JP2024003688A - Imaging element and imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging element and an imaging device for performing local high frame rate imaging or high resolution imaging for each imaging area, reducing artifacts that may occur at boundary portions between areas of different imaging modes, and improving the image quality of an entire screen.

SOLUTION: An imaging device includes an imaging element having a function of grouping adjacent 2×2 pixels each having a photoelectric conversion portion into one pixel block, and performing switching between an operation of binning and reading all charges accumulated in the photoelectric conversion portions of 4 pixels of the pixel block every frame or an operation of sequentially reading out charges corresponding to a plurality of frames accumulated in the photoelectric conversion portion of one pixel of the pixel block pixel by pixel for each frame, and a signal processing unit that performs calculation matched with the mode selection signal for each output signal corresponding to each pixel block and generates a video signal.

SELECTED DRAWING: Figure 1

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an imaging device and an imaging device that acquire video signals.

In recent years, signal processing in imaging devices has been digitized, and output signals from imaging elements have also become digital signals. In order to cope with higher resolution (multiple pixels), the high-speed pixel signal readout circuit of CMOS (Complementary Metal Oxide Semiconductor) image sensors is equipped with an ADC (Analog/Digital Converter) for each column of the pixel array in a column-parallel manner. ADCs are widely used (for example, Patent Document 1).

The signal readout period T _H per pixel row of an image sensor using a column-parallel ADC is inversely proportional to the number of pixels V in the vertical direction (signal readout direction) of the image sensor and the frame rate _ff . Therefore, in order to realize a high-resolution, high-frame-rate image sensor, it is essential to increase the speed of the readout circuit. For example, the signal readout period T _H per row of an 8K/240 fps image sensor using a column-parallel ADC is 1 μs or less. Furthermore, when performing fixed pattern noise removal using digital CDS (correlated double sampling), it is necessary to perform AD conversion of both the pixel reset level and signal level for one readout operation, so it is less than 0.5 μs. High-speed AD conversion is required.

Generally, the operating speed of an ADC is in a trade-off relationship with performance such as noise performance, bit depth, power consumption, and ADC area. In a multi-pixel high-speed image sensor, the time allowed for AD conversion becomes shorter as mentioned above, resulting in noise performance deterioration, insufficient bit depth, increased power consumption, chip There are issues such as an increase in area.

To address this issue, in order to substantially improve image quality with a limited AD conversion speed, image sensors have conventionally been equipped with multiple operation modes specialized for specific imaging performance such as frame rate and resolution. A method has been proposed for selecting an optimal operation mode according to the characteristics of the device (Patent Document 2). For example, select a mode that suits the shooting scene, such as high frame rate/low resolution mode when capturing a fast-moving subject, and low frame rate/high resolution mode when capturing a high-definition subject with little movement. This is intended to substantially improve image quality.

JP2020-188360A Japanese Patent Application Publication No. 2020-141405

However, the conventional mode selection is to set the entire image sensor (imaging area) to an operation mode specialized for specific imaging performance. Even if a mode selection function is provided in an image sensor, it is assumed that in a general photographic scene, a plurality of objects having different characteristics are simultaneously captured on the same photographic screen. In particular, when wide-field imaging such as 360° video is performed, it is assumed that scenes having a plurality of characteristics are mixed at the same time, so this effect becomes significant. At this time, for example, if imaging in a high frame rate mode is selected in accordance with a fast-moving object within the screen, the definition of still, high-definition objects simultaneously reflected within the same screen will be reduced. On the other hand, if high-resolution mode is selected to capture a high-definition object on the screen, a problem arises in that fast-moving objects become blurred and the movement cannot be captured.

Therefore, an object of the present invention, which has been made in view of the above-mentioned problems, is to realize local high frame rate photography or high resolution photography for each imaging area, and to capture images at boundaries between areas with different shooting modes. An object of the present invention is to provide an imaging element and an imaging device that can reduce artifacts that may occur in images and improve the image quality of the entire screen.

In order to solve the above problems, an image sensor according to the present invention includes:
(1) One pixel block is provided for every 2×2 pixels adjacent to each other, and each pixel is provided corresponding to a pixel array constituted by the pixel blocks and a column of the pixel blocks. and a column readout circuit that reads out the output signal of the block, and reads all the charges accumulated in the photoelectric conversion units of the four pixels of the pixel block every frame according to the mode selection signal set for each pixel block. It is characterized by having a function of switching between reading out by binning or sequentially reading out charges for multiple frames accumulated in the photoelectric conversion section of one pixel of the pixel block one pixel at a time for each frame.

In order to solve the above problems, an imaging device according to the present invention includes:
(2) the image sensor and a signal processing unit that performs an operation on the output signal of the image sensor according to the mode selection signal for each output signal corresponding to each pixel block to generate a video signal; It is characterized by being prepared.

(3) In the imaging device of (2) above, the signal processing unit bins and reads out all charges accumulated in the photoelectric conversion units of the four pixels of the pixel block based on the output signal. , generated a video signal in a high-speed shooting mode that outputs a signal every frame, and sequentially read out charges for multiple frames accumulated in the photoelectric conversion section of one pixel of the pixel block one pixel at a time for each frame. It is desirable to generate a video signal in a high-resolution imaging mode with a resolution corresponding to the pixels of the pixel block based on the output signal.

(4) The imaging device of (2) or (3) above includes a prism that separates incident light into each color, a plurality of the imaging elements arranged corresponding to the colors separated by the prism, and the mode a drive signal generation section that outputs a drive signal to the image sensor according to a selection signal, and the signal processing section performs an operation according to the mode selection signal on the output signals of the plurality of image sensors. It is desirable to perform this for each output signal corresponding to each pixel block to generate a video signal.

(5) In the imaging device of (4) above, the prism is a four-plate prism, the four imaging elements are arranged corresponding to Rch, G1ch, G2ch, and Bch, and the drive signal generation section has a high Regarding the pixel blocks in the resolution photography mode, it is desirable that the corresponding pixel blocks of the plurality of image sensors are controlled so that the positions of the output pixels are shifted from each other for each image sensor.

(6) In the imaging device according to (4) or (5) above, the drive signal generation unit determines that, for the pixel blocks in the high-resolution imaging mode, corresponding pixel blocks of the plurality of imaging elements are combined into a Bayer array. It is desirable to perform control to output a signal at a pixel position.

(7) In the imaging device according to any one of (4) to (6) above, the signal processing unit sequentially reads out one pixel per frame of the plurality of image sensors in the high resolution imaging mode. A method of generating a video signal by performing demosaic processing on each frame, and generating a video signal by collecting output signals corresponding to each pixel of the pixel block of the image sensor as a signal at each pixel position every four frames. It is desirable to select one of these methods.

(8) In the imaging device according to any one of (4) to (7) above, the signal processing unit detects a boundary between at least a pixel block in high-speed imaging mode among the pixel blocks in which the high-resolution imaging mode is selected. It is preferable that the pixel blocks of the portion be demosaiced frame by frame to generate a video signal.

According to the image sensor of the present invention, even in an imaging scene where a fast-moving object and a high-definition object are simultaneously captured on the same screen, local high frame rate imaging or high resolution imaging is realized for each imaging area. can. Further, according to the imaging device according to the present invention, it is possible to reduce artifacts that may occur at the boundary between areas in different shooting modes, and improve the image quality of the entire screen.

1 is a configuration example of an imaging device according to an embodiment of the present invention. 1 is a configuration example of an image sensor according to an embodiment of the present invention. This is an example of a circuit configuration of a pixel block of an image sensor. 3 is an example of a timing chart of drive signals in each readout operation of a pixel block. FIG. 3 is a diagram illustrating an example of the operation timing of a certain column of image sensors. It is an example of the output signal of the pixel block corresponding to the feedback signal F. FIG. 3 is a diagram illustrating an example of an output operation of each channel for each frame period. FIG. 2 is a diagram illustrating an example of signal processing using an artifact reduction method. FIG. 2 is a diagram illustrating an example of full resolution signal processing.

Embodiments of the present invention will be described below with reference to the drawings.

(Embodiment)
FIG. 1 is a configuration example of an imaging device according to an embodiment of the present invention. The imaging device includes a lens 10, a four-plate prism 20, a feedback signal (mode selection signal) generation section 30, a drive signal generation section 40, an imaging device 100 for each channel, and a signal processing section 50. Note that the feedback signal generation section 30 may be configured as a device independent of the imaging device.

The lens 10 collects incident light from a subject and guides it to a four-plate prism 20. The four-plate prism 20 separates the incident light into four channels (ch) of R, G1, G2, and B optical signals. G1 and G2 are G signals having the same wavelength band obtained by dividing the G signal into two using a beam splitter or the like. The feedback signal (mode selection signal) generation section 30 generates a feedback signal (mode selection signal) F according to the video. The drive signal generation unit 40 receives the feedback signal F and outputs a drive signal (R, G1, G2, B) according to the feedback signal F to the image sensor 100 of each channel.

Four image sensors 100 are arranged corresponding to the optical signals of each channel of R, G1, G2, and B. As described later, the image sensor 100 of each channel is an image sensor 100 configured in units of 2×2 pixels, and converts a subject image into an image sensor output signal based on a drive signal received from the drive signal generation unit 40. and transmits it to the signal processing section 50. The signal processing unit 50 receives the image sensor output signals (R, G1, G2, B) and the feedback signal F, generates and outputs a video signal.

In this embodiment, four image pickup devices 100 corresponding to 4 channels of R, G1, G2, and B are used using a four-plate prism 20, but the image pickup device of the present invention is designed to acquire color images. Any optical system can be adopted. In addition, in this embodiment, by processing video signals of 4 channels of R, G1, G2, and B, it is possible to utilize signal processing related to the existing Bayer array as described later.

Below, details of each component and the operating principle of the imaging device will be described.

(1) Image sensor FIG. 2 is a configuration example of an image sensor 100 according to an embodiment of the present invention. The image sensor 100 includes a pixel array 110, a row selection circuit 120, a readout column control circuit 130, and a column readout circuit 140.

The pixel array 110 consists of H horizontal x V vertical pixel blocks 111 (horizontal 2H x 2V pixels vertically), and can be controlled for each pixel block 111. In this embodiment, one signal readout line PIXOUT and four readout column control signal lines (TXA to TXD) are vertically wired for each column of pixel blocks. The read column control circuit 130 controls the read column control signal lines (TXA to TXD), and the signal read line PIXOUT is connected to the column read circuit 140 (141, 142, . . . ) corresponding to each column. The column readout circuit 140 reads out signals from each pixel block and performs signal processing such as ADC and digital CDS. In addition, three types of signal lines, row selection SL, transfer gate control TX, and reset RT, are wired horizontally from the row selection circuit 120 for each row of pixel blocks, and the row selection circuit 120 connects these signal lines vertically. The pixel block 111 is scanned and driven row by row.

FIG. 3 is an example of a circuit configuration of the pixel block 111 of the image sensor 100. The pixel block 111 includes four photoelectric conversion units (photodiodes in this embodiment) PD(A) to PD(D) corresponding to four pixels. Four photodiodes PD(A) to PD(D) share a floating diffusion (FD) node. A transfer gate control signal (TX signal) from the row selection circuit 120 is input to the gate electrode of the charge transfer transistor TGT of each photodiode PD via the source/drain terminal of the transfer operation control transistor TGC. Read column control signal lines (TXA to TXD) are connected to the gate electrode of the transfer operation control transistor TGC, and the transfer gate control signal (TX signal) is enabled/disabled for each PD(A) to PD(D). Can be disabled. The potential of the floating diffusion (FD) node is input to the gate electrode of the source follower transistor SFT connected to the power supply PIXVDD. When the selection transistor SLT controlled by the row selection signal (SL signal) becomes conductive, the output signal of the source follower transistor SFT is output to the signal readout line PIXOUT. Note that the reset transistor RTT is controlled by a reset signal (RT signal) and becomes conductive to discharge the charge accumulated in the floating diffusion FD.

FIG. 4 is an example of a timing chart of drive signals in each readout operation of the pixel block 111. The pixel block 111 has five types of read operations. Each timing chart in FIG. 4 corresponding to each readout operation (TXF, TXA, TXB, TXC, TXD) shows the time during which one pixel block row is selected in one frame (1/f _f /V seconds). Each signal is shown.

The row selection signal (SL signal) is common to each readout operation, and is at a level that always makes the selection transistor SLT conductive during the period when a pixel block row is selected (1/f _f /V seconds) (in this embodiment, each transistor The signal that makes it conductive is High.). In addition, the reset signal (RT signal) is common to each readout operation, and becomes High at some timing before the photodiode PD is read out (charge transfer transistor TGT conducts), and the charge of the floating diffusion FD is discharge.

Readout operation TXF is a readout operation used during high-speed shooting (high frame rate shooting), and at a predetermined timing, a transfer gate control signal (TX signal) and four photodiodes PD(A) to PD(D) are read out. The control signals (TXA to TXD) of the transfer operation control transistor TGC all become High. Therefore, in the read operation TXF, the charges of all the photodiodes PD(A) to PD(D) (here, the charges accumulated during one frame period) are read out to the floating diffusion FD. In this way, all charges accumulated in the photoelectric conversion sections of the four pixels of the pixel block 111 can be binned and read out for each frame.

The readout operation TXA is a readout operation used during high-resolution imaging, and at a predetermined timing, a transfer gate control signal (TX signal) and a control signal for the transfer operation control transistor TGC of one photodiode PD(A) are read out. (TXA) becomes High. Therefore, in the read operation TXA, the charge of only the photodiode PD(A) is read out to the floating diffusion FD. Note that during this period, the other photodiodes PD(B) to PD(D) continue to accumulate photocharges. As described later, the read operations TXA to TXD are performed in order, and the read operation TXA is performed every 4 frames, so in this read operation, the charge of the photodiode PD(A) accumulated during the 4 frame period is read out. .

The readout operation TXB is a readout operation used during high-resolution imaging, and at a predetermined timing, a transfer gate control signal (TX signal) and a control signal for the transfer operation control transistor TGC of one photodiode PD(B) are read out. (TXB) becomes High. Therefore, in the read operation TXB, the charge of only the photodiode PD(B) (here, the charge accumulated over four frame periods) is read out to the floating diffusion FD. Note that during this period, the other photodiodes PD(A), PD(C), and PD(D) continue to accumulate photocharges.

The readout operation TXC is a readout operation used during high-resolution imaging, and at a predetermined timing, a transfer gate control signal (TX signal) and a control signal for the transfer operation control transistor TGC of one photodiode PD(C) are read out. (TXC) becomes High. Therefore, in the read operation TXC, the charge of only the photodiode PD(C) (here, the charge accumulated over four frame periods) is read out to the floating diffusion FD. Note that during this period, the other photodiodes PD(A), PD(B), and PD(D) continue to accumulate photocharges.

The readout operation TXD is a readout operation used during high-resolution imaging, and at a predetermined timing, a transfer gate control signal (TX signal) and a control signal for the transfer operation control transistor TGC of one photodiode PD(D) are read out. (TXD) becomes High. Therefore, in the read operation TXD, the charge of only the photodiode PD(D) (here, the charge accumulated over four frame periods) is read out to the floating diffusion FD. Note that during this period, the other photodiodes PD(A) to PD(C) continue to accumulate photocharges.

Immediately after switching the feedback signal from high-speed imaging mode to high-resolution imaging mode, the charge accumulated in each pixel may be less than 4 frame periods, but in the steady state of high-resolution imaging mode, the readout operation TXA to TXD are repeated in order, and charges for four frames are read out from each pixel.

FIG. 5 is an example of the operation timing of a certain column of the image sensor 100. In this figure, for a certain pixel block column (k-th column) of the image sensor 100 of one channel (for example, Rch), high-resolution imaging is performed only for the _m1th row to the _m2th row, and for the other rows. shows an example of operation timing when performing high-speed photography. In FIG. 5, the horizontal axis is the time axis, the vertical axis is the row position, and the diagonal lines indicate which row is operating in which read operation at each time point.

The row selection circuit 120 scans the pixel block from the first row to the Vth row during one frame period T _f =1/f _f (seconds). The control signals TXA to TXD from the read column control circuit 130 differ from frame to frame at a four frame period (f ₁ to f ₄ ) as shown in FIG. That is, in the read operation from m ₁ to m ₂ rows, the first frame (f ₁ ) is the read operation TXA, the second frame (f ₂ ) is the read operation TXB, and the third frame (f ₃ ) is the read operation TXC. , the read operation TXD is performed in the fourth frame (f ₄ ), and the read operation TXF is performed in the read operations of the other rows regardless of the frame.

By performing the above operation, charges for a plurality of frames accumulated in the photoelectric conversion unit of one pixel of the pixel block 111 are sequentially read out one pixel per frame from the m ₁ to m ₂ rows. In other words, a video signal with an exposure time of 4/f _f can be obtained for each charge accumulated in the photodiodes PD(A) to PD(D), resulting in a high-resolution video with a resolution corresponding to each pixel. be done.

In other rows, the charges accumulated in the photodiodes PD(A) to PD(D) are binned, and a video signal with a frame rate f _f is obtained for each pixel block 111, thus realizing high-speed (high frame rate) shooting. be done.

In Figure 5, an example is shown in which images are taken in high resolution mode from _m1 to _m2 rows, but since the operation mode can be arbitrarily specified for each row and can be set independently for each column, two The operation modes of high-resolution photography and high-speed photography can be freely set for each pixel block 111 of ×2 pixels. That is, the image sensor 100 has a function of switching between high-speed imaging and high-resolution imaging in accordance with the feedback signal (mode selection signal) F for each pixel block 111.

(2) Feedback signal (mode selection signal) generation unit The feedback signal (mode selection signal) generation unit 30 outputs a feedback signal F. Note that the feedback signal F is a signal for selecting (designating) the operation mode of each pixel block, and is sometimes referred to as a mode selection signal. The feedback signal F is expressed, for example, in an array of V rows and H columns corresponding to the arrangement of the pixel blocks 111, and determines whether to perform high-speed imaging or high-resolution imaging for each pixel block 111 depending on the value of the array element F _mn . Specify. That is, the operation mode of the pixel block 111 in the mth row and nth column can be specified by the element F _mn . In this embodiment, as an example, high-speed photography is defined when F _mn =0, and high-resolution photography is defined when F _mn =1, but the definition of mode designation is not limited to this.

The method of generating the element F _mn of the feedback signal F is not limited in the present invention. However, in order to improve the effective image quality, it is desirable to capture static objects at high resolution and capture moving objects at high speed. For example, it is desirable to obtain the characteristics (movement) of each region of the image using motion estimation of the subject, and to feed back an appropriate operation mode based on the characteristics to the image sensor 100 as a feedback signal F. Furthermore, in this embodiment, as shown in FIG. 5, in high-resolution imaging, the video signal is acquired using four readout operations TXA to TXD as a set, so the element F _mn is updated once every four frames. It is desirable that

As shown in FIG. 1, the feedback signal (mode selection signal) generation section 30 supplies the same feedback signal F to the drive signal generation section 40 and signal processing section 50 that control the image sensor 100 of each channel. Note that the feedback signal generation section 30 may be provided outside the imaging device. Furthermore, instead of providing the generation unit 30, the feedback signal may be input as an external signal, or may be freely set manually.

Further possible methods for generating the feedback signal F include the following methods (a) to (c).

(a) A value arbitrarily set by the user is input as a fixed value or a value that is updated at a certain update frequency. For example, if the position of a stationary object, a moving object, etc. is fixed on the screen, a feedback signal can be set using the mode of that area as a fixed value.

(b) Determine the area of interest on the screen, focus area, subject movement speed, etc. from the subject's position/location information acquired by the distance sensor and lens parameters (focus position, focal length, f-number), etc., and display the results. A feedback signal is generated in real time based on the

(c) Using image analysis technology such as object recognition, regions are divided for each subject, and an appropriate F _mn is automatically given according to the characteristics of each subject (estimated spatial frequency components and moving speed). For example, by recognizing the subject through image analysis, high-speed photography mode is automatically set for each area if the subject is moving, and high-resolution photography mode is automatically set for each area if the subject is stationary.

Although it is preferable that the feedback signal F be updated once every four frames, an appropriate update period may be selected depending on the moving speed of the subject and the speed of change over time of the photographed scene.

FIG. 6 is an example of the output signal of the pixel block 111 corresponding to the feedback signal F. Here, control using the feedback signal F of one image sensor (for example, Rch) is shown. FIG. 6(a) is an example of the feedback signal (mode selection signal) F. When the element F _mn of the feedback signal F takes a value of 0 or 1, the pixel block 111 of the pixel array 110 is divided into two types of imaging regions with different operation modes.

FIG. 6(b) shows the output signal of each pixel block in each frame (f ₁ to f ₄ ). In the region where the feedback signal F (element F _mn ) is 0, a signal obtained by binning the charges accumulated in PD(A) to PD(D) with an exposure time of 1/ _ff is obtained for each frame, so that the local High frame rate imaging (high-speed shooting) can be achieved. In the area where the element F _mn is 1, a signal at the pixel position shown in the figure is output for each frame. In other words, the charge signals accumulated in each photodiode PD(A) to PD(D) with an exposure time of 4/f _f can be acquired independently for each frame, making it possible to achieve local high-resolution imaging. can.

As described above, according to the image sensor 100 according to the present embodiment, it is possible to specify for each pixel block 111 which performance to give priority to, resolution or frame rate, using the feedback signal (mode selection signal) F. . Therefore, even in a shooting scene where a fast-moving subject and a high-definition subject are captured simultaneously on the same screen, it is possible to improve the image quality of the entire screen by achieving localized high frame rate shooting or high resolution shooting. I can do it.

(3) Drive Signal Generation Unit As shown in FIG. 1, the drive signal generation unit 40 receives a feedback signal (mode selection signal) F and supplies a drive signal (R , G1, G2, B). The drive signals (R, G1, G2, B) control the operation mode (readout operation) of each pixel block 111 of the image sensor 100 of each channel.

The control of each channel to improve the image quality of the entire color image will be described below.

FIG. 7 shows an example of the output operation of each channel for each frame period f ₁ to f ₄ . Here, for a specific pixel block 111, which pixel (photoelectric conversion unit) outputs an output in each frame and the relationship between each channel (R, G1, G2, B) are shown. Since the feedback signal (mode selection signal) F is common to all channels, the pixel blocks 111 at corresponding positions in each channel have the same shooting mode (high-speed shooting or high-resolution shooting). However, in the high-resolution shooting mode, the drive signal generation unit 40 generates drive signals (R, G1, G2, B).

In the pixel block 111 in the high-speed photographing mode, as shown in FIG. 7(a), readout is performed by the readout operation TXF on all channels. That is, for each frame period f ₁ to f ₄ , a signal is output from the pixel blocks 111 of all channels by binning the charges accumulated in PD(A) to PD(D) with an exposure time of 1/ _ff .

In the pixel block 111 in the high resolution shooting mode, as shown in FIG. 7(b), the readout operations TXA to TXD of each channel (R, G1, G2, B) are different in each frame period f ₁ to f ₄ . There is. Therefore, in each frame period f ₁ to f ₄ , charge signals accumulated in photodiodes PD(A) to PD(D) at different positions in each channel (charge signals with exposure time 4/f _f ) are read out. . Under the control of the drive signal generation unit 40, each channel performs read operations TXA to TXD in frame periods f ₁ to f ₄ in which the phases of read operations TXA to TXD are shifted from each other.

In addition, in each frame period f ₁ to f ₄ of the high-resolution shooting mode, the pixel positions of the signals output from each channel are shifted, but the exposure period is the same, and all channels (R, G1, G2, When the output signals of B) are combined, an image equivalent to a Bayer array image is obtained.

(4) Signal Processing Unit The signal processing unit 50 receives the image sensor output signal and the feedback signal (mode selection signal) F of the image sensor 100, and blocks the m-th row and n-th column pixel designated by the element F _mn of the signal F. Based on the operation mode of the image sensor 100, an operation according to the signal F is performed on the output signal corresponding to each pixel block of the image sensor 100, and a video signal is generated. A specific example of signal processing will be described below.

(i) When F _mn = 0 (high-speed shooting) The signal processing unit 50 converts the output signal of the corresponding pixel block (pixel block of m rows and n columns) output from the image sensor 100 of each channel into a high-speed shooting image as it is. Output as a signal. That is, a signal with an exposure time of 1/f _f obtained by binning the charges accumulated in the photoelectric conversion units PD(A) to PD(D) is output as a signal with a resolution of V×H pixels and a frame rate f _f . Note that the signal processing unit 50 can further appropriately convert the signal into a video signal compatible with the display device. For example, the signal processing unit 50 can apply arbitrary up-conversion processing to the output signal of the pixel block in order to display it on a 2V×2H pixel display device.

(ii) When F _mn =1 (high resolution imaging) The image sensor 100 of each channel outputs a signal corresponding to the pixel shown in FIG. 7(b). Therefore, from the pixel block (pixel block of m rows and n columns), the color signal is similar to the image signal obtained when using the Bayer array color filter of a single-chip camera, as shown in FIGS. 8 and 9. signals are acquired with a period of 1/ _ff . The signal processing unit 50 generates a video signal by performing signal processing on the obtained signal by selecting one of the following methods: (a) artifact reduction method or (b) full resolution method. .

(a) Artifact Reduction Method FIG. 8 is a diagram illustrating an example of signal processing of the artifact reduction method. The signal processing unit 50 performs demosaicing processing for each frame on the 4-channel signals corresponding to the Bayer array output in each frame period f ₁ to f ₄ , and the exposure time 4/f _f and the frame A video signal with a rate f _f and a resolution of 2H×2V is generated. Any algorithm can be used for the demosaic processing and is not limited. In this case, the pixel values of all pixels of all channels are estimated by demosaic processing from the Bayer array image (R and B are 1/4 and G is 1/2 of all pixels), so strict high-definition video is not possible. However, instead, a high frame rate video with a frame rate f _f can be obtained. In other words, images obtained using this method are positioned between high-speed photography and high-resolution photography.

Here, the signals of R, G1, G2, and B channels obtained in each frame period have the same exposure period, and there is no deviation in image position even for a moving subject. Therefore, the occurrence of artifacts is reduced in the video obtained by demosaicing the signals of all channels in each frame period.

(b) Full resolution method FIG. 9 is a diagram illustrating an example of signal processing using the full resolution method. The signal processing unit 50 converts the Bayer array signals output during the frame periods f ₁ to f ₄ into photoelectric conversion units PD(A) for each channel (for each R, G1, G2, and B) using a frame memory or the like. By extracting and collecting ~PD(D) signal values, an image with an exposure time of 4/f _f , a frame rate of f _f /4, and a resolution of 2H×2V is generated. The G1 and G2 signals at the same position in the obtained image are averaged to obtain the G signal. Through the above signal processing, a full resolution RGB signal is generated as a video signal. This method allows you to obtain clear, high-resolution images of stationary subjects.

A supplementary explanation will be given regarding the selection of the above two methods. When a video signal is acquired using the full resolution method in (b), there is a difference of 1/f _f in the center position of the exposure time of the photoelectric conversion units PD(A) to PD(D), so the subject may be stationary. This is not a problem if the subject moves even slightly, but if the subject moves even slightly, artifacts due to the exposure time shift and subject movement will occur, causing a problem that will significantly degrade the image quality. For example, when specifying a mode for each region using subject motion estimation, this artifact tends to occur at the boundary between the high-speed shooting mode and the high-resolution shooting mode, where errors in motion estimation tend to occur. In this embodiment, as a countermeasure when the above situation occurs in the signal processing of the method (b), by using the signal processing of the artifact reduction method of (a), it is possible to prevent significant image quality deterioration due to the occurrence of artifacts. Can be suppressed.

The selection of the (a) method and (b) method may be made by inputting a signal to the signal processing unit 50 to select which method, or at least high-speed shooting in the area where the high-resolution shooting mode is selected. The method may be set to be automatically selected such that pixel blocks at the boundary with the mode area are set to (a) the artifact reduction method, and other areas are set to the (b) full resolution method. Furthermore, since the selection of method (a) and method (b) can be made after shooting in the high-resolution shooting mode, for example, the output signal from the image sensor 100 can be saved in a memory etc. A signal processing method that is determined to be preferable may be selected by comparing images generated by the above methods.

As described above, according to the imaging device of the present embodiment, it is possible to achieve local high frame rate shooting or high resolution shooting for each imaging area, and reduce artifacts that may occur at the boundaries between areas with different shooting modes. This can improve the overall image quality of the screen.

(Modified example of embodiment)
The image sensor 100 of this embodiment may be configured as the following modification example.

- Modified example of control unit In Figure 2, the control unit of a pixel block is illustrated as a minimum case of pixels (2 x 2 pixels) that share a floating diffusion FD, but the size of the control unit can be arbitrary. be. For example, if 8×8 pixels (4×4 pixel block) is used as a control unit, the number of readout column control signal lines (TXA to TXD) can be reduced to 1/4. Further, the transfer operation control transistor TGC can be shared by 4×4 pixel blocks, and the number of transistors can be reduced to 1/16, so the pixel area can be significantly reduced.

- Modified Example of Column Readout Circuit In the embodiment shown in FIG. 2, H column readout circuits 140 are provided below the pixel array 110, and the column readout circuits are connected in parallel to the pixel block columns. The number of parallel column readout circuits for pixel block columns is arbitrary. That is, if two column readout circuits (2H in total) are provided corresponding to one pixel block column, the number of pixel blocks 111 processed by one column readout circuit 140 is halved. Also, more column readout circuits can be provided. This allows sufficient processing time for the ADC and improves the performance of the ADC.

Further, in the above embodiments, the configuration and operation of the image sensor and the image capturing apparatus have been described, but the present invention is not limited to this, and may be configured as an image capturing method that generates a video signal. That is, the present invention may be configured as a method of generating a video signal for high-speed imaging or high-resolution imaging using each block of the imaging device according to the data flow shown in FIG.

Although the embodiments described above have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the invention. Therefore, the present invention should not be construed as being limited to the above-described embodiments, and various modifications and changes can be made without departing from the scope of the claims. For example, functions included in each block, each step, etc. described in the embodiments can be rearranged so as not to be logically contradictory, and multiple constituent blocks, steps, etc. can be combined into one, or divided into one. It is possible to do so.

10 Lens 20 Four-plate prism 30 Feedback signal generation unit 40 Drive signal generation unit 50 Signal processing unit 100 Image sensor 110 Pixel array 111 Pixel block 120 Row selection circuit 130 Readout column control circuit 140 Column readout circuit

Claims (8)

A pixel array constituted by the pixel blocks, in which each 2×2 adjacent pixel includes a pixel including a photoelectric conversion unit, and one pixel block is formed of the pixel block;
a column readout circuit that is provided corresponding to the column of the pixel block and reads out an output signal of each pixel block;
Depending on the mode selection signal set for each pixel block, all charges accumulated in the photoelectric conversion units of the four pixels of the pixel block are binned and read out for each frame, or one pixel of the pixel block is binned and read out. An image sensor having a function of switching between sequentially reading out charges for multiple frames, one pixel at a time, for each frame. An image sensor according to claim 1;
a signal processing unit that performs an operation according to the mode selection signal on the output signal of the image sensor for each output signal corresponding to each pixel block, and generates a video signal;
An imaging device comprising: The imaging device according to claim 2,
The signal processing section outputs a signal for each frame based on an output signal read out by binning all the charges accumulated in the photoelectric conversion sections of the four pixels of the pixel block for each frame. generates a video signal of
High-resolution imaging with a resolution corresponding to the pixel of the pixel block is performed based on an output signal that sequentially reads out charges for multiple frames stored in the photoelectric conversion unit of one pixel of the pixel block, one pixel at a time for each frame. An imaging device that generates a mode video signal. The imaging device according to claim 2 or 3,
A prism that separates incident light into each color,
a plurality of the imaging elements arranged corresponding to the colors separated by the prism;
a drive signal generation unit that outputs a drive signal to the image sensor according to the mode selection signal;
Equipped with
The signal processing unit may perform an operation according to the mode selection signal on the output signals of the plurality of image sensors for each output signal corresponding to each pixel block, and generate a video signal. The imaging device according to claim 4,
The prism is a four-plate prism,
The four image sensors are arranged corresponding to Rch, G1ch, G2ch, and Bch,
The drive signal generation unit controls the pixel blocks in the high-resolution imaging mode so that corresponding pixel blocks of the plurality of image sensors output the pixels by shifting the position of the output pixel for each image sensor. . The imaging device according to claim 5,
The drive signal generation unit controls the pixel blocks in the high-resolution imaging mode to output signals at pixel positions where corresponding pixel blocks of the plurality of image sensors collectively form a Bayer array. The imaging device according to claim 6,
The signal processing unit includes a method of generating a video signal by performing demosaic processing on a frame-by-frame basis on output signals sequentially read out one pixel per frame from the plurality of image sensors in the high-resolution shooting mode; An imaging device that selects one of the methods of generating a video signal by collecting output signals corresponding to each pixel of the pixel block of the element as a signal at each pixel position every four frames. The imaging device according to claim 7,
The signal processing unit generates a video signal by performing demosaic processing on a frame-by-frame basis at least at a boundary portion of the pixel block with a pixel block in the high-speed imaging mode among the pixel blocks in which the high-resolution imaging mode is selected. An imaging device that uses a method.

Images