JP2016213740A - Imaging apparatus and imaging system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus, having a wide dynamic range even under high illuminance, enabling high resolution imaging and an imaging system.SOLUTION: The imaging apparatus includes a plurality of pixels arranged over a plurality of rows and a plurality of columns, each of the pixels containing a first pixel, for outputting color information, a second pixel, for outputting brightness information, and a third pixel, having sensitivity different from sensitivity of the second pixel, for outputting brightness information. Each of the first pixel is surrounded by the second pixel and the third pixel.SELECTED DRAWING: Figure 4

Description

  The present invention relates to an imaging apparatus and an imaging system.

  In the solid-state imaging device, a color filter (CF) that transmits light of specific wavelength components (R, G, B) is provided for each pixel in order to obtain a color image. As a color array of color filters, a color filter having a so-called Bayer array is often used.

  On the other hand, in order to improve the sensitivity of the solid-state imaging device, a configuration has been proposed in which the ratio of pixels that easily obtain luminance information is increased. Among them, a solid-state imaging device using white pixels (sometimes referred to as W pixels, White pixels, clear pixels, etc.) that transmit light in the visible light range widely has particularly high sensitivity and a high S / N ratio. An image can be obtained.

  However, white pixels that detect wide wavelength region components have high sensitivity, and therefore are more likely to be saturated than R, G, and B spectral pixels, making it difficult to shoot under high illuminance. This means that the dynamic range is reduced by being easily saturated even under shooting conditions with the same amount of light, and this is common in achieving high sensitivity by detecting non-spectral signals and wide wavelength region component signals. It is a problem.

  In response to this problem, Patent Document 1 proposes a solid-state imaging device having an RBW1W2 array including R and B pixels, which are color pixels, and W pixels having two types of sensitivity. By selecting a low-sensitivity / high-sensitivity luminance signal in accordance with the imaging surface illuminance, an image with a high sensitivity and a wide dynamic range can be captured, and photography can be performed even under high illuminance.

JP 2012-074763 A

  The color filter of the solid-state imaging device described in Patent Document 1 is configured by periodically arranging color pixels R, color pixels B, high-sensitivity white pixels W1, and low-sensitivity white pixels W2. The ratio of the number of pixels of the white pixel W1 and the white pixel W2 is 1: 1. When a color image is actually formed, white information (luminance information) in the color pixel portion is generated by interpolation from pixel values of surrounding white pixels W1 and W2. Then, a ratio between the color data and the interpolated luminance data is obtained in the interpolated pixel, and the ratio is integrated with the pixel value in the surrounding white pixels to calculate the color information of the pixel.

  However, when the white pixel W1 is saturated under high illuminance, it is necessary to calculate the luminance value of the color pixel by interpolation using only the two white pixels W2 adjacent to the color pixel. For this reason, under high illuminance, high-precision interpolation cannot be configured, an interpolation error occurs, and moire occurs.

  An object of the present invention is to provide an imaging apparatus and an imaging system capable of imaging with a wide dynamic range and high resolution even under high illuminance.

  According to an aspect of the present invention, there is provided an imaging device having a plurality of pixels arranged in a plurality of rows and a plurality of columns, wherein the plurality of pixels includes a first pixel that outputs color information, and luminance information. And a third pixel having a sensitivity different from that of the second pixel and outputting luminance information, and each of the first pixels includes the second pixel and the second pixel. There is provided an imaging apparatus characterized by being surrounded by the third pixel.

  According to another aspect of the present invention, the pixel includes a plurality of pixels arranged in a plurality of rows and a plurality of columns, and a readout circuit unit that reads a signal from the plurality of pixels. Includes a first pixel that outputs color information, and a second pixel and a third pixel that output luminance information, and each of the first pixels includes the second pixel and the third pixel. The readout circuit unit reads out a signal from the second pixel after the first accumulation period, and after a second accumulation period different from the first accumulation period. An imaging apparatus is provided that is configured to read a signal from the third pixel.

  According to the present invention, it is possible to realize an imaging device and an imaging system capable of imaging with a wide dynamic range and high resolution even under high illuminance.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. It is a block diagram which shows the structural example of the image pick-up element of the imaging device by 1st Embodiment of this invention. It is a circuit diagram showing an example of composition of an image sensor of an imaging device by a 1st embodiment of the present invention. It is a schematic diagram which shows the color filter arrangement | sequence of the image pick-up element in the imaging device by 1st Embodiment of this invention. It is a schematic diagram which shows an example of the conventional color filter arrangement | sequence. It is the schematic which shows operation | movement of the RGBW12 signal processing part of the imaging device by 1st Embodiment of this invention. It is the schematic which shows the operation | movement in the high precision interpolation part of the imaging device by 1st Embodiment of this invention. It is a graph which shows the relationship between the light quantity in the imaging device by 1st Embodiment of this invention, a signal output, and the resolution. It is a schematic diagram which shows an example of the conventional color filter arrangement | sequence. It is a schematic diagram which shows the color filter arrangement | sequence of the image pick-up element in the imaging device by 2nd Embodiment of this invention. It is the schematic which shows the operation | movement in the high precision interpolation part of the imaging device by 2nd Embodiment of this invention. It is a graph which shows the relationship between the light quantity in the imaging device by 2nd Embodiment of this invention, a signal output, and the resolution. It is a circuit diagram which shows the structural example of the image pick-up element in the imaging device by 3rd Embodiment of this invention. 6 is a timing chart illustrating a method for driving an imaging apparatus according to a third embodiment of the present invention. It is a circuit diagram which shows the structural example of the image pick-up element in the imaging device by 4th Embodiment of this invention. It is a timing chart which shows the drive method of the imaging device by 4th Embodiment of this invention. It is a block diagram which shows the structure of the imaging system by 5th Embodiment of this invention. It is a schematic diagram which shows the color filter arrangement | sequence of the image pick-up element in the imaging device by the modification of embodiment.

[First Embodiment]
An imaging apparatus and a driving method thereof according to a first embodiment of the present invention will be described with reference to FIGS.

  First, a schematic configuration of the imaging apparatus 301 according to the present embodiment will be described with reference to FIGS. 1 to 4. FIG. 1 is a block diagram illustrating a schematic configuration of the imaging apparatus according to the present embodiment. FIG. 2 is a block diagram illustrating a configuration example of the image sensor 101. FIG. 3 is a circuit diagram illustrating a configuration example of the image sensor 101. FIG. 4 is a schematic diagram showing a color filter array used in the image sensor 101.

  As illustrated in FIG. 1, the imaging apparatus 301 according to the present embodiment includes an imaging element 101 and a signal processing unit 102. Note that the image sensor 101 and the signal processing unit 102 may be provided on the same chip, or may be provided on different chips or devices.

  The image sensor 101 converts an optical signal (subject image) incident through an optical system (not shown) into an electrical signal and outputs it. The image sensor 101 is configured by, for example, a so-called single-plate color sensor in which a color filter (hereinafter also referred to as “CF”) is disposed on a CMOS image sensor or a CCD image sensor. An “RGBW12 array” illustrated in FIG. 1 represents a color filter array used in the image sensor 101 of the present embodiment. Details of the RGBW12 array will be described later.

  The signal processing unit 102 is for performing signal processing to be described later on the signal output from the image sensor 101. The signal processing unit 102 includes an RGBW12 signal processing unit 103 and an image processing system unit 106. The RGBW12 signal processing unit 103 includes a pre-processing unit 104 and a high-precision interpolation unit 105.

  The RGBW12 signal processing unit 103 is for processing an output signal from the image sensor 101 having an RGBW12 color filter array. The pre-processing unit 104 appropriately performs preprocessing of signal processing, that is, correction processing such as offset correction and gain correction of each signal, on the output signal from the image sensor 101. The high-precision interpolation unit 105 performs high-precision interpolation processing on the output signal from the pre-stage processing unit 104.

  The image processing system unit 106 uses the output from the RGBW12 signal processing unit 103 to create an output image. Since the image processing system unit 106 is a functional block for creating an RGB color image, it can also be called an RGB signal processing unit. The image processing system unit 106 appropriately performs demosaic processing, color matrix calculation, white balance processing, digital gain, gamma processing, noise reduction processing, and the like in order to make the output from the image sensor 101 a color image. Among these processes, demosaic processing is particularly important for resolution information, and advanced interpolation processing is performed assuming a CF with a Bayer array.

  Next, a configuration example of the image sensor 101 of the imaging device 301 according to the present embodiment will be described with reference to FIGS.

  As illustrated in FIG. 2, the imaging element 101 includes an imaging region 10, a vertical scanning circuit 20, a column amplification unit 40, a horizontal scanning circuit 70, and an output unit 80.

  The imaging area 10 is provided with a pixel array in which a plurality of pixels 12 are arranged in a plurality of rows and a plurality of columns. Each row of the pixel array in the imaging region 10 is provided with a drive signal line 14 extending in the row direction. Each drive signal line 14 is a signal line common to the pixels 12 arranged in the row direction. In addition, each column of the pixel array in the imaging region 10 is provided with a vertical signal line 16 extending in the column direction. Each vertical signal line 16 forms a common signal line for the pixels 12 arranged in the column direction.

  The drive signal line 14 is connected to the vertical scanning circuit 20. As a result, a control signal for controlling the transistor of the pixel 12 to be turned on (conductive state) or off (non-conductive state) from the vertical scanning circuit 20 can be supplied to each drive signal line 14. Further, the column signal amplifier 16 is connected to the vertical signal line 16. A horizontal scanning circuit 70 and an output unit 80 are connected to the column amplification unit 40. As a result, output signals from the plurality of pixels 12 in a predetermined row can be read out for each column. A signal output from the column amplifier 40 via the output unit 80 is input to a signal processing unit 102 (see FIG. 1) outside the image sensor 101, and processing such as analog / digital conversion and correction of input data is performed. .

  As shown in FIG. 3, each pixel 12 includes a photodiode (hereinafter referred to as “PD”) 22, a transfer transistor 24, a reset transistor 26, an amplification transistor 28, and a selection transistor 32. The anode of the PD 22 is connected to a reference voltage line (VSS). The cathode of the PD 22 is connected to the source of the transfer transistor 24. The drain of the transfer transistor 24 is connected to the source of the reset transistor 26 and the gate of the amplification transistor 28. A connection node between the drain of the transfer transistor 24, the source of the reset transistor 26, and the gate of the amplification transistor 28 forms a floating diffusion section (hereinafter referred to as “FD section”). In FIG. 3, the parasitic capacitance of the FD portion is represented by the floating diffusion capacitor 30. The drain of the reset transistor 26 and the drain of the amplification transistor 28 are connected to the power supply voltage line (VDD). The source of the amplification transistor 28 is connected to the drain of the selection transistor 32. The source of the selection transistor 32 that is the output node PDOUT of the pixel 12 is connected to the vertical signal line 16. A current source 18 is connected to the vertical signal line 16.

  The PD 22 is a photoelectric conversion unit that generates and accumulates charges based on incident light by photoelectric conversion. The transfer transistor 24 is a switch for controlling transfer of charge from the PD 22 to the floating diffusion capacitor 30. The transfer transistor 24 is driven by a signal PTX input to the gate, and is turned on when the signal PTX is at a high level (hereinafter referred to as “H level”). The floating diffusion capacitor 30 can hold the charge transferred from the PD 22. The reset transistor 26 is a switch for controlling a reset operation for resetting the potential of the floating diffusion layer 30 to a potential based on the power supply voltage VDD. The transfer transistor 24 is driven by the signal PRES input to the gate, and is turned on when the signal PRES is at the H level. The amplification transistor 28 is for amplifying and outputting a signal based on the signal charge stored in the floating diffusion capacitor 30 connected to the gate. The amplifying transistor 28 is supplied with the power supply voltage VDD at the drain, and supplied with a bias current from the current source 18 via the selection transistor 32 and the vertical signal line 16 at the source, thereby forming a source follower circuit. The selection transistor 32 is a switch for controlling the output of a signal from the amplification transistor 28 to the vertical signal line 16. The selection transistor 32 is driven by the signal PSEL input to the gate, and is turned on when the signal PSEL is at the H level.

  Each drive signal line 14 includes a signal line that supplies a signal PTX, a signal line that supplies a signal PRES, and a signal line that supplies a signal PSEL. The signals PTX, PRES, and PSEL are output from the vertical scanning circuit 20 to the pixel 12 via the drive signal line 14.

  As shown in FIG. 3, the column amplifier 40 includes a column amplifier 42, capacitors C0, C1, CTN, CTS, switches 44, 46, 50, 52, 54, and transistors 56, 58 in each column. . The inverting input terminal of the column amplifier 42 is connected to the vertical signal line 16 via a switch 44 driven by the capacitor C0 and the signal PL. A non-inverting input terminal of the column amplifier 42 is connected to the reference voltage power supply VREF. Between the inverting input terminal and the output terminal of the column amplifier 42, a first feedback path constituted by a series connection body of a switch 46 and a capacitor C1 driven by the signal φC1, and a switch driven by the signal φC A second return path composed of 50 is provided. The capacitor CTN and one main node of the transistor 56 are connected to the output terminal of the column amplifier 42 via the switch 52, and one main node of the capacitor CTS and the transistor 58 is connected via the switch 54, respectively. . Switches 52 and 54 are driven by signals φCTN and φCTS1, respectively. The other main node of the transistor 56 is connected to the horizontal output line 60. The other main node of the transistor 58 is connected to the horizontal output line 62. The horizontal output lines 60 and 62 are connected to the other main nodes of the transistors 56 and 58 in each column, respectively.

  The horizontal scanning circuit 70 outputs a signal φHn to the control node of the transistors 56 and 58 of the column amplifier 40. The output unit 80 includes an output amplifier 82. The output amplifier 82 amplifies and outputs the difference between the signal output to the horizontal output line 60 and the signal output to the horizontal output line 62.

  On each pixel 12 arranged in a two-dimensional matrix, color filters having a predetermined spectral sensitivity characteristic are arranged in the color filter array (hereinafter referred to as “CF array”) shown in FIG. . In FIG. 4, each rectangular area corresponds to one pixel 12. That is, FIG. 4 shows a CF array corresponding to a pixel array of 8 rows × 8 columns. The color filters used in this embodiment include a red filter R, a green filter G, a blue filter B, a white filter W1, and a white filter W2. In the following description, the pixel 12 provided with the red filter R is referred to as “R pixel”, the pixel 12 provided with the green filter G as “G pixel”, and the pixel 12 provided with the blue filter B as “B pixel”. , Respectively. The R pixel, the G pixel, and the B pixel are pixels for mainly outputting color information, and may be referred to as “color pixels” or “RGB pixels”. The pixel 12 provided with the white filter W1 is referred to as “W1 pixel”, and the pixel 12 provided with the white filter W2 is referred to as “W2 pixel”. The W1 pixel and the W2 pixel are pixels for mainly outputting luminance information, and may be referred to as “white pixels”.

  The white pixel is a pixel that directly detects incident light without color separation. The white pixel is characterized by a wide transmission wavelength range in the spectral sensitivity characteristic and high sensitivity compared to the R pixel, G pixel, and B pixel. For example, the half-value width of the transmission wavelength range in the spectral sensitivity characteristic is that of the white pixel. Widest. Typically, the transmission wavelength range in the spectral sensitivity characteristics of the W1 pixel and the W2 pixel includes the transmission wavelength ranges in the high sensitivity characteristics of the R pixel, the G pixel, and the B pixel. The W1 pixel is a pixel that detects the first luminance signal, and the W2 pixel is a pixel that detects the second luminance signal. The W2 pixel is less sensitive to visible light than the W1 pixel.

  In the CF array shown in FIG. 4, a block of 4 rows × 4 columns is the minimum repeating unit. Among the 16 pixels 12 included in the unit block, the ratio of R pixel, G pixel, B pixel, W1 pixel, and W2 pixel is R: G: B: W1: W2 = 1: 2: 1: 8: 4. It has become. A CF array having 12 white pixels in a unit block of 4 rows × 4 columns is referred to as an “RGBW12 array” in this specification. In the RGBW12 array, the ratio of color pixels to white pixels, RGB: W is 1: 3. Compared with the CF array described in Patent Document 1 (see FIG. 9, RGB: W = 1: 1), white The pixel ratio is three times. The characteristics of the RGBW12 array include that all color pixels of the R pixel, G pixel, and B pixel are surrounded by white pixels, and the ratio of white pixels to all pixels is 3/4. The minimum interval between W1 pixels and the minimum interval between W2 pixels are different from each other.

  In other words, the RGBW12 array has color pixels as the first pixels and white pixels as the second pixels, and the total number of second pixel groups is three times the total number of first pixel groups ( 2 times or more). In addition to the effective pixels, the image sensor 101 may include pixels that do not output a signal for forming an image, such as an optical black, a dummy pixel, and a null pixel. These are the first pixels described above. And not included in the second pixel.

  When the RGBW12 array is used, since the color pixels are surrounded by only white pixels, the accuracy in calculating the W value (luminance value) of the color pixel portion is improved. Since the luminance value of the color pixel portion can be interpolated with high accuracy, an image with high resolution can be obtained. Here, a color pixel is surrounded by white pixels. When pixels are schematically shown, all adjacent pixels that touch the sides and vertices of the color pixel are white pixels. Is shown. That is, the four sides and four vertices of the color pixel are in contact with the white pixel W, respectively. For example, taking the G pixel as an example in FIG. 4, the G pixel is in contact with the W1 pixel at four sides and is in contact with the W2 pixel at four vertices. From another viewpoint, when viewed from the color pixel, the right direction, the left direction, the upper direction, the lower direction, the upper right direction, the lower right direction, the upper left direction, and the lower left direction in plan view. It can be said that the white pixels are adjacent to each other. Furthermore, white pixels that are adjacent to the color pixels in the top, bottom, right, and left directions in plan view are diagonally right up, diagonally down right, diagonally up left, and diagonally left. Sensitivity is lower than white pixels adjacent to each of the lower directions. Here, “low sensitivity” means that the signal output from one pixel has a smaller amplitude than the signal output from another pixel when the same amount of light is incident on each pixel. Details will be described later.

  A so-called Bayer array is known as a CF array used when acquiring a color image. As shown in FIG. 5, in the Bayer array, two G pixels are arranged at one diagonal position and two R pixels are arranged at the other diagonal position in a pixel block of 2 rows × 2 columns which is the smallest repeating unit. B pixels are arranged. A predetermined interpolation process is also performed when a color image is formed in a single-plate area sensor using this Bayer array. For example, there is no information about G and B in the R pixel portion. Therefore, the values of G and B in the R pixel portion are interpolated based on information on the G pixel and B pixel around the R pixel. In the Bayer array, the resolution is determined by the largest number of G pixels arranged in a checkered pattern.

  In the RGBW12 array, since the ratio of white pixels that determine resolution is large, the resolution is higher than in the CF array in which pixels that determine resolution are arranged in a checkered pattern, such as the Bayer array described above and the CF array described in Patent Document 1. High image quality can be obtained. That is, it is possible to acquire information having a higher spatial frequency (that is, a smaller pitch) than in the case of a CF array in which pixels that determine resolution are arranged in a checkered pattern. Therefore, an image having a sufficiently high resolution can be obtained simply by obtaining the value of a portion without white pixels (that is, a portion with color pixels) from the average of eight neighboring pixels.

  Although various arrangements of CF are possible, in order to acquire an image with higher resolution in a single-plate image sensor, it is preferable to increase the number of pixels (G pixels in the Bayer array) that mainly create the resolution. In particular, in the Bayer array, G pixels that generate resolution are arranged in a checkered pattern, and interpolation errors may occur. In this regard, since the RGBW12 array has pixels (white pixels) that produce more resolution, the interpolation error can be minimized.

  In the CF array of the image sensor 101 according to the present embodiment, four W1 pixels and four W2 pixels that are less sensitive to visible light than the W1 pixels are provided as white pixels in the unit block. ing. This is to ensure the interpolation accuracy at high illuminance. That is, the white pixel has a higher sensitivity than the color pixel, but has a feature that it is likely to be saturated at high illuminance. By providing white pixels with different sensitivities, even if one white pixel (W1 pixel) is saturated, if the other white pixel (W2 pixel) is not saturated, interpolation is performed using this white pixel (W2 pixel). Processing is possible. In particular, in the CF array of this embodiment shown in FIG. 4, four W2 pixels that interpolate the value of W are arranged around the color pixels, so that interpolation processing can be performed with high accuracy.

  The method of changing the sensitivity between the W1 pixel and the W2 pixel is not particularly limited, and examples thereof include the following methods. The first method is a method of arranging an optical attenuation filter (ND filter: also called a neutral density filter) that absorbs visible light and attenuates each wavelength evenly above the PD 22 of the W2 pixel. It is. Light attenuation filters having different attenuation amounts may be disposed above the W1 pixel and the W2 pixel. In these cases, the filter for the W1 pixel and the filter for the W2 pixel have different transmittances. In the second method, the capacitance value of the floating diffusion capacitor 30 of the W2 pixel is made larger than the capacitance value of the floating diffusion capacitor 30 of the W1 pixel, and the voltage value is reduced when converting the charge from the PD 22 into a voltage. It is. The third method is a method in which the opening for light incidence on the PD 22 of the W2 pixel is made narrower than the opening for light incidence on the PD 22 of the W1 pixel, and the optical signal incident on the PD 22 is reduced. is there. In this case, the opening of the W1 pixel and the opening of the W2 pixel have different aperture ratios. By using these methods to reduce the sensitivity of the W2 pixel compared to the sensitivity of the W1 pixel, it is possible to widen the dynamic range, and it is possible to shoot at high resolution even under high illuminance.

  Next, the driving method of the imaging apparatus according to the present embodiment will be described with reference to FIGS. FIG. 6 is a schematic diagram illustrating the operation of the RGBW12 signal processing unit of the imaging apparatus according to the present embodiment. FIG. 7 is a schematic diagram illustrating the operation of the high-precision interpolation unit of the imaging apparatus according to the present embodiment. FIG. 8 is a graph showing the relationship between the amount of incident light, the signal output, and the resolution in the imaging apparatus according to the present embodiment. 5 and 9 are schematic diagrams showing a conventional color filter array.

  When light enters the PD 22, electrons (charges) are generated by photoelectric conversion. The transfer transistor 24 is turned on when the signal PTX supplied to the gate becomes H level, and transfers the charge generated in the PD 22 to the floating diffusion capacitor 30. The reset transistor 26 is turned on when the signal PRES supplied to the gate becomes H level, and resets the voltage of the floating diffusion capacitor 30 to the reset voltage VDD. By simultaneously turning on the transfer transistor 24 and the reset transistor 26, the potential of the photodiode PD is reset. The selection transistor 32 is turned on when the signal PSEL supplied to the gate becomes H level, and the amplification transistor 28 forms a source follower circuit. As a result, a signal based on the potential of the floating diffusion capacitor 30 is output to the output node PDOUT of the pixel 12.

  The vertical scanning circuit 20 controls the signal levels of the signal PTX, the signal PRES, and the signal PSEL supplied to the pixels 12 to perform vertical scanning, which is a readout operation in units of rows, on the pixels 12 in the imaging region 10. . By the vertical scanning by the vertical scanning circuit 20, a signal based on the reset voltage and a signal based on the charge transferred from the PD 22 to the floating diffusion capacitor 30 are sequentially output from each pixel 12 to the vertical signal line 16 in units of rows. The

  When the signal PL becomes H level and the switch 44 is turned on, the output signal from the pixel 12 is input to the inverting input terminal of the column amplifier 42 via the capacitor C0. By appropriately controlling the switches 46 and 50 by the signal φC1 and the signal φC, the signal input to the inverting input terminal of the column amplifier 42 is amplified with a gain represented by the capacitance ratio of C0 / C1. Is output from the output terminal.

  By turning on the switch 52 in accordance with the timing at which a signal based on the reset voltage is output from the pixel 12, this signal (sometimes referred to as an N signal) is sampled and held in the capacitor CTN. Further, by turning on the switch 54 in accordance with the timing at which a signal based on the voltage when the charge is transferred from the PD 22 to the floating diffusion capacitor 30 is output, this signal (sometimes referred to as S signal) is generated. Sampled and held in the capacitor CTS.

  By sequentially outputting the H level signal φHn from the horizontal scanning circuit 70 for each column, the N signal held in the capacitor CTN and the S signal held in the capacitor CTS are sequentially transferred to the output amplifier 82. The output amplifier 82 amplifies and outputs the difference between the input S signal and N signal, thereby outputting a pixel signal from which a noise component at the time of reset is removed.

  Thereby, the optical signal input into the image pick-up element 101 can be read as an electrical signal.

  The pixel signal output from the image sensor 101 is processed in the signal processing unit 102 according to the flow shown in FIG.

The pixel signal input to the signal processing unit 102 is first input to the upstream processing unit 104 of the RGBW12 signal processing unit 103. The pre-processing unit 104 appropriately performs corrections (pre-processing) such as offset (OFFSET) correction and gain (GAIN) correction of the input signal Din, and creates a corrected output signal Dout (step S101). This process is typically expressed by the following equation.
Dout = (Din−OFFSET) × GAIN

  This correction can be performed in various units. For example, when correction is performed for each pixel 12, correction is performed for each column amplifier 42, correction is performed for each analog-digital conversion (ADC) unit, correction is performed for each output amplifier 80, and the like. By performing the correction, so-called fixed pattern noise can be reduced, and a higher quality image can be obtained.

  The output signal Dout processed by the pre-processing unit 104 is input to the high-precision interpolation unit 105. In the high-precision interpolation unit 105, as shown in FIG. 6, data separation processing (step S102), interpolation processing (step S103), and synthesis processing (step S104) are sequentially performed. In the data separation processing in step S102, the data processed by the pre-processing unit 104 is separated into resolution data Dres and color data Dcol. In the interpolation processing in step S103, interpolation processing is performed on the resolution data Dres. In the combining process in step S104, the resolution data Dint subjected to the interpolation process and the color data Dcol separated in step S102 are combined to generate RGB data Drb.

  The processing in the high-precision interpolation unit 105 will be described more specifically with reference to FIG. FIG. 7A schematically shows output data from a 4 × 4 pixel block, which is the minimum repeating unit of the RGBW12 array. Here, an example in which output data from the pixel block is input to the high-precision interpolation unit 105 and the processes in steps S102 to S104 are performed is described. Actually, output data from all the pixels 12 constituting the imaging region 10 is processed in the same procedure.

  In step S102, the data Dout in FIG. 7A includes resolution data Dres composed of white pixel (W1 pixel and W2 pixel) data and color pixel (R pixel, G pixel, B pixel) data. Separated into color data Dcol. As shown in FIG. 7B, the resolution data Dres after separation includes pixel values (luminances) of four pixels 12 in which color pixels are originally arranged among 16 pixels of 4 rows × 4 columns. Information related data) is unknown (indicated by “?” In the figure). The color data Dcol after separation is data of 4 pixels of 2 rows × 2 columns extracted from 16 pixels of 4 rows × 4 columns, as shown in FIG. This results in low resolution (spatial coarse) data. In FIG. 7D, “Gr” and “Gb” both represent G pixel data. In order to distinguish the data from different G pixels, the notations “Gr” and “Gb” are changed.

  In step S103, interpolation processing is performed on the separated resolution data Dres to compensate for the pixel values of four pixels ("?" In the figure) whose pixel values are unknown. Various methods can be adopted as a method of interpolating pixel values. For example, a method of calculating the average of the surrounding 8 pixels, a method of calculating the average of the upper, lower, left, and right 4 pixels (bilinear method) can be used. Here, as an example, a method for calculating the average of eight surrounding pixels will be described. For convenience of explanation of the interpolation method, FIG. 7C shows the X coordinate and the Y coordinate. For example, a pixel represented as iWb is represented as iWb (3, 3) because it is W data located at the coordinates of (3, 3). In FIG. 7C, “iW” means W data calculated by interpolation, and “r”, “gr”, “gb”, “b” appended to “iW” are This represents the correspondence with the original color pixels.

When both the W1 pixel and the W2 pixel are not saturated, the interpolation data can be obtained using the pixel values of both the W1 pixel and the W2 pixel. At this time, since the sensitivity of the W2 pixel is lower than that of the W1 pixel, the coefficient A (A = (sensitivity of the W1 pixel) / (sensitivity of the W2 pixel) expressed by the ratio between the sensitivity of the W1 pixel and the sensitivity of the W2 pixel. ) Is multiplied by W2 pixel data. For example, pixel data iWb (3, 3) at coordinates (3, 3) is calculated as follows.

On the other hand, if the W2 pixel is not saturated but the W1 pixel is saturated, the pixel value of the W1 pixel can take the maximum value, but is not used as data for interpolation. That is, the pixel data iWb (3, 3) at the coordinates (3, 3) is calculated as follows using only the W2 pixel data.

  In step S103, the interpolated data Dint shown in FIG. 7C and the color data Dcol shown in FIG. 7D are synthesized to generate RGB data Drgb. For example, R color ratio information from the pixel value of the R pixel and the pixel value of iWr, B color ratio information from the pixel value of the B pixel and the pixel value of iWb, the pixel value of the Gr pixel and Gb pixel, and iWgr and iWgb The color ratio information of the G pixel is calculated from the pixel values. Then, using the RGB color ratio information calculated in this way and the pixel values W1, W2, iWr, iWb, iWgr, iWgb of each pixel, RGB data of each pixel is calculated.

  FIG. 8A is a graph showing the relationship between the light amount [lx · sec] and the signal output (after gamma processing) in the white pixel (W1 pixel and W2 pixel). FIG. 8B is a graph showing the relationship between the light amount [lx · sec] and the resolution in the white pixel (W1 pixel and W2 pixel). In the figure, the thick solid line is the case where the RGBW12 array of this embodiment shown in FIG. 4 is used (hereinafter referred to as “Example 1”), and the thin solid line is the case where the CF array shown in FIG. 9 is used ( Hereinafter, it is referred to as a “comparative example”. The amount of light is represented by the product of ambient illuminance [lx] and exposure time [sec], and corresponds to an integrated value of light accumulated by the pixel to output a signal. Here, it is assumed that the imaging elements of Example 1 and the comparative example both have a W1 pixel that is saturated at the light amount (i) and a W2 pixel that is saturated at the light amount (ii). In addition, by performing gamma processing in consideration of these, it is assumed that the dynamic ranges are almost the same.

  Until the incident light amount is less than the light amount (i), that is, until the W1 pixel is saturated, an image can be formed from the data of the W1 pixel and the W2 pixel. Therefore, in the CF array of Example 1, the total number of W1 pixels and W2 pixels used when the amount of light is less than (i) is 12/8 times that of the CF array of the comparative example. For this reason, when the CF array of Example 1 is used, the resolution is improved 12/8 times as compared with the case of using the CF array of the comparative example.

  Since the W1 pixel is saturated when the incident light amount is between the light amount (i) and the light amount (ii), only the data of the W2 pixel is used for interpolation. In the CF array of Example 1 and the CF array of the comparative example, the number of W2 pixels is the same. However, in the CF array of the comparative example, even if the luminance value of the R pixel 401 is to be interpolated from the surrounding white pixels, only the W2 pixel 402 and the W2 pixel 403 positioned above and below are present in the periphery. Even if the luminance value of the B pixel 404 is to be interpolated from the surrounding white pixels, only the W2 pixel 405 and the W2 pixel 406 located on the left and right sides are present in the periphery. This means that pixels that can be interpolated are biased vertically and horizontally depending on the color pixels, and false color due to moire tends to occur, which causes a reduction in resolution. In this regard, in the CF array of the first embodiment, four W2 pixels are arranged around all the color pixels, so that it is possible to improve the interpolation accuracy in both the vertical direction and the horizontal direction. .

  Further, as in the CF array of the first embodiment, if there are interpolation pixels on the top and bottom and the left and right, frequency analysis can be performed in each of the vertical direction and the horizontal direction. As a result, when calculating the interpolation data, it is determined which one of “average of top and bottom”, “average of left and right”, and “average of four pixels” is advantageous in terms of image quality in terms of “noise” and “resolution”. be able to. Thus, the luminance value (W value) of the color pixel portion can be appropriately interpolated, and a good resolution can be obtained. On the other hand, in the CF array of the comparative example, information on only “upper and lower average” when interpolation pixels exist only in the upper and lower sides, and only “left and right average” information when interpolation pixels exist only on the left and right sides. There is no choice but to use. In this case, it is not always possible to perform appropriate interpolation, and good resolution cannot be obtained.

  As described above, according to the present embodiment, it is possible to improve the interpolation accuracy of the luminance value of the color pixel unit and increase the resolution after securing the dynamic range.

[Second Embodiment]
An imaging apparatus and a driving method thereof according to the second embodiment of the present invention will be described with reference to FIGS. Constituent elements similar to those of the image pickup apparatus and its driving method according to the first embodiment shown in FIGS.

  First, the structure of the imaging apparatus 301 according to the present embodiment will be described with reference to FIG. FIG. 10 is a schematic diagram illustrating a color filter array used in the image sensor of the imaging apparatus according to the present embodiment.

  The imaging apparatus 301 according to the present embodiment is the same as the imaging apparatus according to the first embodiment except that the CF array of the imaging element 101 is different. That is, the CF array of the image sensor 101 according to the present embodiment is the same RGBW12 array as the image sensor 101 according to the first embodiment, but the ratio of the W1 pixel and the W2 pixel is reversed. That is, among the 16 pixels 12 included in the unit area, the ratio of R pixel, G pixel, B pixel, W1 pixel, and W2 pixel is R: G: B: W1: W2 = 1: 2: 1: 4: It is eight. Also in the CF array of the image sensor 101 according to the present embodiment, all the color pixels of the R pixel, the G pixel, and the B pixel are surrounded by white pixels, and the ratio of the white pixels to all the pixels becomes 3/4. Yes. Therefore, as in the case of the first embodiment, the sensitivity can be improved by the white pixels, and the accuracy in interpolating the W value of the color pixel portion can be improved.

  Next, the driving method of the imaging apparatus according to the present embodiment will be described with reference to FIGS. FIG. 11 is a schematic diagram illustrating the operation of the high-precision interpolation unit of the imaging apparatus according to the present embodiment. FIG. 12 is a graph showing the relationship between the amount of incident light, the signal output, and the resolution in the imaging apparatus according to the present embodiment. Note that the driving method of the imaging apparatus according to the present embodiment is the same as the driving method of the imaging apparatus according to the first embodiment except that the processing in the high-precision interpolation unit 105 is different.

  FIG. 11A schematically shows output data from a 4 × 4 pixel block, which is the smallest repeating unit of the RGBW12 array. Here, an example in which output data from the pixel block is input to the high-precision interpolation unit 105 and the processes in steps S102 to S104 are performed is described. Actually, output data from all the pixels 12 constituting the imaging region 10 is processed in the same procedure.

  In step S102, the data Dout in FIG. 11A is composed of resolution data Dres composed of data of white pixels (W1 pixel and W2 pixel) and data of color pixels (R pixel, G pixel, B pixel). Separated into color data Dcol. The data Dres for resolution after separation is data (pixel values) of four pixels in which color pixels are originally arranged among 16 pixels of 4 rows × 4 columns, as shown in FIG. 11B. Becomes unknown (indicated by "?" In the figure). The color data Dcol after separation is data of 4 pixels of 2 rows × 2 columns extracted from 16 pixels of 4 rows × 4 columns, as shown in FIG. This results in low resolution (spatial coarse) data.

  In step S103, interpolation processing is performed on the separated resolution data Dres to compensate for the pixel values of four pixels ("?" In the figure) whose pixel values are unknown. Various methods described in the first embodiment can be applied to the method of interpolating the pixel values. Here, as an example, a method of calculating the average of the surrounding eight pixels will be described. For convenience of explanation of the interpolation method, FIG. 11C shows the X coordinate and the Y coordinate. For example, a pixel represented as iWb is represented as iWb (3, 3) because it is W data located at the coordinates of (3, 3). In FIG. 11C, “iW” means W data calculated by interpolation, and “r”, “gr”, “gb”, “b” appended to “iW” are Represents the color of the original color pixel.

When both the W1 pixel and the W2 pixel are not saturated, the interpolation data can be obtained using the pixel values of both the W1 pixel and the W2 pixel. At this time, since the sensitivity of the W2 pixel is lower than that of the W1 pixel, the coefficient A (A = (sensitivity of the W1 pixel) / (sensitivity of the W2 pixel) expressed by the ratio between the sensitivity of the W1 pixel and the sensitivity of the W2 pixel. ) Is multiplied by W2 pixel data. For example, pixel data iWb (3, 3) at coordinates (3, 3) is calculated as follows.

On the other hand, if the W2 pixel is not saturated but the W1 pixel is saturated, the pixel value of the W1 pixel can take the maximum value, but is not used as data for interpolation. That is, the pixel data iWb (3, 3) at the coordinates (3, 3) is calculated as follows using only the W2 pixel data.

  In step S103, the interpolated data Dint shown in FIG. 11C and the color data Dcol shown in FIG. 11D are combined to generate RGB data Drb. For example, R color ratio information from the pixel value of the R pixel and the pixel value of iWr, B color ratio information from the pixel value of the B pixel and the pixel value of iWb, the pixel value of the Gr pixel and Gb pixel, and iWgr and iWgb The color ratio information of the G pixel is calculated from the pixel values. Then, using the RGB color ratio information calculated in this way and the pixel values W1, W2, iWr, iWb, iWgr, iWgb of each pixel, RGB data of each pixel is calculated.

  FIG. 12A is a graph showing the relationship between the light amount [lx · sec] and the signal output (after gamma processing) in the white pixels (W1 pixel and W2 pixel). FIG. 12B is a graph showing the relationship between the light amount [lx · sec] and the resolution in the white pixel (W1 pixel and W2 pixel). In the figure, the thick solid line is the case where the RGBW12 array of this embodiment shown in FIG. 10 is used (hereinafter referred to as “Example 2”), and the thin solid line is the case where the CF array shown in FIG. 9 is used ( Hereinafter, it is referred to as a “comparative example”. Here, it is assumed that the imaging elements of Example 2 and the comparative example both have W1 pixels that are saturated with the light amount (i) and W2 pixels that are saturated with the light amount (ii). In addition, by performing gamma processing in consideration of these, it is assumed that the dynamic ranges are almost the same.

  Until the incident light amount is less than the light amount (i), that is, until the W1 pixel is saturated, an image can be formed from the data of the W1 pixel and the W2 pixel. Therefore, in the CF array of Example 2, the total number of W1 pixels and W2 pixels used when the amount of light is less than (i) is 12/8 times that of the CF array of the comparative example. For this reason, when the CF array of Example 2 is used, the resolution is improved 12/8 times compared to the case of using the CF array of the comparative example.

  Since the W1 pixel is saturated when the incident light amount is between the light amount (i) and the light amount (ii), only the data of the W2 pixel is used for interpolation. Here, the total number of W2 pixels in the CF array of Example 2 is 8/4 times the total number of W2 pixels in the CF array of the comparative example. Further, in the CF array of the comparative example, even if the luminance value of the R pixel 401 is to be interpolated from the surrounding white pixels, only the W2 pixel 402 and the W2 pixel 403 positioned above and below exist in the periphery. Even if the luminance value of the B pixel 404 is to be interpolated from the surrounding white pixels, only the W2 pixel 405 and the W2 pixel 406 located on the left and right sides are present in the periphery. This means that pixels that can be interpolated are biased vertically and horizontally depending on the color pixels, and false color due to moire tends to occur, which causes a reduction in resolution. In this regard, in the CF array of the second embodiment, there are always four W2 pixels around the color pixel. That is, the accuracy of interpolation can be improved both in the vertical direction and in the horizontal direction.

  Therefore, when the incident light quantity is between the light quantity (i) and the light quantity (ii), the CF array of Example 2 can be said to have a resolution that is 8/4 times or higher than that of the CF array of the comparative example.

  As described above, according to the present embodiment, it is possible to improve the interpolation accuracy of the luminance value of the color pixel unit and increase the resolution after securing the dynamic range.

[Third Embodiment]
An imaging apparatus and a driving method thereof according to the third embodiment of the present invention will be described with reference to FIGS. Constituent elements similar to those of the imaging apparatus and the driving method thereof according to the first and second embodiments shown in FIGS.

  FIG. 13 is a circuit diagram illustrating a configuration example of the imaging element of the imaging apparatus according to the present embodiment. FIG. 14 is a timing chart illustrating the driving method of the imaging apparatus according to the present embodiment.

  In the first and second embodiments, the sensitivity of the W2 pixel is made lower than the sensitivity of the W1 pixel by changing the structure of the W1 pixel and the structure of the W2 pixel. However, even if the structures of the W1 pixel and the W2 pixel are the same, a driving method is used. The sensitivity of these pixels can be changed apparently by changing them. In the present embodiment, a driving method of an imaging apparatus that can realize the same effect as in the first and second embodiments when the sensitivities of the W1 pixel and the W2 pixel are the same will be described.

  FIG. 13 shows a circuit configuration of the readout circuit sections of the n-th column and the (n + 1) -th column of the image sensor 101 in the imaging device 301 according to the present embodiment. The circuit configuration of the readout circuit unit in each column is the same as the circuit configuration of the readout circuit unit of the imaging device 101 of the imaging device 301 according to the first embodiment shown in FIG. In the figure, in order to distinguish the constituent elements in the n-th column from the constituent elements in the (n + 1) -th column, “a” is added to the end of the reference numerals of the constituent elements in the n-th column, and the constituent elements in the n + 1-th column. “B” is appended to the end of the reference numeral.

  FIG. 14A shows a timing chart of the entire readout circuit of the image sensor including the readout operation and the reset operation. In FIG. 14A, the horizontal axis indicates time and the vertical axis indicates the row position. In the timing chart of FIG. 14A, it is assumed that signals are sequentially read from the pixels 12a and 12b in the (N + 1) th row from the 0th row to the Nth row. Note that the pixel 12a corresponds to the W1 pixel, and the pixel 12b corresponds to the W2 pixel.

  At time T0, the readout operation from the pixels 12a and 12b in the 0th row starts, and after the readout operation from the pixels 12a and 12b in the 0th row is completed, readout from the pixels 12a and 12b in the first row is completed. Move to operation. Similarly, the read operation from the second row to the Nth row is performed, and the read operation from the pixels 12a and 12b in the Nth row starts from time T3. At time T1, the reset operation of the pixels 12a in the 0th row is performed. Similarly to the readout operation, the reset operation of the pixels 12a is performed in row order, and the reset operation of the pixels 12a in the Nth row is performed at time T5. At time T2, the reset operation of the pixels 12b in the 0th row is performed. Similarly to the readout operation, the reset operation of the pixel 12b is performed in row order, and the reset operation of the pixel 12b in the Nth row is performed at time T6. At time T4, the operation shifts to the reading operation for the next frame, and the same operation is repeated.

  Note that the period from the completion of the reset operation of the pixel 12a to the start of the read operation of the next frame (time T1 to time T4 in the 0th row) is the accumulation period of the pixel 12a (accumulation period 1 in the figure). . Further, the period from the completion of the reset operation of the pixel 12b to the start of the reading operation of the next frame (time T2 to time T4 in the 0th row) is the accumulation period of the pixel 12b (accumulation period 2 in the figure). .

  Next, the read operation from the pixels 12a and 12b will be described more specifically with reference to FIG. Note that the timing chart in FIG. 14B shows a read operation within one horizontal period. In the read operation from the pixels 12a and 12b in the 0th row, the time from the time T0 in FIG. This corresponds to the operation in the period of T1.

  First, at time t0, the vertical scanning circuit 20 sets the signal PRES to the H level and turns on the reset transistors 26a and 26b. As a result, the FD portion is connected to the power supply voltage (VDD) via the reset transistors 26a and 26b, and the potentials of the floating diffusion capacitor 30a of the pixel 12a and the floating diffusion capacitor 30b of the pixel 12b are reset.

  Similarly, at time t0, the vertical scanning circuit 20 sets the signal PSEL to the H level and turns on the selection transistors 32a and 32b. As a result, the amplification transistor 28a outputs a signal based on the reset potential of the floating diffusion capacitor 30a to the vertical signal line 16a via the selection transistor 32a. Similarly, the amplification transistor 28b outputs a signal based on the reset potential of the floating diffusion capacitor 30b to the vertical signal line 16b via the selection transistor 32b.

  Similarly, at time t0, the signal PL is set to H level, and the switches 44a and 44b are turned on. As a result, the vertical signal line 16a is connected to the capacitor C0a, and the vertical signal line 16b is connected to the capacitor C0b.

  Similarly, at time t0, the signals φC1, φC, φCTN, and φCTS are set to the H level, and the switches 46a, 46b, 50a, 50b, 52a, 52b, 54a, and 54b are turned on. As a result, the capacitors C1a, C1b, CTNa, CTNb, CTSa, and CTSb are reset. The potentials of the capacitors CTNa, CTNb, CTSa, and CTSb are VREF.

  Next, at time t1, signals φCTN and φCTS are set to L level, and switches 52a, 52b, 54a, and 54b are turned off. As a result, the reset of the capacitors CTNa, CTNb, CTSa, and CTSb is cancelled.

  Next, at time t2, the vertical scanning circuit 20 sets the signal PRES to the L level and turns off the reset transistors 26a and 26b. As a result, the resetting of the potentials of the floating diffusion capacitors 30a and 30b is released. The floating diffusion capacitors 30a and 30b hold a potential mixed with reset noise (kTC noise). The output signals from the pixels 12a and 12b based on the potentials (reset potentials) of the floating diffusion capacitors 30a and 30b are referred to as N signals.

  Next, at time t3, signal φC is set to L level, and switches 50a and 50b are turned off. Thereby, the reset is released in a state where the column amplifiers 42a and 42b are reset by the N signal. As a result, the column amplifier 42a amplifies the signal from the pixel 12a, and the column amplifier 42b amplifies the signal from the pixel 12b with a gain G1 determined by C0 / C1, and outputs the signal. Further, the potential corresponding to the N signal is clamped to the capacitor C0 by the voltage VREF.

  Next, at time t4, the signal φCTN is set to the H level, and the switches 52a and 52b are turned on. As a result, the output terminal of the column amplifier 42a is connected to the capacitor CTNa, and the output terminal of the column amplifier 42b is connected to the capacitor CTNb.

  Next, at time t5, the signal φCTN is set to the L level, and the switches 52a and 52b are turned off. As a result, signals obtained by amplifying the N signal by the column amplifiers 42a and 42b are sampled and held in the capacitive elements CTNa and CTNb, respectively. At this time, the offsets of the column amplifiers 42a and 42b are simultaneously held.

  Next, in the period from time t6 to time t7, the vertical scanning circuit 20 sets the signals PTXa and PTXb to the H level and turns on the transfer transistors 24a and 24b. As a result, the charge accumulated in the PD 22a is transferred to the floating diffusion capacitor 30a, and the amplification transistor 28a outputs a signal based on the potential of the floating diffusion capacitor 30a to the vertical signal line 16a via the selection transistor 32a. The charge accumulated in the PD 22b is transferred to the floating diffusion capacitor 30b, and the amplification transistor 28b outputs a signal based on the potential of the floating diffusion capacitor 30b to the vertical signal line 16b via the selection transistor 32b. The signals output from the amplification transistors 28a and 28b are signals based on the charges accumulated in the PDs 22a and 22b. A signal output from the amplification transistors 28a and 28b based on the charges accumulated in the PDs 22a and 22b is called an optical signal (sometimes referred to as an S signal). The N signal based on the reset noise of the floating diffusion capacitor 30a and the floating diffusion capacitor 30b is subtracted from the output optical signal by the first CDS circuit.

  Next, in the period from time t8 to time t9, the signal φCTS is set to the H level, and the switches 54a and 54b are turned on. As a result, the signal S1 obtained by amplifying the optical signal output from the column amplifier 42a with the gain G1 is sampled and held in the capacitor CTSa. Further, the signal S1 obtained by amplifying the optical signal output from the column amplifier 42b with the gain G1 is sampled and held in the capacitor CTSb.

  Next, at time t10, the vertical scanning circuit 20 sets the signal PSEL to the L level and turns off the selection transistors 32a and 32b, thereby disconnecting the pixels 12a and 12b from the vertical signal lines 16a and 16b. Further, the input of the column amplifiers 42a and 42b is disconnected by setting the signal PL to the L level and turning off the switches 44a and 44b. Further, the signal φC1 is set to L level, the switches 46a and 46b are turned off, and the amplification operation of the column amplifiers 42a and 42b is stopped.

  Next, during a period from time t11 to time t12, the horizontal scanning circuit 70 performs an operation of sequentially outputting the signal φHn to each column of the column amplification unit 40, that is, horizontal scanning. As a result, the output amplifier 82 sequentially outputs signals based on the signals held in the capacitors CTNa, CTNb, CTSa, and CTSb to the outside. Here, the output signal of the output amplifier 82 is subtracted from the offset component in the gain G1 of the column amplifiers 42a and 42b by the second CDS circuit.

  Next, the reset operation of the pixel 12a will be described more specifically with reference to FIG. Note that the timing chart of FIG. 14C shows the reset operation within one horizontal period, and the reset operation of the pixels 12a in the 0th row is executed at the timing of time T1 in FIG. The

  First, at time t20, the vertical scanning circuit 20 sets the signal PRES to H level and turns on the reset transistor 26a. As a result, the FD portion is connected to the power supply voltage (VDD) via the reset transistor 26a, and the potential of the floating diffusion capacitor 30a is reset. As a result, the FD portion is connected to the power supply voltage (VDD) via the reset transistors 26a and 26b, and the potentials of the floating diffusion capacitor 30a of the pixel 12a and the floating diffusion capacitor 30b of the pixel 12b are reset.

  Next, in the period from time t21 to time t22, the vertical scanning circuit 20 sets the signal PTXa to H level and turns on the transfer transistor 24a. As a result, the cathode of the PD 22a is reset to the same potential as the floating diffusion capacitor 30a, that is, the voltage VDD.

  Next, at time t23, the vertical scanning circuit 20 sets the signal PRES to L level and turns off the reset transistor 26a. As a result, the reset state is released.

  Other signals, that is, signals PSEL, PL, φC, φC1, φCTN, φCTS, and φHn are maintained at the L level during the reset period from time t20 to time t23. The signal PTXb is also maintained at the L level during the reset period from the time t20 to the time t23, and the reset operation of the pixel 12b is not performed.

  The reset operation of the pixel 12b is the same as the reset operation of the pixel 12a except that the reset operation of the pixel 12b in the 0th row is executed at the timing of time T2 in FIG. In the reset operation of the pixel 12b, the signal PTXb is set to the H level and the signal PTXa is set to the L level during the period from the time t21 to the time t22.

  The W1 pixel described in the first and second embodiments is configured by the pixel 12a of the present embodiment, and the W2 pixel described in the first and second embodiments is configured by the pixel 12b of the present embodiment. And the effect similar to 2nd Embodiment can be acquired. In this case, the coefficient A represented by the ratio between the sensitivity of the W1 pixel and the sensitivity of the W2 pixel may be replaced with a coefficient represented by the ratio of the accumulation period 1 and the accumulation period 2.

  As described above, according to the present embodiment, it is possible to improve the interpolation accuracy of the luminance value of the color pixel unit and increase the resolution after securing the dynamic range.

[Fourth Embodiment]
An imaging apparatus and a driving method thereof according to the fourth embodiment of the present invention will be described with reference to FIGS. 15 and 16. Constituent elements similar to those of the image pickup apparatus and its driving method according to the first to third embodiments shown in FIGS. 1 to 14 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  FIG. 15 is a circuit diagram illustrating a configuration example of the imaging element of the imaging apparatus according to the present embodiment. FIG. 16 is a timing chart illustrating the driving method of the imaging apparatus according to the present embodiment.

  As shown in FIG. 15, the image pickup device 101 in the image pickup device 301 according to the present embodiment is different from the circuit configuration of the pixels 12a and 12b in that the readout circuit unit of the image pickup device of the image pickup device according to the third embodiment shown in FIG. The circuit configuration is the same. That is, in the image sensor 101 of the present embodiment, the transistor 34a driven by the signal Cexa is provided between the reset transistor 26a of the pixel 12a and the floating diffusion capacitor 30a. Further, a transistor 34b driven by the signal Cexb is provided between the reset transistor 26b of the pixel 12b and the floating diffusion capacitor 30b. The transistors 34a and 34b are provided with capacitors 36a and 36b, respectively. The capacitors 36a and 36b are configured to be connected to the floating diffusion capacitors 30a and 30b, respectively, when the transistors 34a and 34b are turned on.

  FIG. 16A shows a timing chart of the entire readout circuit of the image sensor including the readout operation and the reset operation. In FIG. 16A, the horizontal axis indicates time, and the vertical axis indicates the row position. In the timing chart of FIG. 14A, it is assumed that signals are sequentially read from the pixels 12a and 12b in the (N + 1) th row from the 0th row to the Nth row.

  At time T0, the readout operation from the pixels 12a and 12b in the 0th row starts, and after the readout operation from the pixels 12a and 12b in the 0th row is completed, readout from the pixels 12a and 12b in the first row is completed. Move to operation. Similarly, the read operation from the second row to the Nth row is performed, and the read operation from the pixels 12a and 12b in the Nth row starts from time T2. At time T1, the reset operation of the pixels 12a and 12b in the 0th row is performed. Similarly to the readout operation, the reset operation of the pixels 12a and 12b is performed in the row order, and the reset operation of the pixels 12a in the Nth row is performed at time T4. At time T3, the operation shifts to the reading operation for the next frame, and the same operation is repeated. Note that the period from the completion of the reset operation of the pixels 12a and 12b to the start of the read operation of the next frame (time T1 to time T3 in the 0th row) is the accumulation period of the pixels 12a and 12b.

  Next, the read operation from the pixels 12a and 12b will be described more specifically with reference to FIG. Note that the timing chart in FIG. 16B shows a read operation within one horizontal period, and in the read operation from the pixels 12a and 12b in the 0th row, the time from the time T0 in FIG. This corresponds to the operation in the period of T1.

  First, at time t0, the vertical scanning circuit 20 sets the signal PRES to the H level and turns on the reset transistors 26a and 26b. Further, the signals Cexa and Cexb are set to the H level, and the transistors 34a and 34b are turned on. As a result, the potentials of the floating diffusion capacitor 30a and the capacitor 36a of the pixel 12a are reset. Further, the potentials of the floating diffusion capacitor 30b and the capacitor 36b of the pixel 12b are reset.

  Similarly, at time t0, the vertical scanning circuit 20 sets the signal PSEL to the H level and turns on the selection transistors 32a and 32b. As a result, the amplification transistor 28a outputs a signal based on the reset potentials of the floating diffusion capacitor 30a and the capacitor 36a to the vertical signal line 16a via the selection transistor 32a. Similarly, the amplification transistor 28b outputs a signal based on the reset potentials of the floating diffusion capacitor 30b and the capacitor 36b to the vertical signal line 16b via the selection transistor 32b.

  Similarly, at time t0, the signal PL is set to H level, and the switches 44a and 44b are turned on. As a result, the vertical signal line 16a is connected to the capacitor C0a, and the vertical signal line 16b is connected to the capacitor C0b.

  Similarly, at time t0, the signals φC1, φC, φCTN, and φCTS are set to the H level, and the switches 46a, 46b, 50a, 50b, 52a, 52b, 54a, and 54b are turned on. As a result, the capacitors C1a, C1b, CTNa, CTNb, CTSa, and CTSb are reset. The potentials of the capacitors CTNa, CTNb, CTSa, and CTSb are VREF.

  Next, at time t1, signals φCTN and φCTS are set to L level, and switches 52a, 52b, 54a, and 54b are turned off. As a result, the reset of the capacitors CTNa, CTNb, CTSa, and CTSb is cancelled.

  Next, at time t2, the vertical scanning circuit 20 sets the signal PRES and the signal Cexa to the L level, and turns off the reset transistors 26a and 26b and the transistor 34a. In the pixel 12a, the signal Cexa is at the L level, and the potential where the reset noise (kTC noise) is mixed in the floating diffusion capacitor 30a is held. In the pixel 12b, since the signal Cexb is at the H level, a potential where reset noise (kTC noise) is mixed is held in the combined capacitance of the capacitance 36b and the floating diffusion capacitance 30b under the gate of the transistor 34b. Note that an output signal of the pixel 12a based on the potential of the floating diffusion capacitor 30a and an output signal from the pixel 12b based on the combined capacitance of the floating diffusion capacitor 30b and the capacitor 36b are referred to as an N signal.

  Next, at time t3, signal φC is set to L level, and switches 50a and 50b are turned off. Thereby, the reset is released in a state where the column amplifiers 42a and 42b are reset by the N signal. As a result, the column amplifier 42a amplifies the signal from the pixel 12a, and the column amplifier 42b amplifies the signal from the pixel 12b with a gain G1 determined by C0 / C1, and outputs the signal. In addition, the potential corresponding to the N signal is clamped to the capacitor C0 by Vref.

  Next, at time t4, the signal φCTN is set to the H level, and the switches 52a and 52b are turned on. As a result, the output terminal of the column amplifier 42a is connected to the capacitor CTNa, and the output terminal of the column amplifier 42b is connected to the capacitor CTNb.

  Next, at time t5, the signal φCTN is set to the L level, and the switches 52a and 52b are turned off. As a result, signals obtained by amplifying the N signal by the column amplifiers 42a and 42b are sampled and held in the capacitive elements CTNa and CTNb, respectively. At this time, the offsets of the column amplifiers 42a and 42b are simultaneously held.

  Next, in the period from time t6 to time t7, the vertical scanning circuit 20 sets the signals PTXa and PTXb to the H level and turns on the transfer transistors 24a and 24b. As a result, the charge accumulated in the PD 22a is transferred to the floating diffusion capacitor 30a, and the amplification transistor 28a outputs a signal based on the potential of the floating diffusion capacitor 30a to the vertical signal line 16a via the selection transistor 32a. The charge accumulated in the PD 22b is transferred to the combined capacitance of the floating diffusion capacitance 30b and the capacitance 36b, and the amplification transistor 28b sends a signal based on the potential of the combined capacitance to the vertical signal line 16b via the selection transistor 32b. Output. The signals output from the amplification transistors 28a and 28b are signals based on the charges accumulated in the PDs 22a and 22b. A signal output from the amplification transistors 28a and 28b based on the charges accumulated in the PDs 22a and 22b is called an optical signal. From the output optical signal, the first CDS circuit subtracts the N signal based on the reset noise of the floating diffusion capacitor 30a and the combined capacitance of the floating diffusion capacitor 30b and the capacitor 36b.

  Next, in the period from time t8 to time t9, the signal φCTS is set to the H level, and the switches 54a and 54b are turned on. As a result, the signal S1 obtained by amplifying the optical signal output from the column amplifier 42a with the gain G1 is sampled and held by the capacitor CTSa. Further, the signal S1 obtained by amplifying the optical signal output from the column amplifier 42b with the gain G1 is sampled and held in the capacitor CTSb.

  Next, at time t10, the vertical scanning circuit 20 sets the signal PSEL to the L level and turns off the selection transistors 32a and 32b, thereby disconnecting the pixels 12a and 12b from the vertical signal lines 16a and 16b. Further, the input of the column amplifiers 42a and 42b is disconnected by setting the signal PL to the L level and turning off the switches 44a and 44b. Further, the signal φC1 is set to L level, the switches 46a and 46b are turned off, and the amplification operation of the column amplifiers 42a and 42b is stopped. Further, the signal Cexb is set to L level to turn off the transistor 34b, thereby preparing for the read operation of the next frame.

  Next, during a period from time t11 to time t12, the horizontal scanning circuit 70 performs an operation of sequentially outputting the signal φHn to each column of the column amplification unit 40, that is, horizontal scanning. As a result, the output amplifier 82 sequentially outputs signals based on the signals held in the capacitors CTNa, CTNb, CTSa, and CTSb to the outside. Here, the output signal of the output amplifier 82 is subtracted from the offset component in the gain G1 of the column amplifiers 42a and 42b by the second CDS circuit.

  Next, the reset operation of the pixels 12a and 12b will be described more specifically with reference to FIG. Note that the timing chart shown in FIG. 16C shows a reset operation in one horizontal period. In the reset operation of the pixels 12a and 12b in the 0th row, the timing at time T1 in FIG. Executed.

  First, at time t20, the vertical scanning circuit 20 sets the signal PRES to the H level and turns on the reset transistors 26a and 26b.

  Next, in the period from time t21 to time t22, the vertical scanning circuit sets the signals Cexa, Cexb, PTXa, and PTXb to the H level, and turns on the transistors 34a and 34b and the transfer transistors 24a and 24b. As a result, the cathodes of the PDs 22a and 22b, the floating diffusion capacitors 30a and 30b, and the capacitors 36a and 36b are reset to the voltage VDD.

  Next, at time t23, the vertical scanning circuit 20 sets the signal PRES to the L level and turns off the reset transistors 26a and 26b. As a result, the reset state is released.

  Other signals, that is, signals PSEL, PL, φC, φC1, φCTN, φCTS, and φHn are maintained at the L level during the reset period from time t20 to time t23.

  In the accumulation time shown in FIG. 16A, the same charge is accumulated in the PD 22a of the pixel 12a and the PD 22b of the pixel 12b. At the time of charge transfer, the pixel 12a is converted to a potential corresponding to the floating diffusion capacitor 30a. On the other hand, in the pixel 12b, the potential is converted into a potential corresponding to the combined capacitance of the floating diffusion capacitor 30b and the capacitor 36b. When the voltage stored in the photodiode is converted into a voltage, it is determined by the integrated value of capacitance and voltage. Therefore, since the magnitude of the voltage value is inversely proportional to the magnitude of the capacitance, the signal value of the pixel 12a is larger than the signal value of the pixel 12b.

  Therefore, by configuring the W1 pixel described in the first and second embodiments with the pixel 12a of the present embodiment and configuring the W2 pixel described in the first and second embodiments with the pixel 12b of the present embodiment, The same effect as in the first and second embodiments can be obtained.

  As described above, according to the present embodiment, it is possible to improve the interpolation accuracy of the luminance value of the color pixel unit and increase the resolution after securing the dynamic range.

[Fifth Embodiment]
An imaging system according to the fifth embodiment of the present invention will be described with reference to FIG. FIG. 17 is a block diagram illustrating a schematic configuration of the imaging system according to the present embodiment. Constituent elements similar to those of the imaging device according to the first to fourth embodiments shown in FIGS. 1 to 16 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  The imaging system 300 according to the present embodiment is not particularly limited, but can be applied to, for example, a digital still camera, a digital camcorder, a surveillance camera, and the like. FIG. 17 shows a block diagram when the imaging apparatus according to the first to fourth embodiments is applied to a digital still camera as an example of the imaging system.

  The imaging system 300 includes a lens 302 that forms an optical image of a subject on the imaging device 301, a barrier 303 for protecting the lens 302, and a diaphragm 304 for adjusting the amount of light passing through the lens 302. In addition, the imaging system includes an output signal processing unit 305 that processes an output signal output from the imaging device 301. As the imaging device 301, the imaging device described in the first to fourth embodiments is used.

  The output signal processing unit 305 includes a digital signal processing unit, and performs an operation of outputting a signal after performing various corrections and compressions on the signal output from the imaging device 301 as necessary. When the signal output from the imaging device 301 is an analog signal, the output signal processing unit 305 may include an analog / digital conversion circuit before the digital signal processing unit.

  In addition, the imaging system 300 includes a buffer memory unit 306 for temporarily storing image data, and a storage medium control interface (I / F) unit 307 for performing recording or reading on a recording medium. Further, the imaging system includes a recording medium 309 that is detachable or built in the imaging system, such as a semiconductor memory for recording or reading imaging data. The imaging system 300 further includes an external interface (I / F) unit 308 for communicating with an external computer and the like, and an overall control / calculation unit 310 for controlling various calculations and the entire digital still camera. Furthermore, the imaging system 300 includes a timing generation unit 311 that outputs various timing signals to the output signal processing unit 305. Note that a control signal such as a timing signal may be input from the outside instead of the timing generator 311. In other words, the imaging system 300 may include at least the imaging device 301 and the output signal processing unit 305 that processes the output signal output from the imaging device 301.

  In this way, by configuring the imaging system 300 to which the imaging device according to the first to fourth embodiments is applied, a high-performance imaging system that can acquire a high-quality image with a wide dynamic range can be realized. .

[Modified Embodiment]
The present invention is not limited to the above embodiment, and various modifications can be made.

  For example, in the above-described embodiment, the image sensor 101 in which the signal charge generated by the PD 22 is an electron has been described as an example, but the same applies to the case of the image sensor 101 in which the signal charge generated by the PD 22 is a hole. be able to. In addition, the circuit configurations of the pixel 12 and the column amplification unit 40 shown in the above embodiment are typical, and can be changed as appropriate.

  In the above embodiment, an example in which the color pixel is configured by the R pixel, the G pixel, and the B pixel has been described. However, the color pixel may be configured by the C pixel, the M pixel, and the Y pixel. In addition, it is not always necessary to configure with three types of color pixels, and it may be configured with arbitrary plural types of color pixels.

  In the first and second embodiments, the sensitivity of these pixels is changed by changing the sensitivity of the W1 pixel and the structure of the W2 pixel. In the third and fourth embodiments, although the structures of the W1 pixel and the W2 pixel are the same, an example is shown in which the sensitivity of these pixels is apparently changed by changing these driving methods. However, these may be combined as appropriate to change the sensitivity of the W1 pixel and the sensitivity of the W2 pixel.

  Further, in order to provide a difference between the output signals of the W1 pixel and the W2 pixel having the same sensitivity, the accumulation time is changed in the third embodiment, and the capacitance value of the capacitance when the charge amount is converted into the voltage in the fourth embodiment. Changed. However, the means for realizing the difference between the output signals is not limited to these. For example, the gains of the amplifiers 42a and 42b may be changed by changing the sizes of the capacitors C1a and C1b (or the capacitors C0a and C0b) in FIG.

  In the fourth embodiment, the capacitors 36a and 36b that load the floating diffusion capacitors 30a and 30b are provided between the reset transistors 26a and 26b and the FD portion. However, the reset transistors 26a and 26b and the FD portion are not necessarily provided. It is not necessary to provide between. For example, a capacitor may be connected to the FD unit via a switch.

  In the first to fourth embodiments, the case where the image sensor 101 has a grid-like pixel array has been described as an example. However, the pixel array of the image sensor 101 does not necessarily have a grid pattern. For example, in a pixel having a honeycomb structure as shown in FIG. 18A, the same effect can be obtained by surrounding the color pixel with the W1 pixel and the W2 pixel. Also, as shown in FIG. 14B, the same effect can be obtained by surrounding the color pixels with W1 pixels and W2 pixels even in a clear bit array obtained by rotating the grid-like pixel arrangement by 45 degrees. Can do.

  The imaging system to which the imaging device according to the first to fourth embodiments can be applied is not limited to the imaging system having the configuration shown in FIG.

  The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

DESCRIPTION OF SYMBOLS 10 Image pick-up area 12 Pixel 14 Drive signal line 16 Vertical signal line 18 Current source 20 Vertical scanning circuit 22 Photodiode 24 Transfer transistor 26 Reset transistor 28 Amplification transistor 30 Floating diffusion capacity 32 Selection transistor 40 Column amplification part 42 Column amplifiers 44, 46, 50, 52, 54 Switch 56, 58 Transistor 60, 62 Horizontal output line 70 Horizontal scanning circuit 80 Output unit 82 Output amplifier 101 Imaging element 102 Signal processing unit 104 Pre-processing unit 105 High-precision interpolation unit 106 Image processing system unit 301 Imaging device

Claims (16)

An imaging apparatus having a plurality of pixels arranged in a plurality of rows and columns,
The plurality of pixels are:
A first pixel that outputs color information;
A second pixel that outputs luminance information;
A third pixel that has a sensitivity different from that of the second pixel and outputs luminance information;
Each of the first pixels is surrounded by the second pixel and the third pixel. An imaging apparatus, wherein: Each of the plurality of pixels includes a photoelectric conversion unit, a capacitor that holds the signal charge generated by the photoelectric conversion unit, and an amplification unit that amplifies and outputs a signal based on the signal charge held in the capacitor Have
The imaging device according to claim 1, wherein the capacitance of the second pixel and the capacitance of the third pixel have different capacitance values. Each of the second pixel and the third pixel includes a photoelectric conversion unit and a filter provided on the photoelectric conversion unit,
The imaging device according to claim 1, wherein the filter of the second pixel and the filter of the third pixel have different transmittances. Each of the second pixel and the third pixel has a photoelectric conversion unit, and an opening for light incidence provided on the photoelectric conversion unit,
The imaging apparatus according to claim 1, wherein an aperture ratio of the second pixel and an aperture ratio of the third pixel are different from each other. A plurality of pixels arranged across a plurality of rows and columns;
A readout circuit unit that reads out signals from the plurality of pixels,
The plurality of pixels are:
A first pixel that outputs color information;
A second pixel and a third pixel that output luminance information;
Each of the first pixels is surrounded by the second pixel and the third pixel,
The readout circuit unit reads a signal from the second pixel after a first accumulation period, and reads a signal from the third pixel after a second accumulation period different from the first accumulation period. It is comprised as follows. The imaging device characterized by the above-mentioned. Each of the plurality of pixels amplifies a photoelectric conversion unit, a capacitor that holds the charge generated by the photoelectric conversion unit, a reset unit that resets the capacitor, and a signal based on the charge held in the capacitor And output the amplification unit,
The solid-state imaging according to claim 5, wherein the readout circuit unit is configured to reset the capacitance of the second pixel and the capacitance of the third pixel at different timings. apparatus. The all pixels with which each said 1st pixel touches by a side or a vertex are said 2nd pixel or said 3rd pixel. The one of Claim 1 thru | or 6 characterized by the above-mentioned. Imaging device. Around each of the first pixels, four second pixels that are in contact with the first pixel at the sides and four third pixels that are in contact with the first pixel at the apex are arranged. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. 9. The imaging device according to claim 1, wherein the total number of the second pixels and the third pixels is twice or more the total number of the first pixels. The imaging apparatus according to claim 1, wherein a minimum interval between the second pixels and a minimum interval between the third pixels are different from each other. Based on the luminance information output from the second pixel adjacent to the first pixel and / or the luminance information output from the signal of the third pixel adjacent to the first pixel, The imaging apparatus according to claim 1, further comprising a signal processing unit that generates luminance information at the position of one pixel. The imaging apparatus according to claim 1, comprising a plurality of types of the first pixels that output the color information of different colors. The said 2nd pixel and the said 3rd pixel output the brightness | luminance information which included the wavelength range of the color corresponding to the said color information which the said several types of said 1st pixel outputs. 12. The imaging device according to 12. The imaging device according to claim 12 or 13, wherein the plurality of types of the first pixels include an R pixel, a G pixel, and a B pixel. The imaging device according to claim 12 or 13, wherein the plurality of types of the first pixels include C pixels, M pixels, and Y pixels. The imaging device according to any one of claims 1 to 15,
An imaging system comprising: an output signal processing unit that processes a signal output from the imaging device.

Images