JP4262290B2 - Image sensor, image processing apparatus, image processing system, and storage medium - Google Patents

Description

  An image sensor for obtaining an image signal from subject light through a color filter array, an image processing apparatus using the image sensor, an image processing system, and a processing step for executing operations of the image sensor The present invention relates to a storage medium storing the program.

Conventionally, there is a color imaging device that generates a color image signal from an imaging device (solid-state imaging device: CCD or the like, hereinafter simply referred to as “imaging device”). In this color imaging apparatus, subject light is incident on a light receiving surface in which a plurality of photodiodes (hereinafter also referred to as “pixels” or “photodetecting elements”) in the imaging element are arranged via a color filter (filter array). It is made so that.
Types of color filters include primary color filters and complementary color filters. The primary color filter has a configuration in which three primary color filters of red (R), green (G), and blue (B) are arranged in a predetermined order corresponding to each pixel on the light receiving surface. Similarly, the complementary color filters have a configuration in which four color filters of complementary colors of cyan (Cy), yellow (Ye), magenta (Mg), and green (G) are arranged in a predetermined order.

  Here, the color filter used for the image sensor is, for example, the above-described complementary color filter, and a signal obtained by the image sensor, processing using the signal, and the like will be described below.

First, in the complementary color filter, the cyan color filter blocks only red light in the visible light range, and the yellow color filter blocks only blue light in the visible light range. The magenta color filter blocks only green light in the visible light range, and the green color filter transmits only green light.
The arrangement of these color filters will be described later.

  Therefore, in an image sensor in which subject light is incident through such a complementary color filter, the pixel signal Cy corresponding to the amount of light received by the pixel through which light is incident through the cyan color filter is transmitted via the yellow color filter. Pixel signal Ye corresponding to the amount of light received by the pixel to which light is incident, pixel signal Mg corresponding to the amount of light received by the pixel through which the light having passed through the magenta color filter, and light having passed through the green color filter are incident. A pixel signal G corresponding to the received light amount of the pixel is obtained.

The pixel signals Cy, Ye, Mg, and G obtained by the imaging device as described above are, for example, signals for iris and white balance adjustment and autofocus detection in a digital still camera, or compression processing in an image processing device. Although used for expansion processing, etc., it is used in the form of a luminance signal Y and color difference signals CB and CR.
The luminance signal Y and the color difference signals CB, CR have pixel signals Cy, Ye, Mg, G and are expressed by the following equations (1) to (3).
Y = Ye + G + Cy + Mg (1)
CB = (G + Ye)-(Mg + Cy) (2)
CR = (Cy + G) − (Ye + Mg) (3)

FIG. 23 and FIG. 24 show an example of the color filter array pattern in the complementary color filter used for the image sensor.
The complementary color filter having the pattern shown in FIG. 23 has two pixels in the horizontal direction (directions C1, C2, C3,... In the figure) and the vertical direction (directions L1, L2, L3,... In the figure). Color filters are arranged so as to have a periodicity of 4 pixels. On the other hand, the complementary color filter of the pattern shown in FIG. 24 has color filters arranged so as to have a periodicity of 2 pixels in the horizontal direction and 8 pixels in the vertical direction.
Regardless of which complementary color filter is used, the luminance signal Y is generated from the pixel signals Cy, Ye, Mg, G (from the pixels in the 4 (2 × 2) pixel block of 2 pixels in the horizontal direction and 2 pixels in the vertical direction. It is obtained by performing the calculation of the above formula (1) using the above-described FIG. 23 and FIG. Similarly, the color difference signals CB and CR can be obtained by performing the above expressions (2) and (3) using the pixel signals Cy, Ye, Mg, and G from the pixels in the 2 × 2 pixel block. .

  However, when the conventional imaging device in which subject light is incident through the color filter as shown in FIGS. 23 and 24 is used in, for example, a digital still camera, there are the following problems.

  (1) In the digital still camera, before the shutter is pressed, the pixel signals (here, the pixel signals Cy, Ye, Mg, G described above) obtained by the image sensor are read out at high speed (at high speed) at the expense of resolution. Read mode), and based on the read signal, the image is displayed on a liquid crystal finder, iris adjustment, white balance adjustment, and the like.

  However, when the color filter shown in FIG. 24 is used for the image sensor, the pixel signal of the image sensor can be read out in the high-speed reading mode, and the white balance can be adjusted based on the pixel signal. When the color filter shown in FIG. 23 is used for the image sensor, pixel signals are read out by thinning out pixels one by one in the vertical direction. Only Cy and the pixel signal Ye were obtained, that is, only pixel signals for two colors of cyan and yellow were obtained, and the white balance could not be adjusted.

  (2) Recently, in an image pickup device such as a CCD, a signal reading method has been devised so that subject light is received by a light receiving surface in the device, and pixel signals of two pixels adjacent in the vertical direction on the light receiving surface are added. In some cases, the sum signal is transferred to a transfer unit in the element and then output. Therefore, the signal output from the image sensor is output in a state where two pixel signals form a pair.

Specifically, for example, when the color filter shown in FIG. 23 is used for the image sensor, first, for the rows L1 and L2, the pixel signal Cy of the pixel located at (C1, L1) and (C1 , L2), a pair of pixel signal Mg of the pixel located at (C2, L1) and a pixel signal G of the pixel located at (C2, L2) are output. A pair is output. Thereafter, the output is sequentially performed in the same manner, and when the output for the rows L1 and L2 is completed, the pixel signals Cy of the pixels located at (C1, L3) and the (C1, L4) for the next rows L3 and L4. After a pair with the pixel signal G of the pixel located is output, a pair of the pixel signal Ye of the pixel located at (C2, L3) and the pixel signal Mg of the pixel located at (C2, L4) is output The Thereafter, output is sequentially performed in the same manner, and output for the rows L3 and L4 is completed.
That is, the output for rows L1 and L2 is
(Cy + Mg), (Ye + G), ...
And the output for the next row L3, L4 is
(Cy + G), (Ye + Mg),...
It becomes.

However, in order to obtain the luminance signal Y and the color difference signals CB and CR from the above output, the above equations (2) and (2) for obtaining the color difference signals CB and CR when the calculations of the above equations (1) to (3) are performed. As for the expression (3), there is an output that cannot be calculated.
That is, the above equation (2) is composed of (G + Ye) and (Mg + Cy), but the output for the rows L3 and L4 is
(Cy + G), (Ye + Mg),...
Therefore, the calculation of the above formula (2) for these rows cannot be performed. Further, the above equation (3) is composed of (Cy + G) and (Ye + Mg), and the outputs for the rows L1 and L2 are as follows:
(Cy + Mg), (Ye + G), ...
Therefore, the calculation of the above equation (3) cannot be performed for these rows.
As described above, the calculation of the above formula (2) can be performed on the outputs for the rows L1 and L2, but cannot be performed on the outputs for the rows L3 and L4. Further, the calculation of the above equation (3) can be performed on the output for the rows L3 and L4, but cannot be performed on the output for the rows L1 and L2.
Therefore, the color difference signals CB and CR for each color can be obtained only for one row of pixel signals from the four rows of pixel signals output from the image sensor, that is, four rows of pixel signals. Since the luminance signal Y and the color difference signals CB and CR are obtained for the first time by using, the vertical resolution is reduced accordingly.

Similarly, when the color filter shown in FIG. 24 is used for the image sensor, first, for the rows L1 and L2,
(Cy + Mg), (Ye + G), ...
For the next rows L3 and L4,
(Cy + Mg), (Ye + G), ...
For the next rows L5 and L6,
(Cy + G), (Ye + Mg),...
For the next rows L7 and L8,
(Cy + G), (Ye + Mg),...
Is output.

However, in order to obtain the luminance signal Y and the color difference signals CB and CR from the above output, when the calculations of the above equations (1) to (3) are performed, also in this case, the above-described color filter of FIG. Similarly to Expression (2) and Expression (3) for obtaining the color difference signals CB and CR, there is an output that cannot be calculated.
That is, the above equation (2) is composed of (G + Ye) and (Mg + Cy), but the outputs for the rows L5 and L6 and the rows L7 and L8 are
(Cy + G), (Ye + Mg),...
Therefore, the calculation of the above formula (2) for these rows cannot be performed. Moreover, although the said Formula (3) consists of (Cy + G) and (Ye + Mg), the output about row L1, L2 and row L3, L4 is as follows.
(Cy + Mg), (Ye + G), ...
Therefore, the calculation of the above equation (3) cannot be performed for these rows.
As described above, the calculation of the above formula (2) can be performed on the outputs for the rows L1 to L4, but cannot be performed on the outputs for the rows L5 to L8. Further, the calculation of the above formula (3) can be performed on the outputs for the rows L5 to L8, but cannot be performed on the outputs for the rows L1 to L4.
Therefore, the color difference signals CB and CR for each color can be obtained only for the pixel signals for two rows from the pixel signals for eight rows output from the image sensor, and the vertical resolution is accordingly increased. Had fallen.

  Therefore, the present invention has been made to eliminate the above-described drawbacks, and implements an imaging device, an image processing apparatus, an image processing system, and an operation for obtaining an image signal with high resolution in both the horizontal and vertical directions. It is an object of the present invention to provide a storage medium storing a program for causing a computer to execute processing steps for the purpose.

  The present invention also provides an image pickup device, an image processing apparatus, an image processing system, and a program storing a program for causing a computer to execute processing steps for performing the operation, which efficiently obtains a high-resolution color image signal. The purpose is to provide a medium.

An image pickup device according to the present invention includes light receiving means for receiving subject light via a predetermined color filter array with a plurality of pixels arranged two-dimensionally and obtaining pixel signals corresponding to the amounts of light received by the plurality of pixels. Reading means for reading out pixel signals from the light receiving means in units of pixel blocks of N rows × N columns (N: integer) while shifting by a predetermined pixel in at least one of the row direction and the column direction; Holding means for holding a pixel signal of a region overlapping with a next read pixel block shifted by a predetermined pixel with respect to the read pixel block among the pixel signals of the current read pixel block read by the means, The reading means uses the pixel signal held in the holding means for reading the pixel signal of the next pixel block.
An image processing apparatus according to the present invention is an image processing apparatus that images a subject with a color image sensor and performs predetermined image processing on an output signal of the image sensor, and the color image sensor includes the above-described image sensor. It is characterized by being.
An image processing system according to the present invention is an image processing system in which an imaging device including a color imaging device and an image processing device that performs predetermined image processing on an output signal of the imaging device are connected at least. The color image sensor is the above-described image sensor.
The computer-readable storage medium according to the present invention receives an object light through a color filter array with a plurality of pixels arranged two-dimensionally, and obtains a pixel signal corresponding to each light reception amount at the plurality of pixels. A computer-readable storage medium storing a program for causing a computer to execute processing steps for operating an element, wherein the processing steps include predetermined pixels in at least one of a row direction and a column direction. A readout step of reading out the pixel signal in the light receiving unit in units of N rows × N columns (N: integer) while shifting by a minute amount, and a pixel signal of the current readout pixel block read out by the readout step Among them, the pixel signal of the area overlapping with the next read pixel block shifted by a predetermined pixel with respect to the read pixel block is stored in the memory in the element. And a holding step of lifting, the reading step, the reading of the pixel signal of the next pixel block, characterized in that it comprises the step of using the pixel signal held in the memory.
An image processing system according to the present invention includes an image sensor and compresses and transmits an information amount of an image signal output from the image sensor without undergoing color processing that performs at least white balance correction or γ correction on color information. An image processing system in which an imaging device and a receiving device configured to perform the color processing after receiving and expanding the compressed image signal are connected, and the imaging device includes the above-described imaging It is an element.

When the pixel signal obtained from the subject light through the color filter array is read out in units of pixel blocks of N rows × N columns, the pixel block to be read next is horizontal with respect to the pixel block from which the previous reading was performed. A high-resolution luminance signal and chrominance signal are obtained by configuring the block to be read out as a block at a position shifted by a predetermined number of pixels in the vertical or horizontal direction, or in the vertical direction along with the horizontal direction. Can do.
For example, when reading out in units of pixel blocks of 2 rows × 2 columns, first, as a result of reading out the first pixel block, the pixel signal Y ₁₁ in the first row and the first column, the first row and the second column Pixel signal Y ₁₂ , second row first column pixel signal Y ₂₁ , and second row second column pixel signal Y ₂₂ are obtained. Of these pixel signals, a region (first row, second column) overlapping with a second pixel block to be read out next (a block at a position shifted in the horizontal direction by one pixel with respect to the first pixel block). And second row, second column) pixel signals Y ₁₂ and Y ₂₁ are held. Then, the held pixel signals Y ₁₂ and Y ₂₁ are used as pixel signals for the overlapping region for reading the next pixel block.
As a result, when a luminance signal and a color difference signal are obtained using a pixel signal of the pixel block in units of 2 × 2 pixel blocks, a luminance signal and a color difference signal corresponding to each pixel in the horizontal direction can be obtained. it can. Therefore, in this case, the horizontal resolution can be doubled.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
The present invention is applied to, for example, a color imaging apparatus 700 as shown in FIG.
This color imaging device (hereinafter simply referred to as “imaging device”) 700 is provided in a digital still camera or the like, and includes an imaging unit 710 that outputs pixel signals obtained from incident subject light, and an imaging unit 710. And a luminance / color difference pre-processing unit 720 that generates a luminance signal Y and color difference signals CB and CR using the pixel signals from.

The imaging unit 710 includes a color filter 711, an imaging device 712 on which subject light is incident via the color filter 711, and a signal reading unit 713 that reads a pixel signal obtained by the imaging device 712. Is supplied to the luminance / color difference pre-processing unit 720.
Here, although details will be described later in the imaging unit 710, four color complementary filters of yellow (Ye), cyan (Cy), magenta (Mg), and green (G) are used as the color filter 711. . Therefore, the image pickup device 712 obtains four-color complementary color pixel signals Ye, Cy, Mg, G, and these pixel signals Ye, Cy, Mg, G are read out by the signal reading unit 713 and before the luminance / color difference. It will be supplied to the processing unit 720.

  The luminance / color difference preprocessing unit 720 obtains the luminance signal Y and the color difference signals CB, CR using the pixel signals Ye, Cy, Mg, G from the imaging unit 710. The luminance signal Y and the color difference signals CB and CR are used for low-resolution screen display on a liquid crystal finder in a digital still camera or the like, iris adjustment, white balance adjustment, autofocus detection signal, and the like.

  Therefore, the most characteristic feature of the imaging apparatus 700 as described above is the imaging unit 710. Hereinafter, the imaging unit 710 will be specifically described.

First, the configuration of the color filter 711 will be described.
FIG. 2 shows a filter arrangement (pattern) of the color filter (complementary color filter) 711 shown in FIG.

  As shown in FIG. 2, the color filter 711 is arranged in a filter so as to have a periodicity of 4 pixels in the horizontal direction and 4 pixels in the vertical direction. In the color filter 711 having such a filter arrangement, color filters G1, Ye1, Cy2, and Mg2 are arranged in order from the left in the first row L1, and Cy1, Mg1 in the second row L2 in order from the left. , G2, Ye2 color filters are arranged, Ye3, G3, Mg4, Cy4 color filters are arranged in order from the left in the third row L3, and Mg3, Cy3, Ye4 in order from the left in the fourth row L4. , G4 color filters are arranged.

As described above, in the color filter 711, there are no duplicate color filters in any row among the four pixels in the horizontal direction and the four pixels in the vertical direction, all the color filters are different, and duplicate in every column. The filter arrangement is such that there is no color filter and all color filters are different.
In the filter arrangement of FIG. 2, the arrangement of the color filters may be reversed left and right or up and down.

  The subject light that has passed through the color filter 711 having the filter arrangement as described above is incident on a light receiving surface (not shown) of the image sensor 712. The image sensor 712 performs photoelectric conversion on each pixel on the light receiving surface. A pixel signal corresponding to the amount of received light is obtained. The signal reading unit 713 reads out the pixel signal obtained by the image sensor 712 according to a reading method described later, and supplies the pixel signal to the luminance / color difference preprocessing unit 720. The luminance / color difference preprocessing unit 720 obtains the luminance signal Y, the blue color difference signal CB, and the red color difference signal CR using the pixel signals of various sizes and shapes from the signal reading unit 713.

  Next, a pixel signal reading method in the signal reading unit 713 and a calculation process in the luminance / color difference preprocessing unit 720 using the pixel signal read in accordance with the reading method will be described.

[Reading method 1]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1), (Cy2 + Mg2),
(G2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from a pair of adjacent pixels (addition signal) in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Cy1 + Mg1)
CB2 =-(Cy2 + Mg2) + (G2 + Ye2)
It is calculated by
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(G1 + Cy1), (Ye1 + Mg1), (Cy2 + G2),
(Mg2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = (G1 + Cy1)-(Ye1 + Mg1)
CR2 = (Cy2 + G2)-(Mg2 + Ye2)
Is obtained by the following calculation.
According to the reading method 1 as described above, there is no occurrence of a row in which the calculation for obtaining the color difference signals CB and CR cannot be performed as in the prior art, and the color difference signals CB and CR are in the horizontal and vertical directions. Both are always obtained for two pixel signals. Therefore, the resolution is high in both the horizontal direction and the vertical direction.

[Reading method 2]
From this reading method 2 to a reading method 5 described later, the luminance signal Y can be output at a high resolution. These are methods suitable for use in autofocusing or the like performed by calculating using only the luminance signal Y.
Therefore, in the readout method 2, the signal readout unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Cy1 + Ye3 + Mg3),
(Ye1 + Mg1 + G3 + Cy3), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = G1 + Cy1 + Ye3 + Mg3
Y2 = Ye1 + Mg1 + G3 + Cy3
Is obtained by the following calculation.
According to such a reading method 2, since one luminance signal Y is obtained for each pixel in the horizontal direction, the resolution is high in the horizontal direction. Therefore, the luminance signal Y generated by this method is particularly suitable for use as an autofocus detection signal when the subject has a high horizontal resolution such as a fine vertical stripe pattern.

[Reading method 3]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Cy1 + Ye3 + Mg3),
(Ye1 + Mg1 + G3 + Cy3), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = (G1 + Cy1 + Ye3 + Mg3)
+ (Ye1 + Mg1 + G3 + Cy3)
Is obtained by the following calculation.
According to such readout method 3, one luminance signal Y is obtained every two pixels in the horizontal direction. In addition, the level of the luminance signal Y increases. Therefore, the luminance signal Y generated by this method is particularly used as an autofocus detection signal when the subject has a high horizontal resolution such as a fine vertical stripe pattern and the subject has low luminance. Ideal for.

[Reading method 4]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1 + Cy2 + Mg2),
(Cy1 + Mg1 + G2 + Ye2), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = G1 + Ye1 + Cy2 + Mg2
Y2 = Cy1 + Mg1 + G2 + Ye2
Is obtained by the following calculation.
According to such a reading method 4, since one luminance signal Y is obtained for each pixel in the vertical direction, the resolution is high in the vertical direction. Therefore, the luminance signal Y generated by this method is particularly suitable for use as an autofocus detection signal when the subject has a high horizontal resolution such as a fine horizontal stripe pattern.

[Reading method 5]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1 + Cy2 + Mg2),
(Cy1 + Mg1 + G2 + Ye2), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = (G1 + Ye1 + Cy2 + Mg2)
+ (Cy1 + Mg1 + G2 + Ye2)
Is obtained by the following calculation.
According to such a reading method 5, one luminance signal Y is obtained every two pixels in the vertical direction. In addition, the level of the luminance signal Y increases. Therefore, the luminance signal Y generated by this method is particularly suitable as a detection signal for autofocus when the subject has a high horizontal resolution such as a fine horizontal stripe pattern and the subject has low luminance. is there.

[Reading method 6]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1), (Cy2 + Mg2),
(G2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Cy1 + Mg1)
CB2 =-(Cy2 + Mg2) + (G2 + Ye2)
Is obtained by the following calculation.
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(Ye3 + Mg3), (G3 + Cy3), (Mg4 + Ye4),
(Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CRl = − (Ye3 + Mg3) + (G3 + Cy3)
CR2 = − (Mg4 + Ye4) + (Cy4 + G4)
Is obtained by the following calculation.
According to such a reading method 6, a color difference line sequential signal can be obtained. This method is optimal when the subject is a moving image and can output an image signal at high speed. This is because the number of readout signals (the number of pixels of the color difference signal to be read) is half that of the above-described readout method 1.

[Reading method 7]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1), (Cy2 + Mg2),
(G2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Cy1 + Mg1)
CB2 =-(Cy2 + Mg2) + (G2 + Ye2)
Is obtained by the following calculation.
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(Cy2 + G2), (Mg2 + Ye2), (Mg4 + Ye4),
(Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = (Cy2 + G2) − (Mg2 + Ye2)
CR2 = − (Mg4 + Ye4) + (Cy4 + G4)
Is obtained by the following calculation.
Similar to the above-described reading method 6, such a reading method 7 is also optimal when the subject is a moving image, and can output an image signal at high speed.

[Reading method 8]
The color difference signal CB is obtained in the same manner as the reading method 7 described above.
For the color difference signal CR, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Cy1), (Ye1 + Mg1), (Ye3 + Mg3),
(G3 + Cy3), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CR1 = (G1 + Cy1) − (Ye1 + Mg1)
CR2 = − (Ye3 + Mg3) + (G3 + Cy3)
Is obtained by the following calculation.
Such readout method 8 is also optimal when the subject is a moving image, and can output an image signal at high speed, as in the above-described readout methods 6 to 7.

[Reading method 9]
The color difference signal CB is obtained in the same manner as the reading method 7 described above.
For the color difference signal CR, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Cy1), (Ye1 + Mg1), (Cy2 + G2),
(Mg2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = (G1 + Cy1)-(Ye1 + Mg1)
CR2 = (Cy2 + G2)-(Mg2 + Ye2)
Is obtained by the following calculation.
Similar to the above-described reading methods 6 to 8, the reading method 9 is optimal when the subject is a moving image, and can output an image signal at high speed. Furthermore, the color difference signal CB and the color difference signal CR are obtained from pixel signals in the same region.

[Reading methods 10 to 13]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1), (Ye3 + G3),
(Mg3 + Cy3), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Cy1 + Mg1)
CB2 = (Ye3 + G3)-(Mg3 + Cy3)
Is obtained by the following calculation.
The color difference signal CR is obtained in the same manner as the above-described reading 6 to 9.
Similar to the above-described reading methods 6 to 9, such reading methods 10 to 13 are optimum when the subject is a moving image, and can output an image signal at high speed.

[Reading method 14]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1),...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Cy1 + Mg1)
Is obtained by the following calculation.
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(Mg4 + Ye4), (Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = − (Mg4 + Ye4) + (Cy4 + G4)
Is obtained by the following calculation.
Similar to the above-described reading methods 6 to 13, such a reading method 14 is also optimal when the subject is a moving image, and can output an image signal at high speed. In particular, in this method, the readout signal (the number of readout pixels necessary for the color difference signal) is ¼ that of the readout method 1 described above, so that an image signal can be output at a higher speed.

[Reading method 15]
In the reading methods 1 and 6 to 14 described above, only the method for generating the color difference signal has been described. However, the luminance signal can be generated by the following reading method 15.
That is, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Cy1 + Mg1), (Cy2 + Mg2),
(G2 + Ye2),
(Ye3 + G3), (Mg3 + Cy3), (Mg4 + Cy4),
(Ye4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the luminance signal Y from the adjacent pairs in the pixel signal series described above. For example,
Y1 = (G1 + Ye1) + (Cy1 + Mg1)
Y2 = (Cy2 + Mg2) + (G2 + Ye2)
Y3 = (Ye3 + G3) + (Mg3 + Cy3)
Y4 = (Mg4 + Cy4) + (Ye4 + G4)
Is obtained by the following calculation.
According to such a readout method 15, a luminance signal can be obtained in units of two pixels in both the horizontal and vertical directions.

[Reading method 16]
In the reading methods 1 and 6 to 14 described above, only the method for generating the color difference signal has been described, but the luminance signal can be generated by the following reading method 16.
That is, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Cy1), (Ye1 + Mg1), (Cy2 + G2),
(Mg2 + Ye2),
(Ye3 + Mg3), (G3 + Cy3), (Mg4 + Ye4),
(Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the luminance signal Y from the adjacent pairs in the pixel signal series described above. For example,
Y1 = (G1 + Cy1) + (Ye1 + Mg1)
Y2 = (Cy2 + G2) + (Mg2 + Ye2)
Y3 = (Ye3 + Mg3) + (G3 + Cy3)
Y4 = (Mg4 + Ye4) + (Cy4 + G4)
Is obtained by the following calculation.
According to such a reading method 16, a luminance signal can be obtained in units of two pixels in both the horizontal and vertical directions.

  In the above-described embodiment, the filter arrangement of the color filters 711 used for the image sensor 712 is as shown in FIG. 2, but for example, as shown in FIG. In the filter arrangement in Fig. 5, the above-described readout methods 1 to 16 can be applied even if the third row L3 and the fourth row L4 are interchanged.

Further, in the above-described reading methods 1 to 16, signals are read in units of 4 × 4 or 2 × 2 pixel blocks, but the present invention is not limited to this. In addition, the pixel blocks to be read are not necessarily closely adjacent to the pixel block from which the previous reading has been performed. For example, these pixel blocks may be spaced apart by a predetermined block unit. good. By doing this, the number of pixels of the signal to be read out (number of pixel signals of the color difference signal) can be further reduced, and further speedup can be achieved.
For example, when performing high-speed output at a low resolution, within an 8 × 8 pixel block in the horizontal direction C1, C2,..., C8 and the vertical direction L1, L2,.
Color difference signal CR1 from (C1, L1), (C1, L2), (C2, L1), (C2, L2)
The color difference signal CB1 is obtained from (C3, L3), (C3, L4), (C4, L3), (C4, L4),
Color difference signal CR2 from (C5, L5), (C5, L6), (C6, L5), (C6, L6)
The color difference signal CB2 from (C7, L7), (C7, L8), (C8, L7), (C8, L8)
You may make it ask.

(Second Embodiment)
In the present embodiment, the filter arrangement of the color filters 711 in the imaging device 700 shown in FIG. 1 is as shown in FIG. 4, for example.

  As shown in FIG. 4, the color filter 711 here has a filter arrangement so as to have a periodicity of 4 pixels in the horizontal direction and 4 pixels in the vertical direction. In the color filter 711 having such a filter arrangement, color filters G1, Ye1, Mg2, and Cy2 are arranged in order from the left in the first row L1, and the first row L1 is arranged in the second row L2. The color filters of Mg1, Cy1, G2, Ye2 are arranged in order from the left in the form of two columns shifted to the left, and the third row L3 is shifted to the left by three columns with respect to the first row L1, Cy3, G3, Ye4, and Mg4 color filters are arranged in order from the left, and Ye3, Mg3, Cy4, and G4 are sequentially arranged from the left in the fourth row L4, shifted by one column to the left with respect to the first row L1. The color filters are arranged.

As described above, in the color filter 711, there are no duplicate color filters in any row among the four pixels in the horizontal direction and the four pixels in the vertical direction, all the color filters are different, and duplicate in every column. The filter arrangement is such that there is no color filter and all color filters are different.
In the filter arrangement shown in FIG. 4, the arrangement of the color filters may be reversed left and right or up and down.

  The subject light that has passed through the color filter 711 having the filter arrangement as described above is incident on the light receiving surface of the image sensor 712 in the same manner as in the first embodiment, and the image sensor 712 performs photoelectric conversion. A pixel signal corresponding to the amount of light received by each pixel on the light receiving surface is obtained. The signal reading unit 713 reads out the pixel signal obtained by the image sensor 712 according to a reading method described later, and supplies the pixel signal to the luminance / color difference preprocessing unit 720. The luminance / color difference preprocessing unit 720 obtains a luminance signal Y, a blue color difference signal CB, and a red color difference signal CR using the pixel signals of various shapes from the signal reading unit 713.

Accordingly, the pixel signal readout method in the signal readout unit 713 and the arithmetic processing in the luminance / color difference preprocessing unit 720 using the pixel signal read out according to the readout method in the present embodiment will be described below. explain.
Note that the overall operation of the imaging apparatus 700 in this embodiment is the same as that in the first embodiment, and is omitted here for the sake of simplicity. Only the structure will be specifically described.

[Reading method 21]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Mg1 + Cy3 + Ye3),
(Ye1 + Cy1 + G3 + Mg3), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = G1 + Mg1 + Cy3 + Ye3
Y2 = Ye1 + Cy1 + G3 + Mg3
Is obtained by the following calculation.
According to such a reading method 21, since one luminance signal Y is obtained for each pixel in the horizontal direction, the resolution becomes high in the horizontal direction. Therefore, the luminance signal Y generated by this method is most suitable for use as an autofocus detection signal when the subject has a high horizontal resolution such as a fine vertical stripe pattern.

[Reading method 22]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Mg1 + Cy3 + Ye3),
(Ye1 + Cy1 + G3 + Mg3), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = (G1 + Mg1 + Cy3 + Ye3)
+ (Ye1 + Cy1 + G3 + Mg3)
Is obtained by the following calculation.
According to such a reading method 22, one luminance signal Y is obtained every two pixels in the horizontal direction. In addition, the level of the luminance signal Y increases. Therefore, the luminance signal Y generated by this method is particularly used as an autofocus detection signal when the subject has a high horizontal resolution such as a fine vertical stripe pattern and the subject has low luminance. Ideal for.

[Reading method 23]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1 + Mg2 + Cy2),
(Mg1 + Cy1 + G2 + Ye2), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = G1 + Ye1 + Mg2 + Cy2
Y2 = Mg1 + Cy1 + G2 + Ye2
Is obtained by the following calculation.
According to such a reading method 23, since one luminance signal Y is obtained for each pixel in the vertical direction, the resolution is high in the vertical direction. Therefore, the luminance signal Y generated by this method is particularly suitable for use as an autofocus detection signal when the subject has a high horizontal resolution such as a fine horizontal stripe pattern.

[Reading method 24]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1 + Mg2 + Cy2),
(Mg1 + Cy1 + G2 + Ye2), ...
Read with. In the pixel signal series, the luminance / color difference preprocessing unit 720 converts the luminance signal Y into, for example,
Y1 = (G1 + Ye1 + Mg2 + Cy2)
+ (Mg1 + Cy1 + G2 + Ye2)
Is obtained by the following calculation.
According to such a reading method 24, one luminance signal Y is obtained every two pixels in the vertical direction. In addition, the level of the luminance signal Y increases. Therefore, the luminance signal Y generated by this method is used as an autofocus detection signal particularly when the subject has a high horizontal resolution such as a fine horizontal stripe pattern and the subject has low luminance. Ideal for.

[Reading method 25]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Mg1 + Cy1), (Mg2 + Cy2),
(G2 + Ye2), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CB1 = (G1 + Ye1) − (Mg1 + Cyl)
CB2 =-(Mg2 + Cy2) + (G2 + Ye2)
Is obtained by the following calculation.
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(Cy3 + G3), (Ye3 + Mg3), (Ye4, Mg4),
(Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = (Cy3 + G3)-(Ye3 + Mg3)
CR2 = − (Ye4 + Mg4) + (Cy4 + G4)
Is obtained by the following calculation.
According to such a reading method 25, a color difference line sequential signal is obtained. In addition, this method is optimal when the subject is a moving image because the number of signals to be read (the number of pixels to be read for the color difference signal) is halved compared to the case of using all pixels as a base. Can output an image signal.

[Reading Method 26]
The signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Mg1 + Cy1),...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CB from adjacent pairs in the pixel signal series described above. For example,
CBl = (G1 + Ye1) − (Mg1 + Cy1)
Is obtained by the following calculation.
In addition, the signal reading unit 713 receives the pixel signal from the image sensor 712.
Pixel signal series:
(Ye4 + Mg4), (Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the color difference signal CR from adjacent pairs in the pixel signal series described above. For example,
CR1 = (Ye4 + Mg4)-(Cy4 + G4)
Is obtained by the following calculation.
Similar to the above-described reading method 25, such a reading method 26 is also optimal when the subject is a moving image, and can output an image signal at high speed. In particular, since the number of signals to be read is ¼ compared to the case where all pixels are used as a basis, an image signal is output at a higher speed.

[Reading Method 27]
In the above-described reading methods 25 and 26, only the method for generating the color difference signal has been described, but the luminance signal can be generated by the following reading method 27.
That is, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Ye1), (Mg1 + Cy1), (Mg2 + Cy2),
(G2 + Ye2),
(Cy3 + G3), (Ye3 + Mg3), (Ye4 + Mg4),
(Cy4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the luminance signal Y from the adjacent pairs in the pixel signal series described above. For example,
Y1 = (C1 + Ye1) + (Mg1 + Cy1)
Y2 = (Mg2 + Cy2) + (G2 + Ye2)
Y3 = (Cy3 + G3) + (Ye3 + Mg3)
Y4 = (Ye4 + Mg4) + (Cy4 + G4)
Is obtained by the following calculation.
According to such a reading method 27, a luminance signal can be obtained in units of two pixels in both the horizontal and vertical directions.

[Reading method 28]
In the above-described reading methods 25 and 26, only the method for generating the color difference signal has been described, but the luminance signal can be generated by the following reading method 28.
That is, the signal reading unit 713 receives the pixel signal from the image sensor 712,
Pixel signal series:
(G1 + Mg1), (Ye1 + Cy1), (Mg2 + G2),
(Cy2 + Ye2),
(Cy3 + Ye3), (G3 + Mg3), (Ye4 + Cy4),
(Mg4 + G4), ...
Read with. The luminance / color difference preprocessing unit 720 obtains the luminance signal Y from the adjacent pairs in the pixel signal series described above. For example,
Y1 = (G1 + Mg1) + (Ye1 + Cy1)
Y2 = (Mg2 + G2) + (Cy2 + Ye2)
Y3 = (Cy3 + Ye3) + (G3 + Mg3)
Y4 = (Ye4 + Cy4) + (Mg4 + G4)
Is obtained by the following calculation.
According to such a reading method 28, a luminance signal can be obtained in units of two pixels in both the horizontal and vertical directions.

  In the above-described embodiment, the filter arrangement of the color filters 711 used for the image sensor 712 is as shown in FIG. 4, but for example, as shown in FIG. In the filter arrangement in Fig. 5, the above-described readout methods 21 to 28 can be applied even if the third row L3 and the fourth row L4 are interchanged.

  In the above-described reading methods 21 to 26, signals are read in units of 4 × 4 pixel blocks, but the present invention is not limited to this. In addition, the pixel blocks to be read are not necessarily closely adjacent to the pixel block from which the previous reading has been performed. For example, these pixel blocks may be spaced apart by a predetermined block unit. good. By doing this, the number of pixels of the signal to be read out (number of pixel signals of the color difference signal) can be further reduced, and further speedup can be achieved.

(Third embodiment)
In the present embodiment, for example, in the imaging apparatus 700 of FIG. 1, the imaging unit 710 is applied to a so-called CMOS sensor. In this case, the readout methods 1 to 28 in the first and second embodiments described above can be realized by setting the circuit configuration of the imaging unit 710 as described below.

FIG. 6 shows a circuit configuration of the imaging unit 710 (hereinafter referred to as “CMOS sensor 710”).
The CMOS sensor 710 detects the average value of the amount of light received (detected light amount) of two pixels (photodetection elements) adjacent in the vertical direction and the detection of two photodetection elements adjacent in the vertical direction in the adjacent column. A first output series for outputting a difference from the average value of the light amount, an average value of the detected light amount of two light detection elements adjacent in the horizontal direction, and two light detectors adjacent in the horizontal direction of the adjacent row And a second output series for outputting the difference between the average values of the detected light amounts. With such a circuit configuration, the CMOS sensor 710 can perform the reading method 1.

Specifically, in FIG. 6, first, reference numeral 1 denotes a vertical scanning circuit that generates an enable signal for sequentially enabling a control signal of each row, which is a signal that becomes sequentially active in the vertical direction. Reference numeral 100 denotes a photodiode as a light detection element that converts incident light into electric charges, and reference numeral 101 denotes a transfer transistor that transfers electric charges generated in the light detection element 100 to the floating diffusion region 102. The floating diffusion region 102 temporarily accumulates electric charges generated in the light detection element 100.
103 is a reset transistor 103 that discharges the electric charge accumulated in the gate of the amplifier transistor 104, 121 is a switch transistor, and 112 is a constant current source transistor determined by a voltage applied from the terminal 7. It is.
Reference numeral 105 denotes a transistor for discharging the electric charges of the capacitors 109, 110, 117, and 118. Reference numeral 106 denotes a distribution transistor that connects the source of the amplifier transistor 104 and the capacitor 109. Reference numeral 107 denotes a distribution transistor that connects the source of the amplifier transistor 104 and the capacitor 110.
Capacitors 109 and 110 are capacitors that function as line memories that are charged by the supply voltage from the amplifier transistor 104.
Reference numeral 108 denotes an averaging transistor for controlling the averaging of the accumulated charge of the capacitor 109 and the accumulated charge of the capacitor 110. Reference numeral 111 denotes a switch transistor for applying the voltage of the capacitor 109 functioning as a line memory to the buffer 123 in the previous stage of the differential amplifier 122.
Reference numeral 122 denotes a differential amplifier that amplifies the difference between the voltage of the capacitor 109 and the voltage of the capacitor 109 ′.
Reference numeral 113 denotes a switch transistor that connects the source of the amplifier transistor 104 and the capacitor 117, and reference numeral 114 denotes a switch transistor that connects the source of the amplifier transistor 104 and the capacitor 118.
Reference numerals 117 and 118 denote capacitors as line memories that are charged by a current supplied from the source of the amplifier transistor 104. Reference numeral 115 denotes a switch transistor for controlling averaging of the accumulated charge of the capacitor 117 and the accumulated charge of the capacitor 117 ′.
Reference numeral 116 denotes a switch capacitor for controlling the averaging of the accumulated charge of the capacitor 118 and the accumulated charge of the capacitor 118 ′. Reference numeral 119 denotes the voltage of the capacitor 117 serving as a line memory. 128 is a switch transistor for applying to 128.
Reference numeral 127 denotes a differential amplifier that amplifies the difference between the voltage of the capacitor 117 and the voltage of the capacitor 118 ′.
The above-described low current source transistor 112 is activated in units of rows, and constitutes an amplifier in combination with the amplifier transistor 104.

  FIG. 7 is a timing chart showing the operation timing of the CMOS sensor 710 of FIG. Hereinafter, the operation of the CMOS sensor 710 will be described with reference to FIGS.

First, at timing T201, the input pulse to the terminal 11 is in a HIGH state. In this state, when the input pulse to the terminals 30, 31, 50, 51 becomes HIGH, the capacitors 109, 110, 117, 118 as line memories are all reset to the initial potential. When the start pulse of the vertical scanning circuit 1 input from the terminal 2 and the scanning pulse input from the terminal 3 become HIGH, the vertical scanning circuit 1 starts scanning, and the first row (R1) is selected. The Further, a HIGH pulse is input to the terminal 8, thereby resetting the floating diffusion region of the pixel portion.
Next, at a timing T202, the reset pulse input to the terminal 8 becomes a LOW state (falls). As a result, the floating diffusion region of the pixels in the first row becomes electrically floating.
Next, when the input pulse to the terminal 9 becomes HIGH at timing T203, the charge is transferred from the photodetecting element in the first row to the floating diffusion region.
Next, when the input pulse to the terminals 10, 30, 50 becomes HIGH at timing T 204, a voltage proportional to the amount of light detected by the light detection element in the first row is generated by the amplifier transistor 104 by the capacitors 109, 117 is read out.
Next, at a timing T205, the vertical scanning pulse input to the terminal 3 is in a LOW state (falls).
Next, at the timing T206, when the vertical pulse input to the terminal 3 becomes HIGH again (rising up), the second row (R2) is selected.
Next, when the reset pulse input to the terminal 8 becomes LOW (falling) at the timing T207, the floating diffusion region of the pixels in the second row is in an electrically floating state.
Next, at the timing T208, as in the above-described timing T203, when the input pulse to the terminal 9 becomes HIGH, the charge is transferred from the photodetecting element in the second row to the floating diffusion region.
Next, also at the timing T209, as in the above-described timing T204, when the input pulse to the terminals 10, 31, 51 becomes HIGH, a voltage proportional to the amount of light detected by the light detection element in the second row is Data is read out from the capacitors 110 and 118 by the amplifier transistor 104 in the second row.
Next, at timing T210, when the input pulse to the terminals 40, 60, 61 becomes a HIGH pulse, the accumulated charge is averaged on the capacitor 117 as a line memory.

At timing T211, the horizontal scanning circuit 4 starts to operate from this timing, and the averaged voltages are sequentially applied to the differential amplifiers (amplifiers) 122 and 127 in the horizontal direction.
As a result, the differential color amplifiers 122 and 127 output the blue color difference signal CB and the red color difference signal CR.
Furthermore, the luminance signal Y is also obtained by adding the outputs of the terminals 70 and 71 or 80 and 81 by an adder (not shown) constituted by an operational amplifier or the like. Further, if the charges of the photodetecting elements in all rows are temporarily stored in the capacitor 109 without performing averaging by the averaging transistor 108, an output for each pixel in the odd-numbered column can be obtained from the terminal 71. Similarly, the output for each pixel in the even column can be obtained from the output terminal 70.

(Fourth embodiment)
In the third embodiment described above, the circuit configuration of the CMOS sensor 710 is as shown in FIG. 6, but in this embodiment, for example, it is configured as shown in FIG.
In the circuit configuration of FIG. 8, the same reference numerals are given to the portions that operate in the same manner as the circuit configuration of FIG. 6, and detailed description thereof is omitted.

  The CMOS sensor 710 here outputs an output series that outputs an average value of the detected light amounts of the two light detecting elements adjacent in the horizontal direction or an average value of the detected light amounts of the four light detecting elements adjacent in the horizontal direction. I have. With such a circuit configuration, the CMOS sensor 710 can perform the readout method 4 and the readout method 23.

Specifically, in FIG. 8, first, 301 is a switch transistor for controlling the averaging of the accumulated charge of the capacitor 109 and the accumulated charge of the capacitor 109 ′, and 302 is the accumulated charge of the capacitor 110. And a switch transistor for controlling the averaging of the charge stored in the capacitor 110 ′.
Reference numeral 301 ′ denotes a switch transistor for controlling the averaging between the accumulated charge of the capacitor 109 ″ and the accumulated charge of the capacitor 109 ″ ′, and 302 ′ denotes the accumulated charge of the capacitor 110 ″ and the capacitor 110 ″ ′. This is a switch transistor for controlling the averaging with the accumulated charge.
Reference numeral 303 denotes a switch transistor for controlling the averaging of the accumulated charge of the capacitor 109 ′ and the accumulated charge of the capacitor 109 ″, and 304 denotes the accumulated charge of the capacitor 110 ′ and the accumulated charge of the capacitor 110 ″. This is a switch transistor for controlling averaging.

If the switch transistors 301, 301 ′, and 303 are interlocked, they are configured to average the accumulated charges of the capacitors 109, 109 ′, 109 ″, and 109 ″ ′. That is, for example, if the switch transistor 303 is turned on after or when the switch transistors 301 and 301 ′ are turned on, they are stored in the capacitors 109, 109 ′, 109 ″, and 109 ″ ′. Average charge.
If the switch transistors 302, 302 ′, and 304 are interlocked, they are configured to average the accumulated charges of the capacitors 110, 110 ′, 110 ″, and 110 ″ ′. That is, for example, if the switch 304 is turned on after or when the switch transistors 302 and 304 ′ are turned on, they are stored in the capacitors 110, 110 ′, 110 ″, and 110 ″ ′. Is averaged.

  FIG. 9 is a timing chart showing the operation timing of the CMOS sensor 710 of FIG. Hereinafter, the operation of the CMOS sensor 710 will be described with reference to FIGS.

First, at timing T401, when the start pulse of the vertical scanning circuit 1 input to the terminal 2 and the vertical scanning pulse input to the terminal 3 become HIGH, the vertical scanning circuit 1 starts scanning. The first line (R1) is selected. At this time, the reset pulse input to the terminal 8 is HIGH, thereby resetting the floating diffusion region of the pixel portion.
Next, at timing T402, when the reset pulse input to the terminal 8 becomes LOW (falls), the floating diffusion region of the pixels in the first row becomes electrically floating.
Next, when the input pulse to the terminal 9 becomes HIGH at timing T403, the charge is transferred from the photodetecting element in the first row to the floating diffusion region.
Next, when the input pulses to the terminals 10 and 50 become HIGH at timing T404, a voltage proportional to the amount of light detected by the light detection element in the first row is applied to the capacitor 109 by the amplifier transistor 104. Read out.
Next, at timing T405, the vertical scanning pulse input to the terminal 3 becomes LOW (falls).
Next, when the vertical scanning pulse input to the terminal 3 becomes HIGH again at timing T406, the second row (R2) is selected.
Next, at timing T407, when the reset pulse input to the terminal 8 becomes LOW (falls), the floating diffusion region of the pixel in the second row becomes electrically floating.
Next, at the timing T408, as in the above-described timing T403, when the input pulse to the terminal 9 becomes HIGH, the charge is transferred from the photodetecting element in the second row to the floating diffusion region.
Next, also at the timing T409, as in the above-described timing T404, when the input pulses to the terminals 10 and 51 are each HIGH, a voltage proportional to the amount of light detected by the light detection element in the second row is Data is read out from the capacitor 110 by the amplifier transistor 104 in the second row.
Next, when the input pulses to the terminals 60, 61, 90, 91 become HIGH at timing T410, the accumulated charges in the capacitors 109, 109 ′, 109 ″, 109 ″ ′ are averaged on the line memory, and The accumulated charges in the capacitors 110, 110 ′, 110 ″, 110 ″ ′ are averaged.

  At timing T411, the horizontal scanning circuit 4 starts operating, and the averaged voltages are sequentially output in the horizontal direction.

  Here, since the purpose is to output the luminance signal Y, in the circuit configuration of FIG. 8, the output from the CMOS sensor 710 is only one system of the terminal 16. For example, FIG. Similar to the circuit configuration shown in FIG. 6, the color difference signals CB and CR can be obtained by providing a plurality of terminals (output lines) from which output is performed as the CMOS sensor 710.

(Fifth embodiment)
In the third embodiment described above, the circuit configuration of the CMOS sensor 710 is as shown in FIG. 6, but in this embodiment, for example, it is configured as shown in FIG.
In the circuit configuration of FIG. 10, the same reference numerals are given to the portions that operate in the same manner as the circuit configuration of FIG. 6, and detailed description thereof is omitted.

  The CMOS sensor 710 here outputs an output series that outputs an average value of the detected light amounts of the two light detecting elements adjacent in the vertical direction or an average value of the detected light amounts of the four light detecting elements adjacent in the vertical direction. I have. With such a circuit configuration, the CMOS sensor 710 can perform the reading method 2 and the reading method 21.

Specifically, in FIG. 10, first, 501, 502, 503, and 504 are distribution transistors that supply the current supplied from the transistor 104 to the corresponding capacitors 508, 509, 510, and 511, respectively.
Reference numeral 508 denotes a capacitor for accumulating charges in the photodetecting elements in the first row (R1). Reference numeral 509 denotes a capacitor for accumulating charges in the photodetecting elements in the second row (R2). Yes, 510 is a capacitor for accumulating charges in the photodetecting elements in the third row (R3), and 511 is for accumulating charges in the photodetecting elements in the fourth row (R4). It is a capacitor.
Reference numeral 505 denotes a switch transistor for controlling the averaging between the accumulated charge of the capacitor 508 and the accumulated charge of the capacitor 509. Reference numeral 506 controls the averaging of the accumulated charge of the capacitor 509 and the accumulated charge of the capacitor 510. 507 is a switch transistor for controlling the averaging of the accumulated charge of the capacitor 510 and the accumulated charge of the capacitor 511.

  Note that if the switches 505, 506, and 507 are linked, they are configured to average the accumulated charges of the capacitors 508, 509, 510, and 511. That is, for example, if the switch transistor 506 is turned on after or when the switch transistors 505 and 507 are turned on, these average the accumulated charges of the capacitors 508, 509, 510, and 511. .

  FIG. 11 is a timing chart showing the operation timing of the CMOS sensor 710 of FIG. Hereinafter, the operation of the CMOS sensor 710 will be described with reference to FIGS.

First, at timing T601, when the start pulse of the vertical scanning circuit 1 input to the terminal 2 and the vertical scanning pulse input to the terminal 3 become HIGH, the vertical scanning circuit 1 starts scanning, and the first The line (R1) is selected. At this time, the reset pulse input to the terminal 8 is HIGH, thereby resetting the floating diffusion region of the pixel portion.
Next, at timing T602, when the reset pulse input to the terminal 8 becomes LOW (falls), the floating diffusion region of the pixels in the first row is in an electrically floating state.
Next, when the input pulse to the terminal 9 becomes HIGH at timing T603, charge transfer from the first row of photodetecting elements to the floating diffusion region is performed.
Next, when the input pulses to the terminals 10 and 30 become HIGH at timing T604, charges proportional to the amount of light detected by the light detection elements in the first row are accumulated in the capacitor 508.
Next, at timing T605, the vertical scanning pulse input to the terminal 3 becomes LOW (falls).
Next, when the vertical scanning pulse input to the terminal 3 becomes HIGH again at timing T606, the second row (R2) is selected.
Next, at timing T607, when the reset pulse input to the terminal 8 becomes LOW (falls), the floating diffusion region of the pixel in the second row is electrically floated.
Next, at timing T608, as in the above-described timing T603, when the input pulse to the terminal 9 becomes HIGH, charges are transferred from the photodetecting elements in the second row to the floating diffusion region.
Next, also at the timing T609, as in the above-described timing T604, when the input pulses to the terminals 10 and 31 are each HIGH, the electric charge proportional to the amount of light detected by the light detection element in the second row is changed to the capacitor. 509 is accumulated.
Similarly, when the input pulses to the terminals 10 and 32 become HIGH at timing T610, charges proportional to the amount of light detected by the light detection elements in the third row are accumulated in the capacitor 510.
Furthermore, when the input pulse to the terminals 10 and 33 becomes HIGH at timing T611, electric charge proportional to the amount of light detected by the light detection elements in the fourth row is accumulated in the capacitor 511.
At timing T612, when the input pulse to the terminals 40 and 41 becomes HIGH, the charges accumulated in the capacitors 508, 509, 510, and 511 are averaged on the line memory.
At timing T613, the horizontal scanning circuit 4 starts operating from this timing, and the averaged voltages are sequentially output in the horizontal direction. Therefore, from the output terminal 70, a voltage proportional to the average value of the detected light amount of the light detection elements from the first row to the fourth row (R1, R2, R3, R4) is sequentially output in the column direction.

  In the circuit configuration shown in FIG. 10, the average value of the first column (C1) and the second column (C2) is obtained from the output terminal 70 by configuring so as not to perform the averaging by the switch transistor 506. The average value of the third column (C3) and the fourth column (C4) can be obtained from the output terminal 71, the average value of the first column (C1) and the second column (C2), and the third value The difference between the average value of the column (C3) and the fourth column (C4) can be obtained from the output terminal 72.

Further, in the operation of the CMOS sensor 710 described with reference to FIGS. 6 to 11, the reset voltage of the diffusion floating region may be read out to another line memory before reading out the charge (optical signal) of the pixel. Good. By taking the difference between the reset voltage and the optical signal output, variation in output voltage due to variation in threshold voltage of the transistor 104 can be eliminated. Therefore, it is possible to obtain a signal having a high S / N ratio in which a noise component due to variation is not mixed in the signal based on the light amount detected by each light detection element.
Further, the vertical and horizontal scans may be thinned out for each block or for a plurality of blocks. As a result, a further compressed signal is obtained.
Further, the present invention can be applied not only to a CMOS sensor but also to other types of photoelectric conversion elements, and in this case as well, the same effects as those of the above embodiments can be obtained.

(Sixth embodiment)
In each of the above-described embodiments, when reading a signal in units of pixel blocks, the pixel block to be read next is closely adjacent to the pixel block that has been previously read. In the embodiment, the pixel block to be read next is shifted by, for example, one pixel with respect to the pixel block that has been previously read (overlap reading).

  For example, FIG. 12 shows an outline of the most characteristic circuit configuration of the imaging unit 710 in the present embodiment when a signal is read out in units of 4 pixel blocks in the imaging device 700 of FIG. FIG. 13 is a diagram illustrating the operation timing of each unit of the imaging unit 710.

As shown in FIGS. 12 and 13, in the imaging unit 710 here, first, when the gate pulse S ₁ is given at the timing when the vertical scanning pulse V ₁ is given, the pixel Y ₁₁ signal (Y ₁₂ signal is also shown). The same is read out and held in the memory capacitor C ₁ . Further, when the gate pulse S ₂ is applied at the timing when the vertical scanning pulse V ₂ is applied, the pixel Y ₂₁ signal (pixel Y ₂₂ signal) is read out and similarly held in the memory capacitor C ₂ .

Next, when the gate pulse H ₁ is given, the pixel Y ₁₁ signal and the pixel Y ₂₁ signal held in the memory capacitor C ₁ are read out.
The pixel Y ₁₂ signal and the pixel Y ₂₂ signal are also read out in the same manner as described above, the pixel Y ₁₁ signal is held in the memory capacitor C ₃ , the pixel Y ₁₂ signal is held in the memory capacitor C ₄ , and the pixel Y _{The 21} signal is held in the memory capacitor C ₅ , and the pixel Y ₂₂ signal is held in the memory capacitor C ₆ .

Then, when the gate pulse T ₁ is given at the time point t ₁ , the pixel signals held in the memory capacitors C _{3 to} C ₆ are simultaneously output, and the pixel signals Y ₁₁ , Y ₁₂ , Y ₂₁ , and Y ₂₂ are output. Output in parallel.

Next, when the gate pulse T ₂ is given at time t ₂ , pixel signals Y ₁₃ , Y ₁₂ , Y ₂₃ , and Y ₂₂ for one block shifted in the horizontal direction by one pixel are output in parallel. In this case, the pixel signals held in the memory capacitors C ₄ and C ₆ are output as the pixel signal Y ₁₂ and the pixel signal Y ₂₂ , respectively.

As described above, in the present embodiment, when the reading of the next pixel block B2 is in front of the reading of the pixel blocks B1, pixel signals Y ₁₂ and Y stored in the memory capacitor C _4, C _{6 22} (signal obtained from the pixel existing in the overlapping portion of each pixel block B1 and B2) is read out, and the pixel block B2 read next to the pixel block B1 is set to 1 in the horizontal direction with respect to the pixel block B1. Signals are sequentially read out in units of pixel blocks as blocks shifted in the horizontal direction by pixels.

By configuring in this way, the following effects can be obtained.
First, in each of the above-described embodiments, as illustrated in FIG. 14, the pixel block B2 ′ to be read next is configured so as to be closely adjacent to the pixel block B1 ′ that has been previously read. From the pixel signal obtained by reading, every other luminance signal Y and color difference signals CB and CR are obtained in the horizontal direction or the vertical direction for each pixel on the image sensor. On the other hand, in the present embodiment, as shown in FIG. 15, since the pixel block B2 to be read next is configured so as to correspond to the previous pixel block 1, an image sensor is obtained from the pixel signal obtained by this reading. The luminance signal Y and the color difference signals CB and CR corresponding to the upper horizontal pixels are obtained. Therefore, the horizontal resolution can be doubled.
Also, two of the four holding capacitors C ₃ , C ₄ , C ₅ , and C ₆ (see the arrow in FIG. 12) are used as holding means for holding the previous readout pixel. Since the pixel signal held in the holding means is output as the pixel signal of the next pixel block to be read out, the pixel block unit output is shifted by one pixel without providing an external memory. Can be obtained.
Furthermore, since it is configured to obtain an output in units of pixel blocks shifted by one pixel inside the imaging unit 710 (on the same element), this output can be performed at high speed.

Here, since a 2 × 2 pixel block is used as a readout unit, two of the four holding capacitors are provided as means for holding the previous readout pixel signal. If the 4-pixel block is a readout unit, 12 capacitors may be provided as the holding means. That is, by providing N × (N−1) holding means for the reading unit of the N × N pixel block, N × N pixel blocks are read in the same manner as the reading of the 2 × 2 pixel block unit described above. Reading in units of pixel blocks can be performed while shifting one pixel in the horizontal direction.
Also, in the vertical direction, if the signal is read in units of pixel blocks while shifting one pixel by the same configuration as in the horizontal direction, the resolution in the vertical direction can be increased.
Furthermore, if it is configured to have a noise removal function, a better image signal can be obtained.

Here, FIG. 16 and FIG. 18 show an example ([Example 1], [Example 2]) of the circuit configuration of the imaging unit 710 that is a more specific embodiment of the present embodiment as described above. 17 and 19 show operation timings in the circuits of FIGS. 16 and 18 described above.
Hereinafter, circuit configurations and operations in [Example 1] and [Example 2] will be sequentially described.

  16 and 18 show the circuit configuration when the imaging unit 710 is applied to a so-called CMOS sensor, as in the third to fifth embodiments described above (hereinafter, also here). The “imaging unit 710” is referred to as “CMOS sensor 710”). For this reason, in each circuit configuration of FIG. 16 and FIG. 18, the same reference numerals are given to portions that operate in the same manner as the circuit configuration of FIG. 6, and detailed description thereof will be omitted.

[Example 1]
The CMOS sensor 710 shown in FIG. 16 can realize the calculation of the pixel signal in the vertical direction to obtain the color difference signal CB. The first column, the second column, the second column, the third column The third column, the fourth column,... Are configured so that the above-described calculation can be performed for each column.

Therefore, in the CMOS sensor 710 here, first, the reset pulse input to the terminal 8 is in a HIGH state. In this state, when the start pulse of the vertical scanning circuit 1 input to the terminal 2 and the scanning pulse of the vertical scanning circuit 1 input to the terminal 3 become HIGH at timing T711, the floating diffusion region of the pixel portion. Is reset. Thereby, the vertical scanning circuit 1 starts scanning, and the first row (R1) is selected. Further, the input pulse to the terminal 11 is also in a HIGH state, and in this state, when each input pulse to the terminals 30 and 31 becomes HIGH, capacitors 221, 222, 223, and 224 that function as line memories (each line , Which corresponds to the capacitors C ₃ , C ₄ , C ₅ , and C ₆ in FIG. 12 (hereinafter referred to as “line memory”).
Next, when the reset pulse input to the terminal 8 becomes LOW at timing T712, the floating diffusion region of the pixels in the first row is thereby electrically floated. Further, the line memory is also floated.
Next, at timing T713, when the input pulses to the terminals 9 and 11 become HIGH, charges are transferred from the photodetecting elements in the first row to the floating diffusion region.
Next, when the input pulses to the terminals 10, 30, and 50 become HIGH at timing T 714, the charge in the first row transferred at timing T 713 is read out to the line memory.
Next, at timing T715, when the vertical scanning pulse input to the terminal 3 is in the LOW state, the second row (R2) is selected.
Next, when the reset pulse input to the terminal 8 becomes LOW at timing T716, the floating diffusion region of the pixels in the second row is thereby electrically floated. Further, the line memory is also floated.
Next, at timing T717, when the input pulses to the terminals 9 and 11 become HIGH, charges are transferred from the photodetecting elements in the second row to the floating diffusion region.
Next, when the input pulses to the terminals 10, 30, and 50 become HIGH at timing T 718, the charge in the second row transferred at timing T 717 is transferred to another line memory (what is a line memory at timing T 714? To another line memory).
Next, when the input pulse to the terminal 40 becomes HIGH at timing T719, mixed signals for two rows of the first row and the second row are generated on the line memory.
At timing T720, the horizontal scanning circuit 4 starts operating from this timing, and the mixed signals in the line memory are sequentially read out. At this time, two adjacent mixed signals are simultaneously output from the terminals 70 and 71 via two horizontal output lines, respectively, and at the same time, are subtracted by a differential circuit (not shown) in the subsequent stage. The color difference signal is directly output from the CMOS sensor 710. Further, if the signals of the two horizontal output lines are added by an adder (not shown) at the subsequent stage, a luminance signal can be obtained.

[Example 2]
The CMOS sensor 710 shown in FIG. 18 can realize the calculation of the pixel signal in the vertical direction to obtain the color difference signal CR. The first column, the second column, the second column, the third column The third column, the fourth column,... Are configured so that the above-described calculation can be performed for each column.

Therefore, in the CMOS sensor 710 here, first, the reset pulse input to the terminal 8 is in a HIGH state. In this state, when the start pulse of the vertical scanning circuit 1 input to the terminal 2 and the scanning pulse of the vertical scanning circuit 1 input to the terminal 3 become HIGH at timing T731, the floating diffusion region of the pixel portion. Is reset. Thereby, the vertical scanning circuit 1 starts scanning, and the first row (R1) is selected. In addition, the input pulse to the terminal 11 is also in a HIGH state, and in this state, when each input pulse to the terminals 50 and 51 becomes HIGH, capacitors 225, 226, 227, and 228 (each line functioning as a line memory). , Which corresponds to the capacitors C ₃ , C ₄ , C ₅ , and C ₆ in FIG. 12 (hereinafter referred to as “line memory”).
Next, at timing T732, when the reset pulse input to the terminal 8, the input pulse to the terminal 11, the input pulse to the terminal 50, and the input pulse to the terminal 51 become LOW, respectively, the first row The floating diffusion region of the eye pixel is electrically floating, and the line memory is also floating.
Next, when the input pulses to the terminals 9 and 11 become HIGH at timing T733, charge transfer from the first row of photodetecting elements to the floating diffusion region is performed.
Next, when the input pulses to the terminals 10 and 50 become HIGH at timing T734, the charge in the first row transferred at timing T733 is read out to the line memory.
Next, when the vertical scanning pulse input to the terminal 3 is in the LOW state at timing T735, the second row (R2) is selected.
Next, at timing T736, when the reset pulse input to the terminal 8 and the input pulse to the terminal 11 are respectively LOW, the floating diffusion region of the pixels in the second row is thereby brought into an electrically floating state. In addition, the line memory is also in a floating state.
Next, at timing T737, when the input pulses to the terminals 9 and 11 become HIGH, charges are transferred from the photodetecting elements in the second row to the floating diffusion region.
Next, when the input pulses to the terminals 10 and 51 become HIGH at timing T738, the charge in the second row transferred at timing T737 is transferred to another line memory (different from the line memory at timing T734). Line memory).
Next, when the input pulses to the terminals 60 and 61 become HIGH at timing T739, mixed signals for two rows of the first row and the second row are generated on the line memory.
At timing T740, the horizontal scanning circuit 4 starts operating from this timing, and the mixed signals in the line memory are sequentially read out. At this time, two adjacent mixed signals are simultaneously output from the terminals 70 and 71 via two horizontal output lines, respectively, and at the same time, are subtracted by a differential circuit (not shown) in the subsequent stage. The color difference signal is directly output from the CMOS sensor 710. Further, if the signals of the two horizontal output lines are added by an adder (not shown) at the subsequent stage, a luminance signal can be obtained.

In the above-mentioned [Example 1] and [Example 2], the specific example for realizing the calculation of the pixel signal in the vertical direction has been described, but the same configuration is also applied to the calculation of the pixel signal in the horizontal direction. The above calculation for each row can be performed as in the first row, the second row, the second row, the third row, the third row, the fourth row, and so on.
Further, if the circuit configuration shown in FIG. 20 is configured using the circuit configuration shown in FIG. 16 and the circuit configuration shown in FIG. 18, for example, and the operation timing thereof is as shown in FIG. CB and the color difference signal CR can be output simultaneously, and a luminance signal can also be output simultaneously.

Further, in the operation of the CMOS sensor 710 described with reference to FIGS. 16 to 21, the reset voltage of the diffusion floating region may be read out to another line memory before reading out the charge (optical signal) of the pixel. Good. By taking the difference between the reset voltage and the optical signal output, variation in output voltage due to variation in threshold voltage of the transistor 104 can be eliminated. Therefore, it is possible to obtain a signal having a high S / N ratio in which a noise component due to variation is not mixed in the signal based on the light amount detected by each light detection element.
Further, the present invention can be applied not only to a CMOS sensor but also to other types of photoelectric conversion elements, and in this case as well, the same effects as those of the above embodiments can be obtained.

(Seventh embodiment)
First, in general, in an image signal processing system that handles a color image signal, first, a predetermined preprocessing is performed on an original signal (pixel signal) output from an image sensor, and a pseudo luminance signal Y ′ and a color difference are processed. Signals U ′ (CR) and V ′ (CB) are generated. Then, using the pseudo luminance signal Y ′ and the pseudo color difference signals U ′ (CR) and V ′ (CB), the color information is subjected to white balance correction, γ correction, other color correction processing, etc. A signal Y and color difference signals U and V are generated. The luminance signal Y and the color difference signals U and V are used for image processing such as compression processing and expansion processing, screen display, and recording on a recording medium.

Therefore, in the present embodiment, the present invention is applied to an image signal processing system that handles color image signals as described above.
For example, as shown in FIG. 22, the image signal processing system to which the present invention is applied includes a transmission side (imaging device) including an imaging element IC chip 810 and a compression processing device 820, an expansion processing device 830, a color processing device 840, In addition, data communication is performed with a receiving side (reproducing device) including the display / recording device 850.

The most characteristic feature of such an image signal processing system 800 resides in the image sensor IC chip 810.
That is, the imaging element IC chip (hereinafter referred to as “imaging unit”) 810 has a function similar to that of the imaging device 700 of FIG. 1 and includes the color filter 711, the imaging element 712, and the like shown in FIG. A signal reading unit 713 and a luminance / color difference preprocessing unit 720 are provided, and these are provided on the same image sensor IC chip.
Hereinafter, the operation of the image signal processing system 800 will be described.

First, on the transmission side, in the imaging unit 810, the subject light is incident on the imaging element (light receiving element) 712 via the color filter configured by the filter array as shown in FIGS.
The pixel signals (original signals) Ye, Cy, Mg, G obtained by the image sensor 712 are read by the signal reading unit 713 as described in the first to seventh embodiments. The luminance / color difference pre-processing unit 720 generates a pseudo luminance signal Y ′ and color difference signals CR ′ (U ′), CB ′ (V ′) from the read original signals Ye, Cy, Mg, G.

The compression processing device 820 does not perform white balance or γ correction on the color information with respect to the pseudo luminance signal Y ′ and the pseudo color difference signals U ′ and V ′ generated by the imaging unit 810, and performs JPEG, MPEG, and H . 261, compression processing using an information compression technique such as vector quantization is performed.
Specifically, for example, first, DCT (Discrete Cosine Transform) is performed for each predetermined pixel block. Next, the DCT pseudo luminance signal Y ′ and pseudo color difference signals U ′ and V ′ are quantized to obtain compressed data from which high frequency components have been removed. Then, the compressed data is encoded. Various methods can be used for the encoding process here, but it is possible to further increase the compression rate by using variable-length encoding that assigns a code length according to the frequency of data generation.
In this way, the data compressed and encoded by the compression processing device 820 is transmitted to the decompression processing device 830 on the receiving side via a medium such as a communication line.

  On the reception side, the information decompression device 830 performs the reverse processing of the DCT and the quantization processing in the compression processing device 820 on the data sent from the transmission side, thereby performing the pseudo luminance signal Y ′ and the pseudo luminance signal Y ′. The color difference signals U ′ and V ′ are restored.

  The color processing device 840 provides good image quality such as color processing such as white balance correction and γ correction for the pseudo luminance signal Y ′ and the pseudo color difference signals U ′ and V ′ obtained by the information decompression device 830. The luminance signal Y and the color difference signals U and V are generated by performing various corrections necessary for obtaining.

  The display / recording device 850 displays the luminance signal Y and the color difference signals U and V generated by the color processing device 840 on a screen or records them on a recording medium.

  As described above, in the present embodiment, the color filter 711 is configured by the filter arrangement as illustrated in FIGS. 2 to 5, and the pixel signals obtained by the image sensor 712 are converted into the first to first pixels described above. As described in each of the seventh embodiment, the reading is performed. As a result, even in an image processing system that handles color image signals, the effects of the first to seventh embodiments described above can be obtained. That is, a high-resolution color image signal in the horizontal direction and the vertical direction can be obtained at high speed.

  Further, a luminance / color difference pre-processing unit 720 for obtaining the pseudo luminance signal Y ′ and the pseudo color difference signals U ′, V ′ from the original signals Ye, Cy, Mg, G obtained by the image sensor 712 is combined with the image sensor 712. It was configured to be provided on the same chip. Thereby, it is possible to speed up the arithmetic processing for obtaining the pseudo luminance signal Y ′ and the pseudo color difference signals U ′ and V ′. Therefore, image processing such as subsequent compression processing and expansion processing can be performed efficiently.

  With the above-described configuration, the processing speed can be increased to improve the accuracy of image processing such as compression processing and decompression processing, and a higher-quality color image signal can be obtained.

  Further, the pseudo luminance signal Y ′ and the pseudo color difference signals U ′ and V ′ obtained by the imaging unit 810 are compressed and encoded by the compression processing device 820 and decompressed by the decompression processing device 830. Later, the color processing device 840 is configured to perform color processing such as white balance correction and γ correction in order to obtain high quality image quality. As a result, it is possible to minimize deterioration in image quality due to block noise and high-frequency noise that occur in association with decompression processing after compression processing. Therefore, it is possible to greatly reduce the amount of information when the display / recording device 850 stores the data in a recording medium or transmits the data via a communication line, and prevents the deterioration of the image quality as much as possible, and the high quality image quality. You can also get

In the seventh embodiment described above, DCT and variable length coding are used as the compression processing in the compression processing device 820. However, the present invention is not limited to this. For example, a codebook compression technique (vector quantization technique) is used. Can also be used.
When using this code book compression technique, first, the compression processing device 820 includes a pseudo luminance signal Y ′ and pseudo color difference signals U ′ and V ′ obtained by the imaging unit 810, and a plurality of code books stored in advance. Compared with (Pattern), the most similar pattern is found, and the code number corresponding to the pattern is transmitted to the decompression processing device 830 on the receiving side.
The decompression processing device 830 reproduces the image signal compressed on the compression processing device 820 side by extracting a plurality of codebooks stored in advance from the pattern corresponding to the code number sent from the compression processing device 820. .

In each of the above embodiments, the color filter 711 is a complementary color filter in which four color filters of complementary colors of yellow (Ye), magenta (Mg), cyan (Cy), and green (G) are arranged. However, the present invention is not limited to this, and other filters may be used as long as a luminance signal or a color difference signal can be obtained.
However, for example, when a primary color filter composed of three color filters of R, G, and B is used as the color filter 711, and composed of four color filters as in the above embodiments. Comparing with the case where the complementary color filter is used as the color filter 711, since each of the color filters constituting the complementary color filter has a color component, the luminance false signal is smaller when the latter complementary color filter is used. Furthermore, it is possible to prevent the disadvantage that the utilization rate of incident light is low and the sensitivity is low. Therefore, a higher effect can be obtained by using the complementary color filter.

According to the embodiment described above, the filter arrangement of the color filter array in which the filters of a plurality of colors are arranged with a periodicity of N rows × N columns is arranged in the cycle unit of N rows × N columns. The colors of the filters in the same row are all different, and the colors of the filters in the same column are all different.
As a result, when obtaining a luminance signal and a color difference signal using a pixel signal of the pixel block in a predetermined pixel block unit, a pixel signal necessary for obtaining the luminance signal and the color difference signal is obtained from any pixel block. Therefore, a high-resolution luminance signal and color difference signal can be obtained. By using this for simple color display, autofocus detection signal, white balance adjustment, image compression processing, or the like, a good color image can be provided.
For example, a color in which four types of filters of the first color to the fourth color (cyan Cy, yellow Ye, magenta Mg, green G, etc.) are arranged with a periodicity of 4 rows × 4 columns. In the case of a filter array, in the cycle unit of 4 rows × 4 columns,
1st line: 1st color, 2nd color, 3rd color, 4th color 2nd line: 3rd color, 4th color, 1st color, 2nd color 3rd line Row: 2nd color, 1st color, 4th color, 3rd color 4th row: 4th color, 3rd color, 2nd color, 1st color, etc. The filters are arranged so that filters of the same color do not overlap in both the row direction and the column direction.
Thereby, when obtaining a luminance signal and a color difference signal using pixel signals of the pixel block in units of pixel blocks of 2 rows × 2 columns, each pixel of the first color to the fourth color is obtained from any pixel block. Since the signal is obtained, that is, the pixel signal necessary for the calculation for obtaining the luminance signal and the color difference signal is obtained, the calculation for obtaining the luminance signal and the color difference signal is performed for a certain row as in the conventional case. There is no such thing as impossible.
Further, when the pixel signal obtained from the subject light through the color filter array is read out in units of pixel blocks of N rows × N columns, the pixel block to be read out next is compared with the pixel block that has been read out last time. Thus, if the block is read out as a block at a position shifted by a predetermined number of pixels in the horizontal direction, the vertical direction, or the vertical direction together with the horizontal direction, the resolution can be further increased.
For example, when reading out in units of pixel blocks of 2 rows × 2 columns, first, as a result of reading out the first pixel block, the pixel signal Y ₁₁ in the first row and the first column, the first row and the second column Pixel signal Y ₁₂ , second row first column pixel signal Y ₂₁ , and second row second column pixel signal Y ₂₂ are obtained. Of these pixel signals, an area (first row, second column, and column) overlapping with a second pixel block to be read next (a block at a position shifted in the horizontal direction by one pixel with respect to the first pixel block). The pixel signals Y ₁₂ and Y ₂₁ in the second row and second column) are held. Then, the held pixel signals Y ₁₂ and Y ₂₁ are used as pixel signals for the overlapping region for reading the next pixel block.
As a result, when a luminance signal and a color difference signal are obtained using a pixel signal of the pixel block in units of 2 × 2 pixel blocks, a luminance signal and a color difference signal corresponding to each pixel in the horizontal direction can be obtained. it can. Therefore, in this case, the horizontal resolution can be doubled.
Further, for example, in the pixel block unit of 2 rows × 2 columns, the addition signal of the pixel signal of the first row and the first column and the pixel signal of the first row and the second column, the pixel signal of the second row and the first column, and the second The difference between the pixel signal of the second row and the second column and the sum signal of the pixel signal of the first row and the first column, the sum signal of the pixel signal of the first row and the first column, the pixel signal of the second row and the first column, and the pixel signal of the first row and the second column And a mode in which the difference between the pixel signal of the second row and the second column and the addition signal of the pixel signals in the second row (reading 1 mode) is used, a color image signal having a color difference signal having a high resolution in both the horizontal direction and the vertical direction can be obtained. can get.
Also, according to the above-described embodiment, there are various readout methods while using the same color filter pattern, so that a mode that can output an image signal that can be used for simple color display, autofocus, and auto white balance at high speed. And a multi-mode such as a mode for outputting a high-resolution image signal.
Further, according to the above-described embodiment, pseudo luminance confidence and color difference signals are obtained from the image pickup device by the above-described readout, and further compressed and decompressed without performing color processing such as white balance and γ correction, and then the above-mentioned after decompression. Since color processing is performed, compression can be performed with high efficiency, and deterioration in image quality after color processing can be prevented.

Another object of the present invention is to supply a storage medium storing software program codes for realizing the functions of the host and terminal of each of the above-described embodiments to a system or apparatus, and the computer (or CPU) of the system or apparatus. Needless to say, this can also be achieved by reading and executing the program code stored in the storage medium.
In this case, the program code itself read from the storage medium implements the functions of the respective embodiments, and the storage medium storing the program code constitutes the present invention.

  As a storage medium for supplying the program code, a ROM, a floppy (registered trademark) disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, or the like is used. it can.

  Further, by executing the program code read by the computer, not only the functions of the embodiment are realized, but also the OS or the like running on the computer performs the actual processing based on the instruction of the program code. Needless to say, the present invention includes a case where part or all of the functions are performed and the functions of the respective embodiments are realized by the processing.

  Further, after the program code read from the storage medium is written to the memory provided in the extension function board inserted in the computer or the function extension unit connected to the computer, the function extension is performed based on the instruction of the program code. It goes without saying that the case where the CPU or the like provided in the board or function expansion unit performs part or all of the actual processing and the functions of the respective embodiments are realized by the processing.

1 is a block diagram illustrating a configuration of an imaging apparatus to which the present invention is applied in a first embodiment. It is a figure for demonstrating an example of the filter arrangement | sequence of the color filter used with the said imaging device. It is a figure for demonstrating another example of the said filter arrangement | sequence. In 2nd Embodiment, it is a figure for demonstrating an example of the filter arrangement | sequence of the color filter used with the said imaging device. It is a figure for demonstrating another example of the said filter arrangement | sequence. In 3rd Embodiment, it is a figure for demonstrating the circuit structure of the said imaging device which implements the reading method of the pixel signal in the said 1st and 2nd embodiment. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. In 4th Embodiment, it is a figure for demonstrating the circuit structure of the said imaging device which implements the reading method 1 of the pixel signal in the said 1st and 2nd embodiment. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. FIG. 16 is a diagram for describing a circuit configuration of the imaging apparatus that performs the pixel signal readout method 1 in the first and second embodiments in the fifth embodiment. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. FIG. 16 is a diagram for describing a circuit configuration of the imaging apparatus that performs overlap reading in the sixth embodiment. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. It is a figure for demonstrating the luminance signal and color difference signal which are obtained from the pixel signal in the case of not performing the said overlap reading. It is a figure for demonstrating the luminance signal and color difference signal which are obtained from the pixel signal at the time of performing the said overlap reading. It is a figure for demonstrating an example (example 1) of the circuit structure which actualized the said imaging device more. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. It is a figure for demonstrating an example (example 2) of the circuit structure which actualized the said imaging device more. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. It is a figure for demonstrating the circuit structure which has both functions of each circuit structure of the said Example 1 and Example 2. FIG. It is a figure for demonstrating operation | movement of the imaging device with the said circuit structure. In 7th Embodiment, it is a block diagram which shows the structure of the image processing system to which this invention is applied. It is a figure for demonstrating the filter arrangement | sequence of the conventional color filter which has a periodicity of 2 pixels in a horizontal direction, and 4 pixels in a vertical direction. It is a figure for demonstrating the filter arrangement | sequence of the conventional color filter which has a periodicity of 2 pixels in a horizontal direction, and 8 pixels in a vertical direction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Photodetection element 101 Transfer transistor 102 Diffusion floating area 103 Reset transistor 104 Ambo transistor 106,107,501,502,503,504 Distribution transistor 121 Switch transistor 105 Discharge transistor 108,115,1 '. 16, 301, 302, 303, 304, 505, 506, 507 Averaging transistor 109, 110, 117, 118 Capacitor (line memory)
122, 127 Differential amplifier 700 Imaging device 710 Imaging unit 711 Color filter 712 Imaging element 713 Signal reading unit 720 Brightness / color difference preprocessing unit

Claims (7)

Light receiving means for receiving subject light via a predetermined color filter array at a plurality of pixels arranged two-dimensionally and obtaining a pixel signal corresponding to each light reception amount at the plurality of pixels;
Reading means for reading out pixel signals from the light receiving means in units of pixel blocks of N rows × N columns (N: integer) while shifting at least a predetermined pixel in either the row direction or the column direction;
Holding means for holding a pixel signal of a region overlapping with a next read pixel block shifted by a predetermined pixel with respect to the read pixel block, out of the pixel signals of the current read pixel block read by the reading means; ,
The image pickup device, wherein the reading unit uses a pixel signal held in the holding unit for reading a pixel signal of a next pixel block.   In the color filter array, filters of a plurality of colors are arranged corresponding to the plurality of pixels with a periodicity of N rows × N columns (N: integer), and each row in the periodic unit of N rows × N columns and 2. The image pickup device according to claim 1, wherein filters are arranged so that no filter of the same color exists in each column. An image processing apparatus that images a subject with a color image sensor and performs predetermined image processing on an output signal of the image sensor,
The image processing apparatus according to claim 1, wherein the color image sensor is an image sensor according to claim 1. An image processing system in which an image pickup apparatus including a color image pickup device and an image processing apparatus that performs predetermined image processing on an output signal of the image pickup apparatus are connected.
The image processing system according to claim 1, wherein the color image sensor is the image sensor according to claim 1. Processing steps for performing the operation of the image sensor that receives subject light through the color filter array with a plurality of pixels arranged two-dimensionally and obtains pixel signals corresponding to the respective amounts of light received by the plurality of pixels are performed by a computer A computer-readable storage medium storing a program to be executed by the computer,
The processing steps are:
A reading step of reading out pixel signals in the light receiving unit in units of pixel blocks of N rows × N columns (N: integer) while shifting by a predetermined pixel in at least one of the row direction and the column direction;
Of the pixel signals of the current readout pixel block read out by the readout step, the pixel signal of the region overlapping with the next readout pixel block shifted by a predetermined pixel with respect to the readout pixel block is held in the memory in the element. Holding step,
The computer-readable storage medium, wherein the reading step includes a step of using a pixel signal held in the memory for reading a pixel signal of a next pixel block.   In the color filter array, filters of a plurality of colors are arranged corresponding to the plurality of pixels with a periodicity of N rows × N columns (N: integer), and each row in the periodic unit of N rows × N columns and 6. The computer-readable storage medium according to claim 5, wherein filters of the same color are arranged so as not to exist in each column. An image pickup apparatus that includes an image pickup element and that compresses and transmits an information amount of an image signal output from the image pickup element without undergoing color processing for performing at least white balance correction or γ correction for color information, and the compressed image An image processing system connected to a receiving device configured to perform color processing after receiving and expanding a signal,
The image processing system according to claim 1, wherein the image sensor is the image sensor according to claim 1.

Images