JP3925479B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging technique capable of capturing still images and moving images, and is particularly effective for capturing high-quality still images and moving images.

A conventional imaging device that performs photoelectric conversion using an imaging device such as a CCD (Charge Coupled Device) and performs digital signal processing on this to obtain a predetermined digital image signal is a video camera that captures a moving image and a still image. It can be classified as a digital still camera for imaging.
In a so-called video camera that captures moving images, a single image sensor is provided for each of the three primary colors RGB and a single plate type in which a mosaic image color filter is provided on a single image sensor to generate a color image. There are three plate types.

  The single-plate type has a simple configuration, but there is a problem in that aliasing noise occurs because the spatial sampling frequency of the color signal is lowered to half or less of the spatial sampling frequency of the pixels of the image sensor. In the three-plate type, the spatial sampling frequency of the RGB three primary color signals is the same as the spatial sampling frequency by the pixels of the image sensor, and aliasing noise hardly occurs, and a high-quality video signal can be generated. However, since three CCDs are used, there is a problem that the structure of the optical system is complicated and the cost is increased.

  An example of a color filter array used for a single-plate image sensor is shown in FIG. In this example, complementary color filters of Mg, G, Cy, and Ye are used. When a moving image is generated using an image pickup device having this color filter, pixel mixing is performed by mixing signals of two pixels adjacent in the vertical direction and output. A video signal in the analog TV signal standard such as NTSC is an interlace signal. In order to generate such an interlace signal, pseudo interlace scanning is performed by changing the combination of rows to be mixed for each field.

  On the other hand, in order to increase the resolution of a still image in a digital still camera, an image sensor in which the number of pixels of the image sensor is increased as compared with an image sensor for a video camera has been proposed. The number of vertical pixels of such an image sensor is larger than the number of scanning lines in the current television system. For example, when the number of effective pixels in the vertical direction is 960, the number of scanning lines in the television system is doubled. FIG. 5B shows an example of the color filter arrangement of such an image sensor for still images. In this filter, three primary colors R (red), G (green), and B (blue) are used.

  Therefore, a method of generating a high-quality moving image comparable to that of a three-plate type using such an image pickup device with a large number of pixels for a digital still camera as a video camera for moving image pickup can be considered.

  However, if the number of pixels of the image sensor is increased in order to obtain a high resolution, there is a problem that image quality deteriorates when a moving image that conforms to a television signal standard such as NTSC is generated. When the number of pixels of the conventional moving image pickup element, for example, the number of effective pixels in the vertical direction is 480, pixel mixing and pseudo-interlacing are performed using the filter array of FIG. If 60 lines of signals can be output per second, a moving image compatible with the NTSC system can be generated in real time.

  However, for example, if the number of pixels is doubled in both horizontal and vertical directions, 240 lines of signals cannot be output in one field unless pixels are thinned out and read at a rate of 1 pixel per 4 pixels in the vertical direction. .

  In this way, in order to read a signal at a ratio of one pixel to four pixels, for example, in the case of the color filter of FIG. 6A, to generate a color image, MG pixel rows and CY pixel rows are alternately arranged. In this case, it is not possible to perform thinning at equal intervals every four pixels. As an example, there is a method of reading CY of the sixth row next to MG of the first row and then reading MG of the ninth row. However, thinning is performed by alternately setting the number of pixels to be thinned out to 5 pixels and 3 pixels. Therefore, aliasing occurs due to irregular sampling, and the image quality deteriorates.

  An object of the present invention is to solve the above-described problems and provide an imaging apparatus capable of generating a high-quality moving image with less aliasing noise in a single-plate imaging apparatus.

  Therefore, in the present invention, in an imaging apparatus using a high-pixel imaging device having a number of pixels approximately N times the number of output lines, signals accumulated in each pixel of the imaging device are thinned out in N pixel cycles in the vertical direction. Alternatively, an image pickup device that outputs a signal of approximately 1 / N lines of pixels in the vertical direction by mixing, and the image pickup device drives the image pickup device to output a pixel signal of a predetermined number of output lines by predetermined thinning and mixing. The image pickup apparatus is configured by a driving circuit that performs this operation and a signal processing circuit that generates a moving image using a pixel signal output from the image pickup element.

  At this time, the color filter array of the image sensor is in the form of a vertical stripe or a horizontal 2 pixels × 8 pixels period, and thinning in the vertical direction can be performed at equal intervals, and there is little aliasing noise. Moreover, the sampling frequency of the color in the vertical direction can be the same as the number of output lines, and a high-quality moving image can be generated.

  ADVANTAGE OF THE INVENTION According to this invention, the imaging device which can produce | generate the high quality moving image with few aliasing noises in a single plate type imaging device can be provided.

  An embodiment of an imaging apparatus according to the present invention will be described. FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to the present invention. In the figure, 1 is a lens, 2 is an image sensor such as a CCD, and 3 is an A / D conversion circuit. A signal processing circuit 4 converts the output signal of the image sensor into a YUV signal and generates a standard television signal such as NTSC or PAL. Reference numeral 5 denotes a drive circuit for driving the image pickup device, and 9 denotes image data recording. In addition to the magnetic tape, a semiconductor memory such as a flash memory, a magnetic disk such as a hard disk, or the like is used. A removable medium such as a PC card may be used. A control circuit 6 controls the operation timing of each block and controls input / output of image data.

  An operation of the imaging apparatus in the present embodiment will be described. The light incident on the lens 1 forms an image on the image pickup surface of the image pickup device 2 through the diaphragm 7. As shown in FIG. 2, the imaging element 2 includes a large number of pixels on the imaging surface. In the figure, reference numeral 20 denotes a pixel, which is usually composed of a photodiode. Although the number of pixels is generally arbitrary, in the present embodiment, the number of effective pixels in the vertical direction is assumed to be 960. A vertical transfer unit 22 uses a four-phase drive CCD. Reference numeral 21 denotes a transfer means for transferring a signal (pixel signal) accumulated in the pixel 20 to the vertical transfer unit 22, which is shared with the gate of the vertical transfer unit 22. The vertical transfer unit 22 is driven by inputting a ternary pulse consisting of three levels of high, middle, and low to the four-phase gate, but a high level is applied to read out the pixel signal from the pixel 10 to the vertical transfer unit. The transfer in the vertical transfer unit is performed by middle and low level four-phase driving. The elementary vertical transfer unit 22 has two gates per pixel, and the number of pixels that can be transferred is 480, which is half the number of vertical pixels.

The basic operation of this image sensor is the same as that of a general interline CCD, but an outline thereof will be described in the case of moving image capturing for video signal generation. During the vertical blanking period, the pixel signal is read to the vertical transfer unit 22, and the vertical transfer unit sequentially transfers the pixel signal in the vertical direction during the horizontal blanking period. The horizontal transfer unit sequentially outputs pixel signals for one line transferred from the vertical transfer unit from the output terminal 25 through the output amplifier 24 during the horizontal scanning period.
In the present invention, in order to make it possible to capture a high-quality moving image, thinning-out reading is performed at equal intervals when reading a signal. For this reason, the four-phase gates V1, V2, V3, and V4 of the vertical transfer unit separate the V1 and V3 gates connected to the pixels into two systems, respectively, and perform thinning readout for each row or every two rows. Added support for types. At this time, in order to make the number of pixels thinned out in the vertical direction at equal intervals, in this embodiment, the V1 gate is alternately separated into V1 and V1 ′ for each pixel, and V3 is similarly separated into V3 and V3 ′ for each pixel. did. In addition, the color filter array of the image pickup element is formed in a vertical stripe shape of RGB three primary colors so that the color signal can always be reproduced when thinned at an arbitrary pixel interval. In the case of the vertical stripe filter, unlike the checkered filter, three types of signals necessary for reproduction of color signals can be obtained from any one row, so that thinning readout at equal intervals is possible.

  FIG. 3 is a timing chart of the driving pulses of the image sensor in the present embodiment, and shows the timing of the driving pulses when outputting, for example, 60 interlaced images suitable for moving image capturing. Waveforms corresponding to V1, V2, V3, V4 and V1 ', V3' are pulses input to each gate of the vertical transfer unit.

When the number of vertical effective lines of 60 interlaced images per second is about 240, and the number of vertical pixels of the image sensor is 960, it may be output by thinning out at a rate of 1 pixel per 4 pixels. Therefore, in this embodiment, in the vertical transfer period in the vertical direction, transfer corresponding to four rows is performed in the vertical blanking period by the four-phase pulse. In the vertical transfer, V1 and V1′V1, and V3 and V3 ′ are equivalent and are driven with the same pulse. During a period during which vertical transfer is not performed (usually a horizontal scanning period), V1 and V2 are at the middle level, and V3 and V4 are at the low level.
To perform vertical transfer for two rows (corresponding to two vertical pixels of the image sensor), at time t1, V1 is set to the low level and V3 is set to the middle level, and the signals under the gates V1 and V2 are transferred under the gates V2 and V3. Subsequently, V2 is set to the low level and V4 is set to the middle level, and transferred below the V4 gate. Further, V1 is set to the middle level, V3 is set to the low level, and signals for one gate phase are further transferred. By setting V2 to the middle level and V4 to the low level at time t2, the transfer for two rows is completed. In order to transfer four rows, the same transfer may be performed again. For this reason, from time t3 to t4, the same operation as the transfer from t1 to t2 is repeated. If such a transfer operation is repeated 240 times for each horizontal blanking period, signals corresponding to the number of effective pixels can be output at a predetermined field rate.

  In the vertical blanking period, the pixel signal is read out to the vertical transfer unit. A period indicated as “read” in FIG. 3 is a pulse waveform at this time. The pixels of the image sensor are connected to the gates V1, V3, V1 ', and V3', and these gates are set to a high level to read out pixel signals.

  The number of vertical pixels of the output image is 240, and it may be read at a rate of one row per four rows. However, if only one of the four pixels is output, half of the pixel signals are not used even when considering interlaced reading, and the signal amount is reduced and the sensitivity is deteriorated. Therefore, in this embodiment, the sensitivity is improved by performing pixel mixing in which signals of two pixels are added in the vertical direction and reading out. The output is an interlaced image, and it is necessary to change the row of pixels to be read for each field. However, in the read field (A field) shown in FIG. 3, the pixel signals connected to the V1 and V3 gates are read. First, at t5, V1 is set to the high level, and the pixel signal is read out under the V1 gate. At t6, V1 is returned to the middle level, and the signal transfer from the pixel to the lower side of the V1 gate is completed. At this time, V3 is set to the middle level and the signal read up to the bottom of the V3 gate is transferred. Thereafter, V3 is set to the middle level and V1 is set to the low level, so that the read pixel signal is transferred under the V2 and V3 gates. Similar to the operation from t5 to t6, the pixel signal connected to V3 is read out by setting V3 to the high level, and the pixel signal connected to the V1 gate read out earlier is mixed. The pixel signals read in the A field as described above are sequentially read out by the vertical transfer described above, and signals for 240 rows in the A field are output through the horizontal transfer unit.

  In the next field, the image pickup device is driven by substantially the same operation, but it is necessary to change the pixel from which the signal is read in order to output the interlace. In the A field, the pixel signals connected to the V1 and V3 gates are read out. In the next field (B field), the pixel signals connected to the V1 'and V3' gates are read out. As a result, in the B field, 1 row, 2 rows, 5 rows, 6 rows, 9 rows, 10 rows,..., 4n + 1, 4n + 2, and 2 pixels are continuously thinned out and read out. Can be read out in the same manner as 3 rows, 4 rows, 7 rows, 8 rows, 11 rows, 12 rows,... 4n + 3, 4n + 4. Two consecutive rows of pixel signals can be mixed in the vertical transfer unit for signals in the same column, thereby outputting an interlaced moving image having 240 effective lines.

  In FIG. 1, the processing after reading out the pixel signal from the image sensor 2 as described above will be described. The pixel arrangement of the image sensor is a horizontal arrangement of the three primary color signals RGB in a stripe shape, and the pixel signal output in each horizontal scanning period is a dot sequential signal of the three primary color signals RGB. These RGB signals are subjected to white balance processing and gamma processing to generate color differences U and V. Further, the luminance signal Y is generated by adding the RGB signals at a predetermined ratio. The YUV signal generated in this way is output as a video signal. When outputting as an analog video signal such as NTSC, NTSC encoding is further performed. The above signal processing is performed in the signal processing circuit 4.

  In the present embodiment, means for transferring the pixel signal to the vertical transfer unit every two rows for signal readout from the image sensor is provided, and it is possible to output a video signal suitable for high-quality moving image generation, In addition, since pixels are thinned out at equal intervals, a high-quality moving image with less aliasing noise can be generated.

  In the present embodiment, the thinning-out readout is possible in the 2-row cycle or 4-row cycle. However, the gates of the vertical transfer units connected to the pixels are separated in accordance with the thinning-out cycle and simultaneously read out pixel signals. By changing the combination, it is possible to perform thinning-out reading of three row periods or thinning-out of arbitrary N row periods. At this time, since a vertical stripe filter is used, a color signal can always be generated even if thinning is performed at equal intervals.

  In this embodiment, the elemental vertical transfer unit 12 of the image sensor whose configuration is shown in FIG. 2 is configured to have two gates per pixel, but a progressive scan type image sensor having three or four gates per pixel. May be used.

  Next, another embodiment of the imaging apparatus according to the present invention will be described with reference to FIG. In the first embodiment, a 240-line interlace signal is generated. In this embodiment, a 480-line non-interlace signal is output.

  In FIG. 4, unlike the case of FIG. 3, the vertical transfer operation transfers one row for one vertical transfer. Therefore, the same transfer from t1 to t2 at the drive pulse timing in FIG. 3 is performed, and the second transfer from t3 to t4 is not performed. For this reason, 2f rows are transferred in one transfer.

  On the other hand, signal readout is performed as follows. In the signal readout period in FIG. 4, first, V1 and V1 ′ are set to the high level to read all the signals in the even-numbered rows, V3 and V3 ′ are set to the middle, and V1 and V1 ′ are set to the low level. Transfer under the gate. Next, V3 and V3 'high level odd row signals are read. As a result, the signals of the first and second lines, the third and fourth lines, the fifth and sixth lines, and so on are mixed in the vertical transfer unit.

  The signals mixed in the vertical transfer unit as described above are transferred one row at a time in the vertical transfer unit. The horizontal transfer unit sequentially outputs signals sent from the vertical transfer unit. In this way, a sequential signal of 480 lines mixed with pixels is output.

  In this embodiment, a non-interlaced signal of 480 lines can be output, and a progressive scan moving image can be generated. In this case, each image is exposed at the same timing, and blurring of each frame image is unlikely to occur during moving image shooting.

  Next, another embodiment of the imaging apparatus according to the present invention will be described. The color filter array of the image sensor of the present embodiment is different. FIG. 6 shows a color filter arrangement, and FIG. 6A shows an RGB vertical stripe configuration used in the embodiment of FIG. 1, which is an arrangement using primary color filters. In this embodiment, Ye ( Yellow) and Cy (cyan) complementary color vertical stripe filters are used, and the filter configuration is Ye, G, Cy or (c) Ye, W, Cy in FIG. The Ye filter is a filter that transmits G and R color light, Cy is a filter that transmits G and B color light, and W is a filter that transmits all RGB colors. The all-color transmission filter may have a characteristic close to magenta by reducing the G sensitivity.

  In the present embodiment, a complementary color filter is used, and high sensitivity can be achieved in addition to high image quality.

  Another embodiment of the present invention will be described. The color filter array of the image pickup element described so far is the vertical stripe shown in FIG. 6, but a color filter array that can be thinned out at equal intervals can be one that repeats two horizontal pixels and eight vertical pixels. . FIG. 7 shows an image sensor according to this embodiment. The configuration is the same as that of the image pickup device in FIG. 2 except that the first row is RG, and the second row to the eighth row are BG, GR, BG, GR, GB, GR, and BG. . In order to output an interlace signal of 240 lines per field from the image sensor using such a color filter array, it is necessary to mix four pixels.

The vertical transfer operation for this is the same as in FIG. 3, and two rows are transferred in one vertical transfer. On the other hand, signal readout is performed as follows. First, V1 and V1 ′ are set to the high level, all the signals in the even-numbered rows are read, V3 and V3 ′ are set to the middle, and V1 and V1 ′ are set to the low level, and the read signals are transferred under the V2 and V3 gates. Next, V3 and V3 ′ high level odd row signals are read. As a result, the signals of the first and second rows, the third and fourth rows, the fifth and sixth rows, and so on are mixed in the vertical transfer section.
As described above, when the signals mixed in the vertical transfer unit are transferred two rows in the vertical direction, the signals of two rows mixed in the vertical transfer unit are further mixed in the horizontal transfer unit from the vertical transfer unit to the horizontal transfer unit. The signals of four rows adjacent in the vertical direction are mixed.

  As described above, the pixel signals for the four rows are mixed by mixing in the vertical transfer unit and mixing in the horizontal transfer unit. At this time, in order to interlace the signals, at the start of the A field and the B field, the transfer is not performed for two lines but only for one line, and the subsequent transfer is performed for two lines. As a result, the combination of rows to be added by the horizontal transfer unit changes for each field and can be interlaced.

  When four-pixel mixing is performed in this way, the output of the mixed line from the first line to the fourth line is a dot sequential output of R + 2B + G and R + 3G, and the next line is a dot sequential signal of 3G + B and 2R + G + B. In this way, the line sequential signal in which the composition of the dot sequential signal changes for each line is obtained. In order to generate a luminance signal and a color signal from such a signal, the following processing may be performed. For the luminance signal, the average composition of each line is 2R + 4G + 2B, which is the same for all the lines, and it is only necessary to perform a normal filtering process.

  As for the color signal, 2B-2G and 2R-2G are generated by subtracting the dot sequential signal of each line. By subtracting the luminance signal Y, two types of color difference signals of BY and RY can be generated, and a color video signal can be obtained.

  In the present embodiment, it is possible to output 240 lines of interlace signals by mixing 4 lines. Since four lines of mixing are performed, higher sensitivity than that of two lines can be achieved. Normally, color signals cannot be generated when four lines are mixed. However, in the present invention, a 2 × 8 pixel configuration filter is used, and even when four lines are mixed, color signals are always generated. Is possible.

  Note that the same 4-pixel mixing is possible even when a vertical stripe filter is used. In addition to the filter arrangement of FIG. 8A described here, a 2 × 8 configuration filter having an arrangement as shown in FIGS. 8B and 8C can also be used. However, in the case of the (b) and (c) filters, not two-pixel mixing but two-pixel mixing is performed in the same manner as the driving in FIG.

  Another embodiment of the present invention will be described. FIG. 9 is a block diagram showing the configuration of the imaging apparatus according to the present invention. The figure performs high-resolution still image capturing in addition to moving images. The same parts as those in FIG. 1 are denoted by the same reference numerals, and description thereof is omitted. Reference numeral 10 denotes a compression / decompression circuit that compresses and decompresses an image using a method such as JPEG (Joint Photographic Expert Group). Compress still image data. Further, a moving image may be recorded after being compressed by a method such as MPEG (Moving Picture Expert Group). When MPEG2 compression is adopted during MPEG compression, a YUV signal that is not subjected to interlace conversion before frame rate conversion may be used as the input to compression, or a signal that has been converted to interlace after frame rate conversion is used. Also good. When performing MPEG compression of a moving image, it is convenient that the number of pixels of the image sensor is 960 vertical effective pixels and 1440 horizontal effective pixels.

  Reference numeral 12 denotes a mode selection switch, which is an interlace mode when the user shoots a movie, a still image, or a movie. Non-interlace (progressive) selection can be performed.

When capturing a still image, unlike when capturing a moving image, all the data in the odd-numbered rows are read out independently in the A field and all the data in the even-numbered rows are read out in the B field without pixel mixing. When the vertical effective pixel is 960, one field here corresponds to 1/30 seconds. The odd line signal read in the first field from the image sensor and the even line signal read in the second field are temporarily written in the memory 10 and sequentially converted by sequentially reading one line and two lines or less. High-resolution images can be reproduced.
Next, a driving method of the image sensor for outputting a signal suitable for still image generation or high-resolution moving image generation will be described. In this case, a signal of 480 lines corresponding to half of all pixels is interlaced read out. A high-resolution image of vertical 960 lines can be generated from the interlaced images of every 480 lines by using an image memory. A driving method in this case will be described with reference to FIG.

  FIG. 10 is a timing chart of the driving pulse of the image sensor, and shows the timing of the driving pulse when outputting a 480-line interlace signal suitable for still image capturing. In the case of 240-line moving image output, vertical transfer is performed once every two lines, but when transferring an image of 480 lines, it is sufficient to perform one line at a time. Therefore, unlike the transfer pulse of FIG. 3, each gate pulse is single for one row of transfer.

  In the signal readout, V1 and V1 'are set to the high level, and in the A field, the pixel signals in the even rows connected to V1 and V3' are read out. Similarly, in the B field, pixel signals in odd rows can be read by setting V3 and V3 'to high level. The pixel signals read out to the vertical transfer unit in this way are output by performing vertical transfer and further horizontal transfer and outputting 480 lines of interlace signals.

  In the above description, at the time of still image capturing, a still image is generated using signals sequentially read from the image sensor, but a still image may be generated from signals sequentially read by mixing pixels. The still image generated in this way has half the number of lines of the still image generated by interlace independent reading, and has a merit that it can be photographed without a mechanical shutter although the resolution is low.

  It can also be used as a freeze function in the moving image mode. When a conventional imaging device is used, when a video is frozen, signals are accumulated for each field, so a high-resolution still image cannot be taken for a moving subject. A still image frozen at an arbitrary timing can obtain a high resolution. It is also possible to use this shooting mode as a continuous shooting function.

As described above, this embodiment can generate a 240-line interlace signal suitable for moving image generation and a 480-line interlace signal suitable for high-resolution moving image and still image generation. Is possible. The 240-line interlace signal is pixel-mixed and has high sensitivity.
In addition, the mode selection switch allows the user to switch between still images and moving images and the moving image mode to interlaced / progressive, and according to this, the signal processing content can be switched, which is convenient.

The block diagram which shows the structure of one Embodiment of the imaging device by this invention 1 is a configuration diagram of an image sensor in an embodiment of an imaging apparatus of the present invention. Drive pulse timing diagram of one embodiment of the present invention Drive pulse timing diagram of one embodiment of the present invention The figure which shows the color filter arrangement | sequence of the conventional image pick-up element The figure which shows the color filter arrangement | sequence of one Embodiment of this invention. Configuration diagram of image sensor in the present invention The figure which shows the color filter arrangement | sequence of one Embodiment of this invention. The block diagram which shows the structure of one Embodiment of the imaging device by this invention Drive pulse timing diagram of one embodiment of the present invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Lens 2 ... Imaging element 3 ... A / D conversion circuit 4 ... Sequential conversion circuit 5 ... Signal processing circuit
8 ... CCD drive circuit 9 ... recording medium
10 ... Memory 11 ... Compression / decompression circuit 12 ... Mode selection switch

Claims (1)

An image sensor that includes a pixel array in which a plurality of types of pixels that respectively convert different types of colored light into electrical signals are periodically arranged in the vertical direction and the horizontal direction, and that outputs a signal accumulated in each pixel in the pixel array An imaging device comprising:
The pixel array is a pixel array that repeats three types of pixels that convert blue, red, and green color light into electrical signals, with 2 pixels in the horizontal direction and 8 pixels in the vertical direction as basic units,
Red and green in the first row, green and red in the third, fifth and seventh rows, blue and green in the second, fourth and eighth rows, and green and blue in the sixth row An imaging apparatus characterized by the above.

Images