JP2003179819A - Image pickup device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image pickup device capable of obtaining a natural linear image where no temporal deviation is caused in an object and no rapid change takes place in a luminance signal even while increasing the dynamic range. <P>SOLUTION: By regularly arranging optical attenuation filters with different transmissivity onto pixels, two kinds of image signals or more with different exposure amounts without temporal deviation are obtained and by composing the image signals, a natural linear image where no temporal deviation is caused in an object and no rapid change takes place in a luminance signal is obtained. In a solid-state imaging element shown in Fig. 1, four kinds of optical attenuation filters are regularly arranged on the pixels and four kinds of image signals with different exposure amounts can be obtained. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image pickup apparatus capable of expanding a dynamic range of an image.

[0002]

2. Description of the Related Art Conventionally, in an image pickup apparatus using a solid-state image pickup element, a method of combining two images having different exposure amounts to obtain an image having a wide dynamic range has been used.

FIG. 9 is a block diagram of a solid-state image pickup element in a conventional image pickup apparatus. As shown in FIG. 9, magenta (Mg), yellow (Ye), cyan (Cy),
Color filters having four different spectral characteristics of green (G) are arranged for each pixel (photoelectric conversion element).

In a conventional image pickup apparatus which obtains an image having a wide dynamic range by synthesizing two images having different exposure amounts,
An electronic shutter or the like is used to switch the exposure time so that the exposure amount for the solid-state image pickup device is different, and images are taken.

FIG. 10 shows the relationship between the exposure time and the exposure amount for one shutter operation. In the conventional image pickup device, for example, the shutter speed required for one image pickup is 1/60
In the case of [sec], an image in which the electronic shutter period (exposure time) is changed within 1/60 [sec] period is taken,
An image signal having a wide dynamic range is obtained by combining the two image signals having different exposure amounts (long-time exposure signal and short-time exposure signal) into one image signal.

As a synthesizing method, first, signals from respective color filters (Mg, Ye, Cy, G) are subjected to signal processing to form a luminance signal and a color signal of a long-time exposure signal and a short-time exposure signal, respectively. Next, the two luminance signals are combined. FIG. 11 shows the luminance signal level with respect to the amount of incident light when the luminance signals are combined. As shown in FIG. 11, when only the luminance signal of the long-time exposure signal is used, the signal is saturated at a certain level due to the narrow dynamic range. Therefore, when the luminance signal of the long-time exposure signal reaches the saturation level, a signal obtained by adding an OFFSET component to the luminance signal of the short-time exposure signal is output. That is, when the amount of incident light is within the saturation level of the brightness signal of the long-time exposure signal, the brightness signal of the long-time exposure signal is used, and when it exceeds the saturation level, the brightness signal of the short-time exposure signal is used and output as a combined brightness signal. Next, the two color signals are combined according to the level and change of the combined luminance signal and output as a combined color signal. Then, these composite luminance signal and performs signal processing on the synthetic color signal, is stored in various memory cards or, or through encoder NTSC (N ational T elevision
S ystem C ommittee) and PAL (P ha
to convert to se A lternation by L ine) signal, such as.

As described above, the conventional image pickup apparatus obtains an image signal having a dynamic range wider than the original dynamic range of the solid-state image pickup element. However, in such a conventional dynamic range expansion method, there is a time difference between two images having different exposure amounts, that is, there is a time difference between the image capturing start times of the first image and the second image. When a fast-moving subject is photographed, there is a problem that time lag occurs between the images and the images become unnatural when they are combined.

There is also a problem that when two images having different exposures are combined, a linear and abrupt change occurs in the luminance signal in the combined joint portion, and the combination becomes unnatural. In order to eliminate the unnaturalness due to the linear and abrupt change of the brightness signal, it is necessary to capture a large number of images with different exposure amounts and combine them to change the brightness signal gently. The maximum time difference (1
Since the time difference between the image capture start times of the first image and the last image in the shutter period increases, the unnaturalness of the joint cannot be eliminated in effect only by combining the two images.

For this reason, although this method has attracted attention as an effective means for expanding the dynamic range, it is not used in a video camera that requires high image quality,
It is only used in some of the narrow surveillance cameras on the market. Further, in recent years, even in electronic still cameras whose markets have been remarkably expanding, the above-mentioned method is drawing attention because of the necessity of expanding the dynamic range. However, since higher image quality is required than a video camcorder and the shutter speed selection is more flexible than a video camcorder,
When shooting with a slow shutter speed, there is a problem that the time difference between the images further increases.

[0010]

SUMMARY OF THE INVENTION In view of the above problems, the present invention is to regularly arrange light attenuation filters having different transmittances on a photoelectric conversion element, or to use photoelectric conversion elements having different aperture ratios. However, while obtaining an image signal with different exposure amount, while expanding the dynamic range,
It is an object of the present invention to provide an image pickup apparatus capable of obtaining a natural image in which there is no time lag of a subject and no abrupt change in luminance signal.

[0011]

[Means for Solving the Problems] Claim 1 in the present invention
The described image pickup device is a solid-state device in which light attenuation filters having different transmittances are regularly arranged on each photoelectric conversion element having the same aperture ratio to form two or more kinds of exposure amounts for each photoelectric conversion element. An image pickup device and a synthesizing unit for synthesizing a plurality of image signals having different exposure amounts obtained by processing signals output from the solid-state image pickup device into one image signal.

An image pickup apparatus according to a second aspect of the present invention is the image pickup apparatus according to the first aspect, characterized in that a color filter is provided on the photoelectric conversion element. An image pickup device according to a third aspect of the present invention is the image pickup device according to the first aspect, characterized in that a color filter having a different transmittance is provided instead of the light attenuation filter.

According to a fourth aspect of the present invention, in the solid-state image pickup device in which photoelectric conversion elements having different aperture ratios are regularly arranged to form two or more kinds of exposure amounts for each photoelectric conversion element. And a synthesizing means for synthesizing a plurality of image signals having different exposure amounts obtained by processing the signals output from the solid-state image pickup device into one image signal.

An image pickup device according to a fifth aspect of the present invention is the image pickup device according to the fourth aspect, characterized in that a color filter having the same transmittance is provided on the photoelectric conversion element. As described above, according to the present invention, the light attenuation filters having different transmittances are regularly arranged on the photoelectric conversion element, or the photoelectric conversion elements having different aperture ratios are regularly arranged to form the solid-state imaging device. With the above configuration, it is possible to obtain two or more types of image signals having different exposure amounts with one shutter operation. As a result, two or more types of images can be obtained without a time difference. By combining these images, the dynamic range can be expanded while the time lag of the subject does not occur on the screen, and the brightness can be obtained at the combined joint portion. It is possible to realize an imaging device that can obtain a natural image without a sudden change in the signal.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION An image pickup apparatus according to an embodiment of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 shows the first embodiment.
3 is a configuration diagram of a solid-state image sensor in the image pickup apparatus of FIG. Figure 1
As shown in FIG. 2, magenta (M
Color filters having four different spectral characteristics of g), yellow (Ye), cyan (Cy), and green (G) are regularly arranged. Further, there are four types of pixels having different transmittances in the same color pixel (photoelectric conversion element). For example, as shown in FIG. 1, a pixel 10 having a transmittance ratio of 1 is provided in one Ye filter.
1, a pixel 102 having a transmissivity ratio of 1/4, a pixel 103 having a transmissivity ratio of 1/16, and a pixel 104 having a transmissivity ratio of 1/32 are regularly arranged. The solid-state image pickup device of the image pickup apparatus is configured by arranging four types of light attenuation filters having different transmittances on pixels in a regular arrangement as shown in FIG. However, the aperture ratio is the same.

The color filter may be arranged and formed on each pixel. In this case, the transmittances of the color filters do not have to be the same, and may be regular. In other words, the transmittance may have a certain regularity in combination with the transmittance of the optical attenuation filter.

The arrangement of the optical attenuation filters is not limited to the arrangement shown in FIG. 1, and the transmittance ratio is 1,
It is not limited to 1/4, 1/16 and 1/32. Further, the transmittance is not limited to four types, and a plurality of types may be used. Further, instead of using the optical attenuation filters having different transmittances, color filters having different transmittances may be used and regularly arranged on each pixel.

The array of color filters is not limited to the array shown in FIG. Further, although the color filter is configured to have four colors of magenta (Mg), yellow (Ye), cyan (Cy), and green (G), the present invention is not limited to this, and green (G), blue (B), and red are provided. A configuration using a primary color filter composed of (R) may be used.

Next, FIG. 2 shows changes in the exposure time and the exposure amount by one shutter operation. As shown in FIG.
In the shutter period required for one image pickup, four kinds of signals having different transmittances are obtained for each color. That is, it is possible to obtain four types of image signals having different exposure amounts. At this time, there is no time lag between the image signals. An image having a wide dynamic range can be obtained by synthesizing these image signals into one image signal by a synthesizing means (not shown), and there is no time lag between the image signals to be synthesized. It is possible to obtain an image that is not unnatural even when combined.

As a synthesizing method, first, signals from respective color filters (Mg, Ye, Cy, G) having the same transmittance are subjected to signal processing, and four kinds of luminance signals (Y) and color signals having different exposure amounts are given. (Cr, Cb) is formed. For example, if the transmittance ratio is 1, Mg, Ye, Cy, G having the transmittance ratio of 1
Signal processing is performed on the signal from the filter to form a luminance signal (Y) and a color signal (Cr, Cb) from the exposure signal having a transmittance ratio of 1. That is, four luminance signals (Y) and color signals (Cr, Cb) having different exposure amounts according to four types of transmittance are formed.

Next, these four luminance signals are combined.
FIG. 3 shows the luminance signal level with respect to the amount of incident light when the luminance signals are combined. As shown in FIG. 3, after the luminance signal having the transmittance ratio 1 reaches a certain level before saturation, the transmittance ratio 1 /
The signal obtained by adding the OFFSET component to the luminance signal of 4 is used as the next luminance signal, and after the luminance signal with the transmittance ratio of 1/4 reaches a certain level before saturation, the luminance signal with the transmittance ratio of 1/16. The signal obtained by adding the OFFSET component to is used as the next luminance signal, and after the luminance signal with the transmittance ratio of 1/16 reaches a certain level before saturation, the OFFSET component is added to the luminance signal with the transmittance ratio of 1/32. The added signal is the next luminance signal,
Output as a combined luminance signal. As shown in FIG. 3, since the brightness signal corresponding to each exposure amount is used, the change of the brightness signal becomes gradual.

Next, the color signals are combined in accordance with the level and change of the combined luminance signal and output as a combined color signal. Then, the combined luminance signal and the combined color signal are subjected to signal processing and stored in various memory cards or passed through an encoder and converted into signals such as NTSC and PAL.

As described above, the image pickup apparatus according to the first embodiment changes the exposure amount of each pixel (photoelectric conversion element) by using the filters having different transmissivities, so that plural kinds of images having different exposure amounts can be obtained. An image with a wide dynamic range is obtained by obtaining signals and combining them. That is, by regularly arranging filters having different transmittances on pixels and obtaining a luminance signal corresponding to each exposure amount, it is possible to obtain a dynamic range larger than the dynamic range that the solid-state imaging device originally has, In addition, it is possible to realize an image pickup apparatus capable of obtaining a natural image on the screen without time lag of the subject and without abrupt change in the luminance signal at the combined joint portion.

(Second Embodiment) An image pickup apparatus according to the second embodiment will be described below with reference to the drawings. Figure 4
FIG. 6 is a configuration diagram of a solid-state image sensor in the image pickup apparatus according to the second embodiment. As shown in FIG. 4, on the solid-state image sensor, magenta (Mg), yellow (Ye), cyan (Cy),
Color filters having four different spectral characteristics of green (G) are regularly arranged, and pixels of the same color (photoelectric conversion element) include four types of pixels having different aperture ratios. For example, as shown in FIG. 4, in one Ye filter, a pixel 401 having an aperture ratio of 1, a pixel 402 having an aperture ratio of 1/4, a pixel 403 having an aperture ratio of 1/16, and an aperture ratio of 1/32 are provided. Of pixels 404 are regularly arranged. However, the filter transmittance is the same.

The arrangement of pixels having different aperture ratios is shown in FIG.
However, the aperture ratio is not limited to 1, 1/4, 1/16, 1/32.
Moreover, the aperture ratio is not limited to four types, and may be a plurality of types.

The array of color filters is not limited to the array shown in FIG. Further, although the color filter is configured to have four colors of magenta (Mg), yellow (Ye), cyan (Cy), and green (G), the present invention is not limited to this, and green (G), blue (B), and red are provided. A configuration using a primary color filter composed of (R) may be used.

Next, FIG. 5 shows changes in the exposure time and the exposure amount by one shutter operation. As shown in FIG.
Four types of signals having different aperture ratios are obtained for each color during the shutter period required for one image pickup. That is, it is possible to obtain four types of image signals having different exposure amounts. At this time, there is no time lag between the image signals. An image having a wide dynamic range can be obtained by synthesizing these image signals into one image signal by a synthesizing means (not shown), and there is no time lag between the image signals to be synthesized. It is possible to obtain an image that is not unnatural even when combined.

As a synthesizing method, first, signals from respective color filters (Mg, Ye, Cy, G) having the same aperture ratio are subjected to signal processing, and four kinds of luminance signals (Y) and color signals having different exposure amounts are given. (Cr, Cb) is formed. For example, if the aperture ratio is 1, Mg, Ye, Cy, G having the aperture ratio of 1
Signal processing is performed on the signal from the filter to form a luminance signal (Y) and a color signal (Cr, Cb) from the exposure signal with an aperture ratio of 1. That is, four luminance signals (Y) and color signals (Cr, Cb) having different exposure amounts according to four types of aperture ratios are formed.

Next, these four luminance signals are combined.
FIG. 6 shows the luminance signal level with respect to the amount of incident light when the luminance signals are combined. As shown in FIG. 6, after the luminance signal with the aperture ratio 1 reaches a certain level before saturation, the aperture ratio 1 /
The signal obtained by adding the OFFSET component to the luminance signal of 4 is used as the next luminance signal, and after the luminance signal of the aperture ratio 1/4 reaches a certain level before saturation, the luminance signal of the aperture ratio 1/16 The signal obtained by adding the OFFSET component to is the next luminance signal, and after the luminance signal with the aperture ratio 1/16 reaches a certain level before saturation, the OFFSET component is added to the luminance signal with the aperture ratio 1/32. The added signal is the next luminance signal,
Output as a combined luminance signal. As shown in FIG. 6, since the brightness signal corresponding to each exposure amount is used, the change in the brightness signal becomes gradual.

Next, the color signals are combined according to the level and change of the combined luminance signal and output as a combined color signal. Then, the combined luminance signal and the combined color signal are subjected to signal processing and stored in various memory cards or passed through an encoder and converted into signals such as NTSC and PAL.

As described above, the image pickup apparatus according to the second embodiment uses the pixels (photoelectric conversion elements) having different aperture ratios to change the exposure amount for each pixel, and to obtain plural kinds of images having different exposure amounts. An image with a wide dynamic range is obtained by obtaining signals and combining them. That is, a solid-state image sensor is configured by regularly arranging pixels having different aperture ratios, and a luminance signal corresponding to each exposure amount is obtained, so that a dynamic range larger than the dynamic range originally possessed by the solid-state image sensor is obtained. It is possible to realize an image pickup apparatus that can obtain a natural image without a time lag of the subject on the screen and without abrupt change in the luminance signal at the combined joint portion.

In the first and second embodiments, the color image pickup device provided with the color filters has been described, but the same effect can be obtained even in the monochrome image pickup device having no color filter. .

In the first and second embodiments, the spatial shift caused by the positional shift of pixels having different transmittance ratios or aperture ratios is interpolated by using an appropriate filter (digital filter). That is, the signal output from each pixel can be A / D converted and digitally filtered so that the pixels are spatially interpolated with each other.

Here, as a spatial interpolation method, for example, as shown in FIG. 7, the area of four pixel signals having different transmittance ratios is considered to be the area of one pixel signal, and the signal of this one pixel is used as the original. , The signal of 1/4, the signal of 1/4, the signal of 1/8,
There is a method of performing spatial interpolation by directly replacing the signal with 1/32 and forming four pixel signals having different transmittances.

Further, as shown in, for example, FIG.
There is a method in which a signal at this position in another transmittance ratio is created by spatial interpolation, with the position of the signal having a transmittance ratio of 1 (pixels with diagonal lines) as a reference. As an example, a procedure for creating a signal having a transmittance ratio of 1/16 of the Ye filter at the position of the diagonal line will be described below.

Here, it is assumed that the nearest two-pixel interpolation is performed. The signal from the pixel 1 (interpolation pixel 1) having the transmittance ratio 1/16 of the Ye filter is S1, and the signal from the pixel 2 (interpolation pixel 2) having the transmittance ratio 1/16 of the Ye filter is S2. R1 is the center distance between the pixel to be interpolated and the pixel (hatched pixel), and the pixel to be interpolated to interpolation pixel 2 (hatched pixel)
If the center distance between and is r2, the pixel signal S to be interpolated (signal with a transmittance ratio of 1/16 from the position of the pixel with diagonal lines) can be expressed by the following equation. That is, the pixel signal to be interpolated can be created by adding the respective signals with the reciprocal of the distance from the interpolated pixel as a weight.

S = ((S1 / r1) + (S2 / r2) /
((1 / r1) + (1 / r2)) In addition to this, it is also possible to perform 4-pixel interpolation or 8-pixel interpolation with an increased number of interpolation pixels. Although the case where the optical attenuation filters having different transmittance ratios are used has been described, the same applies to the case where pixels having different aperture ratios are used.

[0039]

As described above, according to the present invention, the light attenuation filters capable of obtaining two or more kinds of transmittances are regularly arranged on the pixels (photoelectric conversion elements) or formed, or
By arranging pixels (photoelectric conversion elements) having aperture ratios of two or more types regularly to form a solid-state image sensor, the dynamic range can be expanded, and there is no time lag of the subject on the screen and It is possible to realize an imaging device that can obtain a natural image in which a signal does not change suddenly.

[Brief description of drawings]

FIG. 1 is a configuration diagram of a solid-state image pickup element in an image pickup apparatus according to a first embodiment of the present invention.

FIG. 2 is a diagram showing changes in exposure time and exposure amount by one shutter operation in the image pickup apparatus according to the first embodiment of the present invention.

FIG. 3 is a diagram showing the luminance signal level with respect to the amount of incident light when the luminance signals are combined in the image pickup apparatus according to the first embodiment of the present invention.

FIG. 4 is a configuration diagram of a solid-state image pickup element in an image pickup apparatus according to a second embodiment of the present invention.

FIG. 5 is a diagram showing changes in exposure time and exposure amount by one shutter operation in the image pickup apparatus according to the second embodiment of the present invention.

FIG. 6 is a diagram showing a luminance signal level with respect to an amount of incident light when a luminance signal is combined in the image pickup apparatus according to the second embodiment of the present invention.

FIG. 7 is a diagram for explaining an example of spatial interpolation in the image pickup apparatus according to the first and second embodiments of the present invention.

FIG. 8 is a diagram for explaining an example of spatial interpolation in the image pickup apparatus according to the first and second embodiments of the present invention.

FIG. 9 is a configuration diagram of a solid-state imaging device in a conventional imaging device.

FIG. 10 is a diagram showing changes in the exposure time and the exposure amount by one shutter operation in the conventional imaging device.

FIG. 11 is a diagram showing the luminance signal level with respect to the amount of incident light when the luminance signals are combined in the conventional imaging device.

[Explanation of symbols]

101 Pixel with transmittance ratio 1 102 Pixels with transmittance ratio 1/4 103 Pixels with transmittance ratio 1/16 104 Pixels with transmittance ratio 1/32 401 Pixel with 1 aperture ratio 402 Pixel with aperture ratio 1/4 403 Pixel with aperture ratio 1/16 404 Pixel with aperture ratio 1/32

Claims (5)

[Claims] 1. A solid-state imaging device in which light-attenuating filters having different transmittances are regularly arranged and formed on each photoelectric conversion element having the same aperture ratio so that the exposure amount to each photoelectric conversion element is two or more. An image pickup apparatus comprising: an element; and a synthesizing unit for synthesizing a plurality of image signals having different exposure amounts obtained by processing signals output from the solid-state image pickup element into one image signal. 2. The image pickup device according to claim 1, wherein a color filter is provided on the photoelectric conversion element. 3. The image pickup apparatus according to claim 1, further comprising a color filter having a different transmittance in place of the light attenuation filter. 4. A solid-state imaging device having two or more types of exposure amounts for each photoelectric conversion device by forming photoelectric conversion devices having different aperture ratios in a regular arrangement, and outputs from the solid-state imaging device. An image pickup apparatus comprising: a synthesizing unit that synthesizes a plurality of image signals having different exposure amounts obtained by processing the signals into one image signal. 5. The image pickup device according to claim 4, further comprising a color filter having the same transmittance on the photoelectric conversion element.