JP5054659B2 - Imaging device - Google Patents

Description

  The present invention relates to a digital imaging device, and more particularly to an imaging device capable of acquiring a high-quality image.

  In recent years, digital cameras have become popular as imaging devices that acquire still images and moving images. For acquisition of an image, an imaging element in which pixels including photoelectric conversion elements that generate a signal value (pixel value) corresponding to the amount of received light are arranged in a matrix / honeycomb shape is used. Examples of the image pickup device include a complementary metal oxide semiconductor (CMOS) and a charge coupled device (CCD).

When acquiring a color image using an image sensor, for example, incident light related to a subject is separated into red, green, and blue light components by a dielectric multilayer film, and light (components) of each color is separated by different image sensors. There is a way to receive light.
As another method, a Bayer array color filter having a characteristic of selectively transmitting one of red, green, and blue light components corresponding to the pixel of the element is provided on the light receiving surface of the image sensor. There is a method of receiving the light.

  By the way, in order to improve the reproducibility of the color relating to the subject, an imaging device that acquires images of so-called four colors or more has been proposed (see Patent Document 1).

FIG. 11 is a schematic configuration diagram of the imaging apparatus 100 disclosed in Patent Document 1.
Reference numeral 101 denotes a lens that collects light related to the subject S. Reference numeral 102 denotes a spectroscopic prism (hereinafter referred to as a light separation unit), which selectively reflects light components in a predetermined wavelength region from incident light related to the subject S toward a first optical path (hereinafter referred to as a first optical path). Further, the incident light is transmitted toward a second optical path (hereinafter referred to as a second optical path).

  Reference numeral 111 denotes a first image sensor (hereinafter referred to as a first image sensor) arranged in the first optical path, and reference numeral 112 denotes a second image sensor (hereinafter referred to as a second image sensor) arranged in the second optical path. ). The color filter is provided on the light receiving surface of each image sensor (111, 112).

FIG. 12A is a schematic plan view of a color filter provided on the light receiving surface of the first image sensor 111. Red (R), green (corresponding to the pixels (matrix) of the first image sensor 111 is shown. G) shows a state in which a color filter having a characteristic of selectively transmitting one of the light components of blue (B) is provided. Note that the signal values obtained by the pixel “R1 / G1 / B1” in FIG. 12A are only red (R) / green (G) / blue (B).
FIG. 12B is a schematic plan view of a color filter provided on the light receiving surface of the second image sensor 112, similar to FIG.

FIG. 13 is a diagram illustrating the spectral reflectance characteristics of the separation unit 102.
FIG. 14 is a diagram showing the spectral sensitivities of R, G, and B in the first image sensor 111 with a color filter. R in the figure corresponds to the matrix (pixel) “R” shown in FIG. The spectral sensitivity of the pixel of the element 111 in the filter portion to be applied, G is the spectral sensitivity of the pixel of the element 111 in the filter portion corresponding to the matrix “G”, and B is the filter corresponding to the matrix “B”. It is the figure which showed the spectral sensitivity of the pixel of the said element 111 in the part. The same applies to the second image sensor 112.

  The incident light is separated (reflected) by the light separation unit 102 having the spectral reflectance characteristic shown in the spectral reflectance characteristic graph of FIG. 13, and the reflected light of the incident light has the light receiving characteristics shown in the spectral sensitivity graph of FIG. When light is received by the first image sensor 111 having a color filter, the spectral sensitivity of the image sensor 111 is as shown in FIG.

  Similarly, when the incident light is separated (transmitted) by the light separation unit 102 and the transmitted light of the incident light is received by the second imaging element 112 with the same filter, the spectral sensitivity in the imaging element 112 is As shown in FIG.

As shown in FIG. 15 and FIG. 16, the image sensor of the image capturing apparatus 100 disclosed in Patent Document 1 can receive light in six different wavelength regions.
Thereafter, the imaging apparatus 100 converts the signal value of a color other than the color generated in each pixel of the first imaging element 111 illustrated in FIG. 12A to the signal value of the color generated in the peripheral pixels of the pixel. Based on the calculation and generation (demosaic processing), image data is generated.

According to the demosaicing process, the red (R) signal value of the pixel “G1” of the first image sensor 111 is calculated, for example, by taking the average of the signal values of the pixels “R1” adjacent to the left and right. Similarly, the blue (B) signal value is calculated by taking the average of the signal values of the pixels “B1” adjacent to each other in the vertical direction.
In addition, the signal value can be calculated by multiplying the signal value by a predetermined coefficient in accordance with the distance from the pixel “G1” of the first image sensor 111.

The same demosaic process is also performed on the pixels of the second image sensor 112 to generate image data.
Then, the image data generated by the first image sensor 111 and the second image sensor 112 are combined and further subjected to predetermined image processing to generate image data related to the subject S.
JP 2004-172832 A

Thus, if light is separated for each wavelength region and image data is generated, the light use efficiency can be improved and the color reproducibility can be improved.
However, the spectral sensitivity (see FIGS. 15 and 16) in each image sensor (111, 112) of the imaging device 100 is compared with the spectral sensitivity (see FIG. 14) in the pixel corresponding to each filter of the image sensor with color filter. It will decline. This is because the wavelength region is divided into two by the separation unit 102.

  Therefore, when trying to obtain a so-called six-color image using the imaging apparatus 100, the spectral sensitivity is reduced to about half. Therefore, in order to compensate for the lowered sensitivity, the image of the subject is imaged with the gain of the image sensor increased, but this increases noise.

  In particular, as shown in red (R) of FIG. 14, when the sensitivity characteristic is not symmetric with respect to the sensitivity center wavelength, it is difficult to divide the wavelength region by 1: 1, and the spectrum when acquiring images of four colors or more is obtained. The sensitivity may be 30% or less before the wavelength region division. When a dark subject is imaged, the gain is further increased in order to compensate for the decrease in spectral sensitivity. However, in this case, only the noise increases, and the image quality deteriorates.

  The present invention has been made in view of such a situation, and by suppressing noise generation due to gain processing executed for the purpose of compensating for a decrease in the amount of light received by the image sensor, the SN ratio is improved, and the image resolution is further improved. An object of the present invention is to provide an imaging device capable of performing the above.

The first technical means is configured to reflect / transmit incident light from a subject toward the first optical path, and transmit / reflect the incident light toward the second optical path, and to the first optical path according to the amount of received light. A second imaging device including a first imaging device in which a plurality of pixels including photoelectric conversion elements that generate a signal value are arranged, and a plurality of pixels in which the photoelectric conversion devices having different wavelength regions for receiving light are arranged in the second optical path. Light that includes an element and that separates incident light so that the sensitivity received by the first imaging element is a visibility characteristic, and the light separation section reflects / transmits toward the first optical path. An imaging apparatus characterized by variably controlling components .

  According to a second technical means, in the first technical means, the number of pixels of the first image sensor and the number of pixels of the second image sensor are in a one-to-one correspondence.

  According to a third technical means, in the first technical means, the number of pixels of the first image sensor and the number of pixels of the second image sensor are the same, and the pixels of the first image sensor and the second image sensor The correspondence relationship between the pixel of the element and the pixel of the first image sensor is shifted by half a pixel.

According to a fourth technical means, in any one of the first to third technical means, red, green and blue colors are obtained from an image acquired by the first image sensor and an image acquired by the second image sensor. An image having a primary color is generated.

  According to the present invention, it is possible to provide an imaging device with high color reproducibility and image resolution and with little deterioration in image quality.

Example 1
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. The configuration in each drawing is exaggerated for easy understanding, and is different from the actual interval and size.

FIG. 1 is a schematic configuration diagram and a functional block diagram of an imaging apparatus 1 according to the present invention.
The light separation unit 3 selectively reflects the light component indicating the characteristic of spectral luminous efficiency, which is human visual sensitivity, from the incident light related to the subject S collected by the lens 2 toward the first optical path P1. Furthermore, the incident light is transmitted toward the second optical path P2. That is, the light separation unit 3 separates the incident light so that the sensitivity received by the first image sensor 11 becomes the visibility characteristic.

  The first image sensor 11 is an image sensor provided in the first optical path P1, and a plurality of pixels including photoelectric conversion elements that generate a signal value corresponding to the amount of received light are arranged in a matrix or a honeycomb array.

The second imaging element 12 is an imaging element provided in the second optical path P2, and a plurality of pixels including photoelectric conversion elements having different wavelength regions for receiving light are arranged in a matrix or a honeycomb arrangement.
Specifically, the light receiving surface of the second image sensor 12 has a Bayer array color having a characteristic of selectively transmitting one of red, green, and blue light components corresponding to the pixels of the element 12. A filter 12a is provided.
In addition, each image pick-up element (11, 12) is comprised from CMOS, CCD, etc. as mentioned above.

  The image processing unit 13 performs demosaic processing for each pixel of the second image sensor 12. Further, an image having primary colors of red, green, and blue is generated from the image acquired by the first image sensor 11 and the image acquired by the second image sensor 12. In addition, various kinds of image processing may be performed on the image data generated by this generation processing and recorded in a memory (not shown).

  The operation unit 14 includes keys for operating the imaging device 1, and the display unit 15 is a display unit that displays image data recorded in the memory and status information of the imaging device 1.

  The control unit 16 includes a CPU, a ROM, a RAM, and the like, and controls each functional block.

Hereinafter, details of the light separation unit 3, the image sensors (11, 12), and the image processing unit 13 will be described.
FIG. 2 is a diagram showing the reflectance characteristics of the dielectric multilayer film provided in the light separation unit 3. As shown in the reflectance characteristic graph of FIG. 2, the dielectric multilayer film of the light separation unit 3 selectively reflects incident light, particularly light in the vicinity of about 555 nm.

  FIG. 3 is a diagram showing the spectral sensitivity of each image sensor (11, 12). As is apparent from the figure, each of the imaging elements has a high sensitivity characteristic with respect to light around 480 nm.

FIG. 4A is a schematic plan view of a color filter provided on the light receiving surface of the second image sensor 12, where R is a red filter that selectively transmits red light, and G is green light. A green filter that selectively transmits blue light B in the figure indicates a blue filter that selectively transmits blue light.
The color filter has a green filter G as a center, and two blue filters B in the vertical direction and a red filter R in the horizontal direction are arranged in a Bayer array.
A symbol Gx indicates one of the green filters G (pixels).

FIG. 4B shows a schematic plan view of the light receiving surface of the first image sensor 11 in which the color filter is not provided. In the drawing, Y indicates a tristimulus value Y described later.
Note that a symbol Yx indicates a pixel corresponding to the symbol Gx in FIG.

  Thus, the number of pixels of the second image sensor 12 and the number of pixels of the first image sensor 11 correspond one-to-one, and the area of one pixel is the same.

  FIG. 5 is a diagram showing the transmittance of the color filter, in which R is the transmittance of the filter portion corresponding to the pixel “R” shown in FIG. 4A, and G is the pixel “G”. FIG. 7B is a diagram showing the transmittance of the filter portion corresponding to the pixel “B”.

  From FIG. 3 and FIG. 5, the spectral sensitivities of R, G, and B in the second image sensor 12 with the color filter have the characteristics shown in FIG.

  When incident light related to a subject is separated (transmitted) by the light separation unit 3 having reflectance characteristics shown in FIG. 2 and received by the second imaging element 12 with the color filter, the spectral sensitivity of the element 12 2 and FIG. 14 are as shown in FIG.

  Comparing the spectral sensitivities of each wavelength region in FIG. 14 and FIG. 6 by area ratio (integration ratio), the red wavelength region sensitivity in FIG. 6 is about 75%, the green wavelength region sensitivity is about 70%, compared to FIG. The blue wavelength region sensitivity is about 95%. When the conventional method is used, the ratio of each wavelength region sensitivity is about 35% to 60% on average.

  As described above, in the method of this embodiment, when light in the red, green, and blue wavelength regions is received by the image sensor, the spectral sensitivity is higher than that of the conventional method, so that image data with less noise can be obtained. Can do.

  Further, from FIGS. 2 and 3, the spectral sensitivity in the first image sensor 11 without the color filter is as shown by a curve Y in FIG.

  The curve Y indicating the characteristic is similar to the curve V indicating the spectral luminous efficiency characteristic which is human visual sensitivity, and the signal value obtained by the first image sensor 11 corresponds to the tristimulus value Y. To do. In other words, the reflectance characteristic of the dielectric multilayer film is set so that the signal value obtained by the first image sensor 11 has the tristimulus value Y characteristic.

As described above, the spectral sensitivity of the first image sensor 11 is as shown in FIG. 7, but the area ratio of the spectral sensitivity is about the same as the green wavelength region in FIG. This is because the peak of the reflectance characteristic in the dielectric multilayer film is set to about 50%.
Therefore, by setting the reflectance characteristic peak to 45% to 55%, the first image sensor 11 can generate image data with less noise.

  By the way, human vision is more sensitive to changes in brightness than changes in color, and it is known that important factors of image quality such as image resolution and sharpness depend greatly on the resolution of brightness. ing.

  In the imaging apparatus 1 according to the present invention, the color filter (colored pixel array) is a Bayer array as in the conventional example, and therefore, the RGB resolution is not different from the conventional example. However, since the luminance value (brightness) of the pixel can be acquired by the first image sensor 11, image data with a high resolution can be obtained.

Next, an example of image processing executed by the image processing unit 13 will be described.
First, demosaic processing, that is, processing for generating a signal value (pixel value) of a color other than the color generated by the pixel of the second image sensor 12 is executed.
Here, an example of demosaic processing will be described. First, in the G pixel (target pixel) of the second image sensor 12, the signal value average A of four neighboring pixels is calculated. Further, a division value D is calculated by dividing the signal value (Y value) of the pixel of the first image sensor 11 corresponding to the target pixel by the average value of the Y values of the four neighboring pixels of the pixel.
Then, by multiplying the average value A by the division value D, the signal value of the second image sensor 12 can be generated. Accordingly, it is possible to obtain a high-quality G image by performing demosaic processing of G pixels from a high-resolution image.

  A similar method can be applied to the R pixel and the B pixel. In addition, since the R pixel and the B pixel have a smaller number of pixels than the G pixel, only the R pixel and the B pixel may be subjected to general demosaic processing.

  As described above, by using the high-resolution Y value obtained from the first image sensor 11, it is possible to perform the demosaic process on the pixels of the second image sensor 12 with high accuracy.

  Further, it is possible to detect the edge direction based on the Y value generated by the pixel of the first image sensor 11 and to apply it to a demosaic process that considers the edge direction. Thereby, highly accurate demosaic processing can be performed, and a high-quality image can be obtained.

An example of newly generating RGB image data based on the RGB image data generated by the demosaic process and the Y image data generated by the first image sensor 11 will be described.
First, XYZ values are calculated from RGBY values (signal values). This can be obtained by calculation such as (Equation 1).

Here, if the sensitivity of the first image sensor 11 is not different from the spectral luminous efficiency, it is possible to obtain a Y value by setting i = j = k = 0 and giving a constant to l.
On the other hand, when the sensitivity of the first image sensor 11 has an error from the spectral luminous efficiency, the Y value can be slightly corrected by giving appropriate values to i, j, and k. It is possible to improve.

Next, as shown in (Expression 2), conversion from XYZ to RGB is performed. If RGB is a standard such as sRGB, a matrix value already determined as a matrix value may be used.
In other words, assuming that the matrices of (Equation 1) and (Equation 2) are A and B, respectively, in order to calculate RGB from RGBY, it is only necessary to calculate (Equation 3).

  In Equation 3, matrix C = matrix A × matrix B.

With the above method, RGB image data can be obtained from RGBY image data.
Here, the Y value is the highest resolution image that is not demosaiced, has a high contribution rate to the resolution, and the generated RGB image data is a high definition high quality image.

  By capturing the high-resolution brightness image Y by the above method, it is possible to acquire a high-quality image with high color reproducibility and high noise with reduced noise.

(Example 2)
Next, a second embodiment of the imaging apparatus according to the present invention will be described with reference to the drawings. The schematic configuration diagram and functional block diagram of the image pickup apparatus of the second embodiment are the same as those in FIG.

  FIG. 8A is a schematic plan view of a color filter provided on the light receiving surface of the second image sensor 12, similar to FIG. 4A, and FIG. 8B is a diagram of the first image sensor 11. FIG. 2 is a schematic plan view of a light receiving surface.

In the imaging device of Example 1, the number of pixels of the second imaging element 12 and the number of pixels of the first imaging element 11 correspond one-to-one, and the area of one pixel is the same.
However, in the image pickup apparatus according to the second embodiment, the number of pixels of the second image pickup element 12 and the number of pixels of the first image pickup element 11 correspond to each other on a plural basis.
For example, as shown in FIG. 8B, the number of pixels of the second image sensor 12 and the number of pixels of the first image sensor 11 are made to correspond to each other on a one-to-one basis. Is made to be ¼ of the pixel area of the first image sensor 11.

Thus, the pixel area (aperture ratio) of the first image sensor 11 is increased by the amount of decrease in the number of pixels of the first image sensor 11.
Therefore, the amount of light received by one pixel can be increased by about four times, so that even when a subject is imaged in a dark place, a sufficient amount of light can be obtained and image data with reduced noise can be generated. .

Further, since the amount of light is large, the shutter speed can be increased, and even when a moving subject is imaged, high-quality image data with little imaging blur can be generated.
In the first embodiment, since the number of pixels of the first image sensor 11 is large, there is an effect of increasing the resolution. However, when the high resolution by demosaic is sufficient, the first image sensor as in the present embodiment. It is preferable to reduce the number of pixels 11 and increase the pixel area.

In addition, another method for improving the resolution will be described with reference to FIG.
FIG. 9 shows the first image sensor 11 in the case where the pixel (number) of the second image sensor 12 and the pixel (number) of the first image sensor 11 shown in FIG. 3 is a diagram illustrating a state in which a correspondence relationship between a pixel and a pixel of a second image sensor 12 is shifted by a half pixel with respect to a pixel of the first image sensor 11. FIG.

As is apparent from the figure, one R pixel in the second image sensor 12 indicated by R, two G elements indicated by G, for one pixel of the first image sensor 11 indicated by Y. A pixel, one B pixel indicated by reference B, corresponds. In other words, four types of pixels of the first image sensor 11 correspond to one R pixel, G pixel, and B pixel.
By demosaicing using this shift, it is possible to obtain higher resolution image data.

  In the case of this demosaic, the average value of the signal values (Y values) of the four types of pixels of the first image sensor 11 may be used as the Y value described in the first embodiment. Next, using the G pixel acquired by the second image sensor 12 and the G pixel acquired by demosaic processing, the Y pixel acquired by the first image sensor 11 is divided into four. For example, the division is performed according to the ratio of four G pixels corresponding to the target Y pixel. Next, the RGB pixels are divided by the ratio of the divided Y pixel values. Thereby, it is possible to acquire a high resolution image.

  By changing the number of pixels (resolution) of the image sensor by the method described above, an image with reduced noise and image data with less motion blur can be obtained. Further, by shifting the pixel arrangement of the first image sensor and the second image sensor by half a pixel, high resolution can be achieved by demosaic.

(Example 3)
Next, a third embodiment of the imaging apparatus 1 according to the present invention will be described with reference to the drawings. The schematic configuration diagram and functional block diagram of the image pickup apparatus 1 of the third embodiment are the same as those in FIG.

  FIG. 10 is another diagram showing the reflectance characteristics of the dielectric multilayer film provided in the light separating section 3. In Examples 1 and 2, as described with reference to FIG. 2, the reflectance peak of the dielectric multilayer film is fixed to about 50%.

  In Example 3, as shown in FIG. 10, the reflectance peak is variable, for example, 50%, 80%, and 100%. That is, the light separation unit 3 variably controls the light component reflected (transmitted) toward the first image sensor 11 provided in the first optical path P1.

  Since the curves showing the characteristics of these three types of reflectance are similar to the curve V (see FIG. 6) showing the characteristics of the spectral luminous efficiency, which is the human visual sensitivity, the signal value obtained by the first image sensor 11 is obtained. Corresponds to the tristimulus value Y.

  Further, if the reflectance peak is increased, the amount of light incident on the first image sensor 11 increases, so that image data with less noise can be obtained even in a dark environment.

On the other hand, in the second image sensor 12, as the reflectance peak increases, the amount of light incident on the first image sensor 11 decreases.
However, as can be seen from FIG. 10, the wavelength region in which the reflectance characteristics vary greatly is limited. Therefore, even if the reflectance peak is set to 80%, the spectral sensitivity of each color of the second imaging element 12, particularly the spectral sensitivity of blue, is not significantly reduced compared to FIG.

As described above, the amount of light incident on the first image sensor 11 and the second image sensor 12 can be changed by changing the reflectance characteristics of the dielectric multilayer film provided in the light separation unit 3.
Therefore, it is possible to select whether to obtain image data with an emphasis on the noise level or to obtain image data with an emphasis on color reproducibility.
For example, when shooting in fine weather, the peak value of the reflectance is reduced because it is sufficiently bright. Further, when the brightness is insufficient as in indoor shooting, the peak value of the reflectance can be increased. By doing so, it is possible to acquire high-quality image data with reduced noise even when shooting under various environments.

  The reflectivity can be changed using various techniques. For example, the reflectance may be electrically controlled using a liquid crystal material. Alternatively, the light separating unit 3 may be formed of a dichroic mirror, and the dichroic mirror may be moved so that light incident from the lens is irradiated onto a dielectric multilayer film having a desired reflectance. (Movable light separation unit).

  In the above embodiment, a spectroscopic prism is used as the light separation unit 3, but a dichroic mirror or the like having wavelength selectivity may be used.

Further, the reflection characteristic of the dielectric multilayer film of the light separation unit 3 is reversed, and the light component in the predetermined wavelength region is selectively transmitted toward the first optical path from the incident light related to the subject. The light separation unit 3 may be configured to reflect toward the two optical paths.
In this case, the first optical path P1, the first image sensor 11, the second optical path P2, and the second image sensor 12 with the color filter 12a illustrated in the block diagram of FIG.

1 is a schematic configuration diagram and a functional block diagram of an imaging apparatus according to the present invention. It is the figure which showed the reflectance characteristic of the dielectric multilayer. It is the figure which showed the spectral sensitivity of each image sensor. It is an external view of the light-receiving surface of each image sensor. It is the figure which showed the transmittance | permeability of the color filter. It is a figure which shows the spectral sensitivity characteristic in a 2nd image sensor. It is a figure which shows the spectral sensitivity characteristic in a 1st image sensor. It is another external view of the light-receiving surface of each image sensor. It is a figure for demonstrating the other method of improving the resolution. It is another figure which showed the reflectance characteristic of the dielectric multilayer. 1 is a schematic configuration diagram of an imaging apparatus disclosed in Patent Document 1. FIG. FIG. 14 is an external view of a light receiving surface of an imaging element in an imaging apparatus disclosed in Patent Literature 1. It is the figure which showed the spectral reflectance characteristic of the isolation | separation part in the imaging device disclosed by patent document 1. FIG. It is the figure which showed the spectral sensitivity of R, G, B in the 1st image pick-up element with a color filter. It is a figure which shows the spectral sensitivity characteristic in the 1st image sensor with a color filter. It is a figure which shows the spectral sensitivity characteristic in the 2nd image pick-up element with a color filter.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1,100 ... Imaging device, 2,102 ... Lens, 3,103 ... Light separation part, 11, 111 ... Imaging element, 12, 112 ... Imaging element, 12a ... Color filter, 13 ... Image processing part, 14 ... Operation part , 15 ... display section, 16 ... control section, P1 ... first optical path, P2 ... second optical path.

Claims (4)

A light separation unit that reflects / transmits incident light from the subject toward the first optical path and transmits / reflects toward the second optical path;
A first imaging element in which a plurality of pixels including a photoelectric conversion element that generates a signal value corresponding to the amount of received light is arranged in the first optical path;
A second imaging element in which a plurality of pixels including photoelectric conversion elements having different wavelength regions for receiving light are arranged in the second optical path;
The light separating unit separates incident light so that the sensitivity received by the first image sensor becomes a visibility characteristic ,
The imaging apparatus, wherein the light separation unit variably controls a light component reflected / transmitted toward the first optical path .   The image pickup apparatus according to claim 1, wherein the pixels of the first image pickup element and the pixels of the second image pickup element have a one-to-one correspondence.   The number of pixels of the first image sensor is the same as the number of pixels of the second image sensor, and the correspondence relationship between the pixels of the first image sensor and the pixels of the second image sensor is the first image sensor. The imaging apparatus according to claim 1, wherein the imaging apparatus is shifted by a half pixel with respect to the pixel. An image acquired by the first image pickup device, the image acquired by the second image sensor, the red, green, any one of claims 1 to 3, characterized in that to generate the image to primary blue The imaging device according to one item.

Images