JP2011176710A - Imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an imaging apparatus which can obtain a color composite image and distance information using a plurality of imaging apparatuses, the imaging apparatus having improved sensitivity without degrading a resolution, and acquiring more color information. <P>SOLUTION: The imaging apparatus includes: two systems of imaging devices each for capturing a color image and outputting an image signal by performing photoelectric conversion on incident light for acquiring information of a distance to a subject; a distance information acquiring section for acquiring the information of the distance to the subject based on two systems of image signals; and a compositing section for compositing two systems of image signals to obtain a color image. Among the imaging devices, a first imaging device has a pixel constitution including a set of a transparent pixel and first and second primary color pixels among three primary colors of light, and a second imaging device has a pixel constitution including a set of the transparent pixel and a third primary color pixel and the first or second primary color pixel among three primary colors of light. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an imaging apparatus using a plurality of imaging elements that capture color images.

  An imaging device that is currently used generally uses an imaging device having red (R), green (G), and blue (B) color filters. By this color filter, the incident light is separated into red, green and blue components for colorization. As shown in FIG. 9, different color filters are arranged for each pixel of the image sensor, and two green (G) filters, one red (R) filter, and one blue (B) filter are arranged. The entire light receiving surface of the image sensor is covered as a set (area surrounded by a thick line). In the so-called Bayer array type imaging device having such a configuration, incident light is absorbed by the absorption type color filter and the amount of light is greatly reduced. In addition, the pixel size is reduced with recent high definition. Since the amount of light obtained is reduced, there is a problem that the amount of light taken in is reduced and the sensitivity is lowered.

  On the other hand, image sensors using cyan, magenta, and yellow, which are complementary color filters with higher transmittance, are also used as color filters. Since this complementary color filter generates a color signal based on a difference signal between pixels, there is a problem that the color becomes different from the originally assumed three primary colors of light. For example, blue color information can be obtained by adding cyan signals having green and blue information and magenta signals having red and blue information, and subtracting yellow signals having red and green information. In principle, blue information can be obtained in this way, but in reality, information on other wavelength ranges is also included, which becomes incorrect information, resulting in noise.

  In order to solve such a problem in increasing sensitivity by using a complementary color filter, an image sensor having a transparent filter has been proposed (for example, see Patent Document 1). As shown in FIG. 10, a red (R) filter and a blue (B) filter are arranged in an offset structure every three pixels in the horizontal and vertical directions, and a transparent filter (Y) is arranged therebetween. As a result, the sensitivity can be improved.

  Further, as an example of applying an imaging apparatus in addition to photographing a subject, there is a method for obtaining a distance (depth) to a target object such as a subject. Many techniques have been proposed in which two right and left cameras (imaging devices) are used, the amount of positional deviation between the left and right images is obtained, and the three-dimensional position is calculated based on the principle of triangulation (see, for example, Patent Document 2). ).

JP-A-8-23542 Japanese Patent Laid-Open No. 5-114099

  However, in the method of Patent Document 1, although the sensitivity is improved because the transparent pixel is used, the number of pixels for obtaining a color is very small compared to the transparent pixel, and the resolution of the color is greatly reduced. There is a problem that the resolution of the image is reduced. Also, transparent filters, red filters, and blue filters are used, and the green signal can only be obtained from the transparent filter, red filter, and blue filter. As a result, the green signal is obtained from the complementary color, and there is a lot of noise. The problem of becoming remains. Also, an example using a green filter instead of a transparent filter has been presented. In this case, a large number of green pixels are arranged, and the brightness corresponding to the sensitivity (visual sensitivity) of the human eye is improved. However, since only the green wavelength region, which is about 1/3 of the visible light region, can be used as sensitivity to actual light, it cannot be said that the sensitivity is improved as a color image.

  On the other hand, in the method of Patent Document 2, since the amount of positional deviation of each image is calculated by the two right and left imaging devices, two imaging devices are used to compare the same pixel value (video signal value) of the left and right images. The device needs to be an imaging device that exhibits substantially the same characteristics. That is, the image sensors of the two image pickup devices need to have the same filter combination. If one image sensor has a red pixel, a green pixel, and a blue pixel, the other also has the same combination. There is a need. In the case of improving the sensitivity with this configuration, for example, the imaging device of Patent Document 1 may be applied to the left and right imaging devices, but in exchange, the resolution that is a problem of Patent Document 1 is reduced. In addition to the sensitivity improvement, in order to obtain accurate color information, it is necessary to use other color pixels in addition to the red pixel, green pixel, and blue pixel. is there.

  The present invention has been made in view of such circumstances, and in an imaging apparatus capable of obtaining a color composite image and distance information using a plurality of imaging apparatuses, sensitivity is not reduced. It is another object of the present invention to provide an image pickup apparatus that can improve the image quality and can acquire a large amount of color information.

  The present invention is based on two image sensors that photoelectrically convert incident light and output an image signal to acquire a color image and distance information to a subject, and the two image signals. An imaging apparatus comprising: a distance information acquisition unit that obtains distance information to a subject; and a synthesis unit that obtains the color image by synthesizing the two systems of image signals, and the first imaging element of the imaging elements includes: Among the three primary colors of the transparent pixel and light, the first primary color pixel and the second primary color pixel have a set of pixel configurations, and the second imaging element is the third of the three primary colors of the transparent pixel and light. The primary color pixel and one of the first or second primary color pixels have a pixel configuration as a set.

  The present invention is characterized in that the pair of pixel configurations of the first and second imaging elements has two transparent pixels.

  In the present invention, the distance information acquisition unit calculates a parallax amount between the two systems by matching at least transparent pixel signals among the two systems of image signals, and based on the parallax amount, It is characterized by obtaining distance information to a subject.

  The present invention is characterized in that the synthesizer replaces a low illuminance pixel with a transparent pixel signal of any one of the two systems of image signals.

  The present invention generates a signal of the remaining one primary color pixel of the three primary colors based on the signal of the transparent pixel of the first or second image sensor and the signal of the two primary color pixels of the image sensor. And a primary color signal generation unit.

  According to the present invention, in an imaging device that can obtain a color composite image and distance information using a plurality of imaging devices, the sensitivity is improved without causing a decrease in resolution, and more color information is obtained. Can be obtained.

It is a block diagram which shows the structure of the imaging device by one Embodiment of this invention. It is explanatory drawing which shows arrangement | positioning of the color filter of the image pick-up element of this invention. It is a figure which shows the transmission spectral characteristic of the color filter of this invention. It is the calculated color filter transmission spectral characteristic of the image sensor of the present invention. It is a comparison of the red color filter transmission spectral characteristics of the image sensor of the present invention. It is a schematic diagram which shows the difference in the amount of parallax by right and left parallax. It is a conceptual diagram of another form of the image pick-up element of this invention. It is a comparison of the green color filter transmission spectral characteristics of the image sensor of the present invention. It is a conceptual diagram explaining the Bayer arrangement | sequence system of a prior art. It is a conceptual diagram which used the transparent filter (Y) for the Bayer arrangement system of a prior art.

  Hereinafter, an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. In this figure, reference numerals 11 and 12 denote image sensors, which convert incident light into electrical signals and output them. Reference numerals 21 and 22 are color filters provided on the light incident surfaces of the image sensors 11 and 12. Reference numerals 31 and 32 are lenses that form an image on the light incident surfaces of the imaging elements 11 and 12, and may be configured by a plurality of lenses. Reference numeral 4 denotes a signal output from the two image sensors 11 and 12, obtains a correlation for each pixel, detects a parallax, and a parallax signal P indicating the parallax and distance information according to the parallax. A correlation detection unit (distance information acquisition unit) that outputs D. Reference numeral 5 denotes a parallax correction unit that corrects parallax based on the parallax signal P input from the correlation detection unit 4. In the present embodiment, the parallax correction unit 5 corrects the parallax for the red pixel signal and the blue pixel signal based on the image sensor 11 and converts them into signals B′rr and Rrr from the viewpoint of the image sensor 12. Reference numeral 6 denotes a red pixel electrical signal Rc, a green pixel electrical signal Gc, and a blue pixel electrical based on image synthesis based on a signal output from the parallax correction unit 5 and a signal output from the image sensor 12. It is a synthesis unit that outputs a composite image signal composed of the signal Bc.

  Light from the subject is imaged on the incident surfaces of the image sensors 11 and 12 by the lenses 31 and 32. The image sensors 11 and 12 photoelectrically convert the imaged light and output an electrical signal. The color filter 21 includes transparent (W), red (R), and green (G) filters, and the image sensor 11 includes a transparent pixel electrical signal Wr, a red pixel electrical signal Rr, and a green pixel electrical signal Gr. Is output. The color filter 22 includes transparent (W), blue (B), and green (G) filters, and the image sensor 12 includes a transparent pixel electrical signal Wl, a blue pixel electrical signal Bl, and a green pixel electrical signal Gl. Is output. Reference numeral 7 generates a blue pixel electric signal B′r based on the electric signal output from the image sensor 11. Reference numeral 8 generates a red pixel electric signal R′l based on the electric signal output from the image sensor 12. Reference numeral 9 denotes a composite image signal output from the combining unit 6, a blue pixel signal output from the blue signal generation unit 7, and a blue pixel signal output from the blue signal generation unit 8. Society) A color space conversion unit for converting to an XYZ color system which is a standard color system.

  Next, with reference to FIG. 2, the arrangement positions of the filters of the color filters 21 and 22 provided in the image sensors 11 and 12 shown in FIG. 1 will be described. FIG. 2 is a diagram showing a part of each of the image sensors 11 and 12 as viewed from the lenses 31 and 32 side. The image pickup device 11 is an image pickup device used for the right image pickup device of two right and left image pickup devices, for example, two transparent filter pixels (W) (hereinafter referred to as transparent pixels (W)) and one red filter. A pixel (R) (hereinafter referred to as a red pixel (R)) and one green filter pixel (G) (hereinafter referred to as a green pixel (G)) are configured as a set (a thick line region). Yes. On the other hand, the imaging device 12 is an imaging device used for the left imaging device, for example, and includes two transparent pixels (W), one green pixel (G), and one blue filter pixel (B) (hereinafter, blue pixel). (Referred to as (B)) as a set (thick line region).

  Here, the transmission characteristics of the right color filter 21 of the right image sensor 11 will be described. FIG. 3 is a diagram showing the light transmission characteristics of the color filter 21 shown in FIG. In FIG. 3, the horizontal axis indicates the wavelength of light, and the vertical axis indicates the transmittance of the filter. Since the right color filter 21 is composed of transparent pixels (W), red pixels (R), and green pixels (G), the respective transmittances are as shown in FIG. Here, as a simple example, the transmittance of a transparent pixel is set to 100%, and both the red pixel and the green pixel have a maximum transmittance of 100%. The transmittance of the transparent pixel (W) is transparent because it shows 100% transmission over the entire wavelength range without being absorbed in any wavelength range, and the red pixel (R) absorbs in the wavelength range of about 400 nm to 600 nm. Therefore, it becomes red because there is only transmission in the red wavelength region. The green pixel (G) also becomes green due to absorption in the short wavelength blue wavelength region and the long wavelength red wavelength region.

  FIG. 4 is a diagram showing the light transmission characteristics of the color filter 22 shown in FIG. Similarly, in FIG. 4, the horizontal axis indicates the wavelength of light, and the vertical axis indicates the transmittance of the filter. Since the left color filter 22 includes transparent pixels (W), green pixels (G), and blue pixels (B), the respective transmittances are as shown in FIG. Similarly, the maximum transmittance of the blue pixel is 100%.

  Here, the right color filter 21 includes a transparent pixel (W) for acquiring information over the entire wavelength range, and a red pixel (R) and a green pixel (G) among the three primary colors of light. By using it, it is possible to acquire a blue pixel (B) that does not exist in the right image sensor 11 although it is pseudo. For example, the blue signal generation unit 7 artificially subtracts a red pixel signal Rr and a green pixel signal Gr, each of which is multiplied by a preset coefficient, from the transparent pixel signal Wr output from the image sensor 11. A blue pixel signal B′r is obtained. At this time, as the coefficients to be multiplied by the red pixel signal Rr and the green pixel signal Gr, for example, all the wavelengths of the red pixel and the green pixel transmittance with respect to the integral value of the whole wavelength region of the transmittance of the transparent pixel shown in FIG. Use the integral of the region. Similarly, the left color filter 22 includes a transparent pixel (W) that acquires information over the entire wavelength range, and a green pixel (G) and a blue pixel (B) of the three primary colors of light. The red pixel (R) that does not exist in the left image sensor 12 can be obtained although it is pseudo. Similar to the blue signal generation unit 7, the red signal generation unit 8 generates a green pixel signal Gl and a blue pixel signal Bl obtained by multiplying the transparent pixel signal Wl output from the image sensor 12 by a preset coefficient. To obtain a pseudo red pixel signal R′l.

  For example, FIG. 5 shows a result of comparing the characteristics of the pseudo red pixel (R ′) calculated from the left color filter 22 and the red pixel (R) of the right color filter 21 in this way. Although it has a transmission characteristic that transmits a long wavelength and shows red among the three primary colors, the transmission characteristic is somewhat different, and it can be seen that R 'transmits a wider wavelength range than R'. In general, the color artificially calculated from other color information in this way tends to widen the wavelength range, which contributes to the noise described above. As will be described in detail later, if the pseudo color (for example, red here) is associated with the original color, the pseudo color can be replaced with the original color. By taking the difference between the pseudo color and the original color, it can be used as pixel information having a color filter with a new transmission characteristic, and in this case, it greatly contributes to acquisition of accurate color information.

  Next, a method in which the correlation detection unit 4 calculates the distance (depth) to the target object by the stereo matching method using the right imaging element 11 and the left imaging element 12 will be described. The right image sensor 11 and the left image sensor 12 capture the same subject, but are arranged at a certain distance (baseline length) on the left and right, so that parallax occurs and images slightly different on the left and right can be acquired. . For example, as shown in FIG. 6, when the leftmost imaging device 601, the house 602 in the foreground, and the tree 603 in the foreground are acquired by the left and right imaging devices, In the obtained image, the horizontal shift amount between the tree 601 and the house 602 is i, and the horizontal shift amount between the tree 603 and the house 602 is j. On the other hand, in the image obtained by imaging with the left imaging device, the lateral displacement amount between the tree 601 ′ and the house 602 ′ is i ′, and the lateral displacement amount between the tree 603 ′ and the house 602 ′ is j ′. It becomes. At this time, i 'is larger than i, and j' is larger than j. Further, the amount of change is larger for j '/ j than for i' / i. This is because the apparent movement amount due to the parallax is different between the one located in the foreground and the one located in the back, and the one that is in the trouble seems to move more greatly. That is, the distance (depth) to the target object is estimated by looking at the amount of change (referred to as the amount of parallax) in the image of the left image pickup device (referred to as the left image) and the image of the right image pickup device (referred to as the right image). It becomes possible. As the absolute value of the depth, distance information may be calculated from this parallax amount using the principle of a known triangulation method.

Here, the house 602, 602 ′ is used as a reference for calculating the shift amount, but it is not the subject in the image such as the house 602, 602 ′, but the center, left edge, right edge, etc. of each image. A predetermined point or perpendicular line in the image may be used as a reference.
Further, as the amount of parallax, a difference between the left image and the right image at a distance from the above-described reference may be used.
The correlation detection unit 4 outputs a parallax signal P indicating the parallax amount calculated in this way and distance information D. In the present embodiment, as the amount of parallax, for each pixel in the image viewed from the viewpoint of the image sensor 12, a shift to the pixel in the image viewed from the viewpoint of the image sensor 11 corresponding to the pixel is used. For example, in the left image, the amount of parallax for a certain pixel is 10 pixels to the left. For example, the left and right are represented by using positive and negative, such as negative on the left and positive on the right, and the above example of the amount of parallax may be −10 pixels. This is because the amount of parallax is used to correct an image based on a signal output from the image sensor 11 to an image viewed from the viewpoint of the image sensor 12 in the parallax correction unit 5 described later.

  As described above, the depth information is obtained by obtaining the amount of parallax. However, in order to obtain the amount of parallax, the images obtained by the left and right imaging devices are compared and the position (pixel position) of the same image is obtained. Need to be identified. For example, for a subject captured by a pixel in the left image, it is necessary to specify which pixel in the right image captures the same subject. Such association of two images is generally called stereo matching. Various methods have been proposed for stereo matching. In the present embodiment, the correlation detection unit 4 uses a block of 9 pixels in total of 3 pixels in the vertical and horizontal directions, and performs matching while comparing which position in the right image the block of the left image corresponds to. That is, when the subject detected by the pixel A of the left image identifies which pixel of the right image the same subject is captured by the mutual detection unit 4, a block centered on the pixel A of the left image and the right image The normal cross-correlation with each block centered on each of the pixels is calculated, and it is determined that the pixel at the center of the block where the normal cross-correlation has the largest value captures the same subject as the pixel A. Note that the block for which normal cross-correlation is calculated may be limited to a block centered on a pixel in a range corresponding to the position of the pixel A. Since there are many pixels having the same pixel value in pixel matching, matching is performed by setting several pixels in this way, and pixel positions at which all the pixels show a good match are used.

  Moreover, weighting according to the position in the set may be performed for each pixel constituting the set, and importance may be placed on matching of specific pixels in the set. Further, instead of the pixel value itself, a luminance value may be calculated from these pixel values, and matching may be performed using the value. For this matching method, various conventionally proposed methods can be used. In the matching between the left and right images as described above, the same pixel value (image signal value) is searched for on the left and right. Therefore, it is desirable that the left and right images that capture the same position of the subject have the same pixel value. For this purpose, it is desirable that the color filter characteristics of the matching pixels are also the same. Therefore, the correlation detection unit 4 in the present embodiment matches the green pixels (G) and the transparent pixels (W) of the right image sensor 11 and the left image sensor 12. Thereby, stereo matching using three of the four pixels can be performed without matching different red pixels (R) and blue pixels (B) between the left and right imaging elements, and thus the amount of parallax can be estimated from the left and right images. Thus, the parallax map can be calculated by the right image sensor 11 and the left image sensor 12 having different color filter characteristics. The image obtained by the transparent pixel (W) is an image using light in all the wavelength ranges, and the transparent pixel (W) is twice as large as a pixel that has passed through a color filter such as a green pixel (G). Since the obtained image is obtained, it is possible to capture even a dark, low-illuminated subject as compared with an image obtained with pixels through a color filter. Therefore, by using the transparent pixel (W) for matching as described above, it is possible to perform matching of the low luminance part with high accuracy.

Furthermore, for example, the correlation detection unit 4 is transparent for pixels with low illuminance such as an average value of green pixels (G) of a block centered on the pixel in the left image, which is smaller than a preset threshold value. Only the pixels (W) may be matched. In an image obtained with pixels that have passed through a color filter, the influence of noise increases on the pixel value in pixels with low illuminance. Therefore, in this way, by matching only the transparent pixels (W) in the low illuminance pixels, it is possible to perform matching of the low luminance part with high accuracy. Note that the determination of whether or not the illumination is low is not limited to the above-described determination method, and may be another determination method. For example, the determination may be made based on whether the pixel value of the green pixel (G) of the pixel is smaller than the threshold value, or by determining whether the pixel value of the transparent pixel (W) of the pixel is smaller than the threshold value. May be.
Further, if the pixel has a low illuminance, only the transparent pixels (W) may be matched, and if not a low illuminance pixel, only the green pixels (G) may be matched.

  At this time, the red pixel (R) of the right image sensor 11 and the blue pixel (B) of the left image sensor 12 can estimate a substantially accurate amount of parallax based on the matching result of other pixels. For example, if the amount of parallax of surrounding transparent pixels (W) and green pixels (G) is 5 pixels horizontally, the amount of parallax between red pixels (R) and blue pixels (B) is also 5 pixels horizontally. Can be predicted to be minutes. That is, as shown in FIG. 5, the pseudo red pixel (R ′) calculated from the red pixel (R) of the right color filter 21 of the right image sensor 11 and the left color filter 22 of the left image sensor 12. Thus, the same subject can be seen by the red pixel (R) and the pseudo red pixel (R ′) which are different color filters. Similarly, the amount of parallax between the pseudo blue pixel (B ′) of the right color filter 21 of the right image sensor 11 and the blue pixel (B) calculated from the left color filter 22 of the left image sensor 12 is also estimated. be able to. From these, in addition to the green pixel (G) having the same color filter characteristics, five pixels, that is, a blue pixel (B), a pseudo blue pixel (B ′), a red pixel (R), and a pseudo red pixel (R ′). An image having filter characteristics can be acquired. Thereby, it is possible to use R, G, B with reduced noise using the original color instead of the pseudo-calculated color, R ″ obtained by the difference between R ′ and R, or With the information on a new wavelength range of B ″ obtained by the difference between B ′ and B ′, a large amount of color information is obtained by multiband, which is more than the three primary colors, and more accurate color information can be obtained. It becomes possible. Thereby, it is possible to sufficiently cover the lack of color information due to the transparent pixels employed to increase sensitivity.

  In the present embodiment, the parallax correction unit 5 detects the correlation between the signal Rr of the red pixel (R) from the image sensor 11 and the signal B′r of the pseudo blue pixel (B ′) from the blue signal generation unit 7. In accordance with the parallax signal P from the unit 4, the red pixel (R) signal Rrr and pseudo-blue pixel (B ′) signal B′rr of the image from the viewpoint of the image sensor 12 are converted and output.

  The combining unit 6 converts the signal Gl of the green pixel (G) and the signal Bl of the blue pixel (B) from the image sensor 12 and the signal Rrr of the red pixel (R) from the parallax correction unit 5 into red, green, A color image signal Rc, Gc, Bc consisting of the three primary colors of blue is output. However, the combining unit 6 replaces the signals G1, B1, and Rrr with the signal W1 of the transparent pixel (W) from the image sensor 12 and outputs the signals Rc, Gc, and Bc for the low-illuminance pixels. . The image obtained by the transparent pixel (W) is an image using light in all the wavelength ranges, and the transparent pixel (W) is twice as large as a pixel that has passed through a color filter such as a green pixel (G). Since this is an obtained image, replacing the low illuminance area with an image obtained with transparent pixels (W) in this way, red, green, and blue pixels were not captured by noise. A subject having a value relatively close to 0 can also be captured. In addition, since the human eye cannot recognize the color when the illuminance is low, even if an image obtained from a transparent pixel (W) is obtained in such a low illuminance region, human eyes can see the image. Does not reduce the amount of information that can be obtained. In addition, although the method demonstrated in the correlation detection part 4 can be used for the determination method of whether it is a pixel of low illumination intensity, it does not need to be the same method as the correlation detection part 4. FIG.

  The color space conversion unit 9 receives the color image signals Rc, Gc, Bc from the synthesis unit 6, the pseudo blue pixel (B ′) signal B′rr from the parallax correction unit 5, and the red signal generation unit 8. The pseudo red pixel (R ′) signal R′l is converted into the XYZ color system. This conversion is performed by multiplying a preset matrix by a vector composed of these signals Rc, Gc, Bc, R′l, B′rr as shown in the equation (1). Thus, as described above, information on the pseudo red pixel (R ′) and the pseudo blue pixel (B ′) is used in addition to the information on the red pixel (R), the green pixel (G), and the blue pixel (B). Therefore, more color information than the three primary colors is used, and more accurate color information can be acquired. Further, a technique for estimating low-dimensional to high-dimensional information represented by winner estimation may be used.

  The right color filter 21 and the left color filter 22 are the same for the green pixel. In the right image sensor 11 and the left image sensor 12, two transparent pixels and one green pixel are used. Have the same transmission characteristics, and the remaining one pixel in units of four pixels shows different transmission characteristics, but the present invention is not limited to this configuration. For example, as illustrated in FIG. 7, in the right image sensor 71 and the left image sensor 72, two of the transparent pixels (W) have the same transmission characteristics, and the red pixel of the right image sensor 71. Since (R) and the blue pixel (B) of the left image sensor 72 have different colors, they naturally have completely different transmission characteristics. The green pixel (G1) of the right image sensor 71 and the green of the left image sensor 72 are green. The pixel (G2) has the same green color but different transmission characteristics.

  The transmission characteristics of this green pixel are shown in FIG. Although it is green in substantially the same wavelength region, the green pixel (G1) and the green pixel (G2) have slightly different transmission characteristics. In this case, the right imaging device 71 and the left imaging device 72 perform stereo matching between pixels having the same transmission characteristics, and take correspondence between the left and right imaging devices. Thereby, the correspondence between the green pixel (G1) and the green pixel (G2) can be obtained, and the correspondence between the red pixel (R) and the blue pixel (B) can be obtained as described above. As described above, the red pixel (R) and the blue pixel (B) are the pseudo red pixel (R ′) and the red pixel (R) calculated pseudo, or the pseudo blue pixel (B ′) and the blue pixel ( In B), it is possible to increase the color information by taking the difference or the like, but similarly, it is possible to increase the color information in the green pixel (G1) and the green pixel (G2).

  In the above example, two transparent pixels are used. However, one transparent pixel is used, and the right image sensor and the left image sensor have the same transmission characteristics, the red pixels have different transmission characteristics, and the different. A configuration using blue pixels having transmission characteristics may also be used. That is, if at least two pixels having the same transmission characteristics are used for the right image sensor and the left image sensor, stereo matching can be performed accurately, and the remaining two pixels can perform color information by matching based on the matching information. It is possible to obtain. If there are three pixels having the same transmission characteristics in the right image sensor and the left image sensor, the accuracy of stereo matching is further improved.

  In the above description, the transparent pixel (W) is a transparent pixel that transmits in all visible wavelength regions. In order to improve the light utilization efficiency, it is desirable that the transmittance is 100%. However, for example, when an infrared absorption filter is provided to absorb the infrared wavelength region, the visible wavelength region may be absorbed due to the influence. is there. However, if the right imaging element 11 and the left imaging element 12 have the same transmission characteristics, stereo matching is possible, and the transmittance is much higher than that of the red pixel, the green pixel, and the blue pixel. A highly sensitive imaging device can be realized.

  In addition, the arrangement of the pixels shown in FIGS. 1 and 7 is not limited to this, but the transparent pixels having the same transmission characteristics in the image sensor are arranged diagonally to each other as shown in FIGS. It is desirable. This is because the position information can be calculated with higher accuracy when the position information of the subject is processed by the image processing, when the position is diagonal rather than the vertical or horizontal arrangement. In the above description, the example in which the imaging devices are arranged on the left and right has been described. However, it is only necessary that the amount of parallax can be calculated based on the parallax of the imaging device. Any arrangement is possible as long as the same subject can be photographed.

  In the above description, the image pickup device of the image pickup apparatus has been described as an example of a configuration in which four pixels are set as one set, two transparent pixels, one green pixel, and one red pixel or blue pixel. However, an image pickup apparatus in which three pixels are combined may be used. In this case, there is a configuration in which there is one transparent pixel, one green pixel, and one red pixel or blue pixel. In this case, the transparent pixel and the green pixel become pixels having the same transmission characteristics in the two imaging devices, and stereo matching is possible. Then, the pseudo red and the pseudo blue are calculated with the red pixel and the blue pixel, and compared with the red pixel and the pseudo red pixel as described above, and with the blue pixel and the pseudo blue pixel. As a result, a large amount of color information can be acquired, and an accurate image of the color can be acquired even though the sensitivity is high.

  In addition, the same effect can be obtained with the configuration of one transparent pixel, one red pixel, one green pixel or blue pixel, and one transparent pixel, one blue pixel, red pixel or green pixel. One configuration may be used. Stereo matching with two imaging devices using these three pixels as a set can be compared with conventional matching with four pixels as a set, and the same effect can be obtained with fewer pixels. If the pixels have the same size, the number of pixels increases, so a higher-resolution image can be obtained. If the images have the same resolution with the same image sensor, it is possible to arrange large pixels. Therefore, an image pickup device with higher sensitivity can be obtained, and an image having the same pixel size and the same resolution can be realized with a smaller image pickup device because the number of pixels is reduced, resulting in a cost merit.

  In this embodiment, the image (left image) from the viewpoint of the image sensor 12 is generated. However, the image (right image) from the viewpoint of the image sensor 11 may be generated. In that case, the correlation detection unit 4 detects the shift amount of the left image with respect to the right image as the parallax amount. Further, the parallax correction unit 5 corrects the parallax for the signal Bl of the blue pixel (B) from the image sensor 12 and the signal R′l of the pseudo red pixel (R ′) from the red signal generation unit 8. The combining unit 6 also includes a red pixel (R) signal Rr, a green pixel (G) signal Gr, and a transparent pixel (W) signal Wr from the image sensor 11, and the blue that the parallax correction unit 5 has corrected for parallax. A color image is generated from the signal of the pixel (B).

  As described above, by using a transparent pixel, it becomes possible to photograph even a dark subject, so that stereo matching of the left and right imaging devices can be performed even in a dark part. In addition, since a dark part is acquired by a transparent pixel and a bright part is acquired by normal RGB, it is possible to capture a low-luminance part that is difficult to acquire due to noise by a normal RGB filter. In fact, since it becomes impossible to recognize the color in a dark place, there is little problem even if the color information is lost in the dark part.

  1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute image acquisition processing. You may go. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Further, the “computer-readable recording medium” refers to a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

  The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  The distance information to the subject can be acquired by a plurality of imaging devices, and the present invention can be applied to an application in which it is indispensable to combine images obtained by the plurality of imaging devices.

  DESCRIPTION OF SYMBOLS 11 ... Right image sensor, 12 ... Left image sensor, 21 ... Right color filter, 22 ... Left color filter, 31, 32 ... Lens, 4 ... Correlation detection part, 5 ... Parallax correction part, 6 ... Composition part

Claims (5)

In order to acquire a color image and to acquire distance information to the subject, two image pickup devices that photoelectrically convert incident light and output an image signal, and a distance to the subject based on the two image signals An imaging apparatus comprising: a distance information acquisition unit for obtaining information; and a combining unit that combines the image signals of the two systems to obtain the color image
Of the image sensors, the first image sensor has a pixel configuration in which a first primary color pixel and a second primary color pixel are combined as a pair of transparent pixels and the three primary colors of light, and the second image sensor is transparent. An image pickup apparatus having a pixel configuration in which a third primary color pixel and one of the first or second primary color pixels among the three primary colors of pixels and light are combined.   The imaging apparatus according to claim 1, wherein the pair of pixel configurations of the first and second imaging elements includes two transparent pixels. The distance information acquisition unit calculates a parallax amount between the two systems by matching at least transparent pixel signals of the two systems of image signals, and based on the parallax amount, the distance to the subject The imaging apparatus according to claim 1, wherein information is obtained.   The imaging device according to claim 1, wherein the synthesizing unit replaces a low-illuminance pixel with a transparent pixel signal of any one of the two systems of image signals. Primary color signal generation for generating a signal of the remaining one primary color pixel among the three primary colors based on the signal of the transparent pixel of the first or second image sensor and the signal of the two primary color pixels of the image sensor The imaging apparatus according to claim 1, further comprising: a unit.

Images