JP5406151B2 - 3D imaging device - Google Patents

Description

  The present invention relates to a monocular three-dimensional imaging technique for generating a plurality of images having parallax.

  In recent years, there has been a remarkable increase in functionality and performance of digital cameras and digital movies using a solid-state image sensor such as a CCD or CMOS (hereinafter sometimes referred to as “image sensor”). In particular, due to advances in semiconductor manufacturing technology, the pixel structure in a solid-state image sensor has been miniaturized. As a result, higher integration of pixels and drive circuits of solid-state image sensors has been attempted. For this reason, in a few years, the number of pixels of the image sensor has increased significantly from about 1 million pixels to over 10 million pixels. Furthermore, the quality of the image obtained by imaging has improved dramatically. On the other hand, with respect to the display device, a thin liquid crystal display or a plasma display enables high-resolution and high-contrast display without taking up space, and high performance is realized. Such a flow of improving the quality of video is spreading from a two-dimensional image to a three-dimensional image. In recent years, polarized glasses are required, but high-quality three-dimensional display devices are being developed.

  As a representative technique having a simple configuration with respect to the three-dimensional imaging technique, there is a technique of acquiring a right-eye image and a left-eye image using an imaging system including two cameras. In such a so-called two-lens imaging method, since two cameras are used, the imaging apparatus becomes large and the cost can be high. Therefore, a method of acquiring a plurality of images having parallax using one camera has been studied. For example, Patent Document 1 discloses a technique for simultaneously acquiring two images having parallax using a color filter. FIG. 8 is a diagram schematically showing an imaging system using this technique. The imaging system in this technique includes a lens 3, a lens diaphragm 19, a light flux limiting plate 20 on which two color filters 20a and 20b having different transmission wavelength ranges are arranged, and a photosensitive film 21. Here, the color filters 20a and 20b are filters that transmit, for example, red and blue light, respectively.

  With the above configuration, the incident light passes through the lens 3, the lens diaphragm 19, and the light flux limiting plate 20, and forms an image on the photosensitive film. At this time, only the red and blue light beams are transmitted through the two color filters 20a and 20b in the light flux limiting plate 20, respectively. As a result, a magenta color image is formed on the photosensitive film by the light transmitted through the two color filters. Here, since the positions of the color filters 20a and 20b are different, parallax occurs in the image formed on the photosensitive film. Here, when a photograph is made from a photosensitive film and glasses with red and blue films attached for the right eye and the left eye, respectively, an image with a sense of depth can be seen. Thus, according to the technique disclosed in Patent Document 1, an image having parallax can be created using two color filters.

  The technique disclosed in Patent Document 1 forms an image on a photosensitive film and creates a plurality of images having parallax. On the other hand, a technique for acquiring an image having parallax by converting it into an electrical signal is patented. It is disclosed in Document 2. FIG. 9 is a diagram schematically showing a light flux limiting plate in this technique. In this technique, a light flux limiting plate in which an R region 22R that transmits red light, a G region 22G that transmits green light, and a B region 22B that transmits blue light are provided on a plane perpendicular to the optical axis of the imaging optical system. 22 is used. The light transmitted through these areas is received by a color imaging device having R pixels for red, G pixels for green, and B pixels for blue, whereby an image of light transmitted through each area is acquired.

  Patent Document 3 also discloses a technique for acquiring a plurality of images having parallax using a configuration similar to the configuration of FIG. FIG. 10 is a diagram schematically showing the light flux limiting plate disclosed in Patent Document 3. As shown in FIG. Also with this technique, an image with parallax can be created by transmitting incident light through the R region 23R, the G region 23G, and the B region 23B provided on the light flux limiting plate 23.

  According to the techniques disclosed in Patent Documents 1 to 3, an image having parallax can be generated by disposing R, G, and B color filters on the light flux limiting plate. However, since the light flux limiting plate is used, the amount of incident light is reduced. Further, in order to enhance the parallax effect, it is necessary to dispose the R, G, and B color filters at positions separated from each other to reduce their area. However, the incident light quantity is further reduced.

  In contrast to the above technique, Patent Document 4 discloses a technique capable of obtaining a plurality of images having parallax and a normal image having no problem in light quantity using a diaphragm in which R, G, and B color filters are arranged. Is disclosed. In this technique, only the light transmitted through the R, G, and B color filters is received when the diaphragm is closed, and the RGB color filter area is removed from the optical path when the diaphragm is opened. Can receive. As a result, an image having parallax can be acquired when the aperture is closed, and a normal image with a high light utilization rate can be acquired when the aperture is open.

Japanese Patent Laid-Open No. 2-171737 JP 2002-344999 A JP 2009-276294 A JP 2003-134533 A

"Color Lines: Image Specific Color Representation", Ido Omer and Michael Werman, In Proc. CVPR, vol. 2, 946-953.

  According to the prior art, a plurality of images having parallax can be acquired. However, since a color filter is used for the light flux limiting plate, the amount of light received by the image sensor is reduced. In order to ensure a sufficient amount of incident light, a method of obtaining a normal image with a high light utilization rate using a mechanism that removes the color filter from the optical path by mechanical drive, and the number of color filters provided on the light flux limiting plate are minimized. There is a method of suppressing the decrease in light by using two. The former has a problem that the apparatus is increased in size and cost. In the latter case, since the wavelength ranges of the light transmitted through the two color filters are different, the two images obtained have different shades. Therefore, there is a problem that it is difficult to calculate a parallax based on a corresponding point search using a similarity value of gray values such as block matching.

  The present invention provides a three-dimensional imaging technique capable of generating a plurality of images having high light utilization and parallax without performing mechanical driving.

  The three-dimensional imaging device of the present invention includes a light transmission unit having first, second, and third transmission regions having different transmission wavelength ranges, and a photosensitive cell array, and transmits light transmitted through the light transmission unit. A solid-state imaging device arranged to receive the optical system; an optical system that forms an image on an imaging surface of the solid-state imaging device; and a signal processing unit that processes a signal output from the solid-state imaging device. At least one of the first, second, and third transmission regions is formed of a member that transmits light in a wavelength range of cyan, yellow, or magenta, or a transparent member. The photosensitive cell array has a plurality of unit blocks, and each unit block outputs an R detection cell that outputs a first photoelectric conversion signal corresponding to the amount of light in the red wavelength range, and the light in the green wavelength range. A G detection cell that outputs a second photoelectric conversion signal according to the amount and a B detection cell that outputs a third photoelectric conversion signal according to the amount of light in the blue wavelength range are included. The signal processing unit is configured to transmit light incident on each of the first, second, and third transmission regions based on processing including addition / subtraction using the first, second, and third photoelectric conversion signals. An image generation unit that generates three images having parallax by generating three color mixture signals according to the amount, and a parallax estimation unit that estimates parallax between the three images.

  In one embodiment, the signal processing unit further includes a distance information generation unit that generates information indicating the distance of the subject from the parallax estimated by the parallax estimation unit.

  In one embodiment, the parallax estimation unit sets an estimated value of the amount of parallax from among a plurality of candidates for each pixel of the three images, and the position on the image from the three images based on the estimated value. A pixel block extraction unit that extracts three pixel blocks of the same size that are shifted from each other, and the degree of distribution of a point set in a three-dimensional color space defined by a set of pixel values of the three pixel blocks from a straight line A deviation determination unit that determines whether or not there is a deviation; and a parallax amount determination unit that determines the estimated value that minimizes the degree of deviation from the straight line determined by the deviation determination unit as a parallax amount in each pixel.

  In one embodiment, the first transmission region is formed of a member that transmits light in one wavelength region of cyan, yellow, and magenta, and the second transmission region is cyan, yellow, and magenta. The third transmission region is formed of a transparent member. The third transmission region is formed of a transparent member.

  In one embodiment, the first, second, and third transmission regions are formed of members that transmit light in the wavelength bands of cyan, yellow, and magenta, respectively.

  In one embodiment, the light transmission part has a fourth transmission region, and the fourth transmission region is a member that transmits light in a wavelength region of red, green, and blue, or a transparent member Formed with.

  According to the signal processing method of the present invention, a light transmission unit having first, second, and third transmission regions having different transmission wavelength ranges and a light having a photosensitive cell array and receiving light transmitted through the light transmission unit. Is a method for processing a signal output from an image pickup apparatus including a solid-state image pickup element disposed on the image pickup apparatus and an optical system that forms an image on an image pickup surface of the solid-state image pickup element. Here, at least one of the first, second, and third transmission regions is formed of a member that transmits light in a wavelength band of cyan, yellow, or magenta, or a transparent member. The photosensitive cell array has a plurality of unit blocks, and each unit block outputs an R detection cell that outputs a first photoelectric conversion signal corresponding to the amount of light in the red wavelength range, and the light in the green wavelength range. A G detection cell that outputs a second photoelectric conversion signal according to the amount, and a B detection cell that outputs a third photoelectric conversion signal according to the amount of light in the blue wavelength region. The signal processing method of the present invention is incident on each of the first, second, and third transmission regions based on processing including addition / subtraction using the first, second, and third photoelectric conversion signals. Generating three images having parallax by generating three color mixture signals corresponding to the amount of light; and estimating a parallax between the three images.

  The signal processing method of the present invention may further include a step of generating information indicating the distance of the subject from the estimated parallax.

  In one embodiment, the step of estimating the parallax sets an estimated value of the amount of parallax from among a plurality of candidates for each pixel of the three images, and based on the estimated value, The steps of extracting three pixel blocks of the same size whose positions are shifted from each other, and how much the point set distribution in the three-dimensional color space defined by the set of pixel values of the three pixel blocks deviates from the straight line. And determining the estimated value that minimizes the degree of deviation from the determined straight line as the amount of parallax for each pixel.

  According to the present invention, it is possible to generate a plurality of images with high light utilization and parallax. Furthermore, since the amount of parallax can be estimated even when a chromatic color subject is photographed, both estimation of the amount of parallax and acquisition of an image with a high light utilization rate are possible.

1 is a block diagram illustrating an overall configuration of an imaging apparatus according to Embodiment 1. FIG. 2 is a schematic diagram illustrating a schematic configuration of a light-transmitting plate, an optical system, and an image sensor in Embodiment 1. FIG. FIG. 3 is a diagram illustrating an arrangement of transmission regions of a light transmission plate in the first embodiment. 2 is a diagram illustrating a pixel configuration of an image sensor in Embodiment 1. FIG. It is a figure which shows the relationship between the parallax amount d in Embodiment 1, and the distance z. 4 is a diagram illustrating an example of a color image, a Ci1 image, and a Ci2 image in Embodiment 1. FIG. (A) is a figure which shows the example of the straight line in R, G, B color space, (b) is a figure which shows the example of the straight line in Cy, Ye, and Mg color space. FIG. 3 is a flowchart showing a signal processing procedure in the first embodiment. 2 is a diagram illustrating a pixel block according to Embodiment 1. FIG. (A) is a figure which shows the example of distribution of a point set in a three-dimensional color space when a parallax estimated value is comparatively close to a true parallax amount, (b) It is a figure which shows the example of distribution of the point set in the three-dimensional color space in the case of being. 3 is a block diagram illustrating a configuration of a parallax estimation unit according to Embodiment 1. FIG. FIG. 5 is a flowchart showing a parallax amount estimation procedure in the first embodiment. FIG. 6 is a diagram illustrating an arrangement of transmission regions of a light transmission plate in Embodiment 2. It is a figure which shows arrangement | positioning of the permeation | transmission area | region of the translucent board in Embodiment 3. FIG. It is a figure which shows arrangement | positioning of the other permeation | transmission area | region of the light transmission board in Embodiment 3. FIG. FIG. 11 is a schematic diagram of an imaging system in Patent Document 1. It is a schematic diagram of the light beam restricting plate in Patent Document 2. It is a schematic diagram of the light beam restricting plate in Patent Document 3.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, common elements are denoted by the same reference numerals. In the present specification, a signal or information indicating an image may be simply referred to as “image”.

(Embodiment 1)
FIG. 1 is a block diagram showing an overall configuration of a three-dimensional imaging apparatus (hereinafter referred to as “imaging apparatus”) according to the first embodiment of the present invention. The imaging apparatus according to the present embodiment is a digital electronic camera, and includes an imaging unit 100 and a signal processing unit 200 that generates a signal (image signal) indicating an image based on a signal from the imaging unit 100. . The imaging apparatus according to the present embodiment may have a function of shooting a moving image as well as a still image.

  The imaging unit 100 includes an imaging element (image sensor) 1 including a plurality of photosensitive cells arranged on an imaging surface, and a light-transmitting plate (transmission plate) that has three transmission regions having different transmission wavelength ranges and transmits incident light. A light transmitting portion) 2, an optical lens 3 for forming an image on the imaging surface of the imaging device 1, and an infrared cut filter 4. The imaging unit 100 also generates a basic signal for driving the imaging device 1, receives an output signal from the imaging device 1, and sends it to the signal processing unit 200. And an element driving unit 6 that drives the image sensor 1 based on the basic signal generated by the receiving unit 5. The image sensor 1 is typically a CCD or CMOS sensor, and is manufactured by a known semiconductor manufacturing technique. The signal generation / reception unit 5 and the element driving unit 30 are composed of an LSI such as a CCD driver, for example.

  The signal processing unit 200 processes the signal output from the imaging unit 100 to generate an image signal, the memory 30 that stores various data used for generating the image signal, and the generated image signal And an interface (IF) unit 8 for sending the data to the outside. The signal processing unit 200 also includes a parallax estimation unit 40 that estimates parallax between a plurality of images generated by the image generation unit 7, and a distance information generation unit 50 that generates distance information of the subject based on the estimated parallax. It has. The image generation unit 7, the parallax estimation unit 40, and the distance information generation unit 50 are images including hardware such as a known digital signal processor (DSP) and image signal generation processing, parallax estimation processing, and distance information generation processing described later. It can be suitably realized by a combination with software that executes processing. The memory 30 is configured by a DRAM or the like. The memory 30 records the signal obtained from the imaging unit 100 and temporarily records the image data generated by the image generation unit 7 and the compressed image data. These image data are sent to a recording medium (not shown) or a display unit via the interface unit 8.

  Note that the imaging apparatus of the present embodiment may include known components such as an electronic shutter, a viewfinder, a power source (battery), and a flashlight, but a description thereof is omitted because it is not particularly necessary for understanding the present invention.

  Next, the configuration of the imaging unit 100 will be described in more detail with reference to FIGS.

  FIG. 2 is a diagram schematically illustrating an arrangement relationship of the light transmitting plate 2, the lens 3, and the imaging element 1 in the imaging unit 100. In FIG. 2, components other than the translucent plate 2, the lens 3, and the image sensor 1 are omitted. The translucent plate 2 has three transmission regions C1, C2, and C3 having different transmission wavelength ranges, and transmits incident light. The lens 3 is a known lens, collects the light transmitted through the translucent plate 2, and forms an image on the imaging surface 1 a of the imaging device 1. In the following description, on the plane parallel to the imaging surface 1a, the direction from the region C1 to the region C2 is the x direction, and the direction perpendicular to the x direction is the y direction.

  The arrangement relationship of each component shown in FIG. 2 is merely an example, and the present invention is not limited to such an arrangement relationship. For example, the lens 3 may be arranged farther from the imaging element 1 than the translucent plate 2 as long as an image can be formed on the imaging surface 1a. The lens 3 is a lens unit composed of a plurality of lens groups, and the translucent plate 2 may be disposed between them. Further, the lens 3 and the translucent plate 2 do not need to be independent components, and a person may be configured as one integrated optical element. Furthermore, the translucent plate 2 and the imaging surface of the imaging device 1 do not necessarily have to be arranged in parallel. For example, by arranging an optical element that reflects light, such as a mirror or a prism, between the two, the translucent plate 2 and the imaging surface of the imaging element 1 are positioned on a plane intersecting each other. Is also possible.

  FIG. 3 is a front view of the translucent plate 2 in the present embodiment. The shape of the translucent plate 2 in the present embodiment is circular as in the case of the lens 3, but may be other shapes. In the region C1, a color filter (Cy filter) that transmits light in the cyan (Cy) wavelength region (G and B wavelength regions) is disposed. A color filter (Ye filter) that transmits light in the yellow (Ye) wavelength region (R and G wavelength regions) is disposed in the region C2.

  Note that the regions C1 and C2 in the present embodiment are not limited to color filters, as long as they are configured to transmit light in the wavelength regions of Cy and Ye and not transmit light in other wavelength regions, respectively. It may be composed of various members. For example, you may be comprised with optical elements, such as a dichroic mirror which permeate | transmits the light of a one part wavelength range and reflects the light of another wavelength range. Here, the region C1 and the region C2 are arranged apart from each other in the x direction. The distance L between the region C1 and the region C2 is determined according to the size of the lens 3 so that the acquired image has an appropriate parallax. The distance L can be set to several mm to several cm, for example. Further, the other region C3 in the light transmitting plate 2 is a transparent region formed of a transparent member that transmits visible light in the entire wavelength region included in the white light (W). The transparent member may be any member that transmits light with high transmittance. In this embodiment, the areas of the region C1 and the region C2 are equal, and the area of the region C3 is designed to be larger than the areas of the regions C1 and C2.

  On the imaging surface 1a of the imaging device 1 shown in FIG. 2, a photosensitive cell array in which a plurality of photosensitive cells are arranged in a two-dimensional manner and a color filter array arranged to face the photosensitive cell array are formed. . The photosensitive cell array and the color filter array have a plurality of unit blocks, and each unit block includes four photosensitive cells and four color filters facing them. Each photosensitive cell is typically a photodiode, and outputs an electrical signal (hereinafter referred to as “photoelectric conversion signal” or “pixel signal”) corresponding to the amount of received light by photoelectric conversion. Each color filter is manufactured using a known pigment or the like, and is designed to selectively transmit light in a specific wavelength range.

  FIG. 4 is a diagram schematically showing a part of the photosensitive cell array and the color filter array. On the imaging surface 1a, a large number of photosensitive cells 120 and color filters 110 facing them one-to-one are arranged in a matrix. In the present embodiment, four adjacent photosensitive cells 120 constitute one unit block. In each unit block, a color filter (R filter) that transmits light in the red (R) wavelength region is arranged in the first row and the first column. A color filter (G filter) that transmits light in the green (G) wavelength region is arranged in the first row, second column, and second row, first column. A color filter (B filter) that transmits light in the blue wavelength region is arranged in the second row and the second column. Thus, the arrangement of the color filters 110 in the present embodiment is a known Bayer arrangement based on 2 rows and 2 columns. Note that the arrangement of the photosensitive cells 120 and the color filters 110 is not necessarily a Bayer arrangement, and may be any known arrangement.

  With the above configuration, light incident on the imaging device during exposure passes through the light transmitting plate 2, the lens 3, the infrared cut filter 4, and the color filter 110 and enters the light sensing cell 120. Each light sensing cell 120 receives light that has passed through the opposite color filter among the light transmitted through the regions C1, C2, and C3 of the translucent plate 2, and receives the light according to the amount (intensity) of the received light. Output the conversion signal. The photoelectric conversion signal output by each photosensitive cell is sent to the signal processing unit 200 through the signal generation / reception unit 5. The image generation unit 7 in the signal processing unit 200 generates a right-eye image, a left-eye image, and a color image based on the signal transmitted from the imaging unit 100.

  In addition, although the image sensor 1 in this embodiment performs color separation by said color filter array, you may use the image sensor which does not use a color filter array in this invention. For example, an imaging device having a triple well structure disclosed in JP-T-2002-513145 can be used. Thus, each unit block includes an R detection cell that detects light in the red wavelength range, a G detection cell that detects light in the green wavelength range, and a B detection cell that detects light in the blue wavelength range. Any imaging device can be used.

  In the present embodiment, one photosensitive cell detects light in a wavelength range of red, green, or blue, but the wavelength range detected by one photosensitive cell may be further subdivided. For example, the wavelength range λr of red light may be divided into three wavelength ranges λr1, λr2, and λr3, and three photosensitive cells each corresponding to λr1, λr2, and λr3 may be provided. In this case, the sum of the pixel signals of the three photosensitive cells can be processed as a signal corresponding to red light.

  Hereinafter, a photoelectric conversion signal output from each photosensitive cell will be described. First, signals corresponding to the intensities of light incident on the regions C1, C2, and C3 of the translucent plate 2 are denoted by Ci1, Ci2, and Ci3 with the suffix “i”, respectively. In addition, the spectral transmittance of the transparent region C3, the lens 3, and the infrared cut filter 4 in the light transmitting plate 2 is Tw, the spectral transmittance of the Cy filter is Tcy, and the spectral transmittance of the Ye filter is Tye. Similarly, the spectral transmittances of the R, G, and B color filters are expressed as Tr, Tg, and Tb, respectively. Here, Tw, Tcy, Tye, Tr, Tg, and Tb are functions that depend on the wavelength λ of incident light. Signals indicating the intensity of light transmitted through the R, G, and B color filters and received by the opposing photosensitive cells are denoted by Rs, Gs, and Bs with a suffix “s”, respectively. Further, the integral calculation of the spectral transmittance in the wavelength range of visible light is represented by the symbol Σ. For example, the integral operation ∫TwTcyTrdλ for the wavelength λ is represented as ΣTwTcyTr. Here, it is assumed that the integration is performed over the entire wavelength range of visible light. Then, Rs is proportional to the result of adding Ci1ΣTwTcyTr, Ci2ΣTwTyeTr, and Ci3ΣTwTr. Gs is proportional to the sum of Ci1ΣTwTcyTg, Ci2ΣTwTyeTg, and Ci3ΣTwTg. Bs is proportional to the sum of Ci1ΣTwTcyTb, Ci2ΣTwTyeTb, and Ci3ΣTwTb. If the proportionality coefficient in these relationships is 1, Rs, Gs, and Bs can be expressed by the following formulas 1 to 3.

(Formula 1) Rs = Ci1ΣTwTcyTr + Ci2ΣTwTyeTr + Ci3ΣTwTr
(Formula 2) Gs = Ci1ΣTwTcyTg + Ci2ΣTwTyeTg + Ci3ΣTwTg
(Expression 3) Bs = Ci1ΣTwTcyTb + Ci2ΣTwTyeTb + Ci3ΣTwTb
In Equations 1 to 3, ΣTwTcyTr, ΣTwTyTr, and ΣTwTr are represented by Mx11, Mx12, and Mx13, respectively, ΣTwTcyTg, ΣTwTyeTg, and ΣTwTg are represented by Mx21, Mx22, and Mx23, and ΣTwTcyTb, ΣTwTxTb, It will be expressed as Mx33. Then, the relationship between Rs, Gs, and Bs and Ci1, Ci2, and Ci3 can be expressed by the following Expression 4 using a matrix.

ここで、式４における要素Ｍｘ１１〜Ｍｘ３３からなる行列の逆行列の要素を、それぞれｉＭ１１〜ｉＭ３３とすると、式４は次の式５に変形できる。すなわち、領域Ｃ１、Ｃ２、Ｃ３に入射する光の強度を示す信号を、光電変換信号Ｒｓ、Ｇｓ、Ｂｓを用いて表すことができる。

Here, if the elements of the inverse matrix of the matrix composed of the elements Mx11 to Mx33 in Expression 4 are iM11 to iM33, respectively, Expression 4 can be transformed into the following Expression 5. That is, a signal indicating the intensity of light incident on the regions C1, C2, and C3 can be expressed using the photoelectric conversion signals Rs, Gs, and Bs.

画像生成部７（図１）は、式５に基づく信号演算を実行し、信号Ｃｉ１、Ｃｉ２、Ｃｉ３を、単位ブロックごとに生成する。このようにして単位ブロックごとに生成された信号Ｃｉ１、Ｃｉ２、Ｃｉ３は、領域Ｃ１、Ｃ２、Ｃ３のそれぞれに入射する光によって形成される３つの画像を表す。特に、信号Ｃｉ１、Ｃｉ２によって表される画像は、ｘ方向に離れて位置する領域Ｃ１、Ｃ２からそれぞれ被写体を見たときの画像に相当するため、左目用画像および右目用画像として扱うことができる。すなわち、信号Ｃｉ１、Ｃｉ２によって表される２つの画像は、領域Ｃ１、Ｃ２の距離に応じた視差を有する。したがって、これらの画像から被写体の奥行きを示す情報を得ることができる。

The image generation unit 7 (FIG. 1) performs signal calculation based on Expression 5, and generates signals Ci1, Ci2, and Ci3 for each unit block. The signals Ci1, Ci2, and Ci3 generated for each unit block in this way represent three images formed by light incident on each of the regions C1, C2, and C3. In particular, since the images represented by the signals Ci1 and Ci2 correspond to images when the subject is viewed from the regions C1 and C2 that are located apart in the x direction, they can be handled as a left-eye image and a right-eye image. . That is, the two images represented by the signals Ci1 and Ci2 have parallax according to the distance between the regions C1 and C2. Therefore, information indicating the depth of the subject can be obtained from these images.

  Image signals Ci1, Ci2, and Ci3 obtained by the above processing are expressed using photoelectric conversion signals Rs, Gs, and Bs, but these correspond to grayscale images (monochrome images), not color images. In order to obtain a color image instead of a grayscale image, the above signal calculation processing is not performed, and color processing in a normal Bayer array may be performed from each obtained photoelectric conversion signal. At this time, the loss of incident light and the color temperature shift may occur due to the Cy filter and Ye filter disposed on the light-transmitting plate 2. However, since the light transmittance of these color filters is high, the loss of the incident light is reduced. It can be smaller than the case. Further, even if an overall color shift occurs, it can be dealt with by adjusting the white balance. Thus, according to the imaging apparatus of the present embodiment, a good color image with a high light utilization rate can be obtained.

  When obtaining a color image, color information may be obtained by using only the term Ci3 in Equation 4 instead of performing color processing in a normal Bayer array from each photoelectric conversion signal. That is, after obtaining Ci3 based on Equation 5, a color image can also be obtained by setting Mx13 × Ci3 as the R light amount, Mx23 × Ci3 as the G light amount, and Mx33 × Ci3 as the B light amount.

  It is conceivable that a parallax signal is generated using the image signals Ci1 and Ci2 obtained here, and distance information is calculated. If the distance information can be calculated, various applications such as generation of a parallax signal in a color image, generation of an image of an arbitrary viewpoint, and clipping of the foreground and background become possible.

  In general, in order to calculate distance information, it is necessary to obtain a correspondence between each pixel of the image signal Ci1 and each pixel of the image signal Ci2. Obtaining the correspondence of the pixel means searching for which pixel in the image signal Ci2 the point p in the three-dimensional object of the subject that appears in the specific pixel in the image signal Ci1. It is assumed that the subject shown in the pixel represented by the coordinates (x1, y1) in the image signal Ci1 is shown in the pixel represented by the coordinates (x2, y2) in the image signal Ci2. At this time, if the distance between the corresponding pixels (the distance between the coordinates (x1, y1) and the coordinates (x2, y2), such as the Euclidean distance or the city area distance) is obtained, the distance from the camera to the subject is calculated. be able to. The distance between the corresponding pixels is referred to as “parallax amount” in the present embodiment.

Hereinafter, the principle of distance measurement from the camera to the subject will be described with reference to FIG. FIG. 5 is a schematic diagram showing a positional relationship among the light transmitting plate 2, the subject 9, and the image sensor 1. In FIG. 5, it is assumed that the center of the lens 3 (not shown) is located at the center of the translucent plate 2. In FIG. 5, the focal length of the lens 3 is f, the distance from the translucent plate 2 to the subject 9 is z, the amount of parallax generated when the camera captures the subject 9 is d, and the distance between the centers of the regions C1 and C2 is shown. Let L be. At this time, the distance z is obtained by the following equation 6 from the principle of general binocular stereo.
(Formula 6) z = fL / d
Since the distance L between the focal distance f and the areas C1 and C2 is obtained in advance, if the parallax amount d is known, the distance z is obtained. Thereby, distance information to the subject can be obtained.

  As an existing method for obtaining the parallax amount between the feature point of the image signal Ci1 and the feature point of the image signal Ci2 corresponding to the feature point, there is a method of obtaining the coordinates of the corresponding point based on the similarity of the gray value. . For example, there is a method of dividing an image into a plurality of small blocks and calculating SAD (absolute error) or SSD (square error) between the blocks. Since the error calculated from the similar blocks becomes small, the corresponding point between the signal Ci1 and the signal Ci2 can be obtained by determining the combination of the blocks that minimizes the error. Thereby, the parallax amount d can be calculated.

  However, since the image signals Ci1 and Ci2 generally have different gray values at corresponding points, an error occurs when the parallax amount d is obtained by the above method. The image signals Ci1, Ci2, and Ci3 are signals that indicate the intensities of light incident on the transmission regions C1, C2, and C3 from the subject, respectively. Therefore, if the intensity of the light transmitted through the transmission areas C1 and C2 is obtained, the intensity of cyan light (green light and blue light) and yellow light (red light and green light) incident on each area can be obtained. Signals indicating the intensity of light transmitted through the transmission regions C1 and C2 are represented by Ci1ΣTwTcy and Ci2ΣTwTye, respectively. However, since the transmission regions C1 and C2 have different transmission wavelength ranges, when the subject has a chromatic color, the gray values of corresponding points of the signals Ci1ΣTwTcy and Ci2ΣTwTye having parallax are also different from each other.

  6A, 6B, and 6C are diagrams illustrating a captured image (color image), an image represented by Ci1ΣTwTcy, and an image represented by Ci2ΣTwTye, respectively. The background of these images is a color chart (color comparison table). The arrows in FIG. 6A indicate the colors of some sections of the color chart. For example, when the subject is blue, the gray value of the signal Ci1ΣTwTcy indicating the intensity of light transmitted through the Cy filter C1 is larger than the gray value of the signal Ci2ΣTwTye indicating the intensity of light transmitted through the Ye filter C2. Accordingly, the blue section in FIG. 6B is brighter than the blue section in FIG. Conversely, the yellow section in FIG. 6B is darker than the yellow section in FIG. As described above, when the subject is a chromatic color, the gradation value is different between Ci1ΣTwTcy and Ci2ΣTwTye. For this reason, the existing matching method based on the similarity of the gray value cannot be used.

  Patent Document 3 discloses a method of disposing an R, G, B color filter on a diaphragm and obtaining a parallax amount using three images having parallax generated by light transmitted through each color filter. . In this method, in a normal natural image in which no parallax occurs between the R, G, and B components, the three-dimensional distribution of R, G, and B values is locally (R, G, B). The property of being linear in space (three-dimensional color space) is used. That is, when there is no parallax between the R image, G image, and B image, the distribution of pixel values in the three-dimensional space (R, G, B) is linear, and when parallax occurs, The distribution is not linear. Accordingly, when the parallax amount is assumed to be d and the value of d is varied within a certain range, the value of d that minimizes the amount indicating how much the distribution of (R, G, B) is deviated from the straight line. Can be estimated. As described above, a model in which the distribution in the three-dimensional color space is locally linear in a natural image in which no parallax is generated by the color component is referred to as a “color line model”. Details of the linear color model are disclosed in Non-Patent Document 1, for example.

  The imaging apparatus according to the present embodiment obtains parallax between images using a linear color model, as in the method of Patent Document 3 described above. However, in this embodiment, unlike Patent Document 3, each pixel signal Rs, Gs, Bs does not directly correspond to a specific transmission region, but is incident from a plurality of transmission regions C1, C2, C3. Signal components generated by light are superimposed. Therefore, it is necessary to generate three signals corresponding to the intensity of light incident on C1, C2, and C3 and representing different color components.

  In the present embodiment, two types of complementary color filters, Cy and Ye, are used as color filters in the transmission region of the translucent plate 2. Therefore, first, when there is no parallax between the color components, the complementary colors (Cy, magenta (Mg), Ye) on the three-dimensional space are also used in the complementary color system as in the existing method using R, G, B. This indicates that the distribution of pixel values (Cy, Mg, Ye) of each image is linear.

  In order for a set of pixel values having a linear distribution in the primary color system to have a linear distribution in the complementary color system, the primary color and the complementary color need only have a linear relationship. Here, the relationship between the primary colors R, G, and B and the complementary colors Cy, Mg, and Ye is expressed by the following Expression 7.

(Formula 7) (R, G, B) = (bit-Cy, bit-Mg, bit-Ye)
The constant bit is the maximum value of the signal of one pixel, and is, for example, 255 for an 8-bit image. From Equation 7, it can be seen that the linear color can be linearly converted between the complementary color and the primary color, so that the local linearity relationship is also established in the three-dimensional space of the complementary color. FIG. 7 shows an example in which the straight line in the primary color system becomes a straight line even in the complementary color system. FIG. 7A shows straight lines in the three-dimensional color space of R, G, and B, and FIG. 7B shows corresponding straight lines in the three-dimensional color space of Cy, Mg, and Ye. Thus, parallax estimation using a linear color model is also effective in a complementary color system.

  From the above, the amount of parallax can be estimated from three images having complementary color parallax. Hereinafter, a procedure for generating three image signals of Cy, Mg, and Ye having parallax from RGB pixel signals obtained by imaging using the light-transmitting plate 2 including the Cy region, the Ye region, and the transparent region will be described. . Next, a procedure for estimating the amount of parallax between these images and calculating subject distance information will be described.

  FIG. 8 is a flowchart showing a schematic procedure of processing in the signal processing unit 200. When shooting is completed, the image generation unit 7 generates three complementary color images (parallax complementary color images) having parallax from the pixel signals Rs, Gs, and Bs obtained by shooting (S100). Next, the parallax estimation unit 40 estimates the amount of parallax between the three parallax complementary color images using the color linearity in the three-dimensional color space (S200). Finally, the distance information generation unit 50 calculates the distance information of the subject shown in each pixel based on the estimated amount of parallax, using Equation 6 (S300).

  Details of each process will be described below.

  First, a procedure for generating three complementary color image signals Cs, Ys, and Ms having parallax from pixel signals Rs, Gs, and Bs obtained by imaging using the translucent plate 2 having a Cy region, a Ye region, and a transparent region. explain. The image generation unit 7 obtains signals Rt, Gt, and Bt represented by the following Expression 8, Expression 9, and Expression 10, respectively, by dividing Expression 1, Expression 2, and Expression 3 by Tw.

(Expression 8) Rt = Rs / Tw = Ci1ΣTcyTr + Ci2ΣTyeTr + Ci3ΣTr
(Formula 9) Gt = Gs / Tw = Ci1ΣTcyTg + Ci2ΣTyeTg + Ci3ΣTg
(Expression 10) Bt = Bs / Tw = Ci1ΣTcyTb + Ci2ΣTyeTb + Ci3ΣTb
Since Cy has wavelength ranges of G and B, in this embodiment, Gt + Bt−Rt expressed by the following Expression 11 is defined as an image signal Cs of a Cy component.

(Formula 11) Cs = Gt + Bt−Rt
= Ci1 (ΣTcyTg + ΣTcyTb−ΣTcyTr)
+ Ci2 (ΣTyeTg + ΣTyeTb−ΣTyeTr)
+ Ci3 (ΣTg + ΣTb−ΣTr)
Similarly, signals represented by the following expressions 12 and 13 are defined as image signals Ms and Ys of Mg and Ye components, respectively.

(Formula 12) Ms = Rt + Bt−Gt
= Ci1 (ΣTcyTr + ΣTcyTb−ΣTcyTg)
+ Ci2 (ΣTyeTr + ΣTyeTb−ΣTyeTg)
+ Ci3 (ΣTr + ΣTb−ΣTg)
(Formula 13) Ys = Rt + Gt−Bt
= Ci1 (ΣTcyTr + ΣTcyTg−ΣTcyTb)
+ Ci2 (ΣTyeTr + ΣTyeTg−ΣTyeTb)
+ Ci3 (ΣTr + ΣTg−ΣTb)
Here, the R wavelength range is hardly included in the transmission wavelength range of the Cy filter, and the B wavelength range is hardly included in the transmission wavelength range of the Ye filter, so that ΣTcyTr≈ΣTyeTb≈0. Further, it is assumed that the transmission wavelength range of the Cy filter includes the G wavelength range and the B wavelength range substantially equal, and the transmission wavelength range of the Ye filter includes the G wavelength range and the R wavelength range substantially equal. Then, ΣTcyTg≈ΣTcyTb and ΣTyeTg≈ΣTyeTr are established. Furthermore, it is assumed that the integrated value of the spectral transmittance of the color filter facing each pixel of the image sensor 1 is the same for all color components. That is, ΣTr≈ΣTg≈ΣTg is established. Based on the above assumptions, the equations 11, 12, and 13 are rewritten as the following equations 14, 15, and 16, respectively.

(Formula 14) Cs = Gt + Bt−Rt
= Ci1 (ΣTcyTg + ΣTcyTb) + Ci3ΣTg
(Formula 15) Ms = Rt + Bt−Gt
= Ci3ΣTr
(Formula 16) Ys = Rt + Gt−Bt
= Ci2 (ΣTyeTr + ΣTyeTg) + Ci3ΣTg
In Expression 14, the term Ci2 is removed from Expression 11. The first term of Equation 14 represents the amount of light that passes through the Cy region and depends only on Ci1. The second term represents the amount of light that passes through the transparent region Ci3 and passes through the G filter. Here, the light in the G wavelength range (G light) is light that passes through all of the Cy region, the Ye region, and the transparent region. Therefore, the second term indicates the amount of G light transmitted through all the transmission regions multiplied by the ratio of the area of the region C3 to the area of the translucent plate 2, and can be considered as an offset related to brightness. . Even when there is an offset, the distribution of the three-dimensional space of the complementary color system only shifts as a whole, and thus does not affect the calculation of linearity.

  Similarly, in Equation 15, the components Ci1 and Ci2 are removed from Equation 12. In Expression 16, the component Ci1 is removed from Expression 13. In this way, the grayscale values of the three complementary color images having parallax are obtained by performing the calculations shown in Expressions 14 to 16 from the RGB pixel signals.

  Next, a method for estimating the amount of parallax will be described.

  First, the parallax amount of the Cy image and Ye image with respect to the Mg image is set to d pixels. Then, the Cy image value (pixel value) is Icy (x−d, y), the Ye image value (pixel value) is Iye (x + d, y), and the Mg image value (pixel value) is Img (x, y). And the fact that they indicate the same point in the real world may be obtained using the linearity of complementary colors in the local region.

  In order to obtain the parallax amount d using the linearity in the local region in a certain pixel (x, y), the following may be performed. The parallax amount d is obtained by determining whether or not the local pixel value distribution is linearly distributed. A set of pixel values in the vicinity region of the pixel (x, y) is defined by the following Expression 17.

(Expression 17) P = {Icy (s−d, t), Img (s, t), Iye (s + d, t) | (s, t) is a neighboring pixel around (x, y)}
FIG. 9 is a diagram illustrating pixel blocks 60a, 60b, and 60c in the vicinity of corresponding points in each of the Cy image, Mg image, and Ye image. Here, the coordinates of corresponding points in the Mg image are (x, y). Since the parallax amount is d pixels, the Cy image has a corresponding point shifted by −d pixels in the horizontal direction compared to the Mg image. Similarly, the Ye image has a corresponding point shifted by + d pixels in the horizontal direction compared to the Mg image. Therefore, the pixel block 60b in the vicinity of the point (x, y) in the Mg image corresponds to the pixel block 60a in the vicinity of the point (x−d, y) in the Cy image, and the point (x + d, y) in the Ye image. This corresponds to the neighboring pixel block 60c.

  A straight line is fitted to the distribution obtained by Expression 17, and the mean square from the fitted straight line is defined as an error Er (x, y, d) from the linear color model. In order to obtain a straight line, it is only necessary to calculate the main axis, which is the direction in which the distribution spreads, from the distribution of the complementary color three-dimensional space. For this purpose, first, a covariance matrix S of P is obtained. The principal axis of the set P is an eigenvector for the maximum eigenvalue λmax of the covariance matrix. When the distribution is linear, λmax is equal to the sum of the variances in the local regions of the Cy, Ye, and Mg images. That is, the error Er (x, y, d) is expressed by the following equation 18.

(Expression 18) Er (x, y, d) = S00 + S11 + S22−λmax
Here, S00, S11, and S22 are values represented by the following Expressions 19, 20, and 21, which are dispersions of Cy, Mg, and Ye, respectively.

(Expression 19) S00 = Σ (Icy (s−d, t) −avg (Icy)) 2 / N
(Formula 20) S11 = Σ (Img (s, t) −avg (Img)) 2 / N
(Formula 21) S22 = Σ (Iye (s + d, t) −avg (Iye)) 2 / N
Here, N is the number of pixels included in the set P, avg (Icy), avg (Img), and avg (Iye) are average values of the respective components, and are expressed by the following Expression 22, Expression 23, and Expression 24. .

(Formula 22) avg (Icy) = ΣIcy (s−d, t) / N
(Equation 23) avg (Img) = ΣImg (s, t) / N
(Formula 24) avg (Iye) = ΣIye (s + d, t) / N
If the error Er (x, y, d) expressed by Equation 18 is large, it means that the assumption that the amount of parallax is d is likely to be erroneous.

  FIG. 10A shows an example of the distribution of the point set in the three-dimensional color space when the error Er is relatively small. On the other hand, FIG. 10B shows an example of the distribution of the point set in the three-dimensional color space when the error Er is relatively large. In the example of this figure, since the distribution shown in FIG. 10A is closer to the straight line l than the distribution shown in FIG. 10B, the estimated value d of the parallax amount in FIG. It is determined that

  From the above, the estimated value d of the parallax amount is changed within a certain range (for example, d is changed by 1 from −20 to 20), and d at which Er (x, y, d) is minimized is expressed as a coordinate ( What is necessary is just to set it as the amount of parallax in x, y). By performing the above processing for each pixel, the parallax between the three parallax complementary color images can be calculated.

  Hereinafter, the configuration and processing procedure of the parallax estimation unit 40 will be specifically described.

  FIG. 11 is a block diagram illustrating a configuration of the parallax estimation unit 40. The parallax estimation unit 40 extracts a pixel block extraction unit 42 for each pixel from the three parallax complementary color images generated by the image generation unit 7, and the degree of deviation with respect to a straight line from a set of pixel values in each pixel block. A shift determination unit 44 for determination and a parallax amount determination unit 46 for determining a parallax amount based on the determination result are included. The parallax amount d determined for each pixel by the parallax amount determining unit 46 is output to the distance information generating unit 50.

  FIG. 12 is a flowchart illustrating a processing procedure performed by the parallax estimation unit 40. When three parallax complementary color images are generated by the image generation unit 7, the amount of parallax is estimated for each pixel in the following procedure for each pixel. First, in step S202, the pixel block extracting unit 42 sets an estimated value d of the parallax amount indicating how many pixels the Cy image and Ye image are deviated from the Mg image from among a plurality of candidates. Next, in step S203, a pixel block is extracted from each of the three images according to the estimated parallax amount d. Subsequently, in step S204, a point set in the three-dimensional color space is obtained from the set of pixel values of the three pixel blocks based on Expression 17. Next, in step S <b> 205, the deviation determination unit 44 determines how much the point set distribution in the three-dimensional color space is deviated from the straight line based on Expression 18. If determination has not been completed for all candidates for the estimated value d of the parallax amount, the parallax estimated value d is changed to another candidate value in step S207, and steps S203 to S205 are repeated. When the determination is completed for all candidates for the estimated value d of the parallax amount, in step S208, the parallax amount determination unit 46 sets the coordinate d (d) at which the error Er (x, y, d) expressed by Equation 18 is minimized. It is determined as the true amount of parallax in x, y). The parallax estimation unit 40 obtains the parallax between the three parallax complementary color images by executing the above processing for each pixel.

  Through the above processing, a disparity amount distribution (disparity map) for each pixel is obtained, and distance information can be calculated. The distance information generation unit 50 calculates the distance of the subject for each pixel based on Equation 6 from the parallax information obtained by the parallax estimation unit 40.

  As described above, according to the imaging apparatus of the present embodiment, the imaging device includes the Cy region that transmits light in the Cy wavelength region, the Ye region that transmits light in the Ye wavelength region, and the transparent region formed of the transparent member. Imaging is performed using the translucent plate 2. As a result, a plurality of images and color images having parallax can be generated. Furthermore, if the signal processing described above is performed, three complementary color images having parallax can be generated. By obtaining parallax information from the generated three complementary color images, it is possible to obtain subject distance information.

  In addition, although the imaging device of this embodiment produces | generates parallax information and distance information from three parallax complementary color images, the imaging device may be comprised so that only parallax information may be produced | generated. In addition, at least a part of the image generation process, the parallax information generation process, and the distance information generation process by signal calculation may be executed by another device independent of the imaging device. For example, the signal acquired by the imaging apparatus having the imaging unit 100 according to the present embodiment is read by another apparatus, and the program for defining the signal calculation process is executed by the other apparatus as in the present embodiment. The effect of can be obtained.

  In addition, the image generation unit 7 in the present embodiment can generate three parallax complementary color images having parallax, a black and white image and a color image with high light utilization, but it is not necessary to generate all of these images. Absent. The image generation unit 7 may be configured to generate at least three images having parallax.

  In the above description, the transmission region of the light-transmitting plate 2 is configured by a combination of a Cy region, a Ye region, and a transparent region, but the present invention is not limited to such a combination. Even if a combination of Cy and Mg or a combination of Mg and Ye is used instead of the combination of Cy and Ye, the pixel signal can be converted into a complementary color image signal having parallax by the same processing.

(Embodiment 2)
Next, a second embodiment of the present invention will be described with reference to FIG. The imaging apparatus of the present embodiment is the same as that of the first embodiment except that the translucent plate 2 and the three parallax complementary color image generation methods are different from those of the first embodiment. Therefore, in the following description, only points different from the first embodiment will be described, and description of overlapping points will be omitted.

  FIG. 13 is a front view showing the configuration of the light-transmitting plate 2 in the present embodiment. The translucent plate 2 in this embodiment includes a Cy region C1 that transmits Cy light, a Ye region C2 that transmits Ye light, and an Mg region C3 that transmits Mg light. The regions C1, C2, and C3 are formed by a Cy filter, a Ye filter, and an Mg filter, respectively. Moreover, areas other than the areas C1 to C3 of the translucent plate 2 are formed of a light shielding member.

  In the present embodiment, the shape, area, and positional relationship of the transmission regions C1, C2, and C3 are not limited to the example shown in FIG. 13, and can be arbitrarily set.

  Also in this embodiment, the spectral transmittance of each filter is expressed in the same manner as in the first embodiment. Assuming that the spectral transmittance of the Mg filter is Tmg, the signals corresponding to Expressions 8, 9, and 10 are represented by the following Expressions 25, 26, and 27 in this embodiment, respectively.

(Expression 25) Rt = Rs / Tw = Ci1ΣTcyTr + Ci2ΣTyeTr + Ci3ΣTmgTr
(Formula 26) Gt = Gs / Tw = Ci1ΣTcyTg + Ci2ΣTyeTg + Ci3ΣTmgTg
(Expression 27) Bt = Bs / Tw = Bt = Ci1ΣTcyTb + Ci2ΣTyeTb + Ci3ΣTmgTb
Also in this embodiment, Gt + Bt−Rt expressed by the following Expression 28 is defined as an image signal Cs of a Cy component.

(Formula 28) Cs = Gt + Bt−Rt
= Ci1 (ΣTcyTg + ΣTcyTb−ΣTcyTr)
+ Ci2 (ΣTyeTg + ΣTyeTb−ΣTyeTr)
+ Ci3 (ΣTmgTg + ΣTmgTb−ΣTmgTr)
Similarly, signals represented by the following formulas 29 and 30 are defined as image signals Ms and Ys of Mg and Ye components, respectively.

(Formula 29) Ms = Rt + Bt−Gt
= Ci1 (ΣTcyTr + ΣTcyTb−ΣTcyTg)
+ Ci2 (ΣTyeTr + ΣTyeTb−ΣTyeTg)
+ Ci3 (ΣTmgTr + ΣTmgTb−ΣTmgTg)
(Formula 30) Ys = Rt + Gt−Bt
= Ci1 (ΣTcyTr + ΣTcyTg−ΣTcyTb)
+ Ci2 (ΣTyeTr + ΣTyeTg−ΣTyeTb)
+ Ci3 (ΣTmgTr + ΣTmgTg−ΣTmgTb)
Here, since the transmission wavelength ranges of the Cy, Ye, and Mg filters hardly include the R, B, and G wavelength ranges, respectively, ΣTcyTr≈ΣTyeTb≈ΣTmgTg≈0. The transmission wavelength range of the Cy filter includes the G and B wavelength ranges substantially equal, the transmission wavelength range of the Ye filter includes the G and R wavelength ranges substantially equal, and the transmission wavelength range of the Mg filter includes the R and B wavelengths. Suppose that it contains almost equal areas. Then, ΣTcyTg≈ΣTcyTb, ΣTyeTg≈ΣTyeTr, and ΣTmgTr≈ΣTmgTb are established. Furthermore, it is assumed that the integrated value of the spectral transmittance of the color filter facing each pixel of the image sensor 1 is the same for all color components. That is, ΣTr≈ΣTg≈ΣTg is established. Based on the above assumptions, the expressions 28, 29, and 30 are rewritten into the following expressions 31, 32, and 33, respectively.

(Formula 31) Cs = Gt + Bt−Rt
= Ci1 (ΣTcyTg + ΣTcyTb)
(Formula 32) Ms = Rt + Bt−Gt
= Ci3 (ΣTmgTr + ΣTcyTb)
(Formula 33) Ys = Rt + Gt−Bt
= Ci2 (ΣTyeTr + ΣTyeTg)
As described above, in the present embodiment, the signals Cs, Ys, and Ms indicating the three parallax complementary color images are signals corresponding to light incident on the areas Ci1, Ci2, and Ci3, respectively. Since components of light from other regions are not mixed in each signal, the configuration of the present embodiment improves the accuracy of estimating the amount of parallax compared to the configuration of the first embodiment.

(Embodiment 3)
Next, a third embodiment of the present invention will be described with reference to FIG. The imaging apparatus of the present embodiment is the same as that of the first embodiment except that the translucent plate 2 and the three parallax complementary color image generation methods are different from those of the first embodiment. Therefore, in the following description, only points different from the first embodiment will be described, and description of overlapping points will be omitted.

  FIG. 14 is a front view showing the configuration of the light-transmitting plate 2 in the present embodiment. The translucent plate 2 in this embodiment includes a Cy region C1 that transmits Cy light, a Ye region C2 that transmits Ye light, an Mg region C3 that transmits Mg light, and a transparent region C4. Regions C1, C2, C3, and C4 are formed of a Cy filter, a Ye filter, an Mg filter, and a transparent member, respectively.

  In the present embodiment, the translucent plate is divided into four regions, and signal components Ci1, Ci2, Ci3, and Ci4 due to light that has passed through the regions C1, C2, C3, and C4 are used. For example, as shown in FIG. 14, C1 is arranged at the upper part of the translucent plate, C2 is arranged at the lower left, and C3 is arranged at the lower right, and the other areas are configured as C4. Three-direction parallax can be obtained by the regions C1, C2, and C3.

  With the above configuration, signals corresponding to the expressions 8, 9, and 10 are represented by the following expressions 34, 35, and 36, respectively, in the present embodiment.

(Expression 34) Rt = Rs / Tw = Rt = Ci1ΣTcyTr + Ci2ΣTyeTr + Ci3ΣTmgTr + Ci4
(Expression 35) Gt = Gs / Tw = Gt = Ci1ΣTcyTg + Ci2ΣTyeTg + Ci3ΣTmgTg + Ci4
(Expression 36) Bt = Bs / Tw = Bt = Ci1ΣTcyTb + Ci2ΣTyeTb + Ci3ΣTmgTb + Ci4
In the same manner as in the first and second embodiments, the expressions 34, 35, and 36 can be converted into the following expressions 37, 38, and 39.

(Expression 37) Cs = Gt + Bt−Rt = Ci1 (ΣTcyTg + ΣTcyTb) + Ci4
(Formula 38) Ms = Rt + Bt−Gt = Ci3 (ΣTmgTr + ΣTcyTb) −Ci4
(Formula 39) Ys = Gt + Rt + Bt = Ci2 (ΣTyeTg + ΣTyeTr) + Ci4
Through the above processing, a signal obtained by adding or subtracting the signal Ci4 passing through the transparent region to the signals Ci1, Ci2, and Ci3 corresponding to the light incident on the regions C1, C2, and C3 can be obtained.

  According to the translucent plate 2 in the present embodiment, since signals from light incident from three directions can be calculated, the parallax based on the color linearity in consideration of the color shift in both the horizontal direction and the vertical direction shown in FIG. Information can be obtained. This improves the accuracy of parallax estimation in a shape or texture region where it is difficult to obtain features such as a straight line. Further, since the transparent plate has a transparent region, there is an advantage that a highly sensitive two-dimensional color image can be obtained at the same time.

  In addition, although the transparent member is provided in the region C4 in the present embodiment, any one of red, green, and blue color filters may be disposed in the region C4. As an example, FIG. 15 shows a translucent plate 2 in which a green filter is arranged in the region C4. Even when such a light transmitting plate 2 is used, parallax information and depth information can be obtained by the same processing.

  In the above Embodiments 1 to 3, the number of transmission regions in the light transmission plate (light transmission part) 2 is 3 or 4, but the number of transmission regions in the light transmission part in the present invention may be 5 or more. . If the transmission wavelength regions of these transmission regions are different from each other, parallax information can be obtained by the same processing. In addition, the direction in which the parallax can be obtained can be variously changed according to the positional relationship between the transmission regions. The accuracy of parallax estimation is improved by determining an optimal arrangement relationship and estimating the parallax according to the texture, object shape, and color of the shooting scene.

  In Embodiments 1 to 3 described above, a complementary color filter or a transparent member is disposed in each transmission region of the translucent plate 2, but a primary color filter may be disposed in a part of each transmission region. In the present invention, it is only necessary that at least one of the transmission regions has a transmission wavelength region of cyan, yellow, magenta, or transparent. By combining the primary color filter and the complementary color filter, it is possible to change the filter variation and take an image.

  The three-dimensional imaging device of the present invention is effective for all cameras using a solid-state imaging device. For example, it can be used for consumer cameras such as digital still cameras and digital video cameras, and industrial solid-state surveillance cameras.

DESCRIPTION OF SYMBOLS 1 Solid-state image sensor 1a Imaging surface of solid-state image sensor 2 Translucent plate (light transmissive part)
DESCRIPTION OF SYMBOLS 3 Optical lens 4 Infrared cut filter 5 Signal generation / reception part 6 Element drive part 7 Image generation part 8 Interface part 9 Subject 19 Lens diaphragm 20, 22, 23 Light flux limiting plate 20a Color filter 20b which transmits red system light 20b Blue Color filter for transmitting system light 21 Photosensitive film 22R, 23R R light transmission region 22G, 23G G light transmission region 22B of light beam limiting plate B light transmission region 30B of light beam limiting plate 30 Memory 40 Parallax estimation unit 42 pixel block extraction unit 44 displacement determination unit 46 parallax amount determination unit 50 distance information generation unit 60a, 60b, 60c pixel block 100 imaging unit 110 color filter 120 photosensitive cell 200 signal processing unit

Claims (9)

The first, second, and third transmission regions have different transmission wavelength ranges, and at least one of the first, second, and third transmission regions is any one of cyan, yellow, and magenta A member that transmits light in the wavelength range, or a light transmitting portion formed of a transparent member;
A solid-state imaging device having a photosensitive cell array and arranged to receive light transmitted through the light transmission unit, wherein the photosensitive cell array has a plurality of unit blocks, and each unit block is red R detection cell that outputs a first photoelectric conversion signal according to the amount of light in the wavelength range, G detection cell that outputs a second photoelectric conversion signal according to the amount of light in the green wavelength range, and blue wavelength A solid-state imaging device including a B detection cell that outputs a third photoelectric conversion signal corresponding to the amount of light in the region;
An optical system for forming an image on the imaging surface of the solid-state imaging device;
A signal processing unit for processing a signal output from the solid-state imaging device, wherein the first, second, and second signals are based on processing including addition and subtraction using the first, second, and third photoelectric conversion signals. And an image generation unit that generates three images having parallax by generating three color mixing signals corresponding to the amount of light incident on each of the third transmission regions, and estimates the parallax between the three images A signal processing unit having a parallax estimation unit to perform,
A three-dimensional imaging device.   The three-dimensional imaging apparatus according to claim 1, wherein the signal processing unit further includes a distance information generation unit that generates information indicating a distance of a subject from the parallax estimated by the parallax estimation unit. The parallax estimation unit
For each pixel of the three images, an estimated value of the parallax amount is set from among a plurality of candidates, and three pixel blocks of the same size whose positions on the image are shifted from the three images based on the estimated value A pixel block extraction unit for extracting
A shift determination unit that determines how much the distribution of the point set on the three-dimensional color space defined by the set of pixel values of the three pixel blocks is shifted from a straight line;
A parallax amount determination unit that determines, as a parallax amount in each pixel, the estimated value that minimizes the degree of deviation from the straight line determined by the shift determination unit;
The three-dimensional imaging device according to claim 1, comprising:   The first transmission region is formed of a member that transmits light in one wavelength region of cyan, yellow, and magenta, and the second transmission region is the other of cyan, yellow, and magenta. 4. The three-dimensional imaging apparatus according to claim 1, wherein the three-dimensional imaging device is formed of a member that transmits light in one wavelength region, and the third transmission region is formed of a transparent member.   The three-dimensional according to any one of claims 1 to 4, wherein each of the first, second, and third transmission regions is formed of a member that transmits light in a wavelength range of cyan, yellow, and magenta, respectively. Imaging device.   The light transmission part has a fourth transmission region, and the fourth transmission region is formed of a member that transmits light in any one of the wavelength ranges of red, green, and blue, or a transparent member. The three-dimensional imaging device according to claim 5. The first, second, and third transmission regions have different transmission wavelength ranges, and at least one of the first, second, and third transmission regions is any one of cyan, yellow, and magenta A member that transmits light in the wavelength range, or a light transmitting portion formed of a transparent member;
A solid-state imaging device having a photosensitive cell array and arranged to receive light transmitted through the light transmission unit, wherein the photosensitive cell array has a plurality of unit blocks, and each unit block is red R detection cell that outputs a first photoelectric conversion signal according to the amount of light in the wavelength range, G detection cell that outputs a second photoelectric conversion signal according to the amount of light in the green wavelength range, and blue wavelength A solid-state imaging device including a B detection cell that outputs a third photoelectric conversion signal corresponding to the amount of light in the region;
An optical system for forming an image on the imaging surface of the solid-state imaging device;
A signal processing method for processing a signal output from an imaging device comprising:
Based on the processing including addition / subtraction using the first, second, and third photoelectric conversion signals, three values corresponding to the amount of light incident on each of the first, second, and third transmission regions Generating three images having parallax by generating a color mixture signal;
Estimating the parallax between the three images;
A signal processing method including:   The signal processing method according to claim 7, further comprising: generating information indicating a distance of a subject from the estimated parallax. Estimating the parallax comprises:
For each pixel of the three images, an estimated value of the parallax amount is set from among a plurality of candidates, and three pixel blocks of the same size whose positions on the image are shifted from the three images based on the estimated value Extracting each of the
Determining how much the point set distribution in the three-dimensional color space defined by the set of pixel values of the three pixel blocks deviates from a straight line;
Determining the estimated value that minimizes the degree of deviation from the determined straight line as the amount of parallax in each pixel;
The signal processing method of Claim 7 or 8 containing these.

Images