JP6285686B2 - Parallax image generation device - Google Patents

Description

  The present invention relates to a parallax image generation device that generates a parallax image using images taken by a plurality of cameras.

  There are many techniques for generating parallax images in which parallax is estimated for each image from stereo images shot by two color cameras with different shooting positions. However, most of them are obtained by searching for the correspondence of each pixel in the left and right images with the local luminance change of the color image as a clue. For this reason, as is often seen in artifacts, there is a problem in that the estimation accuracy is lowered for a region with a poor texture.

In order to overcome the problem, there is a method of generating a parallax image with high speed and high accuracy by irradiating a known pattern and observing the irradiated pattern.
For example, Patent Document 1 describes a method for obtaining a three-dimensional shape of an object by irradiating a random speckle pattern on the surface of the object and analyzing two-dimensional images photographed by a plurality of cameras.

US Pat. No. 6,1011,269

In the method described in Patent Document 1, a speckle pattern is irradiated to give a texture to the surface of an object that is a subject, and the texture is photographed in an image photographed by a camera. For this reason, when the image captured by the camera is used to generate a parallax image and is used as a reference image when generating an image of a different viewpoint using the parallax image generated from the captured image, the texture itself There is a problem that becomes noise.
In addition, since the speckle pattern to be irradiated cannot always be applied at a high density, it is necessary to refer to a wide area in order to estimate the parallax based only on the applied texture. For this reason, the technique described in Patent Document 1 has a problem that the estimation accuracy of the parallax is reduced in the vicinity of the boundary between the regions where the parallax is greatly different.

  The present invention was devised in view of such a problem, and a parallax image generation device capable of accurately generating a parallax image by giving a texture to a subject having a small texture without affecting the image in the visible light region. It is an issue to provide.

  In order to solve the above-described problem, a parallax image generation device according to an aspect of the present invention provides an infrared image that is an image in the infrared wavelength region and an image in the visible wavelength region with respect to a subject on which an infrared pattern is projected. A parallax image generation device that generates a parallax image using a set of infrared images and a set of visible light images obtained by capturing a certain visible light image with two cameras, and includes an image conversion unit and a correspondence map group generation unit And a parallax image generation processing unit.

According to such a configuration, the parallax image generation device uses the camera parameters for the two cameras to cause the set of infrared images to be parallel to the optical axes of the two cameras by the image conversion unit. Are converted into a set of collimated infrared images, which are images obtained by using the camera parameters for the two cameras, and the set of visible light images is made parallel to the optical axes of the two cameras. It is converted into a set of collimated visible light images which are images obtained at the time.
Next, the parallax image generation device uses the correspondence map group generation unit to generate a parallax between the reference infrared image that is the reference parallelized infrared image and the other non-reference infrared image that is the parallelized infrared image. Is a constant value for the entire image, the degree of correspondence, which is an index indicating the degree of matching between the reference infrared image and the non-reference infrared image for each pixel of the predetermined range image including the pixel, and The parallax between the reference visible light image that is the collimated visible light image having the same optical axis as the reference infrared image and the non-reference visible light image that is the other collimated visible light image is the constant value in the entire image. And the correspondence degree that is an index indicating the degree of matching of the image of the predetermined range including the pixel for each pixel of the reference visible light image and the non-reference visible light image. Two-dimensional arrangement of degrees The degree of correspondence map is, to produce a corresponding degree of map group by obtaining for each parallax predetermined intervals for a predetermined parallax range.

Then, the parallax image generation device selects the parallax for the correspondence map having the highest degree of matching in the correspondence map group for each pixel by the parallax image generation processing unit, so that the parallax is generated for each pixel. A parallax image that is a predetermined image is generated.
As a result, the parallax image generation apparatus accurately estimates the parallax using the infrared stereo image to which the texture of the infrared pattern is given for the subject region with less texture, and uses the visible light stereo image for the subject region having the texture. Estimate the parallax with high accuracy.

  In addition, the parallax image generating device according to another aspect of the present invention is a set of infrared images obtained by photographing an infrared image that is an image in the infrared wavelength region with two infrared cameras for a subject on which an infrared pattern is projected, A parallax image generation device that generates a parallax image using a set of visible light images obtained by capturing a visible light image that is an image in a wavelength region of visible light with two visible light cameras, an image conversion unit, A correspondence map group generation unit and a parallax image generation processing unit are provided.

According to such a configuration, the parallax image generation device uses the camera parameters for the two infrared cameras to cause the pair of infrared images to be parallel to the optical axes of the two infrared cameras. And converting the set of visible light images into the set of collimated infrared images, which are images obtained at the time, and using the camera parameters for the two visible light cameras and the two infrared cameras. The optical axes of the two visible light cameras are converted into a set of collimated visible light images that are images obtained when the optical axes for obtaining the set of collimated infrared images are the same.
Next, the parallax image generation device uses the correspondence map group generation unit to generate a parallax between the reference infrared image that is the reference parallelized infrared image and the other non-reference infrared image that is the parallelized infrared image. Is a constant value for the entire image, the degree of correspondence, which is an index indicating the degree of matching between the reference infrared image and the non-reference infrared image for each pixel of the predetermined range image including the pixel, and The parallax between the reference visible light image that is the collimated visible light image having the same optical axis as the reference infrared image and the non-reference visible light image that is the other collimated visible light image is the constant value in the entire image. And the correspondence degree that is an index indicating the degree of matching of the image of the predetermined range including the pixel for each pixel of the reference visible light image and the non-reference visible light image. Two-dimensional arrangement of degrees The degree of correspondence map is, to produce a corresponding degree of map group by obtaining for each parallax predetermined intervals for a predetermined parallax range.

Then, the parallax image generation device determines the parallax for each pixel by selecting the parallax for the correspondence map having the highest correspondence in the correspondence map group for each pixel by the parallax image generation processing unit. A parallax image which is the obtained image is generated.
As a result, the parallax image generation apparatus accurately estimates the parallax using the infrared stereo image to which the texture of the infrared pattern is given for the subject region with less texture, and uses the visible light stereo image for the subject region having the texture. Estimate the parallax with high accuracy.

  A parallax image generation device according to still another aspect of the present invention provides a set of infrared images obtained by capturing an infrared image, which is an image in the infrared wavelength region, with two infrared cameras for a subject on which an infrared pattern is projected, and a visible image. A parallax image generation device that generates a parallax image using a visible light image obtained by capturing a visible light image that is an image in the wavelength region of light with a visible light camera, the image conversion unit, and a correspondence map group generation unit And a smoothing filter processing unit and a parallax image generation processing unit.

According to such a configuration, when the parallax image generation device makes the set of infrared images parallel to the optical axis of the infrared camera using the camera parameters for the two infrared cameras by the image conversion unit. Converted into a set of collimated infrared images, which are obtained images, and using the camera parameters for the visible light camera and the reference infrared camera, the visible light image is referenced to the optical axis of the visible light camera. Is converted into a reference visible light image which is an image obtained when the same optical axis for obtaining the collimated infrared image is obtained.
Next, the parallax image generation device uses a correspondence map group generation unit to generate a parallax between a reference infrared image that is the reference parallelized infrared image and a non-reference infrared image that is the other parallelized infrared image. 2 is an index indicating the degree of matching between the reference infrared image and the non-reference infrared image for each pixel of a predetermined range image including the pixel, where is a constant value in the entire image. A correspondence map group is generated by obtaining a correspondence map, which is a dimensional array, for each parallax at a predetermined interval in a predetermined parallax range.

Next, the parallax image generating device uses the reference visible light image as a guide image for edge region identification for each correspondence map for the correspondence map group by the smoothing filter processing unit. Smoothing filter processing that holds the edge of the correspondence map corresponding to the edge of the subject is performed.
Then, the parallax image generation device selects, for each pixel, the parallax for the correspondence map having the highest correspondence degree among the correspondence map groups subjected to the smoothing filter processing by the parallax image generation processing unit. A parallax image that is an image in which parallax is determined for each pixel is generated.
Thereby, the parallax image generating apparatus estimates the parallax using the infrared stereo image to which the texture of the subject is given the texture by the infrared pattern. In addition, the parallax image generation device improves the parallax estimation accuracy by performing an edge-holding smoothing filter process using a visible light image as a guide image.

  According to the present invention, since the parallax is estimated using the infrared image and the visible light image obtained by photographing the subject on which the infrared pattern is projected, the parallax image in which the parallax is accurately estimated even for the subject region with less texture is generated. Can do. Further, according to the present invention, since the texture is given by the infrared pattern, the visible light image is not affected.

It is a block diagram which shows the structure of the parallax image generation system provided with the parallax image generation apparatus which concerns on 1st Embodiment of this invention. It is a typical perspective view which shows arrangement | positioning of the camera and infrared pattern irradiation machine in the parallax image generation system provided with the parallax image generation apparatus which concerns on each embodiment of this invention, (a) is 1st Embodiment and 2nd Embodiment. (B) shows an example of the third embodiment, and (c) shows an example of the fourth embodiment. It is a figure for demonstrating the production | generation of a cost volume in the parallax image generation system which concerns on 1st Embodiment of this invention. It is a flowchart which shows operation | movement of the parallax image generation system which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the parallax image generation system which concerns on 2nd Embodiment of this invention. It is a figure for demonstrating the smoothing filter process of a cost volume in the parallax image generation system which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the parallax image generation system which concerns on 2nd Embodiment of this invention. It is a block diagram which shows the structure of the parallax image generation system which concerns on 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the parallax image generation system which concerns on 3rd Embodiment of this invention. It is a block diagram which shows the structure of the parallax image generation system which concerns on 4th Embodiment of this invention. It is a flowchart which shows operation | movement of the parallax image generation system which concerns on 4th Embodiment of this invention. It is a block diagram which shows the structure of the parallax image generation system which concerns on 5th Embodiment of this invention. It is a figure for demonstrating the smoothing filter process of a cost volume in the parallax image generation system which concerns on 5th Embodiment of this invention. It is a figure for demonstrating the smoothing filter process of a cost volume in the parallax image generation system which concerns on 5th Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings as appropriate.
<First Embodiment>
[Configuration of parallax image generation system]
First, the configuration of the parallax image generation system according to the first embodiment will be described with reference to FIG. 1 and FIG.
As illustrated in FIG. 1, the parallax image generation system 100 according to the present embodiment includes a parallax image generation device 1, an infrared pattern irradiator 2, and a photographing device 3. The parallax image generation device 1 includes an image conversion unit 4, a cost volume generation unit 5, and a parallax image generation processing unit 6. In the present embodiment, the photographing apparatus 3 includes two infrared color cameras 31 and 32 that photograph images in the wavelength range of four channels of infrared rays and RGB (red, green, and blue) with the same optical axis. Moreover, as shown to Fig.2 (a), the infrared pattern irradiation machine 2 and the two infrared color cameras 31 and 32 are juxtaposed in the horizontal direction (X-axis direction).

  In this specification, unless otherwise specified, as shown in FIG. 2A, the horizontal direction is the X-axis method, the vertical direction is the Y-axis direction, and a camera (for example, a reference) in the photographing apparatus 3 (for example, The optical axis direction of the infrared color camera 31), that is, the depth direction of the subject will be described as the Z-axis direction.

The parallax image generation system 100 according to the present embodiment gives an infrared texture to a subject by projecting an infrared pattern onto the subject by the infrared pattern irradiator 2. Then, an index corresponding to the distance in the depth direction of the subject by using the set of infrared images obtained by photographing the subject to which the infrared texture is applied by the parallax image generation device 1 and the set of color images that are visible light images. A parallax image that is an image showing parallax is generated.
Hereinafter, each configuration will be sequentially described in detail.

The infrared pattern irradiator 2 is for irradiating an infrared pattern and projecting the infrared pattern onto the subject to impart an infrared texture of the subject.
Although the wavelength of the infrared rays to be irradiated is not particularly limited, it is preferably a wavelength in the near infrared region that is captured in an infrared image and not captured in a color image.

The infrared pattern irradiator 2 can be composed of, for example, a light source that outputs coherent light and a light diffuser. An infrared laser can be used as a light source that outputs coherent light. In addition, a frosted glass can be used as the light diffuser.
By irradiating the rubbing glass with infrared laser light, the diffracted light of the infrared laser light due to the unevenness of the rubbing glass interferes with each other to form a speckle pattern. At this time, since the unevenness of the frosted glass is randomly arranged, the speckle pattern also becomes a random spotted pattern.
Further, the light diffuser may be a hologram having irregularities randomly formed on the surface. Furthermore, the light diffuser may be one that transmits or reflects infrared laser light.

  Further, the infrared pattern irradiator 2 is not limited to a single unit, and may be configured by a plurality of units to irradiate an infrared pattern over a wide range. At this time, the infrared patterns irradiated from the plurality of infrared pattern irradiators 2 may be irradiated so as to overlap so that the infrared patterns are projected onto the subject at a higher density.

The infrared pattern to be irradiated is preferably a random dot pattern such as the speckle pattern described above, but is not limited thereto, and may be another texture shape such as a stripe pattern or a lattice pattern with irregular intervals. . At this time, it is sufficient that the infrared pattern does not generate a similar pattern having a high correlation with each other at least in the vicinity of the epipolar line of the infrared image taken in stereo. That is, it is sufficient that the pattern is random at least in the vicinity of the epipolar line.
Further, it is preferable that the infrared pattern is fine as long as the pattern can be identified in the infrared image captured by the infrared color cameras 31 and 32. The finer the pattern, the higher the resolution of parallax estimation by stereo matching using an infrared image.

  As described above, the photographing device 3 is composed of the two infrared color cameras 31 and 32, and is a camera that individually photographs an infrared image and a color image composed of RGB. In the present embodiment, two infrared color cameras 31 and 32 and an infrared pattern irradiator 2 are juxtaposed in the horizontal direction. Further, the infrared pattern irradiator 2 is disposed between the two infrared color cameras 31 and 32. That is, the infrared pattern irradiator 2 is arranged close to both of the two infrared color cameras 31 and 32. For this reason, when viewed from either of the two infrared color cameras 31 and 32, the infrared pattern irradiated from the infrared pattern irradiator 2 toward the subject is a shadow of a part of the subject and is not projected, that is, Since the area | region where a texture is not provided can be reduced, it is preferable.

  The infrared color cameras 31 and 32 are cameras that shoot an infrared image and a color image composed of RGB on the same optical axis. In the present embodiment, the left viewpoint infrared image and color image captured by the infrared color camera 31 arranged on the left side of the subject are used as the reference images of the respective channels (wavelength regions). Further, an infrared stereo image and a color stereo image are obtained by combining a right viewpoint infrared image and a color image captured by the infrared color camera 32 arranged on the right side with a left viewpoint infrared image and a color image which are reference viewpoints, respectively. And

Hereinafter, a set of infrared images and a set of color images taken by two cameras are appropriately referred to as “infrared stereo image” and “color stereo image”, respectively. The same applies to the case where the infrared camera and the color camera are separate cameras as in other embodiments described later.
The infrared color cameras 31 and 32 output the captured infrared stereo image to the image parallelization processing unit 41 of the parallax image generation device 1 and the captured color stereo image to the image parallelization processing unit 42 of the parallax image generation device 1, respectively. To do.

In this embodiment, the two infrared color cameras 31 and 32 are used for stereo shooting. However, the subject is shot using three or more infrared color cameras. You may make it use the image group image | photographed with the camera as a stereo image, respectively.
In the present embodiment, the infrared color cameras 31 and 32 are used to capture a color image composed of three RGB channels as an image in the visible light region, but a monochrome image in the visible light region, a two-channel image, or 4 You may make it image | photograph the image more than a channel.

Moreover, it is preferable that the infrared color cameras 31 and 32 have sensitivity to the infrared wavelength irradiated by the infrared pattern irradiator 2 in the infrared image, and do not have sensitivity to the infrared wavelength in the color image. That is, in the color image, since the infrared ray has no wavelength, an infrared pattern is not photographed in the color image. Thereby, a color image can be taken without being affected by the irradiation of the infrared pattern.
Therefore, the parallax image generated by the parallax image generating apparatus 1 according to the present embodiment can be used as a color image having a parallax image by combining with the color image captured by the infrared color cameras 31 and 32.

  The parallax image generation device 1 includes an image conversion unit 4, a cost volume generation unit 5, and a parallax image generation processing unit 6, and includes an infrared stereo image, a visible light stereo image, and an infrared color camera that captures these images. A parallax image is generated using the camera parameters 31 and 32.

  The image conversion unit 4 includes image parallelization processing units 41 and 42, and parallelizes the infrared stereo image and the color stereo image input from the infrared color cameras 31 and 32, respectively. Here, the collimation of images means that the coordinates of two images are transformed into images when the optical axes are parallel to each other without changing the respective optical principal points.

The image parallelization processing unit 41 inputs infrared stereo images from the infrared color cameras 31 and 32 and inputs camera parameters of the two infrared color cameras 31 and 32 to parallelize the input infrared stereo images. Is. The camera parameters and the acquisition method will be described later.
The image parallelization processing unit 41 outputs the parallelized infrared stereo image to the cost volume calculation unit 51 of the cost volume generation unit 5.

The image parallelization processing unit 42 inputs color stereo images from the infrared color cameras 31 and 32 and inputs camera parameters of the two infrared color cameras 31 and 32 to parallelize the input color stereo images. Is. The camera parameters and the acquisition method will be described later.
The image parallelization processing unit 42 outputs the parallelized color stereo image to the cost volume calculation unit 52 of the cost volume generation unit 5.

Here, the camera parameters will be described.
The camera parameters include an internal parameter that indicates the state of the camera and an external parameter that indicates the positional relationship of the camera.
The internal parameters include information about the focal length of the camera lens, the pixel pitch, the aspect ratio which is the aspect ratio of the pixel pitch, the image coordinates of the intersection of the optical axis and the imaging surface, and distortion by the camera lens. The external parameters include the camera position and rotation amount in the world coordinate system or the camera coordinate system of the reference camera. Of these, the pixel pitch and aspect ratio can be acquired in advance as known values specific to the camera, and other parameters can be acquired by performing camera calibration.
In the present embodiment, it is assumed that the stereo image is parallelized using the rotation amount of the camera in the world coordinate system for each camera as the external parameter.
When a plurality of cameras having the same specification are used, an average value of internal parameters of the plurality of cameras may be used as an internal parameter common to the cameras.

Camera calibration can be performed by photographing a known pattern such as a marker or a test chart with a camera to be calibrated and analyzing the photographed image. As camera calibration based on such image analysis, for example, the method described in Reference 1 can be used, and thus detailed description thereof is omitted.
In the present embodiment, it is assumed that the camera parameters are obtained in advance by the method described above.
(Reference 1) R. Tsai, “A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Cameras and Lenses,” Journal of Robotics & Automation, RA-3 (4), pp. 323-344, (1987)

Next, image parallelization processing by the image parallelization processing units 41 and 42 will be described.
The image collimation process is a process for converting the coordinates of an image photographed by two cameras in a state where the optical axes are not parallel into an image photographed in a state where the optical axes are parallel.
Even when two cameras are installed so that the optical axes of the cameras are parallel, the stereo image can be processed with higher accuracy by performing the parallelization process using the camera parameters acquired by camera calibration. Can be parallelized.

Specifically, when the internal parameter matrix before the coordinate transformation process is F, the rotation matrix that is the external parameter is R, the internal parameter after the coordinate transformation process is F _r , and the rotation matrix is R _r , Equation (1) is The image can be parallelized by using this.

Here, (u, v) represents the image coordinates before coordinate _{transformation, (u r, v} r) represents the image coordinates after coordinate transformation process, _{P C} and _{P Cr} is and pre coordinate transformation respectively The homogeneous coordinates of the image coordinates after the coordinate conversion process are shown.
Further, the internal parameter matrix F can be expressed as shown in Equation (2).

Here, f is the focal length of the camera lens expressed in units of vertical pixel pitch, a is the aspect ratio calculated by dividing the vertical pixel pitch by the horizontal pixel pitch, and (C _u , C _v ) Indicates the intersection between the optical axis of the camera and the image plane, that is, the image coordinates of the center pixel of the image to be captured.

  In this embodiment, the image parallelization processing unit 41 that parallelizes the infrared stereo image and the image parallelization processing unit 42 that parallelizes the color stereo image are provided independently. An image parallelization processing unit 41 (or 42) may be provided, and the parallelization processing may be sequentially performed on the infrared stereo image and the color stereo image by shifting the timing.

The description of the configuration of the parallax image generation device 1 will be continued.
The cost volume generation unit (correspondence map group generation unit) 5 uses the parallelized infrared stereo image and color stereo image, and assuming that the parallax is a certain value, the left and right images constituting the stereo image A cost volume (Cost Map) that is a two-dimensional array of “cost (Cost)” (correspondence) for each pixel indicating the degree of coincidence for each of the predetermined ranges of disparity Volume) (correspondence map group) is generated.
For this purpose, the cost volume generation unit 5 includes cost volume calculation units 51 and 52 and a cost volume integration unit 53.
The cost volume generation unit 5 outputs the generated cost volume to the parallax image generation processing unit 6.

Cost volume calculation unit 51, using inputs image parallel processing portion 41 infrared stereo image _J r which is collimated from, _{L, J r,} the _R, infrared stereo image _J r input, _{L, J r,} the _R To calculate the cost volume. The cost volume calculation unit 51 outputs the calculated cost volume to the cost volume integration unit 53.

Further, the cost volume calculation unit 52 receives the color stereo images I _{r, L} , I _{r, R} that have been collimated from the image collimation processing unit 42 _, and the input color stereo images I _{r, L} , I _{r, R.} Is used to calculate the cost volume. The cost volume calculation unit 52 outputs the calculated cost volume to the cost volume integration unit 53.

Here, with reference to FIG. 3 (refer to FIG. 1 as appropriate), the cost volume calculation method in the cost volume calculation units 51 and 52 will be described.
First, the cost volume calculation unit 51 will be described.
For each of the parallel infrared stereo images _{Jr, L} , _{Jr, R} , the cost volume calculation unit 51 calculates, for each pixel, the degree of coincidence of a predetermined size image region centered on the pixel, that is, the cost indicating the correlation. calculate. At this time, it is assumed that the parallax between the left-viewpoint infrared image _{Jr, L} that is the reference image and the right-viewpoint infrared image _{Jr, R} that is the other image is a constant value d in the entire image. Thus, the cost is calculated for each pixel.
In the present embodiment, the parallax d denote the difference between the image _J r of the left and right are stereo images, _{L, J r,} the image coordinates of the corresponding points of the subject in _{R (u r, v r)} .

Specifically, as shown in FIG. 3, an image block B _L is an image of a predetermined range of the left image J _r, centered at the pixel p _L of _L which is the reference image, the other image of the stereo image right image J _r, the degree of coincidence between the image blocks B _R of the same size around the corresponding point p _R of _R, i.e., calculates the cost as an index indicating the correlation of the image. In the present embodiment, as the cost of the value calculated is small, matching of the image blocks B _L and the image block B _R (correlation) is high.

Here, the image blocks B _L, the size of B _R is a side length of (number of pixels) is a square region of tau. Further, when the left viewpoint image J _{r, L} is used as a reference, the corresponding point of the subject in the right viewpoint image J _{r, R} is shifted to the left by the parallax d. Therefore, when the image coordinates of the central pixel _{P L} of the image blocks _{B L} and _{(u r, v} r), the image coordinates of the central pixel _{p R} of the image blocks _{B R,} assuming parallax d, only the disparity d left a shifted _{(u r} -d, _v r) to.
In the present embodiment, Expression (3) is used as the cost evaluation expression.

In Expression (3), C _j (u _r , v _r , d) is the reference image of the parallelized left-viewpoint infrared image J _{r, L} assuming that the parallax is d. the image coordinates _{(u r, v} r) indicating the cost at. By calculating the cost C _j for all the pixels, the cost map CM _jd (CM _{j1 to} CM _jN) , which is a two-dimensional array of costs for each pixel, assuming that the parallax is d for the infrared stereo image. Either) is calculated.

Further, a cost map is calculated by the above-described procedure for each parallax d within a predetermined range (d = 1 to N). As a result, an array composed of N cost maps CM _{j1 to} CM _jN , that is, a cost volume CV _j that is a three-dimensional array of costs is calculated.
Here, N is a value predetermined as the maximum value that the parallax d can take. In the present embodiment, d is described as being calculated every “1”. However, d may be calculated as an integer value such as “2”, and every “0.5” or “2.5”. It is good also for every decimal value like.

The image block B _L, the size τ of B _R, captured infrared image J _{r, L, J} r, depending on the fineness of the infrared pattern resolution and infrared pattern irradiator 2 _R is irradiated, as appropriate For example, it can be about 7 to 19 pixels. The image block B _L, the shape of B _R is a simple calculation by the square, not limited to this, rectangular, hexagonal, polygonal such as rhombus, or the like may be used circular.

Further, the cost volume calculation unit 52 calculates the cost volume CV _c for the parallel color stereo images I _{r, L} , I _{r, R in} the same procedure as the cost volume calculation unit 51 described above.
It should be noted that in cost volume calculation unit 52, a color stereoscopic image I _{r, L, I} r, the image blocks B _L when calculating the cost of about each pixel of _R, the size τ and / or shape of B _R is cost volume You may make it differ from the calculation part 51. FIG.

Here, it is assumed that the parallax is d for the color stereo images I _{r, L} , I _{r, R} , and the cost C _c in the image coordinates (u _r , v _r ) of the color image I _{r, L} that is the reference image. (U _r , v _r , d) is calculated by equation (4).

In Equation (4), k represents each of the RGB channels in the color images I _{r, L} , I _{r, R.}
Further, the color image _I r, _{L, I r,} instead of equation (4) using the _R, gray image _M r, _{L, M r,} cost by equation (5) with _{R C} c _(u r, v _r , d) may be calculated. Here, the gray images _{Mr, L} , _{Mr, R} are images composed of one component generated from the color images Ir _{, L} , Ir _{, R.} As the gray images _{Mr, L} , _{Mr, and R} , for example, images showing the luminance components of the color images Ir _{, L} , Ir _{, and R} can be used.

  In the present embodiment, as shown in the equations (3) to (5), the cost is calculated using an evaluation equation generally called SSD (Sum of Squared Difference). You may make it use the other evaluation formula which shows a correlation. As other evaluation formulas, for example, SAD (Sum of Absolute Difference), NCC (Normalized Cross Correlation), ZNCC (Zero-mean Normalized Cross Correlation), SD (Squared Difference), and the like can be used.

In the present embodiment, a cost volume calculation unit 51 that calculates a cost volume using the parallelized infrared stereo images _{Jr, L} , _{Jr, R} , and the parallelized color stereo images Ir _{, L} , The cost volume calculation unit 52 that calculates the cost volume using Ir _{and R} is provided independently. However, one cost volume calculation unit 51 (or 52) is provided, and the timing is shifted. You may comprise so that a cost volume may be calculated sequentially using infrared stereo image _{Jr, L} , _{Jr, R} and color stereo image Ir _{, L} , Ir _{, R.}

Returning to FIG. 1, the description of the configuration of the parallax image generating device 1 will be continued.
The cost volume integration unit 53 receives the cost volume for the infrared stereo images J _{r, L} , J _{r, R} from the cost volume calculation unit 51, and the color stereo images I _{r, L} , I _{r, R} from the cost volume calculation unit 52. Are entered, and the entered two cost volumes are integrated into one. The cost volume integration unit 53 outputs one integrated cost volume to the parallax image generation processing unit 6.

Specifically, the cost volume integration unit 53 adds the weights C _j and C _c of the two cost volumes for each parallax d and for each pixel by weighted addition according to Expression (6). Integrate into a volume.
In Equation (6), λ is a weighting coefficient, and 0 <λ <1.

Also, how to integrate the two cost volume, the cost _C j according to equation (6) is not limited to the weighted sum of _{C c,} the cost _C j, the product of _{C c,} the cost _C j, the _{C c} Other methods such as the sum of the reciprocals and the product of the reciprocals of the costs C _j and C _c may be used.

  In the present embodiment, the cost volume calculation units 51 and 52 first generate two cost volumes and then integrate the two cost volumes into one. However, the present invention is not limited to this. . For example, each time the cost map of the same parallax for the infrared stereo image and the color stereo image is calculated, the cost map of the same parallax and the same pixel for the infrared stereo image and the color stereo image may be integrated into one cost map. Each time the cost is calculated, it may be integrated into one cost.

  The parallax image generation processing unit 6 inputs the cost volume integrated into one from the cost volume integration unit 53, and generates a parallax image in which the parallax is determined for each pixel using the input cost volume. Further, the parallax image generation processing unit 6 outputs the generated parallax image to the outside as an output of the parallax image generation device 1.

Specifically, the parallax image generation processing unit 6 selects the parallax d assumed in the cost map that minimizes the cost for each pixel in the cost volume as the parallax for the pixel. That is, for each pixel of the stereo image taken by the infrared color cameras 31 and 32, the degree of coincidence between the image blocks B _L and B _R (see FIG. 3) having a predetermined size centered on each pixel is the highest. The parallax is determined (so that the cost is minimized).
A parallax image can be generated by selecting the parallax d by the same procedure for all pixels.

[Operation of parallax image generation system]
Next, the operation of the parallax image generation system 100 according to the first embodiment will be described with reference to FIG. 4 (refer to FIG. 1 as appropriate).
As shown in FIG. 4, first, the parallax image generation system 100 irradiates an infrared pattern such as a random dot pattern with the infrared pattern irradiator 2 to give the subject a texture that can be identified by the infrared image (step S11). ).

Next, the parallax image generation system 100 captures an infrared stereo image and a color stereo image with the two infrared color cameras 31 and 32 (step S12).
Next, the parallax image generation system 100 uses the image parallelization processing unit 41 to parallelize the infrared stereo image captured in step S12 (step S13). Further, the parallax image generation system 100 causes the image parallelization processing unit 42 to parallelize the color stereo image captured in step S12 (step S14). Note that either step S13 or step S14 may be executed first or in parallel.

  Next, the parallax image generation system 100 calculates the cost volume for the infrared stereo image parallelized in step S13 by the cost volume calculation unit 51 (step S15). Further, the parallax image generation system 100 calculates the cost volume for the color stereo image parallelized in step S14 by the cost volume calculation unit 52 (step S16). Note that either step S15 or step S16 may be executed first or in parallel.

  Next, the parallax image generation system 100 uses the cost volume integration unit 53 to calculate the cost volume for the infrared stereo image calculated in step S15 and the cost volume for the color stereo image calculated in step S16 using Equation (6). Is used to add the weights for each parallax and each pixel with a weight, thereby integrating them into one cost volume (step S17).

And the parallax image generation system 100 produces | generates a parallax image by selecting the parallax which gives the minimum cost for every pixel by the parallax image generation process part 6 about the cost volume integrated by step S17 (step S18).
Through the above procedure, the parallax image generation system 100 can generate a parallax image.

In the present embodiment, the parallax is estimated using the cost volume obtained by integrating the cost volume calculated using the infrared stereo image and the cost volume calculated using the color image as described above. For this reason, for a subject having a texture, the parallax can be accurately estimated by a cost volume based mainly on a color stereo image in which the original texture is captured. In addition, for a subject that does not originally have a texture, the parallax can be accurately estimated by a cost volume based on an infrared stereo image obtained by photographing an image that is textured mainly by an infrared pattern.
That is, by using an infrared stereo image and a color stereo image, regions with low parallax estimation accuracy can be complemented, and thus a parallax image with high accuracy can be generated.

Second Embodiment
[Configuration of parallax image generation system]
Next, a parallax image generation system according to the second embodiment of the present invention will be described with reference to FIG.
As shown in FIG. 5, the parallax image generation system 100A according to the second embodiment is different from the parallax image generation system 100 according to the first embodiment shown in FIG. Instead of the generation apparatus 1, a difference is that a parallax image generation apparatus 1A including a cost volume generation unit 5A is provided.
In the parallax image generation device 1A of the second embodiment, the cost volume generation unit 5A further includes a cost volume filter processing unit 54. The cost volume integrated by the cost volume integration unit 53 is held for each cost map. The parallax estimation accuracy near the contour of the subject is improved by performing the type smoothing filter processing.

In addition, about the component similar to the parallax image generation system 100 which concerns on 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted suitably.
Hereinafter, the cost volume filter processing unit 54 of the parallax image generation device 1A will be mainly described.

The parallax image generation device 1A includes an image conversion unit 4, a cost volume generation unit 5A, and a parallax image generation processing unit 6, and includes an infrared stereo image, a visible light stereo image, and an infrared color camera that captures these images. A parallax image is generated using the camera parameters 31 and 32.
The cost volume generation unit 5A includes cost volume calculation units 51 and 52, a cost volume integration unit 53, and a cost volume filter processing unit 54.

The cost volume filter processing unit (smoothing filter processing unit) 54 inputs the cost volume integrated from the cost volume integration unit 53 and also uses a reference image among the color stereo images parallelized from the image parallelization processing unit 42. The color image Ir _{, L} is input, and edge retention type smoothing filter processing is performed on the input cost volume using the color image Ir _{, L} as a guide image for each cost map.
The cost volume filter processing unit 54 outputs the cost volume subjected to the smoothing filter processing to the parallax image generation processing unit 6.

  The smoothing filter processing for each cost map by the cost volume filter processing unit 54 is for improving the accuracy of parallax estimation near the contour of the subject as described above. In particular, for a subject that does not originally have a texture, the parallax is estimated mainly based on an infrared stereo image. Here, since the texture of the infrared pattern provided by the infrared pattern irradiator 2 is not necessarily a high-density texture, by performing edge holding type smoothing filter processing, the vicinity of the contour of the subject that does not originally have a texture This is particularly useful for improving the accuracy of parallax estimation.

As the edge holding type smoothing filter process, for example, the methods described in Reference Documents 2 to 4 can be used.
(Reference 2) J. Lu, K. Shi, D. Min, L. Lin and M. Do, “Cross-Based Local Multipoint Filtering”, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 430- 437, (2012)
(Reference 3) K. He, J. Sun, X. Tang, “Guided Image Filtering,” In Proc. Of ECCV Part I, Pages 1-14, (2010)
(Reference 4) G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, K. Toyama, “Digital Photography with Flash and No-Flash Image Pairs,” In Proc. Of SIGGRAPH, (2004 )

Hereinafter, a case where the cost volume filter processing unit 54 according to the present embodiment performs edge holding type smoothing filter processing by CLMF (Cross-based Local Multipoint Filter) described in Reference Document 2 will be described as an example.
In the present embodiment, the cost volume smoothing filter processing by CLMF treats each cost map as a two-dimensional image with the cost as a pixel value, and performs two-dimensional spatial filter processing for each cost map. Further, when the smoothing filter processing is performed on the cost for each pixel with reference to the cost for the peripheral pixels of the pixel, the color image for the reference viewpoint is used as a guide image for identifying the edge region of the subject. Only the cost of a peripheral pixel that is similar in color to the target pixel of the smoothing filter process (the color difference is equal to or smaller than a predetermined threshold) is extracted and used for the smoothing filter process. An area where the parallax is greatly different corresponds to an edge portion of the subject, that is, an area where separate subjects are photographed adjacent to each other, and an edge ( This is used as reference information for maintaining the contour. By using only similar color pixels for the smoothing filter processing, there is an effect of suppressing the “region fattening phenomenon” that occurs when the cost volume is generated by calculating the cost using SSD as an evaluation formula.

Here, the smoothing filter processing by CLMF will be described with reference to FIG.
In FIG. 6, a kernel W _p that is a rectangular image region is a maximum region of peripheral pixels that are referred to when the cost for the center pixel p is subjected to the smoothing filter process. From this kernel W _p, the central pixel p and colors to extract the kernel Omega _p as image regions similar.

To extract the kernel Omega _p, first, the pixel lines arranged in the vertical direction (v direction) from the pixel p, sequentially for each of the vertical direction from the pixel p, determine whether the pixel p and the color is similar To do. At this time, the color images Ir _{, L} are used as a guide image for determination, and it is determined that the colors are similar when the color difference between the pixel p in the guide image and each peripheral pixel is equal to or less than a predetermined threshold.
Sequentially determining a pixel color from a pixel p is similar to the vertical direction, continued determination to the pixel color is not similar within the kernel W _p is detected. Finally, pixels having similar colors are _{defined as} the upper and lower ends of the kernel Ωp.

Next, in the horizontal direction (u-axis direction) from the pixel q (each pixel in the range indicated by the solid arrow line in FIG. 6) arranged in the vertical direction of the pixel p within the range from the upper end to the lower end. It is sequentially determined whether or not the color of the pixel q is similar for the pixels to be arranged. Sequentially determining a pixel color from the pixel q is similar to the left-right direction, continued determination to the pixel color is not similar within the kernel W _p is detected. Finally, the pixels having similar colors are _{defined as} the left end and the right end of the kernel Ωp at the position of the pixel q.
This, by performing for every pixel q, as indicated by a broken line in FIG. 6, it is possible to extract the kernel Omega _p.

  In CLMF, it is assumed that the image S after the filter process has a linear relationship with the guide image G. Therefore, the image S after the filter processing is expressed as shown in Expression (7) using the coefficients a and b.

In the formula (7), the subscript k denotes the respective pixels belonging to the kernel Omega _p, coefficient _a k, _{b k} is calculated by the equation (8).

In Expression (8), Ω k indicates a kernel (image region) extracted by the same procedure as that for extracting Ω p with the pixel k as the central pixel. Further, mu k, is σ k 2, Z 'k and epsilon, the average of the guide image G in kernel Omega k respectively, dispersion, average cost map a target image filtering in kernel Omega k, and the regularization term Show. | Ω k | indicates the number of pixels belonging to the kernel Ω k .

The image S _p the filtered _(i.e., the cost of post-filtering in pixel p) can be approximated as equation (9).

  The cost volume filter processing unit 54 performs the edge holding type smoothing filter processing for all cost maps of the cost volume according to the above procedure.

Returning to FIG. 5, the description of the configuration of the parallax image generating device 1A will be continued.
The parallax image generation processing unit 6 receives the cost volume that has been subjected to the smoothing filter processing from the cost volume filter processing unit 54, and generates a parallax image in which parallax is determined for each pixel using the input cost volume. Further, the parallax image generation processing unit 6 outputs the generated parallax image to the outside as an output of the parallax image generation device 1A.
Since the procedure of the parallax image generation processing by the parallax image generation processing unit 6 is the same as that of the parallax image generation processing unit 6 in the first embodiment, the description thereof is omitted.

[Operation of parallax image generation system]
Next, the operation of the parallax image generation system 100A according to the second embodiment will be described with reference to FIG.
In the parallax image generation by the parallax image generation system 100A according to the second embodiment, from the step S21 of irradiating the infrared pattern to the step S27 of integrating the cost volume, the parallax image generation system according to the first embodiment shown in FIG. Since this is the same as Step S11 to Step S17 in the parallax image generation by 100, description thereof will be omitted.

  After step S27, the parallax image generation system 100A uses, as the guide image, the color image for the reference viewpoint (left viewpoint) parallelized in step S24 for the cost volume integrated in step S27 by the cost volume filter processing unit 54. Then, an edge holding type smoothing filter process is performed for each cost map (step S28).

Then, the parallax image generation system 100A generates a parallax image by selecting the parallax that gives the minimum cost for each pixel for the cost volume that has been subjected to the smoothing filter process in step S28 by the parallax image generation processing unit 6. S29).
Through the above procedure, the parallax image generation system 100A can generate a parallax image.

  In the present embodiment, since the edge holding type smoothing filter process is performed on the cost volume before the parallax image generation process (step S29), the parallax estimation accuracy particularly in the vicinity of the contour of the subject can be improved. it can.

<Third Embodiment>
[Configuration of parallax image generation system]
Next, a parallax image generation system according to the third embodiment of the present invention will be described with reference to FIG. 8 and FIG.
As shown in FIG. 8, the parallax image generation system 100B according to the third embodiment is different from the parallax image generation system 100A according to the second embodiment shown in FIG. The provision differs from the provision of the parallax image generation device 1B instead of the parallax image generation device 1A. Further, the parallax image generation device 1B includes an image conversion unit 4B instead of the image conversion unit 4 and a cost volume generation unit 5B instead of the cost volume generation unit 5A with respect to the parallax image generation device 1A. Is different.

  The parallax image generation system 100B according to the third embodiment shoots an infrared stereo image as the imaging device 3B instead of the two infrared color cameras 31 and 32 in the imaging device 3 of the first embodiment and the second embodiment. Two infrared cameras 33 and 34 for photographing, and two color cameras 35 and 36 for photographing color stereo images. Infrared images and color images can be taken with the same optical axis, but instead of expensive infrared color cameras 31 and 32, a camera for taking infrared images and a camera for taking color images are separately provided, so that it can be taken at low cost. The apparatus 3B can be configured.

  In the present embodiment, since the infrared image and the color image are captured by separate cameras having different optical axes, the light of the color image captured by the color camera 35 and the color image captured by the color camera 36 is captured. In addition to collimating the axes, it is necessary to convert the image coordinates so that the infrared image captured by the infrared camera 33 corresponding to each color image and the infrared image captured by the infrared camera 34 are aligned with the optical principal point. is there. For this purpose, an image coordinate conversion processing unit 43 and an image coordinate conversion processing unit 44 are provided in place of the image parallelization processing unit 42, and a color image captured by the color camera 35 is converted into an infrared ray by the image coordinate conversion processing unit 43. The optical axis is aligned with the infrared image, which is the reference image captured by the camera 33 and parallelized, and the color image captured by the color camera 36 is captured by the infrared camera 34 and parallelized by the image coordinate conversion processing unit 44. The other infrared image is aligned with the optical axis.

A set of color images output from the image coordinate conversion processing unit 43 and the image coordinate conversion processing unit 44 is a color stereo image that has been parallelized and coordinate-converted so that the optical axis coincides with the infrared stereo image. However, the image coordinate conversion process that involves parallelization and movement of the optical principal point depends on the distance in the depth direction of the subject (or the parallax on the infrared stereo image of the conversion destination), and therefore the color camera 35 and the color camera 36 The image coordinate conversion of the captured color image must be performed for each assumed parallax, assuming that the parallax on the infrared stereo image is uniformly assumed.
In the present embodiment, in order to calculate the cost volume using the color image obtained by image coordinate conversion for each parallax, the cost volume calculation unit 52B is used instead of the cost volume calculation unit 52 in the first embodiment and the second embodiment. Is provided.

Since other configurations are the same as those of the parallax image generation system 100A according to the second embodiment, the same components are denoted by the same reference numerals, and description thereof will be omitted as appropriate.
Hereinafter, a configuration different from the second embodiment will be mainly described.

As described above, the photographing apparatus 3B includes the two infrared cameras 33 and 34 and the two color cameras 35 and 36.
As shown in FIG. 2B, the two infrared cameras 33 and 34 are juxtaposed in the horizontal direction (X-axis direction) with the infrared pattern irradiator 2 interposed therebetween. Further, the two color cameras 35 and 36 are respectively arranged in the upward direction (Y-axis direction) of the infrared cameras 33 and 34 that capture images of viewpoints corresponding to the X-axis direction. The infrared cameras 33 and 34 and the color cameras 35 and 36 are preferably arranged so that their optical axes are substantially parallel to each other. Further, it is preferable that the infrared camera 33 and the color camera 35 and the infrared camera 34 and the color camera 36 are arranged in the vicinity so as to be photographed from viewpoints as close as possible. As a result, the amount of conversion by which the coordinate conversion is performed by the image parallelization processing unit 41 and the image coordinate conversion processing units 43 and 44 is reduced, so that an image in which the optical axes are aligned can be calculated with high accuracy.

Moreover, it is preferable that the color cameras 35 and 36 are provided with the infrared cut filter which cuts the infrared rays of the wavelength which the infrared pattern irradiation machine 2 irradiates. As a result, a color image can be taken without being affected by infrared pattern irradiation.
Furthermore, in the present embodiment, a color image composed of three RGB channels is captured as an image in the visible light region by the color cameras 35 and 36, but a monochrome image in the visible light region, a two-channel image, or 4 You may make it image | photograph the image more than a channel.

  The infrared cameras 33 and 34 output the captured infrared stereo image to the image parallelization processing unit 41. The color camera 35 outputs the captured color image to the image coordinate conversion processing unit 43, and the color camera 36 outputs the captured color image to the image coordinate conversion processing unit 44.

  The parallax image generation device 1B includes an image conversion unit 4B, a cost volume generation unit 5B, and a parallax image generation processing unit 6, and includes an infrared stereo image and a visible light stereo image, and an infrared camera 33 that captures these images. 34 and the camera parameters of the color cameras 35 and 36 are used to generate a parallax image.

The image conversion unit 4B includes an image parallelization processing unit 41 and image coordinate conversion processing units 43 and 44.
Since the image parallelization processing unit 41 is the same as that of the second embodiment, description thereof is omitted.
The image coordinate conversion processing unit 43 inputs a color image from the color camera 35 and inputs camera parameters for the color camera 35 and camera parameters for the infrared camera 33. The image coordinate conversion processing unit 43 photographed the input color image by the infrared camera 33 that is parallelized by the image parallelization processing unit 41 using the camera parameters of the color camera 35 and the camera parameters of the infrared camera 33. Convert to the image coordinate system of the same optical axis as the infrared image. The image coordinate conversion processing unit 43 outputs the color image obtained by converting the image coordinates to the cost volume calculation unit 52B as the color image of the left viewpoint that is the reference viewpoint. Furthermore, the image coordinate conversion processing unit 43 outputs the color image obtained by converting the image coordinates to the cost volume filter processing unit 54 as a guide image for smoothing filter processing.

The method for acquiring each camera parameter is the same as in the first embodiment described above, and detailed description thereof is omitted. In this embodiment, since the infrared camera 33 and the color camera 35 have different optical axes, camera calibration is performed for each camera to acquire camera parameters. Here, for the infrared camera 33, camera parameters can be acquired by conventional camera calibration using an image analysis method by handling the captured infrared image as a monochrome image.
The same applies to the camera parameters of the infrared camera 34 and the color camera 36.

  The image coordinate conversion processing unit 44 inputs a color image from the color camera 36 and inputs camera parameters for the color camera 36 and camera parameters for the infrared camera 34. The image coordinate conversion processing unit 44 photographed the input color image by the infrared camera 34 that is parallelized by the image parallelization processing unit 41 using the camera parameters of the color camera 36 and the camera parameters of the infrared camera 34. Convert to the image coordinate system of the same optical axis as the infrared image. The image coordinate conversion processing unit 44 outputs the color image obtained by converting the image coordinates to the cost volume calculation unit 52B as a right viewpoint color image.

Here, the image coordinate conversion processing by the image coordinate conversion processing unit 43 will be described.
The image coordinate conversion processing unit 43 performs coordinate conversion of the color image so that the color image captured by the color camera 35 can be handled in the same coordinate system as the infrared image captured by the infrared camera 33. Here, since the conversion formula depends on the parallax d, a color image corresponding to all the parallaxes d is calculated for each parallax d.
Note that all the parallaxes d are the same as the range and interval of the parallax d for calculating the cost map in the description of the first embodiment. That is, the parallax d is assumed every predetermined interval (for example, “1”) from the minimum value “0” to the maximum value “N” that can be taken as parallax in the parallax image generated by the parallax image generation device 1B. Shall be.

When the image coordinates (u _{r, v r)} of the position p _r of the collimated infrared image for the left viewpoint is the reference viewpoint disparity in the d, homogeneous coordinates of the position p _r is calculated by Equation (10) be able to.

In Expression (10), f represents the focal length in units of pixel pitch of the camera lenses of the two infrared cameras 33 and 34, and B represents the distance between the optical principal points of the two infrared cameras 33 and 34. , (C _u , C _v ) indicate the image coordinates of the center point of the infrared image. In each parameter, the subscript r indicates a parameter for the parallelized image. Further, “T” on the right shoulder indicates a transposed matrix.

Here, the internal parameter matrix F, the rotation matrix R, when the translation vector T, the world coordinates of the point P on the object photographed in the position p _r in the infrared image is represented by the equation (11) . Further, the image coordinates (u _c , v _c ) in the color image corresponding to the point P on the subject are expressed as in Expression (12). In equation (12), the subscript c indicates a parameter for a color image.

For each pixel of the infrared image at the parallel reference viewpoint, the parallax is calculated by calculating the image coordinates (u _c , v _c ) of the corresponding pixel in the color image using the equations (11) and (12). It can be converted to a color image when d is assumed.
The image coordinate conversion processing unit 43 converts color images for all assumed parallaxes d.

  The image coordinate conversion processing unit 44 replaces the infrared image and color image of the left viewpoint with the camera parameters of the camera that captured these images, and the infrared image and color image of the right viewpoint and the camera that captured these images. Using the camera parameters, a color image converted to the coordinate plane of the parallelized right viewpoint infrared image is calculated for all assumed parallaxes in the same procedure.

  The cost volume generation unit 5B includes cost volume calculation units 51 and 52B, a cost volume integration unit 53, and a cost volume filter processing unit 54.

The cost volume calculation unit 52B inputs the color image of the left viewpoint from the image coordinate conversion processing unit 43 and the color image of the right viewpoint from the image coordinate conversion processing unit 44, respectively, and a color stereo image including a set of these color images. Is calculated for each parallax d, a cost volume that is an array of these cost maps is calculated.
The cost volume calculation unit 52B outputs the cost volume for the calculated color stereo image to the cost volume integration unit 53.

  Since the color stereo image input to the cost volume calculation unit 52B is generated for each parallax d, when calculating the cost map, the color stereo image of the corresponding parallax d is used for each pixel. Calculate the cost.

  The cost volume filter processing unit 54 inputs the cost volume integrated from the cost volume integration unit 53 and is converted from the image coordinate conversion processing unit 43 into an image of the same optical axis as the parallel infrared image of the reference viewpoint. A color image is input as a guide image. Then, the cost volume filter processing unit 54 performs the edge holding type smoothing filter processing on the cost volume in the same manner as the cost volume filter processing unit 54 in the second embodiment described above, and the filtered cost volume is obtained. The data is output to the parallax image generation processing unit 6.

Since other components are the same as those of the parallax image generation system 100 according to the second embodiment, description thereof will be omitted.
In the present embodiment, as in the first embodiment, the parallax image generation device 1B does not include the cost volume filter processing unit 54, and uses the cost volume calculated by the cost volume integration unit 53 to generate the parallax image. You may comprise so that a parallax image may be produced | generated by the process part 6. FIG.

[Operation of parallax image generation system]
Next, the operation of the parallax image generation system 100B according to the third embodiment will be described with reference to FIG. 9 (see FIG. 8 as appropriate).
Step S31 of irradiating the infrared pattern is the same as step S11 (see FIG. 4) of the first embodiment, and thus description thereof is omitted.

Next, the parallax image generation system 100B captures an infrared stereo image with the two infrared cameras 33 and 34 and also captures a color stereo image with the two color cameras 35 and 36 (step S32).
Next, in the parallax image generation system 100B, the image parallelization processing unit 41 parallelizes the infrared stereo image captured in step S32 (step S33). Further, the parallax image generation system 100B matches the optical axis of the infrared image at the reference viewpoint (left viewpoint) obtained by parallelizing the color image captured by the color camera 35 in step S32 by the image coordinate conversion processing unit 43 in step S33. In this way, the image coordinate conversion process is performed, and the image coordinate conversion processing unit 44 matches the color image captured by the color camera 36 in step S32 with the optical axis of the right viewpoint infrared image parallelized in step S33. Image coordinate conversion processing is performed (step S34). Note that either step S33 or step S34 may be executed first or in parallel.

  Next, the parallax image generation system 100B calculates a cost volume for the infrared stereo image parallelized in step S33 by the cost volume calculation unit 51 (step S35). Further, the parallax image generation system 100B calculates a cost volume for the color stereo image whose image coordinates are converted in step S34 by the cost volume calculation unit 52B (step S36). Note that either step S35 or step S36 may be executed first or in parallel.

  Next, the parallax image generation system 100B uses the cost volume integration unit 53 to calculate the cost volume for the infrared stereo image calculated in step S35 and the cost volume for the color stereo image calculated in step S36 using Equation (6). Is used to add the weights for each parallax and each pixel with a weight, thereby integrating them into one cost volume (step S37).

  Next, the parallax image generation system 100B uses, as a guide image, a color image for the reference viewpoint (left viewpoint) obtained by converting the image coordinates in step S34 for the cost volume integrated in step S37 by the cost volume filter processing unit 54. Edge-preserving smoothing filter processing is performed for each parallax d, that is, for each cost map (step S38).

Then, the parallax image generation system 100B generates a parallax image by selecting the parallax that gives the minimum cost for each pixel for the cost volume smoothed by the parallax image generation processing unit 6 in step S38 (step S38). S39).
Through the above procedure, the parallax image generation system 100B can generate a parallax image.

  In the present embodiment, the image coordinate conversion processing units 43 and 44 convert the image coordinates of the color stereo image captured by the color cameras 35 and 36 so as to coincide with the optical axis of the parallelized infrared stereo image. A parallax image can be generated from an infrared image and a color image captured using a simple infrared camera and a color camera.

<Fourth embodiment>
[Configuration of parallax image generation system]
Next, a parallax image generation system according to the fourth embodiment of the present invention will be described with reference to FIG. 10 and FIG.
As shown in FIG. 10, the parallax image generation system 100C according to the fourth embodiment is different from the parallax image generation system 100B according to the third embodiment shown in FIG. The provision differs from the provision of the parallax image generation device 1C instead of the parallax image generation device 1B. Further, the parallax image generation device 1C includes an image conversion unit 4C instead of the image conversion unit 4B and a cost volume generation unit 5C instead of the cost volume generation unit 5B with respect to the parallax image generation device 1B. Is different.

  A parallax image generation system 100C according to the fourth embodiment includes two infrared cameras 33 and 34 for capturing an infrared stereo image, and one color camera 35 for capturing a color image as an imaging device 3C. It has. In the present embodiment, the cost volume is calculated only from the infrared stereo image. Then, using the color image coordinate-converted to the image of the same optical axis as the parallel infrared image of the reference viewpoint as a guide image, the cost volume calculated for the infrared stereo image is subjected to edge holding type smoothing filter processing, A parallax image is generated using the cost volume subjected to the smoothing filter process.

For this reason, the parallax image generation system 100C according to the present embodiment does not include the color camera 36 for the right viewpoint in the photographing device 3C as compared to the parallax image generation system 100B according to the third embodiment illustrated in FIG. The image conversion unit 4C of the parallax image generation device 1C does not include the image coordinate conversion processing unit 44 for the right viewpoint, and the cost volume generation unit 5C of the parallax image generation device 1C calculates the cost volume for the color stereo image. The difference is that the cost volume calculation unit 52B and the cost volume integration unit 53 are not provided.
The same components as those in the third embodiment are denoted by the same reference numerals, and the description thereof is omitted as appropriate.

As described above, the photographing apparatus 3C includes two infrared cameras 33 and 34 and one color camera 35.
As shown in FIG. 2C, the two infrared cameras 33 and 34 are juxtaposed in the horizontal direction (X-axis direction) with the infrared pattern irradiator 2 and one color camera 35 interposed therebetween.
The infrared pattern irradiator 2 is preferably disposed between the two infrared cameras 33 and 34. As a result, when viewed from either of the two infrared cameras 33 and 34, the infrared pattern irradiated from the infrared pattern irradiator 2 toward the subject is a shadow of a part of the subject and is not projected, that is, the texture. It is possible to reduce a region where no is given.
The color camera 35 is also preferably disposed between the infrared cameras 33 and 34. Furthermore, it is more preferable that the color camera 35 is disposed near the infrared camera 33 that captures the reference image.
Since the color image taken by the color camera 35 is used as a guide image when the cost volume of the infrared stereo image is subjected to the edge holding type smoothing filter processing, the color image is arranged so that the viewpoint is close to the infrared cameras 33 and 34. Thus, a color image combining the infrared image of the reference viewpoint and the optical axis can be calculated with high accuracy.

  The infrared cameras 33 and 34 output the captured infrared stereo image to the image parallelization processing unit 41. The color camera 35 outputs the captured color image to the image coordinate conversion processing unit 43.

  The parallax image generation device 1 </ b> C includes an image conversion unit 4 </ b> C, a cost volume generation unit 5 </ b> C, and a parallax image generation processing unit 6, and an infrared stereo image and a visible light image, and infrared cameras 33 and 34 that capture these images. And the camera parameters of the color camera 35 are used to generate a parallax image.

The image conversion unit 4 </ b> C includes an image parallelization processing unit 41 and an image coordinate conversion processing unit 43.
The image parallelization processing unit 41 is the same as that in the second embodiment and the third embodiment, and a description thereof will be omitted.
As in the third embodiment, the image coordinate conversion processing unit 43 inputs a color image from the color camera 35, and inputs and inputs camera parameters for the color camera 35 and camera parameters for the infrared camera 33. An image coordinate system having the same optical axis as that of the infrared image captured by the infrared camera 33 parallelized by the image parallelization processing unit 41 using the camera parameters of the color camera 35 and the camera parameters of the infrared camera 33. Convert to The image coordinate conversion processing unit 43 outputs the color image obtained by converting the image coordinates to the cost volume filter processing unit 54 as a guide image for smoothing filter processing.

  The cost volume calculation unit 51 inputs the parallel infrared stereo image from the image parallelization processing unit 41 and calculates the cost volume for the infrared stereo image. The cost volume calculation unit 51 outputs the calculated cost volume to the cost volume filter processing unit 54.

  The cost volume filter processing unit 54 inputs the cost volume for the infrared stereo image from the cost volume calculation unit 51 and coordinates the image with the same optical axis as the reference viewpoint infrared image parallelized from the image coordinate conversion processing unit 43. The converted color image is input as a guide image. Then, the cost volume filter processing unit 54 performs the edge holding type smoothing filter processing on the cost volume in the same manner as the cost volume filter processing unit 54 in the second embodiment and the third embodiment described above. The processed cost volume is output to the parallax image generation processing unit 6.

  The other components are the same as those of the parallax image generation system 100B according to the third embodiment, and thus description thereof is omitted.

[Operation of parallax image generation system]
Next, the operation of the parallax image generation system 100C according to the fourth embodiment will be described with reference to FIG. 11 (see FIG. 10 as appropriate).
Step S41 for irradiating the infrared pattern is the same as step S11 (see FIG. 4) of the first embodiment, and a description thereof will be omitted.

Next, the parallax image generation system 100C captures an infrared stereo image with the two infrared cameras 33 and 34 and also captures a color image with the single color camera 35 (step S42).
Next, in the parallax image generation system 100C, the image parallelization processing unit 41 parallelizes the infrared stereo image captured in step S42 (step S43). In addition, the parallax image generation system 100C matches the optical axis of the infrared image at the reference viewpoint (left viewpoint) obtained by collimating the color image captured by the color camera 35 in step S42 with the image coordinate conversion processing unit 43 in step S43. In this way, image coordinate conversion processing is performed (step S44). Note that either step S43 or step S44 may be executed first or in parallel.

  Next, in the parallax image generation system 100C, the cost volume calculation unit 51 calculates the cost volume for the infrared stereo image parallelized in step S43 (step S45).

  Next, the parallax image generation system 100C uses the cost volume calculated by the cost volume filter processing unit 54 in step S45 to convert the cost volume into an image of the reference viewpoint (left viewpoint) in step S44, and uses the cost map as a guide image. An edge-holding type smoothing filter process is performed every time (step S46).

Then, the parallax image generation system 100C generates a parallax image by selecting the parallax that gives the minimum cost for each pixel for the cost volume smoothed by the parallax image generation processing unit 6 in step S46 (step S46). S47).
Through the above procedure, the parallax image generation system 100C can generate a parallax image.

  In this embodiment, the texture of the infrared pattern is given to the subject by the infrared pattern illuminator 2, and the parallax image is generated based on the cost volume for the infrared stereo image obtained by photographing the subject to which the texture is given. It is possible to accurately estimate the parallax even for a subject that does not exist. In addition, since the cost volume is subjected to an edge-holding smoothing filter process using a color image as a guide image, the parallax can be accurately estimated even for an area that originally has a texture.

<Fifth Embodiment>
[Configuration of parallax image generation system]
Next, with reference to FIG. 12, the parallax image system which concerns on 5th Embodiment of this invention is demonstrated.
A parallax image generation system 100D according to the fifth embodiment shown in FIG. 12 includes a parallax image generation device 1D instead of the parallax image generation device 1A in the parallax image generation system 100A according to the second embodiment shown in FIG. Is. The parallax image generation device 1D includes a cost volume generation unit 5D having a cost volume filter processing unit 54D instead of the cost volume generation unit 5A having the cost volume filter processing unit 54 in the parallax image generation device 1A. is there. Since the other configuration of the parallax image generation system 100D according to the fifth embodiment is the same as that of the parallax image generation system 100A according to the second embodiment, the same reference numerals are given and description thereof is omitted.

The cost volume filter processing unit 54 (see FIG. 5) in the second embodiment smoothes the cost using a kernel Ω _p (see FIG. 6) having a spread in a two-dimensional space. Here, if parallax is estimated for each parallax image, that is, for each frame, with respect to a parallax image composed of a plurality of parallax images (frames) continuous in time series, the continuity of parallax in the time axis direction is not considered. . For this reason, if smoothing processing is performed for each frame and a plurality of time-sequential parallax images are continuously reproduced, flicker may occur in the parallax image. Therefore, the cost volume filter processing unit 54D in the present embodiment performs a smoothing filter process over a three-dimensional space-time by adding a time axis to the two-dimensional space. Thereby, when a plurality of parallax images are continuously reproduced, the occurrence of flicker can be suppressed, and in particular, the depth estimation accuracy for a stationary object can be improved.

Specifically, the cost volume filter processing unit 54D in the present embodiment performs smoothing filter processing using a kernel having a two-dimensional space spread by the cost volume filter processing unit 54 in the second embodiment in the time axis direction. In this way, smoothing filter processing using a kernel having a spread in a three-dimensional space-time is performed.
In the present embodiment, the image conversion unit 4, and the cost volume calculation units 51 and 52 and the cost volume integration unit 53 of the cost volume generation unit 5D perform the processing described in the second embodiment for each frame. Shall. Then, the cost volume filter processing unit 54D uses the cost volume and the guide image for a plurality of frames in a predetermined range in the time axis direction around the frame that is the target of the smoothing filter processing, and performs the smoothing filter processing. I do.

Hereinafter, the cost volume smoothing process performed by the cost volume filter processing unit 54D according to the present embodiment will be described in order with reference to FIGS.
First, edge holding type smoothing filter processing using a two-dimensional kernel will be described. This smoothing filter process is the same as the smoothing filter process performed by the cost volume filter processing unit 54 of the second embodiment. As described above, as the two-dimensional smoothing filter process, the methods described in Reference Document 2 to Reference Document 4 can be used. Here, the case where the CLMF described in Reference 2 is used as the two-dimensional smoothing filter processing using the adaptive kernel will be described again.

[Adaptive kernel]
As described above, the cost volume C _j for one frame output from the cost volume integration unit 53 in the second embodiment can be expressed as C _j (u _r , v _r , d). The cost volume C _j is represented by a three-dimensional array configured by collecting the two-dimensional cost maps CM _{jd of} each parallax d by the number d. In this embodiment, since the parallax is estimated using the cost volume for a plurality of frames that are continuous in time series, the cost volume C _j is 4 as C _j (u _r , v _r , d, t). It can be represented by a dimensional array. Here, t indicates time, and is indicated by time and frame number. For example, when t indicates a frame number, if the number of frames referred to for the smoothing filter process is F, t = 1 to F is assumed.
On the other hand, cost map _{CM jd} for each parallax d _{is, CM jd (u r, v} r, t) , and the represented by three-dimensional array. In addition, since the guide image uses a plurality of frames corresponding to the cost volume C _j , the smoothing filter processing kernel is also three-dimensional.

FIG. 13 shows an example of a two-dimensional kernel for smoothing filter processing. In FIG. 13, the x-axis indicates the horizontal direction, the y-axis direction indicates the vertical direction, and each lattice indicates a pixel. Note that the x-axis and the y-axis in FIG. 13 correspond to the u-axis and the v-axis in FIG. 6, respectively.
The kernel W _p that is a rectangular area is the largest pixel area that can be referred to when performing the smoothing filter process. From this kernel W _p, the pixel region center pixel p and colors are similar, are extracted as adaptive kernel Omega _p.
Further, in the example shown in FIG. 13, it is shown that the hue is different between the shaded area at the center and the other peripheral areas on the guide image. For example, the shaded area is a blue area, and the surrounding area that is not shaded is a yellow area.

Next, the procedure for extracting the adaptive kernel Omega _p for 1 frame.
(Procedure 1)
From the center pixel p, the arm is extended one pixel at a time in the upward direction (minus direction of the y-axis).
(Procedure 2)
When the difference between the pixel value of the tip of the arm and the pixel value of the center pixel becomes equal to or greater than a predetermined threshold, the extension of the arm is stopped and the length from the center pixel p is stored as _{Sp, 1} .
Note that the extension of the arm has a maximum range within the kernel W _p of a rectangle having a predetermined size (when expanded to a three-dimensional space-time, the kernel W _p is a rectangular parallelepiped). The same applies to other procedures described later.
(Procedure 3)
Similarly to the procedure 2, the arm is extended pixel by pixel in the downward direction (plus direction of the y-axis).

(Procedure 4)
When the difference between the pixel value at the tip of the arm and the pixel value of the center pixel p becomes equal to or greater than a predetermined threshold, the extension of the arm is stopped and the length from the center pixel p is stored as _{Sp, 3} .
(Procedure 5)
From a certain pixel (point) q on the line segment (pixel column) V (p) extending in the vertical direction (y-axis direction) represented by Expression (13), the left direction (minus direction of the x-axis) and the right direction Each arm is extended by one pixel in the positive direction of the x-axis.
Incidentally, _{x p} and _{y p,} respectively indicate the x and y coordinates of the central pixel p.

(Procedure 6)
When the difference between the pixel value at the tip of the arm and the pixel value of the center pixel p is equal to or greater than a predetermined threshold, the arm is stopped and the distance (length) from the line segment V (p) is set to S _{q. , 0} and S _{q, 2} . Thereby, a line segment (pixel column) H (q) extending in the horizontal direction (x-axis direction) represented by Expression (14) can be extracted.
Note that x _q and y _q indicate the x coordinate and the y coordinate of the pixel q, respectively.

(Procedure 7)
(Procedure 5) and (Procedure 6) are executed for all pixels q on the line segment V (p) to extract H (q). The kernel U (t) for one frame can be represented by the union of H (q) extracted for each pixel q as shown in Expression (15). Here, t indicates a frame number (time).
Note that the kernel Omega _p indicated by a broken line in FIG. 13 shows a pixel area extracted as U (t).

This process can be efficiently processed by an algorithm using Orthogonal Integral Image. This technique is described in detail in Reference 5.
(Reference 5)
K. Zhang, J. Lu, and G. Lafruit, “Cross-based local stereo matching using orthogonal integral images.” IEEE Transactions on Circuits and Systems for Video Technology, 19 (7): 1073-1079, July 2009.

[Adaptive space-time kernel]
In the present embodiment, this process is expanded in the time axis (t-axis) direction to generate a three-dimensional kernel having a spread in a three-dimensional space-time and used for the smoothing filter process. FIG. 14 is a cross-sectional view of a three-dimensional space-time in which guide images for a plurality of frames are arranged, cut along an xt plane. However, in FIG. 13, the horizontal direction is the x-axis and the vertical direction is the t-axis (time axis). Note that the frame number to which a number is assigned in time series corresponds to the coordinate value of the t-axis.
At this time, three-dimensional kernel Omega _p can be generated by performing the processing (step 10) from below (step 8).

(Procedure 8)
From the center pixel p, the arm is extended by one frame each in the forward direction (minus direction of the t-axis) and the backward direction (plus direction of the t-axis) in the time axis direction.
(Procedure 9)
When the difference between the pixel value of the tip of the arm and the pixel value of the center pixel p is equal to or greater than a predetermined threshold, the extension of the arm is stopped, and the length from the center pixel p is set to S _{q, 4} and S _{q, 5} , respectively. Save as.
As a result, a line segment (pixel column) T (p) extending in the time axis (t-axis) direction represented by Expression (16) can be extracted.

(Procedure 10)
For all the pixels t on the line segment T (p), for each frame including each pixel t, (procedure 1) to (procedure 7) are executed, and the kernel U for each frame shown in Expression (15) is obtained. Extract (t). Then, the three-dimensional kernel Y (p) expanded in space-time can be represented by the union of U (t) extracted for each frame t as shown in Expression (17).
Note that the kernel Omega _p indicated by a broken line in FIG. 14, Y of the pixel area extracted as (p), shows a cross section taken along the x-t plane.

[CLMF]
As described above, in CLMF, smoothed image S, as shown in equation (18), the guide image G _p and the two coefficients a, are assumed to be represented by a linear transformation by and b.

In the equation (18), the subscript k indicates a pixel belonging to the kernel Ω _p , and the coefficients a _k and b _k are constant in the kernel Ω _k centered on the pixel k, and using a linear regression method, It can be calculated by equation (19).

In Expression (19), Z is an image to be smoothed, and corresponds to a cost map in this embodiment. Further, the ^{_{μ k, σ k 2, Z}} 'k, and epsilon, the average of the guide image in the kernel Omega _k respectively, dispersion, average cost map in kernel Omega _k, and a regularization term. | Ω _k | indicates the number of pixels belonging to the kernel Ω _k .

In this case, smoothed image S _p can be approximated by the weighted sum of the S _k defined by formula (20).

  The cost volume filter processing unit 54D performs the three-dimensional space-time smoothing filter processing for each parallax for all cost maps of the cost volume by the above procedure. The cost volume filter processing unit 54 </ b> D outputs the smoothed cost volume to the parallax image generation processing unit 6. Also, the parallax image generation unit 6 receives the smoothed cost volume from the cost volume filter processing unit 54D, and generates a parallax image by selecting the parallax with the lowest cost for each pixel for this cost volume. .

[Operation of parallax image generation stem]
The parallax image generation system 100D according to the present embodiment inputs an infrared image and a color image captured at a predetermined frame frequency by the imaging device 3 to the parallax image generation device 1D. The frame frequency is not particularly limited, but can be, for example, about 30 to 240 Hz.
At this time, the frame of the infrared image and the frame of the color image are input in synchronization with the parallax image generation device 1D. Then, as described above, the parallax image generation device 1D performs the same processing as described in the second embodiment for each frame by the image conversion unit 4, the cost volume calculation units 51 and 52, and the cost volume integration unit 53. . In addition, the parallax image generation device 1D uses the cost volume filter processing unit 54D to center the frame that is the target of the smoothing filter process, and the cost volume for a plurality of frames in a predetermined range in the vicinity in the time axis direction. And the smoothing filter process of the above-mentioned procedure is performed using the guide image. Other operations of the parallax image generation system 100D according to the present embodiment are the same as those of the parallax image generation system 100A according to the second embodiment shown in FIG.

Also, in the third embodiment shown in FIG. 8 and the fourth embodiment shown in FIG. 10, the kernel is changed by replacing the cost volume filter processing unit 54 with the cost volume filter processing unit 54D in the fifth embodiment. Smoothing filter processing extended to a three-dimensional space-time consisting of a two-dimensional space and a time axis can be applied.
In these embodiments, by applying a smoothing filter process over a three-dimensional space-time, the occurrence of flicker can be suppressed when a plurality of parallax images are continuously reproduced, as in the fifth embodiment. At the same time, it is possible to improve the depth estimation accuracy especially for stationary objects.

  As described above, the parallax image generation systems 100, 100 A, 100 B, 100 C, and 100 D according to the embodiments are used for an infrared stereo image, a color stereo image, or a color image of a subject that is textured by the infrared pattern illuminator 2. Can be used to generate a parallax image with high accuracy by the parallax image generation devices 1, 1 </ b> A, 1 </ b> B, 1 </ b> C, and 1 </ b> D. The generated parallax image can be used, for example, for generating a three-dimensional model. Furthermore, the three-dimensional model can be used to generate, for example, an integral photography stereoscopic image.

Note that, in the parallax image generation systems 100, 100A, 100B, 100C, and 100D according to the embodiments of the present invention, image conversion of the parallax image generation devices 1, 1A, 1B, 1C, and 1D that performs arithmetic processing on infrared images and color images. Each component of the units 4, 4B, 4C, the cost volume generators 5, 5A, 5B, 5C, 5D and the parallax image generation processor 6 is a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or a dedicated unit. This hardware circuit can be used.
In addition, the parallax image generation devices 1, 1A, 1B, 1C, and 1D in each embodiment of the present invention are connected to an infrared pattern irradiator 2, a photographing device 3, and the like, and a CPU (Central Processing Unit) and storage means (for example, A computer having hardware resources such as memory, hard disk) and various signal input / output means, the above-described image conversion units 4, 4B, 4C, cost volume generation units 5, 5A, 5B, 5C, 5D and parallax image generation processing It can also be realized by a parallax image generation program for causing each component means of the unit 6 to perform a cooperative operation. This program may be distributed through a communication line, or may be distributed by writing in a recording medium such as an optical disk, a magnetic disk, or a flash memory.

  As described above, the parallax image generation system according to the embodiment of the present invention has been specifically described by the mode for carrying out the invention. However, the gist of the present invention is not limited to these descriptions, and the scope of the claims Should be interpreted broadly based on the description. Needless to say, various changes and modifications based on these descriptions are also included in the spirit of the present invention.

1, 1A, 1B, 1C, 1D Parallax image generation device 2 Infrared pattern irradiator 3, 3B, 3C Imaging device 31, 32 Infrared color camera (camera)
33, 34 Infrared camera 35, 36 Color camera (visible light camera)
4, 4B, 4C Image conversion unit 41, 42 Image parallelization processing unit 43, 44 Image coordinate conversion processing unit 5, 5A, 5B, 5C, 5D Cost volume generation unit (correspondence map group generation unit)
51, 52, 52B Cost volume calculation unit 53 Cost volume integration unit 54, 54D Cost volume filter processing unit (smoothing filter processing unit)
6 Parallax image generation processing unit 100, 100A, 100B, 100C, 100D Parallax image generation system

Claims (6)

A set of infrared images and a set of visible light images obtained by photographing an infrared image that is an image in the infrared wavelength region and a visible light image that is an image in the visible light wavelength region with two cameras. A parallax image generation device that generates a parallax image using
Using the camera parameters for the two cameras, the set of infrared images is converted into a set of parallelized infrared images, which are images obtained when the optical axes of the two cameras are parallel, Using the camera parameters for the two cameras, the set of visible light images is converted into a set of collimated visible light images that are images obtained when the optical axes of the two cameras are parallel. An image converter;
When the parallax between the reference infrared image that is the reference parallelized infrared image and the non-reference infrared image that is the other parallelized infrared image is a constant value in the entire image, the reference infrared image And the non-reference infrared image for each pixel, the correspondence level, which is an index indicating the degree of coincidence of an image in a predetermined range including the pixel, and the collimated visible light image having the same optical axis as the reference infrared image When the parallax between the reference visible light image and the non-reference visible light image that is the other collimated visible light image is the constant value in the entire image, the reference visible light image and the non-reference visible light A correspondence map, which is a two-dimensional array of correspondence obtained by integrating the correspondence, which is an index indicating the degree of coincidence of an image in a predetermined range including the pixel, with respect to each pixel of the light image, for a predetermined parallax range At predetermined intervals A corresponding level map group generation unit for generating a corresponding degree map group by obtaining each difference,
Parallax image generation processing for generating a parallax image that is an image in which the parallax is determined for each pixel by selecting the parallax for the correspondence map having the highest degree of matching in the correspondence map group for each pixel And
A parallax image generating device comprising: For a subject on which an infrared pattern is projected, a pair of infrared images obtained by photographing an infrared image that is an image in the infrared wavelength region with two infrared cameras, and two visible light images that are images in the wavelength region of visible light. A parallax image generation device that generates a parallax image using a set of visible light images captured by a visible light camera,
Using the camera parameters for the two infrared cameras, the set of infrared images is converted into a set of parallelized infrared images that are images obtained when the optical axes of the two infrared cameras are parallel. In addition, using the camera parameters of the two visible light cameras and the two infrared cameras, the set of the visible light images, the optical axis of the two visible light cameras, and the set of the collimated infrared images An image conversion unit for converting into a set of collimated visible light images, which is an image obtained when the same as the optical axis for obtaining
When the parallax between the reference infrared image that is the reference parallelized infrared image and the non-reference infrared image that is the other parallelized infrared image is a constant value in the entire image, the reference infrared image And the non-reference infrared image for each pixel, the correspondence level, which is an index indicating the degree of coincidence of an image in a predetermined range including the pixel, and the collimated visible light image having the same optical axis as the reference infrared image When the parallax between the reference visible light image and the non-reference visible light image that is the other collimated visible light image is the constant value in the entire image, the reference visible light image and the non-reference visible light A correspondence map, which is a two-dimensional array of correspondence obtained by integrating the correspondence, which is an index indicating the degree of coincidence of an image in a predetermined range including the pixel, with respect to each pixel of the light image, for a predetermined parallax range At predetermined intervals A corresponding level map group generation unit for generating a corresponding degree map group by obtaining each difference,
For each pixel, a parallax image generation processing unit that generates a parallax image that is an image in which the parallax is determined for each pixel by selecting the parallax for the correspondence map having the highest correspondence in the correspondence map group When,
A parallax image generating device comprising: The correspondence map group generation unit uses the reference visible light image as a guide image for edge region identification for each correspondence map for the correspondence map group, and corresponds to the edge of the subject in the guide image. A smoothing filter processing unit that performs a smoothing filter process that holds an edge of the correspondence map;
The said parallax image generation process part produces | generates the said parallax image using the correspondence map group smoothed by the said smoothing filter process part, The Claim 1 or Claim 2 characterized by the above-mentioned. A parallax image generating device. For a subject on which an infrared pattern is projected, a set of infrared images obtained by photographing the infrared image that is an image in the infrared wavelength region with two infrared cameras, and a visible light image that is an image in the visible light wavelength region A parallax image generating device that generates a parallax image using a visible light image captured in
Using the camera parameters for the two infrared cameras, the set of infrared images is converted into a set of parallelized infrared images that are images obtained when the optical axes of the infrared cameras are parallel, and Using the camera parameters for the visible light camera and the reference infrared camera, the visible light image is the same as the optical axis for obtaining the collimated infrared image based on the optical axis of the visible light camera. An image conversion unit for converting to a reference visible light image which is an image obtained at the time,
When the parallax between the reference infrared image that is the reference parallelized infrared image and the non-reference infrared image that is the other parallelized infrared image is a constant value in the entire image, the reference infrared image A correspondence map, which is a two-dimensional array of correspondence, which is an index indicating the degree of coincidence of an image of a predetermined range including the pixel for each pixel of the non-reference infrared image and a predetermined interval for a predetermined parallax range. A correspondence map group generation unit that generates a correspondence map group by obtaining each parallax;
For the correspondence map group, for each correspondence map, the reference visible light image is used as a guide image for edge region identification, and the smoothness of the correspondence map corresponding to the edge of the subject in the guide image is retained. A smoothing filter processing unit for performing the smoothing filter processing;
A parallax image that is an image in which the parallax is determined for each pixel is generated by selecting the parallax for the correspondence map having the highest degree of correspondence in the correspondence map group that has been subjected to the smoothing filter process. A parallax image generation processing unit,
A parallax image generating device comprising:   The smoothing filter processing unit extracts peripheral pixels whose color difference from the target pixel of the smoothing filter processing is a predetermined threshold or less with reference to the guide image, and the correspondence degree of the extracted peripheral pixels 5. The parallax image generation device according to claim 3 or 4, wherein smoothing filter processing is performed using. The infrared image and the visible light image are taken at a predetermined frame frequency,
The correspondence map group generation unit generates the correspondence map group for each frame,
The smoothing filter processing unit, for the correspondence map group, the correspondence map group and the guide image for a plurality of frames in the vicinity of a frame corresponding to the correspondence map group that is a target of the smoothing filter processing. The parallax image generating apparatus according to claim 3, wherein the smoothing filter process is performed on a three-dimensional space-time including a two-dimensional space and a time axis.

Images