JP2008216127A - Distance image generation device, distance image generation method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a distance image generation device, a distance image generation method, and a program capable of generating a distance image, having high reliability by acquiring a high-precision operation result in an operation of a subpixel level, while reducing the total processing time. <P>SOLUTION: The distance image generation device includes an image acquisition means for acquiring first image information and second image information to be a comparison object; a first calculation part 15 for comparing a standard image and a reference image by an SAD calculation method to calculate a parallax value in a pixel level; a second calculation part 16 for comparing the standard image and the reference image by a POC calculation method having higher accuracy than that of the SAD calculation method to calculate the parallax value in the subpixel level; and an object area setting part 14 for setting the object area for performing the calculation by the second calculation part 16, on the basis of the calculation result of the parallax value by the first calculation part 15. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a distance image generation device, a distance image generation method, and a program, and more particularly, to a distance image generation device, a distance image generation method, and a program that generate a distance image by capturing a plurality of images of the same object.

  Conventionally, the same object (subject) is imaged from different positions by a plurality of imaging means to obtain a plurality of pieces of image information, and correlation by an SAD (Sum of Absolute Difference) calculation method, an SSD (Sum of Squared Difference) calculation method, or the like. An apparatus for calculating a correlation degree of the image information by performing an operation, obtaining a parallax value for the same object based on the degree of correlation, obtaining a position (distance value) of the object from the parallax value, and generating a distance image Are known.

Further, in such an apparatus, first, a city block distance is calculated for each small region for a plurality of images at a pixel level (pixel level) to obtain a correlation with each other, and a pixel shift that occurs according to the distance to the object. A distance image obtained by imaging (= parallax) as distance data is generated, stereo matching is performed using data of the reference image and the comparison image, and a disparity (subpixel component) of one pixel or less is obtained to obtain a distance image. A technique for interpolating parallax obtained in units of pixels with a resolution of 1 pixel or less is known (see, for example, Patent Document 1).
JP 2000-283755 A

However, although the technique described in Patent Document 1 calculates a parallax using a correlation calculation method using a block such as an SAD calculation method, the calculation method such as the SAD calculation method has high-speed calculation processing. On the other hand, it is difficult to calculate the parallax value with high accuracy in the sub-pixel level calculation, and an accurate distance image cannot be obtained.
When sub-pixel level calculation processing is performed by a correlation calculation method using blocks such as the SAD calculation method, as disclosed in Patent Document 1, the degree of correlation between blocks (city block distance) is correlated. The calculation is performed under the assumption (linear approximation) that the image is symmetric about the minimum point (corresponding point) obtained by the calculation. For this reason, an erroneous result may be obtained in a pixel to which such an assumption does not apply.
Thus, in the correlation calculation method using blocks such as the SAD calculation method, it is difficult to obtain a high-precision result in processing at the subpixel level, and there is a problem that an accurate distance image cannot be generated. there were.

  Accordingly, the present invention has been made to solve the above-described problems, and obtains a highly accurate range image by obtaining a high-precision calculation result in sub-pixel level calculation while reducing the total processing time. It is an object of the present invention to provide a distance image generation apparatus, a distance image generation method, and a program that can generate the image.

In order to solve the above-described problem, a distance image generation device according to claim 1 is provided.
Image acquisition means for acquiring first image information and second image information to be compared;
A first computing means for calculating a parallax value at a pixel level by comparing the first image information and the second image information by a first computing method;
Second calculation means for calculating a parallax value at a sub-pixel level by collating the first image information with the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting means for setting a target area to be calculated by the second calculation means based on the calculation result of the parallax value by the first calculation means;
It is characterized by having.

The invention according to claim 2 is the distance image generation device according to claim 1,
The first calculation method performed by the first calculation means is any one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.

The invention according to claim 3 is the distance image generation device according to claim 1 or 2, wherein
The second calculation method performed by the second calculation means is a POC calculation method.

The distance image generation method according to claim 4,
An image acquisition step of acquiring first image information and second image information to be compared;
A first calculation step of calculating a parallax value at a pixel level by collating the first image information and the second image information by a first calculation method;
A second calculation step of calculating a parallax value at a sub-pixel level by collating the first image information and the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting step for setting a target area to be calculated in the second calculation step based on the calculation result of the parallax value in the first calculation step;
It is characterized by having.

Invention of Claim 5 is the distance image generation method of Claim 4, Comprising:
The first calculation method performed in the first calculation step is any one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.

Invention of Claim 6 is the distance image generation method of Claim 4 or Claim 5, Comprising:
The second calculation method performed in the second calculation step is a POC calculation method.

The invention according to claim 7 is a computer-readable program,
A first calculation function that compares the first image information with the second image information to be compared by the first calculation method and calculates a parallax value at a pixel level;
A second calculation function for calculating a parallax value at a sub-pixel level by collating the first image information with the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting function for setting a target area to be calculated by the second calculation function based on the calculation result of the parallax value by the first calculation function;
Is realized by a computer.

Invention of Claim 8 is a program of Claim 7, Comprising:
The first calculation method is any one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.

Invention of Claim 9 is a program of Claim 7 or Claim 8, Comprising:
The second calculation method is a POC calculation method.

  According to the first, fourth, or seventh aspect of the present invention, the parallax value at the sub-pixel level (one pixel level or less) is calculated based on the calculation result of the parallax value at the pixel level (pixel level). Since the target region to be performed is set, the range for calculating the parallax value at the sub-pixel level (the level of one pixel or less) can be narrowed down, and the calculation process can be performed efficiently.

  In addition, the calculation of the parallax value at the sub-pixel level (the level of one pixel or less) is performed by the second calculation method with higher accuracy than the first calculation method for calculating the parallax value at the pixel level (pixel level). Thus, it is possible to efficiently perform the arithmetic processing to shorten the total processing time of the correlation calculation and to achieve an effect that high-precision processing can be performed.

  According to the invention of claim 2, claim 5 or claim 8, the pixel level (pixel level) is determined by any one of the SAD operation method, the SSD operation method, and the NCC operation method capable of performing high-speed processing. Since the calculation of the parallax value is performed, the corresponding point search can be efficiently performed, and the total processing time of the correlation calculation can be shortened.

  According to the invention of claim 3, 6 or 9, the parallax value at the sub-pixel level (one pixel level or less) is calculated by the POC calculation method which is a high-precision, robust and robust calculation method. Since the calculation is performed, it is possible to obtain a highly reliable calculation result with high reliability and to produce an accurate distance image.

  Hereinafter, an embodiment of a distance image generating apparatus according to the present invention will be described with reference to FIGS. 1 to 11. However, the scope of the invention is not limited to the illustrated examples.

  The distance image generation apparatus 1 according to the present embodiment calculates a positional deviation amount by collating two images, obtains distance information to an object based on the calculated positional deviation amount, and generates a distance image. is there.

As shown in FIG. 1, the distance image generating device 1 includes two imaging devices 2a and 2b that capture an image of a subject (target object).
The imaging devices 2a and 2b each include an imaging element (photoelectric conversion element) 3 and a lens 4 that forms a subject light image on an imaging surface (not shown) of the imaging element 3 as imaging means. The image pickup device 3 is an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The image pickup device 3 photoelectrically converts incident light transmitted through the lens 4 into an electric signal and takes it in. Thus, the subject light image is converted into an analog image signal. The distance image generation device 1 includes an A / D conversion unit 5 that converts an analog image signal obtained by the image sensor 3 into a digital image signal.

  The two imaging devices 2a and 2b capture the same object from different viewpoints and acquire image data as image information, one of which is reference image data (first image information) based on a reference image. And reference image data (second image information) based on an image to be compared (hereinafter referred to as “reference image”). Thus, the two imaging devices 2a and 2b function as an image acquisition unit that acquires image data of the standard image and the reference image.

  In addition, the distance image generation device 1 includes an image processing unit 10 that performs image processing on the image data acquired by each of the imaging devices 2a and 2b, calculates a parallax value, and generates a distance image.

  The image processing unit 10 includes a processing device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a DSP (Digital Signal Processor), a system program, and a distance image generation processing program for performing distance image generation processing described later. Is a computer composed of a ROM (Read Only Memory) for storing various control programs and the like, a RAM (Random Access Memory) for temporarily storing various data (all not shown), etc. A lens distortion correction unit 11 that corrects lens distortion, an image parallelization processing unit 12 that performs parallelization processing of images acquired by the imaging devices 2a and 2b, and a target region setting unit 14 that sets a target region for performing corresponding point search A first calculation unit 15 that performs parallax calculation processing at the pixel level, a second calculation unit 16 that performs parallax calculation processing at the sub-pixel level, and a distance image And a resulting distance image generation unit 17 or the like.

  The lens distortion correction unit 11 corrects distortion (lens distortion) that occurs when an image is formed using the lens 4. Since the lens 4 is distorted, an image cannot be generated correctly, and the image is distorted particularly at the end portion. The lens distortion correction unit 11 performs such lens distortion correction. For example, a lens distortion correction parameter is created and stored in advance, and an image is corrected based on the parameter. The correction method by the lens distortion correction unit 11 is not limited to the one exemplified here, and various known means can be used.

  The image collimation processing unit 12 converts the two images acquired by the respective imaging devices 2a and 2b as if they were taken from a camera moved in parallel so that the epipolar lines become parallel. A parallelization process is performed. Since it is difficult to accurately install the two imaging devices 2a and 2b, the image parallelization process is performed in order to correct image distortion caused by an installation position shift of the two imaging devices 2a and 2b. Is called.

  By performing the parallelization processing on the images, the two images are not shifted in the vertical direction (Y-axis direction in FIG. 2), and when performing correlation calculation processing described later, the horizontal direction (X-axis in FIG. 2). It is sufficient to detect only the deviation in the direction. Note that the method of image collimation processing by the image collimation processing unit 12 is not limited to the one exemplified here, and various known means can be used.

The target area setting unit 14 sets a target area for calculation in the parallax calculation processing by the second calculation unit 16.
In the present embodiment, as will be described later, when the parallax calculation processing at the pixel level by the first calculation unit 15 is completed, the calculation result is output to the target region setting unit 14, and the target region setting The unit 14 sets a calculation target area in the parallax calculation process at the sub-pixel level by the second calculation unit 16 based on the calculation result at the pixel level.

  That is, in the case of the parallax calculation process by the second calculation unit 16, the target area setting unit 14 corresponds points (candidate coordinates (x0, y0)) obtained by the parallax calculation process at the pixel level by the first calculation unit 15. ) Is set as a target region for parallax calculation processing by the first matching unit 6. In the present embodiment, for example, the target region setting unit 14 extracts a region of 32 pixels × 32 pixels around (x0, y0) and sets it as a target region. Note that the range of the target area set by the target area setting unit 14 is not limited to that exemplified here.

  The first calculation unit 15 obtains a correlation level by a SAD (Sum of Absolute Difference) calculation method (first calculation method) for a target region extracted and set from each of the standard image data and the reference image data, and then obtains a pixel level. It is the 1st calculating means which performs the parallax calculation process which calculates the parallax value in (pixel level).

Hereinafter, the parallax calculation processing by the first calculation unit 15 will be specifically described.
The first calculation unit 15 calculates the correlation between the reference image and the reference image using SAD (Sum) when one of the image data acquired by the image acquisition unit 2 is a reference image and the other is a reference image. of absolute differences (total absolute error)) Correlation is calculated by the calculation method, and corresponding points are searched. In the reference image, the amount of deviation between the reference image and the reference image for the region having the highest degree of correlation with the target region of the reference image ( A parallax calculation process for obtaining a (parallax value) is performed.

  The correlation calculation between the reference image and the reference image will be described with reference to FIG. FIG. 2A represents a standard image, and FIG. 2B represents a reference image. In FIG. 2, each square represents one pixel, and the entire screen of the base image and the reference image is composed of 26 pixels in the X-axis direction (horizontal direction) and 20 pixels in the Y-axis direction (vertical direction). Take the case as an example.

  When performing the correlation calculation, the first calculation unit 15 divides the reference image into predetermined unit areas, and sets a certain region in the reference image as a position (target region) for calculating the parallax value. For the setting of the target area, for example, areas in a predetermined range are sequentially set with the upper left (X = 1, Y = 20) in FIG.

That is, when a certain region (for example, 25 pixels of X = 1 to 5, Y = 16 to 20) is set as the target region and the correlation calculation for the target region is completed, for example, in the X-axis direction (the horizontal direction of the image) The next target area (for example, 25 pixels of X = 2 to 6, Y = 16 to 20) is set by sequentially shifting one pixel at a time, and correlation calculation is performed. When all the correlation calculations are completed for all the pixels in the X-axis direction (26 pixels in FIG. 2A), the next target area (for example, X = 1 to 5, Y = 15 to 19) is shifted by one pixel in the Y-axis direction. 25 pixels) is set, and correlation calculation is performed. When the correlation calculation is completed for the target region, the target region is sequentially shifted one pixel at a time in the X-axis direction to set the next target region (for example, 25 pixels of X = 2 to 6, Y = 15 to 19), Correlation is performed.
In this way, the target area is sequentially set by shifting one pixel each in the X-axis direction and the Y-axis direction from the upper left to the lower right of the screen, and the correlation calculation is completed for all the pixels on the entire screen. The correlation operation is repeated. Note that the order in which the correlation calculation is performed (how to set the target region) is not limited to that exemplified here.

Next, a specific method of correlation calculation will be described.
For example, as shown in FIG. 2A, a range surrounded by a broken line in the reference image (25 pixels of X = 9 to 13, Y = 9 to 13) is set as a target region, and correlation calculation is performed. The first calculation unit 15 sequentially sets comparison reference regions in the X-axis direction (horizontal direction of the image) for the same Y-axis range (Y = 9 to 13) as the target region in the reference image, and sequentially performs correlation calculation. Thus, the degree of correlation between the target region and each comparison control region is calculated. As a result, a region having the highest degree of correlation with the target region and a corresponding point are searched for in the reference image.

That is, the first calculation unit 15 firstly sets each pixel constituting the target region (25 pixels in total in FIG. 2A, X = 9, Y = 13, X = 10, Y = 13...). An image data value is calculated. Further, each pixel (for example, a range surrounded by a broken line in FIG. 2B (X = 9, Y = 13, X = 10) in the comparison reference region set as a reference region to be compared with the target region. , Y = 13... For all 25 pixels))). Then, the absolute value is calculated by subtracting the image data value of each pixel constituting the comparison area from the image data value of each pixel constituting the target area.
For example, the image data value of a certain pixel in the target region (X = 9, Y = 10 in FIG. 2A) is 100, and the corresponding pixel in the comparison region in FIG. When the image data value of X = 9 and Y = 10 is 50, 100−50 = 50 and the absolute value is 50.
Further, for example, the image data value of a certain pixel in the target region (X = 13, Y = 10 in FIG. 2A) is 50, and the corresponding pixel in the comparison region is shown in FIG. In the case where the image data value of X = 13 and Y = 10 is 90, 50−90 = −40 and the absolute value is 40.
Such subtraction of the image data value and calculation of the absolute value are performed for all 25 pixels constituting the target region and the comparison region. This calculation may be performed sequentially for each pixel, or may be performed simultaneously for all the pixels in the region.

  Further, the first calculation unit 15 adds all the calculated results (absolute values). The smaller the value after addition is, the higher the degree of correlation is. When the images are the same, the value approaches 0 as much as possible. For example, in the case of FIG. 2, a range surrounded by a broken line in FIG. 2A (25 pixels of X = 9 to 13 and Y = 9 to 13) and a range surrounded by an alternate long and short dash line in FIG. (25 pixels of X = 12 to 16, Y = 9 to 13) is the same image, and the image data value of the corresponding one of the pixels constituting the latter from the image data value of each pixel constituting the former, respectively. When the absolute value is obtained by subtraction and the calculation results are added, the value approaches 0 as much as possible.

In this way, when the reference image is searched for the region having the highest correlation degree with the target region of the reference image (the region where the extreme value of the correlation degree is obtained by the correlation calculation), the first calculation unit 15 searches for the correlation degree. The amount of deviation (parallax value) of how much the highest area is different from the target area of the reference image is calculated, and as shown in FIG. 3, which parallax value corresponds to the area with the highest degree of correlation. Ask.
That is, according to the example of FIG. 2, the target region (25 pixels of X = 9 to 13 and Y = 9 to 13 in FIG. 2A) and the region having the highest degree of correlation are illustrated in FIG. The range surrounded by the one-dot chain line (25 pixels of X = 12 to 16, Y = 9 to 13) is shifted by 3 pixels in the X-axis direction. Therefore, the parallax value in this case is 3.
The first calculation unit 15 outputs the calculation result to the target region setting unit 14 as a parallax value at a pixel level (pixel level).

  Note that the relationship between the parallax value and the distance value is distance value = constant / parallax value, and the distance value is proportional to the reciprocal of the parallax value. In other words, the distance becomes longer as the parallax value becomes smaller, and the distance becomes closer as the parallax value becomes larger. For this reason, the 1st calculating part 15 can acquire a distance value by calculating | requiring a parallax value by the said correlation calculation, and calculating | requiring the reciprocal number of a parallax value.

In addition, the second calculation unit 16 uses a POC (Phase-Only Correlation) calculation method (second calculation method) for a target region extracted and set from each of the standard image data and the reference image data. It is a 2nd calculating means which performs the parallax calculation process which calculates | requires a correlation degree and calculates the parallax value in a sub-pixel level (1 pixel or less level).
The second calculation unit 16 includes a first matching unit 6 that performs a parallax calculation process using a POC calculation method, and a second matching unit 7 that performs a nonlinear fitting process based on the calculation result obtained by the first matching unit 6. ing.

  In the POC calculation method, the base image data and the reference image data to be collated are mathematically processed by Fourier transform to be decomposed into amplitude (grayscale data) and phase (image contour data), of which the phase This is an algorithm that uses only information to obtain the correlation between both images. Unlike the correlation calculation method using amplitude information such as the SAD calculation method, it is resistant to disturbances and can obtain high-precision calculation results.

Hereinafter, the parallax calculation processing by the second calculation unit 16 will be specifically described.
As shown in FIG. 4, the first verification unit 6 of the second calculation unit 16 includes window function units 61a and 61b, FFT (Fast Fourier Transform) units 62a and 62b, and phase information extraction units 63a and 63b. , A combining unit 64, an IFFT (Inverse Fast Fourier Transform) unit 65, and a correlation value calculation unit 66 are provided.

  The window function units 61a and 61b apply a window function to signals corresponding to the target areas of the standard image data and the reference image data. The type of window function is not particularly limited and can be changed as appropriate. Here, a signal obtained by applying a window function to the signal based on the standard image data is denoted by f1, and a signal obtained by applying the window function to the signal based on the reference image data is denoted by f2. The window function units 61a and 61b output the signals f1 and f2 multiplied by the window function to the FFT units 62a and 62b, respectively.

  The FFT units 62a and 62b perform a Fourier transform process on the signals f1 and f2 and output them to the phase information extraction units 63a and 63b. Specifically, the FFT unit 62a performs a two-dimensional discrete Fourier transform on the signal f1 to obtain Fourier image data F1 based on the reference image. Further, the FFT unit 62b performs two-dimensional discrete Fourier transform on the signal f2 to obtain Fourier image data F2 based on the reference image. In addition, there is no restriction | limiting in particular in a Fourier-transform process, For the two-dimensional discrete Fourier transform, the "Introduction to computer image processing, the Japan Industrial Technology Center edition, issue of Soken publication, P44-45" was referred.

The phase information extraction units 63a and 63b are for extracting phase information by removing amplitude components from the signals output from the FFT units 62a and 62b. That is, the phase information extraction units 63a and 63b perform the phase limiting process on the Fourier-transformed signals F1 and F2 to obtain Fourier image data F3 and F4. The phase information extraction units 63a and 63b output the obtained Fourier image data F3 and F4 to the synthesis unit 64.
Note that the method by which the phase information extraction units 63a and 63b extract only the phase information is not particularly limited, and the phase information extraction unit 63a and 63b may extract only the phase by setting the amplitude to 1, or the amplitude component may be extracted by log processing or √ processing. It is good also as removing.

The synthesizer 64 synthesizes Fourier image data F3 and F4 obtained by extracting only phase information from the Fourier-transformed signals F1 and F2, and synthesizes Fourier image data F5 (u, v) = F3 ^* (u, v) F4 (u, v) is obtained. In the formula, “*” represents a complex conjugate, and (u, v) represents a coordinate in Fourier space. The synthesizer 64 outputs the obtained synthesized Fourier image data F5 to the IFFT unit 65.

  The IFFT unit 65 performs inverse Fourier transform on the synthesized Fourier image data F5 obtained by the synthesizing unit 64 to obtain synthesized inverse Fourier image data f5. The IFFT unit 65 outputs the synthesized inverse Fourier image data f5 to the correlation value calculation unit 66.

The correlation value calculation unit 66 performs a correlation calculation between the images from the synthesized inverse Fourier image data f5 to obtain a correlation value (POC value).
FIG. 5 shows an example of a result (correlation strength image data) obtained by performing correlation calculation between images by the POC calculation method. FIG. 5 shows an example in which a search area and a comparison target area of N ₁ × N ₂ pixels are set on the standard image and the reference image, and the correlation of a portion having a high correlation in the N ₁ × N ₂ pixel area. The value (POC value) is large. The position in the comparison target region on the reference image corresponding to this POC value peak (Jc in FIG. 5) corresponds to the corresponding point (candidate coordinates) on the reference image corresponding to the center point in the search region on the standard image. Will be.

  According to the calculation processing using the POC calculation method as described above, the correlation calculation is performed using only the phase component of the image by removing the amplitude component of the image. Corresponding points can be searched with high accuracy.

  When the correlation value calculation unit 66 obtains the calculation result (correlation strength image data), it outputs this to the second verification unit 7.

The second collation unit 7 detects a sub-pixel level displacement amount based on the correlation strength image data obtained by the first collation unit 6. Specifically, the second collation unit 7 correlates the intensity data of 5 pixels × 5 pixels centering on the position where the peak of the correlation intensity image appears (center position of the correlation intensity image data: Jc in FIG. 5). Perform nonlinear fitting using.
As shown in FIG. 6, the second matching unit 7 includes a window function unit 71 and a non-linear fitting unit 72, and the second matching unit 7 has a positional deviation amount detected by the non-linear fitting unit 72 as a threshold value. The amount of misalignment is detected until the following is satisfied.

  The window function unit 71 applies a window function to the correlation intensity image data. The type of window function is not particularly limited, but in the present embodiment, a Hanning window is applied. The window function unit 71 determines the center position of the correlation strength image data based on the amount of positional deviation detected by the nonlinear fitting unit 72.

The non-linear fitting unit 72 approximates the amount of positional deviation at the sub-pixel level by the Levenberg-Merquardt method (non-linear least square method as a non-linear operation) for the correlation strength image data multiplied by the window function. Specifically, the non-linear fitting unit 72 applies the peak model (see FIG. 8) to the correlation intensity image data (see FIG. 7) for 5 pixels × 5 pixels, and obtains the positional deviation amount by the non-linear least square method as the non-linear calculation. It has become. There is no restriction as a non-linear operation, and it can be changed as appropriate to those that perform complicated conditional branch processing.
Further, the non-linear fitting unit 72 determines whether or not the detected positional deviation amount is equal to or less than a predetermined threshold value, and outputs the detection result to the window function unit 71 if it is larger than the threshold value. Further, when the detected positional deviation amount is equal to or smaller than the predetermined threshold, the nonlinear fitting unit 72 outputs a value obtained by adding all the previous detection results to the distance image generating unit 17 as a sub-pixel level positional deviation amount. It has become. Here, the predetermined threshold value is a value that can be arbitrarily set by the user, and the threshold value becomes smaller as precise collation is required.

  When the parallax calculation processing at the sub-pixel level by the second calculation unit 16 ends, the distance image generation unit 17 displays the calculation result of the positional deviation amount (parallax value) between the reference image data and the reference image data in the parallax calculation processing. Based on this, distance image data is generated.

  In addition, the distance image generating apparatus 1 is provided with an output unit 20 that outputs the distance image generated by the distance image generating unit 17. The output unit 20 is a monitor (display means) such as a CRT (Cathode Ray Tube) or LCD (Liquid Crystal Display), for example, and displays a distance image. Note that the output unit 20 is not limited to a monitor, and is, for example, a communication unit for connecting to an external device (output unit) such as a printer, and configured to transmit the generated distance image to the external device and output it. May be.

  Next, a distance image generation method performed by the distance image generation apparatus 1 according to the present embodiment will be described with reference to FIGS. 9 to 11. The distance image generation process is realized by the cooperation of the image processing unit 10 which is a computer and the distance image generation process program.

  As shown in FIG. 9, when the correlation calculation for generating the distance image is performed by the distance image generating device 1, first, the imaging devices 2a and 2b capture the object with the two imaging means 2a and 2b. The image data of the standard image and the reference image is acquired (step S1).

  Image data of the standard image and the reference image acquired by the imaging units 2 a and 2 b are A / D converted by the A / D conversion unit 5 and sent to the image processing unit 10. When the image data of the reference image and the reference image is sent to the image processing unit 10, distortion correction by the lens distortion correction unit 11 and image parallelization processing by the image parallelization processing unit 12 are performed on each image data (step S2). ).

Next, a calculation process (first calculation process) is performed in which the first calculation unit 15 calculates a parallax value at a pixel level for the image data of the reference image and the reference image subjected to the distortion correction process and the image parallelization process. The result is output to the target area setting unit 14 (step S3).
Then, the target area setting unit 14 sets a target area to be calculated based on the calculation result of the pixel level parallax value (step S4). Further, a calculation process (second calculation process) for calculating a sub-pixel level parallax value is performed on the set target area by the second calculation unit 16 (step S5).

  Here, the calculation process (step S3 in FIG. 9) in which the first calculation unit 15 calculates the parallax value at the pixel level will be specifically described with reference to FIG. In the following, an example in which the entire screen of an image is a distance measurement target (that is, all pixels are distance measurement points) and corresponding point search is performed for all pixels in the X-axis direction for one target area. Will be described.

As shown in FIG. 10, the first calculation unit 15 sets a distance measuring point in the reference image (step S11), and a predetermined target area centered on the distance measuring point is set (step S12).
When the target area is set, the first calculation unit 15 sets a comparison target area having the same size as the target area in the reference image (step S13), and performs a correlation calculation by the SAD calculation method, thereby comparing with the target area. The degree of correlation with the target area is calculated (step S14).

  The first calculation unit 15 determines whether or not the parallax calculation processing has been completed for all pixels in the X-axis direction of the comparison target image for the target region (step S15). If not completed (step S15; NO), the first calculation unit 15 shifts the comparison target area in the reference image by one pixel in the X-axis direction (step S16), returns to step 14, and returns to the parallax calculation process. repeat. When the parallax calculation processing has been completed for all the pixels in the X-axis direction (step S15; YES), the pixel having the highest degree of correlation is searched for in the parallax calculation processing, and the parallax value is calculated (step S17).

When the first calculation unit 15 calculates the parallax value for the target region, the first calculation unit 15 further determines whether the parallax calculation processing has been completed for all the ranging points (all pixels to be subjected to ranging) in the reference image (Step S15). S18). In the present embodiment, since the parallax calculation process is performed for all the pixels of the image, it is determined whether the process has been completed for all the pixels.
If there are unprocessed pixels that should be measured in the image (step S18; NO), the unprocessed pixels are set as new ranging points (step S19). A target area centered on this distance measuring point is set (step S12), and the processing from step S13 to step S17 is repeated. On the other hand, if there is no unprocessed distance measuring point in the image (step S18; YES), the pixel level parallax calculation process (first calculation process) is terminated.

  Next, the calculation process (step S5 in FIG. 9) for calculating the sub-pixel level parallax value by the second calculation unit 16 will be specifically described with reference to FIG.

When the target area is set by the target area setting unit 14 based on the calculation result of the parallax value at the pixel level by the first calculation unit 15 (see step S4 in FIG. 9), first, the window function units 61a and 61b are image data. The FFT unit 62a, 62b performs a two-dimensional Fourier transform process with a block size of 32 × 32 pixels centered on the center pixel.
That is, as shown in FIG. 11, the window function unit 61a creates a signal f1 based on the reference image, and 62a performs a Fourier transform process on the signal f1 to convert the signal f1 into the signal F1 (step S21). Further, the window function unit 61b creates a signal f2 based on the reference image, and 62b performs a Fourier transform process on the signal f2 to convert the signal f2 into the signal F2. (Step S22).

Thereafter, the phase information extraction units 63a and 63b remove the amplitude components based on the signals F1 and F2 and extract the phase information F3 from the signal F1 and the phase information F4 from the signal F2 (step S23). Further, the signals F3 and F4 having only phase information are combined by the combining unit 64 to generate a combined signal F5 (step S24). The synthesized signal F5 is output from the synthesizing unit 64 to the IFFT unit 65, and the IFFT unit 65 performs inverse Fourier transform processing on the synthesized signal F5 to convert it to f5 (step S25). Thereafter, the correlation value calculation unit 76 calculates a correlation value max _{x, y} f5 (u, v) based on f5 (step S26), and outputs the calculation result (correlation strength image data) to the second verification unit 7. (Step S27)

  The second collating unit 7 detects the amount of positional deviation at the sub-pixel level. Specifically, first, the window function unit 71 applies a Hanning window to the correlation strength image data output from the first matching unit 6. Thereafter, the non-linear fitting unit 72 applies a peak model (see FIG. 8) to the signal from the window function unit 71 (correlation strength image data for 5 pixels × 5 pixels (see FIG. 7)), and the Levenberg-Merquardt method. The shift amount is approximately calculated by (a nonlinear least square method as a nonlinear operation) (step S28: nonlinear fitting process). The nonlinear fitting unit 72 determines whether or not the result of the approximation calculation is equal to or less than a predetermined threshold (step S29). If the result is larger than the threshold (step S29; NO), the detection result is output to the window function unit 71 (step S29). S30). Then, the nonlinear fitting process (step S28) is performed again. Further, when the detection result is equal to or less than the predetermined threshold (step S29; YES), the nonlinear fitting unit 72 adds all the previous detection results to the distance image generation unit 17 as a sub-pixel level displacement amount. Output (step S31). Thereby, the sub-pixel level parallax calculation processing by the second calculation unit 16 ends.

  Returning to FIG. 9, when the parallax calculation processing by the second calculation unit 16 is completed and the calculation result by the second matching unit 7 is output to the distance image generation unit 17 as a sub-pixel level calculation result (step of FIG. 11). S31), the distance image generation unit 17 generates a distance image based on the calculation result (step S6). And the distance image produced | generated by the distance image production | generation part 17 is output from the output part 20 as needed.

  As described above, according to the present embodiment, in the pixel level parallax calculation processing, the first calculation unit 15 capable of performing high-speed calculation processing performs processing, and in the sub-pixel level parallax calculation processing, The target region to be calculated is determined using the calculation result of the first calculation unit 15 and the calculation processing by the second calculation unit 16 capable of performing high-precision calculation processing is performed, so that the parallax calculation processing is efficiently performed By doing so, it is possible to achieve both reduction of the total processing time of the correlation calculation and generation of a highly accurate distance image.

  In the present embodiment, two image pickup means 2a and 2b are provided, the same object is picked up by the image pickup means 2a and 2b, and standard image data (first image information) and reference image data (second image information). However, the configuration of the imaging devices 2a and 2b is not limited to this. The imaging devices 2a and 2b need only be configured to include at least one imaging unit, and may acquire the standard image data and the reference image data by imaging the same object at different timings by one imaging unit. In addition, two or more imaging units may be provided, and the same object may be captured by each imaging unit to acquire the standard image data and the reference image data.

  Further, in the present embodiment, the case where the imaging devices 2a and 2b are provided as the constituent elements of the distance image generating device 1 has been described. However, the image acquiring unit is provided outside the distance image generating device 1, and the distance image generating device 1 is provided. Alternatively, reference image data (first image information) and reference image data (second image information) may be input from the outside, and processing such as correlation calculation may be performed based on the image data.

  In the present embodiment, the first calculation unit 15 is configured to calculate candidate coordinates based on the correlation value and output the calculated candidate coordinates to the target region setting unit 14 as a search result. The search result output to the area setting unit 14 is not limited to this. For example, the target region setting unit 14 outputs a calculation result indicating a correlation value from the first calculation unit 15 as a search result, and the target region setting unit 14 has the highest degree of correlation based on the calculation result. (Candidate coordinates) may be calculated, and the periphery of the candidate coordinates may be set as the target region.

  Further, in the present embodiment, the case has been described as an example in which the entire screen of the image is the object of distance measurement (that is, all the pixels are the distance measurement points). If it is desired to perform distance measurement only for a portion, the range for performing the parallax calculation processing may be limited, for example, by setting a distance measurement point only for a target portion for which distance measurement is desired and performing parallax calculation processing.

Further, in the present embodiment, the case where the corresponding point search is performed for all the pixels in the X-axis direction for one target region has been described as an example. In some cases, the distance may be obtained. In such a case, the parallax value range corresponding to the distance range according to the application may be used as the calculation range, and the comparison target region may be moved within this range to perform the parallax calculation processing.
For example, when it is sufficient to obtain only a distance that is at a distance of 100 m or more, the comparison target area is moved within this range within a range of parallax values corresponding to a distance value of 100 m or more. Thereby, in the parallax calculation process, the processing time of the correlation calculation can be shortened.

Further, in the present embodiment, the case where the reference image is divided into unit areas of 25 pixels and the target region is set and correlation calculation is performed for each target region has been described as an example. It is not limited to 25 pixels each, and can be set to an arbitrary range.
In this embodiment, the target area is a predetermined area of 25 pixels, and the correlation calculation is performed for each area. For example, the correlation between the reference image and the reference image is calculated for each pixel. It may be.

  In the present embodiment, the first calculation unit 15 that obtains the distance value by the SAD calculation method is provided as the first calculation unit. However, the first calculation unit can perform the process of obtaining the distance value at high speed. Anything is possible as long as it is possible, and the present invention is not limited to this. For example, a calculation unit that obtains a distance value by an SSD (Sum of Squared Differences) calculation method or an NCC (Normalized Cross-Correlation) calculation method may be provided as the first calculation means.

  In the present embodiment, the second calculation unit 16 that searches for corresponding points by the POC calculation method is provided as the second calculation means. However, the second calculation means searches for the corresponding points with high accuracy. However, the present invention is not limited to the one based on the POC calculation method.

Further, in the present embodiment, the case where the image processing unit 10 is a computer including a CPU and the like and a memory such as a ROM and a RAM has been described as an example. You may comprise by wear.
That is, for example, the second arithmetic unit 16 is preferably configured by dedicated hardware that performs phase-only correlation processing. As dedicated hardware, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or the like is applicable.

  In addition, a multi-resolution image composed of a plurality of hierarchical images with different resolutions is generated for the base image and the reference image, and in the calculation in the first calculation unit 15, corresponding points are sequentially applied from the low-resolution hierarchical image to the high-resolution hierarchical image. This search process may be repeated and the search result obtained for the low-resolution hierarchical image may be reflected in the search in the next hierarchy. By adopting such a method, it is possible to perform calculation processing at higher speed and with higher accuracy.

  In addition, it is needless to say that the present invention is not limited to the above embodiment and can be modified as appropriate.

It is a block diagram which shows the whole structure of the correlation calculating apparatus which concerns on this embodiment. FIG. 2A is a diagram illustrating an example of a reference image in the present embodiment, and FIG. 2B is a diagram illustrating an example of a reference image in the present embodiment. It is explanatory drawing which shows the relationship between a correlation degree and a parallax value. It is a block diagram which shows the structure of the 2nd calculating part shown in FIG. It is explanatory drawing which shows the example of the result obtained by the arithmetic processing by a 1st collation part. It is a block diagram which shows the structure of a 2nd collation part. It is a figure which shows an example of correlation intensity | strength image data. It is a figure which shows an example of a peak model. It is a flowchart showing the distance image generation process in this embodiment. It is a flowchart showing the arithmetic processing by the 1st calculating part shown in FIG. It is a flowchart showing the arithmetic processing by the 2nd calculating part shown in FIG.

Explanation of symbols

1 Distance Image Generating Device 2a, 2b Imaging Device (Image Acquisition Unit)
6 First collation unit 7 Second collation unit 10 Image processing unit 11 Lens distortion correction unit 12 Image parallelization processing unit 14 Target region setting unit (target region setting means)
15 1st calculating part (1st calculating means)
16 2nd calculating part (2nd calculating means)
17 Distance image generation unit (distance image generation means)
61a, 61b Window function units 62a, 62b FFT units 63a, 63b Phase information extraction unit 64 Synthesis unit 65 IFFT unit 66 Correlation calculation unit

Claims (9)

Image acquisition means for acquiring first image information and second image information to be compared;
A first computing means for calculating a parallax value at a pixel level by comparing the first image information and the second image information by a first computing method;
Second calculation means for calculating a parallax value at a sub-pixel level by collating the first image information with the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting means for setting a target area to be calculated by the second calculation means based on the calculation result of the parallax value by the first calculation means;
A distance image generating apparatus comprising:   The range image generation apparatus according to claim 1, wherein the first calculation method performed by the first calculation means is one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.   The distance image generation apparatus according to claim 1 or 2, wherein the second calculation method performed by the second calculation means is a POC calculation method. An image acquisition step of acquiring first image information and second image information to be compared;
A first calculation step of calculating a parallax value at a pixel level by collating the first image information and the second image information by a first calculation method;
A second calculation step of calculating a parallax value at a sub-pixel level by collating the first image information and the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting step for setting a target area to be calculated in the second calculation step based on the calculation result of the parallax value in the first calculation step;
A distance image generation method characterized by comprising:   5. The distance image generation method according to claim 4, wherein the first calculation method performed in the first calculation step is any one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.   The distance image generation method according to claim 4 or 5, wherein the second calculation method performed in the second calculation step is a POC calculation method. A first calculation function that compares the first image information with the second image information to be compared by the first calculation method and calculates a parallax value at a pixel level;
A second calculation function for calculating a parallax value at a sub-pixel level by collating the first image information with the second image information by a second calculation method having higher accuracy than the first calculation method; ,
A target area setting function for setting a target area to be calculated by the second calculation function based on the calculation result of the parallax value by the first calculation function;
A computer-readable program characterized by causing a computer to realize the above.   The computer-readable program according to claim 7, wherein the first calculation method is any one of an SAD calculation method, an SSD calculation method, and an NCC calculation method.   The computer-readable program according to claim 7 or 8, wherein the second calculation method is a POC calculation method.

Images