JP2009104366A - Stereo image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a stereo image processing method for reducing mismatching in stereo matching processing due to a noise included in an image picked up by a stereo camera, and for acquiring a highly reliable parallax image and distance image. <P>SOLUTION: In step S1, the luminance value of the pixel of a reference image is compared with a prescribed threshold Nth. In step S2, the luminance value of the pixel of a reference image whose luminance value is determined to be smaller than a prescribed threshold Nth is replaced with 0 value. In step S4, when the luminance value of the pixel of the reference image is 0 value, a parallax vector Vp of the pixel is set to 0 value. In step S5, corresponding points in a comparison image are specified by defining the pixel of the reference image whose luminance value is determined to be equal to or more than the prescribed threshold Nth as a pixel of interest, and the parallax vector Vp of the pixel of interest generated according to a distance Z to a region where a target object 2 exists is obtained, and the processing is ended. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a stereo image processing method for obtaining a disparity vector between corresponding pixel points of a stereo image obtained by capturing a target object and calculating a distance to the target object based on the parallax vector.

  In general, as a three-dimensional measurement technique based on images, a correlation between a pair of images obtained by capturing a target object with a stereo camera including two cameras arranged at different positions is obtained, and the parallax between the two cameras with respect to the target object is obtained from this correlation. Image processing by a so-called stereo method is known in which the position of the target object is calculated from the parallax using the camera parameters such as the mounting position and focal length of the stereo camera based on the principle of triangulation.

  In recent years, a stereo-type vehicle exterior monitoring device using a pair of in-vehicle cameras (stereo cameras) incorporating a solid-state imaging device such as a CCD or CMOS has been attracting attention. In the stereo method, which is one of the three-dimensional measurement techniques, a pixel block having a correlation with a pixel block (small region) in one image is specified in the other image (stereo matching process), and the relative relationship between both pixel blocks is determined. The distance information is calculated from the parallax, which is the shift amount, using the principle of triangulation.

  When a stereo image consisting of a pair of images obtained by capturing a target object with a stereo camera is obtained and one point on the target object is projected onto one image, if the projection destination of the other image at this point is known, The distance to the point on the target object can be calculated based on the principle of triangulation. This is disclosed in Non-Patent Document 1, for example.

  FIG. 11 is a diagram for explaining the principle of calculating the distance by the stereo camera.

  As shown in FIG. 11, a point Q (x, y, z) on the target object 2 is a pair of imaging cameras 3a arranged at a certain distance (= base line length) B between the optical axes Oa and Ob. 3b. Next, the coordinates Qa (xa, ya) and Qb (xb, yb) of the projection point on the XY coordinate plane of the point Q (x, y, z) in the stereo image captured by the cameras 3a and 3b are obtained. It is done. The parallax Qa−Qb = (xa−xb, ya−yb) is obtained from the coordinates Qa (xa, ya) and Qb (xb, yb), and corresponds to the projection point of the point Q (x, y, z). The distance Z to the point Q (x, y, z) on the target object 2 to be calculated is calculated.

  Here, if two cameras 3a and 3b for capturing a stereo image are arranged so that the optical axes Oa and Ob are included in the same XZ plane, Ya = Yb. When the point Q (x, y, z) on the target object 2 is projected onto the coordinates Qa on the image captured by the camera 3a, the image captured by the camera 3b at the point Q (x, y, z) The search (corresponding point search) of the coordinate Qb that is the projection destination on the top is performed on the epipolar lines Epa and Epb where the epipolar plane including the points Q, Qa, and Qb is projected on the images captured by the cameras 3a and 3b. Just do it. That is, if the distance between the optical axes Oa and Ob of the cameras 3a and 3b is B and the focal length is f, the point Q (x, y, z) on the target object 2 is imaged by the two cameras 3a and 3b. When projected onto a point Qa (xa, ya) and a point Qb (xb, yb) on the image, the distance Z to the point Q (x, y, z) on the target object 2 is expressed by [Equation 1]. expressed.

[Equation 1]
Z = B * f / (xa−xb)
Here, xa−xb represents parallax, and when an image captured by the camera 3a is used as a reference image, the disparity vector at the corresponding point Qb (xb, yb) of the comparison image captured by the camera 3b is generally Vp (xa -Xb, ya-yb).

  FIG. 12 is a block diagram showing a configuration of a conventional stereo image processing apparatus.

  As shown in FIG. 12, a conventional stereo image processing apparatus 30 includes a stereo camera 3 including a pair of cameras 3a and 3b that capture a stereo image of a region where the target object 2 exists, and a camera 3a. Each image captured by the camera 3b is obtained from a stereo camera processing unit 5 that obtains a parallax image of a region where the target object 2 exists by stereo matching processing and obtains distance distribution information (distance image).

  The two cameras 3a and 3b constituting the stereo camera 3 are cameras incorporating a solid-state image sensor such as a CCD or a CMOS. One camera 3a is synchronized with each other as a main camera that captures a reference image in the stereo matching process, and the other camera 3b is a sub camera that captures a comparative image in the stereo matching process. The arrangement is such that the vertical axes of the imaging surfaces are parallel to each other. Each of the cameras 3a and 3b includes an amplifier and an A / D converter for processing an analog imaging signal, and various functional circuits for image processing configured by, for example, a highly integrated FPGA, and the captured image is displayed with a predetermined luminance level. Converted into digital image data of the key and output.

  The stereo camera processing unit 5 includes a corresponding point detection unit 6 and a distance calculation unit 7.

  The corresponding point detection unit 6 stores the reference image captured by the camera 3a and the comparison image captured by the camera 3b in the original image memory, and associates the pixels of the comparison image for each pixel of the reference image by stereo matching processing. . In this stereo matching process, for example, a corresponding point search is performed according to an evaluation function such as a correlation function disclosed in Patent Document 1 as Expression (3). In addition, the corresponding point detection unit 6 obtains, as the parallax vector Vp, a pixel shift (= parallax) that occurs according to the distance to the area where the target object 2 exists for each pixel of the reference image using the searched corresponding points. . When this disparity vector Vp is arranged for each pixel constituting the reference image, it constitutes a disparity image.

  The distance calculation unit 7 calculates a distance Z obtained by quantifying distance information, which is perspective information from the disparity vector Vp obtained by the corresponding point detection unit 6 to the area where the target object 2 exists. If this distance Z is arranged for each pixel constituting the reference image, it constitutes distance distribution information (distance image).

  In stereo matching processing in the conventional stereo image processing apparatus 30, a pair of analog images, which are stereo images captured by the cameras 3a and 3b, are digitized for digital processing, and then the values of the digitized images are converted. A corresponding point search is performed using this, and distance information is calculated. Therefore, if noise is included in the digitized image, mismatching (miscorresponding) may occur due to the influence of noise in the stereo matching process, and an accurate parallax vector Vp or distance Z may not be obtained.

  Examples of the noise included in the digital image include fixed pattern noise included in the image signal output from the cameras 3a and 3b, an offset of the A / D conversion unit, and random noise. In a camera (especially a CMOS sensor) incorporating a solid-state imaging device, each pixel arranged in a two-dimensional manner has an amplifier, and noise (fixed pattern noise) is generated due to a difference in gain of each amplifier. Further, the video signal is A / D converted in order to digitize the analog images captured by the cameras 3a and 3b. A quantization error and an offset error occur during the A / D conversion. Furthermore, in addition to fixed pattern noise and offset error, random noise such as thermal noise and spike noise occurs.

  In addition, since a camera with a built-in solid-state imaging device has a limited light receiving sensitivity, auto gain control (AGC) is used to cope with a bright imaging environment or a dark imaging environment. This auto gain control uses a fixed pattern noise subtraction process because the amplification when shooting in an imaging environment illuminated with uniform light is different from the amplification when shooting in a dark or bright imaging environment. However, the effect of noise remains on the image.

  Patent Document 2 discloses a method of removing noise (mainly fixed pattern noise) caused by variations in characteristics of different transfer paths in a camera incorporating a solid-state imaging device such as a CMOS having an amplifier in each pixel or a CCD camera. As described above, the image output of each pixel position of an image captured by irradiating the imaging space where the target object 2 exists with uniform light is recorded in a memory, and noise is subtracted from the image output during normal imaging. A method for eliminating the effects of the above is disclosed. However, this method can be expected to eliminate noise with respect to fixed pattern noise, but cannot eliminate the influence of random noise.

  Patent Document 3 discloses a method for eliminating the influence of random noise by correction using a probabilistic model or the like. However, in this method, noise is corrected by estimation, and cannot be corrected to data for correctly calculating the distance information of the target object 2 to be imaged. In the stereo method, images captured by at least two cameras 3a and 3b are used. If noises of images captured by a plurality of cameras are corrected by estimation, a new miscorrespondence occurs in stereo matching processing. There is a fear.

  Further, Patent Document 4 discloses a method of eliminating the influence of noise by observing the occurrence frequency of the calculated absolute value of the disparity vector and excluding the absolute value of the disparity vector having a frequency equal to or lower than a predetermined value from the processing. Has been. This method has the effect of reducing sudden noise included in the parallax image and smoothing the parallax image as a whole, but a certain degree of correct measurement result is required for smoothing. Therefore, for example, in a dark image in which noise is dominant, the correct parallax vector Vp is hardly obtained, and thus the influence of noise cannot be eliminated.

  Furthermore, in Patent Document 5, a normal stereo matching process and a stereo matching process for an image to which a filter is applied are performed in parallel, the results of both are compared, and if the difference is less than a predetermined value, the former is A method is disclosed in which the latter is selected when the value exceeds a predetermined value. The purpose of this filter is to remove the influence of a sudden luminance change due to noise by using a low-pass filter. However, for example, in a dark image where noise is dominant, the luminance change is very small. It is difficult.

Specifically, the image captured by the camera with the aperture fully closed is black, and all the pixels are searched as corresponding points by the corresponding point search, and parallax cannot be obtained. However, if the image captured by the camera with the aperture fully closed contains noise, the pixel containing the noise is erroneously searched as a corresponding point as long as there is a frequency component that passes through the filter. There is a risk that parallax is required. Although it has been proposed to perform filtering using images with different imaging times, it is effective against random noise, but it cannot obtain the same result for a pair of cameras, The offset error cannot be removed.
JP-A-9-33249 JP-A-5-30350 JP 2006-310999 A JP 2000-35329 A JP 2000-35330 A The Institute of Electronics, Information and Communication Engineers (ed.) Nakashima Masayuki, Yamamoto Masanobu, "Graphics and Vision-Digital Video Processing" Ohmsha, April 25, 1996, p28-31

  FIG. 13 is a diagram for explaining a change in the signal level of the luminance value of the image due to the brightness of the imaging environment and a change in the S / N due to noise.

  As shown in FIG. 13, an image captured in a bright imaging environment has a high luminance value signal level (line Sb), and an image captured in a dark imaging environment has a low luminance value signal level (line Sd). Comparing the signal levels of the luminance values of the images captured in both imaging environments, for example, when both signals include noise (line N) at substantially the same level, the image captured in the dark imaging environment is S / N. Gets worse.

  In addition, when noise in an image cannot be completely removed, in an image including dark pixels, a difference between a signal and noise becomes small and S / N deteriorates. In such an image, pixels are characterized by noise rather than the original signal.

  Furthermore, if the maximum noise value Nmax included in the image is uniformly reduced from all the pixels constituting the image, the luminance value of each pixel after subtraction remains affected by noise.

  In the stereo method, in block matching processing, which is an example of a method for searching for corresponding points between images, an image block (small region) is cut out from images to be compared in order to evaluate the similarity between images, and luminance for the small region is determined. Sum of differences (SAD: Sum of Abs Difference), square sum of luminance differences (SSD: Sum of Squared Difference), normalized cross-correlation (ZNCC), and the like. By searching for the cutout position of a small region that gives the minimum value when using SAD or SSD and using the maximum value when using ZNCC, the displacement between images, that is, the parallax between the camera 3a and the camera 3b can be obtained in units of pixels. .

  Here, in the case of SAD, which is an example of stereo matching processing, the effect of S / N deterioration on the search for corresponding points will be described.

  FIG. 14 is a diagram for explaining the influence of S / N deterioration on the search for corresponding points in stereo matching processing, and FIG. 14A shows an example of the luminance value of a small region of a reference image captured in a bright environment. FIG. 8B is a diagram illustrating an example of a luminance value of a small area of a comparative image captured in a bright environment, and FIG. 10C is a small area of a reference image including noise captured in a bright environment. (D) is a figure which shows an example of the luminance value of the small area | region of the reference | standard image imaged in the dark environment, (E) is a comparison imaged in the dark environment. It is a figure which shows an example of the luminance value of the small area | region of an image, (F) is a figure which shows an example of the luminance value of the small area | region of the reference | standard image containing the noise imaged in the dark environment.

  As illustrated in FIG. 14, as an example of a pixel block (small region) for searching for corresponding points between a reference image and a comparison image by stereo matching processing, a small region corresponding to a size of 3 × 3 pixels is considered. At this time, the attention point in the reference image is a pixel located at the center of the small region.

  As in the small area of the reference image shown in FIG. 14A and the small area of the comparative image shown in FIG. 14B, the arrangement of the luminance values of the small areas of the reference image and the comparative image is the same. In this case, when stereo matching processing is performed by SAD, SAD becomes 0 as shown in [Expression 2]. This SAD value is a correlation value of a small area between the reference image and the comparison image. Since the stereo camera 3 captures the target object 2 that is a three-dimensional object from different line-of-sight directions between the camera 3a and the camera 3b, there is a difference in the projection plane of the target object 2, and the reference image and the comparison image are small. The regions rarely match completely, and in the stereo matching process, association is performed between the small regions where the correlation value is minimized.

[Equation 2]
ABS {(100-100) + (120-120) + (140-140) + (90-90) + (100-100) + (80-80) + (200-200) + (230-230) + (120-120)} = 0
Here, when a noise of about ± 5, for example, is included in the image as in the small region of the reference image shown in FIG. 14C, when stereo matching processing is performed by SAD, SAD is expressed as [Equation 3]. Becomes 15.

[Equation 3]
ABS {(100-105) + (120-115) + (140-145) + (90-95) + (100-95) + (80-85) + (200-205) + (230-225) + (120-125)} = | −15 | = 15

Further, when the stereo matching process is performed on the reference image and the comparison image captured in the dark environment shown in FIGS. 14D and 14E by SAD, SAD becomes 0 as shown in [Expression 4].
[Equation 4]
ABS {(5-5) + (20-20) + (15-15) + (15-15) + (10-10) + (5-5) + (20-20) + (10-10) + (0-0)} = 0

Furthermore, when noise of about ± 5, for example, is included in the image as in the small region of the reference image captured in the dark environment shown in FIG. 14F, when stereo matching processing is performed by SAD, [Equation 5 ], The SAD becomes 15.
[Equation 5]
ABS {(5-10) + (20-15) + (15-20) + (15-20) + (10-5) + (5-10) + (20-25) + (10-5) + (0-5)} = | −15 | = 15

  As shown in FIGS. 14A to 14C, when the signal level is high with respect to noise as in an image captured in a bright environment, and as shown in FIGS. 14D to 14F, the signal level is dark. There is no difference in the correlation value obtained by the stereo matching process when the signal level is small and close to the noise as in the image captured in the environment.

  That is, it is difficult to determine whether the correlation value obtained by the stereo matching process is due to a signal or noise. Then, in the case of an image in which a sufficiently large signal with respect to noise cannot be obtained (S / N is small), even if an incorrect correspondence is searched between the pixels of the reference image and the comparison image due to the influence of noise, From the magnitude of the correlation value between pixels, it is difficult to determine whether or not the correlation value is appropriate, and a parallax vector Vp with low reliability is obtained, and calculation of erroneous distance information cannot be excluded.

  FIG. 15 is a diagram illustrating an example of noise included in an image captured by a camera whose aperture is fully closed.

  If the entire area of the image is black, that is, if the digital value of the luminance is the same (0 value) in the entire area of the image, all points are searched as corresponding points in the corresponding point search. If it does so, parallax cannot be calculated and a parallax image will not be obtained. However, as shown in FIG. 15, if noise is included in the image, that is, if the image includes a luminance value other than 0 value, a small area of the reference image and a small value of the comparison image are detected in the corresponding point search. There is a possibility that the region is associated with a point including noise. Then, a disparity vector is obtained based on the corresponding points searched in error, and an incorrect disparity image is obtained.

  In order to prevent erroneous correspondence due to noise, it is also possible to repeat the removal process of fixed pattern noise, but it is difficult to eliminate the effect of fixed pattern noise and random noise while maintaining the correlation of stereo images, It is difficult to prevent erroneous correspondence in stereo matching processing by repeating noise removal processing.

  Specifically, the case where the conventional stereo image processing apparatus 30 is used as a stereo-type vehicle exterior monitoring apparatus is examined. For example, before sunset when the brightness of each image captured by the cameras 3a and 3b is sufficiently obtained, the influence of noise on the captured image is small. Then, a parallax image is obtained by stereo matching processing based on a stereo image captured by the cameras 3a and 3b, and a distance image of the target object 2 such as a road surface or a vehicle can be acquired using the parallax image. Can be extracted as an obstacle.

  However, for example, after sunset when the brightness of each image captured by the cameras 3a and 3b is not sufficiently obtained, the influence of noise on the captured image increases. Then, when a parallax image is obtained by stereo matching processing based on a stereo image captured by the cameras 3a and 3b, the entire parallax image includes an incorrect parallax vector Vp different from the actual target object 2. It becomes. Since this wrong parallax vector Vp is discontinuous and random parallax, it is impossible to calculate an accurate distance image of a region where the target object 2 such as a road surface or a vehicle exists using the parallax image, and the target The object 2 cannot be extracted as an obstacle.

  The present invention has been made to solve such a problem, and reduces the miscorrespondence of stereo matching processing caused by noise included in an image captured by a stereo camera, and obtains a highly reliable parallax image and distance image. It is an object of the present invention to provide a stereo image processing method that makes it possible to do this.

  In order to solve the above problems, in the present invention, the luminance value of a pixel constituting a reference image captured by a camera constituting a stereo camera is compared with a predetermined threshold value, and the luminance value of the pixel is the threshold value. Is smaller than 0, the luminance value of the pixel is replaced with 0 value, when the luminance value of the pixel is 0 value, the parallax vector of the pixel is set to 0 value, and in other cases, the stereo matching process is performed. There is provided a stereo image processing method characterized in that a corresponding point in a comparison image captured by a camera constituting a camera is searched, and a parallax image generated according to a distance to an area captured by the stereo camera is obtained.

  Further, in the present invention, a parallax image generated according to the distance from the reference image captured by the stereo camera and the comparison image to the area captured by the stereo camera is obtained, and the reference image captured by the camera constituting the stereo camera is configured. When the luminance value of the pixel of the reference image is smaller than the threshold value, the positional information of the pixel of the reference image is recorded and recorded. The position information of the pixel of the reference image is compared with the position information of the pixel of the parallax image, and when the position information matches, the value of the matching pixel position for the parallax image is replaced with a zero value. A stereo image processing method is provided.

  ADVANTAGE OF THE INVENTION According to this invention, the stereo image processing which reduces the miscorrespondence of the stereo matching process resulting from the noise contained in the image imaged with the stereo camera, and can acquire a highly reliable parallax image and distance image Can provide a method.

  A stereo image processing method according to the present invention will be described with reference to the accompanying drawings.

[First Embodiment]
A first embodiment of a stereo image processing method according to the present invention will be described with reference to FIGS.

  FIG. 1 is a block diagram showing the configuration of the stereo image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 1, the stereo image processing apparatus 1 is picked up by a stereo camera 3 composed of a set of two cameras 3a and 3b that pick up a stereo image of a region where the target object 2 exists, and the camera 3a. A noise determination / removal unit 9 that determines and removes noise from the captured image, and an image that has been subjected to noise determination and removal by the noise determination / removal unit 9 and an image captured by the camera 3b by stereo matching processing The stereo camera processing unit 5 obtains a parallax image of a region where the target object 2 exists and acquires distance distribution information (distance image).

  The stereo image processing apparatus 1 is mounted on a moving body such as an automobile or a helicopter, for example, and processes a pair of images obtained by taking a stereo image of a target scene to obtain three-dimensional distance information, and based on the obtained distance information, the surrounding environment Used for devices that recognize the self-position.

  The two cameras 3a and 3b constituting the stereo camera 3 are cameras incorporating a solid-state image sensor such as a CCD or a CMOS. One camera 3a is synchronized with each other as a main camera that captures a reference image in the stereo matching process, and the other camera 3b is a sub camera that captures a comparative image in the stereo matching process. The arrangement is such that the vertical axes of the imaging surfaces are parallel to each other. Each of the cameras 3a and 3b includes an amplifier and an A / D converter for processing an analog imaging signal, and various functional circuits for image processing configured by, for example, a highly integrated FPGA, and the captured image is displayed with a predetermined luminance level. The digital image data is converted into tone digital image data and output to an original image memory (not shown).

  The noise determination / removal unit 9 compares the luminance value of each pixel of the reference image captured by the camera 3a and stored in the original image memory with a predetermined threshold value Nth, and the luminance value of the pixel is a predetermined value. When it is smaller than the threshold value Nth, the luminance value of the pixel is replaced with a predetermined value, for example, 0 value. The predetermined threshold value Nth is obtained by obtaining a histogram for an image captured by the camera 3a in which light incidence is blocked by, for example, fully closing the diaphragm, that is, a dark image captured by the camera 3a. An envelope value in consideration of fluctuations such as random noise is set for the luminance values of all the pixels generated by the offset.

  The stereo camera processing unit 5 </ b> A includes a corresponding point detection unit 6 </ b> A and a distance calculation unit 7.

  The corresponding point detection unit 6A performs noise determination / removal by the noise determination / removal unit 9, the reference image stored in the original image memory, and the comparison image captured by the camera 3b and stored in the original image memory. The pixels of the comparison image are associated with each pixel of the reference image by stereo matching processing. In this stereo matching process, for example, a corresponding point search is performed according to an evaluation function such as a correlation function disclosed in Patent Document 1 as Expression (3). Also, the corresponding point detection unit 6A obtains, as the parallax vector Vp, a pixel shift (= parallax) that occurs according to the distance to the area where the target object 2 exists for each pixel of the reference image using the searched corresponding points. . When this disparity vector Vp is arranged for each pixel constituting the reference image, it constitutes a disparity image.

  The corresponding point detection unit 6A determines and removes noise by the noise determination / removal unit 9, and when the luminance value of the pixel of the reference image stored in the original image memory is zero, The stereo matching process is not performed, and the parallax vector Vp is set to 0 (no parallax).

  The distance calculation unit 7 calculates a distance Z obtained by quantifying distance information, which is perspective information from the disparity vector Vp obtained by the corresponding point detection unit 6A to the area where the target object 2 exists. If this distance Z is arranged for each pixel constituting the reference image, it constitutes distance distribution information (distance image).

  FIG. 2 is a flowchart for explaining the stereo image processing method according to the first embodiment of the present invention.

  As shown in FIG. 2, first, in step S1, the noise determination / removal unit 9 receives an input of a pixel constituting a reference image captured by the camera 3a and stored in the original image memory. And the threshold value Nth. When the luminance value of the pixel is smaller than the predetermined threshold value Nth, the process proceeds to step S2. In other cases, the process proceeds to step S3.

  In step S2, the noise determination / removal unit 9 replaces the luminance value of the reference image pixel, for which the luminance value is determined to be smaller than the predetermined threshold Nth in step S1, with, for example, a zero value, and then proceeds to step S3. move on.

  In step S3, the corresponding point detection unit 6A replaces the luminance value in step S3, or the pixel of the reference image that the noise determination / removal unit 9 determines in step S1 that the luminance value is smaller than the predetermined threshold value Nth. If the luminance value of the pixel is 0, the process proceeds to step S4. In other cases, the process proceeds to step S5.

  In step S4, the corresponding point detection unit 6A sets the disparity vector Vp of the pixel for which the luminance value is determined to be 0 in step S3 to 0 (no parallax), and ends the process.

  In step S5, the corresponding point detection unit 6A uses the reference image stored in the original image memory and the comparison image, and pays attention to the pixel in the reference image determined in step S3 that the luminance value is equal to or greater than the predetermined threshold value Nth. As a pixel, a corresponding point in the comparison image is searched by stereo matching processing. By this corresponding point search, the corresponding point in the comparative image corresponding to the target pixel in the reference image is specified. Next, using the corresponding point in the searched comparison image and the target pixel of the reference image, the pixel shift (= parallax) generated according to the distance Z to the region where the target object 2 exists is calculated as the parallax of the target pixel. The vector Vp is obtained and the process is terminated.

  This stereo matching process is a process for searching for a corresponding point in the comparison image associated with each pixel in the reference image in accordance with an evaluation function such as a correlation function disclosed in Patent Document 1 as Equation (3), for example. This is done by using at least the small area including the target pixel of the reference image stored in the original image memory and the small area including the corresponding point of the comparison image in the corresponding point search, and comparing these areas.

  When the disparity vector Vp is obtained for the pixel on the reference image captured by the camera 3a by the stereo image processing shown in FIG. 2, the distance calculation unit 7 determines each disparity vector Vp based on the output disparity vector Vp of each pixel. The distance Z for each pixel is calculated. The stereo camera processing unit 5 outputs the disparity vector Vp obtained for all the pixels of the reference image as a disparity image, and the calculated distance Z as distance distribution information (distance image).

  Next, setting of the threshold value Nth for specifying a pixel for which stereo matching processing is not performed will be described.

  FIG. 3 is a diagram illustrating an example of a histogram of an image captured in a state where light incidence is blocked by the camera configuring the stereo image processing apparatus according to the first embodiment of the present invention.

  As shown in FIG. 3, when a histogram of an image captured by the camera 3a in which light incidence is blocked by fully closing the aperture, that is, a dark image captured by the camera 3a is obtained, the luminance value is obtained. Generation of noise including an offset in a range of about 10 to 20 can be confirmed.

  In an image captured by the camera 3a with the aperture fully closed, ideally, the histogram concentrates on a zero value. However, the dark image captured by the camera 3a includes pixels whose luminance value is not zero due to noise or offset.

  When a parallax image of an image including noise is obtained by stereo matching processing, a parallax vector Vp with low reliability is obtained over the entire parallax image, and distance distribution information (distance image) with low reliability is calculated as distance information. .

  Therefore, a threshold value Nth that envelops the maximum value of the luminance value that appears in the histogram of a completely dark image is set.

  FIG. 4 is a diagram illustrating an example of the luminance value of a small area of an image captured by the camera constituting the stereo image processing apparatus according to the first embodiment of the present invention, and (A) is captured by the camera. It is a figure which shows an example of the luminance value of the small area | region of the reference | standard image containing noise, (B) is 0 value with the luminance value of the pixel whose luminance value is smaller than a threshold value from the small area shown by (A). It is a figure which shows the example replaced by.

  Here, the threshold value is set to 15 as an example of a threshold value that envelops the maximum value of the luminance value that appears in the histogram of the dark image captured by the camera 3a.

  As shown in FIG. 4A, when the threshold value is set to 15 for a small region of the reference image including noise, the luminance value is smaller than the threshold value 15 as shown in FIG. The pixel is replaced with a zero value.

  In the stereo image processing apparatus 1, when the luminance value of the target pixel of the reference image (the central pixel of the 3 × 3 block in the example of FIG. 4) is 0, the small region including the target pixel is excluded from the stereo matching process. The

  At the pixel position of the reference image whose luminance value is smaller than the threshold value Nth, the parallax vector Vp is set to 0, and the distance Z is not calculated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

  Furthermore, the influence of noise due to auto gain control can be eliminated by setting the value that envelops the maximum value of the histogram of the image captured by the camera with the aperture fully closed as the threshold value Nth. Usually, a CCD camera or a CMOS camera has a low dynamic range of sensitivity. Therefore, an auto gain control is incorporated in order to cope with a bright imaging environment to a dark imaging environment. Since a sufficiently high signal level can be obtained in a bright imaging environment, the amplification by auto gain control is low and the noise level is relatively low. However, in a dark imaging environment, the signal level is low, the amplification level by auto gain control is high, and the noise level is relatively high.

  In this way, by setting the threshold value Nth using an image captured in a dark imaging environment where the relative noise level is high, it is possible to determine whether the image has a high S / N or a low image. The threshold value Nth can eliminate the influence of noise.

  Note that pixels having a luminance value smaller than the threshold value Nth enveloping the maximum noise value obtained from the histogram are excluded from the stereo matching process. Therefore, if this threshold value Nth is set too high, it can be used for the stereo matching process. The number of pixels decreases, and the number of pixels from which the parallax vector Vp and distance information can be calculated decreases. Therefore, it is desirable that the threshold value Nth is not too high with respect to the maximum noise value. Note that a margin (+ α) value may be added to the maximum value of the histogram within the range where there is no problem in calculating the parallax vector Vp and distance information. For example, by setting the value of the margin (+ α) to be larger than the noise variation generated by each of the plurality of cameras, it is possible to set the threshold value Nth of the plurality of cameras as a representative of one camera. .

  Note that in the stereo image processing apparatus 1 and the stereo image processing method according to the present embodiment, instead of replacing the pixel value having a luminance value smaller than the threshold value Nth with a zero value, a signal that is not normally included in the signal at the time of imaging. It may be replaced with the value of. Further, when the value of a pixel having a luminance value smaller than the threshold value Nth is set to 0 value, the stereo matching process may be performed with 0 value.

  In the stereo image processing apparatus 1 and the stereo image processing method according to the present embodiment configured as described above, first, an image is captured under a condition in which light is not received by the solid-state imaging device that configures the camera that captures the reference image. Next, the maximum value of the noise level included in this image is obtained from the histogram. Thereafter, imaging using a set of two cameras is performed, and pixels whose luminance value is smaller than the value that envelopes the maximum noise value are excluded from the stereo matching process from the reference image. Then, the parallax image and the distance information can be obtained by performing the stereo camera process without changing the pixel value whose luminance value is equal to or greater than the maximum noise value. That is, the stereo matching process is performed on the pixels having a large S / N (the influence of noise is small) while maintaining the reliability, and the influence of the pixels having a small S / N (the influence of noise is large) can be eliminated.

  In the stereo image processing apparatus 1 and the stereo image processing method according to the present embodiment, it is possible to eliminate the erroneous correspondence of the stereo matching process due to the influence of noise from the stereo camera image captured in the dark imaging environment where noise is dominant, and to improve reliability. Based on the parallax pixels for which a high parallax vector Vp is obtained, a distance image from which erroneous detection of the distance Z is eliminated can be obtained.

  According to the stereo image processing apparatus 1 and the stereo image processing method according to the present embodiment, it is possible to reduce erroneous correspondence of stereo matching processing caused by noise included in an image captured by a stereo camera, and to provide a highly reliable parallax image and A stereo image processing method that makes it possible to acquire a distance image can be provided.

[Second Embodiment]
A second embodiment of the stereo image processing method according to the present invention will be described with reference to FIGS.

  FIG. 5 is a block diagram showing a configuration of a stereo image processing apparatus according to the second embodiment of the present invention.

  In this stereo image processing apparatus 1A, the same components as those in the stereo image processing apparatus 1 of the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  As shown in FIG. 5, the stereo image processing apparatus 1 </ b> A is picked up by a stereo camera 3 including a pair of cameras 3 a and 3 b that pick up a stereo image of a region where the target object 2 exists, and the camera 3 a. A first noise determination / removal unit 10 that performs noise determination / removal on the captured image, a second noise determination / removal unit 11 that performs noise determination / removal on the image captured by the camera 3b, A parallax image of a region where the target object 2 exists is obtained by stereo matching of an image in which noise is determined and removed by the first noise determination / removal unit 10 and the second noise determination / removal unit 11, and a distance distribution is obtained. It is comprised from the stereo camera process part 5 which acquires information (distance image).

  The first noise determination / removal unit 10 compares the luminance value of each pixel of the reference image captured by the camera 3a and stored in the original image memory with a predetermined threshold value Ntha, and the luminance value of that pixel. Is smaller than a predetermined threshold value Ntha, the luminance value of the pixel is replaced with a predetermined value, for example, 0 value. The predetermined threshold value Ntha is obtained by obtaining a histogram for an image captured by the camera 3a in which light incidence is blocked by, for example, fully closing the diaphragm, that is, a dark image captured by the camera 3a. An envelope value in consideration of fluctuations such as random noise is set for the luminance values of all the pixels generated by the offset.

  The second noise determination / removal unit 11 compares the luminance value of each pixel of the comparison image captured by the camera 3b and stored in the original image memory with a predetermined threshold value Nthb, and the luminance value of that pixel. Is smaller than a predetermined threshold value Nthb, the luminance value of the pixel is replaced with a predetermined value, for example, 0 value. The predetermined threshold value Nth is obtained by obtaining a histogram for an image captured by the camera 3b whose light is blocked by the aperture being fully closed, that is, a dark image captured by the camera 3b. An envelope value in consideration of fluctuations such as random noise is set for the luminance values of all the pixels generated by the offset.

  The stereo camera processing unit 5 includes a corresponding point detection unit 6B and a distance calculation unit 7.

  The corresponding point detection unit 6B performs the determination and removal of noise by the first noise determination / removal unit 10, and determines the noise by the reference image stored in the original image memory and the second noise determination / removal unit 11. Then, the comparison image stored in the original image memory is associated with the pixel of the comparison image for each pixel of the reference image by stereo matching processing. In this stereo matching process, for example, a corresponding point search is performed according to an evaluation function such as a correlation function disclosed in Patent Document 1 as Expression (3). In addition, the corresponding point detection unit 6B obtains, as the parallax vector Vp, a pixel shift (= parallax) that occurs according to the distance to the region where the target object 2 exists for each pixel of the reference image using the searched corresponding points. .

  The corresponding point detection unit 6B determines and removes noise by the first noise determination / removal unit 10, and the luminance value of the pixel of the reference image stored in the original image memory is 0. In this case, the stereo matching process is not performed, and the parallax vector Vp is set to 0 value (no parallax). Further, the corresponding point detection unit 6B determines and removes noise by the second noise determination / removal unit 11, stores the original image memory, and compares the pixels of the comparison image associated with the pixels of the reference image. When the luminance value of the pixel is 0, the stereo matching process is not performed and the parallax vector Vp is set to 0 (no parallax).

  FIG. 6 is a flowchart illustrating a stereo image processing method according to the second embodiment of the present invention.

  As shown in FIG. 6, first, in step S11, the first noise determination / removal unit 10 receives an input of a pixel constituting a reference image captured by the camera 3a and stored in the original image memory, and brightness of the pixel. The value is compared with a predetermined threshold value Ntha. When the luminance value of the pixel is smaller than the predetermined threshold value Ntha, the process proceeds to step S12. In other cases, the process proceeds to step S13.

  In step S12, the first noise determination / removal unit 10 replaces the luminance value with, for example, a zero value for the pixel of the reference image that has been determined in step S11 that the luminance value is smaller than the predetermined threshold value Ntha. Proceed to step S15.

  In step S13, the second noise determination / removal unit 11 receives an input of a pixel constituting a comparison image captured by the camera 3b and stored in the original image memory, and the luminance value of the pixel and a predetermined threshold value Nthb. And compare. When the luminance value of the pixel is smaller than the predetermined threshold value Nthb, the process proceeds to step S14. In other cases, the process proceeds to step S15.

  In step S14, the second noise determination / removal unit 11 replaces the luminance value of the pixel with a zero value, for example, for the pixel of the comparative image that has been determined in step S13 that the luminance value is smaller than the predetermined threshold value Nthb. Then, the process proceeds to step S15.

  In step S15, the corresponding point detection unit 6B determines whether the first noise determination / removal unit 10 has determined that the luminance value is smaller than the predetermined threshold value Ntha in step S11, or the luminance value in step S12. If the luminance value of the pixel of the reference image in which is replaced is 0, the process proceeds to step S16. In other cases, the process proceeds to step S17.

  In step S <b> 16, the corresponding point detection unit 6 </ b> B sets the disparity vector Vp of the pixel for the reference image pixel determined to have a luminance value of 0 in step S <b> 15 to 0 (no parallax), and ends the process.

  In step S17, the corresponding point detection unit 6B uses the reference image stored in the original image memory and the comparison image, and pays attention to the pixel in the reference image determined in step S15 that the luminance value is equal to or greater than the predetermined threshold value Ntha. As a pixel, a corresponding point in the comparison image is searched by stereo matching processing. By this corresponding point search, a corresponding point in the comparative image corresponding to the target pixel in the reference image is specified, and the process proceeds to step S18.

  In step S18, the corresponding point detection unit 6A proceeds to step S19 when the corresponding point in the comparison image corresponding to the target pixel in the reference image identified in step S17 has a 0 brightness value. In other cases, the process proceeds to step S20.

  In step S19, the corresponding point detection unit 6B sets the disparity vector Vp of the pixel to a zero value (disparity) for the pixel of interest in the reference image corresponding to the corresponding point in the comparison image determined to have a luminance value of 0 in step S18. None) and the process is terminated.

  In step S20, the corresponding point detection unit 6A uses the corresponding point in the comparison image corresponding to the target pixel in the reference image specified in step S17 and the region where the target object 2 exists using the target pixel in the reference image. The pixel shift (= parallax) generated according to the distance Z is determined as the parallax vector Vp of the pixel of interest, and the process is terminated.

  This stereo matching process is a process for searching for a corresponding point in the comparison image associated with each pixel in the reference image in accordance with an evaluation function such as a correlation function disclosed in Patent Document 1 as Equation (3), for example. This is done by using at least the small area including the target pixel of the reference image stored in the original image memory and the small area including the corresponding point of the comparison image in the corresponding point search, and comparing these areas.

  When the parallax vector Vp is obtained for the pixel on the reference image captured by the camera 3a by the stereo image processing shown in FIG. 6, the distance calculation unit 7 determines each pixel based on the output parallax vector Vp of each pixel. The distance Z for each pixel is calculated. The stereo camera processing unit 5 outputs the calculated distance Z as distance distribution information (distance image) using the parallax vector Vp obtained for all pixels of the reference image as a parallax image.

  For the threshold values Ntha and Nthb, a margin (+ α) value may be added to the maximum value of the histogram within a range where there is no problem in calculating the parallax vector Vp and the distance information. For example, by setting the value of the margin (+ α) to be larger than the noise variation that occurs between the camera 3a and the camera 3b, the threshold values Ntha and Nthb between the camera 3a and the camera 3b are set to the values of either camera. The threshold value Nth can be set as a representative.

  In the stereo image processing apparatus 1A and the stereo image processing method according to the present embodiment, instead of replacing the pixel values having luminance values smaller than the threshold values Ntha and Nthb with 0 values, signals at the time of imaging are used. May be replaced with a specific value not normally included. Further, when the pixel value having a luminance value smaller than the threshold values Ntha and Nthb is set to 0 value, the stereo matching process may be performed with 0 value.

  In the stereo image processing apparatus 1 and the stereo image processing method according to the first embodiment, the luminance value of the target pixel of the reference image is equal to or higher than the threshold value Ntha, and the luminance value of the corresponding point of the comparison image corresponding to the target pixel is Even when the threshold value is smaller than the threshold value Nthb, the small area of the reference image on which the stereo matching process is performed is associated with the small area of the comparison image.

  In the stereo image processing apparatus 1A and the stereo image processing method according to the present embodiment, when the luminance value of the corresponding point in the comparison image is smaller than the threshold value Nthb, the target pixel of the reference image corresponding to the corresponding point is detected. The parallax vector Vp is not obtained. Therefore, the parallax vector Vp is set to 0 at the pixel position of the reference image corresponding to the pixel of the comparative image whose luminance value is smaller than the threshold value Nthb, and the distance Z is not calculated. The distance Z can be eliminated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

  In the stereo image processing apparatus 1A and the stereo image processing method according to the present embodiment, the reliability of the stereo matching process due to the influence of noise is eliminated from the stereo camera image captured in the dark imaging environment where noise is dominant. Based on the parallax pixels for which a high parallax vector Vp is obtained, a distance image from which erroneous detection of the distance Z is eliminated can be obtained.

  According to the stereo image processing apparatus 1A and the stereo image processing method according to the present embodiment, it is possible to reduce erroneous correspondence of stereo matching processing caused by noise included in an image captured by a stereo camera, and to provide a highly reliable parallax image and A stereo image processing method that makes it possible to acquire a distance image can be provided.

[Third Embodiment]
A third embodiment of the stereo image processing method according to the present invention will be described with reference to FIGS.

  FIG. 7 is a block diagram showing a configuration of a stereo image processing apparatus according to the third embodiment of the present invention.

  In this stereo image processing apparatus 1B, the same components as those in the stereo image processing apparatus 1 of the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  As shown in FIG. 7, the stereo image processing apparatus 1B is picked up by a stereo camera 3 including a pair of cameras 3a and 3b that pick up a stereo image of a region where the target object 2 exists, and the camera 3a. The noise determination unit 13 for determining the noise of the captured image, the noise position recording unit 14 for recording the pixel position information of the noise determined by the noise determination unit 13, and each of the images captured by the camera 3a and the camera 3b A stereo camera processing unit 5 that obtains distance distribution information (distance image) by obtaining a parallax image of an area where the target object 2 exists by stereo matching processing, and pixel position information recorded in the noise position recording unit 14 The noise position data changing unit 15 changes the distance distribution information acquired by the stereo camera processing unit 5.

  The noise determination unit 13 compares the luminance value of each pixel of the reference image captured by the camera 3a and stored in the original image memory with a predetermined threshold value Nth, and the luminance value of the pixel has a predetermined threshold value. When the value is smaller than the value Nth, the pixel position of the pixel is output to the noise position recording unit 14.

  The noise position recording unit 14 records the pixel position of the reference image output from the noise determination unit 13. That is, the noise position recording unit 14 records the pixel positions of pixels in the reference image captured by the camera 3a and having a luminance value smaller than the predetermined threshold value Nth.

  The stereo camera processing unit 5 includes a corresponding point detection unit 6C and a distance calculation unit 7.

  Corresponding point detection unit 6C captures the reference image captured by camera 3a and stored in the original image memory, and the comparison image captured by camera 3b and stored in the original image memory by performing a stereo matching process. The pixels of the comparison image are associated with each other. In addition, the corresponding point detection unit 6 obtains, as the parallax vector Vp, a pixel shift (= parallax) that occurs according to the distance to the area where the target object 2 exists for each pixel of the reference image using the searched corresponding points. .

  The noise position data changing unit 15 changes the value of the pixel position recorded in the noise position recording unit 14 to, for example, 0 value for the distance distribution information (distance image) acquired by the distance calculator 10. The noise position data changing unit 15 can also be configured to change the value of the pixel position recorded in the noise position recording unit 14 to, for example, 0 value for the parallax image calculated by the corresponding point detection unit 6.

  FIG. 8 is a flowchart for explaining a stereo image processing method according to the third embodiment of the present invention.

  As shown in FIG. 8, first, in step S21, the corresponding point detection unit 6C uses the reference image stored in the original image memory and the comparison image, and compares the pixel in the reference image as a target pixel by stereo matching processing. Search for corresponding points in the image. By this corresponding point search, the corresponding point in the comparative image corresponding to the target pixel in the reference image is specified. Next, using the corresponding point in the searched comparison image and the target pixel of the reference image, the pixel shift (= parallax) generated according to the distance Z to the region where the target object 2 exists is calculated as the parallax of the target pixel. Obtained as vector Vp.

  The distance calculation unit 7 calculates the distance Z of the target pixel based on the parallax vector Vp of the target pixel of the reference image. The stereo camera processing unit 5 outputs the disparity vector Vp obtained for all the pixels of the reference image as a disparity image and the calculated distance Z as distance distribution information (distance image), and proceeds to step S22.

  In step S22, the noise determination unit 13 receives an input of a pixel constituting a reference image captured by the camera 3a and stored in the original image memory, and compares the luminance value of the pixel with a predetermined threshold value Nth. If the luminance value of the pixel is smaller than the predetermined threshold value Nth, position information of the pixel in the reference image is output to the noise position recording unit 14, and the process proceeds to step S23. In other cases, the process proceeds to step S24.

  In step S23, the noise position recording unit 14 records the position information of the pixel in the reference image for the pixel of the reference image that has been determined in step S22 that the luminance value is smaller than the predetermined threshold value Nth. Proceed to

  In step S24, the noise position data changing unit 15 uses the position information of the pixel whose luminance value is recorded by the noise position recording unit 14 in step S23 to be smaller than the predetermined threshold value Ntha, and the stereo camera processing unit 5 performs step S21. The output parallax image and the position information of the pixels of the distance image are compared. If the pixel position information recorded by the noise position recording unit 14 matches the pixel position information of the parallax image and the distance image output by the stereo camera processing unit 5, the process proceeds to step S25. In other cases, the process ends.

  In step S25, the noise position data changing unit 15 matches the pixel position information recorded by the noise position recording unit 14 in step S24 with the pixel position information of the parallax image and the distance image output by the stereo camera processing unit 5. The values of the parallax image and the distance image at the pixel position are replaced with 0 values, and the process is terminated.

  In the stereo image processing apparatus 1B and the stereo image processing method according to the present embodiment according to the present embodiment, the luminance value of the reference image is greater than the threshold value Nth from the parallax image and the distance image obtained by the stereo camera processing unit 5. Since the parallax vector Vp and the distance Z at a small pixel position are changed to 0 values, data such as noise removal is not changed in the reference image before performing the stereo matching process. That is, it can be avoided that the correlation value between the small areas to be associated when performing the stereo matching process becomes the minimum value by chance due to the data change of the reference image, and erroneous association is performed.

  According to the stereo image processing apparatus 1B and the stereo image processing method according to the present embodiment, a stereo image is obtained by using the conventional stereo image processing apparatus 30 in which the stereo camera 3 and the stereo camera processing unit 5 are integrated. Based on the luminance value of the image captured by the cameras 3a and 3b output from the processing device 30, and the parallax image and distance distribution information (distance image) acquired by the stereo camera process, the noise determination unit 13 determines the luminance of the reference image. The pixel position whose value is smaller than the threshold value Nth is specified, and the value of the pixel position of the distance image and the parallax image that coincides with this pixel position is set to 0 value. Z can be eliminated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

  Further, according to the stereo image processing device 1B and the stereo image processing method according to the present embodiment, the reference image and the parallax image are acquired from the stereo camera processing unit 5, and the luminance value of the reference image is greater than the threshold value Nth. By specifying a pixel position that becomes smaller and setting the value of the pixel position of the parallax image that coincides with this pixel position to 0, the parallax vector Vp and the distance Z obtained by the influence of noise can be eliminated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

[Fourth Embodiment]
A fourth embodiment of the stereo image processing method according to the present invention will be described with reference to FIGS.

  FIG. 9 is a block diagram showing a configuration of a stereo image processing apparatus according to the fourth embodiment of the present invention.

  In this stereo image processing apparatus 1C, the same components as those in the stereo image processing apparatus 1 of the first embodiment are denoted by the same reference numerals, and redundant description is omitted.

  As shown in FIG. 9, the stereo image processing apparatus 1 </ b> C is picked up by a camera 3 a and a stereo camera 3 including a pair of cameras 3 a and 3 b that pick up a stereo image of a region where the target object 2 exists. A first noise determination unit 17 that performs noise determination of a captured image, a first noise position recording unit 18 that records pixel position information of noise determined by the first noise determination unit 17, and a camera 3b. A second noise determination unit 19 for determining noise of the image captured in step 2; a second noise position recording unit 20 for recording pixel position information of the noise determined by the second noise determination unit 19; Stereo camera processing unit that obtains distance distribution information (distance image) by obtaining a parallax image of a region where the target object 2 exists by stereo matching processing of each image captured by the camera 3a and the camera 3b And a noise position data changing unit that changes the distance distribution information acquired by the stereo camera processing unit 5 based on the pixel position information recorded in the first noise position recording unit 18 and the second noise position recording unit 20. 15A.

  The first noise determination unit 17 compares the luminance value of each pixel of the reference image captured by the camera 3a and stored in the original image memory with a predetermined threshold value Ntha, and the luminance value of the pixel is predetermined. Is smaller than the threshold value Ntha, the pixel position of the pixel is output to the first noise position recording unit 18.

  The first noise position recording unit 18 records the pixel position of the reference image output from the first noise determination unit 17. That is, the first noise position recording unit 18 records the pixel positions of pixels in the reference image captured by the camera 3a and having a luminance value smaller than the predetermined threshold value Ntha.

  The second noise determination unit 19 compares the luminance value of each pixel of the comparison image captured by the camera 3b and stored in the original image memory with a predetermined threshold value Nthb, and the luminance value of the pixel is predetermined. Is smaller than the threshold value Nthb, the pixel position of the pixel is output to the second noise position recording unit 20.

  The second noise position recording unit 20 records the pixel position of the comparison image output from the second noise determination unit 19. That is, the second noise position recording unit 20 records the pixel positions of pixels in the comparison image captured by the camera 3b and having a luminance value smaller than the predetermined threshold value Nthb.

  The stereo camera processing unit 5 includes a corresponding point detection unit 6C and a distance calculation unit 7.

  Corresponding point detection unit 6C captures the reference image captured by camera 3a and stored in the original image memory, and the comparison image captured by camera 3b and stored in the original image memory by performing a stereo matching process. The pixels of the comparison image are associated with each other. In addition, the corresponding point detection unit 6C obtains, as the parallax vector Vp, a pixel shift (= parallax) that occurs according to the distance to the area where the target object 2 exists for each pixel of the reference image using the searched corresponding points. .

  For the distance distribution information (distance image) acquired by the distance calculator 10, the noise position data changing unit 15 </ b> A changes the pixel position value recorded in the first noise position recording unit 18 to, for example, 0 value. In addition, the noise position data changing unit 15 </ b> A uses the pixel position of the reference image associated with the pixel position recorded in the second noise position recording unit 20 for the distance distribution information (distance image) acquired by the distance calculator 10. The value of is changed to 0 value, for example.

  The noise position data changing unit 15A is configured to change the value of the pixel position recorded in the first noise position recording unit 18 to, for example, 0 value for the parallax image calculated by the corresponding point detection unit 6. You can also. In addition, the noise position data changing unit 15A sets the pixel position value of the reference image associated with the pixel position recorded in the second noise position recording unit 20 for the parallax image calculated by the distance calculator 10 to 0, for example. It can also be configured to change to a value.

  FIG. 10 is a flowchart illustrating a stereo image processing method according to the fourth embodiment of the present invention.

  As shown in FIG. 10, first, in step S31, the corresponding point detection unit 6C uses the reference image stored in the original image memory and the comparison image as a pixel of interest and compares them by a stereo matching process. Search for corresponding points in the image. By this corresponding point search, the corresponding point in the comparative image corresponding to the target pixel in the reference image is specified. Next, using the corresponding point in the searched comparison image and the target pixel of the reference image, the pixel shift (= parallax) generated according to the distance Z to the region where the target object 2 exists is calculated as the parallax of the target pixel. Obtained as vector Vp.

  The distance calculation unit 7 calculates the distance Z of the target pixel based on the parallax vector Vp of the target pixel of the reference image. The stereo camera processing unit 5 outputs the disparity vector Vp obtained for all the pixels of the reference image as a disparity image and the calculated distance Z as distance distribution information (distance image), and proceeds to step S32.

  In step S32, the first noise determination unit 17 receives an input of a pixel constituting a reference image captured by the camera 3a and stored in the original image memory, and calculates a luminance value of the pixel and a predetermined threshold value Ntha. Compare. When the luminance value of the pixel is smaller than the predetermined threshold value Ntha, the position information of the pixel in the reference image is output to the first noise position recording unit 18 and the process proceeds to step S33. In other cases, the process proceeds to step S34.

  In step S33, the first noise position recording unit 18 records the position information of the pixel in the reference image for the pixel of the reference image determined in step S32 that the luminance value is smaller than the predetermined threshold value Ntha. The process proceeds to step S34.

  In step S34, the second noise determination unit 19 receives an input of a pixel constituting a comparison image captured by the camera 3b and stored in the original image memory, and calculates a luminance value of the pixel and a predetermined threshold value Nthb. Compare. When the luminance value of the pixel is smaller than the predetermined threshold value Nthb, the position information of the pixel in the comparison image is output to the second noise position recording unit 20 and the process proceeds to step S35. In other cases, the process proceeds to step S36.

  In step S35, the second noise position recording unit 20 records the position information in the comparison image of the pixel of the comparison image that has been determined in step S34 that the luminance value is smaller than the predetermined threshold value Nthb. The process proceeds to step S36.

  In step S36, the noise position data changing unit 15A causes the position information of the pixel whose luminance value is recorded by the first noise position recording unit 18 in step S33 to be smaller than the predetermined threshold value Ntha, and the stereo camera processing unit 5 The position information of the pixels of the parallax image and the distance image output in step S31 is compared. If the pixel position information recorded by the first noise position recording unit 18 matches the pixel position information of the parallax image and the distance image output by the stereo camera processing unit 5, the process proceeds to step S37. In other cases, the process proceeds to step S38.

  In step S37, the noise position data changing unit 15A matches the pixel position information recorded by the first noise position recording unit 18 in step S36 with the pixel position information of the distance image output by the stereo camera processing unit 5. The values of the parallax image and the distance image at the pixel position are replaced with 0 values, and the process is terminated.

  In step S38, the noise position data changing unit 15A causes the stereo camera processing unit 5 to perform step S31 for the pixels of the reference image that the first noise determination unit 17 determines that the luminance value is equal to or greater than the predetermined threshold value Ntha in step S32. The position information of the pixels in the comparison image associated with the pixels of the reference image is acquired using the parallax image output in step S2. Next, the position information of the pixels of the comparison image is compared with the position information of the comparison image recorded by the second noise position recording unit 20 in step S35. If the position information of the pixel of the comparison image matches the position information of the comparison image recorded by the second noise position recording unit 20 in step S35, the process proceeds to step S39. In other cases, the process ends.

  In step S39, the noise position data changing unit 15A, the pixel position information of the reference image in step S38 and the position information of the pixel of the distance image output from the stereo camera processing unit 5 match the disparity image and distance. The value of the image is replaced with 0 value, and the process ends.

  In the stereo image processing device 1B and the stereo image processing method according to the present embodiment, the parallax at the pixel position whose luminance value is smaller than the threshold value Nth in the reference image from the parallax image and the distance image obtained by the stereo camera processing unit 5. Since the vector Vp or the distance Z is changed to a zero value, data such as noise removal is not changed on the reference image before performing the stereo matching process. That is, it can be avoided that the correlation value between the small areas to be associated when performing the stereo matching process becomes the minimum value by chance due to the data change of the reference image, and erroneous association is performed.

  According to the stereo image processing apparatus 1 </ b> C and the stereo image processing method according to the present embodiment, a stereo image is obtained using the conventional stereo image processing apparatus 30 in which the stereo camera 3 and the stereo camera processing unit 5 are integrated. Based on the luminance value of the image captured by the cameras 3a and 3b output from the processing device 30, and the parallax image and distance distribution information (distance image) acquired by the stereo camera process, the noise determination unit 13 determines the luminance of the reference image. By specifying a pixel position whose value is smaller than the threshold value Nth and setting the value of the pixel position of the distance image and the parallax image that coincides with this pixel position to 0, the parallax vector Vp obtained by the influence of noise or The distance Z can be eliminated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

  In the stereo image processing apparatus 1C and the stereo image processing method according to the present embodiment, the reference image and the parallax image are acquired from the stereo camera processing unit 5, and the luminance value of the reference image is smaller than the threshold value Nth. By specifying the pixel position and setting the value of the pixel position of the parallax image that coincides with this pixel position to 0, the parallax vector Vp and the distance Z obtained by the influence of noise can be eliminated. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

  According to the stereo image processing apparatus 1C and the stereo image processing method according to the present embodiment, when the luminance value of the corresponding point in the comparison image is smaller than the threshold value Nthb, the reference image corresponding to the corresponding point is noticed. The pixel parallax vector Vp is not obtained. Therefore, the parallax vector Vp and the distance Z obtained by the influence of noise can be excluded from the pixels of the reference image corresponding to the pixels of the comparative image whose luminance value is smaller than the threshold value Nthb. Therefore, it is possible to acquire a parallax image and a distance image composed of the parallax vector Vp and the distance Z that are less affected by noise and highly reliable.

1 is a block diagram showing a configuration of a stereo image processing apparatus according to a first embodiment of the present invention. The flowchart explaining the stereo image processing method which concerns on 1st Embodiment of this invention. The figure which shows an example of the histogram of the image imaged in the state which incidence | injection of light was interrupted | blocked by the camera which comprises the stereo image processing apparatus which concerns on 1st Embodiment of this invention. It is a figure which shows an example of the luminance value of the small area | region of the image imaged with the camera which comprises the stereo image processing apparatus which concerns on 1st Embodiment of this invention, (A) is a reference | standard including the noise imaged with the camera The figure which shows an example of the luminance value of the small area | region of an image, (B) shows the example which replaced the luminance value of the pixel whose luminance value is smaller than a threshold value into 0 value from the small area shown to (A). Figure. The block diagram which shows the structure of the stereo image processing apparatus which concerns on 2nd Embodiment of this invention. The flowchart explaining the stereo image processing method which concerns on 2nd Embodiment of this invention. The block diagram which shows the structure of the stereo image processing apparatus which concerns on 3rd Embodiment of this invention. The flowchart explaining the stereo image processing method which concerns on 3rd Embodiment of this invention. The block diagram which shows the structure of the stereo image processing apparatus which concerns on 4th Embodiment of this invention. The flowchart explaining the stereo image processing method which concerns on 4th Embodiment of this invention. The figure explaining the principle which calculates distance by a stereo camera. The block diagram which shows the structure of the conventional stereo image processing apparatus. The figure explaining the change of the signal level of the luminance value of the image by the brightness of an imaging environment, and the change of S / N by noise. It is a figure explaining the influence by the deterioration of S / N with respect to the search of the corresponding point of a stereo matching process, (A) is a figure which shows an example of the luminance value of the small area | region of the reference | standard image imaged in the bright environment, (B) ) Is a diagram showing an example of the luminance value of a small area of a comparative image captured in a bright environment, and (C) is a diagram showing an example of the luminance value of a small area of a reference image including noise captured in a bright environment. (D) is a figure which shows an example of the luminance value of the small area of the reference | standard image imaged in the dark environment, (E) is a figure which shows an example of the luminance value of the small area of the comparison image imaged in the dark environment (F) is a figure which shows an example of the luminance value of the small area | region of the reference | standard image containing the noise imaged in the dark environment. The figure which shows an example of the noise contained in the image imaged with the camera which fully closed the aperture stop.

Explanation of symbols

1, 1A, 1, B, 1C Stereo image processing device 2 Target object 3 Stereo camera 3a, 3b Camera 5 Stereo camera processing unit 6, 6A, 6B, 6C Corresponding point detection unit 7 Distance calculation unit 9 Noise removal / removal unit 10 First noise determination / removal unit 11 Second noise determination / removal unit 13 Noise determination unit 14 Noise position recording unit 15, 15A Noise position data change unit 17 First noise determination unit 18 First noise position recording unit 19 Second noise determination unit 20 Second noise position recording unit 30 Conventional stereo image processing apparatus

Claims (2)

Compare the brightness value of the pixels that make up the reference image captured by the camera that makes up the stereo camera with a predetermined threshold,
When the luminance value of the pixel is smaller than the threshold value, the luminance value of the pixel is replaced with a zero value,
When the luminance value of the pixel is 0, the parallax vector of the pixel is set to 0, and in other cases, a corresponding point in a comparison image captured by the camera constituting the stereo camera is searched by stereo matching processing. A stereo image processing method for obtaining a parallax image generated in accordance with a distance to an area captured by the stereo camera. Obtaining a parallax image generated according to the distance from the reference image captured by the stereo camera and the comparison image to the region captured by the stereo camera;
Compare the brightness value of the pixels that make up the reference image captured by the camera that makes up the stereo camera with a predetermined threshold,
When the luminance value of the pixel of the reference image is smaller than the threshold value, the position information of the pixel of the reference image is recorded,
The recorded position information of the pixel of the reference image is compared with the position information of the pixel of the parallax image, and when the position information matches, the value of the matched pixel position of the parallax image is replaced with a zero value. A feature of a stereo image processing method.

Images