JPH10289315A - Parallax calculation device and method, and distance calculation device and method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To avoid occlusion that is caused when a certain object is hidden by another and to obtain the correct parallax information in a wide range in a stereographic image process. SOLUTION: The objects 221 and 222 , are photographed by plural cameras for production of images A to I. An arithmetic processing circuit calculates the SDS(sum of squared differences) between the standard pixel of the image A that is generated by a reference camera and the reference pixels of images B to I generated by other cameras. Then the arithmetic processing circuit properly divides the cameras into groups to calculate the SSSD(sum of SSDs) in each group and detects a reference pixel that secures the minimum value SSSDmin as a corresponding point of the corresponding reference pixel. Thus, the arithmetic processing circuit measures the parallax between the detected corresponding point and the reference pixel and then measures the distances among the points of both objects 221 and 222 based on a trigonometrical survey theory to analyze the shape of each object.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera for photographing a reference image and a reference image of the same object with two or more cameras and associating pixels of these images with each other. A parallax calculating apparatus that obtains, for each pixel, a parallax with respect to the target, and a distance between the target and the two cameras;
The present invention relates to a distance calculation device (so-called stereo image processing device) and a method thereof.

[0002]

2. Description of the Related Art Parallax information between mutually corresponding pixels is obtained between a plurality of images obtained by photographing the same object from two places, and the distance between each point on the object or the object distance is calculated. A so-called stereo image processing device that measures the shape of an object is used.

However, when a conventional stereo image processing apparatus is used, for example, an object far from the camera is hidden behind an object close to the camera, and an occlusion (occlu
), parallax information between pixels, and the distance and shape of the target object may not be measured.

Further, when a conventional stereo image processing apparatus is used, for example, when an object has a repetitive pattern, it is not possible to correctly take correspondence between the pixels of a plurality of images, and Correct measurement of the parallax information, the distance and the shape of the target object may not be possible in some cases.

[0005] For example, see Reference 1, "A Video-rate Stereomachine (A Video-rate Stereomachine; Kanade, Kimura), Journal of the Robotics Society of Japan vol.13 No.3, pp.3
22 to pp. 326 1995) ”includes pixels corresponding to each other in a plurality of images obtained by photographing the same object from two or more places or by photographing with two or more cameras. There is disclosed a stereoscopic image processing apparatus that obtains parallax information between objects and measures the distance and the like of each point on an object. When the stereo image processing apparatus disclosed in Document 1 is used, it is possible to correctly correspond between pixels of a plurality of images when a target object has a repetitive pattern, but a defect due to occlusion is caused. Will still remain.

[0006] Further, for example, reference 2 “Hierarchical D
isparity Estimation Using Multiple Cameras-Hierarchical depth estimation method using multiple cameras-[Park, Inoue,
1996 Video Media Processing Symposium (IMPS9
6) Proposal] ”and Reference 3“ Stereo Matching Using Multiple Baseline Lengths (Okutomi and Kanade, IEICE Transactions D-II Vol. J75-D-II No.8 pp.1317-pp. 1327 19
(August 1992) ", a method using five cameras has been proposed.

However, in the method disclosed in Reference 2, since stereo image processing is performed using four sets of cameras whose epipolar lines corresponding to the directions of parallax are greatly different, for example, occlusion is performed in three directions. When parallax occurs, the parallax obtained from the two sets of cameras corresponding to these two directions can be excluded, but an error occurs in the parallax obtained from the set of cameras corresponding to the remaining one direction. It may have occurred. References 2 and 3
However, the problem caused by the occlusion still remains with the method disclosed in the above.

[0008]

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art, and when there are a plurality of objects, an object located at a far position is replaced with an object located at a near position. It is an object of the present invention to provide a parallax calculating device capable of minimizing a range in which occlusion hidden by an object occurs and obtaining correct parallax information in a wide range. Also, the present invention provides a parallax calculation that can correctly take correspondence between pixels of a plurality of images and obtain correct parallax information, for example, even when a target object has a repeated pattern. It is intended to provide a device.

It is another object of the present invention to provide a distance calculating apparatus capable of obtaining correct parallax information in a wide range and calculating a correct distance and shape of an object.
Another object of the present invention is to provide a parallax / distance calculation method capable of obtaining correct parallax information in a wide range and calculating a correct distance and shape of an object.

[0010]

In order to achieve the above object, a parallax calculating apparatus according to the present invention comprises: first image generating means for photographing an object to generate a first image; A plurality of lines are arranged on a plurality of straight lines passing through the image generating means and divided into a plurality of groups, and each object is photographed to generate four or more second images.
One or more plurality of second image generating means, and a correlation for determining a correlation between each pixel of the generated first image and each pixel of each of the plurality of second images using a predetermined evaluation function. Gender determining means, based on a value of the predetermined evaluation function of a pixel of each of the plurality of second images generated by the plurality of second image generating means included in each of the plurality of groups, Certainty determination for determining which of the groups the pixels of the plurality of second images generated by the plurality of second image generation means correspond to the pixels of the first image most surely Means,
Correspondence between each of the pixels of the first image and the plurality of second images generated by the plurality of second image generation means included in any of the plurality of groups determined to be most reliable. Parallax calculating means for calculating parallax between points.

Preferably, the plurality of second image generation means included in each of the plurality of groups include a pixel position of the first image, the first image generation means and the second image. A virtual straight line (epipolar line) which is given from the positional relationship of the generating means and indicates a range in which the corresponding points of the plurality of second images corresponding to the pixels of the first image exist. , Arranged on any one of the plurality of straight lines, wherein the correlation determining means corresponds to a predetermined pixel block having each pixel of the first image as a central pixel and each pixel of the first image. The correlation between a predetermined pixel block centering on a pixel of each of the plurality of second images on an epipolar line of each of the plurality of second images is determined using the predetermined evaluation function.

Preferably, the correlation determining means includes a first evaluation function whose value changes in accordance with the correlation between pixels of the first image and pixels of each of the plurality of second images. And determining the correlation between the pixels of the first image and the pixels of each of the plurality of second images, wherein the certainty determination unit determines the plurality of second images included in each of the plurality of groups. The plurality of second image generation units included in any of the plurality of groups are based on a total sum of the first evaluation function of the pixels of the plurality of second images generated by the image generation unit. It is determined whether the pixels of each of the generated second images correspond most reliably.

In the parallax calculating apparatus according to the present invention, the first
Image generating means and the second image generating means are arranged at different positions, and each of the five or more image capturing means captures the same object and generates image data (first image data and second image data). Camera 1 or moving image of the same object from different positions to generate image data 1
One camera.

For example, the first image generating means is placed at the intersection of a plurality of straight lines oriented in a plurality of directions on the same plane, and the second image generating means is placed on each of the plurality of straight lines. In a plurality of positions, including the first image generation means,
The cameras are arranged so that three or more cameras are included in one group.

Further, the second image generating means may include, on each of the plurality of straight lines, a position of a pixel of the image generated by the first image generating means, the first image generating means and the second image generating means. , A range in which the corresponding points of the plurality of second images corresponding to the pixels of the first image exist, that is, a virtual straight line (epipolar line) indicating the direction of parallax becomes common. Are arranged as follows.

By disposing the first image generating means and the second image generating means in this way, the parallax is always measured based on images obtained from three or more cameras having the same epipolar line. Therefore, the range in which the occlusion occurs can be reduced, and even if the object has a repetitive pattern, the first
(Hereinafter, also referred to as a reference image) and pixels of a second image (hereinafter, also referred to as a reference image) can be associated with each other, and occurrence of a parallax calculation error can be prevented.

The correlation determining means is given, for example, from the position of the k-th reference pixel and the positional relationship between the first image generating means and the second image generating means. An n × n pixel block having a reference pixel as a center pixel on a virtual straight line (epipolar line) indicating a range in which a corresponding reference pixel (corresponding point) exists;
The sum of squared differences (SSDs) of the differences between the corresponding pixels is sequentially calculated between the pixel block having the n × n configuration and the reference pixel as the center pixel. Evaluate the correlation between pixels.

In other words, the correlation determining means is, for example, SS
Using D as an evaluation function, the pixel value of each pixel included in a pixel block whose center pixel is the k-th reference pixel and the pixels included in a pixel block of the same configuration whose reference pixel on the epipolar line is a center pixel The value is substituted for the evaluation function, and it is determined that the smaller the difference between pixel blocks (the sum of squares of the difference), the higher the correlation.

The certainty evaluation means further calculates, for example, a total sum of SSDs for each group, and
Ds) is calculated. That is, when each group includes n cameras (the first image generation unit and the second image generation unit), the certainty evaluation unit determines one reference pixel and (n−
1) (n-1) SSs between each of the reference pixels
The value of D is cumulatively added for each pixel to calculate SSSD.
Further, the certainty evaluation unit detects a reference pixel having the smallest value of the calculated SSSD as a corresponding point k ′ of the pixel k of the first image.

The parallax calculating means calculates the parallax between the reference pixel k and the corresponding point k 'of the reference pixel k detected as described above, for example, in pixel units.

Further, the distance calculating apparatus according to the present invention includes a first image generating means for photographing an object to generate a first image, and a plurality of straight lines passing through the first image generating means. A plurality of second image generating means for arranging a plurality of second images, each of which is arranged into a plurality of groups, and each of which captures an object to generate four or more plurality of second images; A correlation determining unit configured to determine a correlation between each pixel of the first image and each pixel of each of the plurality of second images using a predetermined evaluation function; and the plurality of pixels included in each of the plurality of groups. The plurality of second image generation units included in any of the plurality of groups based on the value of the predetermined evaluation function of the pixel of each of the plurality of second images generated by the second image generation unit Generated by the plurality Pixels of two images, and certainty determining means for determining whether corresponding to each pixel of the most reliably the first image, and each pixel of the first image,
Parallax for calculating parallax between corresponding points of the plurality of second images generated by the plurality of second image generation units included in any of the plurality of groups determined to be most reliable. A calculating unit, a calculated parallax, and a position of the first image generating unit and a position of the plurality of second image generating units included in any of the plurality of groups determined to be most reliable Based on the relationship between the object and the first image generating means and the plurality of second image generating means included in any of the plurality of groups determined to be most reliable Distance calculating means for calculating the distance.

Preferably, the plurality of second image generation means included in each of the plurality of groups include a pixel position of the first image, the first image generation means and the second image. A virtual straight line (epipolar line) which is given from the positional relationship of the generating means and indicates a range in which the corresponding points of the plurality of second images corresponding to the pixels of the first image exist. , Arranged on any one of the plurality of straight lines, wherein the correlation determining means corresponds to a predetermined pixel block having each pixel of the first image as a central pixel and each pixel of the first image. The correlation between a predetermined pixel block centering on a pixel of each of the plurality of second images on an epipolar line of each of the plurality of second images is determined using the predetermined evaluation function.

Preferably, the correlation determining means includes a first evaluation function whose value changes in accordance with the correlation between the pixels of the first image and the pixels of each of the plurality of second images. And determining the correlation between the pixels of the first image and the pixels of each of the plurality of second images, wherein the certainty determination unit determines the plurality of second images included in each of the plurality of groups. The plurality of second image generation units included in any of the plurality of groups are based on a total sum of the first evaluation function of the pixels of the plurality of second images generated by the image generation unit. It is determined whether the pixels of each of the generated second images correspond most reliably.

In the distance calculating apparatus according to the present invention, the distance calculating means includes a parallax calculated in the same manner as the above-described parallax calculating apparatus according to the present invention, a first image generating means, and an image including a most reliable corresponding point. The distance between each point on the object and these image generating means is calculated based on the principle of triangulation based on the positional relationship of the group that generated the second image generating means with the second image generating means. Analyze the shape of the object.

In the parallax / distance calculation method according to the present invention, an object is photographed from a predetermined position to generate a first image, and the first image is generated and divided into a plurality of straight lines passing through the predetermined position. The object is photographed from each of the three or more included positions to generate four or more second images, and each of the pixels of the generated first image and the plurality of second images are generated. The predetermined evaluation of pixels of each of the plurality of second images generated by photographing at positions included in each of the plurality of groups is determined using a predetermined evaluation function to determine the correlation between each of the pixels of each of the images. Based on the value of the function, it is most certain that the pixels of the first image correspond to the pixels of the plurality of second images generated by photographing from the plurality of positions included in any of the plurality of groups. Determine if A corresponding point of each of the pixels of the first image and a corresponding one of the plurality of second images generated by photographing from the plurality of positions included in any of the plurality of groups determined to be the most reliable. Is calculated, based on the calculated parallax, and the positional relationship between the predetermined position and the plurality of positions included in any of the plurality of groups determined to be most reliable. Then, the distance between the target object and the plurality of positions included in any of the plurality of groups determined to be most certain in the predetermined position and the correspondence is calculated.

Preferably, the plurality of positions included in each of the plurality of groups include a position of a pixel of the first image, the predetermined position, and the plurality of positions included in each of the plurality of groups. And a virtual straight line (epipolar line) indicating a range where the corresponding points of the plurality of second images corresponding to the pixels of the first image are present. A predetermined pixel block which is arranged on any one of the straight lines and has each pixel of the first image as a central pixel, and each of the plurality of second images corresponding to each pixel of the first image. The correlation with a predetermined pixel block centered on the pixel of each of the plurality of second images on the epipolar line,
The determination is made using the predetermined evaluation function.

Preferably, the first evaluation function uses a first evaluation function whose value changes according to the correlation between pixels of the first image and pixels of each of the plurality of second images. A pixel of an image and a correlation between each pixel of the plurality of second images are determined, and each of the plurality of second images generated by shooting from the plurality of positions included in each of the plurality of groups is determined. Based on the sum of the first evaluation function of pixels, the correspondence of the pixels of the plurality of second images generated by photographing from the plurality of positions included in any of the plurality of groups is most certain. Is determined.

[0028]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Stereo image processing In order to facilitate understanding of an embodiment of the present invention, first, FIG.
The stereo image processing will be described with reference to (A) to (C). 1A to 1C are diagrams showing stereo image processing.

As described above, in the stereo image processing, pixels between a plurality of images obtained by photographing the same object with a camera from two or more directions are associated with each other, and parallax information between the corresponding pixels is obtained. The information is converted into distance information from the camera to the object, and the distance and / or shape of the object are measured.

[0030] That is, for example, as shown in FIG. 1 (A), when photographing an object 22 in the two cameras A10 ₁ and camera B10 _2, as shown in FIG. 1 (B), the camera A10 ₁ is Image 24a including image 220a of target object
(Reference image) is obtained, the image from the camera B10 ₂ including an image 220b of the object (reference picture) 24b is obtained,
The images 220a and 220b of the object include pixels (corresponding points k and k ') obtained by photographing the same portion on the object 22.

When the corresponding points between the reference image 24a and the reference image 24b are detected, the parallax between these corresponding points k and k 'can be obtained for each pixel as shown in FIG. The parallax between the corresponding points k and k ′ thus obtained,
Based on the angles (camera angles) and distances of the _two cameras A, B 10 ₁ and 10 ₂ , each point on the object 22 and the cameras A 10 ₁ and B 10 ₂ are applied by applying the principle of triangulation. Can be measured, and the shape of the object 22 can be analyzed from the distance to each point on the object 22.

In the area-based matching stereo image processing, a point (pixel) k on the object image 220a of the reference image 24a and a corresponding pixel (corresponding point) k 'on the object image 220b of the reference image 24b are calculated. For example, an area-based matching method is employed.

The area-based matching method uses the camera A1
For example, as shown by a broken line in FIG. 1 (B), based on the distance and angle (positional relationship) between O ₁ and the camera A10 ₁ and the position of the pixel on the target image 220a of the reference image 24a. Each pixel on the epipolar line, which is a line virtually drawn in the image 24b and indicates the range where the corresponding point k ′ of the pixel k in the reference image 24b exists, is set as a central pixel, and n × n pixels (for example, n = 5), a square pixel block of the reference image 24b including pixels, and a reference image 2 including n × n pixels with pixel k as the center pixel.
The correlation with the square pixel block 4a is determined using a predetermined evaluation function, and the reference image 2 showing the highest correlation is determined.
The central pixel of the pixel block of FIG.
Detected as

The reason for using an n × n pixel block to detect a symmetric point is that the influence of noise is reduced, the characteristics of the pattern of the pixels surrounding the pixel k, and the target point k ′ of the reference image 24b. This is to ensure the detection of the corresponding point by clarifying and judging the correlation with the feature of the pattern of the pixels surrounding the pixel. In particular, for the reference image 24a and the reference image 24b with little change, The use of a larger pixel block increases the certainty of detecting a corresponding point.

In the area-based matching method, as an evaluation function for evaluating the correlation between the pixel blocks, for example, the sum of the squares of the difference values of the pixel data of the corresponding pixels of the two pixel blocks (SSD; 1) Sum of absolute values of difference values of pixel data between corresponding pixels of two pixel blocks, or normalized cross-correlation (norma
lized cross correlation) is used. Hereinafter, a case where an SSD is used as an evaluation function of the area-based matching method will be described as an example.

[0036]

(Equation 1)

However, in equation 1, I (x, y) is
The pixel data of the pixel at the coordinates (x, y) in the reference image 24a is shown, and J (x, y) is the coordinates (x, y) in the reference image 24b.
y indicates pixel data of the pixel, η indicates parallax in the x direction, ξ indicates parallax in the y direction, and w indicates each pixel block.

That is, the stereo image processing is performed by the area-based matching method while changing the parallax (η, ξ) along the epipolar line in the reference image 24b corresponding to the pixel k in the reference image 24a. The SSD between the pixel blocks of the reference image 24a and the reference image 24b is sequentially obtained, the pixel of the reference image 24b whose SSD is the minimum value SSD _min is detected as the corresponding point k ′ of the pixel k, and further, the detected corresponding point k ′ Then, the disparity (η _min , ξ _min ) is substituted into Expression 2 to calculate the disparity ζ. The distance from the cameras A10 ₁ and B10 ₂ to each point on the object 22 can be calculated from the obtained parallax ζ and the positional relationship between the cameras A10 ₁ and B 10 ₂ as described above.

[0039]

(Equation 2)

The value of the evaluation function (SSD) The value of the SSD is based on the reference image 24a and the reference image 2
The smaller the difference between the pixel data values (pixel values) of the pixels included in the pixel block 4b and corresponding to each other, the smaller the difference.

Accordingly, the value of the minimum value SSD _min of the SSD corresponds to the certainty of the detection of the corresponding point k ′ in the reference image 24b of the pixel k of the reference image 24a, and the minimum value SSD _min
Is smaller, the probability that the correspondence between the pixel k and the corresponding point k ′ is correct increases, indicating that the detection is more reliable. Conversely, the larger the value of the minimum value SSD _min is, the larger the pixel k becomes. And the correspondence point k ′ is more likely to be incorrect, indicating that the detection is uncertain.

Problems of Area-Based Matching Method Next, problems of the area-based matching method will be described with reference to FIGS. Figure 2 (A), (B), the reference image 24a obtained by photographing the object 22 placed in front of the background 20 in the camera A10 ₁ and the camera B10 ₂
FIG. 9 is a diagram showing a reference image 24b. FIG. 3 is a diagram illustrating an example of erroneous detection of a corresponding point by the area-based matching method.

[0043] For example, as shown in FIG. 2 (A), obtained by photographing the object 22 which is pre-background 20 in the camera A10 ₁ and camera B10 _2, reference shown in FIG. 2 (B) When the corresponding point detection processing is performed on the image 24a and the reference image 24b using the area-based matching method, the pixel k of the boundary (edge) portion between the target 22 and the background of the reference image 24a shown in FIG. When detecting the corresponding point k ′ in the reference image 24b, erroneous detection may easily occur.

[0044] For example, if the pixel k disparity between corresponding points k '(η, ξ) is very large, as shown in FIG. 3 (a), in the reference image 24a of the camera A10 ₁ is taken, the object Image 200a in contact with image 220a
Background image 200b in the reference image 24b of the pixel k
Original corresponding point k 'of, in the reference picture 24b of the camera B10 ₂ is taken, as shown in FIG. 3 (c), be in a position away from the image 220a of the object.

In such a case, as shown in FIG. 3B, the pixel k of the reference image 24a is set as the central pixel, one of which is included in the image 220a of the object, and the other is the image 200 of the background.
In the reference image 24b, the original corresponding points of the pixels in the two regions of the pixel block having the n × n configuration (hereinafter, an example where n = 5) included in the reference image 24b are included in FIG.
As shown in (d), the parallax (η,
ξ) may be divided into two greatly different areas.

Further, for example, the background images 200a, 200
If the pattern of the object image 220a and the pattern of the object images 220a and 220b are significantly different, the image 220 of the object
a, 220b and the value of the SSD using the pixel block including the background images 200a, 200b
The value is smaller than the value of the pixel block including only a and 200b. Therefore, as shown in FIG.
Original corresponding point k ′ in reference image 24b of pixel k of a
Instead, the pixel k ″ in contact with the image 220b of the object is
The possibility that the pixel k is erroneously detected as a corresponding point in the reference image 24b increases. Note that such a defect is caused by the pattern of the background images 200a and 200b and the image 2 of the target object.
This generally occurs not only when the patterns of the images 20a and 220b are largely different but also when the patterns of the background images 200a and 200b and the images 220a and 220b of the object are different.

Such erroneous detection of corresponding points can be reduced by making the pixel block smaller, but when the pixel block is made smaller, the certainty of detecting the corresponding point by the area-based matching method decreases. . The embodiment of the present invention has been made in view of such a problem, and the effect of erroneous detection of a corresponding point that may occur at an edge portion of an image while maintaining high reliability of detection of a corresponding point by an area-based matching method. Is configured to be reduced.

Configuration of Distance Measuring Apparatus 1 Hereinafter, the configuration of the distance measuring apparatus 1 shown as an embodiment of the present invention will be described. FIG. 4 is a diagram showing a configuration of the distance measuring device 1 according to the present invention.

As shown in FIG. 4, the distance measuring device 1
Is composed from the base of the camera A10 _1, B10 ₂ and the distance measuring unit 12, the distance measuring unit 12, a camera interface (camera IF) circuit 120, the image memory 122,
Arithmetic processing circuit (CPU) 124, recording device (HDD) 1
26 and an output device 128.

The distance measuring device 1 uses these components to form the pixels of the reference image 24a and the reference image 24 obtained by photographing the object 22 with the cameras A10 ₁ and B10 _2.
b, the parallax 間 の between the corresponding pixels is measured, and the distance between the object 22 and the cameras A10 ₁ and B10 ₂ and the object 22 are determined based on the measured parallax ζ.
Is measured.

Components of Distance Measuring Apparatus 1 Each component of the distance measuring apparatus 1 will be described below.

Camera A10 ₁ , Camera B10 ₂ Camera A10 ₁ and camera B10 ₂ are, for example, two or more CCD video cameras fixed at different positions, or one or more movable CCD video cameras. The same background 20 and object 22 from different positions
(FIG. 2) is photographed, and a reference image 24a and a reference image 24b are respectively generated and output to the distance measuring unit 12.

Camera interface circuit 120 The camera interface circuit 120 is connected to the camera A10 _1.
And it receives the reference image 24a and the reference image 24b is input from the camera B10 _2, the image memory circuit 12
2 is output.

Image memory circuit 122 The image memory circuit 122 stores the reference image 24a and the reference image 24b input via the camera interface circuit 120 on a frame-by-frame basis.
Supply to 4

The arithmetic processing circuit 124 processing circuit 124, for example, DSP for image processing,
It comprises a control CPU, ROM, RAM, and other peripheral circuits (not shown), controls each component of the distance measuring device 1, and performs stereo image processing on the reference image 24a and the reference image 24b. , Camera A1
0 ₁ and measurement of parallax ζ of each point on the object 22 with respect to the camera B10 _2, the camera A10 ₁ and the camera B10 ₂
The distance between the object and each point on the object 22 is measured, and the shape of the object 22 is analyzed.
6 or output to the outside via the output device 128.

The stereo image processing performed by the arithmetic processing circuit 124 will be described below with reference to FIG. FIG. 5 (A),
(B) is a diagram showing a process of correcting the parallax に よ る by the arithmetic processing circuit 124 of the distance measuring device 1 (FIG. 4) according to the present invention.

Calculation of Parallax First, the arithmetic processing circuit 124 calculates the area of each pixel of the reference image 24a and each pixel of the reference image 24b using, for example, the SSD shown in Equation 1 as an evaluation function as described above. The disparity ζ ₁ (first disparity) shown in Expression 2 is calculated by the base matching method, the disparity image is generated in association with the corresponding pixel of the reference image 24a or the reference image 24b, and the arithmetic processing circuit 124 In the RAM in the internal memory.

Evaluation of Certainty of Detection of Corresponding Point Next, the arithmetic processing circuit 124 calculates the pixel k of the reference image 24a.
If the minimum value SSD _min of the SSD at the time when the corresponding point k ′ of the reference image 24b is detected is equal to or smaller than a predetermined threshold, the correspondence between the pixel k of the reference image 24a and the corresponding point k ′ is certain. Evaluate and, if not less than the predetermined threshold,
It is evaluated that the correspondence between the pixel k of 4a and the corresponding point k 'is uncertain. The threshold value used for detecting the corresponding point is set to an optimal value by, for example, an experiment.

Reference image 24b of pixel k of reference image 24a
If it is determined that the detection of the corresponding point k ′ is reliable, the arithmetic processing circuit 124 calculates the first parallax ζ ₁
Is the correct parallax そ の ま ま, and the pixel k of the reference image 24a is
When it is determined that the detection of the corresponding point k ′ in the reference image 24b is reliable, the arithmetic processing circuit 124 corrects the parallax よ う な as follows.

As shown in FIG. 5A, the parallax correction arithmetic processing circuit 124 uses the reference pixel k determined as having uncertain correspondence as the center pixel, and uses the n × n pixel used for detecting the corresponding point. Including blocks,
Each reference pixel included in a pixel block having an m × m configuration (m ≧ n; however, m = n is exemplified) larger than this pixel block, and each reference pixel included in a pixel block having this m × m configuration , The reference pixel indicating the smallest SSD value (SSD _min ) and the corresponding point are detected.

That is, the arithmetic processing circuit 124 has an n × n configuration in which each of the m × m reference pixels included in the m × m configuration pixel block having the reference pixel k of the reference image 24a as the center pixel. Of the reference image 24a, and n × m corresponding to the corresponding point of each of the m × m reference pixels included in the m × m configuration pixel block centered on the reference pixel k of the reference image 24a. mx between each pixel block in the reference image 24b having n configurations
Among the m SSD values, the smallest SSD value (SSD
_min ) is determined to be most likely to correspond to the corresponding point in the reference image 24a.

Further, the arithmetic processing circuit 124 calculates the parallax between the pixel of the reference image 24a determined to most accurately correspond to the corresponding point and the corresponding point, and calculates the pixel k in the parallax image and the corresponding point k ″. The disparity ζ ₁ is replaced to correct the disparity ζ.

The arithmetic processing circuit 124 generates a parallax image in which each value of the parallax ζ obtained as described above is associated with each pixel of the reference image 24a or the reference image 24b, and further, based on the parallax image. The distance between the camera A10 ₁ and the camera B10 ₂ and each point on the object 22 is measured, and / or the shape of the object 22 is analyzed, and these results are stored in the recording device 126, or Output to the output device 128.

[0064] As the meaning of the correction of the parallax, the pixel k of the reference image 24a, the first parallax zeta ₁ between its corresponding point k of the first detected reference image 24b as the corresponding point ", the reference In the vicinity of the pixel k of the image 24a, the reference image 24 giving the smallest value of the SSD
Replacing with the disparity between the pixel of a and the corresponding point k ′ means, for example, as shown in FIGS. 3B and 3D, at the boundary (edge) portion between the object 22 and the background 20. An erroneously detected corresponding point k "is discarded, and as shown in FIG. 5B, avoiding an edge portion where erroneous detection of the corresponding point is likely to occur,
This means that the parallax between the most probable corresponding point is obtained by using a pixel block in which the pixel of the background image 200a and the pixel of the object image 220a are not mixed.

[0065] Therefore, than the first parallax zeta ₁ value between the pixel k of the reference image 24a corresponding point k ", the reference image 24
towards the second parallax zeta ₂ values between the corresponding point k 'and the pixel k of a can be considered to be more accurate.

Operation of Distance Measuring Apparatus 1 The operation of the distance measuring apparatus 1 will now be described with reference to FIG. FIG. 6 is a flowchart showing the operation of the distance measuring device 1 according to the present invention.

As shown in FIG. 6, step 100 (S1
00), the distance measuring unit 12 of the distance measuring device 1
Multiple (two) from camera A10 ₁ and camera B10 ₂
(Reference image 24a and reference image 24b).

In step 102 (S102), the arithmetic processing circuit 124 of the distance measuring section 12
Is calculated between the pixel block of the n × n configuration having each of the pixels as the central pixel and the pixel block of the n × n configuration having each of the pixels of the reference image 24b on the epipolar line as the central pixel. The corresponding point is detected by associating the pixel of the image 24a with the central pixel of the pixel block of the reference image 24b giving the minimum value SSD _min . That is,
The arithmetic processing circuit 124 detects a corresponding point between the pixels of the reference image 24a and the pixels of the reference image 24b by the area-based matching method.

In step 104 (S104), the arithmetic processing circuit 124 determines each pixel of the reference image 24a and the reference image 24b detected in the processing of S102.
Calculate the first parallax ζ ₁ between the corresponding point and the reference image 2
A parallax image is generated in association with each pixel of the reference image 4a or the reference image 24b.

[0070] In step 106 (S106), the arithmetic processing circuit 124, corresponding to the calculated parallax zeta ₁ SS
The value of the minimum value SSD _min of D is compared with the above threshold value, and if the value of the minimum value SSD _min is larger than the threshold value, the reference image 24a determined to correspond most accurately to the corresponding point
In the pixel parallax zeta ₂ with its corresponding point, by replacing the parallax zeta ₁ in the parallax image is corrected.

In step 108 (S108), the arithmetic processing circuit 124 corrects the parallax image and the object 22 and the camera A10 ₁ obtained from the corrected parallax image.
And distance between the camera B10 _2, and the data indicating the obtained shape of the object 22 as a result of the analysis, and outputs to the recording device 126 and / or output device 128.

As described above, according to the distance measuring apparatus 1 of the present invention, the accuracy of parallax obtained by stereo matching image processing using the area-based matching method can be greatly improved, and sharpness can be obtained by shape analysis. Images of various objects can be obtained.

[0073] Incidentally, except that in Example, for example, the operation of the distance measuring device 1, the minimum value SSD _min is not only the parallax zeta ₁ showing a value greater than the threshold, the unconditional for all parallax zeta ₁ Can be changed so as to perform correction. In addition, if the calculation process and the correction process of the parallax ζ ₁ are performed in order (sequentially), much processing time is required. For example, when the calculation of the minimum SSD value SSD _min is completed, the parallax ζ ₁ is calculated. The operation of the distance measuring device 1 can be changed so that the calculation process and the correction process are performed in parallel.

In the correction process, S is applied to all the pixel blocks centered on the respective pixels included in the pixel block centered on the corresponding point detected first.
Since calculating the SD requires a lot of processing time, for example, the SSD is calculated only for a pixel block whose center pixel is an outer peripheral pixel of a pixel block whose center pixel is the corresponding point detected first, or The pixels included in the pixel block whose center pixel is the corresponding point detected first are set to S
The correction processing of the distance measuring device 1 can be changed so as to calculate SD.

The respective components of the distance measuring apparatus 1 in the embodiment are merely examples, and for example, as long as the same function and performance can be realized, it is determined whether the measurement is performed by software means or hardware means. It doesn't matter. The configuration of the pixel block used for the corresponding point detection process and the parallax correction process is an example, and the corresponding point detection process and the parallax correction process are performed using a pixel block having another configuration, for example, a rectangular pixel block. You may. Further, the distance measuring device 1 is configured to obtain the reference image 24a and the reference image 24b using two cameras, but the reference image 24b and the reference image 24b are obtained by using more cameras. It is possible to change the configuration of the distance measuring device 1 so as to obtain the image 24a and the reference image 24b, or to move one camera to obtain the reference image 24a and the reference image 24b.

Second Embodiment Stereo Image Processing Using Many Cameras First, referring to FIG. 7, in order to facilitate understanding of a second embodiment of the present invention, a large number of (three or more) cameras are used. Will be described. FIG. 7 is a diagram illustrating an example of the arrangement of cameras when performing stereo image processing using a large number (five or more) of cameras.

As described above, in stereo image processing, an object that is far from the camera may be hidden behind an object that is close to the camera, resulting in occlusion. This occlusion is particularly likely to occur at the contour of an object. For an object in which occlusion has occurred, the parallax ζ and the distance cannot be measured. For example, as shown in FIG. In the image obtained from a combination of two cameras, the range in which occlusion occurs was obtained from the combination of the other two cameras. By supplementing with an image, the range in which parallax, distance, and the like cannot be measured due to occlusion can be reduced.

As described above, in stereo image processing using a large number of cameras, if occlusion occurs in a combination of certain two cameras, an evaluation function of pixels in a range where occlusion occurs [eg, , SS above
By using the fact that the value of D (Equation 1) increases, an image obtained from the combination that gives the smallest SSD value among the camera combinations is selected, and parallax, distance, and the like are selected from the selected image. Measured.

In stereo image processing using two cameras, for example, if a flat object has a repetitive pattern, it may not be possible to detect a correct corresponding point. As shown in (3), in stereo image processing using three or more cameras, the occurrence of such inconvenience is achieved by performing an arithmetic operation using the sum value (SSSD) by cumulatively adding for each corresponding pixel. Can be avoided.

However, for example, three of the four cameras shown in FIG. 7 have occlusions, and a repetitive pattern appears in the range where occlusion occurs in these three cameras. In some cases, even in stereo image processing using a large number of cameras, one set of two
Since only measurement using a single camera can be performed, accurate parallax and distance cannot be obtained for this range. The second embodiment of the present invention is designed to solve such a problem.

Configuration of Distance Measuring Apparatus 2 FIG. 8 is a diagram showing the configuration of the distance measuring apparatus 2 shown as a second embodiment of the present invention. Note that, of the components of the distance measuring device 2 shown in FIG. 8, the same components as those of the distance measuring device 1 shown in FIG. 4 in the first embodiment are denoted by the same reference numerals.

As shown in FIG. 8, the distance measuring device 2 includes cameras A10 ₁ to 10N _n (2 <n) and a distance measuring unit 12. That is, the distance measuring device 2
Are two cameras A10 ₁ of the distance measuring device 1 (FIG. 4),
The configuration in which B10 ₂ is added to n units is adopted, and a process different from that of the distance measuring device 1 is performed. Hereinafter, a case where n = 9 will be described.

FIG. 9 is a diagram exemplifying the arrangement of the cameras A10 _{1 to} I10 ₉ of the distance measuring device 2 shown in FIG. As shown in FIG. 9, the distance measuring device 2 of the camera A10 ₁ ~I
₁₀₉ is arranged in four on baseline perpendicular to the center of the camera A10 _1, Group 1, the camera A10 _1, F10 _6, G10 ₇ including the camera A10 _1, H10 _8, I10 ₉
2, cameras A10 ₁ , D10 ₄ , E10
Group 3 including ₅ and cameras A10 ₁ and C10
₃ and D10 ₄ .

That is, the cameras A10 _{1 to} I10 ₉ are divided into four groups each including three cameras by using the camera A10 ₁ in an overlapping manner.
In each of the groups, three cameras are arranged so that the epipolar lines are the same.

Stereo Image Processing in Distance Measuring Apparatus 2 Hereinafter, stereo image processing performed by the distance measuring apparatus 2 to avoid occlusion will be described.

In the distance measuring device 2, the camera A10 ₁
～I10 ₉ (FIG. 8, FIG. 9), respectively, the distance measuring device 1
As in the case of the cameras A10 ₁ and B10 ₂ , the same object is photographed from different positions, an image is generated, and the distance measurement unit 12 is used.
To the camera interface circuit 120.

The camera interface circuit 120 outputs the images input from the cameras A10 _{1 to} I10 ₉ to the image memory circuit 122. Image memory circuit 122
Stores an image input through the image memory circuit 122 in a frame unit and supplies the image processing circuit 124 with the image.

[0088] processing circuitry 124, for example, as well as in a distance measuring device 1 (FIG. 4), in combination with each camera A10 ₁ and other camera B10 ₂ ~I10 ₉ by two and eight pairs, 8 In each of the sets, the image A generated by the camera A10 ₁ is used as a reference image,
Calculates SSD images B~I other cameras B10 ₂ ~I10 ₉ is generated as a reference image to evaluate the correlation between the pixels of these images.

Further, the arithmetic processing circuit 124 calculates the total value (SSSD) by cumulatively adding the SSD values generated by the cameras included in each group for each corresponding pixel, and based on the calculated SSSD values. Then, it is determined which of the groups of cameras has the most probable (highly correlated) correspondence between the pixels of the image generated by the camera.
The given value SSSSD _min, detected most correspondence likelihood (high correlation) have pixel as the corresponding point of the reference pixel.

The calculation of the SSD, that is, first, the arithmetic processing circuit 124 generates an n × n pixel block centered on the reference pixel k of the pixel A and an epipolar line of the image B corresponding to the reference pixel k of the image A The SSD between the n × n pixel blocks having the upper pixel as the center pixel is sequentially calculated, and the value of SSD _B corresponding to each reference pixel of the image B is calculated.

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block centered on the reference pixel k of the pixel A and a pixel block on the epipolar line of the image C corresponding to the reference pixel k of the image A. The SSD between the pixel block and the pixel block having the n × n configuration with the pixel as the center pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each reference pixel of the image C is calculated.
Calculate the value of (SSD _c ).

Similarly, the arithmetic processing circuit 124 includes an n × n pixel block centered on the reference pixel k of the pixel A, and a pixel block on the epipolar line of the image D corresponding to the reference pixel k of the image A. The SSD between the pixel block and the n × n pixel block having the pixel as the center pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each of the reference pixels of the image D is calculated.
The value of (SSD _D ) is calculated.

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block having the reference pixel k of the pixel A as a center pixel and a pixel block on the epipolar line of the image E corresponding to the reference pixel k of the image A. The SSD between the pixel block and the n × n pixel block having the pixel as the center pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each of the reference pixels of the image E is calculated.
The value of (SSD _E ) is calculated.

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block centered on the reference pixel k of the pixel A and a pixel block on the epipolar line of the image F corresponding to the reference pixel k of the image A. The SSD between the pixel block and the n × n pixel block having the pixel as the center pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each of the reference pixels of the image F is calculated.
Calculate the value of (SSD _F ).

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block having the reference pixel k of the pixel A as the center pixel, and a pixel block on the epipolar line of the image G corresponding to the reference pixel k of the image A. The SSD between the pixel block and the pixel block having the n × n configuration having the central pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each of the reference pixels of the image G is calculated.
Calculate the value of (SSD _G ).

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block centered on the reference pixel k of the pixel A and a pixel block on the epipolar line of the image H corresponding to the reference pixel k of the image A. The SSD between the pixel block and the n × n pixel block having the central pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each of the reference pixels of the image H is calculated.
Calculate the value of (SSD _H ).

Similarly, the arithmetic processing circuit 124 generates an n × n pixel block centered on the reference pixel k of the pixel A and a pixel block on the epipolar line of the image I corresponding to the reference pixel k of the image A. The SSD between the pixel block of the n × n configuration having the pixel as the center pixel is sequentially calculated, and the minimum value SSD _{min of the} SSD corresponding to each reference pixel of the image I is calculated.
Calculate the value of (SSD _I ).

Calculation of SSSD and Detection of Corresponding Point Next, the arithmetic processing circuit 124 calculates the SS of the corresponding pixel of the image generated by the camera included in each of the four groups.
The value of D is cumulatively added to calculate SSSD for each group. If the number of camera sets included in the group is m, the SSSD is defined by Expression 3.

[0099]

(Equation 3)

[0100] That is, the arithmetic processing circuit 124, each reference pixel of the image A in group 1, and the SSD _H between each reference pixel in the image H, each reference pixel of the image A, and the reference pixels of the image I SSD between each
_Is added to each corresponding pixel, and SS of group 1 is added.
To calculate the SD _1.

Similarly, the arithmetic processing circuit 124 determines whether each of the reference pixels of the image A of the group 2 and the image F
And SSD _F between the reference pixels each, each reference pixel of the image A, and, the SSD _G between each reference pixel of the image G, is added to each corresponding pixel group 2
To calculate the SSSD _2.

Similarly, the arithmetic processing circuit 124 determines whether each of the reference pixels of the image A of the group 3 and the image D
And SSD _D between the reference pixels each, each reference pixel of the image A, and, Group 3 and SSD _E between each reference pixel in the image E, is added to each corresponding pixel
To calculate the SSSD _3.

Similarly, the arithmetic processing circuit 124 determines whether each of the reference pixels of the image A of the group 4 and the image B
And SSD _B between the reference pixels each, each reference pixel of the image A, and, the SSD _c between each reference pixel of the image C, and then added to each corresponding pixel group 4
To calculate the SSSD _4.

The arithmetic processing circuit 124 calculates the minimum value of the SSDS _{1 to} SSSD ₄ generated as described above.
_The pixel giving _min is detected as the corresponding point of the reference pixel.

[0105] Examples of occlusion that occurs below using a specific example shown in FIG. 10, further illustrating the calculation and the corresponding point detection SSSD. Figure 10 uses the camera A10 ₁ ~I10 ₉ of the distance measuring device 2 arranged as shown in FIG. 9 (FIG. 8), the camera A10 ₁ ~I10
₉ is a diagram illustrating an image A~I obtained by photographing the object 22 ₂ in the object 22 ₁ and farther in a position close to.

For example, when the cameras A10 _{1 to} I10 ₉ are used to photograph the object 22 ₁ near the cameras A10 _{1 to} I10 ₉ and the object 22 ₂ far from the cameras A10 _{1 to} I10 9, _From each of ₉
The images A to I shown in are obtained, and among these images,
Cameras A10 ₁ , D10 ₄ , E1 included in group 3
0 ₅ there is not any of the occlusion image (image A,
D, E) to generate a camera A1 included in another group.
Occlusion has occurred in the image 0 ₁ other than the camera is generated.

The value of the SSD obtained from the combination of cameras that capture the range in which occlusion occurs is large because there is no corresponding point corresponding to the reference pixel.
Therefore, the value of the SSSD of the group including the camera capturing the range where the occlusion does not occur is larger than the value of the SSSD of the group including the camera capturing the range where the occlusion does not occur.

That is, in the example shown in FIG. 10, as described above, no occlusion has occurred in any of the images generated by the cameras included in the group 3, and the cameras included in the other groups have no occlusion. Since occlusion has occurred in any of the generated images, an occlusion has occurred in any one or more of the images generated by the cameras included in the other groups 1, 2, and 4. Means that the SSSD of group 3 gives the minimum value SSSD _min .

The arithmetic processing circuit 124 calculates the minimum value SSS
A pixel giving D _min is detected as a corresponding point with the reference pixel, the parallax ζ and the distance between points on the objects 22 ₁ and 22 ₂ are measured, and the shapes of the objects 22 ₁ and 22 ₂ are analyzed. . In other words, the arithmetic processing circuit 124 selects one of the four parallaxes 得 obtained from each group, which has the lowest value of the SSSD and is most likely, and uses it for distance measurement and shape analysis.

Operation of Distance Measuring Apparatus 2 The operation of the distance measuring apparatus 2 will be described with reference to FIG. FIG. 11 is a flowchart showing the operation of the distance measuring device 2 shown in FIG.

The cameras A10 _{1 to} I10 ₉ capture images of the objects 22 ₁ and 22 ₂ to generate images A to I, respectively, and supply the images A to the arithmetic processing circuit 124 via the camera interface circuit 120 and the image memory circuit 122, respectively. I do.

As shown in FIG. 11, step 200 (S
200), the arithmetic processing circuit 124 sets the camera A1
0 ₁ and the reference pixel of the image A generated is, cameras B10 ₂ ~
I10 ₉ is SS between the reference pixel of the generated image B~I
D (Equation 1) is calculated.

In step 202 (S202), the arithmetic processing circuit 124 appropriately divides the cameras A10 _{1 to} I10 ₉ into groups and calculates the SSSD (Equation 3) for each group.

In step 204 (S204), the arithmetic processing circuit 124 obtains the minimum value SSSD _min of the SSSD for each group obtained in the process of S202, and substitutes the reference pixel giving the minimum value SSSD _min with the corresponding reference pixel. Detect as a corresponding point.

In step 206 (S206), the arithmetic processing circuit 124 measures the parallax ζ between the detected corresponding point and the reference pixel.

In step 208 (S208), the arithmetic processing circuit 124 executes S200 for all the reference pixels.
To the processing of S206 to generate a parallax image in the same manner as the distance measuring device 1 shown in the first embodiment, and the generated parallax image and the position of the camera of the group to which the minimum value SSSD _min is given in each of the reference pixels The objects 22 ₁ and 22 ₂ are based on the angle and the like (positional relationship) based on the principle of triangulation.
Are measured, and the objects 22 ₁ and 22 ₂ are further measured.
Is analyzed and output to the recording device 126 and / or the output device 128.

The stereo image processing by the distance measuring device 2
Examples below with reference to FIGS. 12 to 16, the distance measuring device 2, the example that was actually performed stereo image processing, in contrast to examples when you did not avoid occlusion will be described.

FIG. 12 is a diagram showing an original image to be subjected to stereo image processing. FIG. 13 shows a range in which the occlusion occurs when stereo image processing is performed on the original image shown in FIG. 12 without avoiding the occlusion shown in the second embodiment. FIG. 4 is a diagram showing an image to be reproduced.

[0119] Figure 14 is a distance measuring device 2 shown in FIG. 8, performs a stereo image processing separately camera A10 ₁ ~I10 ₉ into 4 groups, stereo image processing to show the original image in FIG. 12 FIG. 9 is a diagram showing an image displaying a range in which occlusion occurs when the above operation is performed. In FIGS. 13 and 14, the white portion near the person in the drawings indicates the range in which the occlusion has occurred, and the white portion in the background portion indicates the portion where an error has occurred due to other causes. Is shown. FIG. 15 corresponds to FIGS.
FIG. 5 is a diagram schematically illustrating the image shown in FIG. 4.

For example, using an apparatus having a camera arrangement as shown in FIG. 9 without avoiding the occlusion shown in the second embodiment, FIG. 12 and FIG.
As shown in FIG. 13 and FIG. 15B, when a background having a continuous pattern and a person holding a doll are subjected to stereo pixel processing using the object 22 as a target, the periphery (outline) Occlusion occurs in a large area around ().

On the other hand, occlusion is avoided by using a distance measuring device 2 (FIG. 8) having a camera arrangement and a group configuration as shown in FIG. 9, and shown in FIG. 12 and FIG. When the stereoscopic pixel processing is performed on the target object 22, as shown in FIG. 14 and FIG.
The range in which the occlusion occurs around the person (outline) can be greatly reduced as compared with the case shown in FIG. Therefore, according to the distance measuring device 2, it can be understood that the parallax ζ, the distance, and the shape of the target object can be accurately measured by reducing the range in which the occlusion occurs in the image.

As described above, according to the distance measuring apparatus 2 shown as the second embodiment, the occurrence of occlusion in an image can be reduced, and parallax is always obtained by three or more cameras. Since ζ is measured, even if there is a repeated pattern in the area where the occlusion occurs, the influence of the pattern can be eliminated and accurate measurement of the parallax ζ, the distance and the shape of the object can be performed.

The cameras A10 _{1 to} I1 shown in FIG.
For example, the four groupings of ₀₉ can be changed to five groupings as shown in FIG. Camera A divided into five groups shown in FIG.
According to the distance measuring device 2 using 10 _{1 to I} 10 ₉ , as shown in FIG. 16, the range in which the occlusion occurs is significantly larger than the case where the occlusion is not avoided (FIG. 13). Can be smaller.

Further, the content of the stereo image processing of the distance measuring device 2 is calculated, for example, by calculating the parallax の and the distance of the object for each group at the time of detecting the SSD,
The likelihood of these values may be evaluated by the SSD and changed to select any of these values, and such processing is included in the scope of the technical idea of the present invention. Nor.

The camera A of the distance measuring device 2 (FIG. 8)
The arrangement and grouping of 10 _{1 to I} 10 ₉ (FIG. 9) are examples, and not only can the number of base lines and the number of cameras be increased, but, for example, by making appropriate corrections, The generated image can be handled as a reference image. Further, the stereo image processing by the distance measuring device 1 according to the first embodiment and the stereo image processing by the distance measuring device 2 according to the second embodiment can be appropriately combined and used. Further, the same modification as that of the distance measuring device 1 (FIG. 4) is possible for the distance measuring device 2 (FIG. 8).

[0126]

According to the present invention, when there are a plurality of objects, the range in which occlusion occurs in which a distant object is hidden by a close object is generated as much as possible. The parallax information can be reduced, and correct parallax information can be obtained in a wide range. Further, according to the present invention, for example, even when a target object has a repeated pattern, it is possible to correctly take correspondence between pixels of a plurality of images, and to obtain correct disparity information. . Further, according to the present invention, correct parallax information can be obtained in a wide range, and a correct distance and shape of an object can be measured.

[Brief description of the drawings]

FIGS. 1A to 1C are diagrams illustrating stereo image processing. FIG.

FIGS. 2A and 2B are diagrams showing a reference image and a reference image obtained by photographing an object placed in front of a background with a camera A and a camera B1.

FIG. 3 is a diagram illustrating an example of erroneous detection of a corresponding point by an area-based matching method.

FIG. 4 is a diagram showing a configuration of a distance measuring device shown as a first embodiment of the present invention.

FIGS. 5A and 5B are diagrams showing a process of correcting a parallax に よ る by an arithmetic processing circuit of the distance measuring device (FIG. 4) according to the present invention.

FIG. 6 is a flowchart showing the operation of the distance measuring device according to the present invention.

FIG. 7 is a diagram illustrating the arrangement of cameras when performing stereo image processing using a large number (five or more) of cameras;

FIG. 8 is a diagram showing a configuration of a distance measuring device shown as a second embodiment of the present invention.

9 is a diagram exemplifying an arrangement of cameras A to I of the distance measuring device 2 shown in FIG.

10 is obtained by using the cameras A to I of the distance measuring device (FIG. 8) shown in FIG. 9 to photograph an object at a position close to the cameras A to I or an object at a good distant position. Image A
It is a figure which illustrates -I.

FIG. 11 is a flowchart showing an operation of the distance measuring device shown in FIG. 8;

FIG. 12 is a diagram illustrating an original image to be subjected to stereo image processing;

FIG. 13 shows a range in which occlusion occurs when stereo image processing is performed on the original image shown in FIG. 12 without avoiding occlusion shown in the second embodiment. FIG. 4 is a diagram showing an image to be reproduced.

FIG. 14 shows a camera A using the distance measuring device shown in FIG.
~ I are divided into 4 groups to perform stereo image processing,
FIG. 13 is a diagram showing an image displaying a range in which occlusion occurs when stereo image processing is performed on the original image shown in FIG. 12.

FIG. 15 is a diagram schematically illustrating the images shown in FIGS. 12 to 14;

FIG. 16 shows cameras A to I of the distance measuring device 2 shown in FIG.
It is a figure which illustrates the modification of arrangement | positioning of.

FIG. 17 shows a camera A using the distance measuring device shown in FIG.
~ I are divided into 4 groups to perform stereo image processing,
FIG. 17 is a diagram illustrating an image displaying a range where occlusion occurs when stereo image processing is performed on the original image illustrated in FIG. 16.

[Explanation of symbols]

1, 2, ... distance measuring device, 10 ₁ to 10 ₉ , 10 _n ... cameras A to I, camera N, 12 ... distance measuring unit, 120 ... camera interface circuit, 122 ... image memory circuit,
Reference numeral 124: arithmetic processing circuit, 126: recording device, 128: output device, 20: background, 200a, 200b: background image, 22: target object, 220a, 220b: target image, 24a: reference image, 24b: see image.

Claims (9)

[Claims] 1. A first image generating means for photographing an object to generate a first image, and a plurality of lines arranged on a plurality of straight lines passing through the first image generating means, respectively. Four or more second image generating means, which are divided into groups, each of which captures an object to generate four or more second images, each of the pixels of the generated first image, The plurality of second
Correlation determining means for determining the correlation with each pixel of each image using a predetermined evaluation function; and the second plurality of groups included in each of the plurality of groups.
The plurality of second image generation units included in any of the plurality of groups generate based on the value of the predetermined evaluation function of the pixel of each of the plurality of second images generated by the image generation unit. A certainty determining unit that determines whether the pixels of the plurality of second images correspond to the pixels of the first image most reliably, and the pixels of the first image that are most reliably associated with the pixels of the first image. Parallax calculating means for calculating parallax between corresponding points of the plurality of second images generated by the plurality of second image generating means included in any of the plurality of groups determined to be present Parallax calculation device. 2. The image processing apparatus according to claim 1, wherein the plurality of second image generation units included in each of the plurality of groups include a pixel position of the first image, the first image generation unit and the second image generation unit. And a virtual straight line (epipolar line) indicating a range in which the corresponding points of the plurality of second images corresponding to the pixels of the first image are provided. Arranged on any one of a plurality of straight lines, wherein the correlation determining means corresponds to a predetermined pixel block having each pixel of the first image as a central pixel and each pixel of the first image. The method according to claim 1, wherein a correlation between a predetermined pixel block centered on a pixel of each of the plurality of second images on an epipolar line of each of the plurality of second images is determined using the predetermined evaluation function. The parallax calculation described Output device. 3. The method according to claim 1, wherein the correlation determining unit uses a first evaluation function whose value changes in accordance with a correlation between a pixel of the first image and a pixel of each of the plurality of second images. Determining a correlation between a pixel of the first image and a pixel of each of the plurality of second images, wherein the certainty determination unit determines the plurality of second images included in each of the plurality of groups. The plurality of second image generation units included in any of the plurality of groups generate based on a total sum of the first evaluation function of the pixels of the plurality of second images generated by the generation unit. The parallax calculating apparatus according to claim 1, wherein it is determined whether pixels of each of the plurality of second images correspond most reliably. 4. A first image generating means for photographing an object to generate a first image, and a plurality of lines arranged on a plurality of straight lines passing through the first image generating means, respectively. Four or more second image generating means, which are divided into groups, each of which captures an object to generate four or more second images, each of the pixels of the generated first image, The plurality of second
Correlation determining means for determining the correlation with each pixel of each image using a predetermined evaluation function; and the second plurality of groups included in each of the plurality of groups.
The plurality of second image generation units included in any of the plurality of groups generate based on the value of the predetermined evaluation function of the pixel of each of the plurality of second images generated by the image generation unit. A certainty determining unit that determines whether the pixels of the plurality of second images correspond to the pixels of the first image most reliably, and the pixels of the first image that are most reliably associated with the pixels of the first image. A disparity calculating unit configured to calculate a disparity between corresponding points of the plurality of second images generated by the plurality of second image generation units included in any of the plurality of groups determined to be present; Based on the parallax and the positional relationship between the first image generating means and the plurality of second image generating means included in any of the plurality of groups determined to be most reliable. The object and the front A distance calculating unit that calculates a distance between the first image generating unit and the plurality of second image generating units included in any of the plurality of groups determined to be most reliable. Calculation device. 5. The plurality of second image generating means included in each of the plurality of groups includes: a position of a pixel of the first image; a first image generating means; and a second image generating means. And a virtual straight line (epipolar line) indicating a range in which the corresponding points of the plurality of second images corresponding to the pixels of the first image are provided. Arranged on any one of a plurality of straight lines, wherein the correlation determining means corresponds to a predetermined pixel block having each pixel of the first image as a central pixel and each pixel of the first image. 5. The method according to claim 4, wherein a correlation between a predetermined pixel block centering on a pixel of each of the plurality of second images on an epipolar line of each of the plurality of second images is determined using the predetermined evaluation function. The distance calculation described Output device. 6. The correlation determining means uses a first evaluation function whose value changes in accordance with the correlation between pixels of the first image and pixels of each of the plurality of second images. Determining a correlation between a pixel of the first image and a pixel of each of the plurality of second images, wherein the certainty determination unit determines the plurality of second images included in each of the plurality of groups. The plurality of second image generation units included in any of the plurality of groups generate based on a total sum of the first evaluation function of the pixels of the plurality of second images generated by the generation unit. The distance calculation apparatus according to claim 4, wherein it is determined whether pixels of each of the plurality of second images correspond most reliably. 7. A first image is generated by photographing an object from a predetermined position, and a first image is generated from each of a plurality of three or more positions included in a group divided into a plurality of straight lines passing through the predetermined position. Generating an image of the object to generate a plurality of second images of four or more, and each of the pixels of the generated first image and the plurality of second images.
The correlation between each pixel of each image of the plurality of images is determined using a predetermined evaluation function, and the predetermined pixels of the pixels of each of the plurality of second images generated by photographing at a position included in each of the plurality of groups Based on the value of the evaluation function, the correspondence between each of the pixels of the first image and the pixels of the plurality of second images generated by shooting from the plurality of positions included in any of the plurality of groups is most likely. It is determined whether it is certain, and each of the pixels of the first image, the plurality of images generated by shooting from the plurality of positions included in any of the plurality of groups determined that the correspondence is most certain Calculating a disparity between the corresponding points of the second image, and the calculated disparity, and the predetermined position, and the plurality of groups included in any of the plurality of groups determined to have the most reliable correspondence. Rank Calculating a distance between the object and the plurality of positions included in any of the plurality of groups determined to be most certain in the predetermined position and the correspondence based on the positional relationship between・ Distance calculation method. 8. The plurality of positions included in each of the plurality of groups include a position of a pixel of the first image, and a position between the predetermined position and the plurality of positions included in each of the plurality of groups. The plurality of straight lines are given such that a virtual straight line (epipolar line) indicating a range in which a corresponding point of the plurality of second images corresponding to a pixel of the first image exists is common. A predetermined pixel block centered on each pixel of the first image, and an epipolar line of each of the plurality of second images corresponding to each pixel of the first image The parallax / distance calculation method according to claim 7, wherein the correlation with a predetermined pixel block centering on a pixel of each of the plurality of second images is determined using the predetermined evaluation function. 9. A method according to claim 9, wherein said plurality of pixels of said first image and said plurality of second
Determining a correlation between a pixel of the first image and a pixel of each of the plurality of second images by using a first evaluation function whose value changes in accordance with the correlation with each pixel of the plurality of images. Then, any one of the plurality of groups is determined based on a total value of the first evaluation function of pixels of each of the plurality of second images captured and generated from the plurality of positions included in each of the plurality of groups. The parallax / distance calculation method according to claim 7, wherein it is determined whether or not the correspondence of the pixels of the plurality of second images generated by photographing from the plurality of positions included in the plurality of positions is most reliable.