JP2000149017A - Image processing device, method and supply medium thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To decide the distance from an object with high stability and high accuracy. SOLUTION: Each of matching calculation parts 11-14 calculates plural evaluation curves consisting of evaluation value, corresponding to the certainty of every estimated value of the distance from an object based on the combinations of the reference camera image obtained from a reference camera 1 and the camera images detected by the detection cameras 2-5 respectively. These calculated evaluation curves are supplied to an optimum value detection part 15. At the part 15, one or more evaluation curves are selected and added together. Then the distance to the object is decided, based on the addition result of those evaluation curves.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a providing medium, and more particularly to, for example, highly accurate and stable measurement of the distance, parallax, and shape to an object in a three-dimensional space. And an image processing method, and a providing medium.

[0002]

2. Description of the Related Art As a method of obtaining a distance to an object in a three-dimensional space, corresponding points are searched for a plurality of images obtained by photographing the object from different viewpoints.
Stereo image processing is known in which the distance to a target object is determined using parallax, for example, for each pixel.

In the stereo image processing, a predetermined area (hereinafter referred to as a reference) around a predetermined point (pixel) is obtained from a reference camera image obtained by photographing an object with a reference camera (reference camera). Block) is extracted, and from a detected camera image obtained by photographing the object with another camera (detecting camera), the same size as a reference block centered on a point (pixel) of interest. (Hereinafter, appropriately referred to as a detection block) is extracted, and using the reference block and the detection block, a correlation between a predetermined point on the reference camera image and a point of interest on the detection camera image ( Degree of match)
The calculation is performed while sequentially shifting the point of interest. Then, the point on the detected camera image that has the highest correlation with the predetermined point on the reference camera image (the point on the detected camera image that is focused when the correlation on the predetermined point on the reference camera image is the highest) ) Is detected as a corresponding point of a predetermined point on the reference camera image, and a point on the object in the three-dimensional space (the reference camera image (A point projected to a predetermined point of the image).

In stereo image processing, the above processing is repeated for each pixel on the reference camera image,
For example, 3 pixels projected on each pixel on the reference camera image
A distance image in which a distance to each point in the dimensional space is set as a pixel value is obtained.

[0005]

However, in the conventional stereo image processing, the correspondence of pixels between the reference camera image and the detected camera image becomes inaccurate, and the distance to the object is accurately obtained. May not be possible.

That is, a predetermined point (x,
y) (pixel at coordinates (x, y)), the correlation between the point of interest (x ′, y ′) on the detected camera image is
For example, an evaluation value (error value) e (x,
y).

[0007]

(Equation 1) (1) However, in equation (1), Y _A (x + i, y + j) is
The luminance of the pixel at the point (x + i, y + j) on the reference camera image is represented, and Y _B (x ′ + i, y ′ + j) is the luminance of the pixel at the point (x ′ + i, y ′ + j) on the detected camera image. Represents W represents a reference block and a detection block, and i, j W represents a point in the reference block or the detection block where the point (x + i, y + j) or the point (x ′ + i, y ′ + j) respectively. It represents that.

The evaluation value e (x,
y) becomes smaller as the correlation between the predetermined point (x, y) on the reference camera image and the point of interest (x ′, y ′) on the detected camera image becomes larger. Value e
A point (x ′, y ′) on the detected camera image that minimizes (x, y) is obtained as a corresponding point of the point (x, y) on the reference camera image. In addition, (x ',
From y ′) and (x, y), d obtained according to the following equation is the parallax.

D = ((xx ′) ² + (yy ′) ² ) ^1/2 (2)

The evaluation value e (x, y) of the equation (1) is, for example, [1] when the reference camera image or the detected camera image is a texture in which the texture is indefinite, that is, has little change in luminance or the like. [2] When the reference camera image and the detection camera image have a repetitive pattern, [3] the object is visible from the reference camera but not visible from the detection camera because the shape of the object changes greatly in the depth direction. If there is a point on a hidden object (occlusion is occurring)
In some cases, the point on the detected camera image that is not a true corresponding point is the smallest, and as a result, a point on the detected camera image that is not a true corresponding point is erroneously detected as a corresponding point.

Therefore, the case [1] can be dealt with to some extent by projecting a texture pattern. However, when it is desired to obtain a color image of the object at the same time as obtaining a distance image of the object. Cannot irradiate pattern light, so that a method of projecting a texture pattern cannot be used.

For example, Masatoshi Okutomi and Takeo Kanade, "Stereo Matching Using Multiple Baseline Lengths", Transactions of the Institute of Electronics, Information and Communication Engineers D-II Vol.J75-D-II No.8 pp.1317-1327
In August 1992 (hereinafter referred to as Document 1), a multi-baseline stereo (Multi Baseline Stereo) algorithm was adopted using a reference camera and a plurality of four detection cameras, thereby obtaining [2]. A method for dealing with the case is disclosed. However, it is difficult for the method described in Document 1 to deal with the cases [1] and [3].

Furthermore, for example, Kiyohide SATOH, Yuichi
OHTA, "Occlusion Detectable Stereo -Systematic Co
mparison of Detection Algorithms- ", Proccedings of
International Conference on Pattern Recognition 1
In 996 (ICPR'96) (hereinafter referred to as Reference 2), nine cameras are used (one camera is used as a reference camera, and the remaining eight cameras are used as detection cameras). By combining nine cameras, you can select the best combination with or without hiding,
A method is disclosed which addresses the case [3] and also addresses the case [2] by employing a multi-baseline stereo algorithm.

That is, according to the method described in Document 2, one of the nine cameras is set as C _0.
And eight cameras as detection cameras are denoted by C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C ₆ , C ₇ , and C ₈ , respectively.
Detection cameras (C ₁ , C ₂ , C ₃ ), (C ₂ , C ₃ ,
_{C 4), (C 3,} C 4, C 5), (C 4, C 5, C 6),
_{(C 5, C 6, C} 7), (C 6, C 7, C 8), (C 7, C 8,
(C ₁ ), (C ₈ , C ₁ , C ₂ )
Detection cameras (C ₁ , C ₂ , C ₃ , C ₄ , C ₅ ,
For each of the nine combinations of the detected cameras in total with the combination of C ₆ , C ₇ , C ₈ ), the evaluation value corresponding to the equation (1) is calculated by the multi-baseline stereo algorithm, thereby obtaining the reference camera C _The corresponding point for the pixel of the reference camera image of ₀ is determined.

Then, of the nine combinations of the detection cameras, the corresponding point obtained from the combination having the smallest evaluation value (highest correlation) is adopted, and the distance is obtained.

That is, when there is no occlusion in the target object, the accuracy is improved by using images from many cameras. Therefore, the corresponding points obtained from combinations of all eight detection cameras are used. When the object is hidden, a corresponding point obtained from a combination of certain three detection cameras is adopted according to the direction of the hiding. This allows
The case of [2] and the case of [3] are dealt with.

In the method of Reference 2, since there are three detection cameras and eight detection cameras, it is necessary to use normalized evaluation values when comparing the evaluation values in the two cases. is there.

Therefore, if the nine combinations of detection cameras are represented by t (t = 0, 1,..., 8), the evaluation value _et (x, y, d) for each combination t is For example, it is obtained according to the following equation.

[0019]

(Equation 2) (3) In the equation (3), e _f (x, y, d) is obtained for the detection camera C _f and the reference camera C _{0 in} the combination t of the detection cameras. This represents the evaluation value e (x, y) of (1). Note that the evaluation value e _f (x, y, d)
Is the disparity represented by the equation (2). Also, because it is f t, the Σ of equation (3) represents the summation of the evaluation values obtained for the detection camera each e _f of the combinations of the detection camera t (x, y, d). Further, in equation (3), M _t represents the number of detected cameras in the detected camera combination t, and w _t represents a normalization coefficient for the detected camera combination t.

The normalization coefficient w _t in the equation (3) is, for example, 1 when the combination of the detection cameras is eight detection cameras, and is 1, for example, when the combination t is three detection cameras. A slightly larger value is used. in this way,
The reason why the normalization coefficients for the three detection cameras are set to be larger than the normalization coefficients for the eight detection cameras is that if the normalization coefficients are equal to each other, when there is no occlusion, 8 is always used. No detection cameras are selected, and among the combinations of the three detection cameras, the evaluation value _et (x, y, d)
This is because the smallest one is adopted.

Each combination t (= 0, 1, 1) of the detection cameras
, 8), the evaluation value e _t (x,
After y, d) are obtained, from them,
The minimum evaluation value e (x, y, d) is obtained, and the distance is obtained from the minimum evaluation value e (x, y, d).

E (x, y, d) = min { _et (x, y, d)} (4) where, in the equation (4), min ｛｝ is the minimum value in ｛｝ means.

However, in the method of Reference 2, the normalization coefficient w _t in equation (3), that is, the normalization coefficient for the combination of three detection cameras, is used as the normalization coefficient for the combination of eight detection cameras. The extent to which it is made larger is determined experimentally. Therefore, the normalization coefficient w _t thus experimentally obtained may be appropriate for the reference camera image and the detected camera image to be processed, but may not be appropriate.

As a result, when there is no occlusion, a combination of eight detection cameras is used to determine the distance with high accuracy.
By adopting a combination of two detection cameras,
Sometimes the distance is required with high accuracy, but on the other hand,
There is no hiding, and despite the fact that the distance is required with higher accuracy when using a combination of eight detection cameras, a combination of three detection cameras is used, or conversely, there is hiding and three Although it is better to adopt the combination of the detection cameras, a combination of eight detection cameras may be adopted.

Further, according to the method of Reference 2, depending on the case [1] or the degree of change in luminance, the true corresponding point
Even if there is a combination of detection cameras with the smallest evaluation value, if there is a combination of detection cameras with a smaller evaluation value at the wrong corresponding point, the combination of the detection cameras with the smaller evaluation value will be changed. The combination of the detection cameras that have been adopted and have the smallest evaluation value at the true corresponding point will not be adopted.

As described above, according to the method of Reference 2, it is difficult to obtain the distance stably and accurately.

The present invention has been made in view of such a situation, and addresses the above-mentioned cases [1] to [3], and determines the distance to an object in a three-dimensional space with high accuracy. It is intended to be able to be obtained stably.

[0028]

According to the first aspect of the present invention, there is provided an image processing apparatus, comprising: a first image and one of a plurality of second images;
Based on each of the plurality of combinations of the above, a plurality of evaluation means for calculating an evaluation curve consisting of evaluation values corresponding to the likelihood of each estimated value of the distance to the object, and a plurality of evaluation curves calculated by the plurality of evaluation means Selecting means for selecting one or more of the evaluation curves, adding means for adding the evaluation curves selected by the selecting means, and estimating the distance to the object based on the result of adding the evaluation curves by the adding means. And estimating means.

The image processing method according to the twelfth aspect is characterized in that the first
A plurality of evaluation curves each having an evaluation value corresponding to the likelihood of each estimated value of the distance to the object, based on each of a plurality of combinations of the image and one or more of the plurality of second images. Evaluation step, a selection step of selecting one or more of the plurality of evaluation curves calculated in the plurality of evaluation steps, an addition step of adding the evaluation curves selected in the selection step, An estimating step of estimating a distance to the object based on the addition result.

According to a thirteenth aspect of the present invention, there is provided the providing medium, wherein each of the plurality of combinations of the first image and one or more of the plurality of second images is used for each of the estimated values of the distance to the object. A plurality of evaluation steps for calculating an evaluation curve composed of evaluation values corresponding to the likelihoods; a selection step for selecting one or more of the plurality of evaluation curves calculated for the plurality of evaluation steps; and a selection step. The present invention provides control information including an adding step of adding an evaluation curve, and an estimating step of estimating a distance to an object based on a result of adding the evaluation curves in the adding step.

According to a fourteenth aspect of the present invention, there is provided the providing medium according to each of the plurality of combinations of the first image and one or more of the plurality of second images, each of which is an estimated value of the distance to the object. Calculating a plurality of evaluation curves consisting of evaluation values corresponding to the likelihood, selecting one or more of the plurality of evaluation curves,
The selected evaluation curves are added, and an estimated value of the distance to the object obtained by estimating the distance to the object is provided based on the result of the addition of the evaluation curves.

[0032] In the image processing apparatus according to the first aspect, the plurality of evaluation means are configured to determine the target object based on each of a plurality of combinations of the first image and one or more of the plurality of second images. Calculating an evaluation curve composed of evaluation values corresponding to the likelihood of each estimated value of the distance to, and selecting means for selecting one or more of the plurality of evaluation curves calculated by the plurality of evaluation means. ing. The adding means adds the evaluation curve selected by the selecting means, and the estimating means estimates the distance to the object based on the result of adding the evaluation curves by the adding means.

In the image processing method according to the twelfth aspect, the estimated value of the distance to the object is determined based on each of a plurality of combinations of the first image and one or more of the plurality of second images. Calculates an evaluation curve composed of evaluation values corresponding to the probabilities of each of the above, selects one or more of the plurality of evaluation curves, adds the selected evaluation curves, and, based on the result of the evaluation curve addition, It estimates the distance to an object.

In the provision medium according to the thirteenth aspect,
Based on each of a plurality of combinations of the first image and one or more of the plurality of second images, an evaluation curve including an evaluation value corresponding to the likelihood of each estimated value of the distance to the object is calculated. Then, one or more of the plurality of evaluation curves calculated in the plurality of evaluation steps are selected, the selected evaluation curves are added, and the distance to the object is estimated based on the result of adding the evaluation curves. Control information for causing the information processing apparatus to perform the process is provided.

In the providing medium according to the fourteenth aspect,
A plurality of evaluation curves each having an evaluation value corresponding to a probability for each estimated value of the distance to the object based on each of a plurality of combinations of the first image and one or more of the plurality of second images. Is calculated, one or more of the plurality of evaluation curves are selected, the selected evaluation curves are added, and an object obtained by estimating the distance to the object based on the result of adding the evaluation curves is calculated. It has been made to provide an estimate of the distance to.

[0036]

FIG. 1 shows a configuration example of an embodiment of a distance measuring apparatus to which the present invention is applied.

One reference camera 1 (imaging means) or a plurality of four detection cameras 2 to 5 (imaging means)
Are fixed at different viewpoint positions, for example, CC
A D (Charge Coupled Device) video camera shoots an object from different positions (different line-of-sight directions), and the resulting reference camera image (first image) or four detected camera images (second image) With the memories 6 to 10
Respectively. The memory 6
The reference camera image from the reference camera 1 is temporarily stored in, for example, one frame unit. Memory 7
Nos. 1 to 10 temporarily store the detected camera images from the detected cameras 2 to 5, for example, in units of one frame.

The matching calculation unit 11 (evaluation means) is based on a combination of the reference camera image stored in the memory 6 and the detected camera image stored in the memory 7 for each estimated value of the distance to the object. Stereo processing is performed to calculate an evaluation curve corresponding to the likelihood, for example, an evaluation value obtained by Expression (1).

Similarly to the matching calculation unit 11, the matching calculation units 12 to 14 (evaluation means) also combine the reference camera image stored in the memory 6 with the detected camera images stored in the memories 8 to 10. Based on this, stereo processing is performed.

The evaluation values for each of the estimated distance values, which constitute the evaluation curves obtained as a result of the stereo processing in each of the matching calculation units 11 to 14, are supplied to the optimum value detection unit 15. The optimum value detecting unit 15 (selecting unit) (adding unit) (estimating unit) selects one or more of the plurality of evaluation curves from the matching calculation units 11 to 14, and adds the selected evaluation curves. Thus, the evaluation curve is set to one. Further, the optimum value detecting section 15 estimates and outputs the distance to the object based on the one evaluation curve (the addition result of the selected evaluation curves).

In the distance measuring apparatus configured as described above, an object is photographed by the reference camera 1 and the detection cameras 2 to 5. Further, the resulting one reference camera image and four detected camera images are stored in the memory 6.
To 10 and stored. Then, using the combination of this one reference camera image and each of the four detected camera images, stereo matching is performed in each of the matching calculation units 11 to 14.

Here, the stereo processing is performed by associating pixels between a plurality of images obtained by photographing the same object with a camera from two or more directions (different line-of-sight directions).
The parallax information between corresponding pixels, the distance from the camera to the target, and the shape of the target are obtained.

That is, for example, attention is focused on the reference camera 1 and the detection camera 2 in FIG.
When the object is photographed by the detection camera 2, a reference camera image including a projection image of the object is obtained from the reference camera 1, and a detection camera image including the projection image of the object is also obtained from the detection camera 2. Now, as shown in FIG. 2, if a certain point P on the object is displayed in both the reference camera image and the detected camera image, the position on the reference camera image where the point P is displayed is From the position on the detected camera image, that is, the corresponding point, the reference camera 1 and the detected camera 2
, And the position of the point P in the three-dimensional space (three-dimensional position) can be obtained using the principle of triangulation.

Therefore, in the stereo processing, first, it is necessary to detect a corresponding point. As a detection method, there is, for example, an area-based matching method using an epipolar line.

That is, as shown in FIG. 2, in the reference camera 1, a point P on the object is
On the straight line L connecting the optical center (lens center) O ₁ of it is projected intersection n _a of the imaging surface S ₁ of the base camera 1.

In the detection camera 2, the point P of the object is defined by the image pickup surface S ₂ of the detection camera 2 on a straight line connecting the point P and the optical center (lens center) O ₂ of the detection camera 2. At the intersection n _b of

In this case, the straight line L is a plane passing through the three points of the optical centers O ₁ and O ₂ and the point n _a (or the point P).
As the line of intersection L ₂ of the imaging surface S ₂ of the detection camera image is formed, is projected onto the imaging surface S _2. Point P is a point on the straight line L, thus, the imaging surface S _2, the n _b point obtained through projection of the point P, and lies on the straight line L ₂ obtained by projecting the straight line L, this straight line L _2, epipolar Called line. That, then possible corresponding points n _b of the point n _a is present, a on the epipolar line L _2, therefore, the search for corresponding points n _b may be performed in a subject on the epipolar line L _2.

[0048] Here, the epipolar line, for example, can be considered for each pixel constituting the reference camera image formed on the imaging surface S _1, if the positional relationship between the reference camera 1 and the detection camera 2 known , The epipolar line existing for each pixel can be obtained.

The detection of the corresponding point n _b from the point on the epipolar line L ₂ can be performed, for example, by the following area-based matching.

That is, in the area-based matching, FIG.
(A), the center point n _a of the reference camera image (e.g., diagonal intersection of) the, for example, rectangular reference block is a small block, with withdrawn from the reference camera image, FIG. as shown in 3 (B), on the epipolar line L ₂ projected on the detection camera image, centered at a point, reference block detection block is a small block of the same size and are from the detection camera image It is extracted.

[0051] Here, in the embodiment of FIG. 3 (B), on the epipolar line L _2, a point of the center of the detection block, 6 points of the point n _b1 to n _b6 are provided. These six points n _{b1 to} n _b6 are points that divide the straight line L in the three-dimensional space shown in FIG. 2 into predetermined constant distances such as 1 m, for example, that the distance from the reference camera 1 is 1
m, 2 m, 3 m, 4 m, 5 m, and 6 m are points projected on the imaging surface S ₂ of the detection camera 2, and accordingly, the distance from the reference camera 1 is 1 m, 2 m, 3 m, 4 m, 5 m,
Each point corresponds to a point of 6 m (estimated value of distance) (in this case, it can be said that the distance resolution is 1 m).

[0052] In the area-based matching, the detection camera image detection block around the point n _b1 through n _b6 respectively provided on the epipolar line L ₂ is extracted, and the detection block, the reference block The correlation is
The calculation is performed using a predetermined evaluation function. Then, the point n _b of the central correlation highest detection block and the reference block centered on the point n _a is obtained as the corresponding point of the point n _a.

That is, for example, when a function that takes a smaller value as the correlation is higher as shown in the above equation (1) is used as the evaluation function, the point n on the epipolar line L _{2 It} is assumed that, for example, evaluation values (evaluation function values) as shown in FIG. 4 are obtained for each of _{b1 to} n _b6 . Here, in FIG. 4, the distance numbers corresponding to the distances from the reference camera 1 that are previously assigned to the points in the three-dimensional space corresponding to the points n _{b1 to} n _b6 are set on the horizontal axis, and an evaluation value for a point n _b1 to n _b6) on the epipolar line L ₂ corresponding to, is shown.

[0054] When the evaluation curve made by the evaluation value as shown in FIG. 4 is obtained, the points on the epipolar line L ₂ which evaluation value is the smallest corresponds to the (correlation highest) distance number 3, It is detected as a corresponding point of the point n _a. In FIG. 4, interpolation is performed using the evaluation value (indicated by a black circle in FIG. 4) near the minimum value among the evaluation values obtained for the points corresponding to the distance numbers 1 to 6, and the evaluation value becomes higher. It is also possible to obtain a point that becomes smaller (a point corresponding to 3.3 m indicated by a cross in FIG. 4) and detect that point as a final corresponding point.

In the embodiment of FIG. 3, as described above,
A point on the object in three-dimensional space, a point dividing the line L at predetermined equidistant connecting the optical center O ₁ of the base camera 1, point projected onto the imaging surface S ₂ of the detection camera 2 is set However, this setting can be performed, for example, at the time of calibration of the reference camera 1 and the detection camera 2 (the calibration method is not particularly limited). Then, such a setting is set to the reference camera 1
Performed for each epipolar line exists for each pixel constituting the imaging surface S ₁ of the point set on epipolar line (hereinafter referred to as set point) corresponding to the distance to (distance from the reference camera 1) If a distance number / distance table that associates a distance number with a distance from the reference camera 1 is created in advance, a set point serving as a corresponding point is detected.
By converting the distance number corresponding to the set point with reference to the distance number / distance table, the reference camera 1 is immediately changed.
From the object (estimated value of the distance to the point on the object). That is, the distance can be obtained directly from the corresponding point.

Meanwhile, for the points n _a on the reference camera image, by detecting the corresponding points n _b on the detection camera image can be obtained parallax (disparity information) between the two points n _a and n _b . Further, if the positional relationship between the reference camera 1 and the detection camera 2 is known, from the disparity between the two points n _a and n _b, the principle of triangulation, can determine the distance to the object. The calculation of the distance from the parallax can be performed by performing a predetermined calculation. However, the calculation is performed in advance, and the parallax /
If the distance table is created in advance, the corresponding point can be detected, the parallax can be obtained, and the distance from the reference camera 1 can be immediately obtained by referring to the parallax / distance table.

Here, the parallax corresponds to the distance to the object on a one-to-one basis. Therefore, obtaining the parallax is equivalent to obtaining the distance to the object.

When the distance is directly obtained from the corresponding point, the interval between a set point on the epipolar line corresponding to a distant point in the three-dimensional space and an adjacent set point becomes narrow. Therefore, at a set point where the interval between adjacent set points is smaller than the distance between the pixels of the detected camera image, the accuracy of the obtained distance is deteriorated. When the distance is obtained from the parallax, the parallax does not change much at a close position in the three-dimensional space (a position close to the reference camera 1). Since the maximum accuracy of parallax is basically on a pixel basis, when obtaining a distance from a parallax, the accuracy of the distance deteriorates when obtaining a distance to a near position in a three-dimensional space. However, in any case, the accuracy problem can be solved by setting the set point and detecting the corresponding point in sub-pixel units smaller than the pixel.

[0059] Also, the detection of the corresponding point, to use a reference block and a detection block that it becomes a plurality of pixels blocks, to reduce the effect of noise, on the reference camera image pixel (point) of n _a surrounding pixels and features of the pattern, by utilizing the correlation between the characteristic patterns of the surrounding pixels of the corresponding point (pixel) n _b on the detection camera image, is for the sake of certainty of the corresponding point detection, especially For a reference camera image and a detected camera image with little change, if the size of the block is increased due to the image correlation, the reliability of detection of the corresponding point can be improved.

Further, in the above case, as the evaluation function for evaluating the correlation between the reference block and the detection block, the pixels constituting the reference block represented by the equation (1) and the pixels corresponding to the respective pixels are used. , The sum of the absolute values of the differences between the pixel values of the pixels constituting the detection block is used,
Other evaluation functions include the sum of squares of pixel value differences,
Normalized cross correlation
Etc. can be used.

In the matching calculation section 11 of FIG. 1, the pixels of the reference camera image in the above stereo processing are
A process of obtaining an evaluation value at a set point corresponding to each distance number set on the epipolar line of the detection camera image from the detection camera 2 is performed, and an evaluation curve formed of the evaluation values for each distance number is converted to an optimum value. It is supplied to the detection unit 15.

That is, the flowchart of FIG. 5 shows the stereo processing performed by the matching calculation unit 11.

In the matching calculation section 11, first,
The distance number n is initialized to, for example, 1 and step S2
To the pixel (point) (x ′, x ′, x) corresponding to the distance number n of the epipolar line on the detected camera image from the detection camera 2 corresponding to the pixel (x, y) on the reference camera image.
y ′) is detected. Then, the process proceeds to step S3, using the pixel (x, y) on the reference camera image and the pixel (x ', y') on the detected camera image, for example, by using Expression (1) for the distance number n. The represented evaluation value e (x, y) is calculated. Further, in step S3, the distance number n
And the combination of the evaluation value e (x, y) with respect to the distance number n are supplied to the optimum value detection unit 15 and the process proceeds to step S
Proceed to 4.

In step S4, it is determined whether or not the distance number n is equal to its maximum value N. If not, the process proceeds to step S5, where the distance number n is incremented by one. Then, the process returns to step S2, and thereafter, the same processing is repeated. Here, it is assumed that the distance number n takes an integer value proportional to the distance from the reference camera 1, with 1 as a minimum value, for example.

In step S4, the distance number n
Is determined to be equal to N, that is, when the evaluation values e (x, y) for all the distance numbers 1 to N are obtained for the pixel (x, y) on the reference camera image, the stereo processing is performed. finish.

In the matching calculation section 11, the stereo processing shown in FIG. 5 is performed for each pixel constituting the reference camera image. Further, the other matching calculation units 12 to 1
Also in 4, by performing the stereo processing of FIG. 5, the evaluation value at the set point corresponding to each distance number set on the epipolar line of each of the detected camera images from the detected cameras 3 to 5 is obtained. An evaluation curve including an evaluation value for each distance number is supplied to the optimum value detection unit 15.

Further, here, the matching calculation unit 11
In Nos. 14 to 14, distance numbers are assigned to set points for dividing straight lines in a three-dimensional space corresponding to epipolar lines into equal distances, and an evaluation value for each distance corresponding to the distance number is obtained. , Matching calculation units 11 to 1
In No. 4, it is also possible to obtain an evaluation value for each distance by obtaining an evaluation value for each parallax and converting the parallax to a distance.

Here, the points visible from the reference camera 1 are also visible in each of the detection cameras 2 to 5, so that there is no occlusion and the matching calculation units 11 to 1
4, when the evaluation value that takes the minimum value at the corresponding point is correctly obtained, the evaluation values output by the matching calculation units 11 to 14 are all the same distance number (corresponding to the true corresponding point). Distance number of the distance to do)
Is expected to be minimal.

On the other hand, when the point seen from the reference camera 1 is not seen by any one or more of the detection cameras 2 to 5, but is seen by the other cameras, that is, the detection cameras 2 to 5
Among them, for example, when not visible in the detection camera 2 and visible in the detection cameras 3 to 5,
Generally, the evaluation values output by the matching calculation units 12 to 14 that perform processing using the detected camera images of the visible detection cameras 3 to 5 are the smallest at the same distance number, but are not visible. It is anticipated that the evaluation value output by the matching calculation unit 11 that performs processing using the detected camera image will not be the minimum at the distance number at which the evaluation value output by each of the matching calculation units 12 to 14 becomes the minimum.

In such a case, the matching calculation unit 1
Of the evaluation values output by each of 1 to 14, the evaluation values output by the matching calculation units 12 to 14 that perform processing using the detection camera images of the detection cameras 3 to 5 without occlusion are used for the target object. By estimating the distance to the point, a highly accurate and stable distance can be obtained.

Therefore, the optimum value detecting section 15 in FIG. 1 estimates the distance to a point on the object in this way.

FIG. 6 shows an example of the configuration of the optimum value detector 15 shown in FIG.

The minimum value detection circuit 21 is supplied with a combination of a distance number and an evaluation value for the distance number from the matching calculation unit 11 of FIG.
The minimum value detection circuit 21 temporarily stores the combination of the distance number and the evaluation value, and detects the minimum value of the evaluation value. Further, the minimum value detection circuit 21 calculates the minimum evaluation value (hereinafter, appropriately referred to as minimum evaluation value) and the distance number combined with the minimum evaluation value (hereinafter, appropriately referred to as minimum distance number), A combination of a distance number close to the minimum distance number and an evaluation value for the distance number, that is, a distance number and an evaluation value within a range of ± 2 from the minimum distance number, while being supplied to the memory 25. The five combinations are supplied to the memory 29.

The minimum value detection circuits 22 to 24 are supplied with a combination of the distance number and the evaluation value from the matching calculation units 12 to 14 in FIG. Each detects the minimum value (minimum evaluation value) of the evaluation value supplied thereto in the same manner as the minimum value detection circuit 21 and supplies it to the memories 26 to 28 together with the corresponding minimum distance number. From the numbers, five combinations of the distance number and the evaluation value within the range of ± 2 are supplied to the memories 30 to 32, respectively.

The memories 25 to 28 include the minimum value detection circuit 2
Combinations of the minimum distance number from 1 to 24 and the minimum evaluation value are respectively stored. The memories 29 to 32 store the signals from the minimum value detection circuits 21 to 24,
Five combinations of a distance number and an evaluation value within a range of ± 2 around the minimum distance number are respectively stored.

The selection circuit 33 (selection means) includes the memory 25
Of the evaluation values stored in the memories 29 to 32 based on the storage contents of the memories 29 to 32, a selection signal for making the selection is determined by: The data is supplied to the evaluation value adding circuit 34. The evaluation value adding circuit 34 (adding means) reads out an evaluation value to be selected by the selection signal from the selection circuit 33 from necessary ones of the memories 29 to 32, and reads the read evaluation value into the same memory. Is added for each distance number. Further, the evaluation value addition circuit 34 outputs to the minimum value detection circuit 35 an evaluation curve composed of evaluation values for each distance number obtained as a result of the addition.

The minimum value detecting circuit 35 and the parabola fitting interpolation circuit 36 (estimating means) are adapted to estimate the distance to a point on the object based on the evaluation curve from the evaluation value adding circuit 34. That is, the minimum value detection circuit 35 detects a distance number that minimizes the evaluation value in the evaluation curve from the evaluation value addition circuit 34 and supplies the distance number to the parabola fitting interpolation circuit 36 along with the evaluation curve. The parabola fitting interpolation circuit 36 interpolates between the evaluation values constituting the evaluation curve from the minimum value detection circuit 35 by, for example, parabola fitting (by a quadratic curve), and from the curve obtained as a result of the interpolation, The minimum value is obtained, and a final estimated value of the distance to a point on the object is output.

The minimum evaluation value and the minimum distance number are
Since the optimum value detection unit 15 is stored not only in the memories 25 to 28 but also in the memories 29 to 32, the optimum value detection unit 15 can be configured without the memories 25 to 28.

Next, referring to the flowchart of FIG.
The processing of the optimum value detection unit 15 in FIG. 6 will be described.

In the minimum value detection circuits 21 to 24, the evaluation values for each distance number obtained for a predetermined pixel of the reference camera image from the matching calculation units 11 to 14 in FIG. Supplied and temporarily stored.

As described above, when the supply of the combination of the distance number and the evaluation value obtained for the predetermined pixel of the reference camera image is started, the minimum value detection circuit 21 stores the minimum evaluation value in step S11. Variable S for
_A predetermined large value (a value larger than the maximum value that can be taken as an evaluation value) is set to _min as an initial value, and a variable n representing a distance number is initialized to 1. Further, the minimum value detection circuit 21 determines in step S12
The evaluation value of the distance number n temporarily stored therein is read, and the process proceeds to step S13, where the evaluation value is set as a target evaluation value, and it is determined whether the _target evaluation value is smaller than a variable S _min .

If it is determined in step S13 that the evaluation value of interest is smaller than the variable S _{min, the process} proceeds to step S14.
The minimum value detection circuit 21 sets the variable S _min to the target evaluation value, and sets the variable S _min as the minimum evaluation value together with the corresponding distance number n in the memory 25.
To be stored in an overwritten form. Further, the minimum value detection circuit 21 proceeds to step S15, where the distance number n
The five combinations of the distance number and the evaluation value within a range of ± 2 with respect to are output to the memory 29 and stored in an overwritten form, and the process proceeds to step S16.

On the other hand, if it is determined in step S13 that the evaluation value of interest is not smaller than the variable _Smin , steps S14 and S15 are skipped and step S1 is executed.
Then, it is determined whether or not the distance number n is equal to the maximum value N. If it is determined that the distance number n is not equal, the process proceeds to step S17. In step S17, the distance number n
Is incremented by 1, and the process returns to step S12.
The same processing is repeated until it is determined in step S16 that the distance number n is equal to N.

In other words, the memory 25 stores the minimum value (minimum value) among the evaluation values for each distance number obtained by using the detected camera image output from the detected camera 2 for a predetermined pixel of the reference camera image. Evaluation value) and the distance number (minimum distance number) from which the minimum evaluation value was obtained, and the memory 29 stores the distance number and the evaluation value within a range of ± 2 around the minimum distance number. Five combinations are stored.

Here, the processing of steps S11 to S17 is performed not only by the minimum value detection circuit 21 but also by the minimum value detection circuit 2
Also in 2 to 24, the processing is performed in parallel. This allows
In the memories 26 to 28, the minimum value (minimum evaluation value) of the evaluation values for the predetermined pixels of the reference camera image for each distance number obtained by using the detection camera images output from the detection cameras 3 to 5, respectively. And the distance number (minimum distance number) at which the minimum evaluation value was obtained is stored in each of the memories 30 to 32 within a range of ± 2 around the minimum distance number stored in the memories 26 to 28. Five combinations of a certain distance number and an evaluation value are stored.

The memories 29 to 32 can store evaluation values for all distance numbers (here, distance numbers 1 to N). However, in the present embodiment, from the viewpoint of reducing the storage capacity of the memories 29 to 32, only five combinations of the distance number and the evaluation value within a range of ± 2 around the minimum distance number are stored. I have to.

Thereafter, if it is determined in step S16 that the distance number n is equal to N, the process proceeds to step S18, where the minimum evaluation value and the minimum distance number stored in each of the memories 25 to 28 are read out, and the selection circuit 33, and the process proceeds to step S19.

In the selection circuit 33, in step S19, a selection process is performed to determine which of the memories 29 to 32 should be selected based on the minimum evaluation value and the minimum distance number from each of the memories 25 to 28. , A selection signal for making the selection is supplied to the evaluation value adding circuit 34.
Supplied to The evaluation value adding circuit 34 determines in step S20
In step (3), based on the selection signal from the selection circuit 33, the evaluation value for each distance number stored therein is read from the memory to be selected from the memories 29 to 32, and is added for each of the same distance numbers. Then, the evaluation value adding circuit 34
Outputs the addition result to the minimum value detection circuit 35, and proceeds to step S21.

Here, in the selection processing performed in the selection circuit 33, a selection signal instructing to select at least one of the memories 29 to 32 is generated as described later. Therefore, in step S20, the evaluation value adding circuit 34 reads the evaluation value for each distance number stored therein from only one of the memories 29 to 32 based on the selection signal from the selection circuit 33. However, in this case, it can be considered that 0 is added to the evaluation value.

In step S21, the minimum value detection circuit 35
In, the added evaluation value for each distance number (hereinafter, referred to as
The minimum value (referred to as an addition evaluation value as appropriate) is detected and supplied to the parabola fitting interpolation circuit 36 together with the addition evaluation value for each distance number. In the parabola fitting interpolation circuit 36, in step S22, the output of the minimum value detection circuit 35 is interpolated by parabola fitting, and the minimum value is obtained from a curve obtained as a result of the interpolation. Then, proceeding to step S23, the parabola fitting interpolation circuit 36 determines the distance corresponding to the minimum value of the addition evaluation value obtained by the interpolation on the object in the three-dimensional space projected on a predetermined pixel of the reference camera image. The value is output as the final estimated value of the distance to the point, and the process ends.

The processing shown in the flowchart of FIG. 7 is performed for each pixel of the reference camera image, whereby the optimal value detection unit 15 outputs the three-dimensional space projected to each pixel of the reference camera image. A distance image in which the distance to the upper point is a pixel value is output.

In FIG. 6, the distance to a point in the three-dimensional space is obtained with higher precision by performing interpolation by parabola fitting. It is also possible to output the distance corresponding to the distance number corresponding to the minimum value of the addition evaluation value obtained in the above as the final estimated value of the distance.

Next, FIG. 8 shows a configuration example of the selection circuit 33 of FIG.

As shown in the figure, the selection circuit 33 comprises a minimum value search circuit 41 and comparators 42 to 45. The minimum value search circuit 41 has memories 25 to 28.
The minimum evaluation value and the minimum distance number stored respectively are stored in the comparator 42, the minimum evaluation value and the minimum distance number stored in the memory 25, and the comparator 43 is stored in the memory 26
Are stored in the comparator 44, the minimum evaluation value and the minimum distance number stored in the memory 27 are stored in the comparator 45, and the minimum evaluation value and the minimum distance number stored in the memory 28 are stored in the comparator 45. A minimum distance number is provided for each.

The minimum value search circuit 41 detects the minimum value among the minimum evaluation values from the memories 25 to 28, and supplies the minimum value to the comparators 42 to 45 as a reference evaluation value. The minimum value search circuit 41 supplies the reference evaluation value and a distance number corresponding to the reference evaluation value to the comparators 42 to 45 as a reference distance number.

The comparator 42 compares the reference distance number from the minimum value search circuit 41 with the minimum distance number from the memory 25, and the difference is a predetermined small value, for example, the reference distance number and the minimum distance. It is determined whether the difference from the number is within ± 1. The comparator 42 determines that the difference between the reference distance number and the minimum distance number is within ± 1, that is, the minimum distance number stored in the memory 25 is
The memory 29 paired with the memory 25 only when the distance number (reference distance number) within the range of ± 1 from the distance number (reference distance number) for the minimum value (reference evaluation value) among the minimum evaluation values stored in No. 8 Is output to the evaluation value adding circuit 34.

Similarly, the comparators 43 to 45 store the reference distance number from the minimum value search circuit 41 and the memories 26 to 28
Are compared with the minimum distance number from the memory, and only when the difference is within ± 1, the selection signals indicating selection of the memories 30 to 32 paired with the memories 26 to 28 are evaluated, respectively. Output to the value addition circuit 34.

When the comparators 42 to 45 output the selection signal, they output the selection signal including the reference distance number from the minimum value search circuit 41. Since the reference distance number is one of the minimum distance numbers stored in the memories 25 to 28, at least one of the comparators 42 to 45 always outputs a selection signal.

For example, now, the evaluation value for each distance number obtained from the detected camera images from the detected cameras 2 to 5 is:
It is assumed that the state is as shown in FIGS. 9A to 9D.

That is, in the embodiment shown in FIG. 9, the evaluation values of the six distance numbers 1 to 6 are shown for simplicity.

In FIG. 9A, the evaluation values of distance numbers 1 to 6 obtained from the detected camera images from the detected camera 2 are 30, 20, 10, 20, 30, 40, respectively.
The distance number 3 or the evaluation value 10 corresponding to the distance number 3 is the minimum distance number or the minimum evaluation value, respectively. In FIG. 9B, distance numbers 1 to 6 obtained from the detected camera image from the detected camera 3 are shown.
Are 30, 20, 15, 15, 20,
The distance number 3 or the evaluation value 15 corresponding to the distance number 3 is the minimum distance number or the minimum evaluation value, respectively. Further, in FIG. 9C, the distance number 1 obtained from the detection camera image from the detection camera 4 is shown.
The evaluation values of to 6 are 10, 20, 30, 40,
The distance number 1 and the evaluation value 10 corresponding to the distance number 1 are the minimum distance number and the minimum evaluation value, respectively. In FIG. 9D,
The evaluation values of the distance numbers 1 to 6 obtained from the detection camera images from the detection camera 5 are 50, 40, 30, and 2, respectively.
0, 30, and 40, and the distance number 4 or the evaluation value 20 corresponding to the distance number 4 is the minimum distance number or the minimum evaluation value, respectively.

Here, in FIG. 9B, distance numbers 3 and 4
, The evaluation value is 15, which is the minimum. This is because, in the embodiment of FIG.
_{When the} value is smaller than _min , the evaluation value of interest is substituted for the variable S _min. Therefore, if there are a plurality of minimum values among the evaluation values obtained from the detected camera images, the value with the smaller distance number is used. This is because it becomes the minimum distance number. Therefore, in the embodiment of FIG.
In the case of _min or less, if so assigning the attention evaluation value to a variable S _min, when the minimum evaluation value there are a plurality, greater than the distance number that becomes the minimum distance number, 9
In (B), the distance number 4 or the evaluation value 15 corresponding to the distance number 4 is the minimum distance number or the minimum evaluation value, respectively.

When the above evaluation values are obtained,
The minimum distance number 3 and the minimum evaluation value 10 are stored in the memory 25.
The minimum distance number 3 and the minimum evaluation value 15 are stored in the memory 26.
In the memory 27, the minimum distance number 1 and the minimum evaluation value 10 are stored.
The minimum distance number 4 and the minimum evaluation value 20 are stored in the memory 28.
Each is stored.

The minimum value among these minimum evaluation values is the minimum evaluation value 10 (FIG. 9A) stored in the memory 25. Therefore, the minimum value search circuit 41 selects this as the reference evaluation value. At the same time, the minimum distance number 3 for the reference evaluation value 10 is selected as the reference distance number.

Here, the smallest one of the minimum evaluation values 10, 15, 10, and 20 stored in the memories 25 to 28 is 10 stored in the memories 25 and 27 (FIG. 9A, (FIG. 9C) There are two. As described above, when there are a plurality of minimum values of the minimum evaluation values, the minimum value search circuit 41 outputs, for example, a range of a distance number such as ± 1 of a corresponding minimum distance number among the plurality of minimum values. With the largest number of other minimum distance numbers within
The reference evaluation value is finally selected.

That is, in the embodiment shown in FIG.
Is within the range of ± 1 of the minimum distance number 3 to the minimum evaluation value 10 (FIG. 9A) stored in the memory 2.
There are two other minimum distance numbers, the minimum distance number 3 stored in memory 6 (FIG. 9 (B)) and the minimum distance number 4 stored in memory 28 (FIG. 9 (D)). On the other hand, the minimum evaluation value 10 stored in the memory 27 (FIG. 9)
There is no other minimum distance number within the range of ± 1 distance number of minimum distance number 1 for (C)). For this reason,
In the minimum value search circuit 41, the minimum evaluation value 10 (FIG. 9A) stored in the memory 25 is selected as a reference evaluation value.

In the minimum value search circuit 41, the memory 25
When the minimum evaluation value 10 (FIG. 9A) stored in the memory 25 is selected as the reference evaluation value, the minimum distance number 3 stored in the memory 25 is compared with the reference evaluation value 10 as the reference distance number. Are supplied to the devices 42 to 45.

The comparator compares the reference distance number 3 from the minimum value search circuit 41 with the minimum distance number 3 (FIG. 9A) stored in the memory 25. That is, in this case, the same distance numbers are compared with each other.
5, the minimum distance number 3 is a distance number within ± 1 from the reference distance number 3. Therefore, the comparator 42 stores the memory 29 paired with the memory 25 including the reference distance number 3 in the memory 29. A selection signal instructing selection is output to the evaluation value adding circuit 34.

In the comparator 43, the minimum value search circuit 4
The reference distance number 3 from No. 1 is compared with the minimum distance number 3 (FIG. 9B) stored in the memory 26, and the difference is ±
Since it is within 1, the selection signal including the reference distance number 3 and instructing to select the memory 30 paired with the memory 26 is output to the evaluation value adding circuit 34.

Further, the comparator 44 compares the reference distance number 3 from the minimum value search circuit 41 with the minimum distance number 1 (FIG. 9C) stored in the memory 27, and the difference is ± 1. Otherwise, no selection signal is output.

In the comparator 45, the minimum value search circuit 4
The reference distance number 3 from 1 is compared with the minimum distance number 4 (FIG. 9D) stored in the memory 28, and the difference is ±
Since the distance is within 1, the reference distance number 3 is included in the memory 28.
A selection signal instructing to select the memory 32 paired with is output to the evaluation value adding circuit 34.

Accordingly, the evaluation value adding circuit 34 reads out the evaluation values for each distance number stored in the memories 29, 30, and 32 among the memories 29 to 32, and adds the evaluation values for each of the same distance numbers.

Here, as described above, the memories 29 to 32 have five combinations of distance numbers and evaluation values within a range of ± 2 around the minimum distance number stored in the memories 25 to 28. Only memorized. That is, in the memories 29, 30, and 32 to be added now, only the evaluation values for the distance numbers in the range indicated by the arrows in FIGS. 9A, 9B, and 9D are stored. Absent.

Specifically, the memory 29 stores the distance number 1
Evaluation values 30, 20, 10, 2 for each of
0, 30 (FIG. 9A), and the memory 30 stores the evaluation values 30, 20, 15, 1 for the distance numbers 1 to 5, respectively.
5 and 20 (FIG. 9B), the memory 32 stores the evaluation values 40, 30, 20, and 3 for the distance numbers 2 to 6, respectively.
0 and 40 (FIG. 9D) are respectively stored.

Therefore, when these evaluation values are added for each distance number, the evaluation values are added in the range of distance numbers 1 to 6, but the memories 29 and 30 store the distance number 6 The evaluation value for distance number 1 is not stored in the memory 32 (FIG. 9A, FIG. 9B) (FIG. 9A, FIG. 9B).
(D)).

In such a case, as the evaluation value that is not stored, for example, addition is performed using the maximum value that the evaluation value can take. Therefore, assuming that the maximum value that the evaluation value can take is, for example, 100, the addition result (addition evaluation value) of the evaluation value for each distance number is as shown in FIG.

That is, in this case, for the distance number 1,
The evaluation value 30 in FIG. 9A, the evaluation value 30 in FIG. 9B, and the maximum value 100 that the evaluation value can take are added, and 160 is obtained as an added evaluation value. For the distance number 2, the evaluation value 20 of FIG. 9A, the evaluation value 20 of FIG. 9B, and the evaluation value 40 of FIG. 9D are added, and 80 is obtained as an added evaluation value. Can be Further, for the distance number 3, the evaluation value 10 in FIG. 9A, the evaluation value 15 in FIG. 9B, and the evaluation value 30 in FIG. 9D are added, and 55 is obtained as an added evaluation value. Can be For the distance number 4, the evaluation value 20 in FIG.
The evaluation value 15 of FIG. 9B and the evaluation value 20 of FIG. 9D are added to obtain 55 as the added evaluation value. Further, for the distance number 5, the evaluation value 30 in FIG.
The evaluation value 20 in FIG. 9B and the evaluation value 3 in FIG.
0 is added, and 80 is obtained as an added evaluation value.
For the distance number 6, the maximum value 100 that the evaluation value can take, the maximum value 100 that the evaluation value can take, and FIG.
The evaluation value 40 of (D) is added, and the added evaluation value is 2
40 is required.

The above-mentioned added evaluation values (FIG. 9E) of the distance numbers 1 to 6 are supplied from the evaluation value adding circuit 34 (FIG. 6) via the minimum value detecting circuit 35 to the parabolic fitting interpolation circuit. In the parabolic fitting interpolation circuit 36, the added evaluation value is subjected to interpolation processing by parabola fitting, so that the minimum value of the curve obtained after the interpolation processing (FIG. 9)
A distance corresponding to a distance number (3.5 in FIG. 9E) corresponding to a portion indicated by a cross in (E) is obtained and output.

Since the distance corresponding to the distance number (reference distance number) corresponding to the minimum value among the minimum evaluation values is estimated to be the most probable distance, among the evaluation values output by the detection cameras 2 to 5, It can be said that the evaluation value at which the reference distance number is obtained and the evaluation value at which the distance number adjacent to the reference distance number is the minimum distance number are evaluation values that correctly reflect the distance. By selecting and adding only appropriate evaluation values and estimating the distance based on the resulting added evaluation value, it is possible to obtain the distance stably and with high accuracy.

That is, for example, even in a situation where the corresponding point is erroneously detected due to a texture such as a luminance change of the reference camera image or the detected camera image, it is estimated that the distance is correctly reflected. The correct distance can be stably obtained using only the evaluation value. In addition, detection camera 2
When the arrangement position or lens setting of any one of the detection cameras 2 to 5 fluctuates and becomes a state different from the state at the time of calibration, or when occlusion occurs in any of the detection cameras 2 to 5, The evaluation value obtained from the detection camera image of such a detection camera is not selected, and the correct distance can be obtained stably.

Further, among the four evaluation values obtained from the detected camera images from the four detected cameras 2 to 5, the one estimated to correctly reflect the distance is selected in a so-called adaptive manner. Since only the selected ones are added, addition of combinations that can be selected from the four evaluation values is performed, and compared with the case where any of the added values is selected, the addition is required. The number of circuits can be reduced, and as a result, an increase in the size of the device can be prevented.

That is, the combinations which can be selected from the four evaluation values obtained from the detection camera images from the four detection cameras 2 to 5 respectively are now represented by the four evaluation values e ₁ , e ₂ , e ₃ , If e ₄ , e ₁ , e ₂ , e ₃ , e ₄ , e
₁ and e ₂ , e ₁ and e ₃ , e ₁ and e ₄ , e ₂ and e ₃ , e ₂ and e ₄ , e
₁ and e ₂ and e ₃ , e ₁ and e ₃ and e ₄ , e ₁ and e ₂ and e ₄ , e ₂ and e
There are 14 combinations of ₃ and e ₄ , e ₁ and e ₂ , e ₃ and e ₄ , of which 10 need to be added. Then, the addition of the ten combinations is performed (addition evaluation value is generated), and the fourteen combinations including the _four combinations (e ₁ , e ₂ , e ₃ , e ₄ ) that do not require addition are added. From among them, by the above-described selection processing, one that matches the combination of those selected from the four evaluation values is adopted, and if the distance is obtained based on the adopted addition evaluation value, four evaluation values are obtained. As in the case where one or more of the values are selected and then addition is performed, it is possible to stably obtain a highly accurate distance. However, when the addition is performed first and the selection is performed later, in order to maintain the processing speed to some extent, it is necessary to perform the addition of ten combinations in parallel.
Required. Further, when the addition result is not adopted, the addition processing is useless.

On the other hand, when the addition is performed after selecting one or more of the four evaluation values, only one circuit is required for the addition, and no unnecessary addition is performed. .

Next, FIG. 10 shows another example of the configuration of the selection circuit 33 of FIG.

In the embodiment shown in FIG. 10, the selection circuit 33 comprises comparison counters 51 to 54 and a camera selection circuit 55.
In any of 4, (distance number for the minimum value among the detected camera 2 to 5 were obtained from each of the detection camera image evaluation value) minimum distance number stored in the respective memory 25 or 28 #n ₁ to #n ₄ are being supplied.

The comparison counter 51 calculates the minimum distance number #n
_1, compared with the other minimum distance number #n ₂ to #n ₄ respectively, minute value the difference is predetermined, i.e., for example, the difference between the minimum distance number #n ₁ and the other a minimum distance number, It is determined whether it is within ± 1. Then, the comparator 42 determines that the difference between the minimum distance number #n ₁ and the other minimum distance numbers is within ± 1, that is, the other minimum distance number is ₁ from the minimum distance number #n ₁ . If it is within ± 1 distance number, increments the count value by 1, thereby, a count (number of groups curve) of the minimum distance number #n other minimum distance number in the range within ± 1 of ₁ I do.

FIG. 11 shows an example of the configuration of the comparison counter 51 shown in FIG.

The comparison counter 51 includes comparators 61 to 6
3 and an adder 64, and comparators 61 to 63
Each is a comparator 42 (comparators 43 to 45) in FIG.
It has the same function as.

That is, the minimum distance number #n ₁ is supplied to the comparators 61 to 63.
The minimum distance number #n ₂ is supplied to the comparator 61, the minimum distance number #n ₃ is supplied to the comparator 62, and the minimum distance number #n ₃ is supplied to the comparator 63. I have. Then, the comparator 61 calculates the minimum distance numbers #n ₁ and #n ₂
And the minimum distance number #n ₂ becomes the minimum distance number #n ₁
, Only when the distance number is within ± 1, for example,
The pulse is output to the adder 64. Similarly, the comparator 62
Or 63 is also the minimum distance number #n ₁ and the minimum distance number #
comparing the n ₃ or #n ₄ respectively, the minimum distance number #n
_The pulse is output to the adder 64 only when ₃ or #n ₄ is a distance number within ± ₁ from the minimum distance number #n ₁ .

[0130] Adder 64 counts the number of those output pulses of the comparators 61 to 63, the count value, i.e., other minimum distance from the minimum distance number #n ₁ in the range within ± 1 The number of numbers (hereinafter, appropriately referred to as the number near the minimum distance number) and the minimum distance number are output to the camera selection circuit 55.

[0131] Returning to FIG. 10, other comparative count 52 to 54 are configured similarly to Comparative count 51, thus, the comparator counts 52 to 54, the minimum distance number #n ₂ to #n ₄ The number of neighborhoods of the minimum distance number for each of them is obtained, and is output to the camera selection circuit 55 together with the minimum distance number counted in the neighborhood number of the minimum distance number.

Here, if all the four evaluation values obtained from the detected camera images of the detected cameras 2 to 5 correctly reflect the distance (accordingly, for example, occlusion or corresponding point error). In the case where there is no detection or the like), it is presumed that a minimum evaluation value is obtained from each of the four evaluation values at a distance number having a similar value (ideally, the same). Therefore, in such a case, the number of neighborhoods of the minimum distance number is the number 4-1 of the evaluation value, which is the maximum value.
Further, for example, a case where occlusion occurs in any of the detection cameras 2 to 5 or one of the four evaluation values obtained from the detection camera images of the detection cameras 2 to 5 is used. In the case where an erroneous detection occurs in the corresponding point detection or the like, the number of neighborhoods of the minimum distance number decreases corresponding to the case where the occlusion occurs or the case where the erroneous detection of the corresponding point occurs.

[0133] In the camera selection circuit 55, of the minimum distance number number vicinity of the minimum distance number #n ₁ to #n ₄ respectively from the comparator counts 51 to 54, the largest is chosen. The camera selection circuit 55, among the minimum distance number #n ₁ to #n _4, and a selection of the minimum distance number number of neighbors, the minimum distance number that is counted to the minimum distance number number of neighbors, selected distance Number 29
A selection signal is output to the evaluation value adding circuit 34 to indicate that the one that stores the distance number (minimum distance number) selected as the selected distance number among the selected distance numbers should be selected. .

It should be noted that the camera selection circuit 55 determines whether or not there are a plurality of largest ones among the minimum distance number neighborhood numbers for the minimum distance numbers #n _{1 to} #n ₄ from the comparison counters 51 to 54 respectively. For example, among the plurality of minimum distance number neighborhood numbers, the minimum distance number neighborhood number for the minimum distance number corresponding to the smaller minimum evaluation value is selected.

When the selection circuit 33 is configured as shown in FIG. 10, the evaluation value for each distance number obtained from the detection camera images from the detection cameras 2 to 5 is, for example, as shown in FIG. 9A to 9D, the following evaluation values are selected in the optimum value detection unit 15 (FIG. 6), and the addition of the selected evaluation values is performed. Will be

That is, in this case, as described above, the minimum distance number 3 and the minimum evaluation value 10 are stored in the memory 25, the minimum distance number 3 and the minimum evaluation value 15 are stored in the memory 26, and the memory 27 is stored in the memory 27. The minimum distance number 1 and the minimum evaluation value 10 are stored in the memory 28, and the minimum distance number 4 and the minimum evaluation value 20 are stored in the memory 28, respectively.

The other minimum distance numbers within a range of ± 1 from the minimum distance number 3 (FIG. 9A) stored in the memory 25 are the minimum distance numbers 3 (FIG. 9) stored in the memory 26. 9 (B)) and the minimum distance number 4 stored in the memory 28 (FIG. 9 (D)), so the minimum distance number stored in the memory 25 (hereinafter, appropriately referred to as the minimum distance number of the detection camera 2). The minimum distance number neighbor number for 3 is 2.

The other minimum distance numbers within a range of ± 1 from the minimum distance number 3 (FIG. 9B) stored in the memory 26 are the same as the minimum distance number 3 stored in the memory 25.
(FIG. 9A) and the minimum distance number 4 stored in the memory 28 (FIG. 9D), the minimum distance number stored in the memory 26 (hereinafter, the minimum distance number of the detection camera 3 as appropriate) The number of neighbors of the minimum distance number for 3) is 2.

Further, since there is no other minimum distance number within the range of ± 1 from the minimum distance number 1 (FIG. 9C) stored in the memory 27, the minimum distance number stored in the memory 27 does not exist. The number of neighbors of the minimum distance number for 1 (hereinafter, appropriately referred to as the minimum distance number of the detection camera 4) is 0.

The other minimum distance numbers within the range of ± 1 from the minimum distance number 4 (FIG. 9D) stored in the memory 28 are the minimum distance numbers 3 stored in the memory 25.
(FIG. 9A) and the minimum distance number 3 stored in the memory 26 (FIG. 9B), the minimum distance number stored in the memory 28 (hereinafter, the minimum distance number of the detection camera 5 as appropriate) The number of neighbors of the minimum distance number for 4) is 2.

Therefore, in this case, the maximum value of the number of the minimum distance number neighbors is 2, and the minimum distance number that becomes the maximum number of the minimum distance number neighbors is the minimum distance number 3 of the detection cameras 2, 3, and 5. , 3,4. In such a case,
In the camera selection circuit 55, as described above, the number of the minimum distance numbers near the minimum distance number having the smaller minimum evaluation value is selected.
The number of neighbors of the minimum distance number 3 for the minimum distance number 3 of the detection camera 2 (FIG. 9A) is selected.

The camera selection circuit 55 counts the memory 29 storing the minimum distance number 3 of the detected camera 2 (FIG. 9A) and the number of neighborhoods of the minimum distance number 3 for the minimum distance number 3. Detection camera 3 (FIG. 9 (B))
Minimum distance number 3 or detection camera 5 (FIG. 9 (D))
Is output to the evaluation value adding circuit 34, indicating that a selection should be made between the memory 30 and the memory 32 storing the minimum distance number 4 of.

Therefore, in this case, in the evaluation value adding circuit 34, as in the case where the selecting circuit 33 is configured as shown in FIG.
The evaluation value for each distance number stored in 0, 32 is read and added for each identical distance number. As a result, the addition result (addition evaluation value) of the evaluation value for each distance number is shown in FIG.
As shown in FIG. 9 (E), the optimum value detection unit 15 sets the distance number (3.5 in FIG. 9 (E)) corresponding to the portion indicated by the cross in FIG. 9 (E) as described above. The corresponding distance is determined and output.

In the embodiment shown in FIG. 10, among the evaluation values output from the detected camera images of the detected cameras 2 to 5, those of the detected camera in which the number of neighborhoods of the minimum distance number is the maximum and the minimum distance number are the same. The one of the detection camera counted to the maximum number of the minimum distance numbers is selected, that is,
Only the evaluation value estimated to correctly reflect the distance is selected, and the distance is estimated based on the addition result (addition evaluation value) of the selected evaluation value. The distance can be obtained.

Further, in the embodiment shown in FIG. 10, an evaluation value having a large number of minimum evaluation values in the vicinity is selected, and the distance is obtained based on the result of adding the evaluation values. Since the distance is determined by majority decision, for example, due to a texture such as a luminance change of the reference camera image or the detected camera image, at a point that is not a true corresponding point, an evaluation value may be minimized, Even when an evaluation value that changes suddenly is obtained, the distance can be obtained stably and with high accuracy.

Next, FIG. 12 shows a configuration example of another embodiment of the distance measuring apparatus to which the present invention is applied. In addition,
In the figure, the same components as those in FIG. 1 are denoted by the same reference numerals, and the description thereof will be appropriately omitted below. That is, the distance measuring device of FIG. 12 is configured in the same manner as in FIG. 1 except that matching calculation units 111 to 114 are provided instead of the matching calculation units 11 to 14, respectively.

The matching calculation unit 111 (evaluation means)
By stereo processing, an evaluation value is obtained based on a combination of the reference camera image stored in the memory 6 and the detected camera image stored in the memory 7, and the reference camera image stored in the memory 6 and the memory 8 An evaluation value is obtained based on the combination with the detected camera image stored in the storage device, and the evaluation values are added to each other for the same distance number, and the added value is output as a new evaluation value. . That is, the matching calculation unit 111 obtains an evaluation value by the multi-baseline stereo method using the reference camera image stored in the memory 6 and the detected camera image stored in the memory 7 or 8. I have.

Similarly, the matching calculation unit 112 compares the reference camera image stored in the memory 6 with the memory 8 or 9
The matching calculation unit 113 uses the reference camera image stored in the memory 6 and the detected camera image stored in the memory 9 or 10 using each of the detected camera images stored in the 114 is
Using the reference camera image stored in the memory 6 and each of the detected camera images stored in the memory 10 or 7, an evaluation value based on the multi-baseline stereo method is obtained.

Therefore, according to the embodiment of FIG. 12, even when the reference camera image or the detected camera image has a repetitive pattern, the distance can be obtained stably and with high accuracy.

The distance measuring apparatus to which the present invention is applied has been described above. From the distance image obtained by such a distance measuring apparatus, for example, parallax can be further obtained or the object in a three-dimensional space can be obtained. It is also possible to measure the shape.

In the embodiment shown in FIGS. 1 and 12, for example, as shown in FIG.
, Four detection cameras 2 to 5 can be arranged. Here, in order to cope with the case of having a repetitive pattern, it is desirable that the distance between the reference camera 1 and each of two adjacent detection cameras be different from each other. Further, in the embodiment of FIG. 12, as shown in FIGS. 13B to 13E, the matching calculation unit 111 includes the reference camera 1 and the detection camera 2
And the output images of 3 are output to the matching calculation unit 112.
The output images of the reference camera 1 and the detection cameras 3 and 4 are output to the matching calculation unit 113, and the output images of the reference camera 1 and the detection cameras 4 and 5 are output to the matching calculation unit 114. 2 output images will be supplied respectively.

Further, in the present embodiment, four detection cameras are provided, but the number of detection cameras is not particularly limited.

In the present embodiment, a plurality of cameras are used. However, only one camera is prepared and moved, so that the one camera can be used as a reference camera or a detection camera. It is also possible to use it.

Further, the distance measuring device shown in FIGS.
For example, it can be realized by causing a computer (information processing device) to execute a computer program (control information) for performing the above-described processing, or by using dedicated hardware. In the case of using a computer program, the computer program can be provided by being recorded on, for example, an optical disk, a magneto-optical disk, a magnetic disk, a magnetic tape, a phase change disk, or another recording medium. , The Internet, a CATV (Cable Television) network, a satellite line, a terrestrial wave, and other transmission media.

The distance image obtained by the distance measuring device shown in FIGS. 1 and 12 can be provided to another device or the like via the above-described recording medium or transmission medium.

Further, in the present embodiment, the rectangular block is configured as the reference block and the detection block. However, the reference block and the detection block are not limited to the rectangular block. It is also possible to adopt a circular block composed of pixels within a certain distance from a certain pixel as the reference block and the detection block. In addition, the reference block and the detection block do not need to be formed of adjacent pixels, and may be formed of, for example, a set of pixels every other pixel.

[0157]

According to the image processing apparatus described in claim 1, the image processing method described in claim 12, and the providing medium described in claim 13, a first image and a plurality of second images are provided. Based on each of a plurality of combinations of at least one of the plurality of evaluation curves, a plurality of evaluation curves each including an evaluation value corresponding to the likelihood of each estimated value of the distance to the object are calculated. One or more are selected. Then, the selected evaluation curves are added, and the distance to the object is obtained based on the result of adding the evaluation curves. Therefore,
The distance to the object can be obtained stably and with high accuracy.

According to the fourteenth aspect of the present invention, the estimated value of the distance to the object is based on each of a plurality of combinations of the first image and one or more of the plurality of second images. Calculating a plurality of evaluation curves each having an evaluation value corresponding to the likelihood of each, selecting one or more of the plurality of evaluation curves, adding the selected evaluation curves, and based on an addition result of the evaluation curves The distance to the object obtained by estimating the distance to the object is provided. Therefore, it is possible to provide the distance to the object that is required stably and with high accuracy.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration example of an embodiment of a distance measuring device to which the present invention is applied.

FIG. 2 is a diagram for explaining an epipolar line.

FIG. 3 is a diagram showing a reference camera image and a detected camera image.

FIG. 4 is a diagram showing a transition of an evaluation value.

FIG. 5 is a flowchart for explaining stereo processing.

FIG. 6 is a block diagram illustrating a configuration example of an optimum value detection unit 15 of FIG. 1;

FIG. 7 is a flowchart for explaining processing of an optimum value detection unit 15 of FIG. 6;

8 is a block diagram illustrating a configuration example of a selection circuit 33 in FIG.

FIG. 9 is a diagram for explaining processing of a selection circuit 33 of FIG. 8;

FIG. 10 is a block diagram illustrating another configuration example of the selection circuit 33 of FIG. 6;

11 is a block diagram showing a configuration example of a comparison counter 51 in FIG.

FIG. 12 is a block diagram showing a configuration example of another embodiment of the distance measuring apparatus to which the present invention is applied.

FIG. 13 is a front view showing an arrangement position of a reference camera 1 and detection cameras 2 to 5.

[Explanation of symbols]

REFERENCE NUMERALS 1 reference camera (imaging means), 2 to 5 detection camera (imaging means), 6 to 10 memories, 11 to 14
Matching calculation section (evaluation means), 15 optimal value detection section (selection means) (addition means) (estimation means), 21 to 24 minimum value detection circuits, 25 to 32 memories, 3
3 selection circuit, 34 evaluation value addition circuit, 35 minimum value detection circuit, 36 parabola fitting interpolation circuit, 41 minimum value search circuit, 42 to 45 comparator,
51 to 54 comparison counter, 55 camera selection circuit, 61 to 63 comparator, 64 adder, 1
11 to 114 Matching calculation unit (evaluation means)

Claims (14)

[Claims] 1. An image processing apparatus for obtaining a distance to a target from a first image obtained by photographing a predetermined target and a plurality of second images, wherein the first image And calculating a plurality of evaluation curves each including an evaluation value corresponding to the likelihood of each estimated value of the distance to the object based on each of a plurality of combinations of at least one of the plurality of second images. Evaluation means, selection means for selecting one or more of a plurality of evaluation curves calculated by the plurality of evaluation means, addition means for adding the evaluation curves selected by the selection means, and addition means And an estimating means for estimating a distance to the object based on a result of adding the evaluation curves. 2. The method according to claim 1, wherein the selecting unit includes a plurality of evaluation curves having an evaluation value indicating the most probable value, and indicating that the vicinity of the estimated value corresponding to the evaluation value is most probable. 2. The image processing apparatus according to claim 1, wherein another evaluation curve is selected. 3. The evaluation means, for each set point corresponding to an estimated value of the distance to the object set on the epipolar line, an evaluation corresponding to the likelihood of the estimated value corresponding to the set point. Determining a value, among the plurality of evaluation curves, an evaluation curve having an evaluation value representing the most probable value, an estimated value corresponding to the evaluation value, or a set point corresponding to the estimated value. The image processing apparatus according to claim 2, wherein an estimation value corresponding to a set point adjacent to the other evaluation curve is selected with the other evaluation curve indicating that the estimated value is most likely. 4. The method according to claim 1, wherein the selecting means is, for each of the plurality of evaluation curves, a number of other evaluation curves indicating that the vicinity of the estimated value indicating that the evaluation curve is most likely is the most likely. The image processing apparatus according to claim 1, wherein the number of group curves is obtained, and one or more of the plurality of evaluation curves are selected based on the number of group curves. 5. The method according to claim 1, wherein the selecting unit determines, among the plurality of evaluation curves, an evaluation curve having the largest number of the group curves and an estimation value indicating that the evaluation curve is most likely. The image processing apparatus according to claim 4, wherein another evaluation curve is selected. 6. The selection means, when there are two or more of the plurality of evaluation curves having the largest number of group curves, indicates that the plurality of evaluation curves are most probable among the two or more evaluation curves. The image processing apparatus according to claim 5, wherein one having an evaluation value is selected. 7. The evaluation means, for each set point corresponding to the estimated value of the distance to the object set on the epipolar line, evaluating the likelihood of the estimated value corresponding to the set point. Determining a value, the selection means comprising, among the plurality of evaluation curves, an estimation value indicating that the number of the group curves is the largest, an estimation value indicating that the evaluation curve is most likely, or a setting corresponding to the estimation value. The image processing apparatus according to claim 4, wherein another evaluation curve indicating that an estimated value corresponding to a set point adjacent to the point is most likely is selected. 8. The method according to claim 1, wherein the estimating unit is configured to correspond to a probability for each estimated value of the distance to the object based on a combination of the first image and one of the plurality of second images. The image processing apparatus according to claim 1, wherein an evaluation curve including an evaluation value to be calculated is calculated. 9. The method according to claim 8, wherein the evaluation unit is configured to determine the likelihood of each of the estimated values of the distance to the object based on a combination of the first image and two or more of the plurality of second images. The image processing apparatus according to claim 1, wherein an evaluation curve including a corresponding evaluation value is calculated. 10. The image processing apparatus according to claim 1, wherein the adding means performs addition only on evaluation values near a probable estimated value in the evaluation curve selected by the selection means. apparatus. 11. The image processing apparatus according to claim 1, further comprising an imaging unit configured to photograph the object and output the first image and a plurality of second images. 12. A first object obtained by photographing a predetermined object.
And an image processing method for obtaining a distance to the object from a plurality of second images, wherein a plurality of the first images and one or more of the plurality of second images are obtained. A plurality of evaluation steps for calculating an evaluation curve consisting of evaluation values corresponding to the likelihood of each estimated value of the distance to the object based on each combination of the plurality of evaluations; and a plurality of evaluations calculated in the plurality of evaluation steps A selecting step of selecting one or more of the curves; an adding step of adding the evaluation curves selected in the selecting step; a distance to the object based on a result of adding the evaluation curves in the adding step And an estimation step of estimating. 13. A first object obtained by photographing a predetermined object.
And a plurality of second images, a providing medium that provides control information for causing an information processing device to perform a process of obtaining a distance to the target object, wherein the first image; A plurality of evaluations for calculating an evaluation curve composed of evaluation values corresponding to probabilities of respective estimated values of the distance to the object based on each of a plurality of combinations with one or more of the plurality of second images. A step of selecting one or more of a plurality of evaluation curves calculated in the plurality of evaluation steps; an addition step of adding the evaluation curves selected in the selection step; An estimating step of estimating a distance to the object based on an addition result of the evaluation curves. 14. A first object obtained by photographing a predetermined object.
And a plurality of second images, a providing medium that provides a distance to the object obtained by obtaining a distance to the object, wherein the first image, the plurality of Calculating, based on each of a plurality of combinations with one or more of the second images, a plurality of evaluation curves each including an evaluation value corresponding to the likelihood of each estimated value of the distance to the object; Selecting one or more of the evaluation curves, adding the selected evaluation curves, and estimating a distance to the object based on a result of the addition of the evaluation curves; A providing medium for providing an estimated value of a distance.