JP2000121319A - Image processor, image processing method and supply medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a relatively correct distance even when corresponding arrangement of pixels between a plurality of images is not made correctly. SOLUTION: At a stereo processing part 11, a corresponding point on a detection camera image to pixels of a reference camera image is detected to determine a distance to an object for the pixels of the reference camera image. Moreover, at a correcting pixel selecting part 13, the correcting pixels are selected for the correction of the distance from the pixels of a distance image with the distance as pixel value and at a correction part 14, the distance for the correcting pixels is corrected based on the distance for other pixels of the distance image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus, an image processing method, and a providing medium, and more particularly to, for example, each of pixels of a plurality of images obtained by photographing a predetermined object with a plurality of cameras. The present invention relates to an image processing apparatus and an image processing method for performing so-called stereo image processing, which determine a parallax with respect to an object, a distance to the object, and the like for each pixel.

[0002]

2. Description of the Related Art For example, in the document "Explanation, A Video-rate Stereomachine (Kande)
ade), Kimura, Journal of the Robotics Society of Japan ("Journal of the
Robotics Society of Japan (JRSJ) ") vol.13 No.3, p
p.322-pp.326, 1995) ", by photographing the same object from two or more places, or between a plurality of images obtained by photographing with two or more cameras.
So-called stereo image processing is disclosed, in which parallax information (parallax) between mutually corresponding pixels is obtained, the distance between points on the object is obtained, and the shape of the object is measured.

[0003]

However, in the conventional stereo image processing, the correspondence of pixels between a plurality of images near the outline of the object is incorrect depending on the positional relationship between the background and the object. In some cases, parallax information between pixels, a distance to an object, or a shape of the object cannot be correctly measured.

[0004] The present invention has been made in view of such a situation, and even if it is not possible to accurately associate pixels between a plurality of images, a relatively accurate parallax is obtained. Information and distance can be obtained.

[0005]

An image processing apparatus according to claim 1, wherein a detecting means for detecting a corresponding pixel of a second image corresponding to a pixel of the first image, Processing means for obtaining distance-related information for a pixel of one image; selecting means for selecting, from pixels of the distance-related information image having the distance-related information as a pixel value, a correction target pixel for which distance-related information is to be corrected; And correction means for correcting the distance relationship information about the correction target pixel based on the distance relationship information about other pixels of the distance relationship information image.

[0006] The image processing method according to the twentieth aspect provides a first method.
A detection step of detecting a corresponding pixel of the second image corresponding to a pixel of the image of the second image; a processing step of obtaining distance-related information for a pixel of the first image based on the corresponding pixel;
A selection step of selecting, from the pixels of the distance-related information image whose distance-related information has a pixel value, a correction target pixel for which the distance-related information is to be corrected, and a distance-related information image for the correction-target pixel, And correcting based on the distance relationship information about other pixels.

The providing medium according to claim 38, further comprising: a detecting step of detecting a corresponding pixel of the second image corresponding to a pixel of the first image; From the processing step of obtaining the distance relation information, and from the pixels of the distance relation information image having the distance relation information as a pixel value,
A selection step of selecting a correction target pixel to be corrected, and a correction step of correcting the distance relation information of the correction target pixel based on the distance relation information of other pixels of the distance relation information image. And providing control information comprising:

[0008] The providing medium according to claim 39 detects a corresponding pixel of a second image corresponding to a pixel of the first image,
Based on the corresponding pixel, distance-related information is obtained for a pixel of the first image, and a correction target pixel for which distance-related information is to be corrected is selected from pixels of the distance-related information image having the pixel value of the distance-related information. The distance relation information obtained by correcting the distance relation information about the correction target pixel based on the distance relation information about other pixels of the distance relation information image is provided.

In the image processing apparatus according to the first aspect, the detecting means detects a corresponding pixel of the second image corresponding to the pixel of the first image, and the processing means detects the corresponding pixel of the second image based on the corresponding pixel. Distance relationship information is obtained for a pixel of one image. The selecting means selects a correction target pixel to be subjected to the correction of the distance relation information from the pixels of the distance relation information image having the distance relation information as a pixel value, and the correction means outputs the distance relation information about the correction target pixel. The correction is performed on the basis of the distance-related information on the other pixels of the distance-related information image.

In the image processing method according to the present invention, a corresponding pixel of a second image corresponding to a pixel of the first image is detected, and a distance between pixels of the first image is determined based on the corresponding pixel. From the pixels of the distance-related information image whose pixel value is the distance-related information, a correction target pixel for which the distance-related information is to be corrected is selected. The correction is performed based on the distance relationship information of other pixels of the information image.

[0011] In the providing medium according to claim 38,
A corresponding pixel of the second image corresponding to the pixel of the first image is detected, distance-related information is obtained for the pixel of the first image based on the corresponding pixel, and a distance-related information is set as a pixel value of the distance-related information. From the pixels of the information image, a correction target pixel to be subjected to the correction of the distance relation information is selected, and the distance relation information of the correction target pixel is corrected based on the distance relation information of other pixels of the distance relation information image. To provide the control information for causing the information processing apparatus to perform the processing to be performed.

[0012] In the providing medium according to claim 39,
A corresponding pixel of the second image corresponding to the pixel of the first image is detected, distance-related information is obtained for the pixel of the first image based on the corresponding pixel, and a distance-related information is set as a pixel value of the distance-related information. From the pixels of the information image, a correction target pixel to be subjected to the correction of the distance relation information is selected, and the distance relation information of the correction target pixel is corrected based on the distance relation information of other pixels of the distance relation information image. To provide distance-related information obtained by performing the operation.

[0013]

FIG. 1 shows a configuration example of an embodiment of a distance measuring apparatus to which the present invention is applied.

The cameras 1 and 2 (imaging means) are fixed at different positions, for example, a CCD (Charge Coupled Device).
ice) With a video camera, the object 3 is photographed from different positions (different line-of-sight directions), and the resulting image is
The data is output to the camera interface 4. Here, the camera 1 or 2 functions as a so-called reference camera or detection camera, respectively. Therefore,
Hereinafter, the camera 1 or 2 is referred to as the reference camera 1 or the detection camera 2 as appropriate, and the image output from each is referred to as the reference camera image (first image) or the detection camera image (second image).

The camera interface 4 receives a reference camera image or a detected camera image from the reference camera 1 or the detected camera 2, respectively, and supplies it to the frame memory 5. The frame memory 5 stores a reference camera image and a detected camera image from the camera interface 4 in, for example, a frame unit.

The arithmetic processing circuit 6 calculates, for example, the distance from the reference camera 1 or the detection camera 2 to the object 3 based on the reference camera image and the detection camera image stored in the frame memory 5. Is obtained, and an image having a pixel value corresponding to the distance relationship information is generated.

That is, the arithmetic processing circuit 6 makes correspondence between the pixels of the reference camera image and the pixels of the detected camera image, and for each pixel of the reference camera image, the object 3
An image having a pixel value corresponding to the distance (for example, an image having a larger luminance value as the distance increases) (hereinafter, appropriately referred to as a distance image) is generated. Alternatively, the arithmetic processing circuit 6 measures the parallax 間 の between the corresponding pixels and generates an image having a pixel value corresponding to the parallax （(hereinafter, appropriately referred to as a parallax image). In the arithmetic processing circuit 6, it is also possible to obtain a distance image from the parallax 、, and to analyze the three-dimensional shape of the object 3 from the distance image.

Specifically, the arithmetic processing circuit 6 includes a control CPU (Central Processing Unit) 6A and a ROM (Read O
nly Memory) 6B, RAM (Random Access Memory) 6
C and DSP (Digital Signal Proces) for image processing
sor) 6D and the like. And CPU6A
Executes the application program stored in the storage device 7 to control each block constituting the distance measurement device, and performs stereo processing and correction processing as described later to generate a distance image and the like. Have been made to generate.

The ROM 6B stores, for example, an IPL (In
It stores a program for itial program loading and other necessary data. The RAM 6C temporarily stores a program executed by the CPU 6A, data necessary for the processing of the CPU 6A, and the like. DSP6D
Performs the necessary product-sum operation and the like under the control of the CPU 6A.

The storage device 7 includes, for example, an HD (Hard Dis
k), CD-ROM (Compact Disc Read Only Memor
y), and the like, and the OS (O
perating System), an application program for performing the above-described processing, and the like. Further, the storage device 7 is also configured to store a distance image or the like output from the arithmetic processing circuit 6. The output device 8 includes, for example, a CRT (Cathode Ray Tube), a liquid crystal panel, or the like, and displays a distance image or the like output from the arithmetic processing circuit 6.

In the distance measuring apparatus configured as described above, the reference camera 1 and the detection camera 2
Is taken. Further, the resulting reference camera image and detected camera image are stored in the camera interface 4.
Is supplied to and stored in the frame memory 5. Then, the arithmetic processing circuit 6 performs stereo processing using the reference camera image and the detected camera image.

Here, the stereo processing is performed as described above.
By associating pixels between a plurality of images obtained by photographing the same object with a camera from two or more directions (different line-of-sight directions), parallax information between corresponding pixels and a distance from the camera to the object , To determine the shape of the object.

That is, for example, as shown in FIG. 2, when an object 3 is photographed by two reference cameras 1 and two detection cameras 2, a reference camera image including a projection image of the object 3 is obtained from the reference camera 1. Then, a detection camera image including the projection image of the object 3 is obtained from the detection camera 2. Now, assuming that a point P on the object 3 in FIG. 2 is displayed in both the reference camera image and the detected camera image, the point P
Can be obtained from the position on the reference camera image where is displayed and the position on the detected camera image, that is, the corresponding point (corresponding pixel). Further, the point P is calculated using the principle of triangulation. Position in 3D space (3D position)
Can be requested.

Therefore, in the stereo processing, first, it is necessary to detect a corresponding point. As a detection method, there is, for example, an area-based matching method using an epipolar line.

That is, as shown in FIG. 3, in the reference camera 1, the point P on the object 3 is on a straight line L connecting the point P and the optical center (lens center) O _{1 of the} reference camera 1. ,
It is projected to the intersection n _a of the imaging surface S ₁ of the base camera 1.

In the detection camera 2, the object 3
A point P is an imaging surface S ₂ of the detection camera 2 on a straight line connecting the point P and the optical center (lens center) O ₂ of the detection camera _2.
It is projected to the intersection n _b and.

In this case, a straight line L is defined by _a plane passing through the optical centers O ₁ and O ₂ and the point n _a (or the point P).
As the line of intersection L ₂ of the imaging surface S ₂ of the detection camera image is formed, is projected onto the imaging surface S _2. The point P is a point on the straight line L. Therefore, on the imaging surface S ₂ , the point n _b on which the point P is projected exists on the straight line L ₂ on which the straight line L is projected, and the straight line L ₂ is an epipolar line. Called. That, then possible corresponding points n _b of the point n _a is present, a on the epipolar line L _2, therefore, the search for corresponding points n _b may be performed in a subject on the epipolar line L _2.

[0028] Here, the epipolar line, for example, can be considered for each pixel constituting the reference camera image formed on the imaging surface S _1, if the positional relationship between the reference camera 1 and the detection camera 2 known , The epipolar line existing for each pixel can be obtained.

The detection of the corresponding point n _b from the point on the epipolar line L ₂ can be performed by, for example, the following area-based matching.

That is, in the area-based matching, FIG.
As shown in (A), for example, a small rectangular block (hereinafter, appropriately referred to as a reference block) (first block) having _a point na on the reference camera image as _a center (for example, an intersection of diagonal lines) is provided. , together with the extracted from the reference camera image, as shown in FIG. 4 (B), on the epipolar line L ₂ projected on the detection camera image, centered at a point,
A small block having the same size as the reference block (hereinafter, appropriately referred to as a detection block) (second block) is extracted from the detected camera image.

[0031] Here, in the embodiment of FIG. 4 (B), on the epipolar line L _2, a point of the center of the detection block, 6 points of the point n _b1 to n _b6 are provided. These six points n _{b1 to} n _b6 are points that divide the straight line L in the three-dimensional space shown in FIG.
That is, the distance from the reference camera 1 is, for example, 1 m, 2
m, 3 m, 4 m, 5 m, and 6 m are points projected on the imaging surface S ₂ of the detection camera 2, and accordingly, points at distances from the reference camera 1 of 1 m, 2 m, 3 m, 4 m, 5 m, and 6 m Respectively.

[0032] In the area-based matching, the detection camera image detection block around the point n _b1 through n _b6 respectively provided on the epipolar line L ₂ is extracted, and the detection block, the reference block The correlation is
The calculation is performed using a predetermined evaluation function. Then, the point n _b of the central correlation highest detection block and the reference block centered on the point n _a is obtained as the corresponding point of the point n _a.

[0033] That is, for example, now, as the evaluation function, when a function that takes a smaller value the higher the correlation, the points n _b1 through n _b6 respectively on the epipolar line L _2, for example, as shown in FIG. 5 Evaluation value (value of evaluation function)
Is obtained. In this case, the smallest evaluation value (the correlation is the highest) is point n _b3, is detected as a corresponding point of the point n _a. In FIG. 5, interpolation is performed using the evaluation values near the minimum value among the evaluation values (indicated by ● in FIG. 5) obtained for each of the points n _{b1 to} n _b6 , and a point at which the evaluation value becomes smaller is obtained. (Indicated by a cross in FIG. 5), and that point can be detected as a final corresponding point.

In the embodiment of FIG. 4, as described above,
A point dividing the line L for each predetermined equidistant in a three-dimensional space, but a point obtained by projecting the imaging surface S ₂ of the detection camera 2 has been set, this setting is, for example, the reference camera 1 and the detection camera 2 Can be performed at the time of calibration. Then, such a setting is performed for each epipolar line exists for each pixel constituting the imaging surface S ₁ of the reference camera 1, as shown in FIG. 6 (A), the point set on the epipolar line (hereinafter , Where appropriate, referred to as set points)
If a set point / distance table that associates a distance from the reference camera 1 is created in advance, a set point serving as a corresponding point is detected, and the set point / distance table is referred to.
The distance from the reference camera 1 (the distance to the object 3) can be immediately obtained. That is, the distance can be obtained directly from the corresponding point.

Meanwhile, for the points n _a on the reference camera image, by detecting the corresponding points n _b on the detection camera image can be obtained parallax (disparity information) between the two points n _a and n _b . Further, if the positional relationship between the reference camera 1 and the detection camera 2 is known, from the disparity between the two points n _a and n _b, the principle of triangulation, can determine the distance to the target 3. Although the calculation of the distance from the parallax can be performed by performing a predetermined calculation, the calculation is performed in advance, and the parallax ζ is calculated as shown in FIG.
If a parallax / distance table for associating the distance with the distance is created in advance, the corresponding point is detected, the parallax is obtained, and the parallax /
By referring to the distance table, the distance from the reference camera 1 can be immediately obtained.

Here, the parallax and the distance to the object 3 have a one-to-one correspondence. Therefore, obtaining the parallax and obtaining the distance to the object 3 are equivalent, so to speak. is there.

Further, the detection of the corresponding point, to use a reference block and a detection block that it becomes a plurality of pixels blocks, to reduce the effect of noise, on the reference camera image pixel (point) of n _a surrounding pixels pattern and characteristics of, by determining clarifies a correlation between the feature of the pattern of pixels around the detected corresponding points on the camera image (pixels) n _b,
In order to ensure the detection of the corresponding point, especially for the reference camera image and the detected camera image with little change,
Due to the correlation of the images, the larger the block size, the more reliable the detection of the corresponding point.

Next, in area-based matching,
For example, as an evaluation function for evaluating the correlation between the reference block and the detection block, the sum of the absolute values of the differences between the pixels constituting the reference block and the pixel values of the pixels constituting the detection block corresponding to the respective pixels is provided. And the sum of squares of the differences, normalized cross correl
ation) can be used.

When the sum of the absolute values of the pixels constituting the reference block and the pixel values of the pixels constituting the detection block corresponding to each pixel is used as the evaluation function,
The evaluation value (value of the evaluation function) can be calculated by the following equation, for example.

[0040]

(Equation 1) (1) In Expression (1), x represents a pixel on the reference image (pixel of interest) of interest for which a corresponding point is to be obtained,
Therefore, it represents the central pixel constituting the reference block. Further, y represents a central pixel constituting the detection block,
S (x, y) represents an evaluation value indicating a correlation between a reference block centered on pixel x on the reference camera image and a detection block centered on pixel y on the detected camera image. Furthermore, W _A or W _B represents a reference block or detection block, respectively, j W _A, because it is W _B, x + j is
Represents pixels constituting the reference block W _A, y + j corresponds to the pixel x + j, the pixels constituting the detection block W _B, representing, respectively. In addition, Y _A (x + j) or Y _B (y
+ J) represents the pixel value of pixel x + j or y + j, respectively.

In the case of using the equation (1), for example, the evaluation value S (x, y) is calculated while sequentially shifting the pixel y to the set point set on the epipolar line, and the evaluation value S ( The pixel y (= y _min ) of the detected camera image that minimizes x, y) is detected as the corresponding point of the pixel x of the reference camera image. Note that the parallax is determined by the pixels x and y _min
Is calculated from the position.

For example, when the sum of squares of the difference between the pixels constituting the reference block and the pixel values of the pixels constituting the detection block corresponding to each pixel is used as the evaluation function, the evaluation value is, for example, It can be calculated by the following equation.

[0043]

(Equation 2) (2) In Expression (2), x and y are x coordinates of a pixel and y
Η and η represent parallax in the x direction and parallax in the y direction, respectively. Then, SSD (x, y, η,
ξ) is an evaluation value when the parallax is assumed to be (η, ξ), that is, a reference block centered on a pixel located at coordinates (x, y) on the reference camera image, and an evaluation value on the detected camera image. Represents an evaluation value indicating a correlation with a detection block centering on a pixel located at the coordinates (x−η, y−ξ) of the pixel. Also, W
_A or W _B represents a reference block or detection block, respectively, i, j W _A, because it is _{W B, (x + i,} y
+ J) or (x + i−η, y + j−ξ) represents coordinates on the reference camera image or the detected camera image, respectively.
Y _A (x + i, y + j) represents a pixel value of a pixel located at coordinates (x + i, y + j) on the reference camera image, and Y _B (x + i−η, y + j−ξ) represents a pixel value on the detected camera image. It represents the pixel value of the pixel located at the coordinates (x + i-η, y + j-ξ).

When equation (2) is used, the disparity (η,
ξ) is changed, the evaluation value SSD (x, y, η,
ξ) is calculated, and the pixel (x−η, y−) of the detected camera image that minimizes the evaluation value SSD (x, y, η, ξ) is calculated.
ξ) is detected as the corresponding point of the pixel (x, y) of the reference camera image. Note that the evaluation value SSD (x, y, η, ξ)
The parallax (η, ξ) when minimizing is (η _min ,
ξ _min ), the parallax 距離 as the distance between the pixel of the reference camera and the corresponding point is obtained by the following equation.

[0045]

(Equation 3) ... (3)

Here, the smaller the difference between the pixel value of the reference block and the pixel value of the detection block corresponding to each pixel, the smaller the evaluation value shown in Expressions (1) and (2). Become. Therefore, pixel n of the reference camera image
_As for the pixel n _b of the detected camera image that minimizes the evaluation value for a, the smaller the evaluation value, the larger the pixel n _a
Of greater certainty of being corresponding points, on the contrary, the larger the evaluation value, certainty of being the corresponding point of pixel n _a is the lower.

Next, for example, as shown in FIG.
In front of the background wall (reference camera 1 and detection camera 2
A case where the corresponding point is detected by photographing an object 3 placed in front of the camera 3 with the reference camera 1 and the detection camera 2 and performing area-based matching using an epipolar line is considered.

Here, as described above, obtaining the parallax is equivalent to obtaining the distance to the object 3. In the following, the corresponding points are shown in FIG. 6A, for example. The description will be made assuming that the distance is determined with reference to such a set point / distance correspondence table. Further, in the following, for example, the evaluation value of Expression (1) is used.

In the case shown in FIG. 7A, for example, a reference camera image or a detection camera image as shown in FIG. 7B is obtained from the reference camera 1 or the detection camera 2, respectively.

Now, in the reference camera image shown in FIG. 7B, as shown in FIG.
Of the background k in contact with the foreground, which is a projected image of FIG.
, The detection of a corresponding point k ′ (indicated by a triangle) is considered.

In this case, for example, if the parallax ζ between the pixel k and the corresponding point k ′ is large, as shown in FIG. 8B, the detection camera of the background pixel k in contact with the foreground in the reference camera image The true corresponding point k ′ on the image may be observed at a position distant from the foreground, which is a projection image of the object 3.

In such a case, the reference block whose center pixel is the pixel k of the reference camera image is shown in FIG.
As shown in (C), some pixels include foreground pixels, and others include background pixels. In the embodiment of FIG. 8C, 5 × 5 pixels (horizontal × vertical) centering on pixel k
Are configured as reference blocks, and two columns on the left side are foreground pixels, and three columns on the right side are background pixels.

In the case shown in FIG. 8C, the pixels of the detected camera image corresponding to the two columns on the left side and the three columns on the right side of the reference block are 2 pixels as shown in FIG. 8D.
Divided into two places. That is, the left two columns of foreground pixels in the reference block correspond to the foreground pixel group A in the detected camera image. On the other hand, the right three columns of background pixels of the reference block correspond to the background pixel group B at the true corresponding point k ′ in the detected camera image.

As described above, when the pixels of the detected camera image corresponding to the pixels constituting the reference block are separated, the evaluation value calculated for the reference block is targeted for the detection block including the pixel group B. In some cases, the size may be smaller when the detection block including the pixel group A is targeted. That is, as shown in FIG. 8 (E), in the detected camera image, the detection block formed around the background pixel k ″ in contact with the foreground can minimize the evaluation value between the detection block and the reference block. is there. In this case, a point k ″ that is not a true corresponding point k ′ is detected as the corresponding point of the pixel k, and therefore, the corresponding point is erroneously detected. Do 3
An incorrect value is also obtained for the distance to a point in the dimensional space. That is, if the corresponding point is erroneously detected as described above, the distance to the point in the three-dimensional space corresponding to the pixel k is:
It is obtained as a value close to the distance to the object 3 constituting the foreground. Then, as a result, a distance image having such a distance as a pixel value is as if the width of the object 3 is widened.

Here, the erroneous detection of the corresponding points as described above
If the foreground and background patterns are different, it is relatively easy to occur, and if the patterns are significantly different,
Particularly easy to occur.

The erroneous detection of the corresponding point as described above can be reduced by reducing the size of the reference block (and therefore the detection block) (reducing the number of pixels constituting the reference block). The reliability of detecting a corresponding point by area-based matching is reduced.

Therefore, the arithmetic processing circuit 6 of the distance measuring apparatus shown in FIG. 1 performs the following distance correction processing to maintain the certainty of the corresponding point by area-based matching, and The influence on the distance due to the erroneous detection of the corresponding point occurring at the boundary between the foreground and the background, that is, at the edge, is reduced.

That is, in the arithmetic processing circuit 6, first,
By the stereo processing as described above, for each pixel of the reference camera image, each pixel of the detected camera image is calculated by the equation (1).
Are matched by the area-based matching method using the evaluation value by the evaluation function shown in (the corresponding points are detected),
The distance to a point in the three-dimensional space corresponding to each pixel of the reference camera image is obtained. As a result, a distance image having the distance as a pixel value is generated.

Next, as shown in FIG. 9A, a pixel centered on the pixel k of the reference camera image and optimal for correcting the distance of the pixel k (hereinafter, appropriately referred to as the maximum likelihood pixel) A small block (hereinafter, appropriately referred to as a search block) (indicated by a dotted line in FIG. 9A) (third block) is configured as a region (range) in which is searched. In addition,
In FIG. 9A, the search block is the same as the reference block configured for the pixel k.

Further, among the pixels constituting the search block, the pixel having the smallest evaluation value is selected (detected) as the maximum likelihood pixel. That is, the pixel k of the reference camera image
Between each reference block centered on each pixel included in the search block centered on the corresponding detection point on the detected camera image detected for each pixel included in the search block. Of the evaluation values of, assuming that the pixel in the search block for which the smallest value is obtained is the one that is most reliably associated with the corresponding point,
It is detected as the maximum likelihood pixel. In FIG. 9A, the uppermost pixel k in the search block centered on pixel k
₂ is detected as the maximum likelihood pixel.

Then, the distance as the pixel value of the pixel k of the distance image is replaced with, for example, the distance that is the pixel value of the maximum likelihood pixel detected for the pixel k, thereby obtaining the pixel value of the pixel k. Is corrected. That is, FIG.
In (A), the distance (pixel value) for the pixel k is obtained by defining the upper right pixel k ₂ of the search block as the maximum likelihood pixel.
Its is replaced to the distance (pixel value) for the maximum likelihood pixel k _2.

Accordingly, in this correction processing, the pixel k at the boundary (edge) between the foreground and the background in the reference camera image is used.
Corresponding to the detected camera image k ''
While ignoring (FIG. 8), for example, as shown in FIG. 9 (B), a reference block in which background pixels and foreground pixels are not mixed is used to avoid an edge portion where erroneous detection of a corresponding point is likely to occur. It is estimated that the distance obtained from the corresponding point detected by performing the area-based matching is substantially equal to the distance obtained from the true corresponding point k ′ of the pixel k. Then, the distance for the pixel at which the corresponding point is erroneously detected is replaced.

That is, in FIG. 9, for the pixel k,
A reference block containing a pixel of the foreground pixel and the background is constructed (FIG. 9 (A)), the pixel k ₂ beside it, the reference block is formed only in the pixel in the background (Fig. 9 ( B)). For this reason, while the possibility that the corresponding point of the pixel k is erroneously detected is high, the possibility that the corresponding point of the pixel k ₂ is erroneously detected is low, so that the distance for the pixel k may be more accurate. It is replaced by the distance for high pixel k _2.

Although the size of the search block is not particularly limited, if it is too large, a pixel located far from the pixel k may be replaced by a distance. Then, it is difficult to estimate that the distance for such a distant pixel is substantially equal to the true distance for the pixel k, so it is desirable that the size of the search block is not too large. On the other hand, if the size of the search block is too small, a reference block in which the background pixel and the foreground pixel do not coexist in the reference camera image (a reference block that avoids an edge portion where a corresponding point is likely to be erroneously detected) is formed. Are not included in the search block for the pixel k. That is, the search block does not include a pixel having a low possibility of erroneous detection of the corresponding point, and as a result, replacement may be performed with an erroneous distance obtained from an erroneous corresponding point. Therefore, it is desirable not to make the size of the search block too small. That is, the search block is designed to avoid the above-mentioned inconvenience.
It is desirable to have an appropriate size.

Further, the correction of the distance as the pixel value of the pixels forming the distance image can be performed sequentially on all the pixels forming the distance image, but the correction is performed on the pixels. It is also possible to select a pixel (hereinafter referred to as a correction target pixel as appropriate). This will be described later.

Next, FIG. 10 shows an example of a functional configuration of the arithmetic processing circuit 6 of FIG. 1 for performing the above-described stereo processing and correction processing. Note that this functional configuration is realized, for example, by the arithmetic processing circuit 6 executing an application program stored in the storage device 7.

The stereo processing unit 11 (detection means) (processing means) reads out the reference camera image and the detected camera image stored in the frame memory 5 and performs stereo processing to obtain a distance image and the like. I have. That is, the stereo processing unit 11 performs area-based matching using the epipolar line to detect corresponding points of the detected camera image corresponding to the respective pixels of the reference camera image, and based on the corresponding points, detects the reference camera image. For each pixel of, the distance to a point on the three-dimensional space projected on that pixel is determined. Then, the stereo processing unit 11
A distance image having the distance as a pixel value and an evaluation value obtained when a corresponding point is detected are supplied to the distance image storage unit 12.

The distance image storage unit 12 stores the stereo processing unit 1
The distance image and the evaluation value supplied from 1 are stored.

The correction target pixel selection section 13 (selection means)
From the pixels forming the distance image stored in the distance image storage unit 12, a correction target pixel to be subjected to the correction of the distance having the pixel value (target of correction processing) is selected, and the correction unit 14 is instructed. It is made to do. The correction unit 14 (correction unit) determines a pixel value of a pixel included in the distance image stored in the distance image storage unit 12 that has been instructed by the correction pixel selection unit 13 as a correction target pixel. As the distance is corrected.

The distance image storage unit 15 stores the distance image storage unit 1
2, the distance image after correction by the correction unit 14 is stored.

Next, the stereo processing performed by the arithmetic processing circuit 6 of FIG. 10 will be described with reference to the flowchart of FIG. Note that the stereo processing is performed by the stereo processing unit 11 as described above.

As described above, each of the reference camera 1 and the detection camera 2 captures an image of the object 3, and the resulting reference camera image or detection camera image of, for example, one frame is supplied to the frame memory 5 and stored. Is done.

When the reference camera image and the detected camera image are stored in the frame memory 5, in step S1, a variable i representing a pixel constituting the reference camera image (and the distance image) is initialized to, for example, 1. (I = 1
For example, it represents the upper left pixel of the reference camera image), and proceeds to step S2. In step S2, a point on the object epipolar line (three-dimensional space for the pixel i in the reference camera image, a straight line connecting the optical center O ₁ of the base camera 1, the detection camera image (imaging surface S ₂₎ above The distance number j corresponding to the set point set in advance on the above is initialized to, for example, 1, and the process proceeds to step S3.

Here, the detection of the epipolar line on the detected camera image and the setting of the set point for each pixel of the reference camera image are performed, for example, when the reference camera 1 and the detection camera 2 are calibrated. I do. In addition, on the epipolar line in the three-dimensional space,
The distance between the set points corresponding to the distance numbers j and j + 1 (adjacent set points) is determined, for example, according to the distance resolution required for the distance measuring device in FIG.

In step S3, the pixel n (i, j) at the set point corresponding to the distance number j on the epipolar line for the pixel i of the reference camera image is detected from the detected camera image. Note that the pixel n (i, j) can also have sub-pixel accuracy. The relationship between the distance number j and the corresponding set point n (i, j) is obtained, for example, at the time of calibration, and corresponds to the distance number j by making a table (lookup table). The pixel n (i, j) at the set point can be obtained immediately.

Then, the process proceeds to a step S4, wherein the reference block WA (i) centered on the pixel i of the reference camera image is
Along with being extracted (cut out) from the reference camera image, a detection block WB (i, j) centered on the pixel n (i, j) of the detected camera image is extracted from the detected camera image, and the process proceeds to step S5.

In step S5, the reference block WA
Pixels constituting (i) and detection block WB (i, j)
Is used to calculate the evaluation value S (i, n (i, j)) defined by the equation (1), for example, and the process proceeds to step S6. Here, the evaluation value S (i, n
(I, j)) is represented as S (j). In equation (1), the evaluation value S (j) is calculated using the luminance value of the pixel. Therefore, the evaluation value S (j) is calculated based on the pixel constituting the reference block WA (i) and the detection block. It can be said that the similarity regarding the luminance value between the pixels constituting WB (i, j) is expressed. When the evaluation value S (j) is calculated using the luminance value, the reference camera image and the detected camera image only need to have a luminance component. For example, a so-called black-and-white camera that outputs a grayscale image including only luminance signals (does not output color signals) can be used. However, it is possible to use not only the luminance signal but also the color difference signal in the calculation of the evaluation value according to the equation (1).

In step S6, it is determined whether or not the distance number j is equal to the maximum value J. If it is determined that the distance number j is not equal, the process proceeds to step S7, where the distance number j is incremented by one. Then, returning to step S3,
Hereinafter, the processing of steps S3 to S7 is repeated. That is, thereby, the correlation with the pixel i (the evaluation value represented by the equation (1) here) is calculated for all the set points set on the epipolar line for the pixel i.

On the other hand, in step S6, the distance number j
Is determined to be equal to the maximum value J, that is, when the evaluation value S (j) with the pixel i is calculated for all the set points set on the epipolar line for the pixel i, the process proceeds to step S8. Then, a distance number j that minimizes the evaluation value S (j) (maximizes the correlation) is obtained. Here, the distance number j that minimizes the evaluation value S (j) (maximizes the correlation) is hereinafter appropriately referred to as a minimum distance number _jmax .

[0080] Incidentally, the set point n (i, j _max) corresponding to the minimum distance number j _max is the corresponding point of pixel i, therefore, possible to obtain the minimum distance number j _max, the corresponding point n
This is equivalent to detecting (i, j _max ).

Although the distance number j takes an integer value here, it is also possible to obtain the minimum distance number j _max taking a decimal value by performing interpolation as described with reference to FIG. It is possible.

Thereafter, the process proceeds to step S9, where the distance specified by the minimum distance number j _max is supplied from the stereo processing unit 11 to the distance image storage unit 12 as a pixel value for the pixel i constituting the distance image and stored. Is done. further,
Proceeding to step S10, the evaluation value S (j _max ) is supplied to the distance image storage unit 12, and the final evaluation value S
(I) is stored in association with the distance for the pixel i, and the process proceeds to step S11.

In step S11, it is determined whether the pixel i is equal to the last pixel I (for example, the lower right pixel of the reference camera image) constituting the reference camera image. Proceeding to step S12, the variable i is incremented by 1 and returning to step S2. Then, the pixel to be processed next (for example, the next pixel in the line scan order of the pixel currently being processed)
Similar processing is repeated for i.

In step S11, pixel i
Is determined to be equal to the last pixel I constituting the reference camera image, that is, the distance and the evaluation value (final evaluation value) S (j) for all the pixels constituting the reference camera image.
_{If max} ) is obtained, the stereo processing ends.

In FIG. 11, the stereo processing is performed on all the pixels forming the reference camera image. However, the stereo processing is performed on only some of the pixels forming the reference camera image. It is also possible.

Next, the correction processing performed by the arithmetic processing circuit 6 of FIG. 10 will be described with reference to the flowchart of FIG. Note that the correction processing is performed by the correction target pixel selection unit 13 and the correction unit 14 after the stereo processing is performed.

In the correction processing, first, in step S2
In 1, the variable i representing the pixels constituting the distance image (and the reference camera image) is initialized to, for example, 1, and the process proceeds to step S22. In step S22, in the correction target pixel selection unit 13, the evaluation value S (i) of the pixel i stored in the distance image storage unit 12 (step S22 in FIG. 11).
The final evaluation value S (i) stored at 10 is read, and it is determined whether the evaluation value S (i) is smaller than a predetermined threshold value ε.

If it is determined in step S22 that the evaluation value S (i) of the pixel i is smaller than the threshold value ε,
For the pixel i of the reference camera image, the minimum value S (i) of the evaluation value when the corresponding point is detected (S (j described in FIG. 11)
_max )) is smaller than the predetermined threshold value ε, and therefore, if the detected corresponding point is highly certain that it is a true corresponding point, the process proceeds to step S31. In step S31, the correction unit 14 stores the distance Z (i) of the pixel i in the distance image storage unit 1.
2 and supplied to and stored in the distance image storage unit 15, and the process proceeds to step S29. Therefore, in this case, the distance Z (i) for the pixel i obtained by the stereo processing is obtained.
Is not corrected.

On the other hand, in step S22, when it is determined that the evaluation value S (i) of the pixel i is not smaller than the threshold value ε, that is, for the pixel i of the reference camera image, The minimum value S (i) is equal to a predetermined threshold ε
If the detected corresponding point has low reliability as a true corresponding point, the process proceeds to step S23.
The correction of the distance for the pixel i obtained by the stereo processing is performed as described with reference to FIG.

That is, the pixel-to-be-corrected selecting section 13 selects the pixel i as the pixel to be corrected and notifies the correcting section 14 of the selected pixel. When the correction unit 14 receives a notification from the correction target pixel selection unit 13 that the pixel i is the correction target pixel, in step S23, the variable j representing the pixel in the search block W centered on the pixel i is For example, it is initialized to a value j ₀ representing the upper left pixel in the search block W. Further, in step S23, by referring to the distance image storage unit 12 in the correction unit 14, the variable S _min holding the minimum evaluation value of the pixel in the search block W becomes, for example,
The evaluation value S (j ₀ ) of the pixel j ₀ is initialized, and the variable Z holding the distance is initialized to, for example, the distance Z (j ₀ ) for the pixel j ₀ .

Then, the process proceeds to a step S24, wherein the correction unit 1
4 reads the evaluation value S (j) for the pixel j from the distance image storage unit 12 and sets the variable S _min to the evaluation value S (j).
Determine if it is less than. In step S24, when it is determined that the variable S _min is not smaller than the evaluation value S (j), that is, when the evaluation value S (j) is equal to or less than the variable S _min , the process proceeds to step S25, and the correction unit 14 Is
A variable S _min, with newly set the evaluation value S (j), to the variable Z, newly set distance Z a (j) for the pixel j, the process proceeds to step S26.

In step S24, the variable S
_When it is determined that _min is smaller than the evaluation value S (j),
The process skips step S25 and proceeds to step S26, where it is determined whether or not the pixel j is the last pixel in the search block W (for example, the lower right pixel of the search block W). When it is determined in step S26 that the pixel j is not the last pixel in the search block W,
Proceeding to step S27, the variable j is set to a value representing a pixel whose evaluation value has not yet been compared with the variable S _min among the pixels in the search block W (for example, among the pixels in the search block W). Then, a value representing the next pixel in the line scan order of the pixel to be processed is set), and the process returns to step S24. Then, step S26
, The processes of steps S24 to S27 are repeated until it is determined that the pixel j is the last pixel in the search block W. That is, thereby, the minimum value of the evaluation values of the pixels in the search block W is set to the variable S _min ,
That is, the evaluation value of the maximum likelihood pixel is set, and the distance for the maximum likelihood pixel is set in the variable Z.

Then, in step S26, pixel j
Is determined to be the last pixel in the search block W, the process proceeds to step S28, and the correction unit 14 determines the distance Z (i) for the pixel i stored in the distance image storage unit 12.
Is replaced with the maximum likelihood pixel distance Z, and the process proceeds to step S29. That is, the correction unit 14 writes the maximum likelihood pixel distance Z as the distance of the pixel i into the distance image storage unit 15, and proceeds to step S29.

In step S29, step S of FIG.
11, it is determined whether or not the pixel i is equal to the last pixel I that forms the range image. If it is determined that the pixel i is not equal to the pixel I, the process proceeds to step S30 and the variable i is determined.
Is incremented by 1, and the process returns to step S22.
Therefore, in this case, the same process is repeated for the pixel i to be processed next.

In step S29, pixel i
Is determined to be equal to the last pixel I constituting the range image, the correction process ends.

As described above, a pixel whose evaluation value is not smaller than the predetermined threshold value ε is selected as a correction target pixel, and the distance of the correction target pixel is replaced with the distance of the pixel having the minimum evaluation value in the search block. In this case, the distance of a pixel with low certainty that the detected corresponding point is a true corresponding point is a pixel in the vicinity of the pixel, and the detected corresponding point is a true corresponding point. Since the distance is replaced by the distance having the highest certainty as the corresponding point, the boundary between the foreground and the background, that is, the edge as described with reference to FIG. Reduce the effect on distance caused by erroneous detection of corresponding points,
A relatively accurate distance can be obtained.

The threshold value ε of the evaluation value used to determine whether or not the pixels forming the range image are to be corrected pixels is
For example, an optimal value is set by an experiment or the like. Or,
Further, the threshold value ε can be set adaptively based on the pixel values of the reference camera image and the detected camera image stored in the frame memory 5.

Next, as described above, a pixel whose evaluation value is not smaller than the predetermined threshold value ε is selected as a correction target pixel, and the distance of the correction target pixel is determined by the pixel having the minimum evaluation value (the maximum) in the search block. In the case of replacing with the distance of the maximum likelihood pixel, the distance of the pixel located at the center of the search block including the maximum likelihood pixel is the distance of the maximum likelihood pixel even though the evaluation value is not so large. May be replaced by Therefore, for example, each of the evaluation values of a certain set of pixels is somewhat larger than the threshold ε, and
If a pixel whose evaluation value is somewhat smaller than the threshold value ε exists around the set of pixels, the distance of each of the set of pixels is replaced with the distance of a pixel having a small evaluation value around the set. In this case, since the distance of each pixel in the group is the same, block-like distortion occurs in the distance image.

That is, as shown in FIG. 13, the correction processing described with reference to FIG. 12 is performed on the distance image obtained for the object 3 whose distance (depth) continuously changes, such as a cylinder. In this case, a point obtained by projecting a certain point on the side surface of the cylinder is set as a correction target pixel, and even if the distance between the correction target pixel and the maximum likelihood pixel continuously changes (the Although there is no large difference between the two distances, there is some difference), the distance of the pixel to be corrected may be replaced with the distance of the maximum likelihood pixel in the search block around the pixel. . In this case, since the distance between the pixel set as the pixel to be corrected and the distance of the maximum likelihood pixel are the same, a smooth change in distance (continuity of distance) on the side surface of the cylinder is lost.

Therefore, the correction processing is performed by calculating the distance between a certain pixel and
If the difference between the pixel and the distance from the maximum likelihood pixel in the search block to the center is large to some extent, the pixel can be set as the correction target pixel. in this case,
When the difference between the distance of a certain pixel and the distance between the pixel and the maximum likelihood pixel in the search block centered on the pixel is small, the pixel is no longer a correction target pixel, and as a result, for example,
It is possible to prevent loss of continuity of the distance on the side surface of the cylinder as described above, and to reduce only the influence on the distance due to erroneous detection of the corresponding point occurring at the edge portion.

That is, FIG. 14 shows the operation processing circuit 6 of FIG.
2 shows a flowchart in the case where such a correction process is performed.

In this case, first, in step S41, a variable i representing a pixel constituting the distance image (and the reference camera image) is initialized to, for example, 1, and the process proceeds to step S43. In step S43, as in step S23 of FIG. 12, the variable j representing a pixel in the search block W centered on the pixel i is, for example, a value j ₀ representing the upper left pixel in the search block W. And a variable Z _min that holds the minimum evaluation value of the pixel in the search block W is initialized to, for example, the evaluation value S (j ₀ ) of the pixel j ₀ and a variable Z that holds the distance. Is initialized to a distance Z (j ₀ ) for the pixel j ₀ , for example.

Then, the process proceeds to a step S44, and thereafter, in the steps S44 to S47, the step S2 of FIG.
4 to S27, the minimum value of the evaluation values of the pixels in the search block W, that is, the evaluation value of the maximum likelihood pixel is set as the variable S _min , In Z, the distance for the maximum likelihood pixel is set.

When the evaluation value of the maximum likelihood pixel is set in the variable S _min , and the distance for the maximum likelihood pixel is set in the variable Z, steps S 46 to S 4
8, the distance Z (i) of the pixel i is read from the distance image storage unit 12 by the correction target pixel selection unit 13,
The absolute value of the difference between the distance Z (i) and the variable Z (= | Z
It is determined whether (i) -Z |) is smaller than a predetermined threshold Th.

In step S48, the distance Z of the pixel i
When it is determined that the absolute value of the difference between (i) and the variable Z is not smaller than the threshold Th, that is, the distance Z (i) of the pixel i
And the distance Z between the pixel and the maximum likelihood pixel in the search block W centered on the pixel i is somewhat large, the correction target pixel selection unit 13 selects the pixel i as the correction target pixel. To the correction unit 14. The correction unit 14
When the notification that the pixel i is the correction target pixel is received from the correction target pixel selection unit 13, the process proceeds to step S49, and the distance Z for the pixel i stored in the distance image storage unit 12 is determined.
The correction is performed by replacing (i) with the maximum likelihood pixel distance Z, that is, the maximum likelihood pixel distance Z is written to the distance image storage unit 15 as the distance of the pixel i, and the process proceeds to step S50.

If it is determined in step S48 that the absolute value of the difference between the distance Z (i) of the pixel i and the variable Z is smaller than the threshold Th, the correction target pixel selection unit 13
Without selecting the pixel i as the pixel to be corrected, step S
Go to 52. In step S52, the correction unit 14 reads the distance Z (i) of the pixel i from the distance image storage unit 12, supplies the distance Z (i) to the distance image storage unit 15, and stores the distance Z (i), and proceeds to step S50. Therefore, in this case, the correction of the distance Z (i) for the pixel i obtained by the stereo processing is not performed.

In step S50, step S50 in FIG.
29, it is determined whether or not the pixel i is equal to the last pixel I constituting the range image. If it is determined that the pixel i is not equal, the process proceeds to step S51, where the variable i
Is incremented by 1, and the process returns to step S43.
Then, the same process is repeated for the pixel i to be processed next.

In step S50, pixel i
Is determined to be equal to the last pixel I constituting the range image, the correction process ends.

The distance threshold Th used to determine whether or not the pixels forming the distance image are the correction target pixels is:
For example, the distance can be set to an optimal value by an experiment or the like, or can be set adaptively from the distance as the pixel value of the pixel of the distance image stored in the distance image storage unit 5.

In the embodiment of FIG. 14, the difference between the distance Z (i) of a certain pixel i and the maximum likelihood pixel in the search block W centered on the pixel i is equal to or larger than the threshold Th. Only in the case of, the pixel i is selected as the correction target pixel. However, the distance Z (i) of a certain pixel i and the distance between the maximum likelihood pixel in the search block W centered on the pixel i are Is greater than or equal to the threshold Th and the evaluation value S of the pixel i
When (i) is equal to or larger than the threshold ε, the pixel i can be selected as a correction target pixel. The correction process in this case is as shown in the flowchart of FIG. That is, in the selection of the correction target pixel, it is determined whether or not the evaluation value S (i) of the pixel i is smaller than the threshold value ε after the process of step S41 in FIG. Step S42 is performed, and if the evaluation value S (i) of the pixel i is not smaller than the threshold value ε, the process proceeds to step S43, and if it is smaller, the process proceeds to step S52.

Next, in the embodiment of FIG. 14, the evaluation values of the pixels constituting the search block W centered on a certain pixel i are compared with each other, and the pixel having the minimum evaluation value in the search block W is compared. (Maximum likelihood pixel) (Step S
44 to S47), it is determined whether or not the pixel i is selected as a correction target pixel in consideration of the difference in distance (step S48). However, for example, the search is performed in consideration of the difference in distance. It is possible to select a pixel that can be the maximum likelihood pixel in the block in advance, and to compare the evaluation values only for the selected pixel.

Further, in the embodiment shown in FIG.
Pixel i is selected as a correction target pixel based on the difference (absolute value) between the distance Z (i) of the pixel and the distance Z of the maximum likelihood pixel in the search block W centered on the pixel i. Is determined (step S48).
For example, it is determined whether or not to select the pixel i as a correction target pixel based on the pixel value of the pixel i such as luminance and color information (for example, saturation and hue) in the reference camera image. It is possible.

That is, FIG. 16 shows the operation processing circuit 6 of FIG.
2 shows a flowchart in the case where such a correction process is performed.

In this case, first, in step S61, a variable i representing a pixel constituting the distance image (and the reference camera image) is initialized to, for example, 1, and the process proceeds to step S62. In step S62, the correction target pixel selection unit 13 stores, for example, the luminance Y (i) as the pixel value of the pixel i on the reference camera image in the frame memory 5
Is read out as indicated by the dotted line in FIG.
Further, the correction target pixel selection unit 13 stores the distance Z of the pixel i from the distance image storage unit 12 in step S52.
(I) is read, and the process proceeds to step S63.

In step S63, the variable j representing the pixel in the search block W centered on the pixel i is changed by the correction target pixel selection unit 13 in the same manner as in step S23 in FIG. Is initialized to the value j ₀ representing the upper left pixel in Further, the correction target pixel selecting unit 13 determines in step S63 that the variable S
Initialize _min or Z to a value that cannot be taken as an evaluation value or distance, for example, −1, and step S
Proceed to 64.

In step S64, the correction target pixel selector 13 selects the luminance Y of the pixel j on the reference camera image.
(J) is read from the frame memory 5 and the distance Z (j) is read from the distance image storage unit 12. Further, in step S64, in the correction target pixel storage unit 13, the luminance Y (i) of the pixel i and the pixel j
The absolute value of the difference from the luminance Y (j) (= | Y (i) −Y
(J) |) is smaller than the predetermined threshold value Th _Y, and (a i), the absolute value of the difference between the distance Z (j) of the pixel j (= | Z (i) the distance of the pixel i Z -Z | ) whether greater than the predetermined threshold value Th _Z is determined.

Here, the threshold value Th _Y is set, for example, to an optimum value by an experiment or the like, or is adaptively set from the pixel values of the reference camera image and the detected camera image stored in the frame memory 5. Is possible. The threshold value Th _Z can be set in the same manner as the threshold value Th described with reference to FIG.

In step S64, the luminance Y of pixel i
And (i), the absolute value of the difference between the luminance Y (j) of the pixel j is either not less than the predetermined threshold value Th _Y, or the distance of the pixel i Z (i), the distance of the pixel j Z (j) The absolute value of the difference between
When it is determined that the value is not larger than the predetermined threshold Th _Z ,
Steps S65 to S68 are skipped, and step S
Go to 69.

[0119] Further, in step S64, the luminance Y (i) of the pixel i, the absolute value of the difference between the luminance Y (j) of the pixel j is less than the predetermined threshold value Th _Y, and the distance of the pixel i Z
And (i), the absolute value of the difference between the distance Z (j) of the pixel j is, if it is determined greater than the predetermined threshold value Th _Z, the process proceeds to step S65, the correction target pixel selecting section 13, a variable S _min (Or Z) is determined to be equal to the initial value -1.

In step S65, the variable S _min is
If it is determined that it is equal to 1, the process proceeds to step S66,
The correction target pixel selection unit 13 calculates the evaluation value S for the pixel j.
(J) is read from the distance image storage unit 12 and the variable S
together set to _min, distance of the pixel j Z a (j),
The variable Z is set, and the process proceeds to step S69.

In step S65, the variable S
If it is determined that _min is not equal to -1, step S
Proceeding to 67, the correction target pixel selection unit 13 sets the variable S
_It is determined whether _min is smaller than the evaluation value S (j) for pixel j. In step S67, the variable S
_When it is determined that _min is smaller than the evaluation value S (j),
Proceeding to step S68, the correction target pixel selecting unit 13 sets the evaluation value S (j) for the pixel j in the variable S _min and sets the distance Z (j) of the pixel j in the variable Z. Proceed to S69.

In step S67, the variable S
If it is determined that _min is not smaller than the evaluation value S (j), step S68 is skipped and the process proceeds to step S69, and as in step S26 of FIG.
It is determined whether the pixel j is the last pixel in the search block W. In step S69, pixel j is
If it is determined that the pixel is not the last pixel in the search block W, the process proceeds to step S70, and the evaluation value is still set to the variable j among the pixels in the search block W as in step S27 of FIG. , A value representing a value not compared with the variable S _min is set, and the process returns to step S64. Then, in step S69, the processing of steps S64 to S70 is repeated until it is determined that the pixel j is the last pixel in the search block W.

That is, the search block W
The difference between the luminance Y (i) of the pixel i to be located at the center has a luminance less than the threshold Th _Y, and the distance of the pixel i Z (i)
Is greater than the threshold value Th _Z ,
The pixel j is selected as a pixel appropriate for use in correcting the distance Z (i) of the pixel i (hereinafter, appropriately referred to as an appropriate pixel). Of the suitable pixels in the search block W, as evaluation value is smallest is selected as the maximum likelihood pixel, evaluation value or distance of the maximum likelihood pixel, the variable S _{mi n} or Z, are set, respectively You.

Then, in step S69, pixel j
Is determined to be the last pixel in the search block W, the process proceeds to step S71, and the correction target pixel selection unit 13
Determines whether the variable S _min (or Z) is equal to the initial value −1. When it is determined in step S71 that the variable S _min is not equal to −1, that is,
When the evaluation value of any pixel (the maximum likelihood pixel) in the search block W is set in the variable _min , the correction target pixel selection unit 13 selects the pixel i as the correction target pixel, and 14, and the process proceeds to step S72.

When the correction unit 14 receives a notification from the correction target pixel selection unit 13 that the pixel i is the correction target pixel, in step S72 the distance Z for the pixel i stored in the distance image storage unit 12 is determined. The correction is performed by replacing (i) with the distance (variable) Z of the maximum likelihood pixel, that is, the distance Z of the maximum likelihood pixel is written into the distance image storage unit 15 as the distance of the pixel i, and the process proceeds to step S73.

In step S71, the variable S
_When it is determined that _min is equal to −1, that is, the luminance Y (i) of the pixel i located at the center of the search block W is determined.
The difference between the is case have a luminance less than the threshold Th _Y, and the difference between the distance of the pixel i Z (i) is, if the threshold Th _Z larger pixel j does not exist, that is, the right pixel is not present,
Proceeding to step S75, the correction unit 14 determines the distance Z of the pixel i.
(I) is read from the distance image storage unit 12 and supplied to and stored in the distance image storage unit 15, and the process proceeds to step S73. Therefore, in this case, the pixel i is not selected as a correction target pixel, and as a result, the distance Z (i) is not corrected.

In step S73, step S of FIG.
11, it is determined whether the pixel i is equal to the last pixel I forming the range image. If it is determined that the pixel i is not equal, the process proceeds to step S74, where the variable i
Is incremented by 1, and the process returns to step S62.
Then, the same process is repeated for the pixel i to be processed next.

In step S73, the pixel i
Is determined to be equal to the last pixel I constituting the range image, the correction process ends.

Therefore, according to the embodiment shown in FIG. 16, the luminance Y of the pixel i located at the center of the search block W is determined.
The difference between (i) is the threshold value Th _Y less than a brightness, and the difference between the distance Z (i) of the pixel i is, if there is a threshold Th _Z greater than right pixel, pixel i is the pixel to be corrected Is selected as Then, among the appropriate pixels, the pixel having the smallest evaluation value is selected as the maximum likelihood pixel, and replacement with the distance of the maximum likelihood pixel is performed.
(I) is corrected.

Next, FIG. 17 is a flowchart showing another embodiment of the correction processing, and the same correction processing as that shown in the flowchart of FIG. 16 can be performed by this correction processing.

That is, in this case, first, step S
At 81, a variable i representing a pixel constituting the distance image (and the reference camera image) is initialized to, for example, 1;
Proceed to step S82. In step S82, the correction target pixel selection unit 13 reads, for example, the luminance Y (i) as the pixel value of the pixel i on the reference camera image from the frame memory 5 as indicated by a dotted line in FIG. Further, the correction target pixel selection unit 13 determines in step S
At 82, the distance Z (i) of the pixel i is read from the distance image storage unit 12, and the process proceeds to step S83.

In step S83, the variable j representing the pixel in the search block W centered on the pixel i is changed by the correction target pixel selection unit 13 in the same manner as in step S23 in FIG. Is initialized to the value j ₀ representing the upper left pixel in Further, the correction target pixel selection unit 13 determines in step S83 that the variable S
_min or Z is the evaluation value S (i) of the pixel i or the distance Z
Initialize to (i), respectively, and proceed to step S84.

In step S84, the correction target pixel selector 13 selects the luminance Y of the pixel j on the reference camera image.
(J) is read from the frame memory 5 and the distance Z (j) is read from the distance image storage unit 12. Further, in step S84, in the correction target pixel storage unit 13, the luminance Y (i) of the pixel i and the pixel j
The absolute value of the difference from the luminance Y (j) (= | Y (i) −Y
(J) |) is smaller than the predetermined threshold value Th _Y, and (a i), the absolute value of the difference between the distance Z (j) of the pixel j (= | Z (i) the distance of the pixel i Z -Z | ) whether greater than the predetermined threshold value Th _Z is determined.

In step S84, the luminance Y of the pixel i is determined.
And (i), the absolute value of the difference between the luminance Y (j) of the pixel j is either not less than the predetermined threshold value Th _Y, or the distance of the pixel i Z (i), the distance of the pixel j Z (j) The absolute value of the difference between
When it is determined that the value is not larger than the predetermined threshold Th _Z ,
Steps S85 and 86 are skipped and step S85
Proceed to 87.

[0135] Further, in step S84, the luminance Y (i) of the pixel i, the absolute value of the difference between the luminance Y (j) of the pixel j is less than the predetermined threshold value Th _Y, and the distance of the pixel i Z
And (i), the absolute value of the difference between the distance Z (j) of the pixel j is, if it is determined greater than the predetermined threshold value Th _Z, the process proceeds to step S85, the in the correction target pixel selecting section 13, a variable S _min Is smaller than the evaluation value S (j) for the pixel j. If it is determined in step S85 that the variable S _min is smaller than the evaluation value S (j), the process proceeds to step S86, where the correction target pixel selection unit 13
Sets the evaluation value S (j) of the pixel j in the variable S _min and sets the distance Z (j) of the pixel j in the variable Z.
Is set, and the process proceeds to step S87.

At step S85, the variable S
If it is determined that _min is not smaller than the evaluation value S (j), step S86 is skipped and the process proceeds to step S87, and as in step S26 in FIG.
It is determined whether the pixel j is the last pixel in the search block W. In step S87, the pixel j is
When it is determined that the pixel is not the last pixel in the search block W, the process proceeds to step S88, and the variable j is set to the variable j as in the case of step S27 in FIG. , A value representing a value not compared with the variable S _min is set, and the process returns to step S84. Then, in step S87, the processing in steps S84 to S88 is repeated until it is determined that the pixel j is the last pixel in the search block W.

That is, thereby, an appropriate pixel is selected from the search block W, and the appropriate pixel having the smallest evaluation value is selected as the maximum likelihood pixel, and the evaluation value or the distance of the maximum likelihood pixel is selected. Is set to a variable S _min or Z, respectively. If no appropriate pixel exists in the search block W, the variable S _min or Z is set to the evaluation value S of the pixel i set in step S83.
(I) or the distance Z (i) remains set.

Then, in step S87, the pixel j
Is determined to be the last pixel in the search block W, the process proceeds to step S89, and the correction unit 14 determines the distance Z (i) for the pixel i stored in the distance image storage unit 12.
Is replaced by a variable Z, that is, the variable Z
Is supplied to and stored in the distance image storage unit 15 as the distance of the pixel i, and the process proceeds to step S90.

Here, as described above, the search block W
If there is an appropriate pixel among the pixels, the one with the smallest evaluation value is selected as the maximum likelihood pixel and its distance is set to the variable Z, so that the pixel at the center of the search block W in that case The fact that the distance Z (i) of i is replaced by the variable Z substantially means that the pixel i has been selected as the correction target pixel and the distance Z (i) has been corrected. On the other hand, when there is no appropriate pixel in the search block W, the variable Z is set to the distance Z (i) of the pixel i. When the distance Z (i) of a certain pixel i is a variable Z
Is substantially equivalent to no correction being performed, and therefore the pixel i is
This means that the pixel has not been selected as a correction target pixel.

In step S90, step S90 in FIG.
11, it is determined whether the pixel i is equal to the last pixel I forming the range image, and if it is not, the process proceeds to step S91, where the variable i
Is incremented by 1, and the process returns to step S82.
Then, the same process is repeated for the pixel i to be processed next.

In step S90, pixel i
Is determined to be equal to the last pixel I constituting the range image, the correction process ends.

As described above, based on the difference between the pixel i and the distance Z (i) as well as the difference between the pixel i and the pixel value on the reference camera image, the selection of whether or not the pixel i is the pixel to be corrected is made. The most likely pixel is selected from the appropriate pixels having a small difference in pixel value, and the distance of the pixel i is corrected by the distance of the last pixel, whereby an erroneous correction is performed on the pixel i. Can be prevented.

That is, even when, for example, an image having a low foreground luminance and a high luminance of the background (an image having a dark foreground and a bright background) is obtained as the reference camera image and the detected camera image, as described above. In the edge part,
Corresponding points are erroneously detected for a certain pixel i that constitutes the background, so that the distance of the foreground may be erroneously obtained as the distance.

Of the pixels constituting the search block centered on pixel i, the evaluation value for a pixel whose reference block is composed of only background pixels is generally smaller than the evaluation value of both the foreground and background pixels. As described with reference to FIG. 12, the distance of the pixel i is replaced by the distance of the pixel of the background at which such a small evaluation value is obtained. i
Is corrected from the wrong foreground distance to the correct background distance.

However, when the luminance of the foreground is low and the luminance of the background is high, contrary to the above case, the evaluation value of the pixel forming the reference block with both the foreground and background pixels is smaller. However, in some cases, the evaluation value may be smaller than the evaluation value of a pixel in which a reference block includes only background pixels. Therefore, when the pixel from which such a small evaluation value is obtained is a foreground pixel, as described with reference to FIG.
Replacing the distance of pixel i will result in the distance of pixel i being corrected from the wrong foreground distance to again the wrong foreground distance.

[0146] Therefore, as described with reference to FIGS. 16 and 17, in the search block, the difference between the luminance of the pixel i to be located at the center, right pixels there is a pixel having a luminance less than the threshold Th _Y In such a case, the pixel i is selected as a correction target pixel, and the distance of the correction target pixel i is replaced with the distance of the evaluation value of the appropriate pixel having the smallest evaluation value. Even when the luminance of the background is high, the erroneously obtained foreground distance for pixel i can be replaced with the correct background distance.

That is, the appropriate pixel has a luminance close to that of the pixel i. Therefore, it is highly possible that the appropriate pixel forms the same background as the pixel i in the search block.
Other pixels are more likely to form the foreground.
Therefore, pixels having a high possibility of forming the foreground are ignored, and only the appropriate pixels having a high possibility of forming the background are selected, and the pixel having the smallest evaluation value is selected as the maximum likelihood pixel. The correction target pixel i is determined by the distance of the maximum likelihood pixel.
Is substituted for the pixel i, the erroneously obtained foreground distance is corrected to the correct background distance, thereby preventing erroneous correction from being performed. It becomes possible.

It is preferable that the correction process is performed after noise removal or hole filling is performed on the distance image obtained by the stereo processing. This is because when the reference camera image or the detected camera image has little change in luminance, erroneous detection of the corresponding point is likely to occur in the stereo processing, while the final evaluation value obtained for each pixel is obtained. Is often small, and if the correction processing is performed using the distance image obtained as a result of such stereo processing as it is, an adverse effect such as an increase in noise may occur.

Here, as a method of performing noise removal or padding on a distance image, for example, a set of adjacent pixels having a short distance value is defined as one unit, and a portion having a small area of one unit is subjected to noise reduction. In the set of pixels determined to be noise, pixels adjacent to non-noise pixels are corrected by applying a median filter, and the noise portion is gradually narrowed and finally eliminated. There is something. The details of this method are described, for example, in Japanese Patent Application No. 9-288479 filed earlier by the present applicant.

As described above, the case where the present invention is applied to a distance measuring apparatus for measuring a distance has been described. However, as described above, the present invention relates to a case where parallax is measured and a case where a further distance is obtained from the parallax. The present invention is also applicable to the case of analyzing a three-dimensional shape of an object. Here, in the case of obtaining the distance from the parallax, the same applies to the case where the distance is obtained after performing the correction processing on the parallax, or the case where the distance is obtained from the parallax and the correction processing is performed on the distance. The effect can be obtained.

In this embodiment, the correction processing is performed after the stereo processing is completed. However, the stereo processing is performed until the correction processing becomes possible, and thereafter, the stereo processing and the correction processing are performed in parallel. It is possible to do.

Further, in the present embodiment, the detection of the maximum likelihood pixel from the search block is performed for all the pixels constituting the search block. It is possible to target only the pixels located on the outer periphery of the search block, or to target only every other pixel among the pixels constituting the search block. In this case, detection of the maximum likelihood pixel can be performed at high speed.

Further, in the present embodiment, the stereo processing and the correction processing are performed by causing the arithmetic processing circuit 6 to execute a computer program, but the stereo processing and the correction processing are performed by dedicated hardware. It is also possible to do this.

Further, in the present embodiment, the application program for performing the stereo processing and the correction processing is stored in the storage device 7 and provided. It is also possible to provide by transmitting via a transmission medium.

Further, in this embodiment, rectangular blocks are configured as the reference block, the detection block, and the search block.
The detection block and the search block are not limited to rectangular blocks. For example, a circular block including pixels within a certain distance from a certain pixel is adopted as a reference block, a detection block, and a search block. It is also possible. Further, the reference block, the detection block, and the search block do not need to be configured by adjacent pixels, and may be configured by a set of pixels every other pixel, for example.

Further, in the present embodiment, one camera 1 functioning as a reference camera and one camera 2 functioning as a detection camera are used, but a plurality of detection cameras are used. It is possible. Also,
By preparing only one camera and moving it, it is also possible to use that one camera as both the reference camera and the detection camera.

Further, in the present embodiment, the distance image obtained by the stereo processing and the correction processing is displayed on the output device 8 or stored in the storage device 7. For example, it is possible to transmit the data via a transmission medium such as the Internet and provide the data to another device or the like.

Further, in the present embodiment, the correction is performed by replacing the distance of the pixel to be corrected with the distance of one maximum likelihood pixel. Selecting a plurality of maximum likelihood pixels in the order of smaller values, correcting based on an average value of the distances of the plurality of maximum likelihood pixels, and performing interpolation using the distances of the plurality of maximum likelihood pixels. And so on.

Further, in the present embodiment, the corrected distance image is stored in a distance image storage unit 15 different from distance image storage unit 12.
The distance image after correction is stored in the distance image storage unit 12 in which the distance image before correction is stored.
It is also possible to store the data in an overwritten form.

Further, in the present embodiment, the distance image after the stereo processing is subjected to the correction processing only once, but the correction processing is further applied to the corrected distance image. It is also possible to do so.

[0161]

According to the image processing apparatus according to the first aspect, the image processing method according to the twentieth aspect, and the providing medium according to the thirty-eighth aspect, the pixels correspond to the pixels of the first image.
The corresponding pixel of the second image is detected, and distance-related information is obtained for the pixel of the first image based on the corresponding pixel. Further, from the pixels of the distance-related information image having the pixel value of the distance-related information, a correction target pixel for which the distance-related information is to be corrected is selected. Is corrected based on the distance relationship information for the other pixels. Therefore, the first
Even if it is not possible to accurately associate pixels between the second images, it is possible to obtain highly accurate distance relationship information.

According to the providing medium of claim 39, a corresponding pixel of the second image corresponding to a pixel of the first image is detected, and based on the corresponding pixel, a pixel of the first image is detected.
The distance relationship information is obtained, and a correction target pixel for which the distance relationship information is to be corrected is selected from the pixels of the distance relationship information image having the pixel value of the distance relationship information. Distance relationship information obtained by performing correction based on the distance relationship information for other pixels of the relationship information image is provided. Therefore, highly accurate distance relation information can be provided.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration example of an embodiment of a distance measuring device to which the present invention is applied.

FIG. 2 shows an object 3 with a reference camera 1 and a detection camera 1;
It is a figure showing the state where it is photographing.

FIG. 3 is a diagram for explaining an epipolar line.

FIG. 4 is a diagram showing a reference camera image and a detected camera image.

FIG. 5 is a diagram showing a transition of an evaluation value.

FIG. 6 is a diagram showing a set point / distance table and a parallax / distance table.

FIG. 7 is a diagram showing a state in which an object 3 placed in front of a wall is being photographed.

FIG. 8 is a diagram for explaining erroneous detection of a corresponding point occurring in an edge portion.

FIG. 9 is a diagram for explaining a method of correcting a distance error caused by erroneous detection of a corresponding point.

FIG. 10 is a block diagram illustrating a functional configuration example of an arithmetic processing circuit 6 in FIG. 1;

FIG. 11 is a flowchart illustrating stereo processing.

FIG. 12 is a flowchart illustrating a first embodiment of a correction process.

FIG. 13 is a diagram for explaining a case where distance continuity is lost.

FIG. 14 is a flowchart for explaining a second embodiment of the correction processing.

FIG. 15 is a flowchart illustrating a third embodiment of the correction process.

FIG. 16 is a flowchart for explaining a fourth embodiment of the correction processing.

FIG. 17 is a flowchart for describing a fifth embodiment of the correction processing.

[Explanation of symbols]

1 reference camera (imaging means), 2 detection camera (imaging means), 3 object, 4 camera interface, 5 frame memory, 6 arithmetic processing circuit, 6
A CPU, 6B ROM, 6C RAM, 6D
DSP, 7 storage device, 8 output device, 11 stereo processing unit (detection unit) (processing unit), 12 distance image storage unit, 13 correction target pixel selection unit (selection unit), 14 correction unit (correction unit), 15 distance Image storage unit

Continued on the front page F term (reference) 2F065 AA04 AA06 AA53 BB05 FF05 JJ03 JJ26 QQ24 QQ31 UU05 5B057 DA07 DB03 DC08 DC34 5L096 AA09 CA05 FA34 FA66 FA67 GA53 JA18 9A001 HH23 JJ49 JJ71

Claims (39)

[Claims] 1. A first and a second photographing a predetermined object.
An image processing apparatus that obtains distance-related information related to a distance to the target object based on the image of (a), wherein detection corresponding to pixels of the first image and corresponding pixels of the second image are detected. Means, based on the corresponding pixels, processing means for obtaining the distance relationship information for the pixels of the first image, from the pixels of the distance relationship information image having the distance relationship information as a pixel value, Selecting means for selecting a correction target pixel to be corrected; and correcting means for correcting the distance relation information for the correction target pixel based on the distance relation information for other pixels of the distance relation information image. An image processing apparatus comprising: 2. The image processing apparatus according to claim 1, wherein the processing unit obtains a parallax between a pixel of the first image and a corresponding pixel as the distance relation information. 3. The processing unit, based on the parallax,
3. The image processing apparatus according to claim 2, wherein a distance to the object is further obtained. 4. The image processing apparatus according to claim 1, wherein the processing unit obtains a distance to the object as the distance relation information. 5. The detecting means calculates a predetermined evaluation function using pixel values of pixels of the first and second images, and detects the corresponding pixel based on a result of the calculation. The image processing apparatus according to claim 1, wherein: 6. The image processing apparatus according to claim 5, wherein the selection unit selects the correction target pixel based on a value of the predetermined evaluation function. 7. The image processing apparatus according to claim 1, wherein the selection unit selects the correction target pixel based on the distance relation information. 8. The image processing apparatus according to claim 1, wherein the selection unit selects the correction target pixel based on a pixel value of a pixel of the first image. 9. The method according to claim 1, wherein the correction unit corrects the distance relation information on the correction target pixel based on the distance relation information on a pixel located near the correction target pixel. 2. The image processing device according to 1. 10. The correction unit according to claim 1, wherein the correction unit performs the correction by replacing the distance relation information about the correction target pixel with the distance relation information about another pixel of the distance relation information image. Item 2. The image processing device according to Item 1. 11. The correction means for correcting the distance relationship information for the correction target pixel based on the distance relationship information for another pixel in a predetermined area including the correction target pixel. The image processing apparatus according to claim 1. 12. The correction means selects a pixel in the predetermined area based on the certainty of the correspondence between the pixel of the first image and the corresponding pixel, and selects the distance between the selected pixel and the distance. The image processing apparatus according to claim 11, wherein the distance relation information on the correction target pixel is corrected based on the relation information. 13. The method according to claim 1, wherein the correction unit selects, from among the pixels in the predetermined area, a pixel having the highest certainty in the correspondence between the pixel of the first image and its corresponding pixel. The image processing device according to claim 12. 14. The image processing apparatus according to claim 12, wherein the correction unit selects a part of the pixels in the predetermined area as a selection target. 15. The image processing apparatus according to claim 14, wherein the correction unit selects a pixel located at an outermost periphery of the predetermined area as a selection target. 16. The method according to claim 16, wherein the detecting unit includes the first and second detectors.
Using a pixel value of the pixel of the image of, a predetermined evaluation function is calculated, based on the calculation result, the corresponding pixel is detected, the correction unit, based on the value of the predetermined evaluation function,
13. The image processing apparatus according to claim 12, wherein the certainty of the correspondence between the pixels of the first image and the corresponding pixels is evaluated. 17. The method according to claim 1, wherein the detecting unit determines a predetermined evaluation function for a pixel value of a pixel of the first image and a pixel value of each pixel group of the second image which may correspond to the pixel. Is calculated, and based on the calculation result, the corresponding pixel is detected from the pixel group of the second image. The correction unit is configured to calculate the corresponding pixel based on a value of the predetermined evaluation function.
13. The image processing apparatus according to claim 12, wherein the certainty of the correspondence between the pixels of the first image and the corresponding pixels is evaluated. 18. The method according to claim 17, wherein the detecting unit includes a first pixel block including a pixel of the first image, and a second pixel including a pixel of a pixel group of the second image which may correspond to the pixel. A predetermined evaluation function is calculated using the pixel block of the above, the corresponding pixel is detected based on the calculation result, and the correction unit is configured based on a value of the predetermined evaluation function.
Evaluate the certainty of the correspondence between the pixels of the first image and the corresponding pixels, and select the pixel of the distance relationship information image having the highest certainty from the third pixel block including the correction target pixel 13. The image processing apparatus according to claim 12, wherein the correction is performed by replacing the distance relation information about the correction target pixel with the distance relation information about the selected pixel. 19. The method according to claim 19, further comprising: photographing the predetermined object;
The image processing apparatus according to claim 1, further comprising an imaging unit that outputs a second image. 20. An image processing method for obtaining distance-related information relating to a distance to the target object based on first and second images of a predetermined target object, the image processing method comprising: A detection step of detecting a corresponding pixel of the second image corresponding to a pixel; a processing step of obtaining the distance relation information for a pixel of the first image based on the corresponding pixel; A selection step of selecting a correction target pixel to be subjected to the correction of the distance relation information from the pixels of the distance relation information image having a pixel value; and the distance relation information about the correction target pixel is the distance relation information image. A correction step based on the distance-related information for other pixels. 21. In the processing step, the first
21. The image processing method according to claim 20, wherein a parallax between a pixel of the image and a corresponding pixel is obtained as the distance relation information. 22. The image processing method according to claim 21, wherein in the processing step, a distance to the object is further obtained based on the parallax. 23. The image processing method according to claim 20, wherein in the processing step, a distance to the object is obtained as the distance relation information. 24. In the detecting step, the first
21. The image processing method according to claim 20, wherein a predetermined evaluation function is calculated using pixel values of pixels of the second image and the second image, and the corresponding pixel is detected based on a result of the calculation. 25. The image processing method according to claim 24, wherein, in the selecting step, the correction target pixel is selected based on a value of the predetermined evaluation function. 26. The image processing method according to claim 20, wherein, in the selecting step, the correction target pixel is selected based on the distance relation information. 27. The method according to claim 27, wherein in the selecting step, the first
21. The image processing method according to claim 20, wherein the correction target pixel is selected based on a pixel value of a pixel of the image. 28. The correction step, wherein the distance relation information about the correction target pixel is corrected based on the distance relation information about a pixel located near the correction target pixel. 20. The image processing method according to 20. 29. The method according to claim 29, wherein in the correcting step, the distance-related information about the pixel to be corrected is replaced with the distance-related information about another pixel of the distance-related information image. Item 21. The image processing method according to Item 20, 30. In the correcting step, the distance relation information on the correction target pixel is corrected based on the distance relation information on another pixel in a predetermined area including the correction target pixel. Claim 2
0. The image processing method according to 0. 31. In the correcting step, the first
Based on the certainty of the correspondence between the pixels of the image and the corresponding pixels, a pixel in the predetermined area is selected, and based on the distance relationship information of the selected pixel, The image processing method according to claim 30, wherein the distance relation information is corrected. 32. In the correcting step, among the pixels in the predetermined area, a pixel having the highest certainty of the correspondence between the pixel of the first image and its corresponding pixel is selected. The image processing method according to claim 31. 33. The image processing method according to claim 31, wherein in the correction step, a part of the pixels in the predetermined area is selected. 34. The image processing method according to claim 33, wherein in the correction step, a pixel located at an outermost periphery of the predetermined area is selected. 35. In the detecting step, the first
And a predetermined evaluation function is calculated using the pixel values of the pixels of the second image, and the corresponding pixel is detected based on the calculation result. In the correction step, based on the value of the predetermined evaluation function 32. The method according to claim 31, further comprising: evaluating the certainty of the correspondence between the pixel of the first image and the corresponding pixel.
The image processing method according to 1. 36. In the detecting step, the first
A predetermined evaluation function is calculated for the pixel value of the pixel of the image and the pixel value of each pixel group of the second image that may correspond to the pixel, and based on the calculation result,
Detecting the corresponding pixel from the pixel group of the second image, and, in the correcting step, the correspondence between the pixel of the first image and the corresponding pixel based on the value of the predetermined evaluation function 32. The method according to claim 31, further comprising: evaluating a certainty of the information.
The image processing method according to 1. 37. The method according to claim 37, wherein:
By using a first pixel block including pixels of the image of the second image and a second pixel block including pixels of each pixel group of the second image which may correspond to the pixel, a predetermined evaluation function is calculated. Calculating, based on the result of the calculation, detecting the corresponding pixel, and in the correcting step, based on the value of the predetermined evaluation function, ensuring the correspondence between the pixel of the first image and the corresponding pixel. The third pixel including the pixel to be corrected.
From the pixel block, the pixel having the highest reliability is selected in the distance-related information image, and the distance-related information about the pixel to be corrected is replaced with the distance-related information about the selected pixel. The image processing method according to claim 31, wherein the image processing is performed. 38. Control information for causing an information processing apparatus to perform a process of obtaining distance-related information relating to a distance to an object based on first and second images of a predetermined object. A providing step of detecting a corresponding pixel of the second image, which corresponds to a pixel of the first image, based on the corresponding pixel, for a pixel of the first image, A processing step of obtaining the distance relation information; a selection step of selecting a correction target pixel for which the distance relation information is to be corrected from the pixels of the distance relation information image having the distance relation information as a pixel value; Correcting the distance-related information for the target pixel based on the distance-related information for other pixels of the distance-related information image. Providing medium characterized. 39. A providing medium for providing distance-related information obtained by obtaining distance-related information relating to a distance to an object based on first and second images of a predetermined object. Detecting a corresponding pixel of the second image corresponding to a pixel of the first image, obtaining the distance relationship information for a pixel of the first image based on the corresponding pixel, From the pixels of the distance relationship information image whose pixel value is the distance relationship information, a correction target pixel for which the distance relationship information is to be corrected is selected, and the distance relationship information for the correction target pixel is represented by the distance relationship information. A providing medium for providing the distance relationship information obtained by performing correction based on the distance relationship information for another pixel of an image.