JP2958462B1 - Method and apparatus for correcting luminance of stereo image - Google Patents

Abstract

Kind Code: A1 Abstract: In a stereo image, the brightness of the image is corrected based on the correspondence between the feature amounts of the image. A stereo image is input, and edges of each image are detected (S1). Segments are extracted (S2), and the feature amount of each segment is determined (S3). The correspondence of the segments is obtained between the stereo images (S4), and the occluding outline is removed from the corresponding portion (S5). A luminance correction equation is obtained from the correspondence of luminance obtained by this processing (S6), and the luminance of the stereo image is corrected (S7). Further, the above-described processing is repeated until the correspondence of luminance between the stereo images matches, and the luminance of the stereo image is corrected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a stereo image correcting method and apparatus useful for three-dimensional geometric shape restoration. For example, it can be used to give vision to industrial vision systems and robots.

[0002]

2. Description of the Related Art A method of restoring a three-dimensional geometric shape existing in a scene from a stereo image is based on a known relative positional relationship of a camera at the time of photographing. The three-dimensional information is restored according to the principle described above. As a method of performing the stereo correspondence search and generating three-dimensional information, for example, the following method is known.

(1) First, a part (window) of one image is taken out, and a window having the same luminance distribution is found out from the other, that is, the correlation between the windows is calculated, and the value is high. This is a method of finding a thing and finding correspondence between images and restoring a three-dimensional shape, and is generally called a correlation method. With this method, three-dimensional surface information can be restored. There are roughly two types of correlation methods, a method using a normalized evaluation expression and a method using a difference evaluation expression. As shown in FIG.
The former method is effective for an area where a luminance change is remarkable in a correlation window such as a texture and the like, and since it is normalized, there is no need for absolute luminance coincidence. However, there is a possibility that an erroneous correspondence may occur in a portion where the luminance changes smoothly, such as a shading region. To calculate the correct disparity in the shading area, the latter method is effective,
To do so, the luminance correspondence between images must be correct. That is, in order to obtain an accurate correspondence based on the luminance of all the regions of the image, it is necessary that the luminance adjustment between the images is correctly performed.

(2) Secondly, there is a structural analysis method for detecting a high-order feature amount of an image and obtaining a correspondence between the features.
To detect a portion where the luminance change is large, that is, an edge which is a boundary line of a region, divide it into a segment which is a meaningful line segment by a local shape, and perform a stereo correspondence search using this segment as a unit. Therefore, there is a segment-based stereo method for restoring a three-dimensional shape from the correspondence. With this method, the three-dimensional information of the contour of the object can be restored. The edge portion is a boundary line between regions having different luminances, and edges are detected at almost the same location even when absolute luminances do not match between images. Therefore, the segment-based stereo method using this feature amount has an advantage that it is robust against luminance change. However, this method has a problem that the three-dimensional shape is restored only in the feature amount (edge portion or the like) obtained first, and the plane information cannot be restored.

As described above, the correlation method is effective for restoring three-dimensional surface information, but in order to obtain accurate information, it is essential to adjust the luminance between images. Conventionally, brightness adjustment between images has been performed by performing fine adjustment of the camera at the time of image shooting, but sometimes it can not be adjusted completely,
For images that have already been taken, it was necessary to manually correct the brightness between the images.

[0006]

In general, when the brightness adjustment between stereo images is not accurately performed, the correlation method is used.
Even when the dimensional geometry is restored, accurate information cannot be obtained.

SUMMARY OF THE INVENTION In view of the above, the present invention has been made in view of the above-described problems in the case where the luminance adjustment cannot be performed at the time of photographing an image and the problem of correcting the luminance of a photographed image. Thus, a method is provided in which the correspondence between images of features that are relatively robust to variations in brightness is determined by, for example, a segment-based stereo method, and the brightness of the stereo image is corrected using the brightness information of the corresponding portion. It is intended to be.

[0008]

According to the present invention, there is provided a method for correcting luminance of a stereo image, comprising the steps of detecting edges in a stereo image, dividing the image into segments by a local shape, and obtaining a feature amount of the segment. Based on the amount, obtaining a correspondence relationship between the stereo images by a stereo correspondence search process using a segment as a correspondence unit, removing the occluded contour portion from the corresponding portion in the stereo image, The method has a step of correcting the brightness of an image using the brightness information of a certain point sequence. Further, the brightness correction apparatus for a stereo image according to the present invention includes a means for detecting an edge in a stereo image, dividing the segment into segments by a local shape, and calculating a feature amount of the segment, and mapping the segment based on the feature amount. A means for obtaining a correspondence between stereo images by a stereo correspondence search process as a unit, a means for removing an occluding contour portion from a corresponding part in a stereo image, and a method for obtaining luminance information of a point sequence having a correspondence in a stereo image. And means for correcting the brightness of the image by using the function.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A stereo image luminance correcting method and apparatus according to the present invention detects edges in a stereo image, divides the segment into segments according to a local shape, obtains a feature amount of the segment, and calculates the feature amount of the segment. The correspondence between stereo images is obtained by a stereo correspondence search process using a segment as a correspondence unit based on the correspondence information. In the stereo image, an occluding contour portion is removed from a corresponding portion, and luminance information of a corresponding point sequence in the stereo image is obtained. Is used to correct the brightness of the image. FIG. 2 is a process chart showing the process. Hereinafter, the process of correcting the luminance of the stereo image will be described step by step with reference to FIGS. 3 to 18.

(1) Edge Detection (S1) First, an edge is detected at a boundary dividing a region, that is, a portion where a change in luminance is large. For each image of the stereo image (FIG. 3) composed of a plurality of images, for example, FIG.
By calculating the product sum of the Sobel operator having a 3 × 3 window at each point as shown in, eliminating non-maximal values, and performing extended image processing based on the result of comparison with the threshold value,
An edge portion as shown in FIG. 5 is extracted. The edge referred to here is a group of pixel points, that is, a point sequence. The edge part detected here is a part where the luminance change is large, and is a boundary line that divides the region, and even if there is no absolute luminance match between the stereo images, the edge part is located at almost the same place by the above method. Edges are detected. Therefore, this feature amount has low dependency on the luminance change and is robust.

(2) Segment extraction (S2) The detected edge is divided into segments at the characteristic points shown in FIG. 6: branch points, refraction points, inflection points, and transition points according to the local shape. A branch point is a point where three or more line segments branch from that point, and a refraction point is a point connecting straight lines with different angles. The inflection point is a point connecting the curves with different curvatures, and the transition point is a point connecting the straight line and the curve. A line segment formed by a group of points between the feature points is called a segment.

(3) Acquisition of segment information (S3) Each segment is composed of a sequence of points, which is a group of points, and calculates a feature amount of each segment such as the shape, length, direction, and luminance of the segment. As shown in FIG. 7, the shape of the segment, which is the feature quantity, is classified into three types: a straight line, a concave curve, and a convex curve. The determination of the unevenness of the curve is a concave curve if the area to which the segment belongs looks like a bite into the area when viewed to the right with respect to the direction of the segment,
When the area is wrapped, a convex curve is used. The length of a segment is defined as the number of points in a point sequence forming the segment. The direction of the segment means the direction on the image plane from the start point to the end point of the segment. In addition, a method of obtaining luminance information at each point constituting a segment will be described with reference to FIG. First, a point on the segment which is a boundary line that separates the region 1 and the region 2 is a point that divides the region and belongs to both the region 1 and the region 2. Therefore, both information of the area 1 and information of the area 2 are given as the luminance information of this point. First, a normal line perpendicular to the tangent direction at the point of interest and facing the direction of the region 1 is obtained as information on the region 1. At a nearby point on the normal direction, a differential value is calculated. Among the neighboring points, a point having the smallest differential value is obtained, and the luminance value and the differential value are determined in the attention point area 1.
Brightness information. As described above, the luminance information gives not the luminance information of the edge portion itself but information of a point reflecting the luminance information in the area to which the point belongs. The luminance information on the area 2 of the point of interest can be obtained in the same manner. This processing is performed for each edge point constituting the segment.

(4) Segment Correspondence (S4) In a stereo image, correspondence is determined using the segments determined in steps S2 and S3 as corresponding units. This time, as an example, the correspondence search method by the segment-based stereo method is shown. The processing procedure is shown in FIG. 9 and will be described in this order.

First, a set of segments for which a correspondence is established between stereo images and a correspondence candidate are found. Between the stereo images, the other point corresponding to a certain point in one image is a straight line where the plane intersecting the plane of the focal point of the two cameras and the observed point and the image plane as shown in FIG. Present on. This is called an epipolar line, and the search for the corresponding point need not be performed on the entire image, but only on the epipolar line. This is called an epipolar condition. Therefore, this epipolar condition is used when obtaining the correspondence between segments. A search is made for a set of segments that satisfy this condition and that have similar segment shapes, directions, luminances, and the like, which are the feature amounts of the segment obtained in step S3, and that satisfy all the conditions. Make it a candidate. Then, the local similarity is obtained in consideration of the brightness and length of the correspondence candidate.

Next, the connectivity of the obtained correspondence candidates is examined. In each image, it is determined whether the constituent segments are smoothly connected or form a similar shape. (A) Difference in distance between segments (b) Difference in luminance between segments (c) Evaluation is performed using the three elements of the angle difference to be made, and processing is performed to determine that correspondence candidates satisfying all of these conditions are connected.

Based on the obtained connectivity, a process of obtaining a set of segments similar to the boundary line of one image from another image is performed. That is, a process of finding a boundary line having a similar shape is performed. Here, a union of local similarities of connected segment groups (hereinafter, referred to as paths) is obtained, and the path similarity is given to each corresponding candidate segment forming the path as the path similarity. This is not a local similarity but a process considering the similarity of the shape of the boundary line, and means a global similarity.

Then, using the obtained similarity as an evaluation criterion,
From the selected correspondence candidates, the correspondences are not overlapped, that is, a correct correspondence in which the correspondence relationship is one-to-one is obtained. In other words, if one segment in one image has more than one candidate in the other image,
Sort the previously obtained similarities by size and leave the one with the highest similarity as the correct correspondence, and the other ones with the highest similarity do not overlap the corresponding segment already obtained and the corresponding section That is, a process of leaving a thing. The segment obtained by this processing is the thick line portion in FIG.

(5) Occlusion contour removal (S5) However, the correspondence candidate obtained in step S4 is a contour of a curved surface as shown in FIG. 12, that is, an apparent contour whose appearance is different depending on the viewing direction. Is also included. As shown in the example of FIG. 12, the outlines of the sphere viewed from the viewpoint 1 and the viewpoint 2 actually appear at different positions.
For example, when the movement between the viewpoints is small, the positional deviation hardly causes a problem. Therefore, in the correspondence evaluation described in step S4, the correspondence is left as negligible.
This is because the evaluation of the correspondence is performed based on the connectivity of the segments as described above. However, in the luminance correction, the corresponding portion of the shielding contour is not strictly a true correspondence, and thus the shielding contour should not be used. The correction needs to use only the correspondence of the fixed edge segments that are true contours.

Then, a process for removing the shielding contour is performed next. Generally, since the luminance change in the shielding contour portion is large,
The differential value of the luminance information at each point of the point sequence forming the segment increases. By utilizing this property, the differential value of the luminance information of the corresponding point is compared with a threshold value, thereby removing the occluded contour line from the corresponding segment. Further, in the case of a multi-view stereo image having three or more eyes, as shown in FIG. 13, when the corresponding points obtained from viewpoints 1 and 2 are observed from other viewpoints 3,
If it is not a true outline, no outline can be detected at that part. In other words, an image from another viewpoint can be used as a verification image, and the shielding contour can be removed from the displacement of the detection position. FIG. 14 shows the result after the removal of the shielding outline.

(6) Calculation of brightness correction formula (S6) From the correspondence of the obtained segments, the relationship between the brightness values of the brightness information of the point sequence forming the segment is obtained, and the brightness relationship between the stereo images can be obtained. . Assuming that the reference image of the reference image is I1 and the image undergoing the luminance correction is I2, the luminance distribution of the corresponding points between the images (I1, I2) with respect to the luminance value is as shown in FIG. Find the correction formula. If the corresponding points are correct, the points are substantially distributed on a straight line, and a correction formula for the straight line is calculated.

I2 (0) = a I1 + ba a, b: coefficient Ii (n): image Ii after n-times correction

Ii (0) is the same as Ii. Image I
The least squares method is used to minimize the fitting error using the luminance value of the luminance information of the corresponding points of 1, 1
The coefficients a and b of the correction formula are calculated.

(7) Brightness Correction (S7) The brightness of the image I2 (0) is corrected based on the obtained correction formula to obtain a corrected image I2 (1). Modified image I2
The relational expression between (0) and the modified image I2 (1) is as follows.

I2 (0) = a I2 (1) + b The same processing is repeated from S1 using the reference image I1 and the image I2 (1) whose luminance has been corrected as an input stereo image, and the correction equation is almost I2 (n) = Repeat n times until I1.
The transition of the coefficients a and b of the correction formula of this iterative process is shown in FIG.
Where a = 1 and b = 0. As a result of the repetition processing, the luminance distribution of the images I1 and I2 (n) is shown in FIG.
7 (n = 6 in this example). FIG. 18 shows the corrected image obtained by this processing.

By the above-described processes (1) to (7), it is possible to convert a stereo image into a corrected image whose luminance correspondence is more accurate, using the image data itself.

[0026]

As described above, according to the present invention, the brightness of an image can be corrected using the brightness information of the corresponding portion, based on the correspondence relationship of the feature values relatively robust to brightness variations, There is an effect that the three-dimensional geometric shape can be correctly restored by a correlation method or the like, and the present invention can be used to give vision to an industrial vision system or a robot.

[Brief description of the drawings]

FIG. 1 is a diagram showing a correlation between a texture area and a shading area.

FIG. 2 is a process chart showing image correction according to the present invention.

FIG. 3 is a diagram showing an original image.

FIG. 4 is a diagram showing a Sobel operator.

FIG. 5 is a diagram showing edges.

FIG. 6 is a diagram showing feature points.

FIG. 7 is a diagram showing a shape classification of a segment;

FIG. 8 is a diagram showing luminance information possessed by each point of a point sequence forming a segment.

FIG. 9 is a diagram showing a processing procedure of the segment-based stereo method.

FIG. 10 is a diagram showing epipolar conditions.

FIG. 11 is a diagram showing corresponding segments obtained by the segment-based stereo method.

FIG. 12 is a diagram showing a shielding contour.

FIG. 13 is a diagram illustrating a method of removing a shielding contour.

FIG. 14 is a diagram showing a corresponding segment after removing a shielding contour portion;

FIG. 15 is a diagram illustrating a luminance distribution for an original image.

FIG. 16 is a diagram showing a transition of a coefficient of a luminance correction formula of a repetition process.

FIG. 17 is a diagram showing a luminance distribution after luminance modification.

FIG. 18 is a diagram showing an image whose luminance has been corrected.

──────────────────────────────────────────────────続 き Continued on the front page (58) Fields surveyed (Int. Cl. ⁶ , DB name) G01B 11/00-11/30 102 G01C 3/06 G06T 1/00-7/00

Claims (6)

(57) [Claims] 1. A stereo image, comprising: a step of detecting an edge in a stereo image, dividing the segment into segments by a local shape, and obtaining a feature amount of the segment; and performing a stereo correspondence search process using the segment as a corresponding unit based on the feature amount. A step of obtaining a correspondence between images; a step of removing an occluding contour portion from a corresponding part in a stereo image; and a step of correcting the luminance of the stereo image using the luminance information of a corresponding point sequence. And a brightness correction method for a stereo image, comprising: 2. The method according to claim 1, wherein the step of obtaining the characteristic amount of the segment includes the step of calculating the luminance information of the point sequence on the edge constituting the segment, not the luminance value of the point on the edge itself, but the luminance in the area to which the point belongs. 2. The method of correcting luminance of a stereo image according to claim 1, wherein the information is information of a point reflecting the information. 3. The luminance information of the edge point has luminance information of both regions to be divided, and a normal line perpendicular to a tangent direction at the edge point and directed to each region is obtained. 3. A method according to claim 1, wherein a differential value is calculated at a certain nearby point, a point having the smallest differential value is obtained, and the luminance value and the differential value are used as luminance information of each region of the point of interest. The brightness correction method of the stereo image described in the above. 4. The method according to claim 1, wherein the step of removing the shielding contour portion uses a property that a differential value of luminance information of an edge point on the shielding contour line is larger than a differential value at a fixed edge point on a true contour line. 4. The method of correcting luminance of a stereo image according to claim 1, further comprising removing a corresponding point sequence of a shielding contour portion. 5. The method according to claim 1, wherein the step of correcting the brightness of the image comprises correcting the brightness of the entire image in the stereo image so that the brightness values of the brightness information of the corresponding point sequence match. 5. The method of correcting luminance of a stereo image according to claim 4. 6. A means for detecting an edge in a stereo image and dividing the segment into segments according to a local shape to obtain a feature amount of the segment, and performing a stereo correspondence search process using the segment as a corresponding unit based on the feature amount. Means for determining the correspondence between the images; means for removing the obscured contour part from the corresponding part in the stereo image; means for correcting the luminance of the image using the luminance information of the corresponding point sequence in the stereo image And a brightness correction device for a stereo image comprising: