JP2000213933A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To search a corresponding point with a high speed without enlarging a scale of a hardware. SOLUTION: A corresponding point searching means 6 sets a reference block around one reference picture element under searcing, and an observing block around an observing picture element, when a corresponding picture element is searched, and computes a correlation coefficient between the reference block and the observing block. In this case, a correlation coefficient computing means 10 of the correponding point searching means 6 stops immediately the computing of the correlation coefficient with respect to the reference block when the correlation coefficient under computing exceeds a reference correlation coefficient. On the other hand, when the correlation coefficient under computing is smaller than the reference correlation coefficient, the correlation with respect to the same reference block is continued.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention provides a function for obtaining distance data to an object by a so-called stereo image measuring method based on input images from a plurality of image pickup means installed at different positions on three-dimensional coordinates. The present invention relates to an image processing apparatus having the same.

[0002]

2. Description of the Related Art FIG. 3 is a plan view showing the structure of a conventional image processing apparatus. In this figure, the imaging of the object 1 existing at an arbitrary position on the three-dimensional coordinates is performed by two imaging units 2L.
(Left side) and 2R (right side). The image processing circuit 3
Calculates distance data z on the object 1. The imaging means 2L and 2R are installed at a distance B from each other, and their installation heights are the same.

FIG. 4 is a block diagram showing a configuration of the image processing circuit 3 in FIG. As shown in this figure, the image processing circuit 3 is composed of two image input means 4L and 4R, two feature amount extracting means 5L and 5R, a corresponding point searching means 6, and a distance data calculating means 7.

FIG. 5 is an explanatory diagram for explaining a stereo image measuring method. The operation of FIG. 4 will be described with reference to FIG. In FIG. 5, when the image of the imaging means 2L is S1 and the image of the imaging means 2R is S2, three-dimensional coordinates are set in a space including the object 1 and the images S1 and S2. Note that coordinates in this coordinate system are represented by x, y, and z. [X1, Y1] and [X2, Y2] are used as coordinates indicating positions on the images S1 and S2. In this example, the image S1 of the images S1 and S2 is used as a reference image, the optical axis of the image S1 is located on the z axis, that is, at the position of x = 0, and the optical axis of the image S2 is located at the position of x = B. are doing.
The optical axes of the imaging means 2L and 2R are parallel to each other in the xz plane.

[0005] Image input means 4L and 4R input information of the images S1 and S2 from the image pickup means 2L and 2R, respectively, and output the information to the feature quantity extracting means 5L and 5R. The feature amount extracting means 5L and 5R extract feature amounts indicating the presence of the object 1 from the input image information. As the feature amount, pixel intensity (for example, luminance or shading) is usually used.
As disclosed in JP-A-336669, a value obtained after performing an edge extraction process using a Sobel filter (first derivative) or a Laplacian filter (second derivative) may be used.

The corresponding point searching means 6 includes a feature amount extracting means 5L
, 5R are extracted, and the image S1 is input.
The pixel of the image S2 corresponding to the pixel of the image S1 is searched (this search is performed for all the pixels of the image S1). For example, a point P [x, y, z] of the object 1 is changed to a point P1 on the image S1.
When the image is projected on [X1, Y1], the corresponding point searching means 6 generates a point P2 [X2, X2] corresponding to the point P1 [X1, Y1].
Y2] is searched for from the pixels of the image S2 (in this specification, the pixel at the point P1 is referred to as a target pixel, and the pixel at the point P2 is referred to as a corresponding pixel). In this case, in the image S1, the epipolar line is the scanning line Sc1 where the point P1 [X1, Y1] exists.
Matches. Accordingly, the epipolar line in the image S2 also coincides with the scanning line Sc2 having the same height as the scanning line Sc1, and the point P2 [X2, Y1] corresponding to the point P1 [X1, Y1].
2] must exist at any position on the scanning line Sc2. The search performed by the corresponding point searching means 6 means that the search area is set on the scanning line Sc2 in the image S2 and the point P2 corresponding to a certain point P1 [X1, Y1] on the scanning line Sc1.
[X2, Y2] is obtained from the scanning line Sc2 in the image S2. However, it is difficult to accurately perform the above-described corresponding point search in the stereo image measurement method, and various techniques have been proposed as techniques for this corresponding point search. For example, Japanese Patent Application Laid-Open No. 9-33249 discloses a method in which pixel intensities of both left and right images are calculated to calculate a correlation function representing a correlation between the two pixels, and a corresponding point search is performed according to the value of the correlation function. Have been.

[0007] Corresponding point searching means 6 generates a point P1 [X1, Y1].
After searching for a point P2 [X2, Y2] corresponding to
Parallax data between the two points is obtained. The disparity data is a difference between the positions of the two points in the images S1 and S2, and is represented by D (X2-X1, Y2-Y1). In the above case, since Y1 = Y2, this parallax data is D
(X2−X1,0).

The distance data calculating means 7 calculates distance data z between the object 1 using the parallax data D obtained by the corresponding point searching means 6. That is, assuming that the focal lengths of the imaging means 2L and 2R are f and the distance between the respective optical axes is B, the distance z is represented by f, B and parallax only in the X direction as shown in the following equation (1). Data (X2-X1). The distance data calculating means 7 calculates z for each pixel according to the equation (1).

Z = B · f / (X 2 −X 1) (1) The distance data calculating means 7 outputs the distance data z thus calculated to an external device not shown. The external device is, for example, a surveillance camera device, an industrial robot having a visual function, an NC machine, or the like, and includes devices used in various fields.

The above-described calculation of the correlation coefficient is performed, for example, as follows. That is, when any one of the two images is used as a reference image and a corresponding pixel of the other image corresponding to a target pixel of the reference image is searched, a search area is set in the other image, and the search area is set. One of the pixels in the area is set as a reference pixel.

Further, a reference block, which is a rectangular small area of dX × dY centered on the reference pixel, and a target block, which is a rectangular small area of dX × dY centered on the pixel of interest, are respectively set. Using either (2) or (3), the square of the difference between the feature values of the pixels at the same position in the two blocks, or the product sum operation in the block of the absolute value difference, and the phase obtained as a result The relation number E (X, Y) is defined as a correlation coefficient between the target pixel and the reference pixel (also referred to as a correlation coefficient between the target block and the reference block).

Equations (2) and (3) are based on the point P1 [X1, Y
1] as a target pixel, and a point P2 [X2, Y2] as a reference pixel, where I1 [XA, YB] and I2 [XC,
YD] is the point P1 [XA, YB] and the point P2 [XC,
YD].

[0013]

(Equation 1) After calculating the correlation coefficient between a certain reference pixel (in this case, point P2) and the target pixel (in this case, point P1) as described above, this reference pixel is shifted by one to create a new reference block. Then, the correlation coefficient is similarly obtained between the new reference block and the target block. In this way, the reference pixels are sequentially moved in the search area, and the correlation coefficient between all the pixels in the search area and a certain target pixel is obtained (the number of correlations is equal to the number of set reference pixels). The correlation coefficient having the smallest value is selected from among them.
The reference pixel having the smallest correlation coefficient is determined as the corresponding pixel for the point P1 [X1, Y1]. If the reference pixel having the minimum correlation coefficient is point P2 [X2, Y2], point P2 [X2, Y2]
] Is determined as the corresponding pixel.

Here, a method of obtaining the correlation coefficient E will be described more specifically with reference to FIG. As shown in FIG. 6, a target pixel on the reference image is P, a reference pixel on the reference image is S, and a 3 × 3 target block and a reference block are set around the target pixel. Is as shown in FIG. 6, the correlation coefficient E is represented by one of the following equations (4) and (5). Equation (4) corresponds to equation (2), and equation (5) corresponds to equation (3).
| X | is the absolute value of x, and Si and Pi are the feature values of each pixel. ^{E = (S-P) 2} + (S1-P1) 2 + (S2-P2) 2 + (S3-P3) 2 + (S4-P4) 2 + (S5-P5) 2 + (S6-P6) 2 + (S7−P7) ² + (S8−P8) ² (4) E = | (S−P) | + | (S1−P1) | + | (S2−P2) | + | (S3−P3) | + | (S4−P4) + | (S5−P5) | + | (S6−P6) | + | (S7−P7) | + | (S8−P8) | (5) Fix the target pixel P Then, S is searched for all the pixels in the search range on the reference image, the correlation coefficient E for each reference block is obtained, and the reference pixel when the value of E is minimum is set to the corresponding pixel of the target pixel P. To decide. Further, the above calculation is performed for P on all the pixels on the reference image, and a corresponding pixel for each target pixel is determined.

It should be noted that the correlation coefficient E (X, Y) expressed by the above equation (2) or (3) indicates that the search area and the reference area are both X
The general case of the secondary plane area on the −Y plane has been described, but as described above, in the case of FIG. 5, since the search area is limited on the scanning line Sc2, the Y direction component is What is necessary is to ignore it and consider only the X-direction component.

[0016]

However, in the above-described conventional corresponding point search, the operation of equation (2) or (3) must be performed for all reference pixels in the search area. The amount is very large. As a result, it takes a considerable amount of time to execute this calculation, and a high-speed corresponding point search cannot be performed.

As a technique for performing a high-speed corresponding point search, for example, as disclosed in Japanese Patent Application Laid-Open No. 7-103734, a hierarchical image, which is a lower resolution image, is created and obtained on this image. Techniques for shortening the calculation time by narrowing a search range with reference to disparity data have been conventionally proposed. According to this conventional technique, although the calculation time is reduced to some extent, another hardware problem arises in that a large-capacity memory corresponding to the image of each layer must be provided.

The present invention has been made in view of the above circumstances, and it is an object of the present invention to provide an image processing apparatus capable of performing a corresponding point search at high speed without increasing the scale of hardware.

[0019]

Means for Solving the Problems As means for solving the above problems, the invention according to claim 1 is based on a preset 3
Image input means for inputting image information on an object from a plurality of image pickup means installed so as to have a predetermined positional relationship on dimensional coordinates, and extracting each feature amount in a plurality of images obtained from the image input means A feature amount extracting unit, and using any one of the plurality of images as a reference image, searching for a corresponding pixel of another image corresponding to a target pixel of the reference image based on the feature amount, Corresponding point searching means for obtaining disparity data, and distance data calculating means for calculating distance data for the object based on the disparity data obtained by the corresponding point searching means, wherein the corresponding point searching means comprises: When searching for a corresponding pixel, a search area is set in the other image, and a reference block centered on one reference pixel in the search area and a target block centered on the target pixel are set. Based on calculating the correlation coefficient between the reference block and the block of interest by sequentially integrating the rated value based on the difference between the feature values of the pixels at the corresponding positions in the two blocks, In the image processing apparatus for searching for a corresponding pixel in the search area, the corresponding point searching means calculates a correlation coefficient between the reference block and the target block, The number correlation is compared with a preset reference correlation coefficient, and when the correlation coefficient during the integration becomes larger than the reference correlation coefficient, the calculation of the correlation coefficient with the reference block is immediately performed. Is stopped and the operation proceeds to the next reference block to calculate the correlation coefficient in the same manner. On the other hand, if the correlation coefficient during integration is smaller than the reference correlation coefficient, the correlation between the reference block and the same reference block is calculated. Calculation of relation number continues Further, if the correlation coefficient obtained as a result of performing the calculation of the correlation coefficient between the same reference block to the end without being stopped halfway is smaller than the reference correlation coefficient, When the correlation coefficient is set as a new reference correlation coefficient, and a transition is made to the next reference block to calculate a new correlation coefficient, the correlation coefficient in the course of integration is sequentially changed to the newly set reference phase. The number of relations is compared.

According to a second aspect of the present invention, in the first aspect of the present invention, the corresponding point searching means sets a position of a corresponding pixel corresponding to a target pixel in a previous search as an initial search position in a current search. Is characterized by the following.

[0021]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the drawings. However, the same components as those in FIG. 4 are denoted by the same reference numerals, and redundant description will be omitted. FIG. 1 differs from FIG. 4 in that the corresponding point searching means 6 includes a correlation coefficient calculating means 10 and a corresponding point storing means 11.

Next, the operation of the present embodiment configured as described above will be described with reference to the flowchart of FIG.
First, the correlation coefficient calculating means 10 searches the corresponding point P2 on the image S2 side corresponding to the position P1 of a certain pixel of interest on the image S1 side.
(i-1) is set as an initial search position S (i) (step 1). Here, i is an integer such as i = 1, 2, 3,. In this way, the corresponding point obtained by the previous search is used as the initial search position because the disparity data between adjacent pixels usually has a high correlation. This is because it is considered that there is a high possibility that the corresponding pixel exists.

Next, the correlation coefficient calculating means 10 calculates a phase relationship between the reference block set around the reference pixel at the initial search position S (i) and the target block set around the target pixel. Equation (2) or the number
Calculation is performed using (3), and this correlation coefficient is used as a reference correlation coefficient Emi.
Set as n (step 2).

Here, the reference block set with the position S (i) as the center is defined by the coordinates of the pixel at the position S (i) [X,
Y], besides the pixel of [X, Y], for example, [X + 1, Y], [X-1, Y], [X + 2,
Y] and [X-2, Y]. The same applies to the block of interest set around the pixel of interest. Then, the correlation coefficient calculating means 10 calculates the equation
According to (2) or (3), the square of the feature value difference value of the pixel at the same position of these two blocks and the integration of the absolute value difference within the block are performed.

After the reference correlation coefficient Emin is set as described above, the correlation coefficient calculation means 10 updates the reference pixel (step 3). By this update, for example, step 1
The position moved to the right by, for example, one pixel on the scanning line Sc2 from the initial search position set in the above is set as the center position of the new reference block. Then, the correlation coefficient calculating means 10
Calculates the integrated value of the correlation coefficient E using equation (2) or (3) at this new reference pixel (step 4). The calculation of the integrated value E in step 4 is performed in the same manner as the calculation of the initial correlation coefficient Emin in step 2.

The correlation coefficient calculating means 10 compares the magnitude relation between the integrated value E and the reference correlation coefficient Emin each time the square of the difference value in pixel units or the integration of the absolute value of the difference is performed. (Step 5). In the example of FIG. 6, the first comparison between E and Emin is performed at the stage of E = (SP) 2, and E = (SP) 2+
The second comparison is performed in the integration stage of (S1-P1) 2, and the third comparison is performed in the integration stage of E = (SP) 2+ (S1-P1) 2+ (S2-P2) 2. As described above, the correlation coefficient E in the course of integration is sequentially compared with the reference correlation coefficient Emin.

If the integrated value E is smaller than the reference correlation coefficient Emin, it is determined whether or not the integration in the block has been completed (step 6), and if not, the integration is continued (step 6). Step 4). If it is determined in step 6 that the integration has been completed, the integrated value E of the reference pixel is set as a new reference correlation coefficient Emin (step 7).

Next, the correlation coefficient calculating means 10 determines whether or not the search in the search area has been completed, that is, whether or not the integrated value E has been calculated for all the reference pixels on the scanning line Sc2. (Step 8). If the search has not been completed, the process returns to step 3 to update the reference pixel. If it is determined in step 5 that the reference correlation coefficient Emin is greater than or equal to the integrated value E, the integration of the integrated value E in the block is immediately stopped, and the process proceeds to step 8, where the search area is determined. A determination is made as to whether the search at has been completed. This means that the fact that the value of the integrated value E is equal to or larger than the reference correlation coefficient Emin means that the reference pixel is no longer likely to be the corresponding pixel, so that the integration in the block is continued further. This is because it is obviously useless.

After determining that the search has been completed in step 8, the correlation coefficient calculation means 10 selects the smallest one of the correlation coefficient values calculated so far and selects the point P2 at that time.
The position S (i) of [X2, Y2] is changed to the position P '(i) of the corresponding pixel.
To decide. Then, the position P '(i) of the corresponding pixel is stored in the corresponding point storage means 11 (step 9). The stored position P ′ (i) is set as the initial search position in the next target pixel. When the integrated value E that is smaller than the reference correlation coefficient Emin is calculated in step 7, the minimum value of the correlation coefficient is always the integrated value E.
Is set as a new reference correlation coefficient Emin, so that the reference correlation coefficient Emin is obtained.

The corresponding point searching means 6 determines the point P1
After searching for the point P2 [X2, Y2] corresponding to [X1, Y1], the disparity data D (X2-X1, 0) between the two points is calculated.
(Step 10). The corresponding point search means 6 performs the above-described steps 1 to 10 with all the pixels of the image S1 as target pixels.

As described above, the corresponding point searching means 6 of the present embodiment calculates the integrated value E in a certain block while the integrated value E becomes equal to or larger than the reference correlation coefficient Emin. Immediately stops the operation in the reference block at that point and shifts to the next reference pixel. conventionally,
Regardless of the value of the integrated value E, after the integration is performed for all the pixels in the reference block, the process is shifted to the next reference pixel. By introducing a technique for judging the magnitude relationship with the correlation coefficient Emin, useless computation can be eliminated and the computation time can be reduced.

In this case, the reference correlation coefficient Emin is
It is preferable that the value is as small as possible, that is, the value of the correlation coefficient relating to the corresponding point, since the wasteful calculation to be eliminated is increased. However, in the present embodiment, the initial search position is set as the corresponding point in the previous search. The reference correlation coefficient Emi is likely to be the smallest value from the beginning of the calculation.
Since n can be used, the configuration is more appropriate for reducing the operation time. The configuration of the present embodiment does not require a large-capacity memory or the like, and does not involve an increase in hardware scale.

In the above embodiment, the block set at the position S (i) and the pixel of interest is set as the center.
When the coordinates of the pixel at the position S (i) and the pixel of interest are represented by [X, Y], for example, in addition to the pixel at [X, Y], for example,
[X + 1, Y], [X-1, Y], [X + 2, Y],
Although it has been described that the area is formed by the pixels of [X-2, Y], the range can be freely set in consideration of various conditions, and the order in which the integration is performed in the block. Can also be set freely.

[0034]

As described above, according to the present invention, in the process of calculating the correlation coefficient between the reference block centered on the reference pixel and the target block centered on the target pixel,
The halfway correlation coefficient is compared with a preset reference correlation coefficient. If the halfway calculation correlation coefficient is larger than the reference correlation coefficient, the calculation with the reference block is immediately stopped. At the same time, the process proceeds to the next reference pixel and calculates the correlation coefficient in the same manner. On the other hand, if the correlation coefficient during the calculation is smaller than the reference correlation coefficient, the calculation is continued and the reference block and the If the calculation of the correlation coefficient between is performed to the end without interruption, and the correlation coefficient obtained as a result of the calculation is smaller than the reference correlation coefficient, the correlation After the number is set as a new reference correlation coefficient, and after shifting to the next reference block, the correlation coefficient in the middle of calculation is compared with this newly set reference correlation coefficient. Without expanding
It is possible to perform a corresponding point search at high speed.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of a main part of an embodiment of the present invention.

FIG. 2 is a flowchart for explaining the operation of FIG. 1;

FIG. 3 is a plan view showing a configuration of a conventional image processing apparatus.

FIG. 4 is a block diagram showing a configuration of an image processing circuit 3 in FIG. 3;

FIG. 5 is a diagram illustrating a stereo image measurement method.

FIG. 6 is a diagram illustrating a specific method for obtaining a correlation coefficient.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Object 1 2L, 2R imaging means 3 Image processing circuit 4L, 4R Image input means 5L, 5R Feature extraction means 6 Corresponding point searching means 7 Distance data calculating means 8 Correspondence discriminating means 9 Corresponding point correcting means 10 Correlation coefficient Calculation means 11 Corresponding point storage means

 ──────────────────────────────────────────────────続 き Continuing on the front page F term (reference) 2F065 AA04 AA53 BB05 BB24 DD06 FF01 FF05 JJ03 JJ19 JJ26 QQ21 QQ24 QQ32 QQ36 QQ39 QQ41 RR09 UU05 2F112 AC03 AC06 BA05 CA08 CA12 DA02 FA03 FA21 FA36 FA39 DC043DB

Claims (2)

[Claims] 1. An image input means for inputting image information relating to an object from a plurality of image pickup means installed so as to have a predetermined positional relationship on predetermined three-dimensional coordinates, and obtained from the image input means. A feature amount extracting unit that extracts each feature amount in a plurality of images; and any one of the plurality of images is used as a reference image, and a corresponding pixel of another image corresponding to a target pixel of the reference image is used as the feature amount. A corresponding point searching means for searching for parallax data between the two pixels based on the distance data calculating means for calculating distance data on the object based on the parallax data obtained by the corresponding point searching means. And the corresponding point searching means, when searching for the corresponding pixel, sets a search area in the other image,
A reference block centered on one reference pixel in the search area and a target block centered on the target pixel are set, and a rated value is set based on a difference in feature amount between pixels at corresponding positions in the two blocks. In the image processing apparatus, based on calculating the correlation coefficient between the reference block and the block of interest by sequentially integrating the corresponding blocks, a corresponding pixel is searched in the search area. In the process of calculating the correlation coefficient between the reference block and the block of interest, the successive correlation coefficients in the course of integration are compared with a preset reference correlation coefficient, and the correlation coefficient in the course of integration is used as the reference phase. When the value becomes larger than the number of relations, the calculation of the correlation coefficient with the reference block is immediately stopped, and the operation proceeds to the next reference block to calculate the correlation coefficient in the same manner. On the way When the correlation coefficient is smaller than the reference correlation coefficient, the calculation of the correlation coefficient with the same reference block is continued, and the calculation of the correlation coefficient with the same reference block is stopped halfway. If the correlation coefficient obtained as a result of performing the calculation to the end without being performed is smaller than the reference correlation coefficient, the correlation coefficient is set as a new reference correlation coefficient, and the process proceeds to the next reference block. When calculating a new correlation coefficient, the correlation coefficient in the course of integration is sequentially compared with the newly set reference correlation coefficient. 2. The apparatus according to claim 1, wherein said corresponding point searching means sets a position of a corresponding pixel corresponding to a target pixel in a previous search as an initial search position in a current search. Image processing device