JPH0981755A - Plane estimating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To precisely measure distance data on a plane without being affected by bodies on a road and a floor by performing Hough transform for distance data measured partially on the observation plane, applying two abnormal straight lines running in the observation plane, estimating a plane position by using the straight lines, and acquiring undetermined distance data in the plane. SOLUTION: In photography phase A, two left and right images S1 and S2 are obtained by two image pickup devices. Then those left and right images S1 and S2 are made in correspondence phase B to correspond to each other S3. In the correspondence phase B, one reference image is divided into blocks of m×n pixels and the images are made to correspond to each other, block by block, S3. Parallax in block units which is obtained in the correspondence phase is passed to a trailing-stage plane estimation phase C. In plane estimation phase C, the blocks are grouped horizontally, straight lines running in the observation plane are applied, group by group, through the Hough transform, and the plane is estimated from the obtained straight line group.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a plane estimation method in stereo image processing, in which Hough transform is performed on distance data partially measured on an observation plane, and two lines passing through the observation plane are passed. This is a method of fitting the above straight line, estimating the plane position using this straight line, and obtaining undetermined distance data within the plane.

[0002]

2. Description of the Related Art FIG. 11 is a diagram for explaining the principle of stereo image measurement. In FIG. 11, x, y, and z are used as the coordinates representing the real space, and X and Y are used as the coordinates indicating the position on the image (camera imaging surface). However, X in order to distinguish the left camera 11L, the right camera 11R, as X _L, using a Y _L, coordinates indicating the position on the image plane of the right camera 11R as coordinates indicating the position on the image plane of the left camera 11L _R and Y _R are used. x
The axis is parallel to the X _L axis, the x axis is the X _R axis, the y axis is the Y _L axis, the y axis is the Y _R axis, and the z axis is two left and right cameras 11L,
It is assumed to be parallel to the optical axis of 11R. The origin of the real space coordinate system is set to the midpoint of the projection centers of the left and right cameras 11L and 11R, and the distance between the projection centers is called the baseline length, which is 2a.
Will be represented by Also, the distance (focal length) between the projection center and the image plane is represented by f.

Now, a point p in the real space is a point P on the left screen.
It is assumed that _L (X _L , Y _L ) and the point P _R (X _R , Y _R ) on the right screen are projected. In stereo image measurement, P _L and P _R are determined on the image plane, and the real space coordinates (x, y, z) of the point p are obtained by the principle of triangulation. Here, the two left,
Right camera 11L, the optical axis of 11R are in the same plane, Y _L from the fact that taking parallel the x-axis and the X-axis, Y _R takes the same value. The relationship between the coordinates X _L , Y _L , X _R , Y _R on the image plane and the coordinates x, y, z in the real space is

[0004]

[Equation 1]

Alternatively,

[0006]

[Equation 2]

Is required. here,

[0008]

Equation 3] d = X _L -X _R represents the parallax d. From (Equation 2), since a> 0,

[0009]

X _L > X _R and Y _L = Y _R This is because the corresponding point on the other screen corresponding to one point on one image plane is on the same scanning line which is an epipolar line, and X _L It exists in the range of> X _R. Therefore, a point on the other screen corresponding to one point on one image can be found by examining the similarity of the images for a small area along a straight line where the corresponding point may exist.

Next, a method of evaluating the similarity between both images will be described. As an example, a method of associating the left and right images described in Keiji Mitsuyoshi et al., “Driving Support System Using 3D Image Recognition Technology” (Preliminary Collection 9241992-10, Academic Conference of the Society of Automotive Engineers of Japan) will be described with reference to FIG. To do. First, Fig. 12
The left image 12L as the reference image shown in (1) is M in the horizontal direction,
It is divided into N pieces in the vertical direction and divided into M × N rectangular areas. In the following description, the rectangular area in the left image 12L will be called a block. Enlarged view of this block
The 12 (3) block 1201 is composed of m × n pixels. The brightness of the i-th pixel inside this block 1201 is L _i . Similarly to the left image 12L, a rectangular region of m × n pixels is set in the right image 12R shown in FIG. 12 (2), and the brightness of the i-th pixel inside the rectangular region is set to R _i . The similarity evaluation value C of the rectangular area between these left and right images is given by (Equation 5).

[0011]

(Equation 5)

A right image 12 in which a corresponding area may exist
In the search range 1202 in R, the rectangular area is set to 1 in the horizontal direction.
The similarity evaluation value C is calculated by moving pixel by pixel, and the area where this value is the minimum is set as the corresponding area. In this method, the corresponding area can be determined in block units in the left image 12L, and if the corresponding area is determined, the parallax d can be immediately obtained from the coordinate position of the corresponding area using (Equation 3). . From the parallax d obtained for each block in the reference left image 12L, z (below, distance data K) is obtained using (Equation 1). The distance data K will be shown as follows for convenience of explanation.

Distance data K (X, Y, t) X: horizontal index 1≤X≤MY: vertical index 1≤Y≤N t: time when the image was taken FIG. 13 is necessary for Hough transform FIG. 14 is an explanatory diagram of each straight line parameter, with respect to a straight line 1301 on the xy plane shown in FIG.
If the length of a perpendicular 1302 (distance to the origin which is a parameter of a straight line) drawn from the coordinate origin 0 is ρ, and the angle 1303 between the perpendicular 1302 which is a parameter of a straight line and the x axis is θ, the straight line 1301 Is shown as (Equation 6).

[0014]

[Equation 6] ρ = xcos θ + y sin θ Determined for the point sequence (x _i , y _i ) on the xy plane

[0015]

## EQU7 ## When ρ = x _i sin θ + y _i cos θ is expressed on the ρ-θ plane, a composite trigonometric function is obtained as shown in FIG. 14 (the horizontal axis is θ and the vertical axis is ρ). This sine curve will be called a Hough curve. This Hough curve shows all straight lines passing through the point sequence (x _i , y _i ). A sequence of points (x _i , y on the xy plane
_i ) is on a straight line, this point sequence (x _i , y _i ) is sequentially ρ−
When the Hough curve on the θ plane is used, the Hough curves intersect at one point 1401 as shown in FIG. The point (ρ ₀ , θ ₀ ) where the most curves intersect is the straight line (Equation 6) in the xy plane.
(ρ, θ). The above operation is called straight line fitting by Hough transform. Since the straight line fitting by the Hough transform is a straight line detection method for grasping the global situation of the data point sequence, the straight line detection can be performed even if the points which are not on the straight line are mixed.

[0016]

As described above, when the rectangular areas between the left and right images are associated with each other, the position where the similarity evaluation value C represented by (Equation 5) is the minimum is used as the corresponding area. Since there are few features in the rectangular area on the observation plane, the minimum point of the similarity evaluation value C may not be clearly obtained, and therefore the distance data may not be obtained.

[0017] This can, for example, distance data K obtained at a certain time _{t 0 (X, Y, t} 0) and then fixed distance time obtained for each interval Δt data _{K (X, Y, t 0} + nΔ
Inconvenience occurs when it is attempted to determine the presence or absence of an object on a plane based on the time change of t), (n = 1, 2, 3, ...). The time change of the distance data is | K (X, Y, t ₀ ) −K (X, Y, t _0).
+ NΔt) |, but as will be understood from this, it can be seen from the presence or absence of the object that any one of the distance data K (X, Y, t ₀ ), K (X, Y, t ₀ + nΔt) is uncertain. The judgment cannot be made.

The present invention solves the problem that the presence / absence of an object cannot be determined based on the uncertainty of the distance data K, and enables accurate measurement of distance data on a plane without being influenced by the object. To aim.

[0019]

In order to achieve the above object, the present invention is based on the principle of triangulation by associating corresponding regions between two images obtained by two image pickup means arranged at a predetermined interval. In stereo image processing for measuring the distance to an object, a distance is obtained for each rectangular small region BL (X, Y) 1 ≦ X ≦ M, 1 ≦ Y ≦ N obtained by dividing the image in the horizontal direction M and the vertical direction N. Data K (X, Y) is obtained and the rectangular small area B
For each large area GL (Y) 1 ≤ Y ≤ N in which L (X, Y) is grouped in the horizontal direction, Huffs are obtained from M distance data K (X, Y) included in the large area GL (Y). It is characterized in that the plane passing straight line L (Y) is fitted by conversion, and this is performed for all the large regions GL (Y) to estimate the plane existing in the imaging space.

[0020]

According to the present invention, by approximating the observation plane with a group of straight lines, the distance of the region on the observation plane for which distance data cannot be obtained is estimated. The Hough transform is used to estimate the straight line passing through the observation plane. Furthermore, by utilizing the property that the straight line group is in the same plane, Hough transform is performed again on the straight line group to correct the straight line at the wrong position and improve the reliability of the estimated distance data.

By using this plane estimation method, it is possible to estimate the plane by utilizing the portion where the distance data is obtained, even if the plane has few features such as a road plane or a floor plane. Moreover, since the global property of the Hough transform is used when a straight line is fitted, accurate distance data on the plane can be obtained even if there is an object with a height on the observation plane.

[0022]

DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 is a flow chart for explaining the processing flow from stereo image acquisition to observation plane estimation in one embodiment of the present invention. First, in the imaging phase (A), two imaging devices (for example, the left and right cameras 11L and 11R in FIG. 11) are used.
Two left and right images are obtained at (S1, S2). Next, the left and right images (S1, S2) obtained in the imaging phase (A) are associated with each other in the left and right images in the association phase (B) (S3). In the association phase (B), one reference image is divided into m × n pixel blocks, and the association is performed in block units by, for example, the method described in the conventional example. The parallax in block units obtained as a result of the association is passed to the plane estimation phase (C) in the subsequent stage. Plane estimation phase
In (C), the blocks are grouped in the horizontal direction, a straight line passing through the observation plane is fitted to each group by Hough transform, and a plane is estimated from the resulting straight line group.
(S4).

In the following explanation, as explained in FIG.
It is assumed that the left image (S1) as the reference image is divided into M blocks in the horizontal direction and N blocks in the vertical direction, and the distance data K (X, Y, t) is obtained in the block for which the parallax d can be determined. To do. The plane estimation phase (C) will be described with reference to the flowchart of FIG. Regarding the notation of the distance data K (X, Y, t) described in the conventional example, the time parameter t is omitted, and the distance data K (X, Y) is used as the notation of distance data.

(C) Plane estimation phase In the plane estimation phase (C), the observation plane is estimated using the distance data K (X, Y) of the block for which the parallax is determined,
Acquire undecided data.

(C1) Fitting a plane passing straight line FIG. 3 is a diagram for explaining the arrangement of blocks on the left screen, which is the unit for determining distance data in this embodiment. Here, for convenience of description, the blocks are expressed as follows.

Block: BL (X, Y), 1≤X≤
M, 1 ≦ Y ≦ N In the description of the plane estimation phase (C), three-dimensional K-X-Y reference coordinates as shown in FIG. 4 in which the distance data K (X, Y) is added to the indexes X, Y. Use the system.

In FIG. 3, a region 301L surrounded by a thick line
N (Y) is a block BL (X, Y) 302 composed of m × n pixels
Shows a region grouped in the horizontal direction, which is a region generated by fixing the index Y and changing only the index X. This area 301 is L
It is called N and each LN is described as follows.

Area: LN (Y), 1≤Y≤N LN includes M blocks BL (X, Y) 302. For each block BL (X, Y) 302 included in LN, the vertical axis is the block distance data K as shown in FIG.
The distance data K is plotted with (X, Y) and the horizontal axis as the block index X. When fitting the plane passing straight line L (Y), the index Y is fixed, so K-X
A point on the -Y reference coordinate space is projected on the K-X plane for description. By substituting each plotted point (K (X, Y), X) into (Equation 8) and changing θ in a specific section, one point (K
One Hough curve corresponding to (X, Y), X) is obtained.
The Hough curve corresponding to each point (K (X, Y), X) is ρ−θ
When plotted on a plane, it becomes as shown in FIG.

[0029]

[Equation 8] ρ = Xcos θ + K (X, Y) sin θ, θ _START ≦
θ ≤ θ _END ρ − The point where the Hough curves intersect the most in the plane (ρ _K (Y),
From θ _K (Y)) and (Equation 6), the straight line 501 on the K-X plane is (Equation 9)
Is determined as follows.

[0030]

[Equation 9]

On the ρ-θ plane of FIG. 5B, as a method for searching the points where the Hough curves intersect most, each point on the ρ-θ plane is arranged in a two-dimensional array HG as shown in FIG. 5B. (ρ _j , θ
_j ), and the two-dimensional array HG (ρ _j , θ _j ) is incremented every time the Hough curve passes through the point (ρ _j , θ _j ). By searching the two-dimensional array HG (ρ _j , θ _j ) with the highest frequency, the points (ρ _K (Y), θ _K (Y)) on the ρ-θ plane where the Huff curve passes most are obtained. Is required.

All areas (301) LN (Y), 1≤Y≤N
The same process is performed on the distance data included in the above to obtain a total of N plane passing straight lines L (Y). Each plane passing straight line L
FIG. 6 shows (Y) in K-X-Y reference coordinates. Since the inclination θ _K (Y) of the perpendicular and the distance ρ _K (Y) from the Y axis include the error components θ _E and ρ _E , respectively, for each optimum value θ _R (Y) and ρ _R (Y) hand

[0033]

## EQU10 ## θ _K (Y) = θ _R (Y) + θ _E ρ _K (Y) = ρ _R (Y) + ρ _E Therefore, the plane passing straight line L (Y) is not necessarily in the observation plane, but will be corrected in the subsequent processing.

(C2) Inclination correction of plane passing straight lines If all plane passing straight lines L (Y) are in the same observation plane, the angles formed by the normals to the Y axis of all plane passing straight lines and the X axis are the same. Should be. Therefore, the angle θ _K (Y) formed by the perpendicular of each plane passing straight line and the X axis is sorted by size, and the median value is replaced as the inclination of all plane passing straight lines from the X axis. As a result, the X of the perpendicular line of each plane passing straight line
The inclination from the axis can be unified with the optimum value θ _R (Y). This operation is shown in (Equation 11), and thus the inclination θ _K (Y) of the perpendicular from the X-axis includes an error component for each plane passing straight line. As described above, the error component can be removed from the inclinations of all plane passing straight lines, and the inclinations can be replaced with the most reliable inclination θ _R (Y).

[0035]

[Equation 11]

[0036]

(Equation 12)

By this operation, it is possible to correct the inclination of the plane passing straight line which shows an inclination different from the inclination of the observation plane about the Y axis. A straight line L '(Y) in which this inclination is corrected is shown by (Equation 12). K for each plane passing straight line
In the -XY reference coordinates, all the plane passing straight lines L (Y) have the same inclination as shown in FIG.

(C3) Distance correction of plane passing straight line If there are two planes in the three-dimensional space and the planes are not parallel to each other, the intersection of the two planes should be a straight line. A straight line where the observation plane estimated using this fact and the Y-ρ plane 801 shown in FIG. 8 intersect is fitted using the Hough transform. The distance from the Y axis of each plane passing straight line,
That is, ρ _K (Y) is plotted on the Y-ρ plane as shown in FIG. At the point (Y, ρ _K (Y)) plotted on the Y-ρ plane,
Hough transform is performed using 13).

[0039]

Equation 13 ρ ′ = Ycos θ ′ + ρ _K (Y) sin θ ′, θ ′
_{START ≦ θ '≦ θ' END} ρ'-θ ' that Hough curve is most intersecting a plane ([rho'
_S , θ ′ _S ) is obtained, and the straight line 901 is fitted according to (Equation 6). This line seems to be the most suitable Y of each plane passing line
Since it indicates the distance from the axis, it is called a distance correction straight line. This distance correction straight line exists at KY coordinates as shown by 802 in FIG. The distance correction straight line 802 is
It is shown as in 4).

[0040]

[Equation 14]

If the distance from the Y axis of each plane passing straight line is corrected to the optimum value ρ _R (Y) using the distance correction straight line, (Equation 12)
Is

[0042]

(Equation 15)

Then, the error component is completely removed. Here, the straight line L'corrected for the inclination shown in (Equation 12) is used.
A straight line group L ″ (Y) in which the distance from the Y axis is further corrected with respect to (Y) is expressed by (Equation 15). K-
When shown in XY reference coordinates, they all exist on the same plane as shown in FIG.

(C4) Determining the plane All the plane passing straight lines are present in the observation plane by the above phase, and the observation plane is approximated by the plane passing straight line group. If the distance data K (X, Y) on the observation plane has not been determined, the distance data on the observation plane can be obtained using the plane passage straight line (Equation 15) corresponding to LN (Y).

[0045]

As described above, if the plane estimation method of the present invention is used, the distance data of the entire plane can be obtained if partial distance data can be obtained even on a plane having few features on an image such as a road or a floor. Can be obtained. Furthermore, by using the Hough transform, it is possible to approximate a plane globally, and even if there is a high object on the road or floor, the distance data on the plane can be accurately measured without being affected by the object. You can get well.

[Brief description of drawings]

FIG. 1 is a flowchart illustrating a processing flow from stereo image acquisition to observation plane estimation according to an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a processing flow of a plane estimation phase shown in FIG.

FIG. 3 is an explanatory diagram of an arrangement of blocks on an image, which is a unit for determining distance data, for explaining the plane passing straight line fitting process of FIG. 2;

4 is a diagram showing a three-dimensional K-X-Y reference coordinate system which serves as a reference for explaining the fitting process of the plane passing straight line in FIG.

FIG. 5 is an explanatory diagram of a method of detecting a straight line parameter of a plane passing straight line from a distance data group.

FIG. 6 is a diagram showing the fitted plane passing straight line in a three-dimensional reference coordinate system.

FIG. 7 is a diagram showing a state in which a plane passing straight line corrected for inclination is shown in a three-dimensional reference coordinate system.

FIG. 8 is a diagram showing how a distance correction straight line used to correct the distance of a plane passing straight line from the Y axis is obtained.

FIG. 9 is a diagram showing how a straight line is fitted to a distance value from a Y axis of each plane passing straight line.

FIG. 10 is a diagram showing a state where the plane passing straight line whose inclination is corrected is further corrected for the distance from the Y axis and is shown in a three-dimensional reference coordinate system.

FIG. 11 is an explanatory diagram of the principle of stereo image measurement.

FIG. 12 is an explanatory diagram of a conventional method of associating left and right images.

FIG. 13 is an explanatory diagram of each straight line parameter required in Hough transform.

FIG. 14 is an explanatory diagram that a point at which the Hough curve intersects the most indicates a straight line parameter.

[Explanation of symbols]

(A) ... Imaging phase, (B) ... Correlation phase,
(C) ... Plane estimation phase, (S1) ... Left image, (S2) ...
Right image, (S3) ... Correlation, (S4) ... Plane estimation,
301 ... A region LN (Y) in which blocks BL (X, Y) are grouped in the horizontal direction, 302 ... Block BL (X, Y) consisting of m × n pixels, 501 ... A straight line fitted to the distance data K by Hough transform, 801 ... Y-ρ plane showing index Y and distance value from plane passing straight line to Y axis, 802, 9
01… distance correction straight line used to correct the distance from the Y axis of the plane passing straight line, 11L… left camera, 11R… right camera, 1201… m × n pixel block, 1202… block
A search range of the corresponding area in the right image of 1201, 1301 ... A straight line fitted by Hough transform, 1302 ... A distance to the origin which is a parameter of the straight line, 1303 ... A slope formed by the perpendicular line which is the parameter of the straight line and the X axis, 1401 ... The point where the most curves intersect (ρ ₀ , θ ₀ ).

Claims (3)

[Claims] 1. In stereo image processing for measuring a distance to an object on the principle of triangulation by associating corresponding areas between two images obtained by two image pickup means arranged at a predetermined interval, the images are horizontally aligned. A rectangular small area BL (X, Y) obtained by dividing the rectangular small area B in the vertical direction N by obtaining distance data K (X, Y) for each 1 ≦ X ≦ M, 1 ≦ Y ≦ N
For each large area GL (Y) 1 ≤ Y ≤ N in which L (X, Y) is grouped in the horizontal direction, Huffs are obtained from M distance data K (X, Y) included in the large area GL (Y). A plane estimation method characterized by estimating a plane existing in an imaging space by fitting a plane passing straight line L (Y) by conversion and performing this for all the large regions GL (Y). 2. A plane passing straight line L (Y) fitted to each of the large regions GL (Y), and using a slope θ (Y) of the straight line which is one of the straight line parameters of 1 ≦ Y ≦ N, each plane Straight line L
(Y) The median value θ _m is obtained by sorting the total N slopes θ (Y) of 1 ≦ Y ≦ N by size, and the slopes θ (Y) of all plane passing straight lines L (Y) are calculated. The plane passing straight line L ′ obtained by replacing with the median value θ _m and having the inclination corrected
The plane estimation method according to claim 1, wherein the plane existing in the imaging space is estimated using (Y). 3. A plane passing straight line L ′ whose inclination is corrected
A distance correction straight line h is applied by Hough transform to the distance ρ (Y) from the origin which is a straight line parameter of (Y), and the plane passing straight line L ′ (Y is corrected to have the inclination corrected so as to intersect with the distance correction straight line h. 3. The plane according to claim 1 or 2, wherein the plane existing in the imaging space is estimated by using a plane passing straight line group L ″ (Y), 1 ≦ Y ≦ N obtained by translating Estimation method.