JPH08159716A - Calculation method of vanishing point of road, and position/attitude measuring apparatus for vehicle - Google Patents

Abstract

PURPOSE: To determine a vanishing point of road involved even a curved road by estimating an imaginary straight road extending in the direction of a white line right below a vehicle. CONSTITUTION: A white line displayed along a road is extracted from an image taken and is made to approximate with respect to a hyperbola on an image to determine symptotes of the hyperbola. A vanishing point of the road is determined from the intersection of the symptotes obtained. The position and the attitude of a vehicle is computed using coordinates of the vanishing point of the road and coordinates of the white line or the like as shown on the image taken. This enables the measuring of the position and the attitude of the vehicle more accurately.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention takes an image of the surroundings of a vehicle with an on-vehicle camera fixed to the vehicle body, and based on the taken image, the speed of the vehicle and the position of the road, a preceding vehicle, an obstacle, etc., and the vehicle. The present invention relates to a method of calculating a vanishing point of a road and a position / orientation measuring device of a vehicle for recognizing a relationship and supporting driving.

[0002]

2. Description of the Related Art The front and rear of a vehicle are photographed by an on-vehicle camera fixed to a vehicle body, and white lines (including yellow lines) in the photographed image. Line) and guardrails (hereinafter collectively referred to as "road components") are detected, the point where the road disappears on the screen (road vanishing point) is found from the intersections, and the position coordinates of this road vanishing point and the screen A method of measuring the position / orientation of a vehicle based on the position of a road structure inside is known (Koji Asano et al. "Detection of own vehicle position by white line tracking", Image Electronics Society of Japan, Volume 21, No. 2 113-119, 1992).

According to this method, it is possible to measure the position / orientation of the vehicle which changes momentarily with respect to the road (road coordinate system). Therefore, it is possible to accurately recognize the position and speed of the preceding vehicle and obstacles in the captured image. It is possible to use it for avoiding a collision or the like by combining it with the vehicle speed and steering information.

[0004]

By the way, in the above-mentioned technique, it is assumed that the vehicle is traveling straight, and the road structure is considered to be straight and parallel. Figure 9 (a) shows
FIG. 9 shows a straight white line displayed on a straight road.
(b) shows the state where the white line is displayed on the screen.

However, if the above-mentioned technique is applied to a curved road on which a vehicle is turning, there is the following problem. That is, since the road is a curved road, it is necessary to consider a virtual straight road extending in the direction of the road component directly below the vehicle and extract this from the image (see FIG. 1).
(See (a)).

However, since the imaged image does not include the road component directly under the vehicle (usually, only the road component 1 to 3 m ahead is reflected), the above-mentioned processing is performed based on the road component displayed on the screen. However, it is not possible to find a correct "virtual straight road". Figure 10 (a) shows a white line displayed on a curved road and an incorrect "virtual straight road",
FIG. 10 (b) shows a method of obtaining the road vanishing point based on only the white line displayed on the screen.

Therefore, even if the position of the road vanishing point is obtained, the correct road vanishing point is not obtained and an error occurs.
Even if the position / orientation of the vehicle is measured based on the position of the road vanishing point, it will be incorrect information. Therefore, the present invention provides a road vanishing point calculating method capable of estimating a virtual straight line road extending a direction of a road component directly under a vehicle and determining the road vanishing point even on a curved road. The purpose is to do.

Another object of the present invention is to provide an apparatus for measuring the position / orientation of a vehicle by using the above-mentioned method of calculating a vanishing point of a road.

[0009]

A method of calculating a road vanishing point according to the present invention for achieving the above object extracts a road constituent displayed or installed along a road from a captured image. This is a method in which this is hyperbola-approximated on an image, the asymptote of the hyperbola is obtained, and the road vanishing point is obtained from the intersection of the obtained asymptotes.

Further, the vehicle position / orientation measuring apparatus of the present invention uses the coordinates of the vanishing point of the road calculated by the above method and the coordinates of the road constituents shown in the captured image. It calculates the position and orientation of the.

[0011]

The operation will be described with reference to FIG. Figure 1 (a) shows the road structure displayed on a curved road and the correct "virtual straight road", and Fig. 1 (b) shows the road vanishing point based on the road structure under the vehicle. The method of seeking is shown.

According to the present invention, the road structure near the vehicle is approximated by a hyperbola on the image, and the asymptote is obtained from the hyperbola. The tangential direction of the road structure beneath the vehicle is shown as an asymptote in the image. Therefore, this asymptote is regarded as a "virtual straight road", and its intersection is set as the road vanishing point (see FIG. 2). Note that two asymptotes are generally obtained for one hyperbola, but if the roll angle ψ of the vehicle (see FIG. 3) is 0, then one of them is calculated as shown in FIG. 2 (a).
The book becomes a "virtual straight line", and the other one corresponds to the horizon (see claim 2).

Since the horizon is horizontal on the screen, the other one is horizontal on the screen. Therefore, the 1 of the asymptote
By ² + cx + dy + e + xy = 0 can be adopted as a hyperbolic equation in which the book is horizontal. Here, x is the horizontal coordinate of the screen, y is the vertical coordinate of the screen, and b, c, d, and e are coefficients indicating the shape of the hyperbola. In this case, the equation representing the asymptote is x = -by + bc-d.

Further, if the coordinates of the vanishing point of the road are obtained, the position / orientation of the vehicle can be calculated by using the coordinates of the road constituent shown in the picked-up image (see claim 3).

[0015]

Embodiments will be described in detail below with reference to the accompanying drawings showing embodiments. In the embodiment, it is assumed that there are three white lines on the road at both ends and the center of the road as shown in FIG. 4, and the X axis is horizontal to the direction perpendicular to the road, the Y axis is vertical, the X axis and the Y axis are vertical. Take the Z axis at a right angle to. The road width of one lane is L _W.

FIG. 5 is a block diagram showing the electrical construction of the vehicle position / orientation measuring apparatus. This vehicle position / orientation measuring device includes a vehicle-mounted video camera 1 attached to the front portion of the vehicle or inside the vehicle. The vehicle-mounted video camera 1 is capable of capturing an image of the front of the vehicle. The vehicle-mounted video camera 1 outputs an analog electric signal representing a captured screen. The analog signal is subjected to analog / digital conversion processing in the image input section 2 and converted into image data. This image data is subjected to image processing such as filtering and edge extraction in the image processing unit 3, and then input to the white line extraction unit 4 including a microcomputer and the like.

The white line extraction unit 4 extracts a straight line portion or a curved line portion of a road constituent such as a white line on the road from the edge image obtained by the image processing unit 3. The road vanishing point calculation unit 5 obtains a virtual straight road from the straight line portion or the curved line portion,
Then find the road vanishing point. The vehicle position / orientation calculation unit 6 obtains the vehicle position and orientation from the road vanishing point and the virtual straight road.

Now, assuming that there are three white lines (see FIG. 4), the white line extraction processing, the road vanishing point calculation processing, and the vehicle position / attitude calculation processing will be described in detail. (1) White line extraction processing Since the white line has a certain width, it appears as two contour lines on the edge image. Therefore, raster scanning is performed from the bottom of the screen, and the edge image data for three lines is read into a buffer memory (not shown).

Then, one row of the buffer memory is scanned to check whether or not two or more pixels having a brightness larger than the threshold value are continuous. The next one row is also scanned to check whether or not two or more pixels having a brightness larger than the threshold value continue. Further, the next one row is also scanned to check whether or not two or more pixels having a luminance larger than the threshold value continue.

Two or more consecutive pixels appear in three rows in a row at substantially the same position (x coordinate), and two or more consecutive pixels to the right of the two or more consecutive pixels are in three rows in a row at substantially the same position (x coordinate). When it appears on the x-coordinate), it can be confirmed that it is a parallel edge, and thus two or more consecutive pixels thereof are set as white line edge candidate pixels. Obtaining the average luminance of the portion sandwiched between the white line edge candidate pixels, obtain the average luminance of a predetermined number of peripheral pixels existing on the left and right of the white line edge candidate pixels, the average luminance of the pixels sandwiched between the white line edge candidate pixels, If it is larger than the average luminance of the peripheral pixels by a predetermined value or more, the portion sandwiched by the white line edge candidate pixels is regarded as a white line, and the center pixel thereof is set as a white line candidate point.

When the above processing is completed, the same search is performed for the next three rows, and finally all rows are processed. (2) Road vanishing point calculation processing In this processing, the road vanishing point is calculated by the hyperbolic approximation method based on the white line obtained by the white line extracting unit 4.

The coordinates of the obtained white line candidate points are (x _i ,
_{y i) (i = 0,1,2,} ...) and when the average value x _m of x _i, the average value y _m of y _i, a standard deviation sigma _x of x _i, a standard deviation sigma _y of y _i calculated , Correlation S _xy S _xy = Σ {(x _i −x _m ) / σ _x } {(y _i −y _m ) / σ _y } (1). The total sum Σ is taken over the total number of white line candidate points. Then, with respect to a certain threshold value S _th , when S _xy > S _th , the white line is determined to be a straight line, and when S _xy <S _th , the white line is determined to be a curve.

When the obtained white line is a straight line (when the vehicle is traveling on a straight road), the straight line equation is approximated by x = Ay + B (2), and the coefficients A and B are calculated using the least square method. Ask for.
The above processing is performed on the white lines at both ends of the road and the white line at the center of the road, and straight lines are obtained and the intersections thereof are obtained. This intersection becomes the road vanishing point (x ₀ , y ₀ ). Next, FIG. 6 shows the case where the vehicle is traveling on a curved road having three white lines (when the vehicle is traveling on a curved road). The resulting image is shown. Three curves are displayed in this image. This curve is approximated by a hyperbola. As described above, the hyperbolic equation is represented by by ² + cx + dy + e + xy = 0 (3) when the roll angle ψ of the vehicle is 0. Therefore, the coefficients b, c, d, and e are calculated by the following equation using the least squares method. Σ (sum) is i
= 0 to N-1.

[0024]

[Equation 1]

If the shape of the hyperbola is known, the asymptote of the hyperbola is obtained. There are two hyperbolas in the equation (3), and they are obtained by mathematical solution as x = -by + bc-d (5) y = -c (horizontal line) (6).

In this way, the white line is approximated by a hyperbola, and two asymptotes are obtained for each of these hyperbolas. Of these, the asymptotic line x = −by + bc−d on the non-horizontal side is defined as a “virtual straight road”. For each white line, find the intersection of virtual straight lines, and use this intersection as the road vanishing point (see Fig. 2 (a)).

In this road vanishing point calculation processing, it is determined whether the white line is a straight line or a curved line by _{obtaining the} correlation S _xy as described above. However, the method is not limited to this method. For example, the obtained white line candidate points (x _i , y _i ) are
Using the method of least squares, approximating with a straight line equation x = α + βy,
2 of each white line candidate point (x _i , y _i ) based on this straight line
Multiplying error EE = Σ {(α + βy _i ) −x _i } ² is calculated. The total sum Σ is taken over the total number of white line candidate points. Then, the mean square error E is compared with the threshold value E _th, linear smaller than the threshold value E _th, it may be determined that the greater if the curve.

In this road vanishing point calculation process, the roll angle ψ of the vehicle is set to 0 as described above, and a hyperbolic functional form is assumed to obtain an asymptote. However, the assumption of the roll angle ψ = 0 was made to simplify the function form of the hyperbola and to make it easier to process. Even when the roll angle ψ ≠ 0, the asymptotic line should be found to find the road vanishing point. There is no change in what you can do. FIG. 2 (b) shows the appearance of the white line, asymptote, etc. on the screen when the vehicle rolls to the left.

In this case, the equation corresponding to equation (3) of the hyperbola is ax ² + by ² + cx + dy + e + xy = 0 (6-2), and the corresponding determinant of equation (4) is

[0030]

[Equation 2]

It becomes The asymptotic formula is x = Ay + B,

[0032]

(Equation 3)

It is represented by Of these, the one passing near the vehicle (below the image) is defined as a virtual straight road. (3) Vehicle Position / Attitude Calculation Processing FIG. 7 (a) is a top view of a curved road, and FIG. 7 (b) is a side view. The road coordinate system is (X, Y, Z), the Z axis is parallel to the road surface and extends in the direction of the road, the X axis is parallel to the road surface and perpendicular to the Z axis, and the Y axis is perpendicular to the road surface and upward. The origin of the road coordinate system is on the white line in the center.

The coordinate system (U, V, W) of the camera is
The W axis is taken in the principal axis direction of the lens, the U axis is taken in the lateral direction of the imaging surface, and the V axis is taken in the longitudinal direction of the imaging surface. The image coordinate system (x, y) has an x-axis parallel to the U-axis, and V
Take the y-axis parallel to the axis. The vehicle pitch angle φ, yaw angle θ, and roll angle ψ are set (see FIG. 3), and the rotation direction of the left-hand screw is positive. _Let D _{X be} the lateral deviation of the camera lens from the center white line, and D _{Y be} the height from the road. Note that FIG.
Then, the roll angle ψ is set to 0 for easy illustration, but in reality, a roll occurs and the roll angle ψ has a finite value.

It is possible that the road vanishing point calculation process is inconsistent with the case where the roll angle ψ is set to 0. However, the road vanishing point calculation process only calculates the road vanishing point. After obtaining the road vanishing point, the position / orientation calculation processing of the vehicle can be performed with the roll angle ψ taken into consideration after the road vanishing point is obtained. Road coordinate system (X, Y, Z) to camera coordinate system (U, V, W)
Conversion to the, X direction D _X, after moving D _Y parallel to the Y direction, theta, phi, is rotated in the order of [psi. as a result,

[0036]

[Equation 4]

The conversion formula is obtained as follows. The projection from the camera coordinate system (U, V, W) to the image coordinate system (x, y) is
When the distance between the lens and the screen is F (focal length), x = FU / W (8) y = FV / W (9) using paraxial ray approximation.

From the above equations (7) to (9), the road coordinate system (X,
Y, from Z) image coordinate system (conversion equation into x, y) _{is, x = F (R 11 (} X-D X) + R 12 (Y-D Y) + R 13 Z) / (R 31 (X- D _X ) + R ₃₂ (Y−D _Y ) + R ₃₃ Z) (10) y = F (R ₂₁ (X−D _X ) + R ₂₂ (Y−D _Y ) + R ₂₃ Z) / (R ₃₁ (X−D _X ) + R ₃₂ (Y−D _Y ) + R ₃₃ Z) (11).

The coordinates (x ₀ , y ₀ ) of the road vanishing point are (10)
When Z = infinity in the equation (11), x ₀ = FR ₁₃ / R ₃₃ (12) y ₀ = FR ₂₃ / R ₃₃ (13) Furthermore, _let us consider the expression of the white line on the image in the road coordinate system, where the lateral displacement D _X and height D _Y of the camera are also unknowns.

When the equation (7) is expressed with respect to X, Y and Z,

[0041]

(Equation 5)

It becomes R ⁻¹ is an inverse matrix of the rotation matrix R. Of the three white lines on the screen, an arbitrary point on the central white line is (x _c , y _c ). Convert this to the road coordinate system. Since the inverse transformation of equations (8) and (9) cannot be performed, the transformation is performed as follows using the parameter t.

[0043]

(Equation 6)

Substituting the expression (15) into the expression (14), X = t (R ₁₁ x _c + R ₂₁ y _c + R ₃₁ F) + D _X = 0 (16) Y = t (R ₁₂ x _c + R ₂₂ y _c + R ₃₂ F) + D _Y = 0 (17) is obtained. By deleting t from this, the following equation is obtained.

[0045] _{(R 12 x c + R 22} y c + R 32 F) D X - (R 11 x c + R 21 y c + R 31 F) D Y = 0 (18) Then, any point on the right white line _Be (x _r , y _r ). Convert this to the road coordinate system. Also at this time, using the parameter t, X = t (R ₁₁ x _r + R ₂₁ y _r + R ₃₁ F) + D _X = L _W (19) Y = t (R ₁₂ x _r + R ₂₂ y _r + R ₃₂ F ) + D _Y = 0 (20) is obtained. However, L _W is the road width of one lane and is known (see FIG. 4).

From this, t is deleted to obtain the following equation. (R ₁₂ x _r + R ₂₂ y _r + R ₃₂ F) (D _X −L _W ) −D _Y (R ₁₁ x _r + R ₂₁ y _r + R ₃₁ F) = 0 (21) Point on the left straight line (x _l for the y _l), (21) need only replace the expression of L _W to _{-L W, (R 12 x l} + R 22 y l + R 32 F) (D X + L W) -D Y (R 11 x _l + R _{21 yl} + R ₃₁ F) = 0 (22) is obtained.

These equations (18) (21) (22) and the above (10) (11)
Using the formulas and, the coordinates (x_r, Y_r), (X
_c, Y_c), (X_l, Y_l), Coordinates of road vanishing point
(X₀, Y ₀), Road width L_WIs a known number
Lateral deviation D_X, Height D_Y, Pitch angle φ, yaw angle θ, roll
The angle ψ can be determined. The height of the camera D_YTo
If it can be regarded as constant, the road width L_WIs unknown
The above equations (18) (21) (22) (10) (11) can be applied to
It

Further, the height D _{Y of the} camera can be regarded as constant, and when the road width L _W is known from the road map data or the like, it is possible to use any of the two white lines without using three white lines.
The lateral deviation D _X , the pitch angle φ, the yaw angle θ, and the roll angle ψ of the camera can be obtained only by the information of the white line of the book. <Calculation example> Using a vehicle with a camera mounting height D _Y = 1 m, a road with a radius of curvature of 400 m and a road width L _W = 3.75 m is laterally displaced from the central white line D _X = 0 m and pitch angle φ. =
The vehicle ran with 0 °, yaw angle θ = 0 °, and roll angle ψ = 0 °.

As a result, a screen as shown in FIG. 8 was obtained. A hyperbolic approximation is applied to the obtained screen to obtain the road vanishing point (see FIG. 8 (a)), and the height D _Y , lateral deviation D _X , pitch angle φ, yaw angle θ, roll angle ψ are calculated. It was found that D _Y = 1.
012m, D _X = 0.002m, pitch angle φ = -0.0
546 °, yaw angle θ = −0.0349 °, roll angle ψ =
A value of -0.0135 ° was obtained.

On the other hand, as shown in FIG. 8B, the height D _Y , the lateral deviation D _X , the pitch angle φ, the yaw angle θ, and the roll angle ψ were obtained by tangential approximation, and D _Y = 1 0.004 m, D _X =
0.023 m, pitch angle φ = −0.184 °, yaw angle θ
The numerical values of = -0.725 ° and roll angle ψ = + 0.409 ° were obtained. As compared with the case of the hyperbolic approximation, it can be seen that the error is increased in general and especially the error of the yaw angle θ is increased.

Although the contents of the present invention have been described based on the embodiments, the present invention is not limited to the above embodiments. In the above-mentioned embodiment, the road vanishing point is calculated and the position / orientation of the vehicle is measured with respect to the white line of the road, but a road component such as a guardrail may be used as the target.

[0052]

As described above, according to the present invention, a road constituent is extracted from a picked-up image, and this is hyperbola-approximated on the image, and a road vanishing point is determined from an intersection of asymptotic lines of the hyperbola. Since it is obtained, accurate coordinates of the road vanishing point can be obtained even while the vehicle is traveling on a curved road. Therefore, if the position / orientation of the vehicle is calculated using the coordinates of the road vanishing point and the coordinates of the road constituent shown in the captured image, the position / orientation of the vehicle can be measured more accurately. it can.

In particular, if one of the asymptotic lines of the hyperbola is assumed to be the horizon as described in claim 2, the coordinates of the road vanishing point can be obtained with a smaller amount of calculation, and the calculation time can be shortened. Can be planned.

[Brief description of drawings]

FIG. 1 (a) shows a road structure displayed on a curved road and a correct “virtual straight road”, and (b) shows a road vanishing point based on the road structure directly under the vehicle. It is a figure which shows the method of this invention calculated | required.

FIG. 2 (a) shows how to determine a vanishing point of a road when the roll angle of the vehicle is 0 and one of the asymptotes corresponds to the horizon, and (b) shows the roll angle of the vehicle is not 0. It is a figure which shows how to obtain the road vanishing point in the case.

FIG. 3 is a diagram showing a vehicle pitch angle φ, yaw angle θ, and roll angle ψ.

FIG. 4 is a diagram showing a premise of an embodiment in which there are three white lines on a road at both ends and at the center of the road.

FIG. 5 is a block diagram showing an electrical configuration of a vehicle position / orientation measuring apparatus.

FIG. 6 is a diagram showing an image when traveling on a curved road having three white lines.

FIG. 7A is a view for explaining a coordinate system when a curved road is viewed from above, and FIG. 7B is a view when viewed from the side.

FIG. 8 is a diagram illustrating a method of applying a hyperbola approximation to obtain a road vanishing point and a method of applying a tangent approximation to a road vanishing point on the obtained screen, for comparison.

FIG. 9A is a diagram showing two straight white lines displayed on a straight road, and FIG. 9B is a diagram showing a state in which the white lines are displayed on the screen.

FIG. 10 (a) is a white line displayed on a curved road,
It is a figure which shows the wrong "virtual straight road", and (b) shows the method of calculating | requiring a road vanishing point based only on the white line reflected on the screen.

[Explanation of symbols]

 1 Video Camera 2 Image Input Section 3 Image Processing Section 4 White Line Extraction Section 5 Road Vanishing Point Calculation Section 6 Vehicle Position / Attitude Calculation Section

Continuation of front page (51) Int.Cl. ⁶ Identification code Office reference number FI Technical display location G01C 21/00 A G06T 1/00 7/60 G08G 1/16 C

Claims (3)

[Claims] 1. A method of capturing an image of the surroundings of a vehicle with an in-vehicle camera fixed to the vehicle and calculating a road vanishing point based on the captured image, which is displayed along the road from the captured image. The installed road structure is extracted, and this is hyperbola-approximated on the image, the asymptote of this hyperbola is found, and the road vanishing point is characterized by finding the road vanishing point from the intersection of the asymptotes found. Calculation method. 2. The method of calculating a road vanishing point according to claim 1, wherein one of the asymptotic lines of the hyperbola corresponds to the horizon. 3. A vanishing point calculating means for calculating a road vanishing point according to claim 1, a coordinate of the calculated road vanishing point, and a coordinate of a road constituent shown in a captured image are used. A position / orientation measuring device for a vehicle, comprising: a calculating means for calculating the position / orientation of the vehicle.