JP3319383B2 - Roadway recognition device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a travel path recognition device for a moving object, and more particularly, to an apparatus for mounting an image processing system on a moving object to accurately detect a traveling path from a captured road surface image and to provide a safety device. The present invention relates to a travel path recognition device that enables easy driving support and automatic driving.

[0002]

2. Description of the Related Art Conventionally, as a traveling path detecting apparatus utilizing image processing, a traveling path detecting apparatus for a mobile vehicle disclosed in Japanese Patent Application Laid-Open No. 5-164569 is known. Briefly, an image processing system is mounted on a vehicle, and an image of a guide line drawn on a road surface during traveling is captured. The feature is that a captured image is divided into a plurality of regions and processed. The captured image is composed of two straight lines (L,
R) and where the image is expected to appear,
A plurality of search areas (windows: W1, W2... W5) are set.

In a plurality of areas, in order to detect a guide line,
An edge portion or the like with an intensity change is searched for, and candidate points thereof are obtained. The computer system calculates an approximate straight line passing through the plurality of obtained candidate points (P1,... Pn) and recognizes it as a current guide line.
The approximate straight line recognized two-dimensionally by the imaging device is recognized as a straight line in a three-dimensional space constructed in a computer system by a geometric conversion rule, and used for attitude control of the vehicle.

[0004] Further, since the traveling path changes gradually and continuously, a plurality of search areas (windows: W1,
W2... W5) also need to be changed accordingly. This change is made by applying a correction to the current search area, and the correction amount is set so that the center position of each search area overlaps the current approximate straight line. in this way,
Since the plurality of search areas are corrected so as to always capture the guide line, reliable autonomous traveling can be performed.

[0005] Further, the computer system may erroneously recognize other signs and curbs such as sidewalks drawn on the road surface as guide lines. In such a case, an absolute comparison and an intercomparison of the correction amounts of the plurality of search areas are performed, and the validity is determined. That is, when a correction amount equal to or more than the reference value is calculated in a certain search area, it is determined to be erroneous recognition,
No correction is made, and the current search area is used as it is as the next search area. Therefore, even if something other than a guide line is detected in a certain search area, it is excluded as an erroneous recognition, and the correct traveling road recognition is performed comprehensively.
Safe independent driving is possible.

[0006]

However, the above-mentioned method uses the candidate points detected in the previous search areas, and
This is a kind of feedback system that finally substitutes the approximate straight line for the next running path. In other words, the conventional method is effective for a straight road such as an expressway or a curved road with a small curvature and a very gentle change, but a general curved road with a sudden change and a large curvature has an effect in the image. The displacement amount of the traveling path becomes large and is not always applicable. Therefore, safe independent traveling is not always guaranteed. In addition, large longitudinal vibration or pitch angle vibration may be suddenly applied to the vehicle-mounted camera. In such a case, in the above-described method, the correction amount may exceed the reference value in a plurality of search areas, and the recognition may be determined to be abnormal even though the traveling path is normal.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-described problems. A prediction line at the time of current imaging is calculated based on a previous guidance line parameter and a vehicle traveling state parameter, and the prediction line is used as a reference. Paying attention to the fact that the dispersion state of the guide line candidate points is relatively large when foreign objects on the road surface are detected, and relatively small at locations where the guide line changes suddenly, confusing lines such as roads with large curvature or road markings It is an object of the present invention to provide a travel path recognition device that flexibly follows a general road and detects the travel path with high accuracy.

[0008]

In order to achieve this object, a travel path recognition apparatus according to the present invention captures an image of a travel path using an image pickup device mounted on a moving body, and guides the moving body from the obtained image. A travel path recognition device for recognizing a guide line serving as an index, comprising: a guide line candidate extracting unit configured to extract a guide line candidate serving as a guide line candidate from an image; a guide line obtained during a previous image capturing; From the running state of the image, the guidance line prediction means for predicting the appearance position of the guidance line for the image obtained at the next imaging, and the image processed by the guidance line candidate extraction means, the guidance line prediction means A deviation calculating means for setting a range of a predetermined width around the predicted prediction line, extracting a guide line candidate point by searching the range, and calculating a deviation distribution of the guide line candidate point with respect to the prediction line; Calculated by the calculating means A variance-related value calculating means for calculating a variance-related value related to the variance of the deviation distribution; and a guide line determining means for adopting a guide line candidate as a genuine guide line when the variance-related value is equal to or greater than a predetermined value. I have. The variance-related value calculating means includes an average value calculating means for calculating an average value of the deviation distribution calculated by the deviation calculating means, and a ratio of the deviation distribution existing in a predetermined range around the average value. And means for calculating as

[0009]

The guide line candidate extracting means, which is a component of the present invention, performs image calculation processing on an image captured by an image pickup device mounted on a moving body such as an automobile, and performs a guide line candidate. Extract. The image calculation processing is, for example, enhancement processing by binarization and contour processing by differentiation. For the extracted image, the guide line prediction means
The appearance position of the guide line is predicted for the image obtained at the next image capture from the guide line obtained at the previous image capture and the traveling state of the moving body.

The deviation calculating means sets a range of a predetermined width around the predicted line obtained by the guiding line predicting means for the image processed by the guiding line candidate extracting means. Then, a guide line candidate point is extracted by searching for an image within the range, and a deviation distribution of the guide line candidate point with respect to the prediction line is calculated. The variance-related value calculation means calculates a value related to the variance of the calculated deviation distribution.

Finally, the guide line determining means adopts the guide line candidate as a genuine guide line when the dispersion-related value is equal to or less than a predetermined value. In this way, the deviation distribution between the predicted line and the guide line candidate is obtained, and when the variance of the guide line candidate is small, it is determined that the guide line is a genuine guide line. For example, when a curb or the like is detected on a road surface, the variance of the deviation distribution becomes large. On the other hand, when a genuine guide line such as a white line suddenly changes, the absolute deviation of the guide line candidate itself with respect to the prediction line increases, but the variance of the deviation distribution along the guide line is small. Therefore, even if the guide line fluctuates greatly due to a change in other signs or cameras drawn on the road surface, the guide line can be surely distinguished, and a safer traveling road recognition device can be obtained.

[0012]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a configuration diagram showing an example of the present embodiment.
100, a ROM 110 in which a recognition program is written, and a RAM 12 as a work area memory when the program is executed.
0, an I / O interface 130 as an input / output interface with a control device (not shown), a high-speed A / D conversion of a video signal sent from a camera, and taking it into a frame memory 165 dedicated to an image. It comprises an image input device 160. These components are all address buses, data
The CPU 100 is connected to a system bus 140 composed of various signal lines, and data is exchanged and controlled by various programs written in the CPU 100 and the ROM 110. All means that constitute the present invention
It is formed by the traveling path recognition processing program stored in the OM 110 and the above-described computer system that executes the program.

Next, the operation of the traveling road recognition apparatus according to the present invention will be described with reference to the flowchart of the traveling road recognition program shown in FIG. Below, for simplicity,
Although one guide line will be described, the same process is actually performed for two right and left lines. Here, as the guide line, a white line drawn on both sides of the traveling road is targeted. In step S <b> 100, the road surface image is fetched into the frame memory 165 by the image input device 160 from the camera 150 which is an imaging device. The frame memory has a plurality of Rs.
AM, one unit (one piece)
Is a total of 512 × corresponding to each pixel of the CCD image sensor.
512 is a RAM. Arbitrary pixel is coordinate (x, y)
And intensity I, and the intensity I is digitized in 0 to 255 steps. The center of the screen is the origin of the xy coordinate system, and the origin corresponds to the intersection of the optical axis of the camera and the road surface in the real space.

Next, the process proceeds to step S110. Step S
110 is a guide line candidate extracting means of the present invention. Here, the image taken into the frame memory 165 is first subjected to Sobel operator processing or the like, which is a matrix weighting operation, to calculate an intensity image and an angle image (differential image) representing a density gradient, respectively. A line segment having an intensity equal to or greater than a predetermined threshold is extracted from the intensity image, and a plurality of edges corresponding to the line segment, that is, accurate boundaries between the line segment and the background are extracted from the angle image. If the distance between the edges, that is, the width of the line segment corresponds to the width of the white line drawn on the traveling path, and the angles between the edges are opposite to each other, the line segment extracted in the intensity image is a guide line. It is determined as a candidate.

As shown in FIGS. 4A and 4B, a road coordinate system (XYZ) and a camera coordinate system (X'Y'Z '), which are three-dimensional space coordinates, are set. Road coordinate system (XYZ)
Is a center line between the two guide lines A1 and A2 (hereinafter simply referred to as
This is a coordinate system on a road surface having an origin on the center line (referred to as a center line) and moving along the center line such that the origin changes as the vehicle moves. The tangent direction of the guide lines A1 and A2 is the Z axis,
The direction perpendicular to the road surface (height direction) is the X axis, and the direction parallel to the road surface in the right direction toward the traveling direction is the Y axis. The camera coordinate system (X'Y'Z ') is a coordinate system viewed from a camera rigidly fixed to an automobile. Z 'axis direction is the optical axis direction of the camera, Y' is the horizontal direction with respect to the camera,
X 'is the vertical direction with respect to the camera. (X, y)
Are coordinates in the xy coordinate system described above taken on the screen. The equation of the curve in the road coordinate system (XYZ) of the guide line drawn on the traveling path is expressed by the following equation.

[0016]

X = 0 (1)

Y = cZ _c ² + κW-e (2)

## EQU3 ## Z = Z _c (3)

Here, Z _c is the Z coordinate on the center line, c is の of the curvature of the guide line, W is the interval between the two guide lines A1 and A2, ie, の of the width of the traveling path, κ. Is 1, -1.
The right lead A2 is κ = 1, and the left lead A2 is κ = −.
It is one. e is the Y coordinate of the camera origin in the road coordinate system (XYZ) (positional deviation from the center line of the camera origin). The above equations (1) to (3) are expressed by the Z coordinate Z on the center line.
It is an equation of the curve of the guide lines A1 and A2 using _c as a parameter.

The projection coordinate conversion using the camera parameters such as the position, orientation, and focal length of the camera in the road coordinate system (XYZ) can convert the road coordinate system (XYZ) to the coordinate system (xy) on the screen. is there. Therefore, if the point (X, Y, Z) on the above guide line is represented by coordinates (x, y) on the image, the points (X, Y, Z) are given by equations (4) and (5), into which equations (6) and (7) are substituted. Can be

[0019]

X = x'cos {+ y'sin} (4)

Y = −x′sin {+ y′cos} (5)

[6] _{x '= - F x h /} Z c -F x φ (6)

Equation 7] _{y '= F y (cZ c} 2/2 + κW-e) / Z c -F y θ (7)

Where F _x and F _y are lens focal lengths, h is a Z coordinate (height) in the road coordinate system (XYZ) at the camera origin, θ is a yaw angle, φ is a pitch angle, and ψ is a roll angle. is there. These variables (θ, φ, ψ) are in the road coordinate system (XY
This variable is defined with reference to each axis of Z), and represents the attitude of the camera. However, it is assumed that θ and φ are small.

From the positions of the guide lines A1 and A2 on the screen, the positions of the guide lines A1 and A2 in the road coordinate system (XYZ) can be known. Thus, the position deviation e, posture (θ, φ, ψ) of the vehicle body from the center line, the Z coordinate Z _C on the center line, and the curvature c of the guide line can be obtained. Based on these measured parameters, control and warning of the position and attitude of the vehicle are performed. Since this control method is a known method and is not the gist of the present invention, the description is omitted.

Next, a method of determining a guide line on a screen, which is the gist of the present invention, for controlling the position and attitude of an automobile as described above will be described. Parameters (e (t), c) obtained from the images observed up to the last time
From the locus of (t), W (t), θ (t), φ (t), ψ (t)), the values of these parameters at the next observation are predicted and calculated. That is, in step S160 described later at the previous imaging timing, the values of the above parameters at the time of the next imaging are predicted using the Kalman filter. This will be described later.

In step S120, which is a guide line predicting means, the guide lines A1 and A2 on the screen are based on these predicted parameters at the time of the current imaging.
Is calculated. That is, the predicted line L1 of the guide line A1 on the screen is obtained by the equations (4) to (7) using those parameters. In FIG. 3, the guide line A1
, But the same applies to the guide wire A2.

As shown in FIGS. 3A and 3B, a predicted line L1 calculated based on the predicted values of the parameters is formed on the intensity image from which the guide line candidates are extracted.

Next, steps S130 to S15
At 0, the validity of the guide line candidate extracted at step S110 is checked based on the predicted line L1. Step S130 is a deviation calculating means. Here, the prediction line L
The deviation distribution of the guide line candidate for 1 is obtained. FIG.
As shown in (a) and (b), a first window DW1 having a predetermined search width 2D1 around the prediction line L1 is set, and a guide line candidate B is searched therein.

In the search method, for convenience, an index i is defined for a specific scanning line, and a prediction line L1 intersecting with the scanning line i is defined.
The y-coordinate value ci of the guide line candidate B that intersects with the scanning line i in the same manner as the y-coordinate value bi is obtained. The point where the scanning line i intersects with the guide line candidate B is referred to as a candidate point Ci for convenience.

The difference between them, that is, the position deviation d
i (= ci-bi) is calculated. The position deviation di has a sign, and the sign is, for example, positive on the right with respect to the prediction line L1 and negative on the left with respect to the prediction line L1. Then, as many dis as indicated by the index i (in FIG. 3, i
= 1... N), that is, a deviation distribution is obtained, and the process proceeds to step S140.

Step S140 and the next step S15
0 constitutes the dispersion-related value calculation means. Step S140 is also an average value calculating means of the dispersion related value calculating means. Here, the average value dave of the deviation distribution obtained in step S130 is calculated.

Next, step S150, step S16
Move to 0. First, in Step S150, Step S
In order to determine the validity of each position deviation di obtained in 130, that is, the validity of the candidate point Ci, the deviation average value dav is determined.
Based on e, the dispersion state of each position deviation di is examined. In this method, as shown in FIGS. 3A and 3B, an average value line L2 is newly created by adding the above average value dave to the prediction line L1. Then, a second window DW2 having a predetermined search width 2D2 around the average value line L2.
Is set, and a guide line candidate B is searched in the area.

The search method is described in step S1 shown in FIG.
This is indicated by a subroutine starting from 51. Step S151
Then, the scanning line index i and the counter W are initialized to 0. In step S152, the index i is updated, and in the next step S153, the distance from the average value line L2, | ci-bi-dave |, that is, the absolute value of the deviation of the position deviation with respect to the average position deviation | di-dave |
(= Δi) is calculated.

It is determined whether or not the value Δi is smaller than a predetermined value D2. If it is smaller than D2, step S154 is performed.
Then, the counter W is incremented by 1 and updated. D
If not, the value of the counter W is not updated. Next, the process proceeds to step S155, where it is determined whether or not the index i is the final value N. If not, the process returns to step S152, and the same processing is performed for the next scanning line. If it is determined in step S155 that the value is the final value, the process proceeds to step S156 to determine whether the value of the counter W is equal to or greater than a predetermined value Th. If it is equal to or greater than the predetermined value, the variance of the position deviation from the average value is equal to or less than the predetermined value, and the guide line candidate is determined to be authentic. This step S156 constitutes the guide line determination means.

The value of the counter W corresponds to the variance σ of the position deviation di. Therefore, if the variance σ of the positional deviation is small, there is a unity as a guide line, and the probability that the line segment is a guide line as a whole judgment is high, so that the guide line candidate B is determined as a genuine guide line.

In step S157, the information of the guide line determined as a genuine guide line is obtained as the y coordinate value ci of the guide line candidate corresponding to the index i, ie, the candidate point Ci.
(X, y) coordinates are registered in a predetermined memory area.

FIG. 6 shows the data structure. Where k
Is a constant representing the scanning line interval. For example, when checking at 5 scanning line intervals, it is set to 5. N is the total number of necessary guide line formation points, for example, 50.

If the value of the counter W is equal to or less than the predetermined value Th, it indicates that the variance σ of the position deviation is large. That is, since the guide line candidate is not united as a guide line and the probability of the guide line candidate being a guide line is low as a result of the overall determination of the line segment, the guide line candidate B is not adopted as a genuine guide line.

For example, as shown in FIG. 3B, in the case of an analog of a guide line such as a curb, it is predicted that the guide line is irregularly distributed with respect to the prediction line L1. In this case, the variance of the deviation with respect to the average value of the position deviation described above increases. This makes it possible to accurately distinguish a genuine guide line from a noise line.

After the genuine guide line is determined in this way, step S160 in FIG. 2 is executed. In step S160, to determine the position of the next predicted line,
As described above, the parameters (e (t), c (t), W (t), W (t), θ (t) are obtained from the current position of the guide line obtained in step S150 using the equations (4) to (7). ), Φ
(T), ψ (t)) are calculated. From these parameters, the values of the parameters at the next imaging timing are predicted and calculated. This is calculated using a well-known Kalman filter. The Kalman filter is a technique for sequentially estimating the least-squares estimation of the system state from the dynamics of the system, the statistical properties of noise, and observation data.

In this embodiment, an extended Kalman filter is used. The extended Kalman filter is a method of applying a Kalman filter by linearizing nonlinear dynamics and an observation process. Next, an outline will be given.

The above set of parameters (e, θ, φ,
ψ, c, W) (hereinafter, the description of the time variable t is omitted) is defined by a six-dimensional state vector p as follows.

[0040]

P = [eθφψcW] ^T : T is a transposed matrix (8) The observation target is, for example, the distance d between the prediction line L1 and the guide line candidate shown in FIG.

In general, when there is no abrupt change in parameters, between the k-th state vector p _k and the k + 1-th state vector p _{k + 1} , and between the k-th state vector p _k and k
The following relationship is established with the distance dk.

[Equation 9] _{p k + 1 = f K (} p k) + u k (9)

D _k = h _k (p _k ) + v _k (10) where k: number of updates u _k : noise vector added to the system f _k : function expressing dynamics v _k : function of observation noise term Vector h _k : function representing the observation process

According to the teaching of the Kalman filter, if the functions f _k and h _k can be linearized and can be expressed by a determinant, the equations (9) and (10) can be solved by successively updating k. , And finally the next parameter state vector p _{k + 1} is obtained. In practice, since p _k is the independent variable six column vector, the function f _k, h _k is also expressed by the matrix equation 1 rows and six columns. E, θ, φ, ψ, c, and W representing the state vector p _{k + 1} are parameters expected next time.

In step S160, the parameter value at the time of the next imaging is predicted as described above, and based on the value, the predicted deviation d _k with respect to the previously determined guide line is calculated, and based on the predicted deviation d _k . Next forecast line L
1 is determined on the screen. Such processing is repeatedly executed. If a genuine guide line has not been determined last time, a predicted deviation d _k based on the latest guide line determined in the past is calculated. On the other hand, each time a genuine guide line is determined, the latest parameter is determined, and based on the parameter or a parameter value predicted by a Kalman filter for noise removal, a predetermined position ( As described above, the position and the attitude of the vehicle are controlled so that the attitude is along the positioning travel path at the center line). These control values are output to a control device (not shown) via the I / O interface 130, and are used for controlling the attitude of the vehicle. According to the above-mentioned method, it is possible to reliably predict and capture not only straight roads and gentle curved roads, but also curved roads with large curvatures or sudden changes in guidance lines.

While one embodiment of the present invention has been described above, various other modifications are possible. In the above embodiment, the variance of the position deviation of the guide line candidate with respect to the prediction line is obtained by the number of points on the guide line candidate existing in a predetermined range with respect to the average value of the position deviation. The variance σ of the position deviation may be directly obtained. in short,
By calculating a value related to the variance σ of the position deviation, the effect of the present invention can be achieved.

In step S153, since each position deviation has a sign, the determination is made based on the magnitude of the absolute value of the difference from the deviation average. However, the determination may be made based on the square of the difference. Also,
In the present embodiment, in step S160, all of the parameters related to the guide line and the parameters related to the camera attitude are predicted. However, when the camera attitude, that is, the attitude of the automobile is slowly changing, only the parameter related to the guide line may be used. . Although the Kalman filter is used for the calculation of the prediction line, other methods such as least square approximation may be used. In particular,
When parameters are simplified, Lagrange interpolation, spline interpolation, and least-squares approximation curve interpolation relating to the time of a guide line extracted in the past can be used.

In this embodiment, the image processing is performed by one image pickup device. However, if higher speed processing is required, another image pickup device is provided for each of the left and right guide lines. The same processing may be performed to make a comprehensive judgment. Also,
Although white lines drawn at both ends of the travel path are used as the guide lines, white lines on one side may be used. In addition, any guide line, such as a guardrail, can be used as long as it serves as an index for guiding the vehicle. In addition, various modified examples are conceivable. However, any method may be used as long as the authenticity of the guide line is determined based on the variance of the positional deviation of the guide line candidate from the predicted line, which is the gist of the present invention.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing a configuration of a traveling road recognition device according to one embodiment of the present invention.

FIG. 2 is a flowchart showing a processing procedure of a CPU of the travel path recognition device.

FIG. 3 is an explanatory diagram showing a method for determining a guide line based on a variance of a position deviation with respect to a prediction line.

FIG. 4 is an explanatory diagram showing a relationship among a road coordinate system, a camera coordinate system, and an image sensor coordinate system.

FIG. 5 is a flowchart showing a processing procedure of a CPU for determining a variance-related value of position deviation and a guide line.

FIG. 6 is an explanatory diagram showing a data structure for storing a position of a genuine guide line determined.

FIG. 7 is an explanatory diagram showing a method of obtaining a position of a prediction line using a Kalman filter.

FIG. 8 is an explanatory view showing a conventional method for determining a guide line.

[Explanation of symbols]

 100 CPU 110 ROM 120 RAM 140 System bus 150 Camera 160 Image input device 165 Frame memory L1 Prediction line L2 Average value line Ci Candidate point DW1 First window DW2 Second window

Continuation of front page (56) References JP-A-9-35198 (JP, A) JP-A-6-225308 (JP, A) JP-A-8-261756 (JP, A) JP-A-7-78234 (JP) JP-A 8-320998 (JP, A) JP-A 5-164569 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 1/00 330 B60R 21/00 624 G06T 7/60 200 G08G 1/16

Claims (1)

(57) [Claims] 1. A travel path recognition device which images a travel path by an imaging device mounted on a moving body and recognizes a guide line serving as a guide index of the mobile body from an obtained image, comprising: A guide line candidate extracting means for extracting candidate guide line candidates; a guide line obtained at the time of the previous imaging; and a traveling state of the moving object; A guide line predicting unit for predicting a position, and for an image processed by the guide line candidate extracting unit,
By setting a range of a predetermined width around the predicted line predicted by the guide line predicting means and searching the range, a guide line candidate point is extracted, and a deviation distribution of the guide line candidate point with respect to the predicted line is calculated. And a dispersion-related value calculation means for calculating a variance-related value related to the variance of the deviation distribution calculated by the deviation calculation means. If the variance-related value is equal to or less than a predetermined value, the derivation is performed. A travel line recognition device comprising: a guide line determination unit that employs a line candidate as a genuine guide line.