JPH08315125A - Recognition device for travel path section line for vehicle or the like - Google Patents

Abstract

PURPOSE: To accurately recognize a travel path section line by comparing an extracted segment with road structure models and selecting a road structure model which meets specific matching conditions, selecting a segment which matches with the road structure model, and approximating feature points for every travel section line and generating a section line. CONSTITUTION: A CCD camera 10 picks up the image of a road where the vehicle is traveling and outputs the original image to hardware 30. The hardware 30 scans the input image under the command of a CPU 36 to detect edge points and obtain an edge image, and them perform Hough transformation to detect straight line components. A CPU 36 divides the detected straight lines at their intersections into segments, judges how much the segments match with the edge image, and also judges whether each segment is a broken line or continuous line. Segments which satisfy the certain degree of matching are outputted as travel path section line candidate data together with a corresponding edge dot array in one group and sent to a memory 34 for communication. Namely, a CPU 38 reads out image results which are stored at every specific time and estimates a travel path section line.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a device for recognizing a lane marking of a vehicle, and more specifically, a device for recognizing a lane marking or a road structure of an autonomous vehicle or a vehicle equipped with a lane departure warning device. More specifically, regarding a roadway lane (lane or lane) indicating a roadway on a road, an image is taken by a camera provided inside the vehicle, and the roadway lane marking or road structure is recognized based on the image processing result. Regarding things.

[0002]

2. Description of the Related Art As a device for recognizing such a vehicle lane dividing line, as disclosed in Japanese Patent Laid-Open No. 62-139011, a vehicle lane dividing line (usually indicated by a white line) is formed in the longitudinal direction. After selecting multiple points, projectively transforming multiple points in the selected image to actual positions, find an approximate function that smoothly passes through each of the projective transformed points, and run the function curve A technology to consider it as a road marking line has been proposed.

[0003]

However, when there are a plurality of parallel running road lane markings on the road, the approximation function is individually applied to the point sequence for each running road lane marking according to the teaching of the above-mentioned prior art. , The function curves are not always parallel.

This is because there is an error in the position of the projective-converted point sequence, mainly due to distortion of the optical system of the camera, vibration of the vehicle, and the like. Therefore, in the above-mentioned conventional technique, there is a disadvantage that the road dividing lines that should originally be parallel to each other are detected not to be parallel.

Therefore, the object of the present invention is to solve the above-mentioned problems and to be affected by the distortion of the optical system of the camera, the vibration of the vehicle, etc. even when there are a plurality of parallel lane markings on the road. It is another object of the present invention to provide a vehicle lane marking line recognition device capable of accurately recognizing a lane marking line.

Further, even when there are a plurality of parallel running road lane markings on the road, it is possible to accurately recognize the running road structure without being affected by the distortion of the optical system of the camera or the vibration of the vehicle. An object of the present invention is to provide a road structure recognition device for a vehicle.

[0007]

In order to achieve the above-mentioned object, a traveling road lane marking recognition device according to the present invention comprises:
In Claim 1, a predetermined characteristic point is extracted from the image containing the traveling road surface of the vehicle imaged by the imaging means attached to the vehicle, and a plurality of line segments corresponding to the traveling road lane markings are extracted based on the extracted characteristic points. An image processing unit for extracting, a road structure model storing unit for storing a plurality of road structure models expressing a geometrical relationship between the traveling road lane markings, the plurality of extracted line segments and the plurality of road structures. A model selection unit that compares a model and a road structure model that satisfies a predetermined matching condition and a line segment that matches the selected road structure model from the extracted plurality of line segments. A function curve or a straight line on which the line segment selecting means to be selected and the characteristic points corresponding to the selected line segment are substantially parallel on the road surface for each of the plurality of road dividing lines. An approximate means for the roadway marking lines by approximating, was constructed as consisting of.

According to a second aspect of the present invention, the road structure model is stored in the road structure model storage means as data in the form of a graph consisting of nodes and arcs, and 1 is stored therein.
Each node corresponds to one lane dividing line, and has at least a line segment attribute value indicating the distinction between "solid line and broken line",
One arc represents a relationship between two nodes, and the relationship is configured to include at least "parallel relationship, connection relationship".

According to a third aspect of the present invention, a plurality of points indicating the existing positions of the lane markings are detected on the basis of an image including the traveling road surface of the vehicle, which is picked up by the image pickup means attached to the vehicle. Travel including image processing means and approximation means for obtaining a predetermined function curve or straight line approximating the detected plurality of points, and recognizing the function curve or straight line obtained by the approximation means as a travel road division line In the road lane marking recognition device, the approximating means, when there are a plurality of road lane markings, the plurality of points corresponding to the respective road lane markings are functions parallel to each other in a plane including the road lane. It was configured to approximate.

According to a fourth aspect of the present invention, there is provided a road structure recognizing device for a vehicle, wherein predetermined feature points are extracted from an image including a traveling road surface of the vehicle which is imaged by an image pickup means attached to the vehicle, and An image processing means for extracting a plurality of line segments corresponding to the running road lane markings based on a geometrical relationship between the plurality of line segments, and at least the number of running roads or lanes and the running road on which the vehicle travels. Road structure model storage means for storing a plurality of road structure models distinguished by the position of the lane, and the plurality of extracted line segments and the plurality of road structure models are compared, and the extracted line segments are extracted according to a predetermined rule. Matching evaluation value calculation means for obtaining the matching evaluation value of each model, and the present to past locations of the matching evaluation values for all or some of the plurality of road structure models A history storage means for storing a history of the time duration, and based on the history of the matching evaluation value 1
And a model selection means for selecting one road structure model.

According to a fifth aspect of the present invention, the model selecting means selects one road structure model based on a result of smoothing the history of the matching evaluation values.

[0012]

According to the first aspect of the present invention, predetermined characteristic points are extracted from the image including the traveling road surface of the vehicle imaged by the image pickup means attached to the vehicle, and a plurality of characteristic points corresponding to the traveling road lane markings are extracted based on the extracted characteristic points. Image processing means for extracting line segments, a road structure model storage means for storing a plurality of road structure models expressing a geometrical relationship between the traveling road lane markings, the extracted line segments and the A model selecting unit that compares a plurality of road structure models and selects a road structure model that satisfies a predetermined matching condition, and matches the selected road structure model from the plurality of extracted line segments. A line segment selecting means for selecting a line segment that is present, and the characteristic points corresponding to the selected line segment, for each of the plurality of travel road division lines, are substantially parallel to each other on the travel road surface. Or an approximation means for approximating with a straight line to form a road dividing line, even if there are a plurality of parallel road dividing lines on the road, distortion of the optical system of the camera or vibration of the vehicle It is possible to accurately recognize the lane markings without being affected by the above.

In the above description, "feature points" means edge points of an image or points or places having a specific texture, and "plurality of road structure models expressing geometrical relations" means running roads or lanes. It means a model expressing the road structure such as the number of.

According to a second aspect of the present invention, the road structure model is stored in the road structure model storage means as data in a graph format including nodes and arcs, and 1 is stored therein.
Each node corresponds to one lane dividing line, and has at least a line segment attribute value indicating the distinction between "solid line and broken line",
One arc represents a relationship between two nodes, and the relationship is configured to include at least "parallel relationship, connection relationship". Therefore, in addition to the action and effect described in claim 1, the road structure model The matching between the extracted line segment and the extracted line segment can be solved by reducing it to the matching problem in the graph theory, and the matching can be accurately obtained.

According to a third aspect of the present invention, a plurality of points indicating the existing positions of the lane markings are detected on the basis of an image including the traveling road surface of the vehicle, which is imaged by the image pickup means attached to the vehicle. Travel including image processing means and approximation means for obtaining a predetermined function curve or straight line approximating the detected plurality of points, and recognizing the function curve or straight line obtained by the approximation means as a travel road division line In the road lane marking recognition device, the approximating means, when there are a plurality of road lane markings, the plurality of points corresponding to the respective road lane markings are functions parallel to each other in a plane including the road lane. Since it is configured to approximate, even if there are a plurality of parallel road dividing lines on the road, the road dividing lines can be recognized accurately without being affected by the distortion of the camera optical system or the vibration of the vehicle. Door can be.

In the vehicle road structure recognizing device according to the fourth aspect, a predetermined feature point is extracted from an image including the traveling road surface of the vehicle, which is imaged by the image pickup means attached to the vehicle, and An image processing means for extracting a plurality of line segments corresponding to the running road lane markings based on a geometrical relationship between the plurality of line segments, and at least the number of running roads or lanes and the running road on which the vehicle travels. Road structure model storage means for storing a plurality of road structure models distinguished by the position of the lane, and the plurality of extracted line segments and the plurality of road structure models are compared, and the extracted line segments are extracted according to a predetermined rule. Matching evaluation value calculation means for obtaining the matching evaluation value of each model, and the present to past locations of the matching evaluation values for all or some of the plurality of road structure models A history storage means for storing a history of the time duration, and based on the history of the matching evaluation value 1
Since it is composed of a model selection means for selecting one road structure model, even if there are a plurality of parallel road dividing lines on the road, it is affected by distortion of the optical system of the camera and vibration of the vehicle. Without it, it is possible to accurately recognize the road structure on which the vehicle is traveling.

According to a fifth aspect of the present invention, the model selecting means is configured to select one road structure model based on the result of smoothing the history of the matching evaluation values. In addition to the effect described above, it is possible to more accurately recognize the road structure during traveling.

[0018]

Embodiments of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is an explanatory perspective view generally showing a vehicle equipped with a recognition device for lane markings according to the present invention.

In the figure, the vehicle is equipped with one CCD camera (monochrome TV camera) 10 as the above-mentioned image pickup means. The CCD camera 10 is fixed to the rear-view mirror mounting position above the driver's seat, and allows a monocular view of the vehicle traveling direction. Reference numeral 12 denotes a radar unit including a millimeter wave radar, which includes two front radars mounted in front of the vehicle, and detects the presence of a three-dimensional obstacle such as another vehicle through reflected waves.
A yaw rate sensor 14 is provided near the center of the vehicle compartment to detect angular acceleration about the vertical axis (z axis) of the vehicle.

Further, a vehicle speed sensor 16 including a reed switch is provided near a drive shaft (not shown) of the vehicle.
Is provided to detect the traveling speed of the vehicle, and the steering angle sensor 18 is provided near the steering shaft 20 of the vehicle to detect the steering steering angle.

A steering angle control motor 22 is attached to the steering shaft 20, a throttle actuator 24 composed of a pulse motor is attached to a throttle valve (not shown), and a brake (not shown) is further provided. A brake pressure actuator 26 (not shown in Figure 1) is attached. In this configuration, the vehicle is steered in accordance with the calculated steering angle control amount,
The throttle valve is opened and closed to adjust the vehicle speed, and the brakes are activated as necessary to drive the vehicle.

FIG. 2 is a block diagram showing the above structure in more detail. The output of the CCD camera 10 is sent to the image processing hardware 30, where the linear components are extracted by the detection of edge points and the Hough transform under the instruction of the image processing CPU 36, and the result is passed through the bus 32. Stored in the communication memory 34. Where "edge point"
As is well known, generally means a point in the image where the pixel density greatly changes compared to the surroundings, and a process corresponding to differentiation is performed on the input image to indicate that the differentiation equivalent value (absolute value) is large to some extent. Set as an edge point. The image processing CPU 36 reads the stored value at every predetermined time and determines the travel road lane marking.

On the other hand, the output of the radar unit 12 is transmitted via the radar processing circuit 40 and the bus 32 to the communication memory 34.
Will be stored in. The radar evaluation CPU 42 reads the stored value at every predetermined time and detects the position of the obstacle on the coordinates. The outputs of the vehicle speed sensor 16 and the like are output from the trajectory estimation CPU 4
4 and the movement locus of the own vehicle is estimated. The action plan decision making CPU 50 creates a target route from the store value. The locus of movement of the vehicle estimated as the target route is sent to the locus tracking control CPU 46, where the locus (target route)
The tracking control amount is determined.

Further, the trajectory tracking control CPU 46 calculates a steering angle control amount and outputs it to the steering angle control CPU 52. The steering angle control CPU 52 drives the steering angle control motor 22 via the PWM controller 54 and the driver 56.
The motor drive amount is detected by the encoder 58,
Feedback control is performed.

Further, the action plan decision-making CPU 50 obtains the target acceleration of the vehicle body by its speed / following control unit, and the vehicle speed control C
It is sent to the PU 60. The vehicle speed control CPU 60 drives the throttle actuator 24 via the accelerator pulse motor controller 62 and the driver 64, and drives the brake pressure actuator 26 via the brake solenoid controller 66 and the driver 68. The driving amount is detected via the pressure sensor 70, and the second feedback control is performed.

In the above description, the processing circuit of the sensor such as the waveform shaping circuit is omitted for simplification of illustration.

FIG. 3 functionally shows the block diagram of FIG. FIG. 4 is a block diagram showing a detailed configuration of the image processing hardware of FIG. As shown in the figure, the image processing hardware 30 specifically includes the image input digitizing hardware 30a and the real-time edge detection hardware 30.
b, Hough conversion hardware 30c and image data bus controller 30d.

Next, the operation of the device for recognizing the traveling road lane markings according to this application will be described.

FIG. 5 is a main flow chart (PAD diagram (structured flow chart)) showing the operation. Before the description of FIG. 5, the operation of this apparatus will be described with reference to FIG. Outline. FIG. 6 is a block diagram functionally explaining it.

As described above, the CCD camera 10 captures an image of a road on which the vehicle is running and outputs the original image thus obtained to the image processing hardware 30. The image processing hardware 30 scans the input image under the instruction of the image processing CPU 36, detects edge points to obtain edge images, and then performs Hough transform to detect linear components. They are shown in FIG. The image processing hardware 30 stores the detected value in the communication memory 34 via the bus 32.

The image processing CPU 36 reads the stored value and divides the detection straight line at the intersections thereof to form a line segment. Then, the degree of matching with the edge image is determined for each line segment, and whether the line segment is a broken line (a lane dividing line formed by a broken line drawn on the road) or a continuous line (a running line formed by a solid line drawn on the road) It is determined whether or not it is a road division line. Regarding this determination method, for example, the applicant of the present invention, Japanese Patent Laid-Open No. 3-1589
The one proposed in No. 76 is used.

Then, a line segment satisfying a certain degree of coincidence is paired with the corresponding edge point sequence data, and is output as traveling road lane line candidate data. The output result is sent to the communication memory 34 via the bus 32 and stored there. FIG. 8 shows these processes in the image processing CPU 36.

The operation of the apparatus according to the present invention is performed by the image evaluation CPU 38 in FIG.
As described above, U38 reads out the image processing result (running road lane marking candidate data) stored at every predetermined time, and
The processing shown in (1) is performed to estimate (recognize) the lane marking.

In other words, the stored value may include a line segment that is not related to the travel road division line, or a necessary travel road division line may be missing. In addition, noise components due to the algorithm or the vibration of the vehicle body may be included as the position error of the point sequence. The image evaluation CPU 38 extracts a correct lane marking from the stored value and estimates the true position of the lane marking. The estimated value is fed back to the image processing CPU 36 via the communication memory 34.

Here, the image evaluation CPU 38 previously stores (stores) a road structure model as shown in FIG. 9, a model of 1 lane, 2 lanes, and 3 lanes in the illustrated example, as data in a memory 39 provided side by side. )are doing. More specifically, the road structure model is stored (stored) in the form of a graph in which the elements of the road dividing line (solid line or broken line) are represented by nodes, and the relationship between the nodes is represented by arcs (links). It

The image evaluation CPU 38 converts (symbolizes) the line segment data into a similar graph consisting of nodes and arcs from the read roadway division line candidate data (image processing result). (Referred to as "graph"), and compared with the model (matching judgment) to calculate the degree of matching with the model (matching evaluation value).

Specifically, the comparison is made with a model using graph theory. It should be noted that, since the judgment is erroneous only by the matching result at one time, the comparison results are set as matching solutions in descending order of evaluation value in time series for the past predetermined time (variable according to vehicle speed).
Remember.

When storing, the point sequence data is classified and stored according to the type of model together with the line segment data. Here, the type is defined by the number of lanes (lanes) provided in the model such as two lanes, one lane, and the own vehicle lane (lane).

The classification is performed by determining the continuity between the newly input point sequence data and the stored past point sequence data group (for a predetermined time). Further, upon classification, the newly input point sequence data is subjected to position correction based on the motion trajectory, that is, the origin is aligned with the past point sequence data group.

Subsequently, the evaluation value (of the degree of coincidence) is smoothed for each of the time-series lane markings thus classified, and the candidate having the maximum evaluation value at the current time is selected as the "most probable value". To do. Then, the point sequence data group indicating the lane markings is approximated by a function curve (or straight line).

More specifically, a group of point data representing a plurality of lane markings are approximated by parallel function curves (or straight lines). Then, the traveling road lane marking data obtained by approximation is output as point sequence data. The amount of data to be output differs depending on the vehicle speed.

The apparatus according to the present application will be outlined as above, and will be described in detail below by returning to the flow chart of FIG.
The program shown in the figure is started every predetermined time T.
In this specification, the current time is Tn as described later.

First, in step S10, a straight line component is detected from the road image by image processing, and then a line segment and an edge point sequence which are candidates for the road dividing line are extracted, and their coordinate axes are so-called focal lengths and mounting positions of the camera. Inverse perspective transformation is performed from the camera viewpoint coordinate system to the real plane coordinate system based on the camera parameters.

The "real plane coordinate system" referred to here is a two-dimensional coordinate system in which the host vehicle is the origin and placed on the road plane. The processes performed in S10 are all described in Japanese Patent Application No. 6-182940 previously proposed by the present applicant, and are not the gist of the present invention, and therefore detailed description thereof will be omitted.

Subsequently, the process proceeds to S12, and the road structure model stored in the memory 39 as described above is compared to obtain a running road lane line candidate having a high degree of matching. Specifically, the matching between the road structure model and the input is solved by reducing it to a matching problem in graph theory. That is, it is performed by abstracting the image processing result and expressing it as a graph, and extracting a partial isomorphic graph between the model and the model which is also expressed by the graph.

FIG. 10 is a subroutine flow chart showing the work.

First, in S100, an input graph is created in which the candidate road segment lines are "nodes" and the relationship between them is "arc". The graph is represented by a matrix.

That is, when based on the above-mentioned graph theory,
As shown in FIG. 11, a graph G can be generally expressed as G = (V, E) by using a node V and an arc E representing the property between the nodes. The node V has an attribute and an attribute value P.

When the basic unit forming a road and a possible continuous running line dividing line are made to correspond to the node V in the graph, the road structure graph G shows a solid line and a long broken line ( It has attributes P such as a center line) and a short broken line (such as a highway branch entrance).

The arc stretched between the nodes can be expressed by the positional relationship E between the running road dividing lines such as parallel and connection. That is, P = (p, value) pε {solid line, long broken line, short broken line, unknown} E = (e, value) eε {parallel relation, connection relation, irrelevant} where value is p or e It is a value.

Based on the above, the operation will be described with reference to the flowchart of the subroutine in FIG.
In 200 and S202, Vi, Vj (i ≠ j) for a plurality of candidate road lane dividing line segments (nodes) Vi, Vj
Positional relationship E of Vj, that is, e and (value) described above
And substitute for E.

FIG. 13 is a subroutine flow chart showing the work. First, in S300, the distances L1, L2, L3, L4 between the end points of the plurality of candidate road segment line segments Vi, Vj (nodes) are calculated. Ask (Fig. 14).

Next, in S302, it is determined whether or not any of the obtained distances L1 to L4 is smaller than a threshold value (suitably set) for determining that the nodes are connected to each other. Then, it is determined that the candidate road segments Vi and Vj have a connection relationship, that is, they are connected, and the connection relationship scale is obtained as value. FIG. 15 shows the connection relation scale.

As shown in the figure, when it is determined that the two running road lane marking candidate line segments Vi and Vj are connected to each other at one end point, the connection relation scale is an angle of the connection direction of Vj viewed from Vi. Calculate with θ. Specifically, Vi is centered around the connection point.
Angle V from 0 to Vj measured counterclockwise (0 ≦ θ <
2π) is divided into eight and symbolized. Then S3
Proceeding to 06, the value of E is set from the above and the program is terminated.

On the other hand, if it is determined in S302 that none of the distances L1 to L4 is smaller than the connection determination threshold value, the process proceeds to S308, and the candidate road segments Vi, Vj for the road dividing line are selected.
Widths L5 and L6 are obtained at two appropriate places between them. FIG. 16 shows the work.

Specifically, the candidate road segment Vi, V
By using some of the points constituting j, the straight lines i ′ and j ′ are approximated, and the widths L5 and L6 are set at two positions in an appropriate range of the approximate straight line.
To calculate. Specifically, at appropriate two locations, distances l1, l2, l
3,14 is calculated, and the average value thereof is calculated to obtain widths L5, L6.

Next, in S310, the obtained width L5 is calculated.
It is determined whether or not the ratio of L6 is equal to or larger than a (width threshold value, appropriately set) and is 1 or less (provided that L5 <L6). When the result is affirmative, the process proceeds to S312, where the candidate road segment V
i and Vj are judged to have a parallel relationship, and a parallel relationship scale (va
lue).

Specifically, as shown in FIG. 17, the width L5,
L6 is divided by a predetermined lane width Wo to obtain a quotient (for example, 0.
5) is a parallel relationship scale. Incidentally, as shown in the figure, the quotient is given a positive or negative sign depending on the direction from the reference position (for example, the left end travel road dividing line). Next, in S314, the value thus obtained is set to E. When the result in S310 is negative, S3
Proceed to step 16 and set e = irrelevant and value = none.

Returning to the flow chart of FIG. 12, the program proceeds to S204 and S206, and the value E obtained as above is calculated.
And its negative value −E (inverse relation) are represented by the matrix R [i,
j].

Returning to the flow chart of FIG. 10, the process then proceeds to S102, in which expected road structure models are narrowed down prior to the model matching performed in S104 and subsequent steps. That is, one or more (q) corresponding models are selected from a plurality of (k) models such as one lane and two lanes. This process has a smaller amount of calculation than the model matching process described later, and by selecting a model to be matched in advance in this way, high-speed processing can be realized even on relatively low-performance on-vehicle hardware.

FIG. 18 is a subroutine flow chart showing the work.

Explaining below, first, S400 and S4
In 02, an association graph is created from the input graph (input value) and the model for the i-th model of the k road structure models. This is because, as described above, the problem of whether or not the input value and the model are matched is reduced to the clique extraction problem in the graph theory and solved.

Here, two graphs G1 (V1, E
1), G2 (V2, E2) to one union graph G0
Theoretically, (V0, E0) is created as follows. I. When the attributes p1 and p2 satisfy p1 = p2 for v1εV1 and v2εV2, a node v0εV0 having a label (v1, v2) is provided. B. v0 = (v1, v2), v0 '= (v1', v
2 ′) εV0, E1 (v, v1 ′) = E2 (v
2, v2 ′) holds, v0 and v0 ′ are connected.

FIG. 19 is a subroutine flow chart showing the association graph preparation work.

Explaining below, first, S500 to S5
In 04, if the number of running road dividing lines in the model is m and the number of line segments in the input graph is n (S500), the attributes of m and n (broken line, Solid line) can be combined, that is, with the same kind of line,
Further, it is determined whether or not the positional relationship with respect to the own vehicle is the same, for example, both are on the left side (S502). When the result is affirmative, (m, n) is added to the combination set C (S50
4). Here, C represents a node of the association graph.

Next, proceeding to S506 to S512, for Ci and Cj (i ≠ j) in the combination set C (S506), the relationship between mi and mj and the relationship between ni and nj,
That is, it is determined whether the i-th and j-th relationships of the model line segment m are the same as the i-th and j-th relationship of the input line segment (S
508), when affirmative, A [i, j] is set to 1 (S510), and when negative, it is set to 0 (S51).
2). Where A is the existence of an arc between the nodes of the associative graph (=
1) Indicates nothing (= 0).

Returning to the flow chart of FIG. 18, the process proceeds to S404, in which a matching expected evaluation value is calculated.

FIG. 20 is a subroutine flow chart showing the work.

First, in S600, the total number of arcs in the association graph (the total number of A [i, j] = 1 described above) is set to S1,
Proceeding to S602, the total number of points due to the correspondence between the type of line segment in the input graph and the type of model road dividing line is set as S2. Here, the score is determined as shown in FIG. That is, if both the model and the input value are solid lines, the score is 1.0.
And Next, in S604, the quotient obtained by dividing the sum of the obtained values S1 and S2 by the number of nodes in the association graph is set as the matching expected evaluation value. This is an operation for obtaining an expected value of the size of a "clique" (described later) extracted from the association graph.

The matching is solved by reducing the problem of extracting a clique from the association graph of the input graph and the model graph. At this time, the larger the number of arcs per node of the association graph, the larger the clique may be extracted. Is high,
Therefore, since such a model graph is likely to match the input graph well, the number of arcs per node in the association graph is used as the predicted evaluation value. Actually, as described above, when the line segment attributes of the model and the input match, the predicted evaluation value is further increased.

Next, returning to the flow chart of FIG. 18, S
Proceeding to 406, a model having the highest matching prediction evaluation value is selected (q pieces).

Next, returning to the flow chart of FIG. 10, S
By proceeding to steps 104 and thereafter, and extracting cliques from the association graph, matching solution candidates for the p models that have been narrowed down and the input graph are obtained.

Here, the clique will be described.

The clique is as indicated by the thick line in FIG.
It is a fully connected subgraph whose nodes are arced to every other node. Extracting a clique is nothing but extracting a subgraph of the same shape from two (or three or more) graphs.

That is, as shown in FIG. 23, the series of operations for creating the association graph and finding the clique is to find the part that matches the model and the input value (image processing result), in other words, the model and the input value. It means the work of finding the degree of matching with.

The extraction of this clique is specifically as follows.
`` Dana H. Ballad, Christopher M. Brown, COMPUTER
VISION (PRENTICE HALL), Ch.11 Matching Sec. 11.3
Implementing Graph-Theoretic Algorithm ”. The basic algorithm is shown in the lower part of FIG.

Matching solution candidates are obtained by extracting the cliques in this way. Here, it is assumed that two or more (m) matching solution candidates are obtained. In other words, the larger the number of running road lane markings (input values) that match the model, the better the solution is not always because, for example, how much the width of one lane corresponds to the model is not taken into consideration.

Therefore, only the maximum clique is not extracted as the only matching solution, and for example, a clique with one node less than the maximum clique is left as a solution, and the best matching cue is calculated based on the "matching evaluation value" described below. Determine the solution.

Then, the process proceeds to S110, in which the matching evaluation value is calculated for m matching solution candidates. Figure 2
The calculation method of the matching evaluation value is shown in the lower part of 3. The matching evaluation value is composed of a “unity attribute matching degree” and a “binary attribute matching degree”, and these two matching degrees are defined as follows.

That is, the degree of coincidence of unary attributes indicates how similar the attributes of the corresponding dividing lines are between the input and the model, and the degree of similarity between the nodes of the input graph and the corresponding nodes of the model graph. By comparing the attributes (types of dividing lines), FIG.
I tried to add points as shown in the table inside. In addition, the degree of matching of the binary attribute indicates how parallel the input line segments are if they are parallel to each other, and how linearly they are connected if they are connected. The values of the "parallel relation scale" and the "connection relation scale" described above are added.

Returning to the main flow chart of FIG.
Then, the process proceeds to S14, and the position of the road structure and the running road lane markings currently running is estimated based on the past matching result.

FIG. 24 is a subroutine flow chart showing the work.

Explaining below, first, in S700, the point sequence data of the past traveling road lane markings stored in (or) traveling road structure candidates 1 to n are updated to the current time Tn (described later), and the routine proceeds to S702. Then, the matching solution candidates 1 to m at the current time Tn are classified into any of the traveling road structure candidates 1 to n and are written in the ring buffer.

That is, in order to classify matching solution candidates (m pieces) into past traveling road structure candidates (n pieces), first, the continuity with each road structure candidate is obtained, and the road structure with the largest continuity is obtained. Classify as candidates.

To explain from the background, in the device according to the application, in order to estimate the position of the road structure and the road dividing line which are currently running, as shown in FIG. A table relating to traveling road lane marking position information is created in which the history is recorded together with the positional information of the traveling road lane markings. The above corresponds to the work shown as classification in the block diagram of FIG.

Specifically, for n traveling road structure candidates, the number of the road structure model representing the structure, the number of each traveling road division line defined by the model, and the traveling road division line are shown. It is composed of an approximation function, a matching evaluation value from the current time Tn to N times before, and a ring buffer that stores the roadway section line position matrix element in which the information of the point sequence coordinates is compressed. The value of N is variable depending on the traveling vehicle speed (specifically, the higher the vehicle speed, the smaller the value).

The matrix elements Tn-Nm to Tn for each time are time-corrected at the same time using the trajectory estimated value as will be described later (corresponding to position correction by the motion trajectory shown in FIG. 6).

As described above, S in the flow chart of FIG.
In 702, in order to classify each of the m matching results into the past running road structure candidates 1 to n, the continuity with each candidate is first calculated, and one matching result has the maximum continuity. Select a candidate.

The continuity is calculated mainly from two elements.
The first is the structural continuity, which is performed by judging the degree of continuity of the road structure. Specifically, this is done by quoting the value described in the road structure model. More specifically, if the model is one lane, it is determined that the input value is continuous when the input value is one lane.

The second is the degree of continuity of the point sequence, which is obtained by comparing the roadway division line function estimated in advance from the past point sequence information and the point sequence information of the current roadway division line for the road structure candidate in advance. To do. Specifically, the point sequence information of the running road lane markings estimated up to the previous time is subjected to minimum approximation with a predetermined correction function straight line to obtain the variance.

That is, it is performed by comparing the root mean square of the distance of the point sequence around the function and the variance up to the previous time. It is shown in FIG. Note that the approximation here is used only for determining the continuity of the point sequence, unlike approximation by a function curve described later. Since high precision is not required here,
Use a linear approximation that requires less calculation.

FIG. 27 shows the matching result obtained as described above.

Then, in S704, the evaluation values (time Tn-m to Tn) of the traveling road structure candidates 1 to n are smoothed,
The road structure candidate having the maximum evaluation value is determined as the traveling road structure. For example, when traveling on a lane road as shown in FIG. 28, the evaluation value may be different from one lane in some cases and two lanes in other cases. There is a fear.

Therefore, as shown in FIG. 29, after smoothing the matching evaluation values that change over time, the road structure (model) having the maximum evaluation value at the current time Tn is selected (determined). did. In the case shown,
Of the 1-lane model and the 2-lane model, the 2-lane model having the larger value is selected (determined) as the current road structure.
At this time, the selection (decision) is started only when the difference in evaluation value from other candidates becomes larger than a predetermined threshold value.

In the case shown in the figure, the evaluation value of the two-lane model is slightly higher than that of the one-lane model at time T1, but it is not judged that the vehicle is in the two-lane at this point.
When the difference between the evaluation values exceeds the threshold value at time T2, it is determined to be two lanes. This is done to prevent the road structure determination from being frequently switched due to a temporary decrease in the evaluation value due to a erroneous detection of a lane marking due to image processing. The smoothing is performed to remove noise.

Then, in S706, all the point sequence data of the time Tn-m to Tn of the determined road structure are approximated by mutually parallel function curves.

The approximation using this function curve will be described. The point sequence data (Xij, Yij) on each traveling road lane marking is data for a predetermined time (i: traveling road lane marking number (1 ... N), j: Point sequence data number (1 ... ni) of the road dividing line i. A quadratic function (curve) formed by translating these N running line markings parallel to each other on a plane coordinate fi = ax ² + bx + ci (ci: number of currently recognized running line markings) Then, if the error between each function and the original point sequence (Xij, Yij) is ei, then there is nothing but to find a, b, ci that minimizes e = Σei.

When e is obtained, the result is as shown in formula 1, and thus as shown in formula 2.

[0100]

[Equation 1]

[0101]

[Equation 2]

Therefore, to minimize e
You can solve the normal equation shown in.

[0103]

(Equation 3)

[0104]

[Equation 4]

From the above, the simultaneous linear multiple (N +
You can solve the binary equation. This is done by using, for example, the "Gauss's barefoot method".

[0106]

(Equation 5)

At this time, if the coefficient matrix of the normal equation AiX = Bi for each running road lane marking shown in Equation 6 is used, the coefficient matrix of the simultaneous equations can be written as Equation 7.

[0108]

(Equation 6)

[0109]

(Equation 7)

Here, time correction of the normal matrix elements (see FIG. 24) is performed.
The operation of S700 of the flow chart corresponds to that). The point sequence coordinates from the image processing CPU 36 are different because they are output in a coordinate system whose origin is the camera installation position of the vehicle at the time when the image is captured. In order to collect the time point sequence and apply it to the quadratic curve, the coordinate system including the time point sequence at each time must be corrected to the same time coordinate system according to the movement of the vehicle.

It is appropriate that this correction is always adjusted to the latest image processing result. On the other hand, as described above, since the point sequence at each time is stored in the form of a normal matrix, this matrix element must be directly time corrected.

In general, two xy coordinate systems can be written as in Equation 8 using rotation and translation.

[0113]

(Equation 8)

By rewriting each matrix element of the normal matrix shown in Expression 9 using Expression 8, it becomes as shown in Expression 10.

[0115]

[Equation 9]

[0116]

[Equation 10]

In addition to the elements of the normal matrix, a new Σwixi
³ yi, Σwixi ² yi ² , Σwixiyi ³ , Σwixiyi ² , Σwixiyi ² , Σ
Wiyi ² , Σwiyi ³ , and Σwiyi ⁴ are added to calculate 15 elements, and the time is corrected for each of them, whereby the time of each element of the normal matrix can be corrected.

Summarizing the above, the matrix operation as shown in FIG. 31 is performed.

The time correction will be slightly supplemented.

FIG. 32 shows a coordinate system O having the start time of the trajectory estimation CPU 44 as an absolute origin and a coordinate system at two different times. As can be seen from the figure, the output value from the trajectory estimation CPU 44 is actually shown. When using (t, x, y, θ), let the uncorrected trajectory estimate be (ts, xs, ys, θs) and the corrected trajectory estimate be (td, xd, yd, θd). , The point on the coordinate system before correction (Xs, Ys) is the point on the coordinate system after correction (Xd, Yd)
In order to convert into, it is necessary to use the formula shown in Formula 11.

[0121]

[Equation 11]

Therefore, when the matrix shown in FIG. 31 is used, it is necessary to use dx 'and dy' in Eq.

[0123]

(Equation 12)

In S706 of the flow chart of FIG. 24, the point sequence is regenerated according to the approximate curve obtained as described above.

Subsequently, the flow proceeds to S708, and the regenerated travel road lane line information is written in the communication memory 34 and output.
At this time, as described above, the traveling road lane marking information is output to the front by a distance corresponding to the vehicle speed. In other words, when the vehicle speed is high, the information on the lane markings to the farther distance is output.

Subsequently, the procedure proceeds to S710, in which the data at the oldest time is deleted from the ring buffers of the traveling road structure candidates 1 to n shown in FIG.

Since this embodiment is constructed as described above, when there are a plurality of parallel running road lane markings on the road, each running road lane marking can be recognized accurately, and the road structure can be recognized accurately. can do. FIG. 33 is a data diagram showing a simulation result obtained by comparing the function approximating method according to the application with a conventional technique, and FIG. 33 (a) shows a case where each dividing line of the conventional technique is independently approximated. b) shows a case where each dividing line, which is the technique according to the application, is approximated by a parallel function.

As is clear from the figure, approximate values that are substantially parallel to each other can be obtained by approximating functions that are parallel to each other on the traveling road surface, more accurately, on a plane including the traveling road surface. Accurate recognition of lines or road structures.

The relation with the main claims is as follows. Regarding claim 1, S in the flow chart of FIG.
10 is image processing means, memory 39 in the block diagram of FIG. 2 is road structure model storage means, and S1 of the flow chart of FIG.
The model selection means 04 and subsequent steps correspond to the model selection means, S700 to S704 of the flow chart of FIG. 24 correspond to the line segment selection means, and S706 of the flow chart of FIG. 24 corresponds to the approximation means.

For claim 3, S10 of the flow chart of FIG. 5 corresponds to the image processing means, and S706 of the flow chart of FIG. 24 corresponds to the approximation means. For claim 4, S10 of the flow chart of FIG. Is an image processing means, the memory 39 in the block diagram of FIG. 2 is a road structure model storage means,
The matching evaluation value calculation means is from S104 onward in the flow chart of FIG. 10, the history storage means is from S700 to S704 of the flow chart of FIG. 24, and the model selection means is from S706 of the flow chart of FIG.

In the above description, a function curve obtained by moving in parallel on the plane coordinates is used as the approximate function, but strictly speaking, in order to express parallel running road lane markings, it is shown in FIG. It should be expressed by a function such that each runway dividing line becomes a concentric circle or a concentric ellipse, not a function translated in the y-axis direction.

However, if the curve radius as shown in FIG. 34 is larger than 100R, it can be said that there is no problem in accuracy even if the arc is approximated by a quadratic function. Conversely, 1
Even when traveling on a road having a curve radius of 00R or less, a quadratic function approximation is sufficient when the line-of-sight distance is 50 m.

Furthermore, in many cases, the curve of the road is composed of an arc and a relaxation curve (clothoid curve (quadratic function)) in front of it, so it is best to use both in combination in terms of accuracy. Is. In some cases,
A straight line may be used, or a higher-order function or the like may be used. Further, it may be appropriately selected from the processing speed, the shape of the traveling road (the degree of curvature of the curved road), the vehicle speed, and the like.

Furthermore, although the visual sensor is monocular in the above description, the front radar and the like may be omitted by using distance measurement by binocular vision. Further, although the radar unit 12 is provided only on the front side of the vehicle, it may be provided on the rear side to recognize the lane behind the vehicle. Further, in the above description, the traveling route is also referred to as “road”, “lane”, “traveling lane”, etc., but it goes without saying that the same route is referred to.

[0135]

According to the first aspect of the present invention, even when there are a plurality of parallel running road marking lines on the road, the running road can be accurately run without being affected by the distortion of the optical system of the camera or the vibration of the vehicle. The dividing line can be recognized.

According to the second aspect, in addition to the action and effect described in the first aspect, the matching between the road structure model and the extracted line segment can be reduced to a matching problem in the graph theory and solved. It is possible to accurately determine whether or not there is a match.

According to the third aspect of the present invention, even if there are a plurality of parallel lane markings on the road, the lane can be accurately traveled without being affected by the distortion of the optical system of the camera or the vibration of the vehicle. The dividing line can be recognized.

In the vehicle road structure recognizing apparatus according to the fourth aspect, even if there are a plurality of parallel running road lane markings on the road, the influence of distortion of the optical system of the camera, vibration of the vehicle, etc. It is possible to accurately recognize the road structure on which the vehicle is traveling without receiving it.

According to the fifth aspect, in addition to the effects described above, it is possible to more accurately recognize the traveling road structure.

[Brief description of drawings]

FIG. 1 is an explanatory perspective view generally showing a vehicle equipped with a recognition device for lane markings and the like according to the present invention.

FIG. 2 is a block diagram showing in detail the sensor shown in FIG. 1 and its processing.

3 is a functional block diagram of the configuration of the block diagram of FIG. 2;
It is an explanatory view similar to.

FIG. 4 is a block diagram showing details of image processing hardware of the block diagram of FIG. 2;

FIG. 5 is a main flow chart showing an operation of a recognition device for a lane marking such as a traveling road according to the present application.

FIG. 6 is a block diagram functionally showing the operation of the flow chart of FIG.

FIG. 7 is an explanatory diagram showing a work of extracting a line segment and a sequence of edge points in the flow chart of FIG. 5;

FIG. 8 is an explanatory diagram showing a process of an image processing CPU of FIG.

FIG. 9 is stored (stored) in the memory of the image processing CPU of FIG.
It is explanatory drawing which shows the example of the constructed road structure model.

10 is a subroutine flow chart showing the operation of comparing the flow chart of FIG. 5 with a road structure model and obtaining a candidate roadway lane marking with a high degree of matching.

11 is an explanatory diagram showing an input graph of the flow chart of FIG.

FIG. 12 is a subroutine flow chart showing the input graph creation work of the flow chart of FIG. 10;

13 is a subroutine flow chart showing the input graph creation work of the flow chart of FIG. 12 in more detail.

FIG. 14 is an explanatory diagram showing calculation of a distance for determining the connection relationship in the flow chart of FIG.

FIG. 15 is an explanatory diagram showing a connection relation scale of the flow chart of FIG. 13;

16 is an explanatory diagram showing calculation of a distance for determining the parallel relationship in the flow chart of FIG.

FIG. 17 is an explanatory diagram showing a parallel relation scale of the flow chart of FIG. 13;

FIG. 18 is a subroutine flow chart showing the work of creating the association graph of the flow chart of FIG. 10;

FIG. 19 is a subroutine flow chart showing the associating graph creation work of the flow chart of FIG. 18 in more detail.

FIG. 20 is a subroutine flow chart showing a task of calculating a matching expected evaluation value of the flow chart of FIG. 18;

FIG. 21 is an explanatory diagram showing the scores in the calculation of the matching expected evaluation value in the flow chart of FIG. 20.

22 is an explanatory diagram showing clique extraction work in the flow chart of FIG. 10; FIG.

FIG. 23 is an explanatory diagram showing a task of calculating a matching evaluation value in the flow chart of FIG. 10;

FIG. 24 is a subroutine flow chart showing an operation of estimating the traveling road structure and the position of the lane markings in the flow chart of FIG. 5;

FIG. 25 is an explanatory diagram showing a table of traveling road lane marking position information used for classification work of the flow chart of FIG. 24;

FIG. 26 is an explanatory diagram showing a method of calculating the continuity required in the classification work of the flow chart of FIG. 24;

27 is an explanatory diagram showing a matching result of the flow chart of FIG. 24. FIG.

28 is an explanatory diagram illustrating a matching evaluation value of the flow chart of FIG. 24. FIG.

FIG. 29 is a timing chart showing a change over time in the matching evaluation value of the flow chart of FIG. 24.

FIG. 30 is an explanatory diagram showing function approximation of the flow chart of FIG. 24.

FIG. 31 is an explanatory diagram showing a time correction calculation matrix in the function approximation work of the flow chart of FIG. 24;

FIG. 32 is an explanatory diagram illustrating a time correction operation of the flow chart of FIG. 24.

[Fig. 33] Fig. 33 is an explanatory diagram showing the conventional technique and the technique according to the application in comparison.

34 is an explanatory diagram showing characteristics of a function for approximation of the flow chart of FIG. 24. FIG.

[Explanation of symbols]

 10 CCD camera (monochrome TV camera) 12 Radar unit 14 Yaw rate sensor 16 Vehicle speed sensor 18 Steering angle sensor 30 Image processing hardware 36 Image processing CPU 38 Image evaluation CPU

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location // G05D 1/02 9061-5H G06F 15/70 460E

Claims (5)

[Claims] 1. A. An image processing means for extracting a predetermined feature point from an image including the traveling road surface of the vehicle, which is imaged by the imaging means attached to the vehicle, and for extracting a plurality of line segments corresponding to the traveling road lane markings based on the characteristic point; B. A road structure model storage means for storing a plurality of road structure models expressing the geometrical relationship of the travel lane markings; c. Model selecting means for comparing the extracted line segments with the road structure models and selecting a road structure model satisfying a predetermined matching condition; d. Line segment selecting means for selecting a line segment that matches the selected road structure model from the plurality of extracted line segments; and e. Approximating means for approximating the characteristic points corresponding to the selected line segment by a function curve or a straight line that is substantially parallel on the traveling road surface for each of the plurality of traveling road lane markings, as a traveling road lane marking. And a vehicle lane marking line recognition device. 2. The road structure model is stored in the road structure model storage means as data in the form of a graph consisting of nodes and arcs, wherein one node corresponds to one traveling road lane marking and at least "Solid line, broken line"
2. A line segment attribute value indicating the distinction between two nodes, and one arc indicates a relationship between two nodes, and the relationship includes at least "parallel relationship, connection relationship". Vehicle lane marking recognition device. 3. A. Image processing means for detecting a plurality of points indicating the existing position of the lane marking based on an image including the road surface of the vehicle, which is imaged by the imaging means attached to the vehicle; b. And a means for approximating a predetermined function curve or straight line approximating the detected plurality of points, and recognizing the function curve or straight line obtained by the approximating means as a lane marking. Wherein the approximating means approximates the plurality of points corresponding to the respective traveling road lane markings to functions parallel to each other in a plane including the traveling road surface when the traveling road lane markings are plural. A vehicle lane marking recognition device. 4. A method according to claim 1, wherein An image processing means for extracting a predetermined feature point from an image including the traveling road surface of the vehicle, which is imaged by the imaging means attached to the vehicle, and for extracting a plurality of line segments corresponding to the traveling road lane markings based on the characteristic point; B. Road structure model storage means for storing a plurality of road structure models represented by a geometrical relationship of a plurality of line segments and distinguished by at least the number of roads or lanes and the positions of the roads or lanes on which the vehicle travels; , C. Matching evaluation value calculation means for comparing the extracted line segments with the road structure models and obtaining matching evaluation values for the extracted line segments and each model according to a predetermined rule; d. History storage means for storing a history of the matching evaluation values for a predetermined time in the past for all or some of the plurality of road structure models; and e. A road structure recognition device for a vehicle, comprising: a model selection unit that selects one road structure model based on a history of the matching evaluation values. 5. The model selection means sets 1 based on a result of smoothing the history of the matching evaluation values.
5. The road structure recognition device for a vehicle according to claim 4, wherein one road structure model is selected.