JPH09223218A - Method and device for detecting traveling route - Google Patents

Abstract

PROBLEM TO BE SOLVED: To effectively detect a white line (lane marker) on a road. SOLUTION: A picture signal obtained by a camera 10 is converted into digital data by a conversion part 12. A part of the digital data is segmented as an window and processed by a neural network processing part 16. The processing part 16 has learned the shapes of various white lines and outputs the detection of a white line and its inclination value. Thereby the window can be moved along the white line by determining the position of an window processing part 14 based on the detection result of the processing part 16. The white line on the whole picture can be detected by the setting of a suitable window and the white line can be quickly and surely detected. Especially an almost erased white line, a white line like a broken line, etc., can be also surely detected by learning then.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to the detection of lane markers such as white lines formed on the road surface of a track, and more particularly to the use of neural networks.

[0002]

2. Description of the Related Art In order to realize automatic driving (unmanned driving) of a vehicle, it is necessary to drive the vehicle along a driving lane. Therefore, it is necessary to detect the traveling lane by some means. On the other hand, a white line is usually formed as a lane marker that defines a traveling lane on the road on which the automobile runs. Therefore, the traveling lane can be recognized by photographing the front of the vehicle and detecting the white line from the change in brightness.

However, when the image processing is performed on all the obtained images, most of the processing is performed on the part without the white line, and a lot of wasteful processing takes time. On the other hand, for high-speed automatic piloting, white line detection should be as simple as possible and should be completed in a short time.

Therefore, it has been proposed to estimate the position of the white line and detect the white line. For example, JP-A-5-151
In Japanese Patent No. 341, a window is set in an image, an edge point is detected from a density gradient in the window,
Curve fitting is performed to detect the white line shape. Then, the next window is determined based on the detected white line shape.

As described above, by setting the window, the processing range can be limited, and the travel route can be detected efficiently.

[0006]

The lane marker (white line) may be a continuous line or a broken line. In addition, it is necessary to accurately detect a curve. Therefore, it is difficult to determine the position of the window when the broken line is used in the curve. In addition, the white line formed on the road surface becomes thin with use. In the case of such a white line, the width of the detected white line becomes less than or equal to a predetermined value, and the white line may not be detected.

Further, when the vehicle runs at high speed on a highway, the white line must be detected at a very high speed.
In the processing of the conventional example, the density gradient in the image is detected when the edge point is obtained. Such processing is usually performed by software processing that executes an algorithm including “if to then”. In order to speed up software processing, parallel processing or the like is necessary, and it has been difficult to perform sufficiently high speed processing.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and an object of the present invention is to provide a running road detecting method and apparatus capable of detecting various lane markers accurately and at high speed.

[0009]

SUMMARY OF THE INVENTION The present invention is a method for detecting a lane marker that defines a running path on which a vehicle travels, the method including an imaging step for capturing a road surface image of the running path,
From the extraction step of extracting a part of the road surface image as a window, the input step of inputting the image information in the window to the neural network as the image information of each of a plurality of pixels forming the window, and the image information input in the neural network. A shape determining step of obtaining a shape of a lane marker existing in the window.

Another aspect of the present invention is a track detection device which is mounted on a vehicle and detects a lane marker that defines a track, wherein image pickup means for picking up a road surface image of the road and a window set in a part of the road surface image. Then, the extraction means for extracting the image information in the window and the image information in the extracted window are input as image information for each of a plurality of pixels forming the window, and exist in the window from the input image information. Neural network processing means for obtaining the shape of the lane marker.

Still another aspect of the present invention determines the position of the window in the extraction means according to the shape of the lane marker detected by the neural network processing means, and moves the window along the lane marker in one road surface image. It is characterized by

Still another invention is characterized in that the neural network processing means divides the window vertically into a plurality of regions and calculates a coefficient for the inclination of the lane marker in each region.

As described above, in the present invention, the lane marker such as the white line is detected by using the neural network. This neural network can learn various lane markers in advance. That is, it is possible to learn various types of lane markers such as one that is greatly bent, one that is about to disappear, and one that is in the shape of a broken line. Therefore, it is possible to reliably recognize various lane markers that are difficult to recognize by simply comparing with a threshold value.

Further, by moving the position of the window along the running path from the processing result of the neural network, it is possible to omit the processing for unnecessary places, and it is possible to perform efficient image processing. Therefore, the lane marker can be detected at a sufficient response speed when traveling at high speed. Therefore, using the detection result, steering control or the like can be performed, and unmanned operation control at high speed can be performed.

Further, by dividing the inside of the window vertically into a plurality of regions, the front image taken by a camera or the like can be divided according to the distance from the vehicle. Then, by outputting the coefficient for the lane shape for each region, more appropriate window position setting can be performed.

[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention (hereinafter referred to as embodiments) will be described below with reference to the drawings.

[System Configuration] FIG. 1 is a block diagram showing a system equipped with the road detecting device of the present invention, and has a camera 10 as an image pickup means for photographing a road (road) in front of the vehicle. A CCD camera, for example, is used as the camera 10.

A conversion unit 12 is connected to the camera 10, and the conversion unit 12 converts the video signal supplied from the camera 10 into digital data for each pixel according to its brightness and obtains it. Output digital image data. A window processing unit 14 that functions as an extraction unit is connected to the conversion unit 12. The window processing unit 14 extracts only a predetermined small area (window) from the image data and outputs the data of the extracted portion.

A neural network processing unit 16 is connected to the window processing unit 14. The neural network processing unit 16 outputs the determination result YN regarding the existence of a white line (lane marker) and the determination results LC1, LC2, LC3 regarding the inclination of the lane marker, using the image data for each pixel in the supplied window as an input signal. At the same time, the position of the window with YN = 1 is output as the lane marker position. Then, the determination result of the neural network processing unit 16 is fed back to the window processing unit 14 as a window command. Therefore, the window processing unit 14 changes the window in the same image based on this window command to supply the image data to the neural network processing unit 16 or the position of the window for the newly supplied image data. To decide.

A control unit 18 is connected to the neural network processing unit 16. The control unit 18 controls the traveling of the vehicle according to the lane marker position supplied from the neural network processing unit 16. That is, in this embodiment, three traveling control units, that is, the steering control unit 20, the accelerator control unit 22, and the brake control unit 24 are connected to the control unit 18, and the control unit 18 responds to the detected lane marker position. , The traveling of the vehicle is controlled so that the vehicle can travel along the lane marker at a predetermined speed. The control unit 18 may not directly control the traveling, but may notify the driver of the necessity of steering operation acceleration / deceleration control.

As described above, in the system of this embodiment,
The traveling of the vehicle can be controlled based on the detected lane marker. Here, in the present embodiment, the neural network processing unit 16 that uses a neural network is used to detect the lane marker. Therefore, the processing using the neural network will be described.

"Basic Structure of Neural Network" The basic structure of the neural network is shown in FIG. As described above, this example has a three-layer structure of the input layer, the intermediate layer, and the output layer. The intermediate layer may have any number of layers as long as it is one or more layers.
Then, each layer is adjusted to n according to the number of input data n.
Each of the arithmetic units has an arithmetic unit, and the conversion function of each arithmetic unit is set to f (x) = 2 / (1 + e [-x])-1 (1). Therefore, each arithmetic unit performs weighted addition of multiplying each input data by a predetermined weight wi and then adds the weighted data to calculate x, and then performs the operation of the equation (1) and outputs the operation result. . The weight in each arithmetic unit is set by learning performed in advance.

For example, one window has vertical 9 × horizontal 9 =
In the case of 81 pixels, there are 81 arithmetic units in each of the input layer and the intermediate layer, and the output layer is composed of four arithmetic units according to the number 4 of the outputs. First, the brightness of 81 pixels is directly input to the input layer. Here, this brightness is represented by 0 to 255, for example, if the output of the conversion unit 12 is 8 bits. Further, the relationship between the subject and the value of this image data is the same as that at the time of learning. That is, the levels are adjusted so that the value changes from 0 to 255 during learning, but the value does not change from 0 to 50 during actual processing.

First, each arithmetic unit of the input layer multiplies the inputs by weights wA1 to wA81 preset by the learning process, and from the obtained x, according to equation (1),
Get the output value. Each of the arithmetic units of the intermediate layer has a weight wB1i set in advance for the output of each arithmetic unit of the input layer.
.About.wB81i (i = 1 to 81) are multiplied and then added to obtain x, and then the output is calculated by the equation (1). Similarly, in the output layer, the weights wC1j to wC4j (j = 1 to 8) are added to the outputs of the arithmetic units of the intermediate layer.
After multiplying by 1), four outputs LC1,
LC2, LC3, YN are calculated and output.

"Processing by Neural Network" Camera 1
When the obtained image of the runway is 0 as shown in FIG. 3, the window processing unit 14 cuts out a part of the image as a window and inputs it to the neural network.

Therefore, the processing in this neural network will be described with reference to FIG. First of all, if you get a new image of the track, where is the white line (lane mark)
I don't know if exists. Therefore, the position of the lane marker and its inclination are detected by the initial scanning described later (S11).

Next, a variable n = for the number of lane scans
1 (S12), for one lane marker, lane scanning is performed to detect from the lower end to the upper end of the screen (S).
13). This lane scanning will also be described later. When the lane marker is detected and the lane scanning is completed, n = n + 1 is set (S14). Next, it is determined whether n is equal to or more than the number N of lane markers existing on the runway on the screen (S15). If NO, the process returns to S13,
The next lane scanning and lane marker detection are performed in the same manner. Then, in S15, if YES is obtained and all the lane scanning is completed for the lane marker to be detected, the lane marker detection process for one screen is completed.

Next, the initial scanning of S11 will be described with reference to FIG. In the case of this initial scanning, the window is moved horizontally as shown in the lower part of FIG. That is, first, the window is set at the lower left of the screen (S21). The image data in this window is input to the neural network, and the outputs LC1, LC2, LC3 are output.
And YN are obtained (S22). Then, it is determined whether YN = 1 (S23). If YN = 1, that is, if the lane marker does not exist, the window is shifted to the right by one pixel (S24), and it is determined whether the window is the lower right corner (S2).
5). If it is not lower right, the window shifted by one pixel in S24 is input to the neural network, and it is checked whether the determination result is YN = 1 (S22, S23). And
If YN = 1 in S13, the number of lanes N = N + 1 is set (S26), the window position, LC
1, LC2, LC3 are stored (S27). Note that N
The initial value of is set to 0.

In this way, when the lane marker is detected, the window is slid to the right by p pixels (S2).
8). This is to avoid detecting the same lane marker twice, and this p pixel is set to a value equal to or larger than the lane marker width, for example, half the width of the window. Then, such processing is repeated until the window reaches the lower right. As a result, the number of lane markers detected when the window is moved in the horizontal direction on the screen is stored in N. The position of each detected lane marker is stored as the position of the window, and the values of LC1, LC2, and LC3 at that time are stored for each. It should be noted that the number of lane markers may not be recognized correctly when the lane markers are interrupted with only one row in the horizontal direction, and initial scanning is performed within a certain range (may be skipped) of one screen to detect existing lane markers. It is better to make sure that it is detected.

Next, regarding the lane scanning in S13, FIG.
It will be described based on. First, for one lane marker, the window position where the lane marker is detected, L
C1-LC3 are taken in (S31). If first, n
= The first lane marker, which is stored in the initial scan, is captured. Next, the curvature of the curve is calculated from LC1 to LC3 (S32), the window is tilted by the distance R corresponding to the curvature, and then moved in the curvature direction to set the window (S33).

Then, the image data of this window is input to the neural network, and a lane marker (YN = 1) is searched for in a predetermined range AB (S34). This window is moved by moving it left and right around the initial position. That is, if the lane marker cannot be detected at the initial window position, the pixel is shifted to the left by one pixel, and if it is not found here either, the pixel is shifted to the right of the initial position by one pixel. Then, this is repeated to detect the lane marker around the initial position. The maximum value to be shifted in this way is set to a predetermined value that is somewhat smaller than the width of one traveling lane. Therefore, when the lane marker is not found in this range, the search for the lane marker is ended.

Even if YN = 0, L at that time
The position of the lane marker may be estimated depending on the situation of the values of C1 to LC3. In that case, based on this estimation, the position of the window may be moved pixel by pixel in the estimated direction.

Further, in the case of a sharp curve or the like, YN = 1 may not be obtained because the lane marker is largely bent. In such a case, it is advisable to reduce the width of the window in the vertical direction to 1/2 and determine again whether YN = 1. As a result, the lane marker can be detected in the case shown in FIG. 7, for example.

If the lane marker is found, LC1 to LC3 are updated to the values at that time (S3).
5). Next, when it is updated in S35, and when the lane marker is not found in S34, the window is shifted toward the traveling direction of the window (S3
6) It is determined whether the screen has reached the upper end (S37). If it is not at the upper end, the process returns to S32, and the same lane marker detection is performed based on the window shifted upward in S36. If YES in S37, the lane scanning process in S13 ends.

In this way, as shown in FIG. 8, the window moves upward along the lane marker. If the lane marker is interrupted, the window is shifted in S36 without updating the values of LC1 to LC3 in S35. Therefore, as shown in FIG. 9, when the lane marker appears again after being interrupted, there is a high possibility that the window position deviates from the lane marker. However, S
In 34, since the window is moved within the predetermined range as described above, the lane marker can be reliably detected. In S35, when the lane is not detected, there are two cases, that is, the case where the lane disappears and the case where the lane is discontinued. As described above, when the lane is discontinued, the lane marker can be detected thereafter. Therefore, it is possible to determine the indistinguishable items with one determination.

[Regarding YN and LC1 to LC3] FIG.
Six cases are shown in 0 for the shape of the lane marker on the screen. YN is set to YN = 1 only when penetrating from the lower end to the upper end of the screen. Therefore, FIG.
In (b) and (c), YN = 0. That is, (a)
Then, there is a lane marker only in the right corner, and it does not penetrate to the upper end. Moreover, in (b), it has not penetrated to the lower end. Further, in (c), the lane marker is not detected and it is determined to be blank. In (d) to (f), YN = 1 because the lane marker penetrates vertically. In addition,
As shown in (a), when the upper end does not penetrate, the height of the window may be halved and the determination may be performed again.

Next, as shown in FIG. 11, LC1 to LC3 are values indicating the analog amount about the inclination obtained by dividing the vertical direction into three regions, and LC1 is the highest (far), LC2 is in the middle, and LC3 is the lowest (front). As shown in FIG. 12, the inclination is 0 for a straight straight line, 1.0 for a 45 ° rightward rise,
The downward slope of 45 ° is set to -1.0. This will be described later, including the case of approximation with an ellipse.

When the shape of the lane marker is approximated by an ellipse, the values of LC1 to LC3 are determined as follows. That is, when the minor axis of the ellipse is a and the major axis is b, the equation of the ellipse is (x / a) ² + (y / b) ² = 1.

If the center of the ellipse is moved only on the x-axis, the lane marker is approximated as shown in FIG. 13 (A).
It is approximated by an ellipse as shown in (B). Therefore, LC
1 to LC3 are obtained as the values of the x-coordinates of the intersection points of the ellipse and the line parallel to the x-axis that passes through the centers of the three regions. That is, when the vertical direction of the window is Y and the window is divided into six equal parts, the x-coordinates of the intersections with the three straight lines of y = 5Y / 6, y = Y / 2, y = Y / 6 are LC1 and LC2, respectively. , LC3.

Here, the center of the ellipse is on the x-axis from the origin,
If it is deviated by α (α = a in the figure), when the curve is curved to the left (in the case of FIG. 13A), {(x + α) / a} ² + (y / b) ² = 1 When the curve is bent to the right (in the case of Fig. 13 (B))
Represents an ellipse with {(x-α) / a} ² + (y / b) ² = 1.

When the lane marker is cut to the left or right of the window, there may be no intersection. In that case, as shown in FIGS. 14 (A) and 14 (B), it is set to a value (X / 2) that is 1/2 the width X of the window.

As described above, in this embodiment, the inside of the window is vertically divided into three regions. These three regions are divided into a region closer to the vehicle, a region next closer to the vehicle, and a region farther from the vehicle. And the relationship between the distance on the screen and the actual distance is
The farther the area, the larger the distance on the screen corresponds to the actual distance. Therefore, the shape of the lane marker is also likely to be different in each region. As in the present embodiment, by dividing vertically and detecting the shape in each region, a more appropriate grasp of the shape can be achieved.

[Calculation of Curvature from LC1 to LC3] Further, in S32 of FIG. 6, the curve curvature is calculated from LC1 to LC3. This is obtained as follows.

(I) When elliptical (LC1 <LC2 <LC3
<0, or 0>LC1>LC2> LC3) In this case, a, b, and c may be obtained by solving the following three-dimensional simultaneous linear equation using an inverse matrix.

{(X / 2) LC1-c} ² + (y1 / b) ² {(X / 2) LC2-c} ² + (y2 / b) ² {(X / 2) LC3-c} ² + (Y3 / b) ² where X is the width of the window in the x direction, y1 to y3 are the sizes Y of the window in the y direction, and y1 = 5Y /
6, y2 = Y / 2 and y3 = Y / 6.

(Ii) When the line is straight (| LC1-LC
2 |, | LC2-LC3 |, | LC3-LC1 | are all sufficiently small) In this case, a straight line may be defined by x = {(LC1 + LC2 + LC3) / 3} · X / 2.

"Learning of Neural Network" The weight w in each of the plurality of arithmetic units provided in each layer of the neural network is set by learning with predetermined data. Therefore, this learning will be described based on FIG.

First, weights for inputs in all the operation units of the neural network are randomly set (S41). Then, the pattern generator generates N1 / 3
It occurs with a probability of N, N2 / N, N3 / N and (S4
2) According to this, cases are divided. Note that N1 + N2 +
N3 = N.

In the case of an ellipse, {(x + c) ² / a ² } ² + (y / b) ² = 1 is generated (S43). Here, a is 0 <A1 ≦ a ≦ A
It is a random number in the range of 2 <∞, and b is 0 <B1 ≦ b ≦ B2 <
It is a random number in the range of ∞, and is a random number of −∞ <C1 ≦ c ≦ C2 <∞. In addition, b is set to a predetermined value or more so that two elliptical lines do not appear in the window.

If it is a straight line, a straight line of x = D is generated (S44). Here, D is a random number in the range of −A3 ≦ D ≦ A3. A3 is an arbitrary number, but is set to a sufficiently small value so that the straight line is generated in a range sufficiently close to the origin. On the other hand, if nothing occurs, a blank screen is generated (S45).

When an ellipse or a straight line is generated in S43 and S44, the width t of the lane marker is set to 0 <T1.
It is generated as a random number in the range of ≤t≤T2 <∞ (S46),
A lane marker is drawn with a line width t (S47).

Next, noise is distributed to each pixel on the screen (S48). The magnitude n of this noise is 0 <n1 ≦ n ≦
Set n2 ≦ P. Note that P is the maximum density of noise.

Then, the conditions are classified according to the type of the generated lane marker (S49), and YN and LC1 to LC3 are generated.

First, an ellipse (I) with -A3≤LC2≤A3
In the case of, YN = 1, (X / 2) · LC1 = ± √ [a ² {1- (5Y / 6) ² / b ² }]-c (X / 2) · LC2 = ± √ [a ² {1- (Y / 2) ² / b ² }]-c (X / 2) · LC3 = ± √ [a ² {1- (Y / 6) ² / b ² }]-c is generated ( S50).

Next, in the case of an ellipse (II) with A3 ≦ LC2 or A3 ≦ LC2, YN = 1, LC1 = ± 1 (X / 2) · LC2 = ± √ [a ² {1- (Y / 2) ² / b ² }]-c (X / 2) · LC3 = ± √ [a ² {1- (Y / 6) ² / b ² }]-c is generated (S51).

In the case of a straight line, YN = 1, (X / 2) .LC1 = D (X / 2) .LC2 = D (X / 2) .LC3 = D.

Further, if blank, YN = 0, LC
Generate 1 = LC2 = LC3 = 0. Then, the image obtained in S48 is input to the neural network, and S50-S
Learning is performed by using the values of YN and LC1 to LC3 obtained in 53 as a teacher (S54). Then, it is determined whether the output error is less than or equal to the specified value (S55), and learning is repeated until the error becomes less than or equal to the specified value.

By such learning, the weight in the neural network is set to a predetermined value, and a neural network in which appropriate YN and LC1 to LC3 are obtained for images of various patterns can be obtained.

In addition, A1, A2, A3 used in this example,
B1, B2, C1, C2, D, etc. are set to have appropriate values for the size of the window (horizontal X, vertical Y) and the like.

Further, in the above example, the window is divided into three equal parts in the vertical direction and LC1 to LC3 are obtained, but it is not always necessary to divide the window into equal parts. That is, it is preferable to change the sizes of the three regions according to the curve state of the lane marker so that the curve state is more appropriately expressed. By setting the division appropriately during learning, a solution corresponding to the division can be obtained. Therefore, the curvature corresponding to the value can be calculated from the obtained LC1 to LC3, and more appropriate window can be set. Further, the number of divisions may be four or more.

According to the present embodiment, the window can be moved along the lane marker, so that the window setting becomes appropriate and unnecessary processing can be minimized. Therefore, high processing efficiency and high-speed processing can be performed.
Therefore, sufficient lane marker detection can be performed even during high-speed traveling, and traveling control can be performed based on this.

[Brief description of drawings]

FIG. 1 is a diagram showing an overall configuration.

FIG. 2 is a diagram showing a configuration of a neural network.

FIG. 3 is a diagram showing a window on a screen.

FIG. 4 is a flowchart showing the entire operation.

FIG. 5 is a flowchart showing initial scanning.

FIG. 6 is a flowchart showing lane scanning.

FIG. 7 is a diagram showing a window in a sharp curve.

FIG. 8 is a diagram showing a moving state of a window.

FIG. 9 is a diagram showing a window setting state when a lane marker is interrupted.

FIG. 10: Lane marker state and YN, LC1 to LC
It is a figure which shows the relationship of the value of 3.

FIG. 11 is a diagram showing a divided state of a window.

FIG. 12 is a diagram showing the relationship between the inclination of the lane marker and the values of LC1 to LC3.

FIG. 13 is a diagram showing values of LC1 to LC3.

FIG. 14 is a diagram showing values of LC1 to LC3.

FIG. 15 is a flowchart showing learning of a neural network.

[Explanation of symbols]

10 camera, 12 conversion unit, 14 window processing unit, 16 neural network processing unit, 18 control unit.

Front page continuation (51) Int.Cl. ⁶ Identification code Office reference number FI Technical display location G05D 1/02 G08G 1/16 G08G 1/16 G06F 15/70 460F

Claims (4)

[Claims] 1. A running road detection method for detecting a lane marker that defines a running road on which a vehicle travels, comprising: an imaging step of taking a road surface image of the running road; an extracting step of extracting a part of the road surface image as a window; An input step of inputting the image information in the neural network as image information for each of a plurality of pixels forming the image information; and a shape determining step of obtaining the shape of the lane marker existing in the window from the image information input in the neural network, A method for detecting a road, comprising: 2. A track detecting device which is mounted on a vehicle and detects a lane marker that defines a track, comprising: an image capturing unit that captures a road image of the road; and an extracting unit that extracts a part of the road image as a window. The extracted image information in the window is input as image information for each of a plurality of pixels forming the window, and the neural network processing means for determining the shape of the lane marker existing in the window from the input image information is included. A track detection device. 3. The apparatus according to claim 2, wherein the position of the window in the extraction means is determined according to the shape of the lane marker detected by the neural network processing means, and the window is used as the lane marker in one road surface image. A traveling road detection device characterized by being moved along. 4. The apparatus according to claim 2, wherein the neural network processing means divides the window into a plurality of regions in the vertical direction, and calculates a coefficient for the inclination of the lane marker in each region. A track detection device characterized by the above.