KR20140148171A - Lane Detection method for Advanced Vehicle - Google Patents

Abstract

The present invention relates to a lane detection method for an intelligent vehicle including the steps of: (a) detecting a vanishing point of a road using RANSAC algorithm in the initial frame of a received image; (b) storing a predetermined area around the vanishing point as a template area (TA) which will be used for template matching; (c) estimating a lane while going down from the vanishing point; (d) extracting a perspective transformation coefficient based on the estimated lane; (e) extracting an inverse perspective transformation coefficient by applying inverse perspective transformation to the extracted perspective transformation coefficient, and obtaining an image from which perspective is removed using the inverse perspective transformation coefficient; and (f) detecting a lane by applying a lane filter to the image from which perspective is removed. The present invention provides the strong lane detecting method in real time even under inclined road conditions using the inverse perspective transformation technology, which does not require any camera parameter of images, and the proposed lane filter.

Description

[0001] The present invention relates to a lane detection method for an intelligent vehicle,

The present invention relates to a lane detection method of an intelligent vehicle, and more particularly, to a lane detection method having a performance equivalent to a reverse inverse mapping (IPM) method even in a sloping road environment, ≪ / RTI >

Recently, with the development of IT convergence technology, the technology of smart car which is converged with various IT technologies in the automobile industry is emerging. The key to smart car evolution is the broad acceptance of existing, state-of-the-art telematics services. Advanced Vehicle Safety (AVS) using these services is evaluated as a technology to reduce traffic accidents and damage by protecting drivers and passengers from accidents such as accident prevention, avoidance and collision.

The Lane Departure Warning System, which is one of the components of high safety intelligent vehicle, receives image information from input sensor such as image, radar, laser, etc., It is a technology that implements safe driving function by warning of neglect or drowsiness driving. In order to realize such a function, it is important to utilize various sensors organically well, but it is common to use an image sensor which is a passive sensor.

The real - time lane detection algorithm for detecting lane departure using the image input sensor should be able to use in real time by shortening the processing time by simple and small operation. In the case of the lane detection algorithm, the color information is judged by using the color information [Korea, Publication No. 10-2013-0028610] and the lane edge information, but the color information can be detected more accurately than the edge information [Republic of Korea, No. 10-2011-0001427] Because of the extraction of specific colors, the range of feature extraction choices is limited. In the case of Hough transform using edge information, it is possible to robustly extract straight line to noise or road condition, but it has a disadvantage that it is difficult to apply to real time detection due to a large amount of computation.

There are various methods of vehicle detection. Y. Wang proposed a B-Snake-based lane detection method. This method proposed a robust lane detection method that can detect any type of lane by setting control points using Snake algorithm to find the control points needed for B-Spline. However, it has a limitation in using real time lane detection because of long detection time.

Alberto Broggi proposed a method to detect the lane by generating the top view image by removing the perspective effect of the image using the camera parameters. In order to eliminate the perspective effect of the image, it was possible to shorten the detection time by using the inverse perspective mapping (IPM) method to make the lane parallel and use only the area where the lane appears without using the entire image. However, this method has a disadvantage in that the parameters of the camera must be known in advance in order to perform the inverse projection transformation.

An object of the present invention to solve the above problems is to provide a lane detection method of an intelligent vehicle capable of detecting lanes at a high speed without requiring calculation of camera parameters and robustly detecting lanes in real- .

The proposed method detects the vanishing point in the first frame of the video, predicts the first lane based on the vanishing point, stores the area around the vanishing point as a template, and tracks the vanishing point region for the next frame using the vanishing point. Predicts the lane from the predicted vanishing point region, sets the ideal lane created from the predicted lane, and uses it to extract a homography matrix to be used for perspective transformation. Using the extracted homography matrix, the image with the perspective effect removed is generated, and the lane is detected using the 5x1 lane filter.

The present invention provides a robust real-time lane detection method even in an inclined road environment by using a back-to-front transformation technique which does not require a camera parameter of an image and a proposed lane filter.

In addition, the present invention is robust in lane detection even in a sloping road environment, shows a good lane detection result of about 40 frames per second by reducing the processing area and simplifying the processing procedure.

1 is a block diagram showing a flow of a real-time lane detecting method of an intelligent vehicle according to an embodiment of the present invention,
FIG. 2 is a schematic diagram illustrating a process for obtaining a vanishing point, which is applied to a lane detecting method according to an embodiment of the present invention,
3 is a schematic diagram illustrating an example of perspective transformation applied to a lane detection method according to an embodiment of the present invention,
4 is a photograph showing a problem when applying the same conversion coefficient in a state of a normal road and a state of an inclination,
FIG. 5 is a photograph showing a template region designated on the basis of the vanishing point of an input image and an input image,
6 is a photograph showing the result of prediction of the first lane in the lane detecting method according to the embodiment of the present invention,
7 is a photograph showing a ROI designation applied to an embodiment of the present invention,
8 is a photograph showing a predicted lane and an ideal lane according to an embodiment of the present invention,
FIG. 9 is a schematic diagram showing an example of the inverse perspective transformation applied to the embodiment of the present invention,
10 is a schematic diagram showing a correlation between a lane predicted according to an embodiment of the present invention and an ideal lane,
11 is a schematic diagram showing an example of two types of lanes predicted according to an embodiment of the present invention,
12 is a photograph showing a vanishing point region when the gradient of the road changes,
FIG. 13 is a photograph showing a result of back-to-back transformation using a homography matrix according to an embodiment of the present invention,
14 is a schematic diagram showing an example of a lane filter used in an embodiment of the present invention,
15 is a graph showing gray values in a lane histogram predicted according to an embodiment of the present invention,
16 is a photograph showing a lane detected according to a lane filter applied to an embodiment of the present invention,
17 is a photograph showing an example of lane detection in a situation where the road condition is straight and no obstacle is present.
18 is a photograph showing an experimental result in a case where the state of the road is a curve and the environment of the road is ideal,
Fig. 19 is a photograph showing the results of experiments in the case where road conditions are irregular or shadows and environmental changes or patterns of roads occur. Fig.
20 is a photograph showing an experimental result in the case where the state of the lane is irregular or another vehicle interrupts the lane.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

1 is a block diagram showing a flow of a real-time lane detecting method of an intelligent vehicle according to an embodiment of the present invention. As shown in FIG. 1, a lane detecting method according to an embodiment of the present invention includes: (a) detecting a vanishing point of a road using a RANSAC algorithm at an initial frame of an input image (S110); (b) storing (S130) a certain area around the vanishing point as a template area (TA) to be used for template matching; (c) predicting a lane while descending to the lower end with reference to the vanishing point (S150); (d) extracting perspective transformation coefficients based on the predicted lane (S170); (e) extracting a backsight transformation coefficient by performing backscatter transformation on the extracted perspective transformation coefficient, and obtaining an image in which the perspective is removed by using the backsight transformation coefficient in operation S310; And (f) detecting a lane by applying a lane filter on the image from which the perspective is removed (S330).

The embodiment of the present invention proposes a lane detection method having the same performance as the reverse inverse mapping (IPM) method even in a sloping road environment without using the camera parameters in advance.

More specifically, the lane detection method according to the present invention detects a vanishing point in the first frame of an image, predicts the first lane based on the vanishing point, stores the area around the vanishing point as a template, and tracks the vanishing point region for the next frame . Then, a lane is predicted from the predicted vanishing point region, an ideal lane created from the predicted lane is set, and a homography matrix to be used for perspective transformation is extracted using the ideal lane. Using the extracted homography matrix, the image with the perspective effect removed is generated, and the lane is detected using the 5x1 lane filter.

That is, in the lane detecting method proposed by the present invention, since the algorithm is simple and the interest area is designated, Hough transform is applied in the first frame and then the vanishing point area is tracked as a template. Therefore, And it does not require camera parameters unlike the IPM method. The lane can be detected robustly to the change of the road slope through the disappearing point tracking for each input image. Hereinafter, the entire process will be described in detail with reference to FIG.

RANSAC  Detection of vanishing point using algorithm

The RANSAC algorithm proposed by Fichier and Bolles is a method for predicting model parameters from noise-intensive original data. In general, parameter estimation methods use as much data as possible to remove invalid data, while the RANSAC algorithm uses a method of extending a consistent set of data (consensus sets) using as little initial data as possible do.

The RANSAC algorithm consists of a hypothesis step and a verification step. In the estimation step, arbitrary N sample data is selected from the original data and the selected data is viewed as normal data to predict the model parameters. In the inspection step, the selected data is checked to see if the original data matches the predicted model. If the source data is valid data, add it to the valid data set. These estimation and inspection steps are repeated N times to obtain parameters.

The number of iterations N should be set high enough so that at least one sample set can guarantee a probability P that contains only valid data (Inlier), typically P = 0.99. If the probability that the valid data is selected in the selected sample data is u and the probability of invalid data is v, v = 1 - u. When the number of sample data is m, the number of repetitions N can be obtained by the following equations (1) and (2).

A vanishing point is obtained by finding the intersection of straight lines using the RANSAC algorithm for the input image. 2 is a schematic diagram illustrating a process for obtaining a vanishing point, which is applied to a lane detecting method according to an embodiment of the present invention.

As shown in FIG. 2, the process for acquiring a vanishing point includes steps of 1) converting the input image into a gray image, 2) edge detection using a canyon edge transformation, 3) 4) randomly selecting two line segments S1, S2 and calculating the intersection p from the sample, 5) calculating the set Sp of lines in S passing through the intersection p, and 6 ) And selecting the point p having the largest set Sp as the vanishing point.

Here, the number of repetition N is set as follows. Assuming that a vanishing point exists in the image, there are two or more straight lines that intersect the vanishing point in the image. If the number of straight lines in the image is n, it is possible to set the number of repetitions N by using the following formula (3).

Perspective transformation S170 )

In the present invention, perspective transformation is used to use a method of removing the perspective of an input image. Perspective transformation is to define the relationship between two images resulting from projecting one 2D object onto two different planes. As a result, as shown in FIG. 3, a rectangle image can be converted into a trapezoid image. 3 is a schematic diagram illustrating an example of perspective transformation applied to a lane detection method according to an embodiment of the present invention.

Planar homography refers to a projection transformation that moves one plane to another. The homography matrix is represented by a matrix H having a size of 3x3 as shown in [Equation 4].

When the real-world ideal image is Q, the perspective transformed image is Q 'scale factor, and the homography matrix is H, the following Q' is expressed by Equation (5).

The homography matrix H is a 1x3x3 matrix because there are only 8 degrees of freedom. The homography matrix H can be expressed as Equation (6) by expressing the intra-camera matrix M, the rotation matrix R, and the motion vector t.

Here, M is expressed by the following equation (7).

W is expressed by a rotation matrix R and a motion vector t, and W and R are expressed by Equation (8).

In the rotation matrix of the homography matrix H, R = [r1, r2, r3] is involved in the rotation of the X, Y and Z axes, respectively. Here, if Z = 0 with respect to the object plane, r3 can be eliminated and the following equation (9) can be written.

When the formula is summarized, H is expressed by the following formula (10).

The homography matrix H defines the relationship between a point P _src on the original image plane and a point P _dst on the destination image plane, which is expressed by the following equation (11).

template matching

Template matching is a method of matching the image to be traced by comparing it with the input image. There are three main ways of doing this: Squared Difference, Correlation, Correlation coefficient matching method. In the present invention, a normalized correlation method is used.

Correlation is used to quantify the relationship of each signal in the field of signal processing. In the correlation method, the product of the image stored in the template and the original input image is squared and added. In this case, if the values are perfectly matched, a high value is output, otherwise, a small value or 0 is output. When the input image is I, the template image is represented by T, and the resulting image is represented by R, the correlation expression is expressed by Equation (12), and if a correlation map is generated by applying the expression, And the position of the template region.

Normalization can be done to improve the accuracy of the template matching method and to reduce the interference of illumination. The normalized method significantly reduces the effect of illumination differences between input and template images. The normalization coefficient is expressed by the following equation (13).

The equation for normalizing [Equation 11] can be expressed by Equation (14).

The process of the lane detection method of the intelligent vehicle according to the embodiment of the present invention will be described in detail with reference to FIG.

The first step applies to the start frame of the image. The algorithm is as follows.

[Step 1 algorithm] (S100)

1) Detect the vanishing point existing in the image using the RANSAC algorithm (S110)

2) a certain area is stored as a template area on the basis of the detected vanishing point (S130)

3) A straight line with a certain width nearest to the bottom of the vanishing point is regarded as a lane

4) Designating a region of interest based on the detected vanishing point (S155)

5) extracting the perspective transform coefficients based on the predicted lane (S170)

The second step is applied when the frame is not the start frame of the image or when the first step is performed. The algorithm is as follows.

[2-step algorithm] (S200)

1) Comparing the stored template region with the original image and detecting the similar region (SA)

2) When the template area TA moves on the Y axis, the template area is moved by the changed Y axis (S230)

3) reset the center point of the template area to the vanishing point (s 240)

4) Reset ROI (S250)

5) The perspective transformation coefficient is re-extracted by applying the changed vanishing point (S260)

The three-step algorithm is as follows.

[3-step algorithm]

1) reverse-transforms the image using the extracted coefficients, and generates an image in which the perspective is removed (S310)

2) Detecting a lane from the back-converted image (S330)

In a situation where a vanishing point is detected after the first frame, the lane is detected only by repeating steps 2 and 3.

Vanishing Point Location Based template  appointed

A vanishing point was obtained using the RANSAC algorithm for the input image. In the embodiment of the present invention, the ROI is specified based on the vanishing point and the back-perspective transformation is used. If the inclination point is changed when the inclined road image is input, . Fig. 4 shows the problem when applying the same transform coefficients in the normal road condition and in the inclined road condition.

In order to solve such a problem, in the embodiment of the present invention, a certain peripheral region of the vanishing point is stored in the template region TA and TA is used in the prediction of the vanishing point region of the next frame. The size of the template area used in the embodiment of the present invention is a square having a size of 100 pixels. FIG. 5 shows a template region designated based on a vanishing point of an input image and an input image.

First lane prediction ( S150 )

In the embodiment of the present invention, a lane is predicted in order to obtain a homography matrix to be used for the inverse perspective transformation. In the lane prediction stage, Hough transform and lane characteristics are used for accurate prediction even if the operation speed is slow. The characteristic of the lane is that it is located below the vanishing point and has a certain thickness in the image of the general road. The straight line detected in the lane passes through the vanishing point. Using this feature, a straight line passing through the vanishing point vertically is drawn, and the lower end of the vanishing point is scanned to find a straight line satisfying the characteristics of the lane.

Specify area of interest ( S155 )

It is important to shorten the detection time in the lane detecting method which must be processed in real time. Since the upper part of the vanishing point represents the sky or other background regardless of the road, only the lower part of the vanishing point is designated as the area of interest. The reason for designating such a region of interest is to reduce the amount of computation and to eliminate the unnecessary image region so that only the road region is subjected to the back-projection transformation. The area of interest used in the embodiment of the present invention is shown in Fig.

Reverse Lookup Transformation of Region of Interest S310 )

In the embodiment of the present invention, unlike the IPM method, the present invention proposes a technique of performing a back-to-back conversion based on a predicted lane in a situation in which camera parameters are not needed. To this end, The ideal lane was set based on the center point of the predicted lane in order to generate parallel imaginary ideal lanes. Fig. 8 is a diagram showing the predicted lane and the ideal lane.

A top view image can be created by applying the general concept of perspective transformation inversely. As shown in FIG. 9, it is possible to generate an image in which the perspective effect is removed by applying the inverse perspective transformation on the input image to which the perspective transformation is applied.

A homography matrix to be used for the back-projection transformation can be obtained by using the start and end points of the predicted lane and the ideal lane. The homography matrix H can be obtained from the camera internal parameter M, the rotation matrix R, and the motion vector t as described above. However, if the vertex coordinates of the rectangle made by the predicted lane and the ideal lane coordinate are known as shown in FIG. 9, So that the homography matrix H can be calculated. In FIG. 10, P ₁ is the coordinates of the predicted lane and the coordinates of the ideal lane.

In this case, the formula for the inverse perspective transformation is as shown in the following equation (15). w means the scale factor.

The corresponding relationship of each point is represented by the following equations (16) and (17).

This can be summarized as follows.

In order to know the eight elements of the homography matrix H, it can be obtained by knowing only four points p corresponding to one point P on the image in relation to the above equation. Therefore, the homography matrix H is extracted using the ideal lane point and the predicted lane point. The coefficients of the homography matrix H can be extracted using the following equation (20).

In the embodiment of the present invention, the homography matrix obtained from the two squares in FIG. 10 is as shown in Table 1, where the coordinates of the rectangle formed of the set of predicted lanes P and the coordinates of the rectangle formed by the ideal lanes are shown in FIG.

Table 1. An Example of Homography Matrix H    i One 2 3 a _i -23.2735 -64.1076 8168.701 b _i 2.45E-14 -47.1822 887.467 c _i 8.76E-17 -0.19044 One

template  Area Search

In the embodiment of the present invention, the vanishing point is detected and the area around the vanishing point is stored as a template area (TA) used for template matching. As a method for template matching, it is possible to quickly and precisely match a template and to use a normalized cross-correlation (NCC) matching method that significantly reduces illumination interference between an input image and a template image Respectively.

When the position of the template area is changed to the Y axis by comparing with the position of the detected vanishing point, the changed position is stored as the template area and the center point of the template area is set as the vanishing point. When a new vanishing point is set, the region of interest is designated again based on the set vanishing point. By tracking the vanishing point and resetting the area of interest as described above, it is possible to use the back-to-front transformation robustly even when the vehicle is suddenly changed in inclination while driving on the road.

Fig. 12 shows a redesigned vanishing point area when the road gradient changes. In FIG. 12E, the red dot indicates the initial vanishing point and the blue dot indicates the location of the new vanishing point tracked and the template region based on each vanishing point.

Lane detection ( S330 )

The homography matrix obtained above is used to generate an image in which the perspective of the image is removed. The result of the inverse perspective transformation using the extracted homography matrix is shown in FIG. As shown in the above picture, the top view image is generally represented by a straight line. Therefore, in the present invention, a 5x1 filter is used for lane detection in a perspective-free image. The 5x1 filter can focus on the vertical edges and convert the image to a gray image and apply the filter. In general, the color of the lane is distributed in a lighter color than the road, and the lane is detected by referring to the lane.

In FIG. 14, the sum of each of 2 pixels on the left and 2 pixels on the right side is determined. When the difference between the two values is equal to or larger than (maximum brightness value - image pixel average value), the current pixel is determined as an edge component. I is an image, c is a current pixel, L is a left part of a current pixel, and R is a right part of a current pixel.

As shown in FIG. 15, since the brightness value has a large change amount in the lane portion, the filter is set by using it.

16 shows an edge image detected using a filter. 16 (a) is an input image, and Fig. 16 (b) is a result of an image detected using a lane filter.

Due to the feature of back-to-back conversion, the image information of the upper part of the converted image may be seen because the image information is insufficient. Therefore, using the information located in the lower part of the image can provide more accurate results in the lane extraction, so the lane is detected by searching the lower part of the detected lane image.

Experiment result

The lane detection method proposed in the embodiment of the present invention is carried out by installing a camera on the front window of the vehicle and using a relatively simple highway image using the input image. 17 shows experimental results in an ideal situation in which the road condition is straight and no obstacle exists. The left image is the input image, and the right image is the lane detection result.

Fig. 18 shows an experimental result when the road condition is a curve and the road environment is ideal.

Fig. 19 shows experimental results in the case where the state of the road is irregular or shaded, and when a change in environment or a pattern of the road occurs.

Fig. 20 shows an experimental result in the case where the lane state is irregular or another vehicle intervenes in the lane.

As such, the vehicle is generally driven on a flat road, but an uphill or downhill road may appear depending on the obstacle of the road or the state of the road itself. In this case, since the angle between the road and the camera instantaneously changes in the IPM method, the image in which the top view image appears may not be parallel but may be spread or gathered. These problems may appear brief in time and may not have a significant impact, but they can cause problems in lane departure warning systems. In the present invention, such a problem can be solved by a method of tracking a vanishing point.

The method proposed in the embodiment of the present invention was developed as OpenCV for convenience and the image of 640 * 480 size can be processed in about 40 frames per second in the Intel Core2 Duo 2.6 GHz and 2 GB RAM computer. Table 2 shows the comparison between the method using the B-Snake method and the method using the IPM method according to the embodiment of the present invention in terms of processing speed and camera parameter use.

As described above, according to the present invention, after a lane is predicted using a vanishing point, a vanishing point region is stored as a template, a lower region of the vanishing point reference is set as a region of interest, and a perspective transformation is performed on the region, Detection method. It is possible to designate the area of interest in the lane detection process and to simplify the processing method to remarkably shorten the detection time and to know the camera parameters. In addition, we propose a robust lane detection method that is not affected by uneven road surface condition or sudden uphill or downhill, and it is verified that the feasibility is verified by comparing the processing speed with the existing method.

Claims (7)

A method of detecting a lane of an intelligent vehicle,
(a) detecting a vanishing point of a road using a RANSAC algorithm in an initial frame of an input image;
(b) storing a certain area around the vanishing point as a template area (TA) to be used for template matching;
(c) predicting a lane on the basis of the vanishing point as descending downward;
(d) extracting perspective transformation coefficients based on the predicted lane;
(e) a step of backsight-transforming the extracted perspective transformation coefficients to extract a backsight transformation coefficient, and acquiring a perspective-free image using the backsight transformation coefficients; And
(f) detecting a lane by applying a lane filter on the image from which the perspective is removed.
The method according to claim 1,
The step (a)
Converting an input image into a gray image;
Detecting an edge of the gray image by canyon edge transformation;
Generating a set S of lines in the image using standard Hough transform;
Selecting two arbitrary line segments in the upper set S and calculating an intersection p of the two line segments;
Calculating a set Sp of line segments passing through the intersection point P;
And selecting a point having the largest set of the set Sp as a vanishing point.
3. The method according to claim 1 or 2,
After the step (c)
Further comprising the step of: designating a region of interest (ROI) based on the detected vanishing point.
The method of claim 3,
If the input image is not the first frame or after step (d)
Detecting a similar area (SA) in the stored template area compared with the original image;
Comparing the coordinates of the template with the coordinates of the similar region, and storing the changed region as a new template region when the template region moves on the Y axis;
Resetting the center point of the new template area to a vanishing point;
Resetting a region of interest (ROI) based on the reset vanishing point; And
Further comprising extracting perspective transformation coefficients by applying the reset vanishing point to the real-time lane detecting means.
The method of claim 3,
Wherein the ROI is a lower region based on the vanishing point.
5. The method of claim 4,
The step (C)
And calculating a homography matrix to be used for the reverse inverse transformation.
5. The method of claim 4,
In the template matching,
Wherein a normalized cross-correlation (NCC) matching method is used.

Images