CN109583365B - Method for detecting lane line fitting based on imaging model constrained non-uniform B-spline curve - Google Patents

Abstract

The method comprises the steps of constraining a non-uniform B-spline curve to fit a lane line detection method based on an imaging model, firstly, carrying out median filtering and histogram equalization processing on an image to obtain an enhanced lane line image; secondly, performing edge detection on the image by adopting a Canny operator to obtain a lane line edge image; then, Hough transformation straight line detection is carried out on the edge image, so that the background interference edge is reduced while the edge continuity is improved; deducing a control point estimation model under the constraint of a lane line-camera imaging model on the basis of a camera geometric imaging model based on the assumptions of 'the optical axis of the camera is parallel to the road plane' and 'the left lane line and the right lane line are parallel'; and finally, solving the parameters of the non-uniform B-spline curve model by combining the position information of the pixels at the edge of the lane line to realize the fitting of the lane line. The method can effectively improve the positioning precision of the control points and the detection accuracy of the lane lines, and improve the robustness of the lane line detection algorithm based on curve fitting to background interference.

Description

A lane line detection method based on imaging model-constrained non-uniform B-spline curve fitting

technical field

The invention belongs to the field of traffic video detection, in particular to a lane line detection method based on imaging model constraint non-uniform B-spline curve fitting.

Background technique

At present, driverless technology is a research hotspot in the field of intelligent transportation. The large amount of scientific research investment by many scientific research institutes and enterprises at home and abroad has greatly promoted the rapid development of driverless technology. As the necessary information for the vehicle to keep the lane, change lanes and other driving behaviors, the lane line is an important environmental data in the perception of the driving environment of the unmanned vehicle. The safety of the entire unmanned vehicle driving system has a non-negligible impact.

The main purpose of lane line detection is to extract the location information of lane lines from video images. At present, the commonly used lane line detection methods can be roughly divided into three categories: area-based, feature-based and model-based, among which the model-based lane line detection method is the most common. This kind of method is usually based on the idea that the lane lines of structured roads can be approximated by a specific mathematical model. For lane lines with different orientations such as linear, parabolic, and snake-shaped, straight lines, parabolas, hyperbolas, and splines are used. and other mathematical models to fit, thereby greatly reducing the detection cost while ensuring the accuracy of lane line detection. The spline curve is expressed by a piecewise polynomial, which can accurately fit any shape of the curve, so it has been widely used in lane line detection. Analysis of related research shows that the determination of control points is the key to the B-spline curve fitting lane line. However, the interference of vehicle occlusion, tree shade, building shadow, road damage, etc. increases a lot of difficulty for the extraction of model control points, thus affecting the model. The accuracy of the lane line fitting even causes the fitting to fail. Therefore, how to effectively improve the robustness of control point extraction to background interference (vehicle occlusion, tree shade, building shadow, other road signs, road damage, etc.) The key issue of the efficiency of lane line detection method based on spline curve model.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the problem that the traditional lane line detection method based on spline curve model is susceptible to background interference, resulting in inaccurate or failed control point positioning, and to provide a lane line detection method based on imaging model constraint non-uniform B-spline curve fitting , the method can effectively improve the positioning accuracy of the control point while taking into account the time cost of the algorithm, obtain better detection results, and effectively improve the efficiency of lane line detection.

In order to achieve the above object, the present invention adopts the following technical solutions to realize:

A lane line detection method based on imaging model-constrained non-uniform B-spline curve fitting includes the following steps:

Step 1: Image preprocessing;

Obtain the original lane line image I from the lane line standard image library in the Carnegie Mellon image database, perform median filtering on the original lane line image I to remove salt and pepper noise, and then perform histogram equalization to enhance the brightness and contrast of the image. Make the edge features stand out to obtain the enhanced lane line image I ₁ ;

Step 2: edge detection;

Using the Canny operator to perform edge detection on the enhanced lane line image I ₁ to obtain the initial lane line edge image I ₂ ;

Step 3: Hough line detection;

Perform Hough straight line detection on the initial lane line edge image I ₂ obtained in the step 2, retain the edge containing the straight line detection result, remove the remaining interference edges, and obtain an edge image I ₃ ;

Step 4: Derive imaging model constraints;

The derived imaging model constraints are:

The line segment length Δv in the u-th column in the image coordinate system corresponding to the lane line segment with the length ΔY in the world coordinate system is:

The spacing width Δu in the vth row in the image coordinate system corresponding to the left and right lane line spacing with a width of ΔX in the world coordinate system is:

Step 5: Extraction of non-uniform B-spline curve control points;

Scan lines are set in the image according to the constraints of the imaging model according to the length of ΔY in the corresponding world coordinate system, and the intersection of each scan line and the edge of the left and right lanes is a pair of control points;

Step 6: Lane line fitting;

Obtain the non-uniform B-spline curve control point information in step 5, and then use the NUBS interpolation method to fit the left and right lane lines to complete the detection of the lane lines.

A further improvement of the present invention is that, in step 1, when median filtering is performed, the median filtering function f^(x, y) is used as:

Where f^(x, y) is the output of the median filter, S _xy represents the coordinate group of a rectangular sub-image window whose center is (x, y) and the size is M×N, and f(a, b) is the coordinate group of (a) ,b) pixel gray value.

A further improvement of the present invention is that, in step 1, when performing histogram equalization, the histogram equalization function _sk is:

就是概率论中的频数。where s _k is the histogram equalization output, r _k represents discrete gray level, 0≤r _k ≤255, k=0,1,2,...,n-1, n _i is the gray level _ri appearing in the image number of pixels, n is the total number of pixels in the image,

It is the frequency in probability theory.

A further improvement of the present invention is that the concrete steps of step 2 are as follows:

(1) Smooth the image I ₁ with a Gaussian filter;

The Gaussian smoothing function G(x,y) is:

Convolve G(x, y) with the enhanced lane line image I ₁ to obtain a smooth image f ₁ ;

f ₁ (x,y)=I ₁ (x,y)*G(x,y) (4)

(2) Calculate the magnitude and direction of the gradient by using the finite difference of the first-order partial derivative to obtain the gradient image f ₂ ;

first-order differential convolution template

(3) performing non-maximum suppression on the gradient amplitude to obtain a non-maximum suppression image f ₃ ;

At each point of the gradient image f2, compare the center pixel S of the 8-neighborhood with the _two pixels along the gradient line; if the gradient value of S is not greater than the gradient value of the two adjacent pixels along the gradient line, then let S = 0;

(4) Detect and connect edges with a double threshold algorithm;

Set two thresholds T ₁ and T ₂ for the non-maximum value suppression image f ₃ , T ₁ =0.4T ₂ , set the gray value of the pixel whose gradient value is less than T ₁ to 0, and obtain the image f ₄ ; The gray value of the pixel whose value is less than T ₂ is set to 0, and the image f ₅ is obtained; based on the image f ₅ and supplemented by the image f ₄ , the edges of the image are connected to obtain the initial lane edge image I ₂ .

A further improvement of the present invention is that the concrete steps of step 3 are as follows:

(1) Hough line detection;

For any point A(x ₀ , y ₀ ) in the Cartesian coordinate system, the straight line passing through the point A satisfies

y=kx+l(5)

Where k is the slope and l is the intercept, then the straight line clusters passing through the point A(x ₀ , y ₀ ) in the XY plane are all represented by formula (5), and the slope of the straight line perpendicular to the X-axis is infinite, it cannot be represented; So convert the Cartesian coordinate system to the polar coordinate system;

The equation representing a straight line in polar coordinates is ρ=xcosθ+ysinθ(6)

where ρ is the normal distance from the origin to the straight line, and θ is the positive angle between the normal and the X-axis; then a point in the image space corresponds to a sinusoid in the polar coordinate system ρ-θ; The intersection point is used to detect the straight line in the image space; ρ and θ are discretized, and for each value corresponding to the parameter θ, the value of the corresponding parameter ρ is calculated according to formula (6), and then in the corresponding parameter accumulation unit. Add 1; finally count the value of each accumulating unit, if it is greater than the preset threshold H, it is considered that the group of parameters is the parameter of the straight line in the image space, thereby marking the straight line in the image;

(2) remove the interference edge;

For each edge pixel in the line marked in step (1), search for the entire edge including the pixel and keep it, and remove the edge that has no common pixel point with the marked line, thereby obtaining the edge image I ₃ .

A further improvement of the present invention is that the concrete process of step 4 is:

It is assumed that the optical axis of the camera is parallel to the plane of the vehicle's driving road and the left and right lane lines;

Knowing the world coordinate system (X, Y, Z) and the image coordinate system (U, V), the maximum horizontal viewing angle of the camera is α, the maximum vertical viewing angle is β, and the coordinates of the camera installation position in the world coordinate system are C(d, 0,h), where h is the camera installation height, that is, the value of the camera on the Z axis of the world coordinate system, and d is the horizontal offset of the camera installation, that is, the value of the camera on the X axis in the world coordinate system; the camera optical axis and the vehicle The plane of the driving road is parallel, and the angle between it and the lane line is γ; according to the camera geometric imaging model, a certain point P(x,y,0) on the road in the world coordinate system (X,Y,Z) and the image coordinate system (U,V) ), the mapping model of the relative coordinate point Q(u, v) is:

where H _I and W _I are the horizontal and vertical resolutions of the image after imaging by the camera, respectively;

According to the camera imaging principle, the length of the lane line segment in the image after imaging is shortened with the increase of the distance between the lane line segment in the world coordinate system and the camera. Similarly, the same distance between the left and right lane lines on the road surface in the world coordinate system, in myopia The distance between the lane lines obtained by imaging in the field is wider, and the distance between the lane lines obtained by imaging in the far field of view is narrow; then combined with the geometric camera imaging model, the image coordinate system corresponding to the lane line segment with the length ΔY in the world coordinate system is deduced The line segment length Δv in the u-th column is:

The spacing width Δu in the vth row in the image coordinate system corresponding to the left and right lane line spacing with a width of ΔX in the world coordinate system is:

A further improvement of the present invention is that the concrete process of step 5 is:

Starting from the bottom of the lane line edge image, set the horizontal scan line Line i in line v _i , m≤i≤n, and obtain the control point pair (Li, R _i ) at the intersection of Line _i and the left and right lane lines, where Li's The coordinates are (u _i ,vi ), and the coordinates of R _i are ( _{u i '} ,vi ₎ ; according to the constraints of the imaging model _, the formula for calculating vi is defined as:

According to formulas (8), (10), (11), it is derived:

Wherein v ₁ , Δv ₁ are preset values; Substitute formula (14) into formula (13) to obtain the value of v _i in turn; thus obtain the ordinate of the control point determined by the i-th scan line Line i is equal to v _i ; Search for edge points from the midpoint of the scan line to the left and right sides respectively, and the intersection of the first pair of scan lines and the left and right lane lines is the control point, so as to determine the coordinates (u _i , v _i ) and (u i , v i ) of a pair of control points ′ _i ,v _i ).

A further improvement of the present invention is that the concrete process of step 5 is:

Assuming that the left and right lane lines are parallel, solve the abscissas u _i and u′ _i of the control points _Li and Ri determined by the i-th scan line Line _i ; derive Δu _i+1 according to the constraints of the imaging model and equation (14) The relationship with Δu _i is:

u′ _i =u _i +Δu _i (16)

The angle γ between the optical axis of the camera and the lane line is calculated according to the following formula:

For the situation that the control point is lost due to the lack of the edge of the lane line, when the adjacent control point pairs L ₁ and L ₂ are known, the abscissa u ₂ of the control point is calculated according to formulas (15)-(17); for In the case of wrong positioning of control points caused by false edges, verify whether the ratio of all adjacent control points to the spacing width satisfies Equation (15), so as to detect the coordinates of the wrong control points and reposition them according to Equations (15)-(17) .

A further improvement of the present invention is that the concrete process of step 6 is:

Assuming that the B-spline curve S is composed of n+1 control point sets {P ₀ , P ₁ ,...P _n }, then each point on the curve S satisfies:

where B _i,m (o) is the basic B-spline function, 2≤m≤n+1, t _min ≤u≤t _max , t _j is the node, j=0,...,i+m, when each When the nodes t and _j are equidistant, the B-spline curve is called a uniform B-spline curve, otherwise it is a non-uniform B-spline curve; according to the NUBS interpolation method, if m pairs of control points are known, and m≥3, then the lane The line is fitted with an m-1-order polynomial function; if 4 pairs of control points can be determined, a third-order polynomial function is used to perform NUBS interpolation to fit the lane line; if only 3 pairs of control points are determined, a second-order polynomial is used. Function to fit lane lines.

Compared with the prior art, the present invention has the beneficial effects: firstly, the present invention performs median filtering and histogram equalization processing on the image to obtain an enhanced lane line image; secondly, the Canny operator is used to perform edge detection on the image to obtain The lane line edge image; then the Hough transform line detection is performed on the edge image to improve the edge continuity and reduce the background interference edge; finally, the non-uniform B-spline curve model parameters are solved combined with the lane line edge pixel position information to realize the lane line fitting. The invention can effectively improve the positioning accuracy of the control point and the detection accuracy of the lane line, and improve the robustness of the lane line detection algorithm based on the curve fitting to the background interference.

Further, based on the assumptions of "camera optical axis is parallel to the road plane" and "left and right lane lines are parallel", based on the camera geometric imaging model, the control point estimation model under the constraints of the lane line-camera imaging model is derived; Occlusion, tree shade, building shadow, road damage, and various non-lane road surface markings cause interference to control point determination, improve the robustness of the control point extraction method for non-uniform B-spline curves, and improve the lane line detection accuracy .

Description of drawings

Figure 1 is a schematic diagram of the position of the camera in the world coordinate system;

Figure 2 is a schematic diagram of the control point determination process under the condition of continuous lane lines;

Figure 3 is the first result of lane line detection;

FIG. 4 is a schematic diagram of a control point determination process in the case of discontinuous lane lines;

Figure 5 is the second result of lane line detection;

FIG. 6 is a flowchart of a non-uniform B-spline curve fitting lane line detection algorithm based on imaging model constraints.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings.

The non-uniform B-spline curve fitting lane line detection method based on imaging model constraints provided by the present invention includes the following steps:

Step 1: Image preprocessing;

Obtain the original lane line image I from the lane line standard image library in the Carnegie Mellon image database, perform median filtering on the original lane line image I to remove salt and pepper noise, and then perform histogram equalization to enhance the brightness and contrast of the image. The edge features are highlighted to obtain the enhanced lane line image I ₁ .

Median filter function f^(x,y)

Where f^(x, y) is the output of the median filter, S _xy represents the coordinate group of a rectangular sub-image window whose center is (x, y) and the size is M×N, and f(a, b) is the coordinate group of (a) ,b) pixel gray value.

Histogram equalization function _sk

就是概率论中的频数。where s _k is the histogram equalization output, rk represents discrete gray level (0≤r _k ≤255, _k =0,1,2,...,n-1), n _i is the appearance of _ri in the image The number of pixels in grayscale, n is the total number of pixels in the image,

It is the frequency in probability theory.

Step 2: edge detection;

The Canny operator is used for edge detection on the enhanced lane line image I ₁ to obtain the initial lane line edge image I ₂ .

Specific steps are as follows:

(1) Smooth the image I ₁ with a Gaussian filter;

Gaussian smoothing function G(x,y)

Convolve the image I ₁ with G(x,y) to get the smoothed image f ₁

f ₁ (x,y)=I ₁ (x,y)*G(x,y) (4)

(2) Calculate the magnitude and direction of the gradient by using the finite difference of the first-order partial derivative to obtain the gradient image f ₂ ;

first-order differential convolution template

(3) performing non-maximum suppression on the gradient amplitude to obtain a non-maximum suppression image f ₃ ;

At each point of the gradient image f2, the center pixel S of the ₈ -neighborhood is compared with the two pixels along the gradient line. If the gradient value of S is not greater than the gradient value of two adjacent pixels along the gradient line, then let S=0.

(4) Detect and connect edges with a double-threshold algorithm.

Two thresholds T ₁ and T ₂ are set for the non-maximum suppression image f ₃ , and T ₁ =0.4T ₂ . The gray value of the pixel whose gradient value is less than T ₁ is set to 0, and the image f ₄ is obtained. Then, the gray value of the pixel whose gradient value is less than T ₂ is set to 0, and the image f ₅ is obtained. Based _on the image f5 and supplemented by the image _f4 , the edges of the image are connected to obtain the edge image _I2 .

Step 3: Hough line detection;

Hough line detection is performed on the lane line edge image I ₂ obtained in the second step, only those edges containing the line detection result are retained, and the remaining interference edges are removed to obtain an edge image I ₃ . Specific steps are as follows:

(1) Hough line detection;

For any point A(x ₀ , y ₀ ) in the Cartesian coordinate system, the straight line passing through the point A satisfies

y=kx+l(5)

where k is the slope and l is the intercept. Then the cluster of straight lines passing through the point A(x ₀ , y ₀ ) in the XY plane can be represented by equation (5), but it cannot be represented for the straight line perpendicular to the X-axis whose slope is infinite. So converting the Cartesian coordinate system to the polar coordinate system can solve this special case.

The equation representing a straight line in polar coordinates is ρ=xcosθ+ysinθ(6)

Where ρ is the normal distance from the origin to the straight line, and θ is the positive angle between the normal and the X axis. Then a point in the image space corresponds to a sinusoid in the polar coordinate system ρ-θ. Lines in image space are detected by detecting intersection points in ρ-θ space. Discretize ρ and θ, and at each value corresponding to parameter θ, calculate the value of the corresponding parameter ρ according to formula (6), and then add 1 to the corresponding parameter accumulation unit. Finally, the value of each accumulating unit is counted, and if it is greater than the preset threshold H, it is considered that the group of parameters is the parameter of the straight line in the image space, so as to mark the straight line in the image;

(2) remove the interference edge;

For each edge pixel in the line marked in step (1), search for the entire edge including the pixel and keep it, and remove those edges that have no common pixel point with the marked line, thereby obtaining the edge image I ₃ .

Step 4: Derivation of imaging model constraints;

Based on the camera geometric imaging model, the imaging model constraints are derived based on the assumptions that the optical axis of the camera is parallel to the road plane of the vehicle and that the left and right lane lines are parallel.

Knowing the world coordinate system (X, Y, Z) and the image coordinate system (U, V), the maximum horizontal viewing angle of the camera is α, the maximum vertical viewing angle is β, and the coordinates of the camera installation position in the world coordinate system are C(d, 0, h), where h is the camera installation height, that is, the value of the camera on the Z axis of the world coordinate system, and d is the horizontal offset of the camera installation, that is, the value of the camera on the X axis in the world coordinate system. The optical axis of the camera is parallel to the plane of the road where the vehicle is traveling, and the angle between it and the lane line is γ. According to the camera geometric imaging model, it can be known that a certain point P(x,y,0) on the road in the world coordinate system (X,Y,Z) and the relative coordinate point Q(u,v) in the image coordinate system (U,V) The mapping model is:

where H _I and W _I are the horizontal and vertical resolutions of the image after imaging by the camera, respectively.

According to the camera imaging principle, the length of the lane line segment in the image after imaging is shortened with the increase of the distance between the lane line segment in the world coordinate system and the camera, that is, the lane line segment with the same length on the road surface in the world coordinate system, in myopia The lane line segment obtained by imaging in the field is longer, while the lane line segment obtained by imaging in the far field is shorter. Similarly, with the same left and right lane line spacing on the road surface in the world coordinate system, the lane line spacing obtained by imaging in the near field of view is wider, while the lane line spacing obtained by imaging in the far field of view is narrower. According to this imaging fact, combined with the aforementioned camera imaging model, it can be deduced that the line segment length Δv in the uth column in the image coordinate system corresponding to the lane line segment with length ΔY in the world coordinate system is:

In the world coordinate system, the spacing width Δu in the vth row in the image coordinate system corresponding to the left and right lane line spacing with a width of ΔX is:

Step 5: Extraction of non-uniform B-spline curve control points;

In order to solve the parameters of the NUBS curve model, it is necessary to first determine the appropriate control points. The method is: starting from the bottom of the image of the edge of the lane line, and setting the scan lines in the image according to the imaging model constraints with "corresponding to the length of ΔY in the world coordinate system". The intersection of the scan line with the left and right lane edges is a pair of control points. The mathematical description of the process is as follows: starting from the bottom of the lane line edge image, set the horizontal scan line Line i (m≤i≤n) at line v _i , and obtain the control point pair (Li, R _i ), where the coordinates of _Li are ( _ui ,vi ), and the coordinates of R _i are _{( ui} ',vi ₎ . Since the lane line extends longitudinally in the image as a whole, the ordinate of each control point pair decreases in turn. This trend is not affected by the angle γ between the optical axis of the camera and the lane line. Therefore, in order to simplify the calculation process, the value of γ is ignored. The model constraint definition v _i calculation formula is:

According to formulas (8), (10), (11), it is derived:

Among them, v ₁ and Δv ₁ are preset values. By substituting Equation (14) into Equation (13), the value of _vi can be obtained sequentially. From this, it can be obtained that the ordinate of the control point determined by the i-th scan line Line i is equal to v _i . From the midpoint of the scan line, the edge points are searched for the left and right sides respectively, and the intersection of the first pair of scan lines and the left and right lane lines is the control point, and then the abscissa of the control point pair can be known, so as to determine its coordinates (u _i ,vi ) and ( _u ′ _{i ,} vi ).

The above control point determination process assumes that the left and right lane lines are not blocked, that is, the control point determined by each scan line is the intersection of the scan line and the left and right lane lines. However, the actual detected lane lines often have interfering edges or discontinuous edges caused by vehicle occlusion, tree shadows, building shadows, and road damage. In addition, the edges of the dashed lane lines are also discontinuous. This will result in that the intersection of the scan line with the left lane line or the right lane line is not the actual correct intersection, or there is no intersection.

In view of the above problems, the present invention, on the basis of the aforementioned control point determination method, combined with the assumption of "parallel left and right lane lines", _solves the _{abscissas ui} and u′ _i . According to the constraints of the imaging model and equation (12), the relationship between Δu _i+1 and Δu _i is derived as:

u′ _i =u _i +Δu _i (16)

The angle γ between the optical axis of the camera and the lane line is calculated according to the following formula:

For the case where the control point is lost due to the lack of the edge of the lane line, the abscissa u ₂ of the control point can be calculated according to formulas (15)-(17) when the adjacent control point pairs L ₁ and L ₂ are known. ; For the case of incorrect positioning of control points caused by false edges, it can be verified whether the ratio of all adjacent control points to the spacing width satisfies the formula (15), so as to detect the coordinates of the wrong control points and adjust them according to formulas (15)-(17). to reposition.

Step 6: Lane line fitting.

After the control point information of the edge of the lane line is obtained by the control point determination method in step 5, the left and right lane lines can be fitted by using the NUBS interpolation method. The mathematical model of the B-spline curve is:

Assuming that the B-spline curve S is composed of n+1 control point sets {P ₀ , P ₁ ,...P _n }, then each point on the curve S satisfies:

where B _i,m (o) is the basic B-spline function, 2≤m≤n+1, t _min ≤u≤t _max , t _j (j=0,...,i+m) is the node, when When each node t _j is equidistant, the B-spline curve is called a uniform B-spline curve, otherwise it is a non-uniform B-spline curve. According to the NUBS interpolation method, it can be known that if m (m ≥ 3) pairs of control points are known, the lane line can be fitted with an m-1 order polynomial function. If four pairs of control points can be determined, a third-order polynomial function can be used to perform NUBS interpolation to fit the lane lines; if only three pairs of control points are determined, a second-order polynomial function can be used to fit the lane lines.

Substitute the coordinates of the control point determined in step 5 into formula (18), solve the spline curve S(u) and display it in the original lane line image I, and complete the detection of the lane line.

A specific embodiment will be used for description below.

Referring to Fig. 6, the method of the present invention performs edge detection on a lane line image with a size of W _I × H _I (240 × 256), and determines the control points by combining the image model constraints with the edge position information of the lane line, so as to solve the non-uniformity B-spline curve parameters to realize lane line detection.

The specific steps are as follows:

Step 1: Obtain the original lane line image I from the lane line standard image library in the Carnegie Mellon image database, perform median filtering on the image I to remove salt and pepper noise, and then perform histogram equalization to enhance the brightness and contrast of the image, The edge features are highlighted to obtain the enhanced lane line image I ₁ . In the present invention, a 3×3 rectangular sub-image window is used for median filtering, and the gray scale discrete parameter r _k is in the range of 0≤r _k ≤255 during histogram equalization.

双阈值T₁和T₂为默认值，满足T₁＝0.4T₂。Step 2: Use Canny operator to perform edge detection on the enhanced original lane line image I ₁ to obtain an initial lane line edge image I ₂ . The smoothing parameter σ=1 of the Gaussian smoothing function used for edge detection in the present invention, the first-order differential convolution template

The double thresholds T ₁ and T ₂ are default values, satisfying T ₁ =0.4T ₂ .

Step 3: Use the Hough algorithm to detect the Hough line on the initial lane line edge image I ₂ obtained in step 2. For each edge pixel in the marked line, search for the entire edge containing the pixel and keep it, and remove those with the same pixel. The marked straight line has no edge of common pixel points, so the edge image I ₃ is obtained. The parameters of the Hough algorithm in the present invention are all default values;

Step 4: Starting from the bottom of the lane line edge image _I3 , set the horizontal scan line Line i (2≤i≤6) in the vi line, and obtain the control point pair (Li, R _i ) at the intersection of Line i and the left and right lane lines , where the coordinates of Li are ( _{u i} ,vi ₎ , and the coordinates of R _i are ( _ui ',vi ₎ . In the present invention, v ₁ =0 and Δv ₁ =20 are preset values. By substituting Equation (14) into Equation (13), the value of _vi can be obtained sequentially. From this, it can be obtained that the ordinate of the control point determined by the i-th scan line Line i is equal to v _i . From the midpoint of the scan line, the edge points are searched for the left and right sides respectively, and the intersection of the first pair of scan lines and the left and right lane lines is the control point, and then the abscissa of the control point pair can be known, so as to determine its coordinates (u _i ,vi ) and ( _u ′ _{i ,} vi ).

The actual detected lane lines are interfering edges or discontinuous edges caused by vehicle occlusion, tree shade, building shadows, road damage, or dashed lane lines. For the case where the control point is lost due to the lack of the edge of the lane line, the abscissa u ₂ of the control point can be calculated according to formulas (15)-(17) when the adjacent control point pairs L ₁ and L ₂ are known. ; For the case of incorrect positioning of control points caused by false edges, it can be verified whether the ratio of all adjacent control points to the spacing width satisfies the formula (15), so as to detect the coordinates of the wrong control points and adjust them according to formulas (15)-(17). to reposition.

Step 5: After obtaining the control point information on the edge of the lane line by the control point determination method described in the step 4, the left and right lane lines can be fitted by using the NUBS interpolation method. If 4 pairs of control points can be determined, the third-order polynomial function can be used for NUBS interpolation to fit the lane line; if only 3 pairs of control points are determined, the second-order polynomial function can be used to fit the lane line. Substitute the coordinates of the control point determined in step 4 into formula (18), solve the spline curve S(u) and display it in the original lane line image I, and complete the detection of the lane line.

For a lane line image of size 240×256, median filtering and histogram equalization are performed, and the edge features of lane lines are extracted by Canny edge detection operator. Then perform Hough straight line detection on the image of the lane line edge: Combined with the feature of "the edge of the lane line in the near field is straight", the Hough line detection is performed on the image of the lane line edge, and only those edges containing the straight line detection result are retained, and further elimination is performed accordingly. Those interfering edges formed by background buildings, shade, obstacles, pavement holes, cracks, etc.

Figure 1 is a schematic diagram of the position of the camera in the world coordinate system. According to the principle of camera imaging, the imaging model is derived as equations (11)-(12).

FIG. 2 is a schematic diagram of the control point determination process in the case of continuous lane edge edges. Assuming v ₁ =0, Δv ₁ =20, and substituting equation (14) into equation (13), the values of v _i can be obtained in turn: v ₂ =167, v ₃ =117, v ₄ =78, v ₅ = 46, v ₆ =21. Starting from the bottom of the lane line edge image, set the horizontal scan line Line i (2≤i≤6) on the v _i line, and obtain the control point pair (Li,R _i ) at the intersection of Line i and the left and right lane lines, where Li _i The coordinates of is (u _i ,vi _{), and the coordinates of R i are (u′ i , vi )} . From this, it can be obtained that the ordinate of the control point determined by the i-th scan line Line i is equal to v _i . From the midpoint of the scan line, the edge points are searched for the left and right sides respectively, and the intersection of the first pair of scan lines and the left and right lane lines is the control point, and then the abscissa of the control point pair can be known, so as to determine its coordinates (u _i ,vi ₎ and ( _ui ',vi ₎ . According to this, 5 sets of control point pairs of coordinates {(92,21)(117,21)}, {(80,46)(145,46)}, {(67,78)(175,78)}, {( 51,117)(212,117)}, {(33,167)(254,167)}. Then, according to the NUBS interpolation algorithm, the fourth-order polynomial function is used for NUBS interpolation, and the obtained fitting curve is displayed in the lane line image as shown in Figure 3.

FIG. 4 is a schematic diagram of a control point determination process when the edge of the lane line is discontinuous. Similarly, assuming v ₁ =0, Δv ₁ =20, and substituting Equation (14) into Equation (13), the values of v _i can be obtained in turn: v ₂ =167, v ₃ =117, v ₄ =78, v ₅ =46, v ₆ =21. Starting from the bottom of the edge image of the lane line, set the horizontal scan line Line i (2≤i≤6) in the v _i line, in which the intersection of Line 5 and Line 6 and the left lane line is missing, and only 3 left lane line controls are preliminarily determined. The points are {(15,167),(39,117),(59,78)}; the five right lane control points are {(253,167),(213,117),(181,78),(155,46),( 134,21)}. According to equations (10)-(12), it is estimated that Δu ₅ =80, u ₅ =75; Δu ₆ =46, u ₆ =88. Therefore, it is determined that the coordinates of the two control points missing the left lane line are (75, 46), (88, 21) respectively. Then, according to the NUBS interpolation algorithm, the fourth-order polynomial function is used to perform NUBS interpolation, and the obtained fitting curve is displayed in the lane line image as shown in Figure 5.

It can be seen from Figures 3 and 5 that the lane line detection is carried out according to the above method, and better detection results are achieved. This example shows that the solution of the present invention can effectively improve the control point positioning accuracy and success rate of the non-uniform B-spline curve model, while taking into account the amount of calculation, it has better detection accuracy and real-time performance.

The specific embodiments of the present invention are given above. It should be noted that the present invention is not limited to the following specific embodiments, and all equivalent transformations made on the basis of the solution of the present application fall into the protection scope of the present invention.

Claims (7)

1. The method for detecting the lane line based on the imaging model constrained non-uniform B-spline curve fitting is characterized by comprising the following steps:

the method comprises the following steps: preprocessing an image;

acquiring an original lane line image I from a lane line standard image library in an intra-card Chiilong image database, performing median filtering on the original lane line image I to remove salt-pepper noise, performing histogram equalization to enhance the brightness and contrast of the image to make edge features prominent, and obtaining an enhanced lane line image I ₁ ；

Step two: detecting edges;

for the enhanced lane line image I ₁ Adopting a Canny operator to carry out edge detection to obtain an initial lane line edge image I ₂ ；

Step three: detecting a Hough straight line;

for the initial lane line edge image I obtained in the step two ₂ Performing Hough linear detection, reserving edges containing linear detection results, removing other interference edges to obtain an edge image I ₃ ；

Step four: deducing constraint conditions of the imaging model;

the derived imaging model constraints are as follows:

the length delta v of the line segment in the u-th column in the image coordinate system corresponding to the lane line segment with the length delta Y in the world coordinate system is as follows:

the pitch width Δ u in the v-th line in the image coordinate system corresponding to the left and right lane line pitch having the width Δ X in the world coordinate system is:

the concrete process of the step four is as follows:

the optical axis of the camera is assumed to be parallel to the plane of the driving road of the vehicle and parallel to the left lane line and the right lane line;

knowing a world coordinate system (X, Y, Z) and an image coordinate system (U, V), the maximum horizontal viewing angle of the camera is alpha, the maximum vertical viewing angle is beta, and the coordinates of the camera mounting position in the world coordinate system are C (d,0, h), where h is the camera mounting height, i.e., the value of the camera on the Z axis of the world coordinate system, and d is the camera mounting horizontal offset, i.e., the value of the camera on the X axis of the world coordinate system; the optical axis of the camera is parallel to the plane of the driving road of the vehicle, and the included angle between the optical axis of the camera and the lane line is gamma; according to the camera geometric imaging model, the mapping model of a certain point P (X, Y,0) on the road surface in the world coordinate system (X, Y, Z) and a relative coordinate point Q (U, V) in the image coordinate system (U, V) is:

in the formula H _I ，W _I Respectively the horizontal resolution and the vertical resolution of the image after the camera imaging;

according to the camera imaging principle, the length of the lane line segment in the imaged image is shortened along with the increase of the distance between the lane line segment and the camera in the world coordinate system, and similarly, the same left lane line and right lane line spacing on the road surface in the world coordinate system is wider in the lane line spacing obtained by imaging in the near vision field and narrower in the lane line spacing obtained by imaging in the far vision field; and deducing the length delta v of a line segment in the u-th column in an image coordinate system corresponding to the lane line segment with the length delta Y in the world coordinate system by combining a geometric camera imaging model, wherein the length delta v is as follows:

the pitch width Δ u in the v-th line in the image coordinate system corresponding to the left and right lane line pitch having the width Δ X in the world coordinate system is:

step five: extracting control points of the non-uniform B-spline curve;

setting scanning lines in an image according to the length of delta Y in a corresponding world coordinate system and the constraint of an imaging model, wherein the intersection point of each scanning line and the edges of the left lane and the right lane is a pair of control points; the specific process is as follows:

starting from the bottom of the lane line edge image at v _i Horizontal scanning lines Line i are arranged in rows, m is less than or equal to i and less than or equal to n, and control point pairs (Li, R) are obtained at the intersection points of the Line i and the left and right lane lines _i ) Wherein L is _i Has the coordinates of (u) _i ,v _i )，R _i The coordinate is (u) _i ',v _i ) (ii) a According to the imaging modelConstraint definition v _i The calculation formula is as follows:

derived from equations (8), (10), (11):

wherein v is ₁ ，Δv ₁ Is a preset value; v is obtained by substituting formula (14) for formula (13) _i A value of (d); thereby finding that the vertical coordinate of the control point determined by the ith scanning Line i is equal to v _i (ii) a Searching edge points from the middle point of the scanning line to the left and right sides respectively, and determining the coordinates (u) of a pair of control points by using the intersection points of the first pair of scanning lines and the left and right lane lines as the control points _i ,v _i ) And (u) _i ',v _i )；

Step six: fitting a lane line;

and fifthly, obtaining the non-uniform B-spline curve control point information, and fitting the left lane line and the right lane line by using a NUBS interpolation method to complete the detection of the lane lines.

2. The imaging model constraint-based non-uniform B-spline curve fitting lane line detection method according to claim 1, wherein in the step one, during median filtering, a median filtering function f ^ (x, y) is adopted as:

wherein f is ^{^} (x, y) is the median filtered output, S _xy Representing the set of coordinates of a rectangular sub-image window centered at (x, y) and of size M × N, and f (a, b) is the pixel gray scale value with coordinates (a, b).

3. According to the claimSolving 1 the method for detecting the lane line based on the imaging model constrained non-uniform B-spline curve fitting is characterized in that in the step one, when histogram equalization is carried out, a histogram equalization function s _k Comprises the following steps:

wherein s is _k For histogram equalized output, r _k Representing discrete gray levels, 0 ≦ r _k ≤255，k＝0,1,2,…,n-1，n _i For the occurrence of grey levels r in an image _i N is the total number of pixels in the image,

is the frequency in probability theory.

4. The imaging model constraint-based non-uniform B-spline curve fitting lane line detection method according to claim 1, wherein the specific steps of step two are as follows:

(1) smoothing image I with Gaussian filter ₁ ；

The gaussian smoothing function G (x, y) is:

using G (x, y) and the enhanced lane line image I ₁ Performing convolution to obtain a smooth image f ₁ ；

f ₁ (x,y)＝I ₁ (x,y)*G(x,y)(4)

(2) Calculating the amplitude and direction of the gradient by using finite difference of first-order partial derivatives to obtain a gradient image f ₂ ；

First order differential convolution template

(3) For gradient amplitudeThe non-maximum value is suppressed to obtain a non-maximum value suppressed image f ₃ ；

In the gradient image f ₂ Compares the central pixel S of the 8 neighbourhood with the two pixels along the gradient line at each point of (a); if the gradient value of S is not larger than the gradient values of two adjacent pixels along the gradient line, making S equal to 0;

(4) detecting and connecting edges by using a dual-threshold algorithm;

suppression of non-maximum images f ₃ Setting two thresholds T ₁ And T ₂ ，T ₁ ＝0.4T ₂ Making the gradient value less than T ₁ The gray value of the pixel (d) is set to 0 to obtain an image f ₄ (ii) a Then the gradient value is less than T ₂ The gray value of the pixel (d) is set to 0 to obtain an image f ₅ (ii) a With image f ₅ Based on the image f ₄ For supplement, the edges of the images are connected to obtain an initial lane edge image I ₂ 。

5. The imaging model constraint-based non-uniform B-spline curve fitting lane line detection method according to claim 1, characterized in that the concrete steps of step three are as follows:

(1) detecting a Hough straight line;

for any point A (x) in the rectangular coordinate system ₀ ,y ₀ ) Straight line passing through point A satisfies

y＝kx+l(5)

Where k is the slope and l is the intercept, then the point A (X) is crossed in the X-Y plane ₀ ,y ₀ ) The linear clusters of (a) are all expressed by formula (5), and cannot be expressed if the slope of a straight line perpendicular to the X axis is infinite; therefore, the rectangular coordinate system is converted into a polar coordinate system;

the equation representing a straight line in a polar coordinate system is

ρ＝xcosθ+ysinθ(6)

Wherein rho is the normal distance from the original point to the straight line, and theta is the positive included angle between the normal line and the X axis; one point in the image space corresponds to one sine curve in the polar coordinate system rho-theta; detecting a straight line in an image space by detecting an intersection point of the ρ - θ space; discretizing rho and theta, respectively calculating the value of the corresponding parameter rho according to a formula (6) at each value corresponding to the parameter theta, and then adding 1 to the corresponding parameter accumulation unit; finally, counting the value of each accumulation unit, and if the value is larger than a preset threshold value H, considering that the parameter is the parameter of a straight line in an image space, so as to mark the straight line in the image;

(2) removing interference edges;

searching and reserving the whole edge containing the pixel for each edge pixel in the straight line marked in the step (1), and eliminating the edge without common pixel points with the marked straight line so as to obtain an edge image I ₃ 。

6. The imaging model constraint-based non-uniform B-spline curve fitting lane line detection method according to claim 1, characterized in that the concrete process of step five is as follows:

assuming that the left lane Line and the right lane Line are parallel, solving the control point L determined by the ith scanning Line i _i 、R _i Abscissa u of _i And u' _i (ii) a Deducing delta u according to the constraint condition of the imaging model and the formula (14) _i+1 And Δ u _i The relation of (A) is as follows:

u’ _i ＝u _i +Δu _i (16)

the included angle gamma between the optical axis of the camera and the lane line is calculated according to the following formula:

for the case of control point loss due to lane line edge loss, L is paired with adjacent control points ₁ 、L ₂ When known, the abscissa u of the control point is calculated according to equations (15) to (17) ₂ (ii) a For the case of control point mislocation caused by false edges, it is verified whether all adjacent control point pair interval width ratios satisfyEquation (15), thereby detecting the erroneous control point coordinates and repositioning them according to equations (15) - (17).

7. The imaging model constraint-based non-uniform B-spline curve fitting lane line detection method according to claim 1, wherein the specific process of the sixth step is as follows:

suppose B-spline curve S is composed of n +1 control point sets { P } ₀ ,P ₁ ,...P _n And (c) then each point on the curve S satisfies:

wherein B is _i,m (o) is a basic B-spline function, m is more than or equal to 2 and less than or equal to n +1, t _min ≤u≤t _max ，t _j Is a node, j is 0.. times.i + m, when each node t is _j When the distance between the two splines is equal, the B spline curve is called as a uniform B spline curve, otherwise, the B spline curve is called as a non-uniform B spline curve; according to the NUBS interpolation method, if m is known to be more than or equal to 3 for the control points, the lane lines are fitted by adopting an m-1 order polynomial function; if 4 pairs of control points can be determined, carrying out NUBS interpolation by adopting a third-order polynomial function so as to fit out a lane line; if only 3 pairs of control points are determined, a second order polynomial function is used to fit the lane lines.

Images