CN116721396A - Lane line detection method, device and storage medium - Google Patents
Lane line detection method, device and storage medium Download PDFInfo
- Publication number
- CN116721396A CN116721396A CN202310739916.8A CN202310739916A CN116721396A CN 116721396 A CN116721396 A CN 116721396A CN 202310739916 A CN202310739916 A CN 202310739916A CN 116721396 A CN116721396 A CN 116721396A
- Authority
- CN
- China
- Prior art keywords
- lane line
- lane
- fitting
- points
- scene graph
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 94
- 238000000034 method Methods 0.000 claims abstract description 25
- 230000011218 segmentation Effects 0.000 claims abstract description 24
- 238000007499 fusion processing Methods 0.000 claims abstract description 8
- 238000012545 processing Methods 0.000 claims description 22
- 238000004590 computer program Methods 0.000 claims description 21
- 238000006243 chemical reaction Methods 0.000 claims description 16
- 230000002776 aggregation Effects 0.000 claims description 4
- 238000004220 aggregation Methods 0.000 claims description 4
- 238000012163 sequencing technique Methods 0.000 claims description 2
- 230000006870 function Effects 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000000605 extraction Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 230000002159 abnormal effect Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The application provides a lane line detection method, a lane line detection device and a storage medium, wherein the method comprises the following steps: inputting the acquired lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain first lane line information, wherein the lane line network detection model is used for outputting key points of lane lines in the lane line scene graph; inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information; and carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph. The lane line detection method provided by the application can improve the accuracy of lane line detection.
Description
Technical Field
The application relates to the technical field of intelligent driving, in particular to a lane line detection method, lane line detection equipment and a storage medium.
Background
With the development of intelligent driving technology, lane line detection has become one of important basic functions of automobile auxiliary driving and unmanned driving, and accurate detection and identification of lane lines are important preconditions for carrying out lane departure early warning, lane keeping, lane changing and other functions.
The current lane line detection method mainly comprises the steps of detecting through a semantic segmentation model, performing post-processing on a detected result, and finally performing fitting processing on the post-processed result to obtain a lane line.
However, the current detection method has the situations of losing the lane line and inaccurate prediction, and how to improve the detection accuracy of the lane line becomes a technical problem to be solved in the present technology.
Disclosure of Invention
The embodiment of the application provides a lane line detection method, a lane line detection device and a storage medium, which are used for solving the problem of low accuracy of current lane line detection.
In a first aspect, an embodiment of the present application provides a lane line detection method, including:
inputting the acquired lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain first lane line information, wherein the lane line network detection model is used for outputting key points of lane lines in the lane line scene graph;
inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information, wherein the lane line semantic segmentation model is used for classifying all pixel points in the lane line scene graph;
and carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph.
In one possible implementation manner, the fusing processing is performed on the first lane line information and the second lane line information, and the determining the lane line information corresponding to the lane line scene graph includes:
sequencing all the lane lines based on the distances between all the lane lines in the first lane line information and the second lane line information and the cameras for shooting the lane line scene graph under the world coordinate system;
when the distance between two adjacent lane lines is smaller than a first preset distance, determining the lane line to be deleted based on the respective confidence degrees of the two adjacent lane lines;
deleting the lane lines to be deleted from the first lane line information and the second lane line information to obtain lane line information corresponding to the lane line scene graph.
In one possible implementation, the confidence of each of the two adjacent lane lines includes the confidence of the number of points of each lane line at the point of fitting and/or the confidence of the fitting residual of the points.
In one possible implementation, the confidence of the fit residual for the points includes the confidence of the fit residual when the points in the lane line scene graph are fit and/or the confidence of the fit residual when the points in world coordinates are fit.
In one possible implementation manner, inputting the obtained lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information, including:
inputting the lane line scene graph into a lane line semantic segmentation model, and extracting the contour of the obtained lane line semantic label graph to obtain the contour of the lane line;
determining fitting points for fitting second lane line information based on the contour of the lane line and key points output by the lane line network detection model;
fitting the fitting point of the second lane line information subjected to world coordinate conversion to obtain second lane line information.
In one possible implementation, determining the fitting point for fitting the second lane line information based on the contour of the lane line and the key points output by the lane line network detection model includes:
merging the contours of the lane lines to obtain processed contour lines;
and setting a point when the positions of the pixel points on the processing contour line and the key points output by the lane line network detection model are overlapped as a fitting point of the second lane line information based on the pixel points on the processing contour line and the key points output by the lane line network detection model.
In one possible implementation manner, before determining the fitting point for fitting the second lane line information based on the contour of the lane line and the key point output by the lane line network detection model, the method further includes:
the contours of the lane lines are grouped, and whether the contours in the target group are merged into other groups is determined based on the coincidence degree of the contours in the target group and the contours in other groups in the Y direction and the distance between the contours in the target group and the fitting lines of the contours in other groups.
In one possible implementation manner, inputting the obtained lane line scene graph into a pre-trained lane line network detection model, and performing fitting processing on the obtained result to obtain first lane line information, including:
world coordinate conversion is carried out on the key points based on the coordinates of the key points and the first vanishing points, wherein the key points are output by the lane line network detection model, and the first vanishing points are aggregation points of lane lines on an image output by the lane line network detection model;
and fitting the key points subjected to world coordinate conversion to obtain first lane line information.
In a second aspect, an embodiment of the present application provides a lane line detection apparatus, including:
the first acquisition module is used for inputting the acquired lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain first lane line information, wherein the lane line network detection model is used for outputting key points of lane lines in the lane line scene graph;
the second acquisition module is used for inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, extracting and fitting the obtained result to obtain second lane line information, wherein the lane line semantic segmentation model is used for classifying all pixel points in the lane line scene graph;
the determining module is used for carrying out fusion processing on the first lane line information and the second lane line information and determining lane line information corresponding to the lane line scene graph.
In one possible implementation manner, the determining module is configured to sort all lane lines based on distances between all lane lines in the first lane line information and the second lane line information and a camera that shoots a lane line scene graph under a world coordinate system;
when the distance between two adjacent lane lines is smaller than a first preset distance, determining the lane line to be deleted based on the respective confidence degrees of the two adjacent lane lines;
deleting the lane lines to be deleted from the first lane line information and the second lane line information to obtain lane line information corresponding to the lane line scene graph.
In one possible implementation, the confidence of each of the two adjacent lane lines includes the confidence of the number of points of each lane line at the point of fitting and/or the confidence of the fitting residual of the points.
In one possible implementation, the confidence of the fit residual for the points includes the confidence of the fit residual when the points in the lane line scene graph are fit and/or the confidence of the fit residual when the points in world coordinates are fit.
In one possible implementation manner, the second obtaining module is configured to input the lane line scene graph into a lane line semantic segmentation model, and perform contour extraction on the obtained lane line semantic label graph to obtain a contour of a lane line;
determining fitting points for fitting second lane line information based on the contour of the lane line and key points output by the lane line network detection model;
fitting the fitting point of the second lane line information subjected to world coordinate conversion to obtain second lane line information.
In one possible implementation manner, the second obtaining module is configured to combine the contours of the lane lines to obtain a processed contour line;
and setting a point when the positions of the pixel points on the processing contour line and the key points output by the lane line network detection model are overlapped as a fitting point of the second lane line information based on the pixel points on the processing contour line and the key points output by the lane line network detection model.
In one possible implementation, the second obtaining module is configured to group the outlines of the lane lines, and determine whether the outlines in the target group are merged into other groups based on the overlap ratio of the outlines in the target group with the outlines in the other groups in the Y direction, and the distance between the outlines in the target group and the fitting lines of the outlines in the other groups.
In one possible implementation manner, the first obtaining module is configured to perform world coordinate conversion on the key points based on the coordinates of the key points and the first vanishing points output by the lane line network detection model, where the first vanishing points are aggregation points of lane lines on the image output by the lane line network detection model;
and fitting the key points subjected to world coordinate conversion to obtain first lane line information.
In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to the first aspect or any one of the possible implementations of the first aspect, when the computer program is executed by the processor.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method as described above in the first aspect or any one of the possible implementations of the first aspect.
The embodiment of the application provides a lane line detection method, a lane line detection device and a storage medium. And then, inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information. And finally, carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph.
According to the method, the obtained lane line scene graph is detected by adopting two different types of lane line detection models, and the results output by the two models are fused, so that the lane line can be more accurately determined, the problem that the lane line of a single lane line detection model is lost or inaccurate in prediction due to inaccurate data processing results in the later period can be avoided, and the stability, namely the safety, of automatic driving is improved.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of an implementation of a lane line detection method according to an embodiment of the present application;
FIG. 2 is a block diagram of a lane line detection method provided by an embodiment of the present application;
FIG. 3 is a block diagram of contour extraction provided by an embodiment of the present application;
FIG. 4 is a block diagram of contour merging provided by an embodiment of the present application;
fig. 5 is a schematic structural diagram of a lane line detection apparatus according to an embodiment of the present application;
fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
For the purpose of making the objects, technical solutions and advantages of the present application more apparent, the following description will be made by way of specific embodiments with reference to the accompanying drawings.
As described in the background art, at present, most of lane lines are detected by using a single lane line detection model, but inaccurate prediction, such as the problem of losing the lane line or adhering the lane line, occurs in the detection process, so that the accuracy of the lane line detection result is reduced, and therefore, the vehicle cannot be effectively positioned, and the running safety of the vehicle is ensured.
In order to solve the problems in the prior art, the embodiment of the application provides a lane line detection method, a lane line detection device and a storage medium. The lane line detection method provided by the embodiment of the application is first described below.
The application is applicable to both passenger vehicles and commercial vehicles, and is shown in fig. 1 and 2, which show a flowchart of the implementation of the method provided by the embodiment of the application, and is described in detail as follows:
step S110, inputting the acquired lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain first lane line information.
The lane line scene graph is a road scene graph containing lane lines, and can be obtained by shooting a front road by a camera or a radar device installed on a vehicle during the running process of the vehicle.
The lane line network detection model is used for outputting key points of lane lines in the lane line scene graph, and is trained by using a large number of lane line scene graphs marked with the key points.
In some embodiments, the first lane line information may be obtained based on a keypoint fit detected by the lane line network detection model. The specific steps are as follows:
step 1101, inputting the obtained lane line scene graph into a pre-trained lane line network detection model, and outputting key points on all lane lines in the lane line scene graph.
And step 1102, determining a pitch angle based on coordinates of key points and vanishing points output by the lane line network detection model.
Since the vehicle has a change in pitch during driving, it is necessary to determine the pitch angle before fitting the lane lines.
The pitch angle is determined by:
firstly, fitting key points in a lane line scene graph output by a lane line network detection model based on a least square method to obtain a fitting equation of lane lines in the lane line scene graph;
then, based on a fitting equation of the lane lines in the lane line scene graph, coordinates of the first vanishing point are determined. The vanishing point refers to the intersection point of two lane lines in the same direction on the image output by the lane line network detection model.
Finally, the pitch angle of the shooting equipment for shooting the lane line scene graph can be obtained based on the coordinates of the first vanishing point.
Step S1103, performing world coordinate transformation on the key points on all the lane lines based on the coordinates of the first vanishing point.
And step 1104, fitting the key points subjected to world coordinate conversion to obtain first lane line information.
Based on the steps S1101-S1104, the lane line information can be determined according to the key points of all the lane lines in the lane line scene graph output by the lane line network detection model.
And step S120, inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information.
The lane line semantic segmentation model is used for classifying all pixel points in the input image. If the pixels are classified into two classes in advance, judging whether the pixels belong to the lane lines or the background. For example, lane lines may be indicated in white and background in brown.
In some embodiments, lane line information in a lane line scene graph may be determined by a lane line semantic segmentation model based on the following steps:
step S1201, inputting the lane line scene graph into a lane line semantic segmentation model to obtain a lane line semantic tag graph.
The lane line semantic label graph comprises classification identifications of all pixel points in the lane line scene graph, namely, lane lines and backgrounds are classified by adopting different identifications.
Step S1202, traversing all pixel points on the lane line semantic tag map, and extracting the contour of the lane line in the lane line scene map based on the classification identifications of all pixel points and the position relations among all pixel points.
As shown in fig. 3, a specific traversal procedure is as follows:
traversing all pixel points from left to right and from bottom to top from the lower left corner of the semantic label graph, firstly judging whether the pixel point is a lane line according to the identification type of the current point, then determining whether the pixel point is a point on the same lane line according to the position relation between the current point and the existing contour, if the pixel point is adjacent to the existing contour, the pixel point belongs to the existing contour and is a lane line with the existing contour. If it is not adjacent to an existing contour, a new contour needs to be added, i.e. the point is on another lane. And finally, traversing all the points to determine the grouping of the contours, wherein the grouping of the contours corresponds to the contours of different lane lines.
Step S1203, merging and fitting the contours of the lane lines to obtain second lane line information.
In order to obtain the second lane line information, a plurality of points are selected from the contour of the lane line to be fitted, so as to obtain a lane line equation, and the specific steps are as follows:
step S12031, determining fitting points for fitting the second lane line information based on the contour of the lane line and the key points output by the lane line network model.
Since there are many points on the contour of the lane line, a plurality of points need to be extracted from the points of the contour for fitting, and in order to ensure the accuracy of fitting, the reliability of selecting fitting points must be ensured.
In some embodiments, the contours of the lane lines may first be combined to obtain a processed contour line. And then, setting a point when the pixel points on the processing contour line are overlapped with the positions of the key points output by the lane line network model as fitting points of the second lane line information based on the pixel points on the processing contour line and the key points output by the lane line network model.
In this embodiment, when the lane line semantic segmentation model is used to detect the obtained lane line scene graph when the road is broken or blocked, a partial contour may deviate from other normal contours when contour extraction is performed in step S1202, and the deviated contours need to be combined. The contours of the lane lines may be grouped, and whether the contours within the target group merge into other groups may be determined based on the degree of overlap in the Y direction of the contours within the target group with the contours within other groups, and the distance between the contours within the target group and the fitted lines of the contours within other groups. The target groupings are off-profile groupings.
As shown in fig. 4, a specific merging process is as follows:
the deviated outlines are set as one group, the outlines of other lane lines are respectively one group, and traversing processing is carried out on the groups deviated from the outlines.
For example, the maximum value and the minimum value of the longitudinal coordinates of the existing grouping profile are Y respectively 11 And Y 12 The maximum and minimum values of the longitudinal coordinates of the deviated profile are Y respectively 21 And Y 22 The method for calculating the overlap ratio may be based on Max (Y 21 ,Y 11 )-Min(Y 22 ,Y 12 ) And Y is equal to 11 –Y 12 +Y 21 –Y 22 The extent of the overlap is determined by the size of (a). Such as Max (Y) 21 ,Y 11 )-Min(Y 22 ,Y 12 ) Less than Y 11 –Y 12 +Y 21 –Y 22 It is indicated that the deviated profile coincides with the existing grouping profile in the Y direction and that it is impossible to place the deviated profile on one lane line, and that it is necessary to add one lane line profile grouping. If Max (Y) 21 ,Y 11 )-Min(Y 22 ,Y 12 ) Greater than Y 11 –Y 12 +Y 21 –Y 22 It is stated that the deviated contour does not coincide with the existing grouping contour in the Y direction, and that the deviated contour belongs to one of the existing grouping contours.
Then, it is determined which existing grouping profile the deviated profile belongs to by the distance from the point on the deviated profile to the existing grouping profile.
Finally, when the distance from the point on the deviated contour to the existing grouping contour is smaller than the preset distance, the deviated contour belongs to the grouping.
And step S12032, performing world coordinate conversion on the fitting point based on the fitting point of the second lane line information and the vanishing point on the lane line semantic label graph.
Fitting equations can be obtained based on fitting points of second lane line information in the lane line scene graph, and vanishing points on the lane line semantic tag graph can be obtained through the fitting equations.
And obtaining a vanishing point on the lane line semantic label graph, and obtaining a pitch angle according to the coordinate of the vanishing point, so as to perform world coordinate conversion on the fitting point of the second lane line information.
And step S12033, fitting the fitting point after world coordinate conversion to obtain second lane line information.
Based on the steps, the second lane line information determined based on the detection result of the lane line semantic segmentation model can be obtained.
And step S130, carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph.
When the road is blocked or damaged, the first lane line information and the second lane line information deviate, and abnormal lane lines need to be processed.
In some embodiments, the abnormal lane lines may be determined from the distance of each lane line from the camera that captured the lane line scene graph. The specific steps are as follows:
and S1301, sorting all the lane lines based on the distances between all the lane lines in the first lane line information and the second lane line information and the cameras for shooting the lane line scene graph under the world coordinate system.
Step S1302, when the distance between two adjacent lane lines is smaller than a first preset distance, determining the lane line to be deleted based on the respective confidence degrees of the two adjacent lane lines.
The confidence of the lane lines may be calculated from the points on each lane line.
In this embodiment, the confidence of each lane line may be determined according to the confidence of the number of points of each lane line at the point of fitting and/or the confidence of the fitting residual of the points.
Specifically, in order to improve accuracy, the confidence of each lane line may be evaluated as a comprehensive confidence according to the weighted value of the confidence of the number of each lane line at the fitting time point and the confidence of the fitting residual of the points.
For example, when the confidence coefficient of the point on each lane line is obtained, the number of the points selected during fitting of each lane line equation may be scored, the value is 0-100 minutes, for example, 0 points correspond to 0 minutes, the maximum points correspond to 100 minutes, and the confidence coefficient equation of the points may be obtained through linear fitting, so that the confidence coefficient of the points of each lane line may be obtained according to the points of each lane line equation during fitting.
The confidence of the fit residuals for the points includes the confidence of the residuals when fitting the points in the lane line scene graph and the confidence of the fit residuals when fitting the points at world coordinates. The confidence of the residual error when the points in the lane line scene graph are fitted and the confidence of the fitted residual error when the points in the world coordinates are fitted are both 0-100 points, the score when the fitted residual error is 0 is 100 points, the score when the fitted residual error is 1000 is 0 points, and the score greater than 1000 points is also 0 points. After linear fitting is carried out on the fitting residual errors of all the points, the confidence of the residual errors when the points in the lane line scene graph of each lane line are fitted and the confidence of the fitting residual errors when the points under the world coordinates are fitted can be obtained.
The final integrated confidence is a weighted sum of the confidence of the number of points of each lane line, the confidence of the residual errors when fitting the points in the lane line scene graph, and the confidence of the fitted residual errors when fitting the points under world coordinates.
And finally, determining the lane line with low comprehensive confidence as the lane line to be deleted according to the respective comprehensive confidence of the two adjacent lane lines.
And step S1303, deleting the lane lines to be deleted from the first lane line information and the second lane line information to obtain lane line information corresponding to the lane line scene graph.
And determining a lane line to be deleted according to step S1302, deleting the lane line from the first lane line information and the second lane line information, and finally obtaining lane line information corresponding to the lane line scene graph.
According to the lane line detection method, firstly, an obtained lane line scene graph is input into a pre-trained lane line network detection model, and fitting processing is carried out on the obtained result to obtain first lane line information. And then, inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information. And finally, carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph.
According to the method, the obtained lane line scene graph is detected by adopting two different types of lane line detection models, and the results output by the two models are fused, so that the lane line can be more accurately determined, the problem that the lane line of a single lane line detection model is lost or inaccurate in prediction due to inaccurate data processing results in the later period can be avoided, and the stability, namely the safety, of automatic driving is improved.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present application.
Based on the lane line detection method provided by the embodiment, correspondingly, the application further provides a specific implementation mode of the lane line detection device applied to the lane line detection method. Please refer to the following examples.
As shown in fig. 5, there is provided a lane line detection apparatus 500 including:
the first obtaining module 510 is configured to input the obtained lane line scene graph into a pre-trained lane line network detection model, and perform fitting processing on the obtained result to obtain first lane line information, where the lane line network detection model is used to output key points of lane lines in the lane line scene graph;
the second obtaining module 520 is configured to input the obtained lane line scene graph into a pre-trained lane line semantic segmentation model, and extract and fit the obtained result to obtain second lane line information, where the lane line semantic segmentation model is used to classify all pixel points in the lane line scene graph;
the determining module 530 is configured to perform fusion processing on the first lane line information and the second lane line information, and determine lane line information corresponding to the lane line scene graph.
In a possible implementation manner, the determining module 530 is configured to sort all lane lines based on distances between all lane lines in the first lane line information and the second lane line information and a camera that shoots a lane line scene graph under a world coordinate system;
when the distance between two adjacent lane lines is smaller than a first preset distance, determining the lane line to be deleted based on the respective confidence degrees of the two adjacent lane lines;
deleting the lane lines to be deleted from the first lane line information and the second lane line information to obtain lane line information corresponding to the lane line scene graph.
In one possible implementation, the confidence of each of the two adjacent lane lines includes the confidence of the number of points of each lane line at the point of fitting and/or the confidence of the fitting residual of the points.
In one possible implementation, the confidence of the fit residual for the points includes the confidence of the fit residual when the points in the lane line scene graph are fit and/or the confidence of the fit residual when the points in world coordinates are fit.
In a possible implementation manner, the second obtaining module 520 is configured to input the lane line scene graph into a lane line semantic segmentation model, and perform contour extraction on the obtained lane line semantic label graph to obtain a contour of a lane line;
determining fitting points for fitting second lane line information based on the contour of the lane line and key points output by the lane line network detection model;
fitting the fitting point of the second lane line information subjected to world coordinate conversion to obtain second lane line information.
In a possible implementation manner, the second obtaining module 520 is configured to combine the contours of the lane lines to obtain a processed contour line;
and setting a point when the positions of the pixel points on the processing contour line and the key points output by the lane line network detection model are overlapped as a fitting point of the second lane line information based on the pixel points on the processing contour line and the key points output by the lane line network detection model.
In one possible implementation, the second obtaining module 520 is configured to group the outlines of the lane lines, and determine whether the outlines in the target group are merged into other groups based on the overlap ratio of the outlines in the target group with the outlines in the other groups in the Y direction, and the distance between the outlines in the target group and the fitting lines of the outlines in the other groups.
In one possible implementation manner, the first obtaining module 510 is configured to perform world coordinate conversion on the key points based on the coordinates of the key points output by the lane line network detection model and the first vanishing points, where the first vanishing points are aggregation points of lane lines on the image output by the lane line network detection model;
and fitting the key points subjected to world coordinate conversion to obtain first lane line information. Fig. 6 is a schematic diagram of an electronic device according to an embodiment of the present application. As shown in fig. 6, the electronic device 6 of this embodiment includes: a processor 60, a memory 61 and a computer program 62 stored in said memory 61 and executable on said processor 60. The processor 60, when executing the computer program 62, implements the steps of the various lane line detection method embodiments described above, such as steps 110 through 130 shown in fig. 1. Alternatively, the processor 60, when executing the computer program 62, performs the functions of the modules in the apparatus embodiments described above, such as the functions of the modules 510-530 shown in fig. 5.
Illustratively, the computer program 62 may be partitioned into one or more modules that are stored in the memory 61 and executed by the processor 60 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of performing the specified functions, which instruction segments describe the execution of the computer program 62 in the electronic device 6. For example, the computer program 62 may be partitioned into modules 510 through 530 shown in FIG. 5.
The electronic device 6 may include, but is not limited to, a processor 60, a memory 61. It will be appreciated by those skilled in the art that fig. 6 is merely an example of the electronic device 6 and is not meant to be limiting as the electronic device 6 may include more or fewer components than shown, or may combine certain components, or different components, e.g., the electronic device may further include an input-output device, a network access device, a bus, etc.
The processor 60 may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field-programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or a memory of the electronic device 6. The memory 61 may be an external storage device of the electronic device 6, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card) or the like, which are provided on the electronic device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used for storing the computer program and other programs and data required by the electronic device. The memory 61 may also be used for temporarily storing data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/electronic device and method may be implemented in other manners. For example, the apparatus/electronic device embodiments described above are merely illustrative, e.g., the division of the modules or units is merely a logical function division, and there may be additional divisions in actual implementation, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by instructing related hardware by a computer program, where the computer program may be stored in a computer readable storage medium, and the computer program may implement the steps of the method embodiment of lane line detection method when executed by a processor. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (10)
1. A lane line detection method, characterized by comprising:
inputting the acquired lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain first lane line information, wherein the lane line network detection model is used for outputting key points of lane lines in the lane line scene graph;
inputting the acquired lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain second lane line information, wherein the lane line semantic segmentation model is used for classifying all pixel points in the lane line scene graph;
and carrying out fusion processing on the first lane line information and the second lane line information, and determining lane line information corresponding to the lane line scene graph.
2. The lane line detection method according to claim 1, wherein the fusing the first lane line information and the second lane line information to determine lane line information corresponding to the lane line scene graph includes:
sequencing all lane lines based on the distance between all lane lines in the first lane line information and the second lane line information and the distance between cameras for shooting the lane line scene graph under a world coordinate system;
when the distance between two adjacent lane lines is smaller than a first preset distance, determining the lane line to be deleted based on the respective confidence degrees of the two adjacent lane lines;
and deleting the lane lines to be deleted from the first lane line information and the second lane line information to obtain lane line information corresponding to the lane line scene graph.
3. The lane-line detection method according to claim 2, wherein the respective confidence levels of the adjacent two lane lines include a confidence level of the number of points at which each lane line is fitted and/or a confidence level of a fitting residual of the points.
4. The lane line detection method of claim 3 wherein the confidence of the fit residual for the points comprises the confidence of the fit residual when fitting the points in the lane line scene graph and/or the confidence of the fit residual when fitting the points in world coordinates.
5. The lane line detection method according to claim 1, wherein the inputting the obtained lane line scene graph into a pre-trained lane line semantic segmentation model, and extracting and fitting the obtained result to obtain the second lane line information comprises:
inputting the lane line scene graph into the lane line semantic segmentation model, and extracting the contour of the obtained lane line semantic label graph to obtain the contour of the lane line;
determining fitting points for fitting the second lane line information based on the contour of the lane line and key points output by the lane line network detection model;
fitting the fitting point of the second lane line information after world coordinate conversion to obtain the second lane line information.
6. The lane-line detection method according to claim 5, wherein the determining of the fitting point for fitting the second lane-line information based on the contour of the lane line and the key points output by the lane-line network detection model comprises:
merging the contours of the lane lines to obtain processed contour lines;
and setting a point when the pixel point on the processing contour line is overlapped with the position of the key point output by the lane line network detection model as a fitting point of the second lane line information based on the pixel point on the processing contour line and the key point output by the lane line network detection model.
7. The lane-line detection method according to claim 5 or 6, wherein before determining the fitting point for fitting the second lane-line information based on the contour of the lane line and the key point output by the lane-line network detection model, further comprising:
and grouping the outlines of the lane lines, and determining whether the outlines in the target group are merged into other groups or not based on the coincidence degree of the outlines in the target group and the outlines in other groups in the Y direction and the distance between the outlines in the target group and the fitting lines of the outlines in other groups.
8. The lane line detection method according to claim 1, wherein inputting the obtained lane line scene graph into a pre-trained lane line network detection model, and fitting the obtained result to obtain the first lane line information, comprises:
world coordinate conversion is carried out on the key points based on the coordinates of the key points and the first vanishing points, wherein the key points are output by the lane line network detection model, and the first vanishing points are aggregation points of lane lines on an image output by the lane line network detection model;
fitting the key points subjected to world coordinate conversion to obtain the first lane line information.
9. An electronic device comprising a memory for storing a computer program and a processor for invoking and running the computer program stored in the memory to perform the method of any of claims 1 to 8.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310739916.8A CN116721396A (en) | 2023-06-20 | 2023-06-20 | Lane line detection method, device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310739916.8A CN116721396A (en) | 2023-06-20 | 2023-06-20 | Lane line detection method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116721396A true CN116721396A (en) | 2023-09-08 |
Family
ID=87875016
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310739916.8A Pending CN116721396A (en) | 2023-06-20 | 2023-06-20 | Lane line detection method, device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116721396A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117576649A (en) * | 2023-12-26 | 2024-02-20 | 华东师范大学 | Lane line detection method and system based on segmentation points and dual-feature enhancement |
| CN117576649B (en) * | 2023-12-26 | 2024-04-30 | 华东师范大学 | Lane line detection method and system based on segmentation points and dual-feature enhancement |
-
2023
- 2023-06-20 CN CN202310739916.8A patent/CN116721396A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117576649A (en) * | 2023-12-26 | 2024-02-20 | 华东师范大学 | Lane line detection method and system based on segmentation points and dual-feature enhancement |
| CN117576649B (en) * | 2023-12-26 | 2024-04-30 | 华东师范大学 | Lane line detection method and system based on segmentation points and dual-feature enhancement |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Marzougui et al. | A lane tracking method based on progressive probabilistic Hough transform | |
| US11455805B2 (en) | Method and apparatus for detecting parking space usage condition, electronic device, and storage medium | |
| CN110163176B (en) | Lane line change position identification method, device, equipment and medium | |
| CN111295666A (en) | Lane line detection method, device, control equipment and storage medium | |
| CN112036385B (en) | Library position correction method and device, electronic equipment and readable storage medium | |
| CN114820679A (en) | Image annotation method and device, electronic equipment and storage medium | |
| Sahu et al. | A comparative analysis of deep learning approach for automatic number plate recognition | |
| CN113297939A (en) | Obstacle detection method, system, terminal device and storage medium | |
| CN111898540A (en) | Lane line detection method, lane line detection device, computer equipment and computer-readable storage medium | |
| CN116343148A (en) | Lane line detection method, device, vehicle and storage medium | |
| CN111160183A (en) | Method and device for detecting red light running of vehicle | |
| CN113591543B (en) | Traffic sign recognition method, device, electronic equipment and computer storage medium | |
| CN115249407B (en) | Indicator light state identification method and device, electronic equipment, storage medium and product | |
| CN116721396A (en) | Lane line detection method, device and storage medium | |
| CN114724119A (en) | Lane line extraction method, lane line detection apparatus, and storage medium | |
| CN113435350A (en) | Traffic marking detection method, device, equipment and medium | |
| CN115410105A (en) | Container mark identification method, device, computer equipment and storage medium | |
| CN114118188A (en) | Processing system, method and storage medium for moving objects in an image to be detected | |
| CN112257555A (en) | Information processing method, device, equipment and storage medium | |
| CN111639640A (en) | License plate recognition method, device and equipment based on artificial intelligence | |
| CN116503695B (en) | Training method of target detection model, target detection method and device | |
| CN112446375A (en) | License plate recognition method, device, equipment and storage medium | |
| CN112348105B (en) | Unmanned aerial vehicle image matching optimization method | |
| CN116152761B (en) | Lane line detection method and device | |
| CN113807293B (en) | Deceleration strip detection method, deceleration strip detection system, deceleration strip detection equipment and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |