CN116895060A - Three-dimensional lane marking method and device based on scene reconstruction - Google Patents
Three-dimensional lane marking method and device based on scene reconstruction Download PDFInfo
- Publication number
- CN116895060A CN116895060A CN202310846537.9A CN202310846537A CN116895060A CN 116895060 A CN116895060 A CN 116895060A CN 202310846537 A CN202310846537 A CN 202310846537A CN 116895060 A CN116895060 A CN 116895060A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- lane line
- lane
- data
- instance
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000011218 segmentation Effects 0.000 claims abstract description 45
- 238000002372 labelling Methods 0.000 claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000013500 data storage Methods 0.000 claims description 20
- 238000007781 pre-processing Methods 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 11
- 238000010276 construction Methods 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 238000012163 sequencing technique Methods 0.000 claims description 4
- 238000012790 confirmation Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 description 11
- 238000004590 computer program Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 5
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 230000001788 irregular Effects 0.000 description 2
- 239000000203 mixture Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 235000004522 Pentaglottis sempervirens Nutrition 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000003064 k means clustering Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a three-dimensional lane marking method and a device based on scene reconstruction, which relate to the technical field of point cloud data processing and comprise the steps of acquiring first laser point cloud data comprising a plurality of data sequences and two-dimensional image data acquired by a corresponding camera; acquiring a two-dimensional lane line semantic segmentation label based on the two-dimensional image data; two-dimensionally projecting the first laser point cloud data onto two-dimensional image data to obtain first semantic lane line point clouds; converting the first semantic lane line point cloud of the continuous frame from a vehicle coordinate system to a world coordinate system, and carrying out clustering treatment to obtain a first instance lane line point cloud; performing collinear fitting on the first instance lane line point cloud to obtain a second instance lane line point cloud; and two-dimensionally projecting the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud with consistent data sequences. The method has the advantages of dense lane line point cloud labeling results, high labeling precision and efficiency, strong consistency of sequence labeling results and the like.
Description
Technical Field
The invention relates to the technical field of point cloud data processing, in particular to a three-dimensional lane line labeling method and device based on scene reconstruction.
Background
Lane line detection is receiving increasing attention and importance in intelligent assisted driving awareness tasks. In order to improve safety redundancy at the vehicle end and limit of related regulations, intelligent auxiliary driving tends to adopt a route of heavy perception and light map, namely, the application of a high-precision map in auxiliary driving is avoided or weakened by improving the perception capability of an intelligent driving automobile on environmental information such as roads, vehicles, pedestrians and the like.
Marking three-dimensional lane lines and BEV (Bird's Eye View) lane lines by means of laser point clouds acquired by a laser radar is a main marking mode at present. At present, the scheme commonly used in the industry is to collect multi-mode data including laser point cloud, looking-around images, GPS, IMU, various conversion matrixes and the like by means of a data acquisition vehicle, and then semi-automatically label lane lines.
The reasons for the difficulty in marking the laser point cloud lane lines mainly comprise sparsity and irregular distribution of the laser point cloud, the difficulty in marking is caused by the lack of color and texture information, and the application requirement cannot be met by single-frame marking; secondly, the task of marking the lane lines is very challenging, the lane lines are generally of an elongated structure, the number of laser points scanned on each lane line is very rare, and the number of cloud data of the lane lines is very rare.
Disclosure of Invention
The invention provides a three-dimensional lane marking method and device based on scene reconstruction, which aims to solve the problems of inaccurate three-dimensional lane marking result, low marking efficiency, high marking cost, small number of lane point clouds and the like.
In a first aspect, the three-dimensional lane line labeling method based on scene reconstruction provided by the invention adopts the following technical scheme:
a three-dimensional lane marking method based on scene reconstruction comprises the following steps:
acquiring first laser point cloud data comprising a plurality of data sequences and two-dimensional image data acquired by a corresponding camera;
acquiring a two-dimensional lane line semantic segmentation label based on the two-dimensional image data;
two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain a first semantic lane line point cloud;
converting the first semantic lane line point cloud of the continuous frame from a vehicle coordinate system to a world coordinate system, and carrying out clustering treatment to obtain a first instance lane line point cloud;
performing collinear fitting on the first instance lane line point cloud to obtain a second instance lane line point cloud;
and carrying out two-dimensional projection on the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud with consistent data sequence.
Further, in the three-dimensional lane marking method based on scene reconstruction, the obtaining the first laser point cloud data including a plurality of data sequences and the corresponding two-dimensional image data includes:
collecting road data and establishing a self-collecting data set containing a plurality of data sequences;
and preprocessing the original laser point cloud in the self-acquisition data set to obtain the first laser point cloud data.
Further, in the three-dimensional lane line labeling method based on scene reconstruction, the two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain a first semantic lane line point cloud includes:
projecting the first laser point cloud data from a laser radar coordinate system to a camera coordinate system, so that the first laser point cloud data obtain lane line semantic segmentation labels, and a laser point cloud set with the lane line semantic segmentation labels is the first semantic lane line point cloud.
Further, in the three-dimensional lane line labeling method based on scene reconstruction, the converting the first semantic lane line point cloud of the continuous frame from the vehicle coordinate system to the world coordinate system, and performing clustering processing to obtain a first instance lane line point cloud includes:
projecting the first semantic lane line point cloud to a world coordinate system to obtain a second semantic lane line point cloud;
clustering the second semantic lane line point cloud, and dividing the second semantic lane line point cloud into a plurality of subsets with relatively close distances to obtain the first instance lane line point cloud.
Further, in the three-dimensional lane line labeling method based on scene reconstruction, the performing collinear fitting on the first example lane line point cloud to obtain a second example lane line point cloud includes:
performing curve fitting on each lane line in the first instance lane line point cloud;
comprehensively sequencing the fitted lane lines according to the length and the number of laser points;
judging whether the short line examples and the long line examples are collinear one by one, and if so, merging and updating fitting results;
obtaining a second instance lane line point cloud until merging cannot be achieved, and setting a unique identification code for each lane line point cloud in the second instance lane line point cloud.
In a second aspect, the invention further provides a three-dimensional lane line labeling device based on scene reconstruction, and the following technical scheme is adopted:
a three-dimensional lane marking device based on scene reconstruction, comprising:
the data acquisition and acquisition module acquires first laser point cloud data containing a plurality of data sequences and two-dimensional image data acquired by a corresponding camera;
the data storage module is used for storing the first laser point cloud data and the two-dimensional image data;
the lane line segmentation model training and marking module is used for training a two-dimensional lane line semantic segmentation model and marking lane lines on the two-dimensional image data by using the segmentation model to obtain lane line semantic segmentation labels;
the laser point cloud preprocessing module is used for preprocessing the collected first laser point cloud data;
the lane line point cloud semantic acquisition module is used for projecting the first laser point cloud data to a coordinate system where the two-dimensional image data are located, so as to obtain a first semantic lane line point cloud;
the lane line example construction module is used for processing the first semantic lane line point cloud to obtain a second example lane line point cloud;
and the laser point cloud re-projection module is used for projecting the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud.
Further, in the three-dimensional lane line labeling device based on scene reconstruction, the lane line instance construction module comprises a laser point cloud four-dimensional lane line reconstruction module, a laser point cloud clustering and fitting module and a laser point cloud collineation merging module;
the laser point cloud four-dimensional lane line reconstruction module is used for transforming the first semantic lane line point cloud into a world coordinate system to obtain a second semantic lane line point cloud;
the laser point cloud clustering and fitting module is used for clustering the second semantic lane line point cloud to obtain a first instance lane line point cloud;
and the laser point cloud collineation merging module is used for carrying out collineation judgment on the lane lines in the first instance lane line point cloud, carrying out collineation fitting on the collinear lane lines, and generating a second instance lane line point cloud.
Further, in the three-dimensional lane line labeling device based on scene reconstruction, the device further includes:
and the four-dimensional interactive display and correction module is used for displaying the labeling result for verification and confirmation of labeling personnel.
In a third aspect, the present invention further provides a readable storage medium, where the following technical solutions are sampled:
a readable storage medium storing computer instructions which when executed by a processor implement a three-dimensional lane marking method based on scene reconstruction as claimed in any one of the first aspects above.
In summary, the present invention includes at least one of the following beneficial technical effects:
1. according to the laser point cloud three-dimensional lane marking method based on scene reconstruction, the laser point cloud and the two-dimensional image data are fused, so that automatic marking of the three-dimensional lane is realized. The method has the advantages of dense lane line point cloud labeling results, high labeling precision and efficiency, strong consistency of sequence labeling results and the like;
2. in the labeling process, the labeled lane lines not only contain laser points of three-dimensional space positions, but also comprise all laser point clouds of different times. Therefore, the marking effect realizes four-dimensional scene reconstruction including time sequences, the lane lines marked based on the four-dimensional reconstruction scene are marked on continuous multi-frame point clouds, and the point cloud data volume of each lane line instance is greatly improved.
Drawings
FIG. 1 is a flow diagram of one embodiment of a three-dimensional lane marking method based on scene reconstruction of the present invention.
FIG. 2 is a block flow diagram of one embodiment of a three-dimensional lane marking method step S1 based on scene reconstruction according to the present invention.
FIG. 3 is a block flow diagram of one embodiment of a three-dimensional lane marking method step S2 based on scene reconstruction according to the present invention.
FIG. 4 is a block flow diagram of one embodiment of a three-dimensional lane marking method step S4 based on scene reconstruction according to the present invention.
FIG. 5 is a block flow diagram of one embodiment of a three-dimensional lane marking method step S5 based on scene reconstruction according to the present invention.
FIG. 6 is a schematic structural diagram of an embodiment of a three-dimensional lane marking apparatus based on scene reconstruction according to the present invention.
Reference numerals illustrate: 101. a data acquisition and acquisition module; 102. a data storage module; 103. the lane line segmentation model training and marking module; 104. a laser point cloud preprocessing module; 105. the lane line point cloud semantic acquisition module; 106. the laser point cloud four-dimensional lane line reconstruction module; 107. a laser point cloud clustering and fitting module; 108. a laser point cloud collinearly combining module; 109. the laser point cloud re-projection module; 110. and a four-dimensional interactive display and correction module.
Detailed Description
For more clearly illustrating the objects, technical solutions and advantages of the embodiments of the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The execution sequence of the method steps in the embodiments of the present invention may be performed according to the sequence described in the specific embodiments, or the execution sequence of each step may be adjusted according to actual needs on the premise of solving the technical problem, which is not listed here.
The reasons for the difficulty in marking the laser point cloud lane lines mainly comprise sparsity and irregular distribution of the laser point cloud, and lack of color and texture information; the marking is difficult, and the single-frame marking cannot meet the application requirements; secondly, the task of marking lane lines is very challenging, lane lines are generally elongated structures, and the number of laser points scanned onto each lane line is very rare.
Aiming at the problems of difficult guarantee of marking quality, low marking efficiency, high marking cost and the like of the current laser point cloud three-dimensional lane marking, the invention provides a three-dimensional lane marking method and device based on scene reconstruction by means of related technologies such as scene reconstruction, semantic segmentation, data processing and the like.
The invention is described in further detail below with reference to fig. 1-6.
The embodiment of the invention discloses a three-dimensional lane marking method based on scene reconstruction, which comprises the following steps of:
s1, acquiring first laser point cloud data comprising a plurality of data sequences and two-dimensional image data acquired by a corresponding camera;
s2, acquiring a two-dimensional lane line semantic segmentation label based on the two-dimensional image data;
s3, two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain first semantic lane line point clouds;
s4, converting the first semantic lane line point cloud of the continuous frame from a vehicle coordinate system to a world coordinate system, and carrying out clustering treatment to obtain a first instance lane line point cloud;
s5, performing collinear fitting on the first instance lane line point cloud to obtain a second instance lane line point cloud;
and S6, projecting the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud.
During the running process of the vehicle, first, a data sequence comprising first laser point cloud data and two-dimensional image data is acquired by using sensors such as a laser radar, a camera and the like, wherein each frame of data comprises one laser point cloud and one corresponding two-dimensional image. And then, segmenting the two-dimensional image by using a deep learning model to obtain the lane line semantic segmentation label. And then, projecting the two dimensions of the laser point cloud onto a corresponding two-dimensional image plane, and obtaining a first semantic lane line point cloud based on the lane line semantic segmentation labels. And then, performing three-dimensional projection, clustering and other processing on the first semantic lane line point cloud to obtain an instantiation representation of the lane line, and obtaining the first instance lane line point cloud. And then, carrying out collinear fitting on each lane line example in the lane line point cloud of the first example to obtain a lane line point cloud of the second example. And finally, projecting the second instance lane line point cloud onto each two-dimensional image to obtain a third instance lane line point cloud with consistent whole sequence.
According to the laser point cloud three-dimensional lane marking method based on scene reconstruction, the laser point cloud and the two-dimensional image data are fused, so that the three-dimensional automatic marking of the lane is realized. And after automatic marking, carrying out collinear fitting on the preliminarily marked lane lines to obtain a second instance lane line point cloud and a third instance lane line point cloud which are continuous and consistent in sequence. The method has the advantages of dense lane line point cloud labeling results, high labeling precision and efficiency, strong consistency of sequence labeling results and the like, and can be widely applied to the fields of automatic driving, intelligent traffic and the like.
Further, referring to fig. 2 as an embodiment of the present invention, step S1, acquiring the first laser point cloud data including a plurality of data sequences and the two-dimensional image data acquired by the corresponding cameras includes:
s11, collecting road data and establishing a self-collected data set containing a plurality of data sequences;
s12, preprocessing the original laser point cloud in the self-acquisition data set to obtain the first laser point cloud data.
Further, in step S11, the road data at least includes: laser point cloud data, image data of an all-around camera at corresponding time, a conversion matrix from a vehicle coordinate system to a laser radar coordinate system, a conversion matrix of the laser radar coordinate system and a camera coordinate system, an internal reference matrix of the all-around camera, a pose matrix for collecting vehicle quantity at each collecting time and the like, and collecting the road data to form the self-collection data set.
Optionally, in order to improve the labeling efficiency and ensure the labeling effect, data cutting processing can be performed according to a certain time length (for example, 30 seconds), namely, the self-sampling data set is divided into a plurality of self-sampling data sequences; before each data acquisition, the pose, the external parameter matrix and the like are calibrated and calibrated, and software synchronization is carried out on the trigger signals acquired by the laser radar and the camera.
Further, in step S12, the preprocessing method includes motion compensation, data rejection, and the like. The motion compensation preprocessing refers to that the laser radar performs distance measurement in a scanning mode, the scanning frequency is generally 10Hz, the acquisition vehicle runs at a certain speed, the acquisition time of each azimuth angle of a single-frame point cloud is different, the point cloud is distorted, and compensation is needed to reduce the influence of the point cloud on scene accuracy. The data rejection preprocessing refers to performing null value rejection, infinite value rejection, abnormal value statistical rejection and the like on the laser point cloud in sequence so as to reduce data noise and improve data quality.
Further, as an embodiment of the present invention, referring to fig. 3, step S2 of obtaining a two-dimensional lane line semantic segmentation tag based on the two-dimensional image data includes:
s21, determining a two-dimensional lane line segmentation model;
s22, training the two-dimensional lane line segmentation model based on a lane line self-acquisition data set and/or a public data set;
s23, carrying out semantic segmentation on the two-dimensional image data by using the trained lane line segmentation model to obtain the lane line semantic segmentation label.
Further, as an embodiment of the present invention, the two-dimensional lane line segmentation model may use a semantic segmentation model such as BiSeNet, internImage, HRNet.
Further, as an embodiment of the present invention, the two-dimensional lane line segmentation model may also use an example segmentation model such as LaneNet, oneFormer, panoptic-deep lab.
It should be noted that, the algorithm for performing segmentation based on the semantic segmentation model outputs pixels of all lane lines in the image, but cannot distinguish which pixels belong to the same lane line. The algorithm output for segmentation based on the above example segmentation model is lane line examples and corresponding pixels thereof, and the number of lane lines contained in each image and the pixel position of each lane line can be obtained, however, the examples only refer to single frame images, and it is also impossible to obtain exactly which lane lines in continuous frame images are the same examples.
Further, as an embodiment of the present invention, step S3 of two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain a first semantic lane line point cloud includes:
projecting the first laser point cloud data from a laser radar coordinate system to a camera coordinate system, so that the first laser point cloud data obtain lane line semantic segmentation labels, and a laser point cloud set with the lane line semantic segmentation labels is the first semantic lane line point cloud.
Alternatively, the first semantic lane line point cloud may be represented asK represents the total frame number of the laser radar point cloud in the sequence, < >>，N i Represents the total number of lane line semantics of the ith frame, [ x ] j y j z j ] T Representing the laser spot.
Further, referring to fig. 4 as an embodiment of the present invention, step S4 of converting a first semantic lane line point cloud of consecutive frames from a vehicle coordinate system to a world coordinate system and performing clustering processing to obtain a first instance lane line point cloud includes:
s41, three-dimensionally projecting the first semantic lane line point cloud to a world coordinate system to obtain a second semantic lane line point cloud;
s42, clustering the second semantic lane line point cloud, and dividing the second semantic lane line point cloud into a plurality of subsets with relatively close distances to obtain the first instance lane line point cloud.
Further, step S41 specifically includes: according to a transformation matrix from a laser radar coordinate system to a vehicle body coordinate system, the vehicle pose and the like, the first semantic lane line point cloud is transformed to a world coordinate system and is recorded as a second semantic lane line point cloud, and the second semantic lane line point cloud is a four-dimensional reconstruction scene containing lane lines.
Alternatively, the second semantic lane line point cloud may be specifically expressed as，N seq The total number of laser points representing the semantics of all lane lines in the whole self-acquired data sequence S laneseg-world Is>Not only comprises laser points in three-dimensional space positions, but also is a mixture of laser point clouds in different times, so that the method is a four-dimensional reconstruction scene, and lane lines marked on the basis of the four-dimensional reconstruction scene are marked on continuous multi-frame point clouds with consistent sequences.
Optionally, the clustering method in step S42 may be processed by using algorithms such as gaussian mixture model clustering, density-based spatial clustering, K-means clustering, and the like; after the above processing is completed, S can be laneseg-world Dividing the first instance lane line point cloud into a plurality of pseudo lane line instances, and recording the pseudo lane line instances as a first instance lane line point cloud; the pseudo lane line example is a very rough processing mode, and only rough division is performed, but if the two-dimensional lane line division model used in semantic division of the two-dimensional image data is an example division model, step S42 may be omitted, and the obtained second semantic lane line point cloud is the first example lane line point cloud after the first semantic lane line point cloud is transformed to the world coordinate system.
Alternatively, the first example lane line point cloud may be represented as，N laneinstance1 Laser point cloud set for representing number of clustered pseudo-examples and lane line pseudo-examples>Including a number of laser spots.
Further, as an embodiment of the present invention, referring to fig. 5, step S5, performing co-line fitting on the first example lane line point cloud to obtain a second example lane line point cloud includes:
s51, performing curve fitting on each lane line in the first instance lane line point cloud;
s52, comprehensively sequencing the fitted lane lines according to the length and the number of laser points;
s53, judging whether the short line examples and the long line examples are collinear one by one, and performing merging fitting if the short line examples and the long line examples are collinear; and obtaining a second instance lane line point cloud until the second instance lane line point cloud cannot be merged.
Alternatively, the curve fitting in step S51 may be performed by polynomial fitting, spline curve fitting, bezier curve fitting, or the like.
Alternatively, the comprehensive ordering in step S52 may take the euclidean distance measures of its two endpoints in the world coordinate system. Furthermore, in order to cope with the scene that the lane lines are curves, the quantity of point clouds in each lane line instance can be added to carry out weighted comprehensive sequencing.
Alternatively, the collineation in step S53 means that the two curves are relatively close. Specifically, the Euclidean distance average of all laser points in the short line to the long line fitting curve can be used for measuring the index. Because of the interference of curves and other factors, the threshold can be set below a certain range, i.e. the two lane line instances are considered to be collinear.
Optionally, a second example lane line point cloud S laneinstance2 Is represented by a first instance lane line point cloud S laneinstance1 In a similar fashion, with the difference that the number of instances N laneinstance2 Fewer, more accurate lane line examples. Generally, taking a 30s self-acquired data sequence as an example, the real world acquisition distance reaches about 500 meters, N laneinstance2 The number is generally between 10 and 50, N laneinstance1 The number is then related to the clustering method, typically between 200-3000.
Further, as a specific embodiment of the present invention, step S5, performing collinear fitting on the first example lane line point cloud to obtain a second example lane line point cloud, further includes:
s54, setting a unique identification code for each lane line point cloud in the lane line point cloud of the second example.
Specifically, the second example lane line point cloud S laneinstance2 Each N of (2) laneinstance2 Assigning unique identification codes throughout the self-acquired data sequence in order to ensure that each N laneinstance2 There is a unique identification code and a number or character string may be used as the identification code. Each lane line may be assigned an increasing integer as an identification code using a self-increasing number, e.g. starting from 1.
Optionally, the result of the second instance lane line point cloud may be displayed interactively after being visualized, so that the labeling personnel can confirm and correct the second instance lane line point cloud.
Further, the step S6 specifically includes: and projecting the lane line point cloud to each camera picture according to the relation of the pose matrix, the laser radar-acquisition vehicle transformation matrix, the camera internal and external parameter matrix and the like, and obtaining a third instance lane line point cloud with consistent whole sequence. The term (data acquisition) sequence is consistent, which means that any picture acquired by the looking-around camera at different moments belongs to the same lane line in the physical world, and in the image labeling result, the unique identifier is the same, specifically, any instance is included in the N of the lane line point cloud of the second instance laneinstance2 In one example; the consistency of the sequences can ensure the consistency of the examples of the looking-around cameras.
Furthermore, the labeling results of the first example lane line point cloud and/or the second example lane line point cloud and/or the third lane line example point cloud can be stored in a file format such as Json, xml, txt, so that subsequent reading and use are convenient.
Based on the three-dimensional lane marking method based on scene reconstruction, the embodiment of the invention also discloses a three-dimensional lane marking device based on scene reconstruction.
Referring to fig. 6, a three-dimensional lane marking device based on scene reconstruction includes a data acquisition and acquisition module 101, a data storage module 102, a lane segmentation model training and marking module 103, a laser point cloud preprocessing module 104, a lane point cloud semantic acquisition module 105, a lane instance construction module, and a laser point cloud re-projection module 109.
The data acquisition and acquisition module 101 is used to acquire radar data, image data, and other body data for calculation. Specifically, the module comprises a vehicle end and a cloud end part, wherein a data acquisition vehicle is generally provided with a laser radar, a looking-around camera, a GPS/IMU and the like, and the vehicle end part of the data acquisition module can collect acquisition information of all sensors; the cloud part of the data acquisition module receives the information stored in the vehicle end part on one hand and can download the lane line public data set from the public website on the other hand; the above data is stored in the data storage module 102.
The data storage module 102 is used for storing various data in the lane marking process. Specifically, the module is responsible for storing all data in the laser point cloud three-dimensional lane marking device based on scene reconstruction, including but not limited to collected laser point cloud, looking around images, lane line public data sets, depth pre-training models, depth model network models, laser point cloud and image intermediate processing results, marking result data, intermediate data in the software running process, log data and the like.
The lane line segmentation model training and labeling module 103 is used for training a two-dimensional lane line semantic segmentation model and carrying out lane line labeling on the two-dimensional image data by using the segmentation model to obtain lane line semantic segmentation labels. Specifically, the module reads the public data from the data storage module 102, completes the training of the lane line semantic segmentation model, and further uses the trained lane line semantic segmentation model to automatically label the looking-around picture in the self-acquired data, and the lane line segmentation label is stored in the data storage module 102.
The laser point cloud preprocessing module 104 is configured to preprocess the collected first laser point cloud data. Specifically, the module reads laser point cloud original data from the data storage module 102, and performs preprocessing operations such as motion compensation, outlier rejection and the like on the laser radar point cloud data.
The lane line point cloud semantic acquisition module 105 is configured to project the first laser point cloud data to a coordinate system where the two-dimensional image data is located, so as to obtain a first semantic lane line point cloud. Specifically, the module reads the first laser point cloud data and the related conversion matrix from the data storage module 102, completes the steps of lane line segmentation label projection and the like, obtains the first semantic lane line point cloud, and stores the first semantic lane line point cloud in the data storage module 102.
The lane line example construction module is used for processing the first semantic lane line point cloud to obtain a first example lane line point cloud. Specifically, the lane line instance construction module includes a laser point cloud four-dimensional lane line reconstruction module 106, a laser point cloud clustering and fitting module 107, and a laser point cloud collineation merging module 108.
The laser point cloud four-dimensional lane line reconstruction module 106 is configured to transform the first semantic lane line point cloud to a world coordinate system to obtain a second semantic lane line point cloud. Specifically, the module sequentially reads the first semantic lane line point cloud and the related conversion matrix of each sequence from the data storage module 102, completes the four-dimensional reconstruction steps of world system transformation and lane line point cloud, and stores the second semantic lane line point cloud into the data storage module 102.
The laser point cloud clustering and fitting module 107 is configured to cluster the second semantic lane line point cloud to obtain a first instance lane line point cloud. Specifically, the module reads the second semantic lane line point cloud from the data storage module 102, realizes point cloud clustering and fitting sorting, and stores the sorted first instance lane line point cloud into the data storage module 102.
The laser point cloud collineation merging module 108 is configured to perform collineation determination on the lane lines in the first instance lane line point cloud, and merge and fit the collinear lane lines to generate a second instance lane line point cloud. Specifically, the module reads a first instance lane line point cloud from the data storage module 102, performs collineation judgment and merging on lane line instance laser point clouds in the first instance lane line point cloud, and stores a second instance lane line point cloud in the data storage module 102.
The laser point cloud re-projection module 109 is configured to project the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud. Specifically, the module reads the second instance lane line point cloud from the data storage module 102, re-projects the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud, and stores the third instance lane line point cloud in the data storage module 102.
Further, the laser point cloud three-dimensional lane line labeling device based on scene reconstruction further comprises:
the four-dimensional interactive display and correction module 110 is configured to display the labeling result for verification and confirmation by a labeling person. Specifically, the module reads the second instance lane line point cloud and the third instance lane line point cloud from the data storage module 102, and displays the labeling result on the front-end device of the computer. The labeling personnel can check and confirm the labeling result through front-end interaction software.
The embodiment of the invention also discloses a computer readable storage medium.
A computer readable storage medium storing a computer program which when executed by a processor performs the steps of a three-dimensional lane marking method based on scene reconstruction as set forth in any one of the above embodiments. The computer readable storage medium may include: any entity or device capable of carrying a computer program, a recording medium, a USB flash disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a software distribution medium, and so forth. The computer program comprises computer program code. The computer program code may be in the form of source code, object code, executable files, or in some intermediate form, among others. The computer readable storage medium may include: any entity or device capable of carrying computer program code, a recording medium, a USB flash disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a Read-only memory (ROM), a random access memory (RAM, random Access Memory), a software distribution medium, and so forth.
Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and further implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, system that includes a processing module, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (9)
1. The three-dimensional lane marking method based on scene reconstruction is characterized by comprising the following steps of:
acquiring first laser point cloud data comprising a plurality of data sequences and two-dimensional image data acquired by a corresponding camera;
acquiring a two-dimensional lane line semantic segmentation label based on the two-dimensional image data;
two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain a first semantic lane line point cloud;
converting the first semantic lane line point cloud of the continuous frame from a vehicle coordinate system to a world coordinate system, and carrying out clustering treatment to obtain a first instance lane line point cloud;
performing collinear fitting on the first instance lane line point cloud to obtain a second instance lane line point cloud;
and carrying out two-dimensional projection on the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud with consistent data sequence.
2. The method for labeling three-dimensional lane lines based on scene reconstruction according to claim 1, wherein the acquiring first laser point cloud data including a plurality of data sequences and corresponding two-dimensional image data thereof comprises:
collecting road data and establishing a self-collecting data set containing a plurality of data sequences;
and preprocessing the original laser point cloud in the self-acquisition data set to obtain the first laser point cloud data.
3. The method for labeling three-dimensional lane lines based on scene reconstruction according to claim 1, wherein the two-dimensionally projecting the first laser point cloud data onto the two-dimensional image data to obtain a first semantic lane line point cloud comprises:
projecting the first laser point cloud data from a laser radar coordinate system to a camera coordinate system, so that the first laser point cloud data obtain lane line semantic segmentation labels, and a laser point cloud set with the lane line semantic segmentation labels is the first semantic lane line point cloud.
4. The method for labeling three-dimensional lane lines based on scene reconstruction according to claim 1, wherein the converting the first semantic lane line point cloud of the continuous frames from the vehicle coordinate system to the world coordinate system and performing clustering processing to obtain the first instance lane line point cloud comprises:
projecting the first semantic lane line point cloud to a world coordinate system to obtain a second semantic lane line point cloud;
clustering the second semantic lane line point cloud, and dividing the second semantic lane line point cloud into a plurality of subsets with relatively close distances to obtain the first instance lane line point cloud.
5. The three-dimensional lane marking method based on scene reconstruction according to claim 1, wherein said performing collinear fitting on the first instance lane line point cloud to obtain a second instance lane line point cloud comprises:
performing curve fitting on each lane line in the first instance lane line point cloud;
comprehensively sequencing the fitted lane lines according to the length and the number of laser points;
judging whether the short line examples and the long line examples are collinear one by one, and if so, merging and updating fitting results;
obtaining a second instance lane line point cloud until merging cannot be achieved, and setting a unique identification code for each lane line point cloud in the second instance lane line point cloud.
6. The utility model provides a three-dimensional lane line marking device based on scene rebuilds which characterized in that includes:
the data acquisition and acquisition module acquires first laser point cloud data containing a plurality of data sequences and two-dimensional image data acquired by a corresponding camera;
the data storage module is used for storing the first laser point cloud data and the two-dimensional image data;
the lane line segmentation model training and marking module is used for training a two-dimensional lane line semantic segmentation model and marking lane lines on the two-dimensional image data by using the segmentation model to obtain lane line semantic segmentation labels;
the laser point cloud preprocessing module is used for preprocessing the collected first laser point cloud data;
the lane line point cloud semantic acquisition module is used for projecting the first laser point cloud data to a coordinate system where the two-dimensional image data are located, so as to obtain a first semantic lane line point cloud;
the lane line example construction module is used for processing the first semantic lane line point cloud to obtain a second example lane line point cloud;
and the laser point cloud re-projection module is used for projecting the second instance lane line point cloud to the two-dimensional image data to obtain a third instance lane line point cloud.
7. The three-dimensional lane marking device based on scene reconstruction according to claim 6, wherein the lane instance construction module comprises a laser point cloud four-dimensional lane reconstruction module, a laser point cloud clustering and fitting module and a laser point cloud collineation merging module;
the laser point cloud four-dimensional lane line reconstruction module is used for transforming the first semantic lane line point cloud into a world coordinate system to obtain a second semantic lane line point cloud;
the laser point cloud clustering and fitting module is used for clustering the second semantic lane line point cloud to obtain a first instance lane line point cloud;
the laser point cloud collineation merging module is used for carrying out collineation judgment on the lane lines in the first instance lane line point cloud, carrying out collineation fitting on the collineation lane lines, and generating a second instance lane line point cloud.
8. The three-dimensional lane marking apparatus based on scene reconstruction as claimed in any one of claims 6 to 7, further comprising:
and the four-dimensional interactive display and correction module is used for displaying the labeling result for verification and confirmation of labeling personnel.
9. A readable storage medium storing computer instructions which when executed by a processor implement a three-dimensional lane marking method based on scene reconstruction as claimed in any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310846537.9A CN116895060A (en) | 2023-07-11 | 2023-07-11 | Three-dimensional lane marking method and device based on scene reconstruction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310846537.9A CN116895060A (en) | 2023-07-11 | 2023-07-11 | Three-dimensional lane marking method and device based on scene reconstruction |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116895060A true CN116895060A (en) | 2023-10-17 |
Family
ID=88313110
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310846537.9A Pending CN116895060A (en) | 2023-07-11 | 2023-07-11 | Three-dimensional lane marking method and device based on scene reconstruction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116895060A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117252992A (en) * | 2023-11-13 | 2023-12-19 | 整数智能信息技术(杭州)有限责任公司 | 4D road scene labeling method and device based on time sequence data and electronic equipment |
| CN118196747A (en) * | 2024-05-14 | 2024-06-14 | 纽劢科技(上海)有限公司 | 4D lane line marking method and device, electronic equipment and readable storage medium |
-
2023
- 2023-07-11 CN CN202310846537.9A patent/CN116895060A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117252992A (en) * | 2023-11-13 | 2023-12-19 | 整数智能信息技术(杭州)有限责任公司 | 4D road scene labeling method and device based on time sequence data and electronic equipment |
| CN117252992B (en) * | 2023-11-13 | 2024-02-23 | 整数智能信息技术(杭州)有限责任公司 | 4D road scene labeling method and device based on time sequence data and electronic equipment |
| CN118196747A (en) * | 2024-05-14 | 2024-06-14 | 纽劢科技(上海)有限公司 | 4D lane line marking method and device, electronic equipment and readable storage medium |
| CN118196747B (en) * | 2024-05-14 | 2024-08-13 | 纽劢科技(上海)有限公司 | 4D lane line marking method and device, electronic equipment and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111830953B (en) | Vehicle self-positioning method, device and system | |
| CN116895060A (en) | Three-dimensional lane marking method and device based on scene reconstruction | |
| CN110738121A (en) | front vehicle detection method and detection system | |
| CN114359181B (en) | Intelligent traffic target fusion detection method and system based on image and point cloud | |
| CN114413881B (en) | Construction method, device and storage medium of high-precision vector map | |
| CN115082674B (en) | Multi-mode data fusion three-dimensional target detection method based on attention mechanism | |
| CN104766058A (en) | Method and device for obtaining lane line | |
| US20230005278A1 (en) | Lane extraction method using projection transformation of three-dimensional point cloud map | |
| CN112419512B (en) | Air three-dimensional model repairing system and method based on semantic information | |
| Jiang et al. | Target detection algorithm based on MMW radar and camera fusion | |
| CN114359562A (en) | Automatic semantic segmentation and labeling system and method for four-dimensional point cloud | |
| CN116071747A (en) | 3D point cloud data and 2D image data fusion matching semantic segmentation method | |
| CN115909245A (en) | Visual multi-task processing method based on deep learning | |
| CN115761701A (en) | Laser radar point cloud data enhancement method, device, equipment and storage medium | |
| CN116597270A (en) | Road damage target detection method based on attention mechanism integrated learning network | |
| CN118379588A (en) | Detection method and device based on feature fusion, electronic equipment and storage medium | |
| CN116309943B (en) | Parking lot semantic map road network construction method and device and electronic equipment | |
| CN111626971B (en) | Smart city CIM real-time imaging method with image semantic perception | |
| CN112712098B (en) | Image data processing method and device | |
| CN112528918A (en) | Road element identification method, map marking method and device and vehicle | |
| CN112580489A (en) | Traffic light detection method and device, electronic equipment and storage medium | |
| CN116794650A (en) | Millimeter wave radar and camera data fusion target detection method and device | |
| Du | Lane line detection and vehicle identification using monocular camera based on matlab | |
| CN114359147A (en) | Crack detection method, crack detection device, server and storage medium | |
| CN111539279A (en) | Road height limit height detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |