CN115372990A - High-precision semantic map building method and device and unmanned vehicle - Google Patents
High-precision semantic map building method and device and unmanned vehicle Download PDFInfo
- Publication number
- CN115372990A CN115372990A CN202211039764.2A CN202211039764A CN115372990A CN 115372990 A CN115372990 A CN 115372990A CN 202211039764 A CN202211039764 A CN 202211039764A CN 115372990 A CN115372990 A CN 115372990A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- obstacle
- image
- information
- semantic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Electromagnetism (AREA)
- General Physics & Mathematics (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to the field of automatic driving, in particular to a high-precision semantic map building method, which comprises the following steps: acquiring original point cloud through the laser radar, and acquiring an image through the vehicle-mounted camera; projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud; acquiring detection information of a dynamic obstacle in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline; calculating the contact ratio of the projection point cloud and the two-dimensional contour; when the position coincidence degree is greater than a preset threshold value, filtering out the point cloud corresponding to the projection point cloud from the original point cloud; determining semantic information of the filtered residual point cloud based on the original image; and acquiring the high-precision semantic map according to the semantic information of the residual point cloud. The method can accurately remove the point cloud of the dynamic barrier in the point cloud map, establish a high-precision semantic map and realize high-precision positioning.
Description
Technical Field
The invention relates to the field related to unmanned driving technology, in particular to a high-precision semantic map construction method and device and an unmanned vehicle.
Background
With the rapid development of intelligent transportation, automatic driving has become the strategic development direction of the global automobile industry, and the mapping and positioning technology is one of the core technologies in the field of automatic driving. In an unmanned scene, firstly, an environment needs to be mapped, then a positioning function is realized in the built map, the mapping precision influences the positioning precision, and if only a low-precision map is built, high-precision positioning cannot be realized.
In the process of implementing the embodiment of the present invention, the inventors found that: the point cloud map is obtained by combining data obtained by scanning environments at different moments by the laser radar, but the point cloud of the dynamic obstacle in the point cloud map seriously affects the efficiency and quality of map building, so that the positioning accuracy is affected, and therefore the interference of the dynamic obstacle needs to be filtered when the map is built.
Disclosure of Invention
The implementation mode of the invention mainly solves the technical problems of accurately removing the point cloud of the dynamic barrier in the point cloud map and establishing the high-precision semantic map.
In view of the above problems, embodiments of the present invention provide a mapping method and apparatus for a high-precision semantic map, and an unmanned vehicle, which overcome or at least partially solve the above problems.
According to one aspect of the embodiment of the invention, a mapping method of a high-precision semantic map is provided and applied to an unmanned vehicle, and the method comprises the following steps: acquiring an original point cloud through the laser radar, and acquiring an image through the vehicle-mounted camera; projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud; acquiring detection information of a dynamic obstacle in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline; calculating the contact ratio of the projection point cloud and the two-dimensional contour; when the position contact ratio is greater than a preset threshold value, filtering out the point cloud corresponding to the projection point cloud from the original point cloud; determining semantic information of the filtered residual point cloud based on the original image; and acquiring the high-precision semantic map according to the semantic information of the residual point cloud.
Optionally, the projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud includes: identifying the original point cloud to obtain a ground point cloud and a non-ground point cloud; removing the ground point cloud, and reserving the non-ground point cloud as an obstacle point cloud; and projecting the obstacle point cloud into the image to obtain a projection point cloud.
Optionally, the projecting the obstacle point cloud into the image to obtain a projection point cloud includes: obtaining relevant parameters according to internal and external parameters of the calibration camera and the radar external parameter matrix; determining a mapping relation between a radar coordinate system and a camera coordinate system according to the related parameters; and mapping the obstacle point cloud into an image based on the mapping relation to obtain the projection point cloud.
Optionally, the acquiring the detection information of the dynamic obstacle in the image includes: acquiring a detection frame, category information and a two-dimensional outline of the obstacle based on an example segmentation network; judging a dynamic barrier and a static barrier according to the category information; and removing the detection information of the static obstacle and keeping the detection information of the dynamic obstacle.
Optionally, the obtaining, by the example-based segmentation network, the detection frame, the category information, and the two-dimensional contour of the obstacle includes: extracting the features of the image based on a convolutional neural network, and acquiring a high-level feature map and a low-level feature map; carrying out up-sampling on a high-level feature map through the pyramid network FPN, and carrying out top-down connection on the up-sampling result and a low-level feature map so as to carry out multi-scale prediction; wherein instance mask prediction is performed on an obstacle by the Head network at the time of the multi-scale prediction.
Optionally, the determining semantic information of the filtered residual point cloud based on the original image includes: projecting the filtered residual point cloud to the original image; and acquiring semantic information of pixels of the filtered residual point cloud in the original image, and setting the semantic information of the pixels as the semantic information of the filtered residual point cloud.
Optionally, the obtaining the high-precision semantic map according to the semantic information of the remaining point cloud includes: performing data association processing on the sensor data and the semantic information of the residual point cloud to obtain data; performing three-dimensional reconstruction based on the data to obtain a point cloud map; and vectorizing the point cloud according to the direction of the lane line in the point cloud map to obtain the high-precision semantic map.
According to another aspect of the embodiment of the invention, a high-precision semantic map construction device is provided, which is applied to an unmanned vehicle, wherein the unmanned vehicle is provided with a laser radar and a vehicle-mounted camera, and the high-precision semantic map construction device comprises: the original information acquisition module is used for acquiring an original point cloud through the laser radar and acquiring an image through the vehicle-mounted camera; the projection point cloud obtaining module is used for projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud; the detection information acquisition module is used for acquiring detection information of the dynamic barrier in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline; the coincidence degree calculation module is used for calculating the coincidence degree of the projection point cloud and the two-dimensional contour; the dynamic point cloud filtering module is used for filtering the point cloud corresponding to the projection point cloud from the original point cloud when the position coincidence degree is greater than a preset threshold value; the semantic processing module is used for determining semantic information of the filtered residual point cloud based on the original image; and the map building module is used for acquiring the high-precision semantic map according to the semantic information of the residual point cloud.
According to yet another aspect of an embodiment of the present invention, there is provided an unmanned vehicle including: a laser radar; a vehicle-mounted camera; at least one processor, and a memory, the lidar, and the onboard camera each communicatively coupled to the processor, the memory storing instructions executable by the at least one processor, the instructions being executable by the at least one processor to enable the at least one processor to perform the method as described above.
According to yet another aspect of embodiments of the present invention, there is provided a non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by an unmanned vehicle, cause the unmanned vehicle to perform the method as described above.
The method and the device for establishing the high-precision semantic map and the unmanned vehicle are different from the related technology, the original point cloud is obtained through the laser radar, and the image is obtained through the vehicle-mounted camera; projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud; acquiring detection information of a dynamic obstacle in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline; calculating the contact ratio of the projection point cloud and the two-dimensional contour; when the position contact ratio is greater than a preset threshold value, filtering out the point cloud corresponding to the projection point cloud from the original point cloud; determining semantic information of the filtered residual point cloud based on the original image; according to the semantic information of the residual point cloud, the high-precision semantic map is obtained, the point cloud of the dynamic barrier in the point cloud map can be accurately removed, the high-precision semantic map is built, and high-precision positioning is achieved.
Drawings
One or more embodiments are illustrated in drawings corresponding to, and not limiting to, the embodiments, in which elements having the same reference number designation may be represented as similar elements, unless specifically noted, the drawings in the figures are not to scale.
Fig. 1 is a flowchart of a method for creating a high-precision semantic map according to an embodiment of the present invention;
FIG. 2 is a flowchart of a method for obtaining a projection point cloud based on an original point cloud according to an embodiment of the present invention;
fig. 3 is a flowchart of a method for acquiring a projection point cloud based on an obstacle point cloud according to an embodiment of the present invention;
fig. 4 is a flowchart of a method for acquiring detection information of a dynamic obstacle according to an embodiment of the present invention;
fig. 5 is a flowchart of a method for obtaining detection information of an obstacle according to an embodiment of the present invention;
FIG. 6 is a flowchart of a method for determining semantic information of a filtered residual point cloud according to an embodiment of the present invention;
FIG. 7 is a flowchart of a method for obtaining a high-precision semantic map based on semantic information of a remaining point cloud according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of a map creating apparatus for a high-precision semantic map according to an embodiment of the present invention;
fig. 9 is a schematic view of an unmanned vehicle according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It should be noted that, if not conflicting, various features of the embodiments of the present invention may be combined with each other within the scope of the present invention. Additionally, while functional block divisions are performed in the device diagrams, with logical sequences shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than the block divisions in the device diagrams, or the flowcharts.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
The embodiment of the present invention provides a method for creating a high-precision semantic map, which is applied to an unmanned vehicle, please refer to fig. 1, where fig. 1 is a flowchart of the method for creating a high-precision semantic map according to the embodiment of the present invention, and includes:
s11, acquiring original point cloud through the laser radar, and acquiring an image through the vehicle-mounted camera.
And the laser radar continuously scans and collects the environment of the current position of the unmanned vehicle to obtain the original point cloud. The detection angle of the vehicle-mounted camera may be 120 degrees, and if a cloud-eye camera is used, the detection angle may be 180 degrees. The radar is divided into a top radar and a front radar, the top radar is mounted on a vehicle roof, the detection angle can be 360 degrees, the front radar is mounted below a headlamp, and the detection angle can be 270 degrees. The laser radar and the vehicle-mounted camera can be selected according to actual conditions, the detection angle is related to product parameters of the laser radar and the vehicle-mounted camera, and the laser radar and the vehicle-mounted camera are not limited by the disclosure.
And S12, projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud. The method converts the obstacle point cloud from a lidar coordinate system to a camera coordinate system, the camera coordinate system being a coordinate system of the vehicle-mounted camera. Referring to fig. 2, the projecting the original point cloud corresponding to the obstacle into the image to obtain a projected point cloud includes:
and S121, identifying the original point cloud to obtain a ground point cloud and a non-ground point cloud. The obstacle sensing is usually only interested in obstacles on the road, and the ground point cloud is easy to affect the obstacle clustering, so the ground point cloud and the non-ground point cloud are usually separated before the obstacle clustering. The obstacles are clustered by dividing the same or similar obstacles with different characteristic forms under one concept, and dividing the obstacles with different characteristic forms under different concepts, such as: the obstacles in the image are divided into a car and a person according to the original point cloud, and the car and the person have different characteristic forms, so that the obstacles belong to different concepts, and the car can be a car, a truck, a tricycle and the like.
And S122, removing the ground point cloud, and reserving the non-ground point cloud to be used as an obstacle point cloud. The removal of the ground point cloud can reduce the calculation amount and eliminate interference.
And S123, projecting the obstacle point cloud into the image to obtain a projection point cloud. The point cloud of the obstacle is in the laser radar coordinate system, the obstacle on the image acquired by the camera is in the camera coordinate system, and the point cloud data and the image are fused to convert the data into the same coordinate system. Referring to fig. 3, the projecting the obstacle point cloud into the image to obtain a projection point cloud includes:
and S1231, obtaining relevant parameters according to the internal and external parameters of the calibration camera and the radar external parameter matrix. The internal reference matrix of the camera is a parameter related to its own characteristics, and determines the projection position of the actual position of the obstacle on the imaging plane, for example: focal length, pixel size, etc. of the camera; the camera's external reference matrix is a transformation from the world coordinate system to the camera coordinate system, involving the pose of the camera in physical space, such as: mounting height, rotation direction, etc.; the external parameter matrix of the radar is the relative transformation relation of a radar coordinate system relative to a camera coordinate system. The internal reference calibration of the camera is to eliminate distortion generated when the camera takes a picture, so that the length and the width measured on the image are normal and correct. The external parameter calibration of the camera and the radar is mainly related to the relative position of the camera and the radar, the acquired data are deviated due to the difference of the installation positions of the camera and the radar, the difference can be reduced by calibrating the external parameter to perform coordinate conversion, and the data are converted into the same coordinate system.
And S1232, determining the mapping relation between the radar coordinate system and the camera coordinate system according to the relevant parameters. The coordinates of the radar coordinate system are three-dimensional coordinates, the coordinates of the camera coordinate system are two-dimensional coordinates, certain conversion rules exist when the three-dimensional coordinates are converted into the two-dimensional coordinates, and the conversion rules are the mapping relation.
And S1233, mapping the obstacle point cloud to an image based on the mapping relation, and acquiring the projection point cloud. Wherein the acquiring the obstacle projection point cloud comprises: and converting the point cloud from the laser radar coordinate system to a camera coordinate system, and converting the camera coordinate system to a pixel coordinate system to obtain a projection point cloud.
And S13, acquiring detection information of the dynamic barrier in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline. Wherein the two-dimensional profile is an integral body formed by connecting the edges of the dynamic barriers, and different dynamic barriers in different categories are endowed with different colors in the two-dimensional profile, such as: the two-dimensional outline of the common-class car A is red, and the two-dimensional outline of the common-class car B is yellow; the detection frame is a rectangular surrounding frame outside the dynamic barrier; the dynamic obstacles include, but are not limited to, occlusion type obstacles and small target obstacles, which are obstacles where the image features are not fully revealed, such as: one vehicle only exposes the head of the vehicle, and the rest is blocked, and the small target obstacles are defined differently, for example: the obstacle size, which is 0.1 of the original image size or an obstacle having a size smaller than 32 × 32 pixels, may be defined as the small target obstacle, which is not specifically limited by the present disclosure; the class information includes a class and a confidence level, which is a confidence level of an output class, and the example segmentation network may identify the obstacle as a plurality of classes when outputting a class with the highest confidence level, for example: the example segmentation network identifies an obstacle as a vehicle or a stone, the confidence that the obstacle is the vehicle is 0.97, the confidence that the obstacle is the stone is 0.74, outputs the type of the obstacle as the vehicle, and outputs the confidence that the type of the obstacle is the vehicle on the basis of the output of the type of the obstacle as the vehicle, wherein the confidence that the obstacle is the vehicle is 0.97. Referring to fig. 4, the acquiring the detection information of the dynamic obstacle in the image includes:
s131, acquiring a detection frame, category information and a two-dimensional outline of the obstacle based on the example segmentation network. The method comprises the following steps: carrying out up-sampling on a high-level feature map through the pyramid network FPN, and carrying out top-down connection on the up-sampling result and a low-level feature map so as to carry out multi-scale prediction; wherein instance mask prediction is performed on an obstacle by the Head network at the time of the multi-scale prediction. And the detection frame, the class information and the two-dimensional outline of the obstacle are simultaneously output through the example segmentation network. Referring to fig. 5, the obtaining of the detection frame, the category information, and the two-dimensional contour of the obstacle based on the example segmentation network includes:
s1311, extracting the features of the image based on the convolutional neural network, and obtaining a high-level feature map and a low-level feature map.
The convolutional neural network is a neural network used for extracting features in deep learning, the detection network yolox is adopted by the trunk network, and the backbone CSPDarkenet is used as a feature extractor to extract the features of the image, and the feature extractor has less calculated amount and parameter amount and can improve the running speed of the model. The feature extraction is divided into low-level feature extraction and high-level semantic feature extraction, the high-level semantic feature extraction generally focuses on features of semantic levels, such as human recognition, image classification and the like in a recognition task, the low-level feature extraction generally focuses on general features of colors, textures, shapes and the like of images, so that feature layers of multiple sizes are obtained, each size corresponds to one resolution, and multiple feature layers with different resolutions are obtained, the feature layers visually obtain a high-level feature map and a low-level feature map, the low-level feature map has detail features, and the high-level feature map has semantic information. Different feature layers that convolutional neural network drawed reduce by low to high size gradually, formed the characteristic pyramid naturally, the information abundance that the feature layer was drawed by low to high is different, and the semantic information that the feature layer was drawed more high more is more, and the feature layer reduces by low to high resolution ratio gradually, and the reception field crescent, low level feature can detect little target barrier, but the semantic information of low level feature is less, leads to detecting accuracy lower, high level feature has strong semantic information, can be used for categorised, nevertheless the reception field on high level feature layer is too big, is difficult to discern little target barrier.
S1312, performing up-sampling on the high-level feature map through the pyramid network FPN, and performing top-down connection on the up-sampling result and the low-level feature map to perform multi-scale prediction; wherein instance mask prediction is performed on an obstacle by the Head network at the time of the multi-scale prediction.
And the FPN network carries out multi-scale prediction on the basis of the characteristic pyramid, so that the detection precision of the small target obstacle is improved. The up-sampling is to increase the size of the high-level feature map, the multi-scale prediction is to fuse the high-level feature map to the low-level feature map, the high-level feature map is up-sampled to the size same as that of the low-level feature map, then the up-sampling is performed, the feature maps in multiple stages are fused together, and not only can the semantic features of the high-level feature map be extracted, but also the two-dimensional contour features of the low-level feature map can be extracted.
The Head network is a neural network that transforms the fused information into a final prediction result. The instance mask predicts the emphasis of the instance segmentation, and assigns a class to each pixel in the instance according to the location and size of the instance, thereby converting the instance segmentation well into a classifiable class problem. Inputting the fused information into the Head network to carry out category information analysis and example mask prediction, acquiring multiple kinds of mask information with different confidence degrees, screening and removing redundant mask information with lower confidence degrees, reserving mask information with the highest confidence degree and category information, endowing two-dimensional contour features with different colors for different examples in different categories, mapping the two-dimensional contour features back to an original image by corresponding characters to corresponding category names, and outputting the image with color mask information and category information.
And S132, judging the dynamic obstacle and the static obstacle according to the category information. And determining the person or object with potential dynamic or movable in the category information as the dynamic obstacle, and determining the person or object without potential dynamic or movable in the category information as the static obstacle. For example: the category information includes a vehicle and a stone block, and the vehicle is determined as the dynamic obstacle and the stone block is determined as the static obstacle regardless of whether the vehicle is in a moving or stationary state.
And S133, removing the detection information of the static obstacle and keeping the detection information of the dynamic obstacle. For constructing the high-precision semantic map, the dynamic obstacles are all interference information, such as: and vehicles and people need to filter the interference information to eliminate the interference of the dynamic barrier on the high-precision semantic map construction. The point cloud corresponding to the dynamic obstacle needs to be determined before the interference of the dynamic obstacle is filtered, the projection point cloud not only contains the dynamic obstacle but also contains a static obstacle, and the point cloud corresponding to the dynamic obstacle is difficult to be directly distinguished, so that the point cloud corresponding to the dynamic obstacle needs to be determined by retaining the detection information of the dynamic obstacle output by the example segmentation network.
And S14, calculating the coincidence degree of the projection point cloud and the two-dimensional contour. And calculating the coincidence degree of the projection point cloud and the two-dimensional outline according to the positions of the projection point cloud and the two-dimensional outline in a pixel coordinate system.
In the prior art, the coincidence degree calculation is performed by adopting the output detection frame and the projection point cloud, the detection frame introduces background information or the incompleteness of the detection frame of the shielding type obstacle can cause the point cloud projected on the image to be endowed with wrong semantic information, the example segmentation network can output a fine two-dimensional outline, and redundant background information can not be introduced.
And S15, when the position coincidence degree is greater than a preset threshold value, filtering out the point cloud corresponding to the projection point cloud from the original point cloud. The position contact ratio is greater than a preset threshold value, which represents that the projection point cloud corresponds to the same dynamic barrier as the two-dimensional contour, and the preset threshold value can be set according to actual conditions or manual operation, specifically can be 0.9, for example: the instance segmentation network outputs the type of the dynamic barrier as a vehicle, the position coincidence degree of the projection point cloud of the barrier and the two-dimensional outline is 0.93, the position coincidence degree is greater than a preset threshold value 0.9, and the type of the dynamic barrier is endowed as the vehicle. If the projection point cloud is not in the two-dimensional contour, the contact ratio of the projection point cloud and the two-dimensional contour is 0, the projection point cloud and the two-dimensional contour are not matched, and the obstacle corresponding to the projection point cloud and the dynamic obstacle corresponding to the two-dimensional contour are not the same obstacle. When the dynamic barrier exists in the environment, data generated by scanning of the sensor can be subjected to error matching, so that an inconsistent semantic map is created, positioning failure of the unmanned vehicle in a complex dynamic environment is caused, the point cloud corresponding to the dynamic barrier needs to be removed, and the semantic map is constructed by utilizing the point cloud in the residual static background environment.
And S16, determining semantic information of the filtered residual point cloud based on the original image. The original image and the filtered residual point cloud are acquired at the same time, and the semantic information of the filtered residual point cloud is the semantic information of the point cloud under the static background environment. Referring to fig. 6, the determining semantic information of the filtered residual point cloud based on the original image includes:
and S161, projecting the filtered residual point cloud to the original image. The method carries out data association processing on the original image and the filtered residual point cloud, so that each frame of point cloud has a frame of image corresponding to the frame of point cloud.
And S162, obtaining semantic information of pixels of the filtered residual point cloud in the original image, and setting the semantic information of the pixels as the semantic information of the filtered residual point cloud. If the point cloud is sparse, it is difficult to directly perform precise semantic segmentation on the point cloud, and the semantic information obtained based on the image is more accurate than the semantic information directly obtained based on the point cloud, so in this embodiment, the semantic information of the filtered remaining point cloud is determined by the original image with a semantic tag, and the semantic tag may be specifically semantic information of a pixel, for example: projecting the filtered residual point clouds into the original image, wherein each frame of point cloud corresponds to a pixel in the time-synchronized image, and further determining semantic information of the point clouds based on the semantic information of the pixels, specifically, the semantic information can be an RGB value of the pixel, and the RGB value of the pixel without a semantic label is 0.
And S17, acquiring the high-precision semantic map according to the semantic information of the residual point cloud. And performing three-dimensional reconstruction based on the semantic information of the residual point cloud to obtain a point cloud map, and vectorizing the point cloud in the point cloud map to obtain the high-precision semantic map. Referring to fig. 7, the obtaining the high-precision semantic map according to the semantic information of the remaining point cloud includes:
and S171, performing data association processing on the sensor data and the semantic information of the residual point cloud to obtain data. The sensor may in particular be a GPS sensor for locating the current position coordinates of the vehicle.
And S172, performing three-dimensional reconstruction based on the data to obtain a point cloud map. And the three-dimensional reconstruction is to determine the pose of each frame of point cloud through the associated data, and then combine all the frames of point cloud together based on the poses of the point cloud to obtain the point cloud map. Wherein the pose is a position and a pose of the point cloud under the coordinate system.
And S173, vectorizing the point cloud according to the direction of the lane line in the point cloud map to obtain the high-precision semantic map. In the high-precision semantic map, the point clouds are clustered to form point cloud clusters, and the point cloud clusters of different categories display different colors, for example, a vehicle displays red and a tree displays yellow. The direction of the lane line may be determined by an end point method, a random sampling consistency algorithm, and the like, which is not specifically limited by this disclosure. Processing the points in the point cloud cluster into vectorized lines according to the direction of the lane line, and adding semantic attributes to the lines according to the semantic labels of the points, such as: and the center point, the length and the like are obtained to obtain the high-precision semantic map.
In the embodiment of the invention, the ground point cloud is removed from the original point cloud obtained by the laser radar, the non-ground point cloud is projected on an image as the obstacle point cloud to obtain the obstacle projection point cloud, the image obtained by the vehicle-mounted camera outputs detection information through an example segmentation network, a dynamic obstacle is determined according to category information in the detection information, the two-dimensional outline of the dynamic obstacle is reserved, the coincidence degree of the two-dimensional outline of the dynamic obstacle and the projection point cloud is calculated, if the two-dimensional outline and the projection point cloud are greater than a preset threshold value, the same dynamic obstacle is determined to correspond to the two-dimensional outline and the projection point cloud, the point cloud corresponding to the projection point cloud is filtered out from the original point cloud, semantic information of the residual point cloud after filtering is determined based on the original image, the point cloud map is obtained through three-dimensional reconstruction, and the point cloud map is vectorized to obtain the high-precision semantic map. The embodiment of the invention can accurately remove the point cloud of the dynamic barrier in the point cloud map, establish the high-precision semantic map and realize high-precision positioning.
Referring to fig. 8, an embodiment of the present invention provides an apparatus 200 for creating a high-precision semantic map, where the apparatus 200 includes:
and the original information acquisition module 21 is configured to acquire an original point cloud through the laser radar and acquire an image through the vehicle-mounted camera.
And the projection point cloud obtaining module 22 is configured to project the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud.
And the detection information acquisition module 23 is configured to acquire detection information of a dynamic obstacle in the image, where the detection information includes a detection frame, category information, and a two-dimensional contour.
And the coincidence degree calculating module 24 is used for calculating the coincidence degree of the projection point cloud and the two-dimensional contour.
And the dynamic point cloud filtering module 25 is configured to filter the point cloud corresponding to the projection point cloud from the original point cloud when the position contact ratio is greater than a preset threshold.
And the semantic processing module 26 is used for determining semantic information of the filtered residual point cloud based on the original image.
And the map building module 27 is configured to obtain the high-precision semantic map according to the semantic information of the remaining point cloud.
It should be noted that the mapping device for a high-precision semantic map can execute the mapping method for a high-precision semantic map provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects of the execution method. Technical details which are not described in detail in the embodiment of the mapping device of the high-precision semantic map can be referred to the mapping method of the high-precision semantic map provided by the embodiment of the invention.
An embodiment of the present invention further provides an unmanned vehicle, please refer to fig. 9, which shows a hardware structure of an unmanned vehicle capable of executing the high-precision semantic map building method shown in fig. 1 to 7.
The unmanned vehicle 300 includes: a laser radar 31; a vehicle-mounted camera 32; at least one processor 33, and a memory 34, wherein the memory 34, the lidar 31, and the onboard camera 32 are respectively in communication connection with the processor 33, and the memory 34 stores instructions executable by the at least one processor 33, and the instructions are executed by the at least one processor 33, so that the at least one processor 33 can execute the mapping method of high-precision semantic maps described in the above embodiments.
The processor 33 and the memory 34 may be connected by a bus or other means, and in fig. 6, the memory 34 is taken as a non-volatile computer-readable storage medium for storing non-volatile software programs, non-volatile computer-executable programs, and modules, for example. The processor 33 executes various functional applications and data processing by running the nonvolatile software programs, instructions and modules stored in the memory 34, that is, implements the mapping method of the high-precision semantic map described in the above embodiments.
The memory 34 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function; the storage data area may store data created by use of the unmanned vehicle, and the like. Further, the memory 34 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some embodiments, memory 34 optionally includes memory 34 remotely located from processor 33. These remote memories 34 may be connected to the unmanned vehicle 300 via a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The one or more modules are stored in the memory 34 and, when executed by the one or more processors 33, perform the method of mapping high precision semantic maps of any of the embodiments described above, e.g., performing the method steps of fig. 1-7.
The product can execute the mapping method of the high-precision semantic map provided by the embodiment of the invention and is provided with a corresponding functional module for executing the mapping method of the high-precision semantic map. For technical details that are not described in detail in this embodiment, reference may be made to the method for creating a high-precision semantic map provided by the embodiment of the present invention.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a general hardware platform, and certainly can also be implemented by hardware. It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware related to instructions of a computer program, which can be stored in a computer readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; within the idea of the invention, also technical features in the above embodiments or in different embodiments may be combined, steps may be implemented in any order, and there are many other variations of the different aspects of the invention as described above, which are not provided in detail for the sake of brevity; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present application.
Claims (10)
1. A mapping method of a high-precision semantic map is applied to an unmanned vehicle, the unmanned vehicle is provided with a laser radar and a vehicle-mounted camera, and the mapping method is characterized by comprising the following steps:
acquiring original point cloud through the laser radar, and acquiring an image through the vehicle-mounted camera;
projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud;
acquiring detection information of a dynamic obstacle in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline;
calculating the contact ratio of the projection point cloud and the two-dimensional contour;
when the position contact ratio is greater than a preset threshold value, filtering out the point cloud corresponding to the projection point cloud from the original point cloud;
determining semantic information of the filtered residual point cloud based on the original image;
and acquiring the high-precision semantic map according to the semantic information of the residual point cloud.
2. The method of claim 1, wherein projecting the original point cloud corresponding to the obstacle into the image to obtain a projected point cloud comprises:
identifying ground point clouds and non-ground point clouds in the original point clouds;
removing the ground point cloud, and keeping the non-ground point cloud as an obstacle point cloud;
and projecting the obstacle point cloud into the image to obtain a projection point cloud.
3. The method of claim 2, wherein projecting the obstacle point cloud into the image results in a projected point cloud, comprising:
obtaining relevant parameters according to the internal and external parameters of the calibration camera and the radar external parameter matrix;
determining a mapping relation between a radar coordinate system and a camera coordinate system according to the related parameters;
and mapping the obstacle point cloud into an image based on the mapping relation, and acquiring the projection point cloud.
4. The method according to claim 1, wherein the obtaining of the detection information of the dynamic obstacle in the image comprises:
acquiring a detection frame, category information and a two-dimensional outline of the obstacle based on an example segmentation network;
judging a dynamic barrier and a static barrier according to the category information;
and removing the detection information of the static obstacle and keeping the detection information of the dynamic obstacle.
5. The method of claim 4, wherein the obtaining the detection frame, the class information and the two-dimensional profile of the obstacle based on the example segmentation network comprises:
extracting the features of the image based on a convolutional neural network, and acquiring a high-level feature map and a low-level feature map;
carrying out up-sampling on a high-level feature map through the pyramid network FPN, and carrying out top-down connection on the up-sampling result and a low-level feature map so as to carry out multi-scale prediction; wherein instance mask prediction is performed on an obstacle by the Head network at the time of the multi-scale prediction.
6. The method of claim 1, wherein determining semantic information of the filtered residual point cloud based on the original image comprises:
projecting the filtered residual point cloud to the original image;
and acquiring semantic information of pixels of the filtered residual point cloud in the original image, and setting the semantic information of the pixels as the semantic information of the filtered residual point cloud.
7. The method of claim 1, wherein the obtaining the high-precision semantic map according to the semantic information of the remaining point cloud comprises:
performing data association processing on the sensor data and the semantic information of the residual point cloud to obtain data;
performing three-dimensional reconstruction based on the data to obtain a point cloud map;
and vectorizing the point cloud according to the direction of the lane line in the point cloud map to obtain the high-precision semantic map.
8. The utility model provides a device of drawing of building of high accuracy semantic map, is applied to unmanned car, unmanned car is equipped with laser radar and on-vehicle camera, its characterized in that, the device of drawing of building of high accuracy semantic map includes:
the original information acquisition module is used for acquiring original point cloud through the laser radar and acquiring an image through the vehicle-mounted camera;
the projection point cloud acquisition module is used for projecting the original point cloud corresponding to the obstacle into the image to obtain a projection point cloud;
the detection information acquisition module is used for acquiring detection information of the dynamic barrier in the image, wherein the detection information comprises a detection frame, category information and a two-dimensional outline;
the coincidence degree calculation module is used for calculating the coincidence degree of the projection point cloud and the two-dimensional contour;
the dynamic point cloud filtering module is used for filtering the point cloud corresponding to the projection point cloud from the original point cloud when the position contact ratio is greater than a preset threshold value;
the semantic processing module is used for determining semantic information of the filtered residual point cloud based on the original image;
and the map construction module is used for acquiring the high-precision semantic map according to the semantic information of the residual point cloud.
9. An unmanned vehicle, comprising:
a laser radar;
a vehicle-mounted camera;
at least one processor, and a memory, the lidar, and the onboard camera each communicatively coupled to the processor, the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the method of any of claims 1-7.
10. A non-transitory computer-readable storage medium having stored thereon computer-executable instructions that, when executed by an unmanned vehicle, cause the unmanned vehicle to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211039764.2A CN115372990A (en) | 2022-08-29 | 2022-08-29 | High-precision semantic map building method and device and unmanned vehicle |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211039764.2A CN115372990A (en) | 2022-08-29 | 2022-08-29 | High-precision semantic map building method and device and unmanned vehicle |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115372990A true CN115372990A (en) | 2022-11-22 |
Family
ID=84069894
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211039764.2A Pending CN115372990A (en) | 2022-08-29 | 2022-08-29 | High-precision semantic map building method and device and unmanned vehicle |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115372990A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116088503A (en) * | 2022-12-16 | 2023-05-09 | 深圳市普渡科技有限公司 | Dynamic obstacle detection method and robot |
| CN116182831A (en) * | 2022-12-07 | 2023-05-30 | 北京斯年智驾科技有限公司 | Vehicle positioning method, device, equipment, medium and vehicle |
| CN116755441A (en) * | 2023-06-19 | 2023-09-15 | 国广顺能(上海)能源科技有限公司 | Obstacle avoidance method, device, equipment and medium of mobile robot |
| CN117237924A (en) * | 2023-11-13 | 2023-12-15 | 深圳元戎启行科技有限公司 | Obstacle visibility analysis method and device, intelligent terminal and storage medium |
-
2022
- 2022-08-29 CN CN202211039764.2A patent/CN115372990A/en active Pending
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116182831A (en) * | 2022-12-07 | 2023-05-30 | 北京斯年智驾科技有限公司 | Vehicle positioning method, device, equipment, medium and vehicle |
| CN116088503A (en) * | 2022-12-16 | 2023-05-09 | 深圳市普渡科技有限公司 | Dynamic obstacle detection method and robot |
| CN116755441A (en) * | 2023-06-19 | 2023-09-15 | 国广顺能(上海)能源科技有限公司 | Obstacle avoidance method, device, equipment and medium of mobile robot |
| CN116755441B (en) * | 2023-06-19 | 2024-03-12 | 国广顺能(上海)能源科技有限公司 | Obstacle avoidance method, device, equipment and medium of mobile robot |
| CN117237924A (en) * | 2023-11-13 | 2023-12-15 | 深圳元戎启行科技有限公司 | Obstacle visibility analysis method and device, intelligent terminal and storage medium |
| CN117237924B (en) * | 2023-11-13 | 2024-03-29 | 深圳元戎启行科技有限公司 | Obstacle visibility analysis method and device, intelligent terminal and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107507167B (en) | Cargo tray detection method and system based on point cloud plane contour matching | |
| CN115372990A (en) | High-precision semantic map building method and device and unmanned vehicle | |
| CN112581612B (en) | Vehicle-mounted grid map generation method and system based on fusion of laser radar and all-round-looking camera | |
| CN112740225B (en) | Method and device for determining road surface elements | |
| US10867403B2 (en) | Vehicle external recognition apparatus | |
| CN112287860A (en) | Training method and device of object recognition model, and object recognition method and system | |
| KR102363719B1 (en) | Lane extraction method using projection transformation of 3D point cloud map | |
| CN115457358A (en) | Image and point cloud fusion processing method and device and unmanned vehicle | |
| CN114792416A (en) | Target detection method and device | |
| CN112907626A (en) | Moving object extraction method based on satellite time-exceeding phase data multi-source information | |
| CN115327572A (en) | Method for detecting obstacle in front of vehicle | |
| CN114639085A (en) | Traffic signal lamp identification method and device, computer equipment and storage medium | |
| CN107220632B (en) | Road surface image segmentation method based on normal characteristic | |
| CN113219472B (en) | Ranging system and method | |
| CN114445793A (en) | Intelligent driving auxiliary system based on artificial intelligence and computer vision | |
| CN114118247A (en) | Anchor-frame-free 3D target detection method based on multi-sensor fusion | |
| CN112529011A (en) | Target detection method and related device | |
| KR20170106823A (en) | Image processing device identifying object of interest based on partial depth map | |
| CN111191482A (en) | Brake lamp identification method and device and electronic equipment | |
| Barua et al. | An Efficient Method of Lane Detection and Tracking for Highway Safety | |
| CN112733678A (en) | Ranging method, ranging device, computer equipment and storage medium | |
| CN114841910A (en) | Vehicle-mounted lens shielding identification method and device | |
| CN117970341A (en) | Mining area electric shovel cable detection method and system based on multi-data fusion | |
| CN113611008B (en) | Vehicle driving scene acquisition method, device, equipment and medium | |
| CN114332814A (en) | Parking frame identification method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |