CN115421158A - Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device - Google Patents

Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device Download PDF

Info

Publication number
CN115421158A
CN115421158A CN202211387608.5A CN202211387608A CN115421158A CN 115421158 A CN115421158 A CN 115421158A CN 202211387608 A CN202211387608 A CN 202211387608A CN 115421158 A CN115421158 A CN 115421158A
Authority
CN
China
Prior art keywords
point cloud
dimensional
semantic
state
self
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202211387608.5A
Other languages
Chinese (zh)
Other versions
CN115421158B (en
Inventor
陈昶昊
李方言
王雄飞
何晓峰
褚超群
潘献飞
毛军
张礼廉
范晨
胡小平
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
National University of Defense Technology
Original Assignee
National University of Defense Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by National University of Defense Technology filed Critical National University of Defense Technology
Priority to CN202211387608.5A priority Critical patent/CN115421158B/en
Publication of CN115421158A publication Critical patent/CN115421158A/en
Application granted granted Critical
Publication of CN115421158B publication Critical patent/CN115421158B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00—Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/38—Electronic maps specially adapted for navigation; Updating thereof
    • G01C21/3804—Creation or updating of map data
    • G01C21/3833—Creation or updating of map data characterised by the source of data
    • G01C21/3841—Data obtained from two or more sources, e.g. probe vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/86—Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
    • Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Remote Sensing (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Biomedical Technology (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Electromagnetism (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Automation & Control Theory (AREA)
  • Optics & Photonics (AREA)
  • Optical Radar Systems And Details Thereof (AREA)

Abstract

The invention discloses a solid-state laser radar three-dimensional semantic map building method and device for self-supervised learning. The light-weight laser radar/inertia combined mapping algorithm based on Kalman filtering carries out scene three-dimensional reconstruction, and the three-dimensional point cloud map is semantically segmented through the point cloud semantic segmentation model, so that the semantic segmentation effect on the three-dimensional coordinate point cloud is good, and the point cloud segmentation understanding capability and the semantic segmentation precision are improved.

Description

Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device
Technical Field
The invention relates to the technical field of three-dimensional point cloud semantic segmentation, in particular to a solid-state laser radar three-dimensional semantic mapping method and device for self-supervised learning.
Background
Semantic segmentation is a typical computer vision problem that involves taking some raw data (e.g., flat images) as input and converting them into segmented regions with semantic meaning, forming a semantic map that facilitates understanding by robots and humans. The appearance of lightweight solid-state lidar has reduced lidar's cost and weight size, and is different with rotation type lidar, and small-size solid-state lidar's the angle of vision is less, and the scanning mode is irregular. The invention relates to a method and a device for constructing a graph aiming at dense three-dimensional point cloud semantics of a light and small solid-state laser radar. The method and apparatus may be used for mobile robots as well as hand-held devices.
Along with the rapid popularization of robots, the application of logistics distribution, household robots, medical robots and the like requires that the robots can independently navigate and establish images in various indoor and outdoor complex environments. The execution of the robot task needs to have the capabilities of autonomous navigation, map building and scene understanding. The laser radar can directly acquire information such as three-dimensional geometric position and reflection intensity of a target, the measuring distance can reach hundreds of meters, and the measuring precision is high. However, it is not enough for an indoor robot autonomous navigation to obtain only a three-dimensional point cloud map. The human body can distinguish roads, walls and other sundries from the point cloud map, but for the robot, the point cloud map in the eyes is a group of points with unknown meanings. And when the robot is required to effectively identify objects and regions with semantic meanings in the point cloud, semantic segmentation and image building are required. In recent years, the rapid increase of computational power enables the three-dimensional point cloud semantic segmentation algorithm based on deep learning to rapidly develop and improve the performance, and the research on the aspect also becomes a research hotspot in recent years.
The three-dimensional semantic mapping can be divided into two steps of semantic segmentation understanding and three-dimensional dense mapping of three-dimensional point cloud. In recent years, relevant research results focus on two aspects of three-dimensional point cloud mapping and point cloud semantic segmentation. In the aspect of three-dimensional point cloud mapping, zhang et al provides a laser radar odometer and mapping algorithm based on batch optimization, and real-time three-dimensional point cloud mapping is realized. Lin et al further proposes a positioning and mapping algorithm suitable for small solid-state laser radars, and realizes feature extraction under a limited field angle and motion compensation and point cloud matching under an irregular sampling condition. And Xu et al, fusing the point cloud characteristics and inertial vision of the laser radar by adopting extended Kalman filtering, and realizing positioning and mapping with higher efficiency and robustness. In the aspect of three-dimensional point cloud matching, qi et al. And (3) projecting the point cloud into a strip picture, performing semantic segmentation on the picture and mapping the point cloud back to realize the construction and segmentation of the point cloud, wherein the algorithm is only suitable for the rotary laser radar. At present, a three-dimensional dense semantic mapping method for small solid-state lidar does not exist.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a solid-state laser radar three-dimensional semantic mapping method and device for self-supervision learning, which can be applied to mobile robots and handheld devices. The method constructs a coding-decoding deep neural network, semantically segments real-time input three-dimensional point cloud based on a self-supervision learning mode, and adopts Kalman filtering to fuse laser radar point cloud data, semantic segmentation results and inertial data to construct a map so as to realize scene three-dimensional reconstruction. In addition, the reflectivity of the laser radar signal is introduced into a point cloud segmentation network of self-supervision learning, and therefore the point cloud segmentation understanding capability is effectively improved.
In order to achieve the purpose, the invention provides a solid-state laser radar three-dimensional semantic mapping method for self-supervision learning, which comprises the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and carrying out self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time inertia data through a small solid-state laser radar and an inertia measurement element, fusing the real-time three-dimensional point cloud and the real-time inertia data based on a Kalman filter, and outputting a dense three-dimensional point cloud map composed of the real-time point cloud data under global coordinates;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on a computing device with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, performing three-dimensional point cloud and inertia data acquisition and map updating according to fixed frequency, performing real-time semantic segmentation, and generating a three-dimensional semantic map. Because the point cloud segmentation network is trained, the RGB images do not need to be synchronously acquired.
In order to achieve the above object, the present invention further provides a solid-state lidar three-dimensional semantic graph building apparatus for self-supervised learning, comprising:
the system comprises an RGB camera, a laser radar and an inertia measuring device, wherein the RGB camera, the laser radar and the inertia measuring device are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring RGB images, three-dimensional point cloud data and inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertial data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
and the three-dimensional semantic map building module is connected with the point cloud semantic division model and the three-dimensional point cloud map building module and is used for generating a three-dimensional semantic map by corresponding the point cloud division result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate.
The invention provides a self-supervised learning solid-state laser radar three-dimensional semantic mapping method and device. In addition, the reflectivity of the laser radar signal is introduced into the point cloud segmentation network, so that the point cloud segmentation understanding capability is improved.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
FIG. 1 is a flow chart of a solid-state lidar three-dimensional semantic mapping method in an embodiment of the invention;
fig. 2 is a block diagram of a solid-state lidar three-dimensional semantic graph establishing device in the embodiment of the invention.
The implementation, functional features and advantages of the present invention will be further described with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive step based on the embodiments of the present invention, are within the scope of protection of the present invention.
In addition, the descriptions related to "first", "second", etc. in the present invention are only for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise.
In addition, the technical solutions in the embodiments of the present invention may be combined with each other, but it must be based on the realization of the technical solutions by those skilled in the art, and when the technical solutions are contradictory to each other or cannot be realized, such a combination of the technical solutions should not be considered to exist, and is not within the protection scope of the present invention.
Example 1
Fig. 1 shows a method for building a three-dimensional semantic map of a solid-state lidar for self-supervised learning, which includes the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and performing self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time Inertial data through a small solid-state laser radar and an Inertial Measurement Unit (IMU), fusing the real-time three-dimensional point cloud and the real-time Inertial data based on a Kalman filter, and outputting a dense three-dimensional point cloud map composed of the real-time point cloud data under the global coordinate;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on a computing device with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, performing three-dimensional point cloud and inertia data acquisition and map updating according to the 50Hz frequency, performing real-time semantic segmentation, and generating a three-dimensional semantic map.
In this embodiment, the point cloud semantic segmentation model includes an encoding layer, a decoding layer, and a semantic prediction layer. The point cloud semantic segmentation model is input as
Figure 950697DEST_PATH_IMAGE001
The point cloud set of (a) is,
Figure 868975DEST_PATH_IMAGE002
the number of the dots is the number of the dots,
Figure 238776DEST_PATH_IMAGE003
for the characteristic dimension of each point, 4 dimensions in this embodiment, including three-dimensional coordinates
Figure 473579DEST_PATH_IMAGE004
And reflectivity
Figure 801793DEST_PATH_IMAGE005
。
The coding layer adopts a 3-layer coder and is mainly used for carrying out feature extraction on input three-dimensional point clouds, sequentially processing point cloud data, reducing the number of the point clouds, increasing the dimension of each point and finally counting the number of the point clouds
Figure 941787DEST_PATH_IMAGE006
Is reduced to
Figure 397170DEST_PATH_IMAGE007
And promoting the dimension of the point cloud feature from 4 dimensions to 128 dimensions. The encoder consists of a local feature aggregator and a random sampling layer, and specifically comprises:
the local feature aggregator consists of a local space encoder and an attention mechanism and aims to expand the receptive field of each point so as to extract more effective features;
the random sampling layer is used for accelerating the extraction of point features and improving the operation efficiency.
And the decoding layer designs a decoder with 3 layers, the nearest K adjacent points are found for each input point by adopting a K nearest neighbor algorithm (K nearest neighbors are found to represent) through the encoder of each layer, and the point cloud characteristic set is up-sampled by a nearest neighbor difference algorithm. Finally, the features obtained by the up-sampling are connected with the features obtained by the encoder to obtain the feature vector output by the decoding layer
Figure 673431DEST_PATH_IMAGE008
。
The semantic prediction layer is used for mapping the feature vector obtained by the decoding layer to the full-link layer and the linear occipital flow function ReLU
Figure 969283DEST_PATH_IMAGE009
，
Figure 347306DEST_PATH_IMAGE010
Are a category of semantics, in common
Figure 324489DEST_PATH_IMAGE011
The class of the user is a generic class,
Figure 720835DEST_PATH_IMAGE012
the number of the middle points of the three-dimensional point cloud is shown.
In the specific implementation process, the self-supervision training of the point cloud semantic segmentation model specifically comprises the following steps:
step 1.1, corresponding three-dimensional point cloud coordinates and RGB image pixels one by one through a geometric relationship, obtaining a semantic segmentation result of each pixel in an RGB image by adopting a trained RGB image segmentation model, and obtaining a point cloud semantic label through a corresponding relationship between the three-dimensional point cloud coordinates and the RGB image pixels, wherein the label obtained through the corresponding relationship of the RGB image is defined as an automatic supervision semantic label in the embodiment;
step 1.2, point characteristics output by point cloud segmentation network
Figure 938321DEST_PATH_IMAGE013
Comparing with the self-supervision semantic labels obtained through the RGB image corresponding relation, and iteratively adjusting parameters of each level of the point cloud semantic segmentation model to enable semantic results output by the point cloud semantic segmentation model to approach the self-supervision semantic labels, wherein the process of iteratively adjusting the parameters is self-supervision learning training;
and step 1.3, after iteration for a certain number of times, a model with an output score closer to the self-supervision semantic label is left, so that a point cloud semantic segmentation model is formed.
It should be noted that the RGB image is only used for self-supervised training of the point cloud semantic segmentation model, and the trained point cloud semantic segmentation model can perform point cloud semantic segmentation only by inputting point cloud data, so as to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current time.
In this embodiment, the map building process of the dense three-dimensional point cloud map specifically includes:
step 2.1, performing inertial integration on the real-time inertial data to obtain a first initial state quantity (including position, attitude, speed and IMU error quantity) of the system, wherein the implementation process comprises the following steps:
IMU-based measurement noise
Figure 52908DEST_PATH_IMAGE014
0, the integral of inertia is:
Figure 833782DEST_PATH_IMAGE015
in the formula
Figure 350214DEST_PATH_IMAGE016
Show about
Figure 941863DEST_PATH_IMAGE017
、
Figure 543746DEST_PATH_IMAGE018
、
Figure 128311DEST_PATH_IMAGE019
(Here, the
Figure 499249DEST_PATH_IMAGE020
Is 0) is calculated as a function of the integral of inertia,
Figure 261800DEST_PATH_IMAGE021
representing an exponential function, the combination of which represents the inertial integration state update process;
Figure 85400DEST_PATH_IMAGE022
i.e. two adjacent IMU sample times in a radar scan
Figure 208076DEST_PATH_IMAGE023
And
Figure 433521DEST_PATH_IMAGE024
the difference between the two (IMU sampling interval),
Figure 366974DEST_PATH_IMAGE025
is the label of the ith IMU measurement data sample.
Figure 943448DEST_PATH_IMAGE026
Is measurement data of the IMU (i.e. IMU)
Figure 135395DEST_PATH_IMAGE027
Inertial data representing the ith IMU sample);
Figure 215347DEST_PATH_IMAGE028
indicating the state of the system in which, among other things,
Figure 585279DEST_PATH_IMAGE029
the presentation system is in
Figure 586733DEST_PATH_IMAGE030
The first state quantity at the sub-sampling instant,
Figure 51213DEST_PATH_IMAGE031
the presentation system is in
Figure 251250DEST_PATH_IMAGE032
A first state quantity at the sub-sampling time;
step 2.2, based on the real-time three-dimensional point cloud, performing state updating on the first state quantity by adopting Kalman filtering to obtain a second state quantity of the system, wherein the implementation process comprises the following steps:
firstly, calculating the point cloud residual error of the laser radar three-dimensional point cloud
Figure 306931DEST_PATH_IMAGE033
The method comprises the following steps:
Figure 77572DEST_PATH_IMAGE034
in the formula (I), the compound is shown in the specification,
Figure 345742DEST_PATH_IMAGE035
is the location of the nearest feature point on the map,
Figure 400286DEST_PATH_IMAGE036
is a point
Figure 643179DEST_PATH_IMAGE037
The normal vector (or edge orientation) of the respective plane (or edge) on which it lies,
Figure 415963DEST_PATH_IMAGE038
is the IMU estimates the point cloud location;
iterative updating is carried out on the state estimation by adopting an iterative Kalman filter until the point cloud residual error
Figure 487824DEST_PATH_IMAGE039
And converging, wherein the process of iteratively updating the state estimation comprises the following steps:
Figure 131295DEST_PATH_IMAGE040
in the formula
Figure 810670DEST_PATH_IMAGE041
Is shown in
Figure 8433DEST_PATH_IMAGE042
Time (
Figure 883985DEST_PATH_IMAGE042
Is shown as
Figure 913121DEST_PATH_IMAGE043
Scan end time of sub lidar scan) to
Figure 701079DEST_PATH_IMAGE044
State after sub-Kalman filtering
Figure 448455DEST_PATH_IMAGE045
Is used to generate a value of (a) to (b),
Figure 862119DEST_PATH_IMAGE046
is shown in
Figure 11341DEST_PATH_IMAGE047
At the first moment
Figure 958482DEST_PATH_IMAGE048
State after sub-Kalman filtering
Figure 927575DEST_PATH_IMAGE049
The generated value of (a) is,Ithe unit matrix is represented by a matrix of units,
Figure 348192DEST_PATH_IMAGE050
which represents the observation matrix, is shown,
Figure 86341DEST_PATH_IMAGE051
to represent
Figure 730949DEST_PATH_IMAGE052
About
Figure 203650DEST_PATH_IMAGE053
The partial derivative of (a) of (b),
Figure 959116DEST_PATH_IMAGE054
in order to be an exponential function of the,
Figure 20613DEST_PATH_IMAGE055
in the form of a function of a logarithm,
Figure 836122DEST_PATH_IMAGE056
represent
Figure 514228DEST_PATH_IMAGE057
State of the moment
Figure 89697DEST_PATH_IMAGE058
True value of
Figure 802438DEST_PATH_IMAGE059
And generating a value
Figure 788849DEST_PATH_IMAGE060
A dynamic model of the state of error between,
Figure 423092DEST_PATH_IMAGE061
representing a generated value
Figure 520361DEST_PATH_IMAGE062
And the estimated value
Figure 838341DEST_PATH_IMAGE063
An error therebetween;
Figure 261232DEST_PATH_IMAGE064
and representing a Kalman filtering gain matrix, wherein the calculation process is as follows:
Figure 913931DEST_PATH_IMAGE065
Figure 814891DEST_PATH_IMAGE066
Figure 705486DEST_PATH_IMAGE067
in the formula (I), the compound is shown in the specification,
Figure 784432DEST_PATH_IMAGE068
a covariance matrix is represented by a matrix of covariance,
Figure 924426DEST_PATH_IMAGE069
is shown as
Figure 629077DEST_PATH_IMAGE070
The IMU state covariance at the sub-sampling instant,
Figure 170917DEST_PATH_IMAGE071
denotes the first
Figure 420764DEST_PATH_IMAGE072
The IMU state covariance at the sub-sampling instant,
Figure 251316DEST_PATH_IMAGE073
representing a covariance matrix
Figure 494079DEST_PATH_IMAGE074
Sum partial derivative matrix
Figure 156004DEST_PATH_IMAGE075
The intermediate variable that is generated is,
Figure 91599DEST_PATH_IMAGE076
and
Figure 691339DEST_PATH_IMAGE077
respectively represent
Figure 737792DEST_PATH_IMAGE078
To pair
Figure 457487DEST_PATH_IMAGE079
And
Figure 298404DEST_PATH_IMAGE080
the partial derivative matrix of (a) is,
Figure 634707DEST_PATH_IMAGE081
representing IMU noise
Figure 235584DEST_PATH_IMAGE082
Covariance of (2), superscript
Figure 606523DEST_PATH_IMAGE083
Represents a transpose of a matrix;
in the above iterative update of the state estimate, the cloud residual is present at the point
Figure 618341DEST_PATH_IMAGE084
After convergence, an optimal state estimate, i.e. a second state quantity, can be obtained as follows:
Figure 910782DEST_PATH_IMAGE085
in the formula (I), the compound is shown in the specification,
Figure 33459DEST_PATH_IMAGE086
the presentation system is in
Figure 275215DEST_PATH_IMAGE087
A second state quantity at the sub-sampling time;
and 2.3, updating the reverse state of the second state quantity to obtain a third state quantity of the system so as to optimize the estimation of the position and the attitude and improve the positioning accuracy, wherein the implementation process of the reverse state updating is as follows:
Figure 457935DEST_PATH_IMAGE088
in the formula (I), the compound is shown in the specification,
Figure 34410DEST_PATH_IMAGE089
the presentation system is in
Figure 960778DEST_PATH_IMAGE087
A third quantity of states at the sub-sampling instants,
Figure 57041DEST_PATH_IMAGE090
the presentation system is in
Figure 613924DEST_PATH_IMAGE091
A third state quantity at the sub-sampling time;
step 2.4, obtaining a conversion matrix from the laser radar coordinate system to the IMU coordinate system based on the third state quantity
Figure 412116DEST_PATH_IMAGE092
And
Figure 142174DEST_PATH_IMAGE093
IMU coordinate system of time of day
Figure 342211DEST_PATH_IMAGE094
To a global coordinate system
Figure 883045DEST_PATH_IMAGE095
Is estimated to update the transformation matrix
Figure 168533DEST_PATH_IMAGE096
And pass through
Figure 702283DEST_PATH_IMAGE097
And
Figure 960089DEST_PATH_IMAGE098
converting point cloud coordinates of each frame of self coordinate system in one scanning of the laser radar into coordinates of a coordinate system at the scanning end time to obtain global coordinates, wherein the global coordinates are as follows:
Figure 655512DEST_PATH_IMAGE099
in the formula (I), the compound is shown in the specification,
Figure 179029DEST_PATH_IMAGE100
representing the coordinates of the laser point cloud,
Figure 516469DEST_PATH_IMAGE101
to represent
Figure 425519DEST_PATH_IMAGE102
The frame of the laser radar is scanned for the second time,
Figure 760686DEST_PATH_IMAGE103
a global coordinate system is represented, and,
Figure 755186DEST_PATH_IMAGE104
is shown as
Figure 381471DEST_PATH_IMAGE105
The number of the characteristic points is one,
Figure 145028DEST_PATH_IMAGE106
is shown as
Figure 447833DEST_PATH_IMAGE107
Is characterized by
Figure 195209DEST_PATH_IMAGE108
The coordinates of the sub-scan lidar frame,
Figure 359605DEST_PATH_IMAGE109
is shown as
Figure 243248DEST_PATH_IMAGE110
Global coordinates of the feature points after coordinate conversion;
Figure 716954DEST_PATH_IMAGE111
a coordinate transformation matrix representing the coordinate system from the lower right corner mark to the upper left corner mark,
Figure 889310DEST_PATH_IMAGE112
the coordinate system of the IMU is represented,
Figure 106664DEST_PATH_IMAGE113
which represents the laser radar coordinate system and which,
Figure 595546DEST_PATH_IMAGE114
a transformation matrix representing the lidar coordinate system to the IMU coordinate system,
Figure 505733DEST_PATH_IMAGE115
to represent
Figure 962122DEST_PATH_IMAGE116
IMU coordinate system of time of day
Figure 717588DEST_PATH_IMAGE117
To the global coordinate system
Figure 779085DEST_PATH_IMAGE118
Updating the transformation matrix;
Figure 79748DEST_PATH_IMAGE119
the number of point clouds;
and 2.5, adding all the feature points at each time to the existing map according to the global coordinate to obtain the three-dimensional point cloud map under the global coordinate system.
In the global mapping process of step 2.5, each point cloud has a fixed serial number and a feature quantity, and the feature quantity is a point cloud segmentation result obtained by the semantic segmentation model in step 3. And (3) corresponding the local point cloud semantic segmentation result obtained in the step (3) with the generated global three-dimensional point cloud map coordinate, thus obtaining a global three-dimensional semantic map.
Example 2
On the basis of the solid-state laser radar three-dimensional semantic map building method for the self-supervised learning in the embodiment 1, the embodiment also discloses a solid-state laser radar three-dimensional semantic map building device for the self-supervised learning, and referring to fig. 2, the device mainly comprises an RGB camera, a laser radar, an inertia measuring device, a data acquisition module, a point cloud semantic segmentation model, a self-supervised training module, a three-dimensional point cloud map building module and a three-dimensional semantic map building module. Specifically, the method comprises the following steps:
the RGB camera, the laser radar and the inertia measuring device are carried on a carrier such as a mobile robot or handheld equipment and are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring the acquired RGB image, the three-dimensional point cloud data and the inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertial data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
the three-dimensional semantic map building module is connected with the point cloud semantic segmentation model and the three-dimensional point cloud map building module and is used for enabling the point cloud segmentation result corresponding to each point cloud coordinate to correspond to the dense three-dimensional point cloud map coordinate to generate the three-dimensional semantic map.
In a specific application process, each functional module of the solid-state lidar three-dimensional semantic map building device is the same as that in embodiment 1, and therefore, the description thereof is omitted in this embodiment.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.

Claims (9)

1. A solid-state laser radar three-dimensional semantic mapping method for self-supervision learning is characterized by comprising the following steps:
step 1, constructing a point cloud semantic segmentation model of a coding-decoding structure, and carrying out self-supervision training on the point cloud semantic segmentation model based on a synchronously acquired RGB image and a three-dimensional point cloud data set;
step 2, collecting real-time three-dimensional point cloud and real-time inertia data through a small solid-state laser radar and an inertia measurement element, fusing the real-time three-dimensional point cloud and the real-time inertia data based on a Kalman filter, and outputting a dense three-dimensional point cloud map composed of the real-time point cloud data under global coordinates;
step 3, inputting the real-time three-dimensional point cloud into the trained point cloud semantic segmentation model to obtain a point cloud segmentation result corresponding to each point cloud coordinate at the current moment, and corresponding the point cloud segmentation result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate to generate a three-dimensional semantic map;
and 4, deploying the trained point cloud semantic segmentation model on computing equipment with an ARM + GPU architecture, constructing a semantic map building system together with a small solid-state laser radar and an inertia measurement element, carrying out three-dimensional point cloud, inertia data acquisition and map updating according to fixed frequency, carrying out real-time semantic segmentation, and generating a three-dimensional semantic map.
2. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 1, wherein in the step 1, the point cloud semantic segmentation model comprises:
the encoding layer is used for extracting the characteristics of the input three-dimensional point cloud;
the decoding layer is used for finding K nearest adjacent points for each input point on the basis of the characteristics extracted by the coding layer, then carrying out up-sampling on the point cloud characteristic set through a nearest neighbor difference algorithm, and connecting the characteristics obtained by the up-sampling with the characteristics obtained by the coder to obtain a characteristic vector output by the decoding layer;
a semantic prediction layer for mapping the feature vectors obtained from the decoding layer to the full-link layer and the linear anchor function ReLU
Figure 223828DEST_PATH_IMAGE001
，
Figure 219597DEST_PATH_IMAGE002
Are a category of semantics, in common
Figure 530492DEST_PATH_IMAGE003
The class of the user is a generic class,
Figure 660122DEST_PATH_IMAGE004
the number of the midpoints of the three-dimensional point cloud is shown.
3. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 2, wherein in the step 1, the point cloud semantic segmentation model is self-supervised trained, specifically:
step 1.1, corresponding three-dimensional point cloud coordinates and RGB image pixels one by one through a geometric relationship, obtaining a semantic segmentation result of each pixel in an RGB image by adopting a trained RGB image segmentation model, and obtaining a self-supervision semantic label through the corresponding relationship between the three-dimensional point cloud coordinates and the RGB image pixels;
step 1.2, point characteristics output by point cloud segmentation network
Figure 5653DEST_PATH_IMAGE005
Comparing with the self-supervision semantic label obtained through the corresponding relation of the RGB image, and enabling the semantic result output by the point cloud semantic segmentation model to approach the self-supervision semantic label by iteratively adjusting the parameters of each level of the point cloud semantic segmentation model;
and step 1.3, after iteration for a certain number of times, a model with an output score closer to the self-supervision semantic label is left, so that a point cloud semantic segmentation model is formed.
4. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 1, 2 or 3, wherein the step 2 specifically comprises:
step 2.1, performing inertia integration on the real-time inertia data to obtain a first initial state quantity of the system;
2.2, based on the real-time three-dimensional point cloud, performing state updating on the first state quantity by adopting Kalman filtering to obtain a second state quantity;
step 2.3, updating the reverse state of the second state quantity to obtain a third state quantity of the system;
step 2.4, obtaining a conversion matrix from the laser radar coordinate system to an inertial coordinate system and an estimation updating conversion matrix from the inertial coordinate system to a global coordinate system based on the third state quantity, and converting point cloud coordinates of each frame of self coordinate system in one scanning of the laser radar to coordinates of a coordinate system at the last scanning moment to obtain global coordinates;
and 2.5, adding all the feature points at each time to the existing map according to the global coordinate to obtain the dense three-dimensional point cloud map under the global coordinate system.
5. Self-supervision according to claim 4The three-dimensional semantic mapping method for the learned solid-state laser radar is characterized in that in step 2.1, inertial measurement noise is set
Figure 110006DEST_PATH_IMAGE006
0, the integral of inertia is:
Figure 908198DEST_PATH_IMAGE007
in the formula (I), the compound is shown in the specification,
Figure 638257DEST_PATH_IMAGE008
show about
Figure 838294DEST_PATH_IMAGE009
、
Figure 113548DEST_PATH_IMAGE010
、
Figure 399036DEST_PATH_IMAGE011
Is a function of the integral of the inertia of,
Figure 667207DEST_PATH_IMAGE012
representing an exponential operator, the combination of which represents the inertia integral state update process;
Figure 721750DEST_PATH_IMAGE013
i.e. two adjacent sampling instants in a radar scan
Figure 433485DEST_PATH_IMAGE014
And with
Figure 3007DEST_PATH_IMAGE015
The difference between them;
Figure 809289DEST_PATH_IMAGE016
is the first
Figure 734651DEST_PATH_IMAGE017
The labels of the secondary inertial data samples,
Figure 600976DEST_PATH_IMAGE018
is shown as
Figure 595476DEST_PATH_IMAGE019
Real-time inertial data sampled by the secondary IMU;
Figure 205449DEST_PATH_IMAGE020
indicating the state of the system in which, among other things,
Figure 250897DEST_PATH_IMAGE021
the presentation system is in
Figure 553702DEST_PATH_IMAGE022
The first state quantity at the sub-sampling instant,
Figure 769920DEST_PATH_IMAGE023
the presentation system is in
Figure 449163DEST_PATH_IMAGE024
A first state quantity at a sub-sampling instant.
6. The self-supervised learning solid-state lidar three-dimensional semantic mapping method according to claim 5, wherein the step 2.2 is specifically as follows:
firstly, calculating the point cloud residual error of the real-time three-dimensional point cloud
Figure 83538DEST_PATH_IMAGE025
The method comprises the following steps:
Figure 557244DEST_PATH_IMAGE026
in the formula (I), the compound is shown in the specification,
Figure 526337DEST_PATH_IMAGE027
is the location of the nearest feature point on the map,
Figure 478113DEST_PATH_IMAGE028
is a point
Figure 232573DEST_PATH_IMAGE029
The normal vector or edge orientation of the respective plane or edge in which it lies,
Figure 877181DEST_PATH_IMAGE030
estimating the point cloud position by the IMU;
iterative updating is carried out on the state estimation by adopting an iterative Kalman filter until the point cloud residual error
Figure 333571DEST_PATH_IMAGE031
And converging, wherein the process of iteratively updating the state estimation comprises the following steps:
Figure 89037DEST_PATH_IMAGE032
in the formula (I), the compound is shown in the specification,
Figure 884955DEST_PATH_IMAGE033
is shown in
Figure 275415DEST_PATH_IMAGE034
At the first moment
Figure 219100DEST_PATH_IMAGE035
State after sub-Kalman filtering
Figure 778258DEST_PATH_IMAGE036
The generated value of (a) is,
Figure 490999DEST_PATH_IMAGE037
is shown in
Figure 962562DEST_PATH_IMAGE038
At the first moment
Figure 393544DEST_PATH_IMAGE039
State after sub-Kalman filtering
Figure 490813DEST_PATH_IMAGE040
The generated value of (a) is,
Figure 323640DEST_PATH_IMAGE041
is shown as
Figure 966105DEST_PATH_IMAGE042
The scan end time of the secondary lidar scan,Ithe unit matrix is represented by a matrix of units,
Figure 884382DEST_PATH_IMAGE043
which represents the observation matrix, is shown,
Figure 785342DEST_PATH_IMAGE044
to represent
Figure 207096DEST_PATH_IMAGE045
About
Figure 286042DEST_PATH_IMAGE046
The partial derivative of (a) of (b),
Figure 426036DEST_PATH_IMAGE047
in the form of a function of a logarithm,
Figure 130687DEST_PATH_IMAGE048
to represent
Figure 938106DEST_PATH_IMAGE049
State of the moment
Figure 187953DEST_PATH_IMAGE050
True value of
Figure 815243DEST_PATH_IMAGE051
And generating a value
Figure 58006DEST_PATH_IMAGE052
The dynamic model of the error state between the two,
Figure 454352DEST_PATH_IMAGE053
representing a generated value
Figure 875100DEST_PATH_IMAGE054
And the estimated value
Figure 989687DEST_PATH_IMAGE055
The error between;
Figure 770561DEST_PATH_IMAGE056
and representing a Kalman filtering gain matrix, wherein the calculation process is as follows:
Figure 286993DEST_PATH_IMAGE057
Figure 878642DEST_PATH_IMAGE058
Figure 480525DEST_PATH_IMAGE059
in the formula (I), the compound is shown in the specification,
Figure 65090DEST_PATH_IMAGE060
a covariance matrix is represented by a matrix of covariance,
Figure 436028DEST_PATH_IMAGE061
is shown as
Figure 933000DEST_PATH_IMAGE062
The IMU state covariance at the sub-sampling instant,
Figure 22179DEST_PATH_IMAGE063
is shown as
Figure 410435DEST_PATH_IMAGE064
The IMU state covariance at the sub-sampling instant,
Figure 635880DEST_PATH_IMAGE065
representing a covariance matrix
Figure 303753DEST_PATH_IMAGE066
And partial derivative matrix
Figure 145807DEST_PATH_IMAGE067
The intermediate variable that is generated is,
Figure 72174DEST_PATH_IMAGE068
and
Figure 152126DEST_PATH_IMAGE069
respectively represent
Figure 522058DEST_PATH_IMAGE070
To pair
Figure 320250DEST_PATH_IMAGE071
And
Figure 50309DEST_PATH_IMAGE072
the partial derivative matrix of (a) is,
Figure 984767DEST_PATH_IMAGE073
representing IMU noise
Figure 525601DEST_PATH_IMAGE074
Covariance of (2), superscript
Figure 14351DEST_PATH_IMAGE075
Represents a transpose of a matrix;
in the above iterative update process of the state estimation, when the cloud residual error is in point
Figure 282521DEST_PATH_IMAGE076
After convergence, an optimal state estimate, i.e. a second state quantity, can be obtained as follows:
Figure 337065DEST_PATH_IMAGE077
in the formula (I), the compound is shown in the specification,
Figure 298067DEST_PATH_IMAGE078
the presentation system is in
Figure 556005DEST_PATH_IMAGE079
A second state quantity at the sub-sampling instant.
7. The method according to claim 6, wherein in step 2.3, the updating of the reverse state of the second state quantity is specifically:
Figure 627866DEST_PATH_IMAGE080
in the formula (I), the compound is shown in the specification,
Figure 802495DEST_PATH_IMAGE081
the presentation system is in
Figure 668820DEST_PATH_IMAGE082
A third quantity of states at the sub-sampling instants,
Figure 679632DEST_PATH_IMAGE083
the presentation system is in
Figure 289605DEST_PATH_IMAGE084
A third state quantity at the sub-sampling instant.
8. The method for building a three-dimensional semantic map of a solid-state lidar based on self-supervised learning according to claim 4, wherein in step 2.4, converting the point cloud coordinates of the own coordinate system of each frame in a scan of the lidar into the coordinates of the global coordinate system specifically comprises:
Figure 318741DEST_PATH_IMAGE085
in the formula (I), the compound is shown in the specification,
Figure 355967DEST_PATH_IMAGE086
representing the coordinates of the laser point cloud,
Figure 854076DEST_PATH_IMAGE087
to represent
Figure 267740DEST_PATH_IMAGE088
The frame of the laser radar is scanned for the second time,
Figure 151382DEST_PATH_IMAGE089
a global coordinate system is represented, and,
Figure 625089DEST_PATH_IMAGE090
is shown as
Figure 344914DEST_PATH_IMAGE091
The number of the characteristic points is one,
Figure 296690DEST_PATH_IMAGE092
is shown as
Figure 300418DEST_PATH_IMAGE093
A characteristic point is
Figure 679447DEST_PATH_IMAGE094
The coordinates of the sub-scan lidar frame,
Figure 886568DEST_PATH_IMAGE095
is shown as
Figure 907614DEST_PATH_IMAGE096
Global coordinates of the feature points after coordinate conversion;
Figure 765848DEST_PATH_IMAGE097
a coordinate transformation matrix representing the coordinate system from the lower right corner mark to the upper left corner mark,
Figure 581358DEST_PATH_IMAGE098
the inertial coordinate system is represented by a coordinate system,
Figure 264057DEST_PATH_IMAGE099
which represents the coordinate system of the laser radar,
Figure 557635DEST_PATH_IMAGE100
a transformation matrix representing the lidar coordinate system to an inertial coordinate system,
Figure 270376DEST_PATH_IMAGE101
represent
Figure 256786DEST_PATH_IMAGE102
Inertial frame of time
Figure 438500DEST_PATH_IMAGE103
To a global coordinate system
Figure 535769DEST_PATH_IMAGE104
Updating the transformation matrix;
Figure 103017DEST_PATH_IMAGE105
is the number of lidar point clouds.
9. The utility model provides a solid-state laser radar three-dimensional semantic map building device of self-supervised learning which characterized in that includes:
the system comprises an RGB camera, a laser radar and an inertia measuring device, wherein the RGB camera, the laser radar and the inertia measuring device are respectively used for collecting RGB images, three-dimensional point cloud data and inertia data;
the data acquisition module is connected with the RGB camera, the laser radar and the inertia measurement device and is used for acquiring RGB images, three-dimensional point cloud data and inertia data;
the point cloud semantic segmentation model is connected with the data acquisition module and is used for performing point cloud semantic segmentation on the three-dimensional point cloud to obtain a point cloud segmentation result corresponding to each point cloud coordinate;
the self-supervision training module is connected with the data acquisition module and the point cloud semantic segmentation model and is used for carrying out self-supervision training on the point cloud semantic segmentation model according to the synchronously acquired RGB image and the three-dimensional point cloud data set;
the three-dimensional point cloud map building module is connected with the data acquisition module and used for fusing the real-time three-dimensional point cloud and the real-time inertial data and outputting a dense three-dimensional point cloud map consisting of the real-time point cloud data under the global coordinate;
and the three-dimensional semantic map building module is connected with the point cloud semantic division model and the three-dimensional point cloud map building module and is used for generating a three-dimensional semantic map by corresponding the point cloud division result corresponding to each point cloud coordinate with the dense three-dimensional point cloud map coordinate.
CN202211387608.5A 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device Active CN115421158B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211387608.5A CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211387608.5A CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Publications (2)

Publication Number Publication Date
CN115421158A true CN115421158A (en) 2022-12-02
CN115421158B CN115421158B (en) 2023-04-07

Family

ID=84207166

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211387608.5A Active CN115421158B (en) 2022-11-07 2022-11-07 Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device

Country Status (1)

Country Link
CN (1) CN115421158B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115638788A (en) * 2022-12-23 2023-01-24 安徽蔚来智驾科技有限公司 Semantic vector map construction method, computer equipment and storage medium
CN116229057A (en) * 2022-12-22 2023-06-06 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning
CN116778162A (en) * 2023-06-25 2023-09-19 南京航空航天大学 Weak supervision large aircraft appearance point cloud semantic segmentation method based on geometric feature guidance
CN117517864A (en) * 2023-11-08 2024-02-06 南京航空航天大学 Laser radar-based power transmission line near electricity early warning method and device

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018094719A1 (en) * 2016-11-28 2018-05-31 深圳市大疆创新科技有限公司 Method for generating point cloud map, computer system, and device
CN110097553A (en) * 2019-04-10 2019-08-06 东南大学 The semanteme for building figure and three-dimensional semantic segmentation based on instant positioning builds drawing system
US10929694B1 (en) * 2020-01-22 2021-02-23 Tsinghua University Lane detection method and system based on vision and lidar multi-level fusion
CN112634451A (en) * 2021-01-11 2021-04-09 福州大学 Outdoor large-scene three-dimensional mapping method integrating multiple sensors
CN113128591A (en) * 2021-04-14 2021-07-16 中山大学 Rotation robust point cloud classification method based on self-supervision learning
CN113763423A (en) * 2021-08-03 2021-12-07 中国北方车辆研究所 Multi-mode data based systematic target recognition and tracking method
CN114898322A (en) * 2022-06-13 2022-08-12 中国第一汽车股份有限公司 Driving environment identification method and device, vehicle and storage medium
CN114926469A (en) * 2022-04-26 2022-08-19 中南大学 Semantic segmentation model training method, semantic segmentation method, storage medium and terminal
CN115222919A (en) * 2022-07-27 2022-10-21 徐州徐工矿业机械有限公司 Sensing system and method for constructing color point cloud map of mobile machine

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018094719A1 (en) * 2016-11-28 2018-05-31 深圳市大疆创新科技有限公司 Method for generating point cloud map, computer system, and device
CN110097553A (en) * 2019-04-10 2019-08-06 东南大学 The semanteme for building figure and three-dimensional semantic segmentation based on instant positioning builds drawing system
US10929694B1 (en) * 2020-01-22 2021-02-23 Tsinghua University Lane detection method and system based on vision and lidar multi-level fusion
CN112634451A (en) * 2021-01-11 2021-04-09 福州大学 Outdoor large-scene three-dimensional mapping method integrating multiple sensors
CN113128591A (en) * 2021-04-14 2021-07-16 中山大学 Rotation robust point cloud classification method based on self-supervision learning
CN113763423A (en) * 2021-08-03 2021-12-07 中国北方车辆研究所 Multi-mode data based systematic target recognition and tracking method
CN114926469A (en) * 2022-04-26 2022-08-19 中南大学 Semantic segmentation model training method, semantic segmentation method, storage medium and terminal
CN114898322A (en) * 2022-06-13 2022-08-12 中国第一汽车股份有限公司 Driving environment identification method and device, vehicle and storage medium
CN115222919A (en) * 2022-07-27 2022-10-21 徐州徐工矿业机械有限公司 Sensing system and method for constructing color point cloud map of mobile machine

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
ŁUKASZ SOBCZAK等: ""LiDAR Point Cloud Generation for SLAM Algorithm Evaluation"", 《SENSOR》 *
王金科等: ""多源融合SLAM 的现状与挑战"", 《中国象形图形学报》 *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116229057A (en) * 2022-12-22 2023-06-06 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning
CN116229057B (en) * 2022-12-22 2023-10-27 之江实验室 Method and device for three-dimensional laser radar point cloud semantic segmentation based on deep learning
WO2024130776A1 (en) * 2022-12-22 2024-06-27 之江实验室 Three-dimensional lidar point cloud semantic segmentation method and apparatus based on deep learning
CN115638788A (en) * 2022-12-23 2023-01-24 安徽蔚来智驾科技有限公司 Semantic vector map construction method, computer equipment and storage medium
CN115638788B (en) * 2022-12-23 2023-03-21 安徽蔚来智驾科技有限公司 Semantic vector map construction method, computer equipment and storage medium
CN116778162A (en) * 2023-06-25 2023-09-19 南京航空航天大学 Weak supervision large aircraft appearance point cloud semantic segmentation method based on geometric feature guidance
CN117517864A (en) * 2023-11-08 2024-02-06 南京航空航天大学 Laser radar-based power transmission line near electricity early warning method and device
CN117517864B (en) * 2023-11-08 2024-04-26 南京航空航天大学 Laser radar-based power transmission line near electricity early warning method and device

Also Published As

Publication number Publication date
CN115421158B (en) 2023-04-07

Similar Documents

Publication Publication Date Title
CN115421158B (en) Self-supervision learning solid-state laser radar three-dimensional semantic mapping method and device
CN108665496B (en) End-to-end semantic instant positioning and mapping method based on deep learning
JP7236565B2 (en) POSITION AND ATTITUDE DETERMINATION METHOD, APPARATUS, ELECTRONIC DEVICE, STORAGE MEDIUM AND COMPUTER PROGRAM
CN113393522B (en) 6D pose estimation method based on monocular RGB camera regression depth information
US7747106B2 (en) Method and system for filtering, registering, and matching 2.5D normal maps
CN115655262B (en) Deep learning perception-based multi-level semantic map construction method and device
CN107462897B (en) Three-dimensional mapping method based on laser radar
JP2014523572A (en) Generating map data
US11948310B2 (en) Systems and methods for jointly training a machine-learning-based monocular optical flow, depth, and scene flow estimator
CN116222577B (en) Closed loop detection method, training method, system, electronic equipment and storage medium
CN115267724B (en) Position re-identification method of mobile robot capable of estimating pose based on laser radar
CN112907557A (en) Road detection method, road detection device, computing equipment and storage medium
Li et al. FDnCNN-based image denoising for multi-labfel localization measurement
Li et al. Vehicle object detection based on rgb-camera and radar sensor fusion
CN117570968A (en) Map construction and maintenance method and device based on visual road sign and storage medium
CN117496312A (en) Three-dimensional multi-target detection method based on multi-mode fusion algorithm
US11741631B2 (en) Real-time alignment of multiple point clouds to video capture
Milli et al. Multi-modal multi-task (3mt) road segmentation
CN114782689A (en) Point cloud plane segmentation method based on multi-frame data fusion
CN113554754A (en) Indoor positioning method based on computer vision
Mo et al. Cross-based dense depth estimation by fusing stereo vision with measured sparse depth
CN114417946A (en) Target detection method and device
WO2024011455A1 (en) Method for position re-recognition of mobile robot based on lidar estimable pose
Wang et al. Improved simultaneous localization and mapping by stereo camera and SURF
CN114323038A (en) Outdoor positioning method fusing binocular vision and 2D laser radar

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant