CN116071410A - Point cloud registration method, system, equipment and medium based on deep learning - Google Patents
Point cloud registration method, system, equipment and medium based on deep learning Download PDFInfo
- Publication number
- CN116071410A CN116071410A CN202310239900.0A CN202310239900A CN116071410A CN 116071410 A CN116071410 A CN 116071410A CN 202310239900 A CN202310239900 A CN 202310239900A CN 116071410 A CN116071410 A CN 116071410A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- network
- point
- determining
- adopting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 48
- 238000013135 deep learning Methods 0.000 title claims abstract description 26
- 239000011159 matrix material Substances 0.000 claims abstract description 56
- 230000009466 transformation Effects 0.000 claims abstract description 32
- 238000013528 artificial neural network Methods 0.000 claims abstract description 16
- 230000007246 mechanism Effects 0.000 claims description 14
- 230000004927 fusion Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 9
- 238000000354 decomposition reaction Methods 0.000 claims description 9
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000005259 measurement Methods 0.000 abstract description 6
- 230000006870 function Effects 0.000 description 8
- 239000000284 extract Substances 0.000 description 4
- 239000004973 liquid crystal related substance Substances 0.000 description 4
- 238000005070 sampling Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 3
- 239000008280 blood Substances 0.000 description 2
- 210000004369 blood Anatomy 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004807 localization Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Abstract
The invention discloses a point cloud registration method, a system, equipment and a medium based on deep learning, and relates to the field of computer vision and laser radar measurement calibration. The method comprises the following steps: acquiring laser radar data, and generating a point cloud data set according to the laser radar data; determining a first point cloud and a second point cloud from the point cloud dataset; determining geometric descriptors and key scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network; determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; a trained matching pair consistency learning network is adopted to determine the confidence coefficient of each matching pair; calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient; registering the first point cloud and the second point cloud according to the transformation matrix. The method and the device can improve the point cloud registration accuracy, thereby improving the accuracy of laser radar measurement calibration.
Description
Technical Field
The invention relates to the field of computer vision and laser radar measurement calibration, in particular to a point cloud registration method, a system, equipment and a medium based on deep learning.
Background
Three-dimensional point cloud registration refers to a process of giving two frames of point clouds, obtaining a transformation matrix by calculating relative pose between the two frames of point clouds, and aligning the two frames of point clouds to the same coordinate system.
In recent years, fields such as autopilot and digital man have become a hot spot for research in the industry and academia. Point cloud registration techniques play an important role therein. While traditional visual live localization and map creation (Simultaneous Localization andMapping, SLAM) has evolved over the years, it is inevitably limited by lighting conditions and texture quality. Algorithms based on iterative closest points (Iterative Closest Point, ICP) require a good initial value in point cloud registration, otherwise are very prone to falling into local optima. Therefore, the point cloud registration method based on deep learning is gradually becoming a research hotspot.
The deep learning-based point cloud registration method can be generally divided into two major categories, namely, a local feature descriptor is used for obtaining a matching relation, a random sampling consistency (Random Sample Consensus, RANSAC) algorithm is used for calculating a transformation matrix from the matching relation, the method requires that the local feature descriptor needs to exclude pose information of a radar and a series of differences (such as self-shielding) caused by different sampling angles from robustness, and even if the two conditions are met, the information of the light-dependent local surface is difficult to obtain a precise matching relation. The second method is based on regression, namely, the neural network directly outputs a transformation matrix between two input frames of point clouds, and the method has different solutions, but the final precision is difficult to exceed the first method.
In order to solve the precision problem existing in the prior art in the field, a new point cloud registration method needs to be provided.
Disclosure of Invention
The invention aims to provide a point cloud registration method, a system, equipment and a medium based on deep learning, so as to improve the point cloud registration precision and improve the accuracy of laser radar measurement calibration.
In order to achieve the above object, the present invention provides the following solutions:
a point cloud registration method based on deep learning, the point cloud registration method comprising:
acquiring laser radar data, and generating a point cloud data set according to the laser radar data;
determining a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively represent laser radar data of different frames;
determining geometric descriptors and key scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network;
determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes the one-to-one correspondence of points in the first point cloud and points in the second point cloud;
determining the confidence coefficient of each matched pair by adopting a trained matched pair consistency learning network;
calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient;
registering the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
Optionally, the trained twin neural network comprises: an encoder network, a mutual attention network, and a decoder network; determining geometric descriptors and key scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network, wherein the method specifically comprises the following steps of:
respectively downsampling the first point cloud and the second point cloud by adopting an encoder network to obtain a first local shape feature and a second local shape feature;
adopting a mutual attention network to exchange information between the first local shape feature and the second local shape feature to obtain a first fusion feature and a second fusion feature;
and respectively upsampling the first fusion feature and the second fusion feature by adopting a decoder network to obtain the geometric descriptor and the criticality score of each point in the first point cloud and the second point cloud.
Optionally, the trained matching pair consistency learning network comprises: a low-order consistency determination network, a high-order consistency determination network based on a self-attention mechanism, and a fully connected network; the method for determining the confidence coefficient of each matched pair by adopting a trained matched pair consistency learning network specifically comprises the following steps:
adopting a low-order consistency determination network, and determining the relative equidistance between every two matching pairs according to the geometric descriptors of the corresponding points in every matching pair to obtain a low-order consistency matrix;
adopting a high-order consistency determination network based on a self-attention mechanism, and determining high-order consistency characteristics of each matched pair according to the low-order consistency matrix;
and adopting a fully connected network, and respectively carrying out classification judgment on each matching pair according to each high-order consistency characteristic to obtain the confidence coefficient of each matching pair.
Optionally, calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient specifically includes:
respectively carrying out decentration on the first point cloud and the second point cloud according to the confidence degrees to obtain a decentration result;
calculating a covariance matrix according to each confidence coefficient and the decentralization result;
and carrying out singular value decomposition on the covariance matrix, and determining a transformation matrix between the first point cloud and the second point cloud according to a singular value decomposition result.
Optionally, determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score specifically includes:
selecting a plurality of points with the highest key scores from the first point cloud and the second point cloud respectively to serve as a first point set to be matched and a second point set to be matched;
and matching the points in the first point set to be matched and the second point set to be matched according to the geometric descriptor to obtain a plurality of matching pairs.
Optionally, determining the first point cloud and the second point cloud according to the point cloud data set specifically includes:
and selecting any two frames of point clouds with the overlapping degree of 30% -60% from the point cloud data set, and respectively carrying out data enhancement on the two frames of point clouds to obtain a first point cloud and a second point cloud.
Optionally, the encoder network and the decoder network both adopt a KPConv convolution mode, and the encoder network and the decoder network adopt a U-Net mode to perform cross-layer connection.
A point cloud registration system based on deep learning, the point cloud registration system comprising:
the data set generation module is used for acquiring laser radar data and generating a point cloud data set according to the laser radar data;
the point cloud determining module is used for determining a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively represent laser radar data of different frames;
the geometric descriptor and criticality score determining module is used for determining geometric descriptors and criticality scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network;
the matching pair determining module is used for determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes the one-to-one correspondence of points in the first point cloud and points in the second point cloud;
the confidence determining module is used for determining the confidence of each matched pair by adopting a trained matched pair consistency learning network;
a transformation matrix calculation module, configured to calculate a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient;
the point cloud registration module is used for registering the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the point cloud registration method described above.
A computer readable storage medium storing a computer program which when executed by a processor implements the point cloud registration method described above.
According to the specific embodiment provided by the invention, the invention discloses the following technical effects:
according to the point cloud registration method based on deep learning, a trained twin neural network is adopted, geometric descriptors and key scores of all points in the point clouds are determined, a plurality of matching pairs are determined from the first point cloud and the second point cloud according to the geometric descriptors and the key scores, and key points with the most characteristics in the first point cloud and the second point cloud can be extracted for matching; the trained matching pair consistency learning network is adopted to determine the confidence coefficient of each matching pair, and more accurate matching pairs can be further determined from the matching pairs based on the confidence coefficient, so that a transformation matrix can be accurately calculated, and point cloud registration is realized. Therefore, the method and the device can improve the point cloud registration accuracy, so that the accuracy of laser radar measurement calibration is improved, and more accurate positioning service, more real scene simulation and the like are provided in application scenes such as virtual reality, automatic driving, three-dimensional reconstruction and the like.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a point cloud registration method based on deep learning provided by the invention;
FIG. 2 is a network structure diagram of the point cloud registration method based on deep learning provided by the invention;
fig. 3 is a block diagram of a point cloud registration system based on deep learning provided by the invention.
Symbol description:
the system comprises a data set generation module-1, a point cloud determination module-2, a geometric descriptor and criticality score determination module-3, a matching pair determination module-4, a confidence degree determination module-5, a transformation matrix calculation module-6 and a point cloud registration module-7.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention aims to provide a point cloud registration method, a system, equipment and a medium based on deep learning so as to improve the point cloud registration precision.
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to the appended drawings and appended detailed description.
Example 1
The embodiment of the invention provides a point cloud registration method based on deep learning. As shown in fig. 1 and 2, the point cloud registration method includes:
step S1: and acquiring laser radar data, and generating a point cloud data set according to the laser radar data. Specifically, a point cloud dataset is generated using data historically collected by a lidar system.
Step S2: determining a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively characterize laser radar data of different frames. Preferably, a point cloud with 30% -60% overlapping degree of any two frames is selected from the point cloud data set, and data enhancement is performed on the two frames of point clouds respectively to obtain a first point cloud (denoted as point cloud S) and a second point cloud (denoted as point cloud T).
Step S3: and determining the geometric descriptor and the criticality Score of each point in the first point cloud and the second point cloud by adopting the trained twin neural network. The twin neural network is obtained by training in a supervision training mode. The trained twin neural network comprises: an encoder network, a mutual attention network, and a decoder network.
Preferably, the encoder network and the decoder network both adopt a convolution mode of core point convolution (Kernel Point Convolution, KPConv), and the encoder network and the decoder network adopt a U-Net mode to carry out cross-layer connection, so that information loss can be avoided.
The step S3 specifically comprises the following steps:
step S3.1: and respectively downsampling the first point cloud and the second point cloud by adopting an encoder network to obtain a first local shape feature and a second local shape feature.
Specifically, the input of the encoder network is point cloud data, the point cloud data is downsampled to generate low-resolution local shape characteristics, and the outputs corresponding to the point cloud S and the point cloud T are respectively
And->
Step S3.2: and adopting a mutual attention network to exchange information between the first local shape feature and the second local shape feature to obtain a first fusion feature and a second fusion feature.
In particular, a mutual-attention network transducer to
And->
And for input, exchanging information for the two-frame point cloud. The specific steps of information exchange are as follows: 1. in the transducer, with +.>
As query, in->
As key and value, from ∈ ->
Extracts information from the blood and fuses to +.>
In, and output->
2. In the transducer, with +.>
As query, in->
As key and value, from ∈ ->
Extracts information from the blood and fuses to +.>
In, and output->
To this end (I)>
And->
All contain each other's information, i.e. the information exchange is completed.
Step S3.3: and respectively upsampling the first fusion feature and the second fusion feature by adopting a decoder network to obtain the geometric descriptor and the criticality score of each point in the first point cloud and the second point cloud.
In particular, the inputs to the decoder network are respectively
And->
Characterizing low resolution point cloud>
And->
Resampling to obtain the geometric descriptor F of each point S And F T A criticality Score.
Step S4: determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes a one-to-one correspondence of points in the first point cloud and points in the second point cloud.
The step S4 specifically comprises the following steps:
step S4.1: and selecting a plurality of points with the highest key scores from the first point cloud and the second point cloud respectively to serve as a first point set to be matched and a second point set to be matched.
Step S4.2: and matching the points in the first point set to be matched and the second point set to be matched according to the geometric descriptor to obtain a plurality of matching pairs.
Specifically, the points in the first point cloud and the second point cloud are respectively ranked from high to low according to the obtained criticality score, the points with the highest K 'scores are respectively selected from the two frames of point clouds according to the ranking result to serve as key points, and rough matching pairs are generated according to the corresponding geometric descriptors (namely, initial matching relations are formed according to the geometric descriptors corresponding to the K' key points of the point cloud S and the point cloud T).
K' key points are selected from two frames of point clouds respectively, and two overlapping area points in the two frames of point clouds are: common points (such as points in a plane) and feature points (such as corner points) with information truly, and only the latter are repeatable key points, namely the repeatability can be judged as key points in a multi-frame continuous sampling point cloud.
In the embodiment, the twin neural network strengthens information communication between two frame point clouds by adopting a mutual attention network, and provides a path and a basis for a decoder network to dig out repeatable characteristic points in the two frame point clouds and endow the repeatable characteristic points with a higher key Score.
The decoder network extracts feature points which appear in two frames of point clouds, in order to extract correct feature points, a mutual attention network is introduced into forward propagation of the network to provide a path for the feature points, and an encouraging mechanism for the key score of the correct feature points and a punishment mechanism for the geometric descriptor of common points are introduced into a loss function trained by the network, wherein the encouraging mechanism is expressed in the following form:
wherein the method comprises the steps of
Ith point S representing point cloud S i It should be noted that here T i Representing the point S i Correctly matched points in the point cloud T by means of labels,/->
Then it represents T i Is a criticality score of (2); m is m i Representing the distinguishability of this match to the descriptor, the expression is m i ＝D i- -D i+ Wherein D is i+ Is S i And T i Euclidean distance between respective descriptors, D i- Is the distance S i The descriptors of the nearest points outside a range r and the euclidean distance between them (i.e. for S i Taking the point cloud S and S i Nearest point S j And meet S i And S is j The Euclidean distance between the two is larger than r, D i- Is S i And S is j Euclidean distance between the respective descriptors).
According to D i+ And D i- As can be seen from the definition of D i+ Describe S i Finding the difficulty level of correct matching pair from the point cloud T, D i+ Smaller means easier to find the sum S i Correctly matched T i The more difficult the representation is on the contrary; and D is i- Describe point S i Local saliency inside the point cloud S, D i- Smaller represents the point S i The less pronounced (possibly in the smooth region), whereas the point S is indicated i The more pronounced (possibly at the corner point). From the expression, we know that when m i >At 0, indicating that the matched pair is highly differentiated, the loss function will tend to be
This score gets larger, letting m at the same time i Further becoming larger; otherwise let->
Becomes smaller, m i Further becomes smaller. Based on this mechanism, encouragement of the criticality score of the correct feature points and penalties for common point geometry descriptors can be achieved.
Step S5: and determining the confidence coefficient of each matched pair by adopting a trained matched pair consistency learning network. Wherein the trained matching pair consistency learning network comprises: a low-order consistency determination network, a high-order consistency determination network based on a self-attention mechanism, and a fully connected network.
Specifically, the matching pair consistency learning network performs contrast analysis on input point cloud rough matching pairs by using a self-attention mechanism of a transformer, extracts high-dimensional consistency between the matching pairs, and outputs a confidence degree of whether each matching pair is correct or not.
The step S5 specifically comprises the following steps:
step S5.1: and determining the relative equidistance between every two matching pairs according to the geometric descriptors of the corresponding points in every matching pair by adopting a low-order consistency determination network to obtain a low-order consistency matrix.
Specifically, in the matching pairs of the K' pair inputs, for any two pairs of matching pairs (x i ，y i ) And (x) j ，y j ) The corresponding decoder outputs the geometric descriptors of which are respectively
And->
The low-order consistency of the two matched pairs can be described as +.>
Wherein->
||·|| 2 Representing an L2 norm operation. As can be seen, C ij Representing the relative equidistance of the two matching pairs in the geometric descriptor is therefore an imperfect variable that does not accurately describe whether the two matching pairs are exactly identical, and is therefore referred to as low-order. Finally, a K '×K' low-order consistency matrix C is obtained.
Step S5.2: and determining the high-order consistency characteristics of each matched pair according to the low-order consistency matrix by adopting a high-order consistency determination network based on a self-attention mechanism.
In particular, the higher order consistency based on the self-attention mechanism determines that the network is a multi-layer transducer network structure, the input of which is
K'. Times.6 point cloud data of (1), the dot product of Q and K in the original transducer operates as: />
Here, the dot product operation of Q and K for each layer of transformers
The modification is as follows:
i.e. multiplying the low-order consistency matrix C on the original basis, so that C can be used to guide the outcome of the attention mechanism. Specifically described is C ij When larger, the i and j pairs of matching pairs are low-order consistent in geometric descriptor, so that the corresponding attention A can be properly enlarged ij And vice versa. Except for the first layer of transformers, the input of each layer of transformers is then the output of the previous layer with a size of K' x D features (D is the number of channels) for a total of 12 layers. The final layer output is characterized by F and a higher order consistency matrix m=f '·f' T Wherein
Step S5.3: and adopting a fully connected network, and respectively carrying out classification judgment on each matching pair according to each high-order consistency characteristic to obtain the confidence coefficient of each matching pair.
Specifically, the final fully connected network performs classification judgment on the correctness of the input point cloud matching pairs according to the high-order consistency characteristic to obtain the confidence (sigma) of each matching pair i I=1, 2, …, K'), the specific formula is: sigma (sigma) i ＝FC(F i ) FC (·) represents a fully connected network, with the input being the feature F that matches the output to the consistency learning network.
In this embodiment, the input to the consistency learning network is a point cloud matching pair containing both correct and incorrect
The matching pair consistency learning network builds a low-order consistency relation through the relation between descriptors of input matching pairs, continuously optimizes and forms a high-order and more accurate consistency relation through a self-attention mechanism of a transducer, and then carries out classification judgment on the correctness of the input point cloud matching pairs by using the last fully connected network as a basis to obtain the confidence sigma of each matching pair i I=1, 2, …, K'). The training loss function of the matching pair consistency learning network comprises the following two items.
First is a cross entropy function on the higher order consistency matrix M:
wherein, the liquid crystal display device comprises a liquid crystal display device,
representing the matching pair labels.
The next is a cross entropy function matching the opposite confidence σ:
wherein, the liquid crystal display device comprises a liquid crystal display device,
for confidence labels, cross entropy () represents the cross entropy function.
Thus, the total loss function is loss = loss M +loss σ 。
Step S6: and calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence degree. Specifically, after obtaining the confidence coefficient of each point cloud matching pair, taking the confidence coefficient as the weight of the matching pair, and calculating a transformation matrix between two frames of point clouds through Singular Value Decomposition (SVD) with weight.
The step S6 specifically comprises the following steps:
step S6.1: and respectively carrying out decentration on the first point cloud and the second point cloud according to the confidence degrees to obtain a decentration result.
Specifically, two frames of point clouds are respectively subjected to decentration, and the specific formula is as follows:
x' i 、y' i respectively, the result of the decentralization, wherein +.>
And->
Is the centroid point.
Step S6.2: and calculating a covariance matrix according to each confidence coefficient and the decentralization result.
Specifically, the formula is adopted
A covariance matrix is calculated. Wherein x' i T Represents x i Is a transpose of (a).
Step S6.3: and carrying out singular value decomposition on the covariance matrix, and determining a transformation matrix between the first point cloud and the second point cloud according to a singular value decomposition result.
Specifically, singular value decomposition of covariance matrix: h=uΣv T Obtaining singular value decomposition results, namely a left unitary matrix U, a diagonal matrix sigma and a right unitary matrix V, wherein the rotation matrix is R=VU T The translation vector is
Further, the transformation matrix is determined as T= [ R|t T ]Wherein U is T Representing the transpose of the left unitary matrix U, R T Representing the transpose, t, of the rotation matrix R T Representing the transpose of the translation vector t.
Step S7: registering the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
Specifically, the obtained rough matching pairs are input into a matching pair consistency learning network, more accurate matching pairs can be extracted, and a transformation matrix between two frames of point clouds can be calculated, so that point cloud registration is completed. For the point cloud with the overlapping area of two frames acquired by the laser radar, the transformation matrix between the two frames can be calculated so as to align the two frames to the same coordinate system.
Example two
In order to execute the corresponding method of the above embodiment to achieve the corresponding functions and technical effects, a point cloud registration system based on deep learning is provided below. As shown in fig. 3, the point cloud registration system includes:
the data set generation module 1 is used for acquiring laser radar data and generating a point cloud data set according to the laser radar data.
A point cloud determining module 2, configured to determine a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively characterize laser radar data of different frames.
A geometric descriptor and criticality score determination module 3 for determining a geometric descriptor and criticality score for each point in the first and second point clouds using the trained twin neural network.
A matching pair determining module 4, configured to determine a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes a one-to-one correspondence of points in the first point cloud and points in the second point cloud.
And the confidence determining module 5 is used for determining the confidence of each matched pair by adopting the trained matched pair consistency learning network.
A transformation matrix calculation module 6, configured to calculate a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient.
The point cloud registration module 7 is configured to register the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
Example III
The embodiment of the invention also provides electronic equipment, which comprises a memory and a processor, wherein the memory is used for storing a computer program, and the processor is used for running the computer program to enable the electronic equipment to execute the point cloud registration method in the first embodiment. The electronic device may be a server.
In addition, the present invention also provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the point cloud registration method in the first embodiment.
In summary, the method, system, device and medium for point cloud registration based on deep learning provided by the invention adopt a deep learning network structure composed of an encoder, a mutual attention network, a decoder and a matching pair consistency learning network, and firstly, the encoder is used for downsampling the point cloud to learn low-resolution local shape characteristics; then, the mutual attention network is used for further enhancing the fusion of global information to obtain high-level features; then up-sampling by using a geometric feature decoder to obtain geometric descriptors and key scores of each point, sorting and taking out some key points with the most features according to the key scores, and calculating point cloud rough matching pairs according to the corresponding geometric descriptors; and finally, extracting more accurate matching pairs through a matching pair consistency learning network, so as to calculate a transformation matrix, realize point cloud registration, and improve the precision of the point cloud registration, thereby improving the accuracy of laser radar measurement calibration.
In the present specification, each embodiment is described in a progressive manner, and each embodiment is mainly described in a different point from other embodiments, and identical and similar parts between the embodiments are all enough to refer to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant points refer to the description of the method section.
The principles and embodiments of the present invention have been described herein with reference to specific examples, the description of which is intended only to assist in understanding the methods of the present invention and the core ideas thereof; also, it is within the scope of the present invention to be modified by those of ordinary skill in the art in light of the present teachings. In view of the foregoing, this description should not be construed as limiting the invention.
Claims (10)
1. The point cloud registration method based on the deep learning is characterized by comprising the following steps of:
acquiring laser radar data, and generating a point cloud data set according to the laser radar data;
determining a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively represent laser radar data of different frames;
determining geometric descriptors and key scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network;
determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes the one-to-one correspondence of points in the first point cloud and points in the second point cloud;
determining the confidence coefficient of each matched pair by adopting a trained matched pair consistency learning network;
calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient;
registering the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
2. The deep learning based point cloud registration method of claim 1, wherein the trained twin neural network comprises: an encoder network, a mutual attention network, and a decoder network; determining geometric descriptors and key scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network, wherein the method specifically comprises the following steps of:
respectively downsampling the first point cloud and the second point cloud by adopting an encoder network to obtain a first local shape feature and a second local shape feature;
adopting a mutual attention network to exchange information between the first local shape feature and the second local shape feature to obtain a first fusion feature and a second fusion feature;
and respectively upsampling the first fusion feature and the second fusion feature by adopting a decoder network to obtain the geometric descriptor and the criticality score of each point in the first point cloud and the second point cloud.
3. The deep learning based point cloud registration method of claim 1, wherein the trained matching pair consistency learning network comprises: a low-order consistency determination network, a high-order consistency determination network based on a self-attention mechanism, and a fully connected network; the method for determining the confidence coefficient of each matched pair by adopting a trained matched pair consistency learning network specifically comprises the following steps:
adopting a low-order consistency determination network, and determining the relative equidistance between every two matching pairs according to the geometric descriptors of the corresponding points in every matching pair to obtain a low-order consistency matrix;
adopting a high-order consistency determination network based on a self-attention mechanism, and determining high-order consistency characteristics of each matched pair according to the low-order consistency matrix;
and adopting a fully connected network, and respectively carrying out classification judgment on each matching pair according to each high-order consistency characteristic to obtain the confidence coefficient of each matching pair.
4. The deep learning-based point cloud registration method according to claim 1, wherein calculating a transformation matrix between the first point cloud and the second point cloud according to each confidence level specifically comprises:
respectively carrying out decentration on the first point cloud and the second point cloud according to the confidence degrees to obtain a decentration result;
calculating a covariance matrix according to each confidence coefficient and the decentralization result;
and carrying out singular value decomposition on the covariance matrix, and determining a transformation matrix between the first point cloud and the second point cloud according to a singular value decomposition result.
5. The deep learning based point cloud registration method of claim 1, wherein determining a number of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score, comprises:
selecting a plurality of points with the highest key scores from the first point cloud and the second point cloud respectively to serve as a first point set to be matched and a second point set to be matched;
and matching the points in the first point set to be matched and the second point set to be matched according to the geometric descriptor to obtain a plurality of matching pairs.
6. The deep learning based point cloud registration method of claim 1, wherein determining a first point cloud and a second point cloud from the point cloud dataset comprises:
and selecting any two frames of point clouds with the overlapping degree of 30% -60% from the point cloud data set, and respectively carrying out data enhancement on the two frames of point clouds to obtain a first point cloud and a second point cloud.
7. The deep learning-based point cloud registration method according to claim 2, wherein the encoder network and the decoder network both adopt a KPConv convolution mode, and the encoder network and the decoder network adopt a U-Net mode to perform cross-layer connection.
8. A point cloud registration system based on deep learning, the point cloud registration system comprising:
the data set generation module is used for acquiring laser radar data and generating a point cloud data set according to the laser radar data;
the point cloud determining module is used for determining a first point cloud and a second point cloud according to the point cloud data set; the first point cloud and the second point cloud respectively represent laser radar data of different frames;
the geometric descriptor and criticality score determining module is used for determining geometric descriptors and criticality scores of each point in the first point cloud and the second point cloud by adopting a trained twin neural network;
the matching pair determining module is used for determining a plurality of matching pairs from the first point cloud and the second point cloud according to the geometric descriptor and the criticality score; the matching pair characterizes the one-to-one correspondence of points in the first point cloud and points in the second point cloud;
the confidence determining module is used for determining the confidence of each matched pair by adopting a trained matched pair consistency learning network;
a transformation matrix calculation module, configured to calculate a transformation matrix between the first point cloud and the second point cloud according to each confidence coefficient;
the point cloud registration module is used for registering the first point cloud and the second point cloud according to the transformation matrix to obtain a registration point cloud; the registration point cloud is used for constructing a point cloud map of the laser radar scanning area.
9. An electronic device comprising a memory for storing a computer program and a processor that runs the computer program to cause the electronic device to perform the point cloud registration method of any of claims 1 to 7.
10. A computer-readable storage medium, characterized in that it stores a computer program which, when executed by a processor, implements the point cloud registration method according to any of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310239900.0A CN116071410A (en) | 2023-03-14 | 2023-03-14 | Point cloud registration method, system, equipment and medium based on deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310239900.0A CN116071410A (en) | 2023-03-14 | 2023-03-14 | Point cloud registration method, system, equipment and medium based on deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116071410A true CN116071410A (en) | 2023-05-05 |
Family
ID=86171681
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310239900.0A Pending CN116071410A (en) | 2023-03-14 | 2023-03-14 | Point cloud registration method, system, equipment and medium based on deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116071410A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116942313A (en) * | 2023-06-05 | 2023-10-27 | 北京长木谷医疗科技股份有限公司 | Surgical robot registration method and device based on reinforcement learning and electronic equipment |
-
2023
- 2023-03-14 CN CN202310239900.0A patent/CN116071410A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116942313A (en) * | 2023-06-05 | 2023-10-27 | 北京长木谷医疗科技股份有限公司 | Surgical robot registration method and device based on reinforcement learning and electronic equipment |
| CN116942313B (en) * | 2023-06-05 | 2024-01-16 | 北京长木谷医疗科技股份有限公司 | Surgical robot registration method and device based on reinforcement learning and electronic equipment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111709409B (en) | Face living body detection method, device, equipment and medium | |
| CN103729885B (en) | Various visual angles projection registers united Freehandhand-drawing scene three-dimensional modeling method with three-dimensional | |
| CN110069656B (en) | Method for searching three-dimensional model based on two-dimensional picture of generated countermeasure network | |
| CN105913487A (en) | Human eye image iris contour analyzing and matching-based viewing direction calculating method | |
| CN110009674A (en) | Monocular image depth of field real-time computing technique based on unsupervised deep learning | |
| CN113361560B (en) | Semantic-based multi-pose virtual fitting method | |
| JP2001283229A (en) | Method for calculating position and direction of object in three-dimensional space | |
| CN110287873A (en) | Noncooperative target pose measuring method, system and terminal device based on deep neural network | |
| CN112784782B (en) | Three-dimensional object identification method based on multi-view double-attention network | |
| CN106844620B (en) | View-based feature matching three-dimensional model retrieval method | |
| CN105701455A (en) | Active shape model (ASM) algorithm-based face characteristic point acquisition and three dimensional face modeling method | |
| CN109886351A (en) | High-spectral data and high-definition picture fusion method | |
| CN111652273A (en) | Deep learning-based RGB-D image classification method | |
| CN116071410A (en) | Point cloud registration method, system, equipment and medium based on deep learning | |
| CN113743417A (en) | Semantic segmentation method and semantic segmentation device | |
| CN111507184B (en) | Human body posture detection method based on parallel cavity convolution and body structure constraint | |
| CN114782503A (en) | Point cloud registration method and system based on multi-scale feature similarity constraint | |
| CN105975906A (en) | PCA static gesture recognition method based on area characteristic | |
| CN115222954A (en) | Weak perception target detection method and related equipment | |
| CN113516693B (en) | Rapid and universal image registration method | |
| Zhang | Innovation of English teaching model based on machine learning neural network and image super resolution | |
| CN112215285B (en) | Cross-media-characteristic-based automatic fundus image labeling method | |
| CN111709269B (en) | Human hand segmentation method and device based on two-dimensional joint information in depth image | |
| CN117274388A (en) | Unsupervised three-dimensional visual positioning method and system based on visual text relation alignment | |
| CN111127532B (en) | Medical image deformation registration method and system based on deep learning characteristic optical flow |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |