CN116523979B - Point cloud registration method and device based on deep learning and electronic equipment - Google Patents
Point cloud registration method and device based on deep learning and electronic equipment Download PDFInfo
- Publication number
- CN116523979B CN116523979B CN202310457614.1A CN202310457614A CN116523979B CN 116523979 B CN116523979 B CN 116523979B CN 202310457614 A CN202310457614 A CN 202310457614A CN 116523979 B CN116523979 B CN 116523979B
- Authority
- CN
- China
- Prior art keywords
- registration
- point set
- feature map
- points
- deep learning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000013135 deep learning Methods 0.000 title claims abstract description 25
- 239000011159 matrix material Substances 0.000 claims abstract description 58
- 238000013136 deep learning model Methods 0.000 claims abstract description 26
- 238000012545 processing Methods 0.000 claims description 31
- 238000010586 diagram Methods 0.000 claims description 25
- 238000011176 pooling Methods 0.000 claims description 19
- 238000000605 extraction Methods 0.000 claims description 11
- 238000004590 computer program Methods 0.000 claims description 8
- 230000006870 function Effects 0.000 description 12
- 230000008569 process Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 238000013519 translation Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 230000001902 propagating effect Effects 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 208000036487 Arthropathies Diseases 0.000 description 1
- RYGMFSIKBFXOCR-UHFFFAOYSA-N Copper Chemical compound [Cu] RYGMFSIKBFXOCR-UHFFFAOYSA-N 0.000 description 1
- 208000012659 Joint disease Diseases 0.000 description 1
- 206010003246 arthritis Diseases 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 229910052802 copper Inorganic materials 0.000 description 1
- 239000010949 copper Substances 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000001624 hip Anatomy 0.000 description 1
- 210000004394 hip joint Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000017074 necrotic cell death Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000001356 surgical procedure Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0012—Biomedical image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30004—Biomedical image processing
- G06T2207/30008—Bone
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Molecular Biology (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Quality & Reliability (AREA)
- Databases & Information Systems (AREA)
- Radiology & Medical Imaging (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a point cloud registration method and device based on deep learning and electronic equipment, wherein the method comprises the following steps: acquiring a first registration point set, and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that in the first registration point set; based on a deep learning model, extracting features of the first registration point set and the second registration point set to obtain a registration feature map; determining a target registration matrix based on the registration feature map and a third registration point set, wherein the third registration point set and the first registration point set are obtained based on different registration stages; based on the target registration matrix, corresponding relations between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set are determined. The point cloud registration method can reduce errors, improve accuracy of registration results, reduce difficulty of registration operation and improve use experience of operators.
Description
Technical Field
The present invention relates to the field of medical treatment, and in particular, to a point cloud registration method and apparatus based on deep learning, and an electronic device.
Background
With the continuous development of medical technology, more and more surgical tools are appeared to assist doctors in performing operations, and particularly with the appearance of surgical robots, great convenience is brought to doctors.
For example, as surgical robots are becoming popular for acetabular surgery, there is an increasing demand for accuracy in pre-operative and intra-operative registration of the hip joints. However, there is a large uncertainty in the error in acetabular registration due to the diversity of hip arthropathy, such as fractures, femoral head necrosis, arthritis, etc. Meanwhile, because the number of the registration points in the acetabular registration process is small (namely sparse point cloud), the registration difficulty is higher than that of the traditional point cloud registration, and a great challenge is caused.
Therefore, how to solve the above-mentioned problems is considered.
Disclosure of Invention
The invention provides a point cloud registration method and device based on deep learning and electronic equipment, which are used for solving the problems.
In a first aspect, the present invention provides a point cloud registration method based on deep learning, including:
acquiring a first registration point set, and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that in the first registration point set;
based on a deep learning model, extracting features of the first registration point set and the second registration point set to obtain a registration feature map;
determining a target registration matrix based on the registration feature map and a third registration point set, wherein the third registration point set and the first registration point set are obtained based on different registration stages;
based on the target registration matrix, corresponding relations between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set are determined.
Optionally, based on the deep learning model, feature extraction is performed on the first registration point set and the second registration point set to obtain a registration feature map, including:
performing multi-layer sensing processing operation and pooling processing operation on the first registration point set based on a deep learning model to obtain a global feature map;
performing multi-layer sensing processing operation and pooling processing operation on the second registration point set based on a deep learning model to obtain a local feature map;
and obtaining the registration feature map based on the global feature map and the local feature map.
Optionally, the obtaining the registration feature map based on the global feature map and the local feature map includes:
performing Value operation on the local feature map to obtain a first feature map;
performing Query operation and Key operation on the global feature map respectively to obtain a second feature map and a third feature map;
performing Softmax operation on the second characteristic diagram and the third characteristic diagram to obtain a self-attention coefficient;
and performing point multiplication operation on the first characteristic map and the self-attention coefficient to obtain the registration characteristic map.
Optionally, the determining the target registration matrix based on the registration feature map and the third registration point set includes:
determining a first center point of a point set in the registration feature map and a second center point of the third registration point set;
calculating a first distance between each alignment point in the registration feature map and the first center point and a second distance between each alignment point in the third alignment point set and the second center point;
determining a rotational portion and a translational portion of a target registration matrix based on the first distance and the second distance;
and obtaining the target registration matrix based on the rotating part and the translating part.
Optionally, the determining a rotation portion and a translation portion of the target registration matrix based on the first distance and the second distance includes:
determining a first numerical value based on the first distance and the coordinate value of each registration point in the registration feature map;
determining a second value based on the second distance and the coordinate value of each registration point in the third registration point set;
and determining a rotating part and a translating part of the target registration matrix when the difference value between the first numerical value and the second numerical value is zero.
Optionally, the determining, based on the target registration matrix, correspondence between the plurality of registration points in the first registration point set and the plurality of registration points in the third registration point set includes:
obtaining a plurality of registration points corresponding to the third registration point set based on the product of the plurality of registration points in the first registration point set and the target matrix; or,
and obtaining a plurality of registration points corresponding to the first registration point set based on the product of the plurality of registration points in the third registration point set and the inverse matrix of the target matrix.
Optionally, the number of registration points in the second registration point set is less than one half of the number of registration points in the first registration point set.
In a second aspect, the present invention provides a point cloud registration device based on deep learning, including:
the acquisition module is used for acquiring a first registration point set and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that of registration points in the first registration point set;
the extraction module is used for extracting features of the first registration point set and the second registration point set based on a deep learning model to obtain a registration feature map;
a determining module, configured to determine a target registration matrix based on the registration feature map and a third registration point set, where the third registration point set and the first registration point set are obtained based on different registration stages;
the determining module is further configured to determine correspondence between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set based on the target registration matrix.
In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a deep learning based point cloud registration method as described above when executing the program.
In a fourth aspect, the present invention provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements a deep learning based point cloud registration method as described above.
The technical scheme of the invention has at least the following beneficial effects:
according to the point cloud registration method based on the deep learning, the second registration point set is obtained by selecting part of the quasi points from the first registration point set, and the feature extraction is carried out on the first registration point set and the second registration point set through the deep learning model, so that the features of each registration point in the first registration point set and the second registration point set are fully combined, and the obtained registration feature map can be more accurate. And the target registration matrix obtained based on the registration feature map and the third registration point set is more accurate, so that errors can be reduced when the registration points in the first registration point set and the registration points in the third registration point set are determined based on the target registration matrix, the accuracy of registration results is improved, the difficulty of registration operation is reduced, and the use experience of operators is improved.
Drawings
In order to more clearly illustrate the invention or the technical solutions of the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of a point cloud registration method based on deep learning provided by the invention;
FIG. 2 is a schematic structural diagram of a deep learning model according to the present invention;
fig. 3 is a schematic block diagram of a point cloud registration device based on deep learning provided by the invention;
fig. 4 is a schematic structural diagram of an electronic device according to the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein.
It should be understood that, in various embodiments of the present invention, the sequence number of each process does not mean that the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
It should be understood that in the present invention, "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements that are expressly listed or inherent to such process, method, article, or apparatus.
It should be understood that in the present invention, "plurality" means two or more. "and/or" is merely an association relationship describing an association object, and means that three relationships may exist, for example, and/or B may mean: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship. "comprising A, B and C", "comprising A, B, C" means that all three of A, B, C comprise, "comprising A, B or C" means that one of the three comprises A, B, C, and "comprising A, B and/or C" means that any 1 or any 2 or 3 of the three comprises A, B, C.
It should be understood that in the present invention, "B corresponding to a", "a corresponding to B", or "B corresponding to a" means that B is associated with a, from which B can be determined. Determining B from a does not mean determining B from a alone, but may also determine B from a and/or other information. The matching of A and B is that the similarity of A and B is larger than or equal to a preset threshold value.
As used herein, "if" may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to detection" depending on the context.
The technical scheme of the invention is described in detail below by specific examples. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
Referring to fig. 1, a schematic flow chart of a point cloud registration method based on deep learning is provided. The point cloud registration method comprises the following steps:
s11: and acquiring a first registration point set, and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that in the first registration point set.
It should be noted that the first registration point set may also be referred to as an intra-operative registration point set, i.e. a plurality of registration points in the first registration point set are acquired intra-operatively. The alignment points in the second alignment point set are part of the alignment points in the first alignment point set. The second set of registration points may also be referred to as an intraoperative local set of registration points.
S12: and carrying out feature extraction on the first registration point set and the second registration point set based on a deep learning model to obtain a registration feature map.
It should be noted that, by extracting features from the first registration point set and the second registration point set, the obtained registration feature map can retain more features, and the result is more accurate.
S13: and determining a target registration matrix based on the registration feature map and a third registration point set, wherein the third registration point set and the first registration point set are obtained based on different registration stages.
It should be noted that the third registration point set may also be referred to as a pre-operation registration point set, that is, the plurality of registration points in the third registration point set are points that have been planned before operation.
S14: based on the target registration matrix, corresponding relations between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set are determined.
Optionally, the number of registration points in the first set of registration points is the same as the number of registration points in the third set of registration points.
According to the point cloud registration method based on the deep learning, the second registration point set is obtained by selecting part of the quasi points from the first registration point set, and the feature extraction is carried out on the first registration point set and the second registration point set through the deep learning model, so that the features of each registration point in the first registration point set and the second registration point set are fully combined, and the obtained registration feature map can be more accurate. And the target registration matrix obtained based on the registration feature map and the third registration point set is more accurate, so that errors can be reduced when the registration points in the first registration point set and the registration points in the third registration point set are determined based on the target registration matrix, the accuracy of registration results is improved, the difficulty of registration operation is reduced, and the use experience of operators is improved.
For example, referring to fig. 2, a schematic diagram of a network structure of a deep learning model is provided in the present invention. And the deep learning model can be described by taking a transducer network model as an example.
The feature extraction is performed on the first registration point set (represented by an intra-operative registration point set in fig. 2) and the second registration point set (represented by an intra-operative local registration point set in fig. 2) based on a deep learning model, so as to obtain a registration feature map, which comprises the following steps:
s21: and performing multi-layer sensing MLP processing operation and pooling processing operation on the first registration point set based on the deep learning model to obtain a global feature map.
Alternatively, the global registration feature map may be represented by FG.
S22: and carrying out multi-layer sensing processing operation and pooling processing operation on the second registration point set based on the deep learning model to obtain a local feature map.
Alternatively, the local feature map may be represented by FL.
The pooling operation has the following functions: 1) And downsampling, dimension reduction and redundant information removal are performed, meanwhile, the perception view is increased, the characteristic information of the feature map is reserved, and the parameter quantity is reduced. 2) The nonlinearity is realized, and the occurrence of overfitting can be prevented to a certain extent. 3) Feature invariance (feature invariant) may be achieved, wherein feature invariance includes translational invariance, rotational invariance, and dimensional invariance. The pooling operation makes the model more concerned about whether certain features exist rather than feature-specific locations. Can be regarded as a strong prior, so that the feature learning contains a certain degree of freedom and can tolerate some tiny displacement of the features.
Optionally, the pooling operation includes an average pooling average-pooling operation and a maximum pooling max-pooling operation. The average-scaling can reduce errors generated by the variance of the estimated values caused by the limited size of the neighborhood, so that the background information of the image is reserved more. max-pooling can reduce errors caused by deviation of the estimated mean value due to errors of parameters of a convolution layer, so that texture information is reserved more.
S23: and obtaining the registration feature map based on the global feature map and the local feature map.
As the global feature map and the local feature map are subjected to multi-layer perception processing operation and pooling processing, the obtained registration feature map can retain more feature information, errors are reduced, and the result is more accurate.
By way of example, with continued reference to fig. 2, the obtaining the registration feature map based on the global feature map and the local feature map includes the steps of:
s31: and performing Value operation on the local feature map FG to obtain a first feature map.
S32: and respectively performing Query operation and Key operation on the global feature map to obtain a second feature map and a third feature map.
Alternatively, value operation, query operation, and Key operation are denoted by V, Q and K, respectively, in fig. 2.
S33: and carrying out Softmax operation on the second characteristic diagram and the third characteristic diagram to obtain a self-attention coefficient.
Alternatively, the Softmax operation is represented in fig. 2, and the self-attention coefficient is represented in fig. 2 by c_v.
S34: and performing point multiplication operation on the first characteristic map and the self-attention coefficient to obtain the registration characteristic map.
Alternatively, the dot product operation is shown in FIG. 2Expressed, the registration feature map is expressed as Output.
Specifically, the registration feature map may be calculated as follows:
C_V(K,Q)＝Softmax(K T Q)
illustratively, the registration feature map is based onAnd a third set of registration points, determining a target registration matrix, comprising:
and determining a first central point of the point set in the registration feature map and a second central point of the third registration point set.
Specifically, the first center point and the second center point are respectively calculated by the following modes:
wherein (1)>For registering the i-th point in the feature map point set, < > j->For the i-th point in the third set of registration points,/->Center of point set representing registration feature, +.>Representing the center of the third set of registration points,point and->The points each have their corresponding coordinate values.
And calculating a first distance between each alignment point in the registration feature map and the first center point and a second distance between each alignment point in the third alignment point set and the second center point.
In particular, the method comprises the steps of,wherein (1)>For registering the set of distances of each alignment point in the feature map point set to the first center point,/>A set of distances from the second center point for each of the third set of registration points.
Determining a rotational portion and a translational portion of a target registration matrix based on the first distance and the second distance;
and obtaining the target registration matrix based on the rotating part and the translating part.
For example, the determining a rotation portion and a translation portion of the target registration matrix based on the first distance and the second distance includes:
determining a first numerical value based on the first distance and the coordinate value of each registration point in the registration feature map;
determining a second value based on the second distance and the coordinate value of each registration point in the third registration point set;
and determining a rotating part and a translating part of the target registration matrix when the difference value between the first numerical value and the second numerical value is zero.
Optionally, an optimal objective function E is defined, and when the optimal objective function E is zero, a rotation part and a translation part of the target registration matrix are determined.
In particular, the method comprises the steps of,
where R represents the rotated portion of the target registration matrix and T represents the translated portion of the target registration matrix. When E is zero, the rotation part R and the translation part T can be obtained by substituting the numerical calculation.
After the target registration matrix is obtained, the registration points in the preoperative registration point set can be obtained based on registration points in the operative registration point set, or the registration points in the operative registration point set can be obtained based on registration points in the preoperative registration point set.
For example, the determining, based on the target registration matrix, correspondence between the plurality of registration points in the first set of registration points and the plurality of registration points in the third set of registration points includes:
obtaining a plurality of registration points corresponding to the third registration point set based on the product of the plurality of registration points in the first registration point set and the target matrix; or,
and obtaining a plurality of registration points corresponding to the first registration point set based on the product of the plurality of registration points in the third registration point set and the inverse matrix of the target matrix.
For example, the number of registration points in the second set of registration points is less than one half of the number of registration points in the first set of registration points.
Alternatively, the number of registration points in the first set of registration points may be between 30 and 40, such as 32, 35, 38, and the number of registration points in the second set of registration points may be 15, 16, 17. The registration operation is carried out through a plurality of registration points, so that the registration accuracy can be ensured, and the registration reliability is improved.
Referring next to fig. 3, based on the same technical concept as the above-mentioned point cloud registration method based on deep learning, the present invention further provides a point cloud registration device based on deep learning, where the function of the point cloud registration device is the same as that of the above-mentioned point cloud registration method, and will not be described herein.
The point cloud registration device comprises:
an obtaining module 31, configured to obtain a first set of registration points, and obtain a second set of registration points based on the first set of registration points, where the number of registration points in the second set of registration points is less than the number of registration points in the first set of registration points;
an extracting module 32, configured to perform feature extraction on the first registration point set and the second registration point set based on a deep learning model, so as to obtain a registration feature map;
a determining module 33, configured to determine a target registration matrix based on the registration feature map and a third registration point set, where the third registration point set and the first registration point set are obtained based on different registration phases;
the determining module 33 is further configured to determine correspondence between a plurality of registration points in the first set of registration points and a plurality of registration points in the third set of registration points based on the target registration matrix.
Optionally, the extracting module 31 is specifically configured to, when performing feature extraction on the first registration point set and the second registration point set based on a deep learning model to obtain a registration feature map:
performing multi-layer sensing processing operation and pooling processing operation on the first registration point set based on a deep learning model to obtain a global feature map;
performing multi-layer sensing processing operation and pooling processing operation on the second registration point set based on a deep learning model to obtain a local feature map;
and obtaining the registration feature map based on the global feature map and the local feature map.
Optionally, the extracting module 31 is specifically configured to, when obtaining the registration feature map based on the global feature map and the local feature map:
performing Value operation on the local feature map to obtain a first feature map;
performing Query operation and Key operation on the global feature map respectively to obtain a second feature map and a third feature map;
performing Softmax operation on the second characteristic diagram and the third characteristic diagram to obtain a self-attention coefficient;
and performing point multiplication operation on the first characteristic map and the self-attention coefficient to obtain the registration characteristic map.
Optionally, the determining module 33 is specifically configured to, when determining the target registration matrix based on the registration feature map and the third registration point set:
determining a first center point of a point set in the registration feature map and a second center point of the third registration point set;
calculating a first distance between each alignment point in the registration feature map and the first center point and a second distance between each alignment point in the third alignment point set and the second center point;
determining a rotational portion and a translational portion of a target registration matrix based on the first distance and the second distance;
and obtaining the target registration matrix based on the rotating part and the translating part.
Optionally, the determining module 33 is specifically configured to, when determining the rotating part and the translating part of the target registration matrix based on the first distance and the second distance:
determining a first numerical value based on the first distance and the coordinate value of each registration point in the registration feature map;
determining a second value based on the second distance and the coordinate value of each registration point in the third registration point set;
and determining a rotating part and a translating part of the target registration matrix when the difference value between the first numerical value and the second numerical value is zero.
Optionally, the determining module 33 is specifically configured to, when determining, based on the target registration matrix, correspondence between a plurality of registration points in the first set of registration points and a plurality of registration points in the third set of registration points:
obtaining a plurality of registration points corresponding to the third registration point set based on the product of the plurality of registration points in the first registration point set and the target matrix; or,
and obtaining a plurality of registration points corresponding to the first registration point set based on the product of the plurality of registration points in the third registration point set and the inverse matrix of the target matrix.
Optionally, the number of registration points in the second registration point set is less than one half of the number of registration points in the first registration point set.
Next, referring to fig. 4, a schematic structural diagram of an electronic device according to the present invention is provided. The electronic device may include: processor 410, communication interface (Communications Interface) 420, memory 430 and communication bus 440, wherein processor 410, communication interface 420 and memory 430 communicate with each other via communication bus 440. The processor 410 may invoke logic instructions in the memory 430 to perform the depth-learning based point cloud registration method provided by the methods described above.
Further, the logic instructions in the memory 430 described above may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as a stand-alone product. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
In another aspect of the invention, a computer readable storage medium is provided, having stored thereon computer program instructions which, when executed by a processor, implement a deep learning based point cloud registration method as described above.
The computer readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: portable computer disks, hard disks, random Access Memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static Random Access Memory (SRAM), portable compact disk read-only memory (CD-ROM), digital Versatile Disks (DVD), memory sticks, floppy disks, mechanical coding devices, punch cards or in-groove structures such as punch cards or grooves having instructions stored thereon, and any suitable combination of the foregoing. Computer-readable storage media, as used herein, are not to be construed as transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., optical pulses through fiber optic cables), or electrical signals transmitted through wires.
The computer readable program instructions described herein may be downloaded from a computer readable storage medium to a respective computing/processing device or to an external computer or external storage device over a network, such as the internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers and/or edge servers. The network interface card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium in the respective computing/processing device.
Computer program instructions for carrying out operations of the present invention may be assembly instructions, instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, c++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, aspects of the present invention are implemented by personalizing electronic circuitry, such as programmable logic circuitry, field Programmable Gate Arrays (FPGAs), or Programmable Logic Arrays (PLAs), with state information for computer readable program instructions, which can execute the computer readable program instructions.
Various aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.
These computer readable program instructions may be provided to a processing unit of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, programmable data processing apparatus, and/or other devices to function in a particular manner, such that the computer readable medium having the instructions stored therein includes an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.
The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer, other programmable apparatus or other devices implement the functions/acts specified in the flowchart and/or block diagram block or blocks.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
Note that all features disclosed in this specification (including any accompanying claims, abstract and drawings) may be replaced by alternative features serving the same, equivalent or similar purpose, unless expressly stated otherwise. Thus, unless expressly stated otherwise, each feature disclosed is one example only of a generic set of equivalent or similar features. Where used, further, preferably, still further and preferably, the brief description of the other embodiment is provided on the basis of the foregoing embodiment, and further, preferably, further or more preferably, the combination of the contents of the rear band with the foregoing embodiment is provided as a complete construct of the other embodiment. A further embodiment is composed of several further, preferably, still further or preferably arrangements of the strips after the same embodiment, which may be combined arbitrarily.
It will be appreciated by persons skilled in the art that the embodiments of the invention described above and shown in the drawings are by way of example only and are not limiting. The objects of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been shown and described in the examples and embodiments of the invention may be modified or practiced without departing from the principles described.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present disclosure, and not for limiting the same; although the present disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present disclosure.
Claims (8)
1. A point cloud registration method based on deep learning, comprising:
acquiring a first registration point set, and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that in the first registration point set;
based on a deep learning model, extracting features of the first registration point set and the second registration point set to obtain a registration feature map;
determining a target registration matrix based on the registration feature map and a third registration point set, wherein the third registration point set and the first registration point set are obtained based on different registration stages;
based on the target registration matrix, determining corresponding relations between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set;
the feature extraction is performed on the first registration point set and the second registration point set based on the deep learning model to obtain a registration feature map, including:
performing multi-layer sensing processing operation and pooling processing operation on the first registration point set based on a deep learning model to obtain a global feature map;
performing multi-layer sensing processing operation and pooling processing operation on the second registration point set based on a deep learning model to obtain a local feature map;
obtaining the registration feature map based on the global feature map and the local feature map;
the obtaining the registration feature map based on the global feature map and the local feature map includes:
performing Value operation on the local feature map to obtain a first feature map;
performing Query operation and Key operation on the global feature map respectively to obtain a second feature map and a third feature map;
performing Softmax operation on the second characteristic diagram and the third characteristic diagram to obtain a self-attention coefficient;
and performing point multiplication operation on the first characteristic map and the self-attention coefficient to obtain the registration characteristic map.
2. The deep learning based point cloud registration method of claim 1, wherein the determining a target registration matrix based on the registration feature map and a third registration point set comprises:
determining a first center point of a point set in the registration feature map and a second center point of the third registration point set;
calculating a first distance between each alignment point in the registration feature map and the first center point and a second distance between each alignment point in the third alignment point set and the second center point;
determining a rotational portion and a translational portion of a target registration matrix based on the first distance and the second distance;
and obtaining the target registration matrix based on the rotating part and the translating part.
3. The deep learning based point cloud registration method of claim 2, wherein the determining a rotational portion and a translational portion of a target registration matrix based on the first distance and the second distance comprises:
determining a first numerical value based on the first distance and the coordinate value of each registration point in the registration feature map;
determining a second value based on the second distance and the coordinate value of each registration point in the third registration point set;
and determining a rotating part and a translating part of the target registration matrix when the difference value between the first numerical value and the second numerical value is zero.
4. The deep learning based point cloud registration method of claim 1, wherein the determining, based on the target registration matrix, correspondence between the plurality of registration points in the first set of registration points and the plurality of registration points in the third set of registration points includes:
obtaining a plurality of registration points corresponding to the third registration point set based on the product of the plurality of registration points in the first registration point set and the target registration matrix; or,
and obtaining a plurality of registration points corresponding to the first registration point set based on the product of the plurality of registration points in the third registration point set and the inverse matrix of the target registration matrix.
5. The deep learning based point cloud registration method of any of claims 1 to 4, wherein the number of registration points in the second set of registration points is less than one half of the number of registration points in the first set of registration points.
6. A point cloud registration device based on deep learning, comprising:
the acquisition module is used for acquiring a first registration point set and acquiring a second registration point set based on the first registration point set, wherein the number of registration points in the second registration point set is smaller than that of registration points in the first registration point set;
the extraction module is used for extracting features of the first registration point set and the second registration point set based on a deep learning model to obtain a registration feature map;
a determining module, configured to determine a target registration matrix based on the registration feature map and a third registration point set, where the third registration point set and the first registration point set are obtained based on different registration stages;
the determining module is further used for determining corresponding relations between a plurality of registration points in the first registration point set and a plurality of registration points in the third registration point set based on the target registration matrix;
the extraction module is also used for: performing multi-layer sensing processing operation and pooling processing operation on the first registration point set based on a deep learning model to obtain a global feature map; performing multi-layer sensing processing operation and pooling processing operation on the second registration point set based on a deep learning model to obtain a local feature map; obtaining the registration feature map based on the global feature map and the local feature map;
the obtaining the registration feature map based on the global feature map and the local feature map includes:
performing Value operation on the local feature map to obtain a first feature map; performing Query operation and Key operation on the global feature map respectively to obtain a second feature map and a third feature map; performing Softmax operation on the second characteristic diagram and the third characteristic diagram to obtain a self-attention coefficient; and performing point multiplication operation on the first characteristic map and the self-attention coefficient to obtain the registration characteristic map.
7. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the deep learning based point cloud registration method of any of claims 1 to 5 when the program is executed.
8. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the deep learning based point cloud registration method according to any of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310457614.1A CN116523979B (en) | 2023-04-24 | 2023-04-24 | Point cloud registration method and device based on deep learning and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310457614.1A CN116523979B (en) | 2023-04-24 | 2023-04-24 | Point cloud registration method and device based on deep learning and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116523979A CN116523979A (en) | 2023-08-01 |
| CN116523979B true CN116523979B (en) | 2024-01-30 |
Family
ID=87391545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310457614.1A Active CN116523979B (en) | 2023-04-24 | 2023-04-24 | Point cloud registration method and device based on deep learning and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116523979B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106780459A (en) * | 2016-12-12 | 2017-05-31 | 华中科技大学 | A kind of three dimensional point cloud autoegistration method |
| CN113256692A (en) * | 2021-05-14 | 2021-08-13 | 西安交通大学 | Rigid body registration method and equipment based on adaptive neighborhood weight learning |
| CN114419570A (en) * | 2022-03-28 | 2022-04-29 | 苏州浪潮智能科技有限公司 | Point cloud data identification method and device, electronic equipment and storage medium |
| CN115908514A (en) * | 2022-10-18 | 2023-04-04 | 西安电子科技大学 | Point cloud registration method based on fusion of global features and local features |
| CN115994933A (en) * | 2023-02-10 | 2023-04-21 | 重庆邮电大学 | Partial point cloud registration method based on consistency learning |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11875424B2 (en) * | 2021-03-15 | 2024-01-16 | Shenzhen University | Point cloud data processing method and device, computer device, and storage medium |
-
2023
- 2023-04-24 CN CN202310457614.1A patent/CN116523979B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106780459A (en) * | 2016-12-12 | 2017-05-31 | 华中科技大学 | A kind of three dimensional point cloud autoegistration method |
| CN113256692A (en) * | 2021-05-14 | 2021-08-13 | 西安交通大学 | Rigid body registration method and equipment based on adaptive neighborhood weight learning |
| CN114419570A (en) * | 2022-03-28 | 2022-04-29 | 苏州浪潮智能科技有限公司 | Point cloud data identification method and device, electronic equipment and storage medium |
| CN115908514A (en) * | 2022-10-18 | 2023-04-04 | 西安电子科技大学 | Point cloud registration method based on fusion of global features and local features |
| CN115994933A (en) * | 2023-02-10 | 2023-04-21 | 重庆邮电大学 | Partial point cloud registration method based on consistency learning |
Non-Patent Citations (1)
| Title |
|---|
| 基于特征融合的点云配准技术研究;杨珊珊;中国优秀硕士学位论文全文数据库;第2022卷(第07期);1-71 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116523979A (en) | 2023-08-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220114750A1 (en) | Map constructing method, positioning method and wireless communication terminal | |
| JP4985516B2 (en) | Information processing apparatus, information processing method, and computer program | |
| US20100080470A1 (en) | Tagging images by determining a set of similar pre-tagged images and extracting prominent tags from that set | |
| EP3968285A2 (en) | Model training method and apparatus, keypoint positioning method and apparatus, device and medium | |
| JP2020524355A5 (en) | ||
| KR20220062622A (en) | Data processing methods and related devices | |
| CN112382359B (en) | Patient registration method and device, electronic equipment and computer readable medium | |
| WO2022156425A1 (en) | Minimally invasive surgery instrument positioning method and system | |
| CN111784776A (en) | Visual positioning method and device, computer readable medium and electronic equipment | |
| KR20220074782A (en) | Method and device for simultaneous localization and mapping (slam) | |
| CN114219717A (en) | Point cloud registration method and device, electronic equipment and storage medium | |
| CN113592015B (en) | Method and device for positioning and training feature matching network | |
| CN111815593A (en) | Lung nodule domain adaptive segmentation method and device based on counterstudy and storage medium | |
| CN111368860B (en) | Repositioning method and terminal equipment | |
| CN111739071A (en) | Rapid iterative registration method, medium, terminal and device based on initial value | |
| Hutchcroft et al. | CoVisPose: Co-visibility pose transformer for wide-baseline relative pose estimation in 360∘ indoor panoramas | |
| CN116523979B (en) | Point cloud registration method and device based on deep learning and electronic equipment | |
| CN115239776B (en) | Point cloud registration method, device, equipment and medium | |
| CN114674328B (en) | Map generation method, map generation device, electronic device, storage medium, and vehicle | |
| EP4086853A2 (en) | Method and apparatus for generating object model, electronic device and storage medium | |
| CN116012875A (en) | Human body posture estimation method and related device | |
| CN113240743B (en) | Heterogeneous image pose estimation and registration method, device and medium based on neural network | |
| CN113744236B (en) | Loop detection method, device, storage medium and computer program product | |
| CN116523982B (en) | Sparse point cloud registration method and device based on similarity compatibility measurement | |
| CN116563356B (en) | Global 3D registration method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |