CN116258835A - Point cloud data three-dimensional reconstruction method and system based on deep learning - Google Patents

Point cloud data three-dimensional reconstruction method and system based on deep learning Download PDF

Info

Publication number
CN116258835A
CN116258835A CN202310483537.7A CN202310483537A CN116258835A CN 116258835 A CN116258835 A CN 116258835A CN 202310483537 A CN202310483537 A CN 202310483537A CN 116258835 A CN116258835 A CN 116258835A
Authority
CN
China
Prior art keywords
point cloud
point
points
cloud data
vector
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202310483537.7A
Other languages
Chinese (zh)
Other versions
CN116258835B (en
Inventor
石小川
马超
张典
李烺琛
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University WHU
Original Assignee
Wuhan University WHU
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University WHU filed Critical Wuhan University WHU
Priority to CN202310483537.7A priority Critical patent/CN116258835B/en
Publication of CN116258835A publication Critical patent/CN116258835A/en
Application granted granted Critical
Publication of CN116258835B publication Critical patent/CN116258835B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G06V10/42—Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • General Health & Medical Sciences (AREA)
  • Evolutionary Computation (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Databases & Information Systems (AREA)
  • Computational Linguistics (AREA)
  • Computer Graphics (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Geometry (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a point cloud data three-dimensional reconstruction method and system based on deep learning, comprising the following steps: step 1, aiming at an object to be scanned, carrying out multiple scanning from a plurality of different angles by utilizing a three-dimensional laser scanner to obtain a plurality of point cloud data of the same object to be scanned; step 2, preprocessing the collected multiple point cloud data, including removing discrete points and recovering noise points to correct positions; step 3, calculating a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, aligning the source point cloud to the coordinate system where the target point cloud is located, thereby realizing the registration of the point clouds and finally obtaining a complete point cloud image; and 4, carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and carrying out optimization and rendering to obtain a three-dimensional model approaching to a real object. The invention can convert the real non-matter cultural heritage into a digital model which is permanently stored in a computer.

Description

Point cloud data three-dimensional reconstruction method and system based on deep learning
Technical Field
The invention relates to the technical field of information, and mainly aims at real non-material cultural heritage, and provides an automatic method and system for constructing point cloud data into three-dimensional modeling by three steps of denoising, registration and surface reconstruction by using a deep learning method.
Background
Because the cultural relics are easy to damage, and most cultural relics have natural damage phenomena (oxidation, corrosion, rust and the like), a large amount of resources are required to be input for protecting the physical objects of the cultural relics. In recent years, with the continuous development of laser scanning technology and computer technology, digital cultural relic protection is increasingly applied, such as three-dimensional modeling of a cultural relic, virtual reality, digital cultural relic restoration and the like. The three-dimensional modeling of the cultural relics converts the physical objects of the cultural relics into digital models which can be permanently stored in a computer, has the advantages of low cost, high efficiency, easiness in display and the like, and is a currently mainstream digital cultural protection mode. The three-dimensional modeling of the cultural relics needs to use point clouds, wherein the point clouds are original data obtained by laser scanning and are a set of points in space, and each point comprises information such as color, normal vector, intensity value, time and the like besides three-dimensional coordinate information of the point.
In the aspect of modeling of scanned point cloud data, the existing point cloud LOD model building technology has some defects, such as large memory consumption, long processing time, poor stability and the like, and is difficult to process massive point cloud data, and the generated three-dimensional model has poor effect. To address these issues, a more efficient, more stable point cloud processing technique is required. In this respect, deep learning techniques provide a good solution. The deep learning has high efficiency of processing large data sets and autonomy of extracting features, and is very suitable for point cloud data processing. The deep learning technology is successfully applied to the fields of image, voice, natural language processing and the like, and a new thought is provided for point cloud processing. The point cloud processing technology based on deep learning can realize efficient processing and modeling of point cloud, and provides powerful support for protection and inheritance of non-material cultural heritage.
However, the specificity of the point cloud data makes the conventional deep learning model not directly applicable to the processing of the point cloud data. Point cloud data is a special three-dimensional data representation method, which consists of a large number of discrete points, each point containing three coordinate values and other possible attribute values, such as colors, normal vectors, and the like. The specificity of the point cloud data is mainly represented in two aspects, compared with the conventional two-dimensional image data, and has data disorder and affine variation independence. First, there are many points in a cloud of points, which appear in the cloud file in whatever order, and the information they refer to is not changed, but rather, the points in a picture are already arranged in the image in the order they are inherent in. Second, the point cloud data exists in a three-dimensional space with affine variation independence, i.e., rotational invariance, which results in that a conventional convolutional neural network for a two-dimensional image cannot be used for the point cloud data. Because of these special properties, it is difficult for the conventional convolutional neural network model to directly process the point cloud data, and there is no standard flow for deeply processing the point cloud data.
Disclosure of Invention
In order to solve the defects of the prior art, the invention provides a set of three-dimensional modeling method for the point cloud data, which comprises three steps of denoising, registering and surface reconstruction, and the method is used for processing the point cloud data by using a deep learning technology through an improved neural network model so as to obtain a final three-dimensional grid model.
In order to achieve the above purpose, the technical scheme provided by the invention is as follows: the three-dimensional reconstruction method of the point cloud data based on the deep learning comprises the following steps:
step 1, aiming at an object to be scanned, carrying out multiple scanning from a plurality of different angles by utilizing a three-dimensional laser scanner to obtain a plurality of point cloud data of the same object to be scanned;
step 2, preprocessing the collected multiple point cloud data, including removing discrete points and recovering noise points to correct positions;
step 3, calculating a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, aligning the source point cloud to the coordinate system where the target point cloud is located, thereby realizing the registration of the point clouds and finally obtaining a complete point cloud image;
and 4, carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and carrying out optimization and rendering to obtain a three-dimensional model approaching to a real object.
In step 2, firstly, dividing the point cloud data into a plurality of local patches, wherein each local patch takes a central point as the center and comprises all points within a range of a distance r;
secondly, inputting each local patch into a quaternary space transfer network, learning a four-dimensional vector, transferring an original point cloud to a new pose state through the four-dimensional vector, changing the dimension of the pose state through a full connection layer, extracting the characteristics of point clouds one by using a multi-layer perceptron network, and inversely transforming the characteristic vector of each point cloud to restore the original pose; after the processed point-by-point feature vectors are obtained, the feature vectors of each point in one patch are fused to obtain the feature vector of one local patch;
And finally, carrying out regression processing on the fused feature vectors by using a plurality of full connection layers to obtain a final result.
Further, the discrete points are removed and the noise points are recovered by changing the number of channels of the last full connection layer, and the outlier is judged: the number of channels of the last full connection layer is set to 1, and the process description function
Figure SMS_2
，
Figure SMS_4
，
Figure SMS_6
Representing the probability that the point is an outlier, a threshold value is set by itself +.>
Figure SMS_8
I.e. +.>
Figure SMS_10
In the case of->
Figure SMS_12
For outliers, it is removed and the set of all outliers is denoted +.>
Figure SMS_13
The method comprises the steps of carrying out a first treatment on the surface of the And (3) judging noise points: the number of channels of the last full-connection layer is set to 3, the procedure description function +.>
Figure SMS_1
，
Figure SMS_3
，
Figure SMS_5
Representing the offset vector of each learned point, the corrected point is
Figure SMS_7
，
Figure SMS_9
Representing the original noise point cloud, ++>
Figure SMS_11
Representing one point in the original noisy point cloud.
Further, the specific implementation manner of registering any two pieces of point cloud data in the step 3 is as follows;
firstly, respectively extracting features of a source point cloud and a target point cloud to obtain two global features, and then splicing the two global features to obtain a combined feature of the two point clouds, wherein the splicing operation is to correlate the source point cloud and the target point cloud so as to better match and align in subsequent processing;
Then, taking the combined characteristics as input, calculating the transformation relation between the point clouds through five full-connection layers, and finally outputting a seven-dimensional vector, wherein the first three bits are translation vectors, the last four bits are unit quaternions, converting the unit quaternions into a rotation matrix, and carrying out translation and rotation operations on the source point clouds according to the output translation vectors and the rotation matrix, namely aligning the source point clouds to a coordinate system where the target point clouds are positioned;
and finally, fusing the aligned source point cloud and the aligned target point cloud.
Further, in step 4, the surface shape points, the shape outer points and the shape inner points of the point cloud are distinguished through a neural network, then, the voxel surface reconstruction is carried out on each frame of the point cloud on the surface shape points, a triangular mesh model of the point cloud data is obtained, and the shape outer points and the shape inner points do not participate in the reconstruction; the specific implementation of the surface reconstruction is as follows;
an empty triangle mesh is first initialized, and then the following operations are performed:
41, obtaining a cube of the current voxel at (x, y, z), wherein the cube comprises 8 vertexes;
42, acquiring a boundary set of the current cube, wherein the boundary set is a set of points with all occupation probabilities in a [0.5-tau,0.5+tau ] interval, and tau is a set threshold value;
43, for each point in the set, the following is performed:
i. vertices v1 and v2 are selected, where p1<0.5, p2>0.5;
interpolation (v 1, v2, p1, p 2) by interpolation function to obtain a new interpolation point vertex with occupation probability equal to 0.5;
adding interpolation points vertex into a triangle mesh M;
44, constructing a triangle face list of the current cube, wherein the triangle face list comprises points with the occupation probability of 0.5 in the current cube and the vertex generated immediately;
45, for each triangle face, add it to triangle mesh M, when all triangle faces have been added to triangle mesh, the algorithm ends;
further, the neural network learning process is as follows;
(1) Potential vector coding: first for each input point using a point convolution method
Figure SMS_14
Generating a potential vector
Figure SMS_15
Encoder->
Figure SMS_16
Implemented by point cloud convolution network, only the number of channels of the last layer is required to be changed to control vector +.>
Figure SMS_17
Of (1), wherein>
Figure SMS_18
N represents the number of input points for the points in the registered point cloud;
(2) Relative latent vector coding: given an arbitrary query point
Figure SMS_20
Query Point->
Figure SMS_21
From the input points, construct a set of point set neighbors +.>
Figure SMS_24
It comprises->
Figure SMS_26
The nearest k points are set as +. >
Figure SMS_28
Then use the query point +.>
Figure SMS_30
Relative to->
Figure SMS_32
Is>
Figure SMS_19
For each point set neighbor +>
Figure SMS_22
Points of->
Figure SMS_23
Is>
Figure SMS_25
Enhancement is performed, and the potential vectors of these enhancements are subjected to +.>
Figure SMS_27
Processing to obtain relative potential vector->
Figure SMS_29
Wherein->
Figure SMS_31
Is a cascading operation; the query is completed for each input point through the operation;
(3) Feature weighting: through learning
Figure SMS_34
A separate linear layer consisting of->
Figure SMS_36
A corresponding weight vector->
Figure SMS_38
Parameterization, producing->
Figure SMS_40
Relative weight->
Figure SMS_41
，
Figure SMS_43
Representing the dot multiplication operation, wherein the weight sum is 1; finally the relative weights are +.>
Figure SMS_44
Become weight +.>
Figure SMS_33
And average +.>
Figure SMS_35
Query Point->
Figure SMS_37
Feature vector at
Figure SMS_39
From adjacent points->
Figure SMS_42
Is>
Figure SMS_45
Weighted summation results in:
Figure SMS_46
；
(4) Decoding into occupancy probability: linear layer
Figure SMS_47
Feature vector +.>
Figure SMS_48
Decoding to occupancy score->
Figure SMS_49
This is a full connection layer output with dimension 2, which is then converted into occupancy probability by softmax>
Figure SMS_50
；
(5) Voxel surface reconstruction: obtaining the occupation probability of each point
Figure SMS_51
Afterwards, the occupancy probability is->
Figure SMS_52
Is 0.5->
Figure SMS_53
the point cloud of tau is reconstructed as a voxel grid.
Further, the three-dimensional laser scanner adopted in the step 1 comprises a laser pulse scanner and a phase ranging scanner.
The invention also provides a point cloud data three-dimensional reconstruction system based on deep learning, which comprises the following modules:
the point cloud acquisition module is used for scanning the scanned object for multiple times from multiple different angles by utilizing the three-dimensional laser scanner to obtain multiple pieces of point cloud data of the same object to be scanned;
the preprocessing module is used for preprocessing the collected multiple point cloud data, and comprises the steps of removing discrete points and recovering noise points to correct positions;
the registration module calculates a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, and aligns the source point cloud to the coordinate system of the target point cloud, so that the registration of the point clouds is realized, and a complete point cloud image is finally obtained;
and the reconstruction module is used for carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and then carrying out optimization and rendering to obtain a three-dimensional model which approximates to a real object.
Further, in the preprocessing module, firstly, local division is performed, each piece of point cloud data is divided into a plurality of local patches, and each local patch takes a central point as a center and comprises all points within a range of a distance r;
Secondly, inputting each local patch into a quaternary space transfer network, learning a four-dimensional vector, transferring an original point cloud to a new pose state through the four-dimensional vector, changing the dimension of the pose state through a full connection layer, extracting the characteristics of point clouds one by using a multi-layer perceptron network, and inversely transforming the characteristic vector of each point cloud to restore the original pose; after the processed point-by-point feature vectors are obtained, the feature vectors of each point in one patch are fused to obtain the feature vector of one local patch;
and finally, carrying out regression processing on the fused feature vectors by using a plurality of full connection layers to obtain a final result, wherein the specific implementation mode is as follows:
the discrete points are removed and the noise points are recovered by changing the number of channels of the last full-connection layer, and the outlier is judged: the number of channels of the last full connection layer is set to 1, and the process description function
Figure SMS_55
，
Figure SMS_56
，
Figure SMS_58
Representing the probability that the point is an outlier, a threshold value is set by itself +.>
Figure SMS_61
I.e. +.>
Figure SMS_63
In the case of->
Figure SMS_65
For outliers, it is removed and the set of all outliers is denoted +.>
Figure SMS_66
The method comprises the steps of carrying out a first treatment on the surface of the And (3) judging noise points: the number of channels of the last full-connection layer is set to 3, the procedure description function +. >
Figure SMS_54
，
Figure SMS_57
，
Figure SMS_59
Representing the offset vector of each learned point, the corrected point is +.>
Figure SMS_60
，
Figure SMS_62
Representing the original noise point cloud, ++>
Figure SMS_64
Representing one point in the original noisy point cloud.
Further, in the reconstruction module, the surface shape points, the shape outer points and the shape inner points of the point cloud are distinguished through a neural network, then, voxel surface reconstruction is carried out on each frame of the point cloud on the surface shape points, a triangular mesh model of the point cloud data is obtained, and the shape outer points and the shape inner points do not participate in the reconstruction; the specific implementation of the surface reconstruction is as follows;
an empty triangle mesh is first initialized, and then the following operations are performed:
41, obtaining a cube of the current voxel at (x, y, z), wherein the cube comprises 8 vertexes;
42, acquiring a boundary set of the current cube, wherein the boundary set is a set of points with all occupation probabilities in a [0.5-tau,0.5+tau ] interval, and tau is a set threshold value;
43, for each point in the set, the following is performed:
i. vertices v1 and v2 are selected, where p1<0.5, p2>0.5;
interpolation (v 1, v2, p1, p 2) by interpolation function to obtain a new interpolation point vertex with occupation probability equal to 0.5;
adding interpolation points vertex into a triangle mesh M;
44, constructing a triangle face list of the current cube, wherein the triangle face list comprises points with the occupation probability of 0.5 in the current cube and the vertex generated immediately;
45, for each triangle face, add it to triangle mesh M, when all triangle faces have been added to triangle mesh, the algorithm ends;
the neural network learning process is as follows;
(1) Potential vector coding: first for each input point using a point convolution method
Figure SMS_67
Generating a potential vector
Figure SMS_68
Encoder->
Figure SMS_69
Implemented by point cloud convolution network, only the number of channels of the last layer is required to be changed to control vector +.>
Figure SMS_70
Of (1), wherein>
Figure SMS_71
N represents the number of input points for the points in the registered point cloud;
(2) Relative latent vector coding: given an arbitrary query point
Figure SMS_73
Query Point->
Figure SMS_75
From the input points, construct a set of point set neighbors +.>
Figure SMS_77
It comprises->
Figure SMS_79
The nearest k points are set as +.>
Figure SMS_81
Then use the query point +.>
Figure SMS_83
Relative to->
Figure SMS_85
Is>
Figure SMS_72
For each point set neighbor +>
Figure SMS_74
Points of->
Figure SMS_76
Is>
Figure SMS_78
Enhancement is performed, and the potential vectors of these enhancements are subjected to +.>
Figure SMS_80
Processing to obtain relative potential vector->
Figure SMS_82
Wherein->
Figure SMS_84
Is a cascading operation; the query is completed for each input point through the operation;
(3) Feature weighting: through learning
Figure SMS_87
A separate linear layer consisting of->
Figure SMS_88
A corresponding weight vector->
Figure SMS_90
Parameterization, producing->
Figure SMS_93
Relative weight->
Figure SMS_94
，
Figure SMS_97
Representing the dot multiplication operation, wherein the weight sum is 1; finally the relative weights are +.>
Figure SMS_99
Become weight +.>
Figure SMS_86
And average +.>
Figure SMS_89
Query Point->
Figure SMS_91
Feature vector at
Figure SMS_92
From adjacent points->
Figure SMS_95
Is>
Figure SMS_96
Weighted summation results in:
Figure SMS_98
；
(4) Decoding into occupancy probability: linear layer
Figure SMS_100
Feature vector +.>
Figure SMS_101
Decoding to occupancy score->
Figure SMS_102
This is a full connection layer output with dimension 2, which is then converted into occupancy probability by softmax>
Figure SMS_103
；
(5) Voxel surface reconstruction: obtaining the occupation probability of each point
Figure SMS_104
Afterwards, the occupancy probability is->
Figure SMS_105
Is 0.5->
Figure SMS_106
the point cloud of tau is reconstructed as a voxel grid.
Compared with the prior art, the invention has the advantages and beneficial effects that:
(1) Through the thought of local division, a symmetric function and a space transfer network are introduced into the convolutional neural network model, so that the deep learning model can adapt to the disorder and affine change independence of the point cloud data, is suitable for denoising the point cloud data, improves the quality of the point cloud data, and achieves a better three-dimensional model construction effect.
(2) The feature extraction is carried out on the point cloud data through the multi-layer perceptron MLP and the point potential vector relative coding method, so that the influence of sparsity of the point cloud data on the feature extraction is reduced, and the local shape information of the point cloud can be extracted more accurately.
(3) The method introduces an occupancy probability concept and a characteristic weighting method based on an attribute mechanism to determine the shape of the point cloud surface, and solves the problems of low efficiency, low expandability, low generation precision and the like of the traditional three-dimensional model construction method.
The invention creates an automatic system for modeling the point cloud data into a three-dimensional model by using a deep learning technology, and converts the real non-material cultural heritage into a digital model which can be permanently stored in a computer so as to realize the protection and inheritance of the non-material cultural heritage.
Drawings
FIG. 1 is a flow chart of the present invention.
Fig. 2 is a schematic diagram of a denoising module according to the present invention.
Fig. 3 is a schematic view of a surface reconstruction module according to the present invention.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings and examples.
As shown in fig. 1, the method for reconstructing the point cloud data based on the deep learning, provided by the invention, comprises the following steps:
step one, acquiring point cloud data
The point cloud is obtained, and three-dimensional coordinates of the sampling points on the surface of the object are obtained. The most commonly used equipment for acquiring the point cloud data can divide a three-dimensional laser scanner into:
(1) The laser pulse scanner is based on pulse ranging method. The laser generator emits light pulses that strike the surface of the object under test and are reflected back to be received by a receiver on the scanner. The light speed is known, the distance between the scanner and the sampling points can be calculated by the propagation time, and the three-dimensional coordinate of each sampling point can be accurately calculated by combining the position of the scanner and the scanning angle. This pulse ranging method based on time measurement requires a clock system with high accuracy. The method is characterized by wide ranging range, flexible use scene and relatively low precision, and is mostly used in long-distance scanning measurement.
(2) The phase ranging scanner is based on a phase ranging method. Phase ranging is also essentially a pulse ranging method, except that it is based on the phase difference of light waves rather than time for distance calculation. Since light is also a wave, there is a phase difference between the emitted light and the received light, and the propagation time of the light can be calculated indirectly from this phase difference, thereby calculating the distance. The device has higher scanning precision, is generally used for medium-distance or short-distance scanning measurement, and has the precision of millimeter level.
In the application scene of the invention, the scanning object is a non-genetic cultural relic entity or a place for holding the non-genetic cultural activity, and a scanning mode of combining a ground fixed laser scanning system and a phase ranging scanner is adopted.
In an actual scanning scene, because the volume of an object to be scanned is large (such as a building), a shielding object exists, or the scanning angle of the scanner is limited, point cloud data of the whole object surface cannot be acquired in a single scanning, multiple scanning stations are required to be used for scanning for multiple times at multiple different angles, and then multiple pieces of point cloud data of the same object to be scanned are obtained.
Step two, denoising module
In the original process of point cloud acquisition, due to the influence of acquisition environment (light, water vapor in air, equipment shake and shielding), scanning equipment (equipment fault and low scanning precision) and scanned objects (the surfaces of the objects have tiny concave-convex), noise points generally exist in the point cloud obtained by scanning, the noise points not only increase the data volume of the point cloud, but also influence the accuracy of the later point cloud processing step, and thus the final modeling effect is influenced. The first step in the processing effort is to remove these noise points.
The initial input of the module is a frame of point cloud data, and before the point cloud data is input to a network, the point cloud data needs to be divided locally, the whole point cloud is divided into a plurality of local patches, and each local patch takes a central point as a center and comprises all points within a range of a distance r. Therefore, the point cloud data can be divided into a plurality of smaller blocks, and network learning is facilitated.
Next, a four-dimensional vector is learned by using a quaternary space transfer network, and the original point cloud is transferred to a new pose state through the vector so as to ensure affine change independence of the model on a specific space. Then, the multi-layer perceptron network is used for extracting the characteristics of each point, and the characteristic vector of each point is subjected to similar inverse transformation to reach the original pose. In order to ensure that the model can adapt to the disorder of the point cloud data, the feature fusion is carried out on the features of each point by using a symmetrical function, and the features of each point are combined into a feature vector of a local patch.
And after the fusion result is obtained, a plurality of full-connection layers are used for processing the fused local feature vectors, and a final result of network output is obtained. It should be noted that, since the point cloud data often contains outliers and noise points, the number of channels of the last regression layer can be changed to adapt to the required output size, and at the same time, the robustness of the model is enhanced.
The above process is formulated as:
Figure SMS_107
，
Figure SMS_108
is each point in a frame of point cloud data.
Figure SMS_109
Is the target function desired to be obtained by training, +.>
Figure SMS_110
Is a symmetric function, < >>
Figure SMS_111
Is a feature extraction function. The basic idea is to use +.>
Figure SMS_112
Respectively processing and then sending the symmetric function>
Figure SMS_113
To achieve permutation invariance.
The points to be removed by the present invention include two types, one is called outliers, which are points far from the main part of the point cloud, and can be directly removed, and the other is called noise points, which are actually formed by shifting the correct points, so that the points are removed, not deleted, but an inverse shift vector is applied to the points, and the points are restored to the correct positions. The whole denoising process comprises the following steps:
Figure SMS_114
wherein the method comprises the steps of
Figure SMS_115
Representing the predicted point cloud result after passing through the denoising module,/->
Figure SMS_116
Representing a set of outliers predicted by the denoising module, +.>
Figure SMS_117
Representing the original noise point cloud, ++>
Figure SMS_118
Representing the originalIs one point in the noise point cloud, +.>
Figure SMS_119
Representing the learned offset vector for each point. The specific process is as follows:
(1) And (5) local division. In order to grasp the local characteristics of the point cloud data, the whole point cloud needs to be divided into a plurality of patches, and one point cloud is given
Figure SMS_120
One local patch is dotted +.>
Figure SMS_121
For the center, all points within a range r from the center are included, denoted +.>
Figure SMS_122
Let therein +.>
Figure SMS_123
A point.
(2) And extracting point-by-point characteristics. The initial input is a set of all point cloud data of a frame, each point cloud is represented as one
Figure SMS_124
A dimension vector, where N represents the number of points and 3 corresponds to XYZ coordinates. As shown in FIG. 2, the data is first shared according to the patch of the last step +.>
Figure SMS_126
Each time a patch is input, a pre-trained quaternary space transfer network (STN 1 ) Outputting parameters of a four-dimensional vector (rotation) of pose transformation:
Figure SMS_129
Through the operation, affine change independence of the model on a specific space is ensured, and the original point cloud is migrated to a new pose state favorable for network learning:
Figure SMS_130
And then through a full connection layer (FNN 1 ) Changing dimensions facilitates feature extraction, then point-by-point features are extracted and processed using a multi-layer perceptron network:
Figure SMS_132
Finally, similar inverse transformation is performed on the point-by-point characteristic vector to reach the original pose, so that a quaternary space transfer network (STN 2 ) And a full link layer (FNN) 2 ）:
Figure SMS_134
Note here +. >
Figure SMS_136
And->
Figure SMS_125
Are all in common->
Figure SMS_127
Personal, because they are->
Figure SMS_128
Middle->
Figure SMS_131
Feature vector of individual points, but +.>
Figure SMS_133
And->
Figure SMS_135
Is different in dimension.
(3) And (5) feature fusion. After the processed point-by-point feature vectors are obtained, the feature vectors of each point in a patch are summed (namely, the feature vectors are realized through a symmetrical function) so that the feature vectors are combined into the feature vector of the patch, the switching law of addition ensures that the combined feature vectors are identical no matter the sequence of input points, namely, the model is ensured to adapt to the disorder of point cloud data, and meanwhile, more information is reserved:
Figure SMS_137
，
Figure SMS_138
and->
Figure SMS_139
Is the same.
(4) And (5) a regression module. After the result of feature fusion is obtained, a plurality of full connection layers are used for fused feature vectors
Figure SMS_141
) And processing to obtain a final result of network output. The number of channels of the last regression layer (i.e., the full connection layer) can be varied to accommodate the size of the desired output. For outlier judgment, the number of channels of the regression layer is set to be 1, the probability that the outlier is represented by the process description function +.>
Figure SMS_142
，
Figure SMS_144
Can set threshold value +.>
Figure SMS_146
I.e. +.>
Figure SMS_147
In the case of->
Figure SMS_148
For outliers, it is removed and the set of all outliers is denoted +.>
Figure SMS_149
The method comprises the steps of carrying out a first treatment on the surface of the For noise point determination, the number of channels of the regression layer is set to 3, representing the inverse offset vector applied to that point, the process description function +. >
Figure SMS_140
，
Figure SMS_143
The corrected point is
Figure SMS_145
。
Step three, registration module
Because the coordinate systems of the point clouds obtained by each scanning are different, the point clouds cannot be simply added, and different point clouds can be spliced only by placing the different point clouds under the same coordinate system through a certain rotation and translation operation, and the process is called point cloud registration.
The initial input of the module is the denoised multi-frame point cloud data processed by the previous module, and for convenience of description, the two-frame point cloud data are assumed to be two-frame point cloud data, and are divided into a source point cloud and a target point cloud, and the task of the module is to align the two-frame point cloud data to the same coordinate system so that the two-frame point cloud data can be effectively fused in subsequent processing. To achieve this, we need to calculate a transformation relationship to align the source point cloud to the coordinate system in which the target point cloud is located.
To extract global features of the point cloud, we use a multi-layer perceptron MLP approach. Firstly, we respectively perform feature extraction on a source point cloud and a target point cloud to obtain two global features. Then, we splice the two features together to get the merged feature of the two point clouds. This stitching operation is to correlate the source point cloud and the target point cloud to better match and align in subsequent processing.
Next, we compute the transformation relationship between the point clouds through five fully connected layers, taking the merged features as input. The final output is a seven-dimensional vector, where the first three bits are translation vectors and the last four bits are unit quaternions. The unit quaternion can be converted into a rotation matrix, so we can represent the translational and rotational transformations of the source point cloud relative to the target point cloud by this vector. The source point cloud is subjected to translation and rotation operations, and can be aligned to the coordinate system of the target point cloud.
And finally, fusing the aligned source point cloud and target point cloud to obtain a complete point cloud image, wherein in subsequent processing, the three-dimensional reconstruction task can be performed by using the point cloud image.
The function of the module is to calculate a transformation relation according to the source point cloud and the target point cloud, align the source point cloud to the coordinate system where the target point cloud is located, and then add the two point clouds to obtain a combined point cloud (for a plurality of point clouds, namely, adding the two point clouds together). In a specific registration task, two point clouds to be registered must contain overlapping portions, otherwise registration cannot be performed. Therefore, when the point clouds to be registered are selected, two point clouds which are relatively close to each other at the scanning site are usually selected for registration, so that the overlapping parts of the two point clouds are more, and the matching relation between the points is easy to find. The specific process is as follows:
(1) And (5) extracting characteristics. The method comprises the steps of respectively carrying out a source point cloud and a target point cloud, and extracting global characteristics of the point cloud by using a multi-layer perceptron method and a maximum pooling layer:
Figure SMS_150
，
Figure SMS_151
。
(2) And (5) splicing. The two global features are spliced to obtain the merging features of the two point clouds, the merging features are used as the input of the following five full-connection layers, the splicing operation is performed to enable the source point clouds and the target point clouds to be associated, and therefore the regression function predicts transformation by comprehensively considering the structural information of the source point clouds and the target point clouds:
Figure SMS_152
+
Figure SMS_153
。
(3) And (5) a full connection layer. And inputting the spliced combined characteristics into a full-connection layer, wherein the final output is a seven-dimensional vector, the first three bits of the vector are translation vectors, the last four bits are unit quaternions, and the unit quaternions can be converted into a rotation matrix. The source point cloud is in the same coordinate system with the target point cloud through translation and rotation.
The registration of the plurality of point clouds can be regarded as two-by-two registration, the registration of the No. 1 and the No. 2 is firstly carried out to form one, then the registration of the two is carried out with the registration of the No. 3, and finally, the images after the registration of the plurality of point clouds are synthesized into a complete point cloud image.
Step four, a surface reconstruction module
The point cloud surface reconstruction is to reconstruct a curved surface or a polygonal grid from unstructured point clouds, so that the curved surface or the polygonal grid is fitted to the original object surface as much as possible, and a three-dimensional model approaching to a real object can be obtained through a curved surface model or a polygonal grid model obtained through surface reconstruction and a certain optimization and rendering. The initial input of the module is one frame of complete point cloud data spliced by the previous module, and the function of the module is to distinguish the surface shape points, the shape outer points and the shape inner points of the point cloud by using a designed neural network, and then to reconstruct the voxel surface of each frame of point cloud on the surface shape points, so as to obtain a triangular grid model of the point cloud data, wherein the shape outer points and the shape inner points do not participate in reconstruction.
As shown in FIG. 3, the present invention introduces occupancy probabilities
Figure SMS_154
Given as input a set of registered 3D point clouds, the goal of this module is to construct an implicit function +.>
Figure SMS_155
Indicate +.>
Figure SMS_156
Occupancy probability->
Figure SMS_157
The function is learned by using the neural network by using the data which contains the point cloud in the whole space and is provided with the label, wherein the label value of 0 indicates that the point is outside the shape of the object, namely the shape outer point, and can be regarded as background or noise, and the label value of 1 indicates that the point is inside the shape of the object, namely the shape inner point, and does not participate in the surface reconstruction of the object, so that the efficiency of the surface reconstruction is greatly improved. The surface of the shape can then be extracted to have an occupancy level of 0Implicit function of 5->
Figure SMS_158
The most probable surface occupying level 0.5, the isosurfaces being composed of points of 0.5.
The neural network learning process is designed as follows:
(1) Potential vector coding: first for each input point using a point convolution method
Figure SMS_159
Generating a potential vector
Figure SMS_160
Encoder->
Figure SMS_161
Can be realized by any point cloud convolution network, and only the channel number of the last layer is required to be changed to control the vector +.>
Figure SMS_162
Of (1), wherein >
Figure SMS_163
For points in the registered point cloud, n represents the number of input points.
(2) Relative potential vector encoding. Given an arbitrary query point
Figure SMS_165
Query Point->
Figure SMS_167
From the input points, construct a set of point set neighbors +.>
Figure SMS_169
It comprises->
Figure SMS_170
The nearest k points, wherein k can be set according to the actual situation, and the point in the adjacent set of the set points is +.>
Figure SMS_172
. Then use query point +.>
Figure SMS_174
Relative to->
Figure SMS_176
Is>
Figure SMS_164
For each point set neighbor +>
Figure SMS_166
Points of->
Figure SMS_168
Is>
Figure SMS_171
Enhancement is performed, and the potential vectors of these enhancements are subjected to +.>
Figure SMS_173
Processing to obtain relative potential vectors
Figure SMS_175
Wherein->
Figure SMS_177
Is a cascading operation. Where the query point is merely illustrative of the starting of the query from this input point, and eventually a query needs to be completed for each input point. The introduction of the relative potential vector enables the characteristics of the query point to be linked with the points in the neighbor point set, so that the neural network is more accurate in grasping the local shape characteristics of the object, and for the generation of complex shapes, the relative potential vector can be used for combining the local shapes of the structure, and the expandability of model reconstruction is greatly improved.
(3) And (5) weighting the characteristics. Relative potential vector
Figure SMS_194
Is +. >
Figure SMS_195
For determining inquiry point->
Figure SMS_196
Is relevant for the importance of the occupancy of (1), whereby it can be used to infer the relative latent variable +.>
Figure SMS_197
Importance weight of (c). Thereby introducing a mechanism of attention, relative to the potential vector +.>
Figure SMS_198
Through weight vector->
Figure SMS_199
Parameterized linear layer, generating relative weights +.>
Figure SMS_200
，
Figure SMS_179
Representing the dot product operation, these weights are at +.>
Figure SMS_181
Normalizing the inner part to obtain the interpolation weight of each part>
Figure SMS_182
The sum of these interpolation weights is 1. In practice the feature weighting is by learning +.>
Figure SMS_185
A separate linear layer consisting of->
Figure SMS_187
Corresponding weight vectors
Figure SMS_188
Parameterization, producing->
Figure SMS_190
Relative weight->
Figure SMS_193
Finally +.>
Figure SMS_178
Become weight +.>
Figure SMS_180
And average +.>
Figure SMS_183
. Query Point->
Figure SMS_184
Feature vector +.>
Figure SMS_186
From adjacent points->
Figure SMS_189
Is>
Figure SMS_191
Weighted summation results in:
Figure SMS_192
。
(4) Decoding into occupancy probabilities. Linear layer
Figure SMS_201
Feature vector +.>
Figure SMS_202
Decoding to occupancy score->
Figure SMS_203
This is a full connection layer output with dimension 2, which is then converted into occupancy probability by softmax>
Figure SMS_204
A label value of 0 indicates that this point is outside the shape of the object, which can be considered as background or noise, a label value of 1 indicates that this point is inside the shape of the object, no participantThe surface of the body is reconstructed.
(5) Voxel surface reconstruction. Obtaining the occupation probability of each point
Figure SMS_205
Then, reconstructing the point cloud as a voxel grid, firstly initializing an empty triangle grid, and then executing the following operations:
1. a cube of the current voxel at (x, y, z) is obtained, containing 8 vertices.
2. A set of boundaries of the current cube is obtained, the boundaries being a set of all points within the interval of 0.5-tau,0.5+ tau of the occupancy probability.
3. For each point in the set, the following is performed:
i. vertices v1 and v2 are selected, where p1 and p2 are the occupancy probabilities of vertices v1 and v2, respectively, p1<0.5, p2>0.5.
interpolation (v 1, v2, p1, p 2) by interpolation function to obtain a new interpolation point vertex with occupation probability equal to 0.5.
interpolation points vertex are added to the triangle mesh M.
4. And constructing a triangle face list faces of the current cube, wherein the list comprises points with the occupation probability of 0.5 in the current cube and the vertex generated just.
5. For each triangle face, it is added to the triangle mesh M, and when all triangle faces have been added to the triangle mesh, the algorithm ends.
The pseudo code is as follows:
input: 3D voxel grid V (size nx, ny, nz), occupancy probability threshold tau
And (3) outputting: triangle mesh M
M=empty grid
for x in range(nx-1):
for y in range(ny-1):
for z in range(nz-1):
cube = acquisition cube (x, y, z, V)
set = acquisition cube set (cube, tau)
for point in set:
if p1<0.5 and p2>0.5
vertex=interpolation (v 1, v2, p1, p 2)
M. adding vertex (vertex)
faces = build triangle face list (cube, set)
for face in faces:
M. triangle (face)
When all triangle facets have been added to the triangle mesh, the algorithm ends
Returning to M, wherein the acquisition cube function "acquisition cube (x, y, z, V)" returns a cube containing eight vertices according to voxel coordinates, the acquisition cube boundary function "acquisition cube set (cube, tau)" returns a set containing all points within the occupation probability [0.5-tau,0.5+tau ] interval, "interpolation (V1, V2, p1, p 2)" returns an interpolation point with an occupation probability equal to 0.5, and "build triangle function build triangle (cube, set)" returns a list containing all triangle faces with an occupation probability equal to 0.5.
The invention also provides a point cloud data three-dimensional reconstruction system based on deep learning, which comprises the following modules:
the point cloud acquisition module is used for scanning the scanned object for multiple times from multiple different angles by utilizing the three-dimensional laser scanner to obtain multiple pieces of point cloud data of the same object to be scanned;
The preprocessing module is used for preprocessing the collected multiple point cloud data, and comprises the steps of removing discrete points and recovering noise points to correct positions;
the registration module calculates a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, and aligns the source point cloud to the coordinate system of the target point cloud, so that the registration of the point clouds is realized, and a complete point cloud image is finally obtained;
and the reconstruction module is used for carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and then carrying out optimization and rendering to obtain a three-dimensional model which approximates to a real object.
The specific implementation manner of each module corresponds to each step, and the invention is not written.
The specific embodiments described herein are offered by way of example only to illustrate the spirit of the invention. Those skilled in the art may make various modifications or additions to the described embodiments or substitutions thereof without departing from the spirit of the invention or exceeding the scope of the invention as defined in the accompanying claims.

Claims (10)

1. The three-dimensional reconstruction method of the point cloud data based on the deep learning is characterized by comprising the following steps of:
Step 1, aiming at an object to be scanned, carrying out multiple scanning from a plurality of different angles by utilizing a three-dimensional laser scanner to obtain a plurality of point cloud data of the same object to be scanned;
step 2, preprocessing the collected multiple point cloud data, including removing discrete points and recovering noise points to correct positions;
step 3, calculating a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, aligning the source point cloud to the coordinate system where the target point cloud is located, thereby realizing the registration of the point clouds and finally obtaining a complete point cloud image;
and 4, carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and carrying out optimization and rendering to obtain a three-dimensional model approaching to a real object.
2. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 1, wherein: in the step 2, firstly, carrying out local division, dividing each point cloud data into a plurality of local patches, wherein each local patch takes a central point as a center and comprises all points within a range of a distance r from the center;
secondly, inputting each local patch into a quaternary space transfer network, learning a four-dimensional vector, transferring an original point cloud to a new pose state through the four-dimensional vector, changing the dimension of the pose state through a full connection layer, extracting the characteristics of point clouds one by using a multi-layer perceptron network, and inversely transforming the characteristic vector of each point cloud to restore the original pose; after the processed point-by-point feature vectors are obtained, the feature vectors of each point in one patch are fused to obtain the feature vector of one local patch;
And finally, carrying out regression processing on the fused feature vectors by using a plurality of full connection layers to obtain a final result.
3. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 2, wherein: the discrete points are removed and the noise points are recovered by changing the number of channels of the last full-connection layer, and the outlier is judged: the number of channels of the last full connection layer is set to 1, and the process description function
Figure QLYQS_2
，
Figure QLYQS_4
，
Figure QLYQS_5
Representing the probability that the point is an outlier, a threshold value is set by itself +.>
Figure QLYQS_7
I.e. +.>
Figure QLYQS_9
In the case of->
Figure QLYQS_12
For outliers, it is removed and the set of all outliers is denoted +.>
Figure QLYQS_13
The method comprises the steps of carrying out a first treatment on the surface of the And (3) judging noise points: the number of channels of the last full-connection layer is set to 3, the procedure description function +.>
Figure QLYQS_1
，
Figure QLYQS_3
，
Figure QLYQS_6
Representing the offset vector of each learned point, the corrected point is +.>
Figure QLYQS_8
，
Figure QLYQS_10
Representing the original noise point cloud, ++>
Figure QLYQS_11
Representing one point in the original noisy point cloud.
4. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 1, wherein: the specific implementation mode of registering any two pieces of point cloud data in the step 3 is as follows;
firstly, respectively extracting features of a source point cloud and a target point cloud to obtain two global features, and then splicing the two global features to obtain a combined feature of the two point clouds, wherein the splicing operation is to correlate the source point cloud and the target point cloud so as to better match and align in subsequent processing;
Then, taking the combined characteristics as input, calculating the transformation relation between the point clouds through five full-connection layers, and finally outputting a seven-dimensional vector, wherein the first three bits are translation vectors, the last four bits are unit quaternions, converting the unit quaternions into a rotation matrix, and carrying out translation and rotation operations on the source point clouds according to the output translation vectors and the rotation matrix, namely aligning the source point clouds to a coordinate system where the target point clouds are positioned;
and finally, fusing the aligned source point cloud and the aligned target point cloud.
5. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 1, wherein: in step 4, distinguishing the surface shape points, the shape outer points and the shape inner points of the point cloud through a neural network, and then carrying out voxel surface reconstruction on each frame of point cloud on the surface shape points to obtain a triangular grid model of point cloud data, wherein the shape outer points and the shape inner points do not participate in reconstruction; the specific implementation of the surface reconstruction is as follows;
an empty triangle mesh is first initialized, and then the following operations are performed:
41, obtaining a cube of the current voxel at (x, y, z), wherein the cube comprises 8 vertexes;
42, acquiring a boundary set of the current cube, wherein the boundary set is a set of points with all occupation probabilities in a [0.5-tau,0.5+tau ] interval, and tau is a set threshold value;
43, for each point in the set, the following is performed:
i. selecting vertexes v1 and v2, wherein p1 and p2 are the occupation probabilities of vertexes v1 and v2, and p1 is less than 0.5 and p2 is more than 0.5;
interpolation (v 1, v2, p1, p 2) by interpolation function to obtain a new interpolation point vertex with occupation probability equal to 0.5;
adding interpolation points vertex into a triangle mesh M;
44, constructing a triangle face list of the current cube, wherein the triangle face list comprises points with the occupation probability of 0.5 in the current cube and the vertex generated immediately;
45, for each triangle face, it is added to the triangle mesh M, when all triangle faces have been added to the triangle mesh, the algorithm ends.
6. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 5, wherein: the neural network learning process is as follows;
(1) Potential vector coding: first for each input point using a point convolution method
Figure QLYQS_14
Generating a potential vector
Figure QLYQS_15
Encoder->
Figure QLYQS_16
Implemented by point cloud convolution network, only the number of channels of the last layer is required to be changed to control vector +.>
Figure QLYQS_17
Of (1), wherein>
Figure QLYQS_18
N represents the number of input points for the points in the registered point cloud;
(2) Relative latent vector coding: given an arbitrary query point
Figure QLYQS_19
Query Point->
Figure QLYQS_21
From the input points, construct a set of point set neighbors +.>
Figure QLYQS_24
It comprises->
Figure QLYQS_26
The nearest k points are set as +.>
Figure QLYQS_28
Then use the query point +.>
Figure QLYQS_29
Relative to->
Figure QLYQS_31
Is>
Figure QLYQS_20
For each point set neighbor +>
Figure QLYQS_22
Points of->
Figure QLYQS_23
Is>
Figure QLYQS_25
Enhancement is performed, and the potential vectors of these enhancements are subjected to +.>
Figure QLYQS_27
Processing to obtain relative potential vector->
Figure QLYQS_30
Wherein->
Figure QLYQS_32
Is a cascading operation; the query is completed for each input point through the operation;
(3) Feature weighting: through learning
Figure QLYQS_34
A separate linear layer consisting of->
Figure QLYQS_36
A corresponding weight vector->
Figure QLYQS_38
Parameterization, producing->
Figure QLYQS_39
Relative weight->
Figure QLYQS_41
，
Figure QLYQS_43
Representing the dot multiplication operation, wherein the weight sum is 1; finally through softmax relative weight->
Figure QLYQS_45
Become weight +.>
Figure QLYQS_33
And average +.>
Figure QLYQS_35
Query Point->
Figure QLYQS_37
Feature vector +.>
Figure QLYQS_40
From adjacent points->
Figure QLYQS_42
Is>
Figure QLYQS_44
Weighted summation results in:
Figure QLYQS_46
；
(4) Decoding into occupancy probability: linear layer
Figure QLYQS_47
Feature vector +.>
Figure QLYQS_48
Decoding to occupancy score->
Figure QLYQS_49
This is a full connection layer output with dimension 2, which is then converted into occupancy probability by softmax>
Figure QLYQS_50
；
(5) Voxel surface reconstruction: obtaining the occupation probability of each point
Figure QLYQS_51
Afterwards, the occupancy probability is->
Figure QLYQS_52
Is 0.5->
Figure QLYQS_53
the point cloud of tau is reconstructed as a voxel grid.
7. The deep learning-based point cloud data three-dimensional reconstruction method as set forth in claim 1, wherein: the three-dimensional laser scanner adopted in the step 1 comprises a laser pulse scanner and a phase ranging scanner.
8. The point cloud data three-dimensional reconstruction system based on the deep learning is characterized by comprising the following modules:
the point cloud acquisition module is used for scanning the scanned object for multiple times from multiple different angles by utilizing the three-dimensional laser scanner to obtain multiple pieces of point cloud data of the same object to be scanned;
the preprocessing module is used for preprocessing the collected multiple point cloud data, and comprises the steps of removing discrete points and recovering noise points to correct positions;
the registration module calculates a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, and aligns the source point cloud to the coordinate system of the target point cloud, so that the registration of the point clouds is realized, and a complete point cloud image is finally obtained;
and the reconstruction module is used for carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and then carrying out optimization and rendering to obtain a three-dimensional model which approximates to a real object.
9. The deep learning-based point cloud data three-dimensional reconstruction system of claim 7, wherein: in the preprocessing module, firstly, local division is carried out, each point cloud data is divided into a plurality of local patches, and each local patch takes a central point as the center and comprises all points within a range of a distance r from the center;
Secondly, inputting each local patch into a quaternary space transfer network, learning a four-dimensional vector, transferring an original point cloud to a new pose state through the four-dimensional vector, changing the dimension of the pose state through a full connection layer, extracting the characteristics of point clouds one by using a multi-layer perceptron network, and inversely transforming the characteristic vector of each point cloud to restore the original pose; after the processed point-by-point feature vectors are obtained, the feature vectors of each point in one patch are fused to obtain the feature vector of one local patch;
and finally, carrying out regression processing on the fused feature vectors by using a plurality of full connection layers to obtain a final result, wherein the specific implementation mode is as follows:
the discrete points are removed and the noise points are recovered by changing the number of channels of the last full-connection layer, and the outlier is judged: the number of channels of the last full connection layer is set to 1, and the process description function
Figure QLYQS_55
，
Figure QLYQS_57
，
Figure QLYQS_58
Representing the probability that the point is an outlier, a threshold value is set by itself +.>
Figure QLYQS_60
I.e. +.>
Figure QLYQS_62
In the case of->
Figure QLYQS_63
For outliers, it is removed and the set of all outliers is denoted +.>
Figure QLYQS_65
The method comprises the steps of carrying out a first treatment on the surface of the And (3) judging noise points: the number of channels of the last full-connection layer is set to 3, the procedure description function +. >
Figure QLYQS_54
，
Figure QLYQS_56
，
Figure QLYQS_59
Representing the offset vector of each learned point, the corrected point is +.>
Figure QLYQS_61
，
Figure QLYQS_64
Representing the original noise point cloud, ++>
Figure QLYQS_66
Representing one point in the original noisy point cloud.
10. The deep learning-based point cloud data three-dimensional reconstruction system of claim 7, wherein: in the reconstruction module, the surface shape points, the shape outer points and the shape inner points of the point cloud are distinguished through a neural network, then voxel surface reconstruction is carried out on each frame of point cloud on the surface shape points, a triangular mesh model of point cloud data is obtained, and the shape outer points and the shape inner points do not participate in reconstruction; the specific implementation of the surface reconstruction is as follows;
an empty triangle mesh is first initialized, and then the following operations are performed:
41, obtaining a cube of the current voxel at (x, y, z), wherein the cube comprises 8 vertexes;
42, acquiring a boundary set of the current cube, wherein the boundary set is a set of points with all occupation probabilities in a [0.5-tau,0.5+tau ] interval, and tau is a set threshold value;
43, for each point in the set, the following is performed:
i. vertices v1 and v2 are selected, where p1<0.5, p2>0.5;
interpolation (v 1, v2, p1, p 2) by interpolation function to obtain a new interpolation point vertex with occupation probability equal to 0.5;
Adding interpolation points vertex into a triangle mesh M;
44, constructing a triangle face list of the current cube, wherein the triangle face list comprises points with the occupation probability of 0.5 in the current cube and the vertex generated immediately;
45, for each triangle face, add it to triangle mesh M, when all triangle faces have been added to triangle mesh, the algorithm ends;
the neural network learning process is as follows;
(1) Potential vector coding: first for each input point using a point convolution method
Figure QLYQS_67
Generating a potential vector
Figure QLYQS_68
Encoder->
Figure QLYQS_69
Implemented by point cloud convolution network, only the number of channels of the last layer is required to be changed to control vector +.>
Figure QLYQS_70
Of (1), wherein>
Figure QLYQS_71
N represents the number of input points for the points in the registered point cloud;
(2) Relative latent vector coding: given an arbitrary query point
Figure QLYQS_73
Query Point->
Figure QLYQS_75
From the input points, construct a set of point set neighbors +.>
Figure QLYQS_77
It comprises->
Figure QLYQS_79
The nearest k points are set as +.>
Figure QLYQS_81
Then use the query point +.>
Figure QLYQS_82
Relative to->
Figure QLYQS_85
Is>
Figure QLYQS_72
For each point set neighbor +>
Figure QLYQS_74
Points of->
Figure QLYQS_76
Is>
Figure QLYQS_78
Enhancement is performed, and the potential vectors of these enhancements are subjected to +.>
Figure QLYQS_80
Processing to obtain relative potential vector- >
Figure QLYQS_83
Wherein->
Figure QLYQS_84
Is a cascading operation; by the above operationOne input point completes one query;
(3) Feature weighting: through learning
Figure QLYQS_86
A separate linear layer consisting of->
Figure QLYQS_89
A corresponding weight vector->
Figure QLYQS_91
Parameterization, producing->
Figure QLYQS_93
Relative weight->
Figure QLYQS_96
，
Figure QLYQS_98
Representing the dot multiplication operation, wherein the weight sum is 1; finally the relative weights are +.>
Figure QLYQS_99
Become weight +.>
Figure QLYQS_87
And average +.>
Figure QLYQS_88
Query Point->
Figure QLYQS_90
Feature vector +.>
Figure QLYQS_92
From adjacent points->
Figure QLYQS_94
Is>
Figure QLYQS_95
Weighted summation results in:
Figure QLYQS_97
；
(4) Decoding into occupancy probability: linear layer
Figure QLYQS_100
Feature vector +.>
Figure QLYQS_101
Decoding to occupancy score->
Figure QLYQS_102
This is a full connection layer output with dimension 2, which is then converted into occupancy probability by softmax>
Figure QLYQS_103
；
(5) Voxel surface reconstruction: obtaining the occupation probability of each point
Figure QLYQS_104
Afterwards, the occupancy probability is->
Figure QLYQS_105
Is 0.5->
Figure QLYQS_106
the point cloud of tau is reconstructed as a voxel grid. />
CN202310483537.7A 2023-05-04 2023-05-04 Point cloud data three-dimensional reconstruction method and system based on deep learning Active CN116258835B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310483537.7A CN116258835B (en) 2023-05-04 2023-05-04 Point cloud data three-dimensional reconstruction method and system based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310483537.7A CN116258835B (en) 2023-05-04 2023-05-04 Point cloud data three-dimensional reconstruction method and system based on deep learning

Publications (2)

Publication Number Publication Date
CN116258835A true CN116258835A (en) 2023-06-13
CN116258835B CN116258835B (en) 2023-07-28

Family

ID=86679625

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310483537.7A Active CN116258835B (en) 2023-05-04 2023-05-04 Point cloud data three-dimensional reconstruction method and system based on deep learning

Country Status (1)

Country Link
CN (1) CN116258835B (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116781837A (en) * 2023-08-25 2023-09-19 中南大学 Automatic change laser three-dimensional scanning system
CN116817771A (en) * 2023-08-28 2023-09-29 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN117078912A (en) * 2023-07-10 2023-11-17 中国地质大学（武汉） Laser point cloud and image alignment method and device based on deep learning
CN117292067A (en) * 2023-11-24 2023-12-26 中影年年(北京)文化传媒有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117473105A (en) * 2023-12-28 2024-01-30 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117690095A (en) * 2024-02-03 2024-03-12 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117710977A (en) * 2024-02-02 2024-03-15 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117953167A (en) * 2024-03-27 2024-04-30 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN118212368A (en) * 2024-05-22 2024-06-18 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011085433A1 (en) * 2010-01-14 2011-07-21 The University Of Sydney Acceptation/rejection of a classification of an object or terrain feature
CN105574929A (en) * 2015-12-15 2016-05-11 电子科技大学 Single vegetation three-dimensional modeling method based on ground LiDAR point cloud data
CN109118582A (en) * 2018-09-19 2019-01-01 东北大学 A kind of commodity three-dimensional reconstruction system and method for reconstructing
WO2020146547A1 (en) * 2019-01-08 2020-07-16 Apple Inc. Auxiliary information signaling and reference management for projection-based point cloud compression
CN111833449A (en) * 2020-06-30 2020-10-27 南京航空航天大学 Three-dimensional reconstruction and intelligent disease identification method for internal environment of subway tunnel
CN111899328A (en) * 2020-07-10 2020-11-06 西北工业大学 Point cloud three-dimensional reconstruction method based on RGB data and generation countermeasure network
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
CN113358061A (en) * 2021-05-31 2021-09-07 东南大学 Single stripe three-dimensional point cloud measuring method for end-to-end calibration of deep learning network
WO2021175050A1 (en) * 2020-03-04 2021-09-10 华为技术有限公司 Three-dimensional reconstruction method and three-dimensional reconstruction device
WO2021179593A1 (en) * 2020-03-10 2021-09-16 山东大学 Deep learning-based three-dimensional pipeline reconstruction method, system, medium, and apparatus

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011085433A1 (en) * 2010-01-14 2011-07-21 The University Of Sydney Acceptation/rejection of a classification of an object or terrain feature
CN105574929A (en) * 2015-12-15 2016-05-11 电子科技大学 Single vegetation three-dimensional modeling method based on ground LiDAR point cloud data
CN109118582A (en) * 2018-09-19 2019-01-01 东北大学 A kind of commodity three-dimensional reconstruction system and method for reconstructing
WO2020146547A1 (en) * 2019-01-08 2020-07-16 Apple Inc. Auxiliary information signaling and reference management for projection-based point cloud compression
WO2021175050A1 (en) * 2020-03-04 2021-09-10 华为技术有限公司 Three-dimensional reconstruction method and three-dimensional reconstruction device
WO2021179593A1 (en) * 2020-03-10 2021-09-16 山东大学 Deep learning-based three-dimensional pipeline reconstruction method, system, medium, and apparatus
CN111833449A (en) * 2020-06-30 2020-10-27 南京航空航天大学 Three-dimensional reconstruction and intelligent disease identification method for internal environment of subway tunnel
CN111899328A (en) * 2020-07-10 2020-11-06 西北工业大学 Point cloud three-dimensional reconstruction method based on RGB data and generation countermeasure network
CN112967219A (en) * 2021-03-17 2021-06-15 复旦大学附属华山医院 Two-stage dental point cloud completion method and system based on deep learning network
CN113358061A (en) * 2021-05-31 2021-09-07 东南大学 Single stripe three-dimensional point cloud measuring method for end-to-end calibration of deep learning network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
陈辉;马世伟;ANDREAS NUECHTER;: "基于激光扫描和SFM的非同步点云三维重构方法", 仪器仪表学报, no. 05, pages 1148 - 1157 *
黄明伟;方莉娜;唐丽玉;王思洁;: "改进泊松算法的图像三维重建点云模型网格化", 测绘科学, no. 04, pages 23 - 28 *

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117078912A (en) * 2023-07-10 2023-11-17 中国地质大学（武汉） Laser point cloud and image alignment method and device based on deep learning
CN116781837B (en) * 2023-08-25 2023-11-14 中南大学 Automatic change laser three-dimensional scanning system
CN116781837A (en) * 2023-08-25 2023-09-19 中南大学 Automatic change laser three-dimensional scanning system
CN116817771A (en) * 2023-08-28 2023-09-29 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN116817771B (en) * 2023-08-28 2023-11-17 南京航空航天大学 Aerospace part coating thickness measurement method based on cylindrical voxel characteristics
CN117292067A (en) * 2023-11-24 2023-12-26 中影年年(北京)文化传媒有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117292067B (en) * 2023-11-24 2024-03-05 中影年年(北京)科技有限公司 Virtual 3D model method and system based on scanning real object acquisition
CN117473105A (en) * 2023-12-28 2024-01-30 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117473105B (en) * 2023-12-28 2024-04-05 浪潮电子信息产业股份有限公司 Three-dimensional content generation method based on multi-mode pre-training model and related components
CN117710977B (en) * 2024-02-02 2024-04-26 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117710977A (en) * 2024-02-02 2024-03-15 西南石油大学 Dam BIM three-dimensional model semantic quick extraction method and system based on point cloud data
CN117690095A (en) * 2024-02-03 2024-03-12 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117690095B (en) * 2024-02-03 2024-05-03 成都坤舆空间科技有限公司 Intelligent community management system based on three-dimensional scene
CN117953167A (en) * 2024-03-27 2024-04-30 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN117953167B (en) * 2024-03-27 2024-05-28 贵州道坦坦科技股份有限公司 Expressway auxiliary facility modeling method and system based on point cloud data
CN118212368A (en) * 2024-05-22 2024-06-18 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data
CN118212368B (en) * 2024-05-22 2024-08-13 堆知(北京)科技集团有限公司 Method and system for reconstructing three-dimensional image based on incomplete contour point cloud data

Also Published As

Publication number Publication date
CN116258835B (en) 2023-07-28

Similar Documents

Publication Publication Date Title
CN116258835B (en) Point cloud data three-dimensional reconstruction method and system based on deep learning
Chen et al. Point-based multi-view stereo network
Tu et al. SWCGAN: Generative adversarial network combining swin transformer and CNN for remote sensing image super-resolution
CN110197505B (en) Remote sensing image binocular stereo matching method based on depth network and semantic information
CN114782634B (en) Monocular image dressing human body reconstruction method and system based on surface hidden function
CN116071278A (en) Unmanned aerial vehicle aerial image synthesis method, system, computer equipment and storage medium
CN114359509A (en) Multi-view natural scene reconstruction method based on deep learning
CN115619951A (en) Dense synchronous positioning and mapping method based on voxel neural implicit surface
CN110992366A (en) Image semantic segmentation method and device and storage medium
CN114429531A (en) Virtual viewpoint image generation method and device
CN114677479A (en) Natural landscape multi-view three-dimensional reconstruction method based on deep learning
CN115272599A (en) Three-dimensional semantic map construction method oriented to city information model
CN117745932A (en) Neural implicit curved surface reconstruction method based on depth fusion constraint
Yan et al. Efficient implicit neural reconstruction using lidar
CN113723294B (en) Data processing method and device and object recognition method and device
CN115049794A (en) Method and system for generating dense global point cloud picture through deep completion
Xu et al. Deep learning guided building reconstruction from satellite imagery-derived point clouds
CN117274446A (en) Scene video processing method, device, equipment and storage medium
CN116993926A (en) Single-view human body three-dimensional reconstruction method
Tang et al. Encoder-decoder structure with the feature pyramid for depth estimation from a single image
CN117115343A (en) Dynamic scene autonomous positioning and on-line high-precision three-dimensional reconstruction method
CN115731344A (en) Image processing model training method and three-dimensional object model construction method
CN117934752A (en) Method, electronic device and computer program product for generating a target object
CN112634135B (en) Remote sensing image super-resolution reconstruction method based on super-resolution style migration network
CN114998405A (en) Digital human body model construction method based on image drive

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant