CN116258835A - Method and system for 3D reconstruction of point cloud data based on deep learning - Google Patents

Abstract

The invention provides a point cloud data three-dimensional reconstruction method and system based on deep learning, comprising the following steps: step 1, aiming at an object to be scanned, carrying out multiple scanning from a plurality of different angles by utilizing a three-dimensional laser scanner to obtain a plurality of point cloud data of the same object to be scanned; step 2, preprocessing the collected multiple point cloud data, including removing discrete points and recovering noise points to correct positions; step 3, calculating a transformation relation according to the source point cloud and the target point cloud for any two pieces of point cloud data, aligning the source point cloud to the coordinate system where the target point cloud is located, thereby realizing the registration of the point clouds and finally obtaining a complete point cloud image; and 4, carrying out surface reconstruction on the unstructured point cloud image to obtain a triangular grid model of the point cloud data, and carrying out optimization and rendering to obtain a three-dimensional model approaching to a real object. The invention can convert the real non-matter cultural heritage into a digital model which is permanently stored in a computer.

Description

Method and system for 3D reconstruction of point cloud data based on deep learning

Technical Field

The present invention relates to the field of information technology, and is mainly aimed at physical intangible cultural heritage. It proposes an automated method and system for constructing point cloud data into three-dimensional modeling through three steps of denoising, registration and surface reconstruction using a deep learning method.

Background Art

Since cultural relics are fragile and most of them are subject to natural damage (oxidation, corrosion, rust, etc.), the physical protection of cultural relics requires a lot of resources. In recent years, with the continuous development of laser scanning technology and computer technology, digital cultural relic protection has been increasingly applied, such as cultural relic 3D modeling, virtual reality, digital cultural relic restoration, etc. Among them, cultural relic 3D modeling converts the physical objects of cultural relics into digital models that can be permanently stored in computers. It has the advantages of low cost, high efficiency, and easy display. It is the current mainstream digital cultural protection method. Cultural relic 3D modeling requires point cloud, which is the raw data obtained by laser scanning. It is a collection of points in space. In addition to the 3D coordinate information of each point, each point also contains information such as color, normal vector, intensity value, time, etc.

In terms of scanning point cloud data modeling, the existing point cloud LOD model building technology has some shortcomings, such as high memory consumption, long processing time and poor stability. It is difficult to process massive point cloud data, and the generated 3D model is not good. In order to solve these problems, a more efficient and stable point cloud processing technology is needed. In this regard, deep learning technology provides a good solution. Deep learning has the efficiency of processing large data sets and the autonomy of extracting features, which is very suitable for point cloud data processing. The successful application of deep learning technology in image, speech, natural language processing and other fields also provides a new idea for point cloud processing. Point cloud processing technology based on deep learning can realize efficient processing and modeling of point clouds, providing strong support for the protection and inheritance of intangible cultural heritage.

However, the particularity of point cloud data makes it impossible for traditional deep learning models to be directly applied to the processing of point cloud data. Point cloud data is a special three-dimensional data representation method, which consists of a large number of discrete points, each of which contains three coordinate values and other possible attribute values, such as color, normal vector, etc. Compared with traditional two-dimensional image data, the particularity of point cloud data is mainly manifested in two aspects. It has data disorder and affine change independence. First, there are many point data in a group of point cloud data. No matter in what order these points appear in the point cloud file, the information they refer to does not change. On the contrary, the points in a picture are arranged in the image according to the inherent order. Secondly, point cloud data exists in three-dimensional space and has affine change independence, that is, rotation invariance, which makes the traditional convolutional neural network used for two-dimensional images unable to be used for point cloud data. Due to these special properties, it is difficult for traditional convolutional neural network models to directly process point cloud data, and there has been no standard process for deep processing of point cloud data.

Summary of the invention

In order to address the shortcomings of the prior art and effectively model 3D point cloud data, the present invention proposes a set of point cloud data three-dimensional modeling methods, including three steps of denoising, alignment and surface reconstruction, and through an improved neural network model, it realizes the use of deep learning technology to process point cloud data to obtain the final three-dimensional mesh model.

In order to achieve the above object, the technical solution provided by the present invention is: a three-dimensional reconstruction method of point cloud data based on deep learning, comprising the following steps:

Step 1: Scan the object to be scanned multiple times from multiple different angles using a three-dimensional laser scanner to obtain multiple point cloud data for the same object to be scanned;

Step 2: preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

Step 3: For any two point cloud data, a transformation relationship is calculated based on the source point cloud and the target point cloud, and the source point cloud is aligned to the coordinate system of the target point cloud, so as to achieve point cloud registration and finally obtain a complete point cloud image;

Step 4: reconstruct the surface of the unstructured point cloud image to obtain a triangular mesh model of the point cloud data, and then optimize and render it to obtain a three-dimensional model that is close to the real object.

Furthermore, in step 2, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and contains all points within a range r from the center;

Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch.

Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result.

，

，

表示该点为离群点的概率，自行设置阈值

，即

时，判定

为离群点，将其去除，所有离群点的集合记为

；对噪声点判定：最后一个全连接层的通道数量设定为3，过程描述函数

，

，

表示学习到的每个点的偏移向量，修正后的点即为

，

表示原始的噪声点云，

表示原始的噪声点云中的一个点。Furthermore, by changing the number of channels in the last fully connected layer, the removal of discrete points and the recovery of noise points are achieved. For the determination of outliers, the number of channels in the last fully connected layer is set to 1, and the process description function

,

,

Indicates the probability that the point is an outlier, and sets the threshold by yourself

,Right now

When

is an outlier, remove it, and the set of all outliers is recorded as

; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function

,

,

Represents the offset vector of each learned point, and the corrected point is

,

represents the original noisy point cloud,

Represents a point in the original noisy point cloud.

Furthermore, the specific implementation method of registering any two point cloud data in step 3 is as follows;

First, feature extraction is performed on the source point cloud and the target point cloud respectively to obtain two global features. Then, the two global features are spliced to obtain the merged features of the two point clouds. The splicing operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing.

Next, the merged features are used as input, and the transformation relationship between point clouds is calculated through five fully connected layers. The final output is a seven-dimensional vector, of which the first three are translation vectors and the last four are unit quaternions. The unit quaternion is converted into a rotation matrix. The source point cloud is translated and rotated according to the output translation vector and rotation matrix, that is, the source point cloud is aligned to the coordinate system of the target point cloud.

Finally, the aligned source point cloud and target point cloud are fused.

Furthermore, in step 4, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows;

First initialize an empty triangle mesh, then do the following:

41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices;

42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold;

43. For each point in the set, do the following:

i. Select vertices v1 and v2, where p1<0.5, p2>0.5;

ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5;

iii. Add the interpolation point vertex to the triangle mesh M;

44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points;

45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

Further, the neural network learning process is as follows;

产生一个潜在向量

，编码器

由点云卷积网络实现，只需要改变最后一层的通道数来控制向量

的维度，其中

为经过配准后的点云中的点，n表示输入点的个数；(1) Latent vector encoding: Use the point convolution method to first encode each input point

Generate a latent vector

, encoder

Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector

The dimensions of

is the point in the registered point cloud, and n represents the number of input points;

，查询点

来自输入点，构造一组点集邻居

，它包括离

最近的k个点，设点集邻居中的点为

，而后用查询点

相对于

的局部坐标

，对每个点集邻居

中的点

的潜在向量

进行增强，这些增强的潜在向量经过

处理得到相对潜在向量

，其中

为级联操作；通过以上操作对每一个输入点都完成一次查询；(2) Relative latent vector encoding: Given an arbitrary query point

, query point

From the input point, construct a set of point set neighbors

, which includes

The nearest k points, let the points in the point set neighbors be

, and then use the query point

Relative to

The local coordinates of

, for each point set neighbor

Points in

The latent vector

These enhanced latent vectors are enhanced by

Processing to obtain the relative potential vector

,in

It is a cascade operation; through the above operation, a query is completed for each input point;

个独立的线性层，由

个相应的权重向量

参数化，产生

个相对权重

，

表示点乘操作，权重总和为1；最后通过softmax将相对权重

变为权重

，并平均为

，查询点

处的特征向量

由相邻点

的相对潜在向量

加权求和得到：

；(3) Feature weighting: through learning

independent linear layers, consisting of

The corresponding weight vector

Parameterization, generating

Relative weight

,

represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to

Becomes weight

, and the average is

, query point

The eigenvector at

By adjacent points

The relative potential vector

The weighted summation yields:

;

将特征向量

解码为占据得分

，这是一个维度为2的全连接层输出，然后通过softmax转换为占用概率

；(4) Decoding into occupancy probability: linear layer

The feature vector

Decoded as occupancy score

, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax

;

后，将占用概率

为0.5

tau的点云当作一个体素格子进行重构。(5) Voxel surface reconstruction: obtain the occupancy probability of each point

After that, the occupation probability

0.5

The point cloud of tau is reconstructed as a voxel grid.

Furthermore, the three-dimensional laser scanner used in step 1 includes a laser pulse scanner and a phase ranging scanner.

The present invention also provides a point cloud data 3D reconstruction system based on deep learning, comprising the following modules:

The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned;

The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image;

The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering.

Furthermore, in the preprocessing module, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and contains all points within a range r from the center;

Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch.

Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. The specific implementation method is as follows:

，

，

表示该点为离群点的概率，自行设置阈值

，即

时，判定

为离群点，将其去除，所有离群点的集合记为

；对噪声点判定：最后一个全连接层的通道数量设定为3，过程描述函数

，

，

表示学习到的每个点的偏移向量，修正后的点即为

，

表示原始的噪声点云，

表示原始的噪声点云中的一个点。By changing the number of channels in the last fully connected layer, the discrete points can be removed and the noise points can be restored. For the determination of outliers, the number of channels in the last fully connected layer is set to 1. The process description function

,

,

Indicates the probability that the point is an outlier, and sets the threshold by yourself

,Right now

When

is an outlier, remove it, and the set of all outliers is recorded as

; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function

,

,

Represents the offset vector of each learned point, and the corrected point is

,

represents the original noisy point cloud,

Represents a point in the original noisy point cloud.

Furthermore, in the reconstruction module, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows;

First initialize an empty triangle mesh, then do the following:

41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices;

42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold;

43. For each point in the set, do the following:

i. Select vertices v1 and v2, where p1<0.5, p2>0.5;

ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5;

iii. Add the interpolation point vertex to the triangle mesh M;

44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points;

45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

The neural network learning process is as follows:

产生一个潜在向量

，编码器

由点云卷积网络实现，只需要改变最后一层的通道数来控制向量

的维度，其中

为经过配准后的点云中的点，n表示输入点的个数；(1) Latent vector encoding: Use the point convolution method to first encode each input point

Generate a latent vector

, encoder

Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector

The dimensions of

is the point in the registered point cloud, and n represents the number of input points;

，查询点

来自输入点，构造一组点集邻居

，它包括离

最近的k个点，设点集邻居中的点为

，而后用查询点

相对于

的局部坐标

，对每个点集邻居

中的点

的潜在向量

进行增强，这些增强的潜在向量经过

处理得到相对潜在向量

，其中

为级联操作；通过以上操作对每一个输入点都完成一次查询；(2) Relative latent vector encoding: Given an arbitrary query point

, query point

From the input point, construct a set of point set neighbors

, which includes

The nearest k points, let the points in the point set neighbors be

, and then use the query point

Relative to

The local coordinates of

, for each point set neighbor

Points in

The latent vector

These enhanced latent vectors are enhanced by

Processing to obtain the relative potential vector

,in

It is a cascade operation; through the above operation, a query is completed for each input point;

个独立的线性层，由

个相应的权重向量

参数化，产生

个相对权重

，

表示点乘操作，权重总和为1；最后通过softmax将相对权重

变为权重

，并平均为

，查询点

处的特征向量

由相邻点

的相对潜在向量

加权求和得到：

；(3) Feature weighting: through learning

independent linear layers, consisting of

The corresponding weight vector

Parameterization, generating

Relative weight

,

represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to

Becomes weight

, and the average is

, query point

The eigenvector at

By adjacent points

The relative potential vector

The weighted summation yields:

;

将特征向量

解码为占据得分

，这是一个维度为2的全连接层输出，然后通过softmax转换为占用概率

；(4) Decoding into occupancy probability: linear layer

The feature vector

Decoded as occupancy score

, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax

;

后，将占用概率

为0.5

tau的点云当作一个体素格子进行重构。(5) Voxel surface reconstruction: obtain the occupancy probability of each point

After that, the occupation probability

0.5

The point cloud of tau is reconstructed as a voxel grid.

Compared with the prior art, the advantages and beneficial effects of the present invention are:

(1) Through the idea of local division, symmetric functions and spatial transfer networks are introduced into the convolutional neural network model, so that the deep learning model can adapt to the disorder and affine change independence of point cloud data, making it suitable for point cloud data denoising and improving the quality of point cloud data to achieve better 3D model construction effect.

(2) Feature extraction of point cloud data is performed through a multi-layer perceptron (MLP) and a point latent vector relative encoding method, which reduces the impact of the sparsity of point cloud data on feature extraction and enables more accurate extraction of the local shape information of the point cloud.

(3) The concept of occupancy probability and a feature weighting method based on the attention mechanism are introduced to determine the surface shape of the point cloud, solving the problems of low efficiency, poor scalability and low generation accuracy of traditional 3D model construction methods.

The present invention has created an automated system that uses deep learning technology to model point cloud data into three-dimensional models, converting physical intangible cultural heritage into digital models that can be permanently stored in computers, so as to achieve the protection and inheritance of intangible cultural heritage.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the present invention.

FIG. 2 is a schematic diagram of the denoising module of the present invention.

FIG. 3 is a schematic diagram of a surface reconstruction module in the present invention.

DETAILED DESCRIPTION

The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

As shown in FIG1 , the method for three-dimensional reconstruction of point cloud data based on deep learning provided by the present invention comprises the following steps:

Step 1: Point cloud data acquisition

Acquiring point cloud actually means obtaining the three-dimensional coordinates of sampling points on the surface of an object. The most commonly used equipment for acquiring point cloud data can be divided into three types according to different distance measurement principles:

(1) Laser pulse scanner, based on pulse ranging method. The laser generator emits a light pulse, which hits the surface of the object to be measured and reflects back, and is received by the receiver on the scanner. The speed of light is known, and the distance between the scanner and the sampling point can be calculated from the propagation time. Combined with the position of the scanner and the scanning angle, the three-dimensional coordinates of each sampling point can be accurately calculated. This pulse ranging method based on time measurement requires a very accurate clock system. It is characterized by a wide ranging range and flexible usage scenarios, but relatively low accuracy, and is mostly used in long-distance scanning measurements.

(2) Phase ranging scanner, based on phase ranging method. Phase ranging method is essentially a pulse ranging method, except that its distance calculation is based on the phase difference of light waves rather than time. Since light is also a wave, there is a phase difference between the emitted light and the received light. Through this phase difference, the propagation time of light can be indirectly calculated, and thus the distance can be calculated. It has higher scanning accuracy and is generally used for medium-distance or short-distance scanning measurements, with an accuracy of up to millimeter level.

In the application scenario of the present invention, the scanning object is an intangible cultural heritage entity or a place for holding intangible cultural heritage activities, and a scanning method that combines a ground-fixed laser scanning system and a phase ranging scanner is adopted.

In actual scanning scenarios, due to the large size of the object to be scanned (such as a building), the presence of obstructions, or the limited scanning angle of the scanner itself, a single scan cannot obtain the point cloud data of the complete object surface. Multiple scanning stations are required to perform multiple scans at multiple different angles, thus obtaining multiple point cloud data for the same object to be scanned.

Step 2: Denoising module

In the original process of point cloud acquisition, due to the influence of the acquisition environment (light, water vapor in the air, equipment jitter, obstructions), scanning equipment (equipment failure, low scanning accuracy) and scanned objects (small bumps on the surface of the object), there are usually some noise points in the scanned point cloud. These noise points not only increase the amount of point cloud data, but also affect the accuracy of the subsequent point cloud processing steps, thereby affecting the final modeling effect. Therefore, the first step in the processing work is to remove these noise points.

The initial input of this module is a frame of point cloud data. Before the point cloud data is input into the network, it needs to be locally divided first, dividing the entire point cloud into multiple local patches. Each local patch is centered on a central point and contains all points within a range of r from the center. In this way, the point cloud data can be divided into multiple smaller blocks, which is convenient for network learning.

Next, the quaternion space transfer network is used to learn a four-dimensional vector, through which the original point cloud is transferred to a new posture state to ensure that the model is independent of affine changes in a specific space. Then, a multi-layer perceptron network is used to extract the features of each point, and a similar inverse transformation is performed on the feature vector of each point to reach the original posture. In order to ensure that the model can adapt to the disorder of the point cloud data, a symmetric function is used to fuse the features of each point and merge the features of each point into a feature vector of a local patch.

After obtaining the fusion result, several fully connected layers are used to process the fused local feature vectors to obtain the final result of the network output. It should be noted that since point cloud data often contains outliers and noise points, the number of channels in the last regression layer can be changed to adapt to the required output size and enhance the robustness of the model.

，

是一帧点云数据中的每一个点。

是希望通过训练得到的目标函数，

是对称函数，

是特征提取函数。基本思路就是对各个元素（即点云中的各个点）使用

分别处理，再送入对称函数

中处理，以实现排列不变性。The above process is expressed as follows:

,

It is each point in a frame of point cloud data.

is the objective function we hope to obtain through training,

is a symmetric function,

is a feature extraction function. The basic idea is to use

Process them separately and then send them to the symmetric function

to achieve permutation invariance.

The points to be removed by the present invention include two types: one is called outlier points, which are points far away from the main part of the point cloud and can be directly removed; the other is called noise points, which are actually formed by the deviation of the correct points. Therefore, removing them does not mean deleting them, but applying an inverse deviation vector to them to restore them to the correct position. The whole denoising process:

表示通过去噪模块后预测的点云结果，

表示去噪模块预测出的离群点的集合，

表示原始的噪声点云，

表示原始的噪声点云中的一个点，

表示学习到的每个点的偏移向量。具体过程如下：in

Represents the predicted point cloud result after passing through the denoising module,

represents the set of outliers predicted by the denoising module,

represents the original noisy point cloud,

represents a point in the original noise point cloud,

Represents the offset vector of each point learned. The specific process is as follows:

，一个局部patch以点

为中心，包含距离中心r范围内的所有点，记为

，假设其中有

个点。(1) Local division. In order to grasp the local features of point cloud data, it is necessary to divide the entire point cloud into multiple patches.

, a local patch is composed of points

As the center, it includes all points within the range of r from the center, recorded as

, assuming that there is

points.

维向量，其中N代表点的数量，3对应XYZ坐标。如图2所示，首先将数据按照上一步分好的patch，共有

个，每次输入一个patch，通过一个预训练好的四元空间转移网络（STN₁），输出一个位姿变换的四维向量（旋转）的参数：

，通过这个操作，保证了模型对特定空间的仿射变化无关性，并且将原始点云迁移到一个新的有利于网络学习的位姿状态：

，而后通过一个全连接层（FNN₁）更改维度便于特征提取，然后使用多层感知机网络提取逐个点的特征并处理：

，最后也要对逐点的特征向量做类似的逆变换，到达原本的位姿，因此还需要引入一个四元空间转移网络（STN₂）和一个全连接层（FNN₂）:

，注意这里的

与

都共有

个，因为他们是

中

个点的特征向量，但是

与

的维度不同。(2) Point-by-point feature extraction. The initial input is a set of all point cloud data of a frame, and each point cloud is represented by a

dimensional vector, where N represents the number of points and 3 corresponds to the XYZ coordinates. As shown in Figure 2, first divide the data into patches according to the previous step.

Each time a patch is input, it is passed through a pre-trained four-element spatial transfer network (STN ₁ ) to output the parameters of a four-dimensional vector (rotation) of pose transformation:

Through this operation, the model is guaranteed to be independent of affine changes in a specific space, and the original point cloud is migrated to a new pose state that is conducive to network learning:

, and then a fully connected layer (FNN ₁ ) is used to change the dimension for feature extraction, and then a multi-layer perceptron network is used to extract and process the features of each point:

Finally, a similar inverse transformation is performed on the point-by-point feature vector to reach the original position, so a quaternion space transfer network (STN ₂ ) and a fully connected layer (FNN ₂ ) are also introduced:

, note that

and

All in common

because they are

middle

The eigenvectors of points, but

and

The dimensions are different.

，

与

的维度相同。(3) Feature fusion. After obtaining the processed point-by-point feature vectors, the network sums the feature vectors of each point in a patch (i.e., implemented through a symmetric function), thus merging them into a feature vector of a patch. Regardless of the order of the input points, the commutative law of addition ensures that the merged feature vectors are the same, which ensures that the model can adapt to the disorder of point cloud data, while retaining more information:

,

and

The dimensions are the same.

）处理，获得网络输出的最终结果。可以通过改变最后一个回归层（即全连接层）的通道数量，以适应所需输出的大小。对离群点判定，回归层的通道数量设定为1，表示该点为离群点的概率，过程描述函数

，

，可自行设置阈值

，即

时，判定

为离群点，将其去除，所有离群点的集合记为

；对噪声点判定，回归层的通道数量设定为3，表示应对该点施加的逆偏移向量，过程描述函数

，

，修正后的点即为

。(4) Regression module. After obtaining the result of feature fusion, the fused feature vector is processed using several fully connected layers (

) to obtain the final result of the network output. The number of channels in the last regression layer (i.e., the fully connected layer) can be changed to adapt to the required output size. For outlier determination, the number of channels in the regression layer is set to 1, indicating the probability that the point is an outlier. The process description function

,

, you can set the threshold yourself

,Right now

When

is an outlier, remove it, and the set of all outliers is recorded as

; For noise point determination, the number of channels in the regression layer is set to 3, indicating the inverse offset vector to be applied to the point, and the process description function

,

, the corrected point is

.

Step 3: Registration module

Since the coordinate systems of the point clouds obtained from each scan are not the same, the point clouds cannot be simply added together. Instead, they must go through certain rotation and translation operations to place different point clouds in the same coordinate system before they can be spliced. This process is called point cloud registration.

The initial input of this module is the denoised multi-frame point cloud data processed by the previous module. For the convenience of introduction, it is assumed that there are two frames of point cloud data. Here they are divided into source point cloud and target point cloud. The task of this module is to align the two frames of point cloud data to the same coordinate system so that they can be effectively fused in subsequent processing. To achieve this goal, we need to calculate a transformation relationship to align the source point cloud to the coordinate system of the target point cloud.

In order to extract the global features of the point cloud, we use the multi-layer perceptron MLP method. First, we extract features from the source point cloud and the target point cloud respectively to obtain two global features. Then, we concatenate these two features to obtain the merged features of the two point clouds. This concatenation operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing.

Next, we use the merged features as input and calculate the transformation relationship between point clouds through five fully connected layers. The final output is a seven-dimensional vector, where the first three are the translation vector and the last four are the unit quaternion. The unit quaternion can be converted to a rotation matrix, so we can use this vector to represent the translation and rotation transformation of the source point cloud relative to the target point cloud. By translating and rotating the source point cloud, it can be aligned to the coordinate system of the target point cloud.

Finally, the aligned source point cloud and target point cloud are fused to obtain a complete point cloud image. In subsequent processing, we can use this point cloud image to perform 3D reconstruction tasks.

The function of this module is to calculate a transformation relationship based on the source point cloud and the target point cloud, align the source point cloud to the coordinate system of the target point cloud, and then add the two point clouds to obtain a merged point cloud (for multiple point clouds, the results of adding two by two are added together). In the specific registration task, the two point clouds to be registered must contain overlapping parts, otherwise the registration cannot be performed. Therefore, when selecting the point clouds to be registered, two point clouds with relatively close scanning sites are usually selected for registration, so that the two point clouds have more overlapping parts and it is easy to find the matching relationship between the points. The specific process is as follows:

，

。(1) Feature extraction. This is done for the source point cloud and the target point cloud respectively, using the multi-layer perceptron method and the maximum pooling layer to extract the global features of the point cloud:

,

.

+

。(2) Splicing. The two global features are spliced to obtain the merged features of the two point clouds, which are used as the input of the following five fully connected layers. The purpose of the splicing operation is to associate the source point cloud with the target point cloud, so that the regression function can comprehensively consider the structural information of the two and predict the transformation:

+

.

(3) Fully connected layer. The combined features obtained by concatenation are input into the fully connected layer. The final output is a seven-dimensional vector. The first three digits of the vector are the translation vector, and the last four digits are the unit quaternion. The unit quaternion can be converted into a rotation matrix. After translation and rotation, the source point cloud is in the same coordinate system as the target point cloud.

The registration of multiple point clouds can also be viewed as pairwise registration. First, register point clouds 1 and 2 into one image, and then register them with point cloud 3. Finally, the images after multiple point cloud registrations are synthesized into a complete point cloud image.

Step 4: Surface reconstruction module

Point cloud surface reconstruction is to reconstruct a surface or polygonal mesh from an unstructured point cloud so that it fits the original object surface as closely as possible. The surface model or polygonal mesh model obtained by surface reconstruction, after certain optimization and rendering, can obtain a three-dimensional model close to the real object. The initial input of this module is a frame of complete point cloud data spliced by the previous module. Its function is to use the designed neural network to distinguish the surface shape points, shape external points and shape internal points of the point cloud, and then perform voxel surface reconstruction on each frame of point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data. The shape external points and shape internal points do not participate in the reconstruction.

的概念，给定一组经过配准的3D点云作为输入，本模块的目标是构建一个隐式函数

，指示在任意给定查询点

处的占用概率

，使用包含在整个空间中的点云并带有标签的数据来使用神经网络学习这个函数，标签值为0表示这个点在物体形状的外部，即形状外点，可以把它当作背景或者噪声，标签值为1表示这个点在物体形状的内部，即形状内点，不参与物体的表面重构，这样大大提升了表面重构的效率。而后，可以将形状的表面提取为占用水平为0.5的隐式函数

的等值面，占用水平0.5的最可能是表面，构成的等值面即都是0.5的点构成的表面。As shown in FIG3 , the present invention introduces the occupation probability

Given a set of registered 3D point clouds as input, the goal of this module is to construct an implicit function

, indicating that at any given query point

Occupancy probability

, using the point cloud contained in the entire space and labeled data to use the neural network to learn this function. The label value of 0 means that the point is outside the shape of the object, that is, the shape outside point, which can be regarded as background or noise. The label value of 1 means that the point is inside the shape of the object, that is, the shape inside point, and does not participate in the surface reconstruction of the object. This greatly improves the efficiency of surface reconstruction. Then, the surface of the shape can be extracted as an implicit function with an occupancy level of 0.5

The isosurface of the occupancy level 0.5 is most likely the surface, and the isosurface is composed of points with a value of 0.5.

The designed neural network learning process is as follows:

产生一个潜在向量

，编码器

可以由任何点云卷积网络实现，只需要改变最后一层的通道数来控制向量

的维度，其中

为经过配准后的点云中的点，n表示输入点的个数。(1) Latent vector encoding: Use the point convolution method to first encode each input point

Generate a latent vector

, encoder

It can be implemented by any point cloud convolutional network, just need to change the number of channels in the last layer to control the vector

The dimensions of

is the point in the point cloud after registration, and n represents the number of input points.

，查询点

来自输入点，构造一组点集邻居

，它包括离

最近的k个点，这里的k可根据实际情况自行设置，设点集邻居中的点为

。而后用查询点

相对于

的局部坐标

，对每个点集邻居

中的点

的潜在向量

进行增强，这些增强的潜在向量经过

处理得到相对潜在向量

，其中

为级联操作。其中查询点只是说明从这个输入点开始查询，最终需要对每一个输入点都完成一次查询。相对潜在向量的引入，使得查询点的特征与其邻居点集内的点产生联系，使得神经网络对于物体的局部形状特征的把握更加精确，对于复杂形状的生成，同样可以利用相对潜在向量将其结构为局部形状的组合，这样做大大提升了模型重构的可扩展性。(2) Relative latent vector encoding. Given an arbitrary query point

, query point

From the input point, construct a set of point set neighbors

, which includes

The nearest k points, where k can be set according to the actual situation, and the points in the point set neighbors are

Then use the query point

Relative to

The local coordinates of

, for each point set neighbor

Points in

The latent vector

These enhanced latent vectors are enhanced by

Processing to obtain the relative potential vector

,in

It is a cascade operation. The query point only indicates that the query starts from this input point, and eventually a query needs to be completed for each input point. The introduction of relative latent vectors connects the features of the query point with the points in its neighbor point set, making the neural network more accurate in grasping the local shape features of the object. For the generation of complex shapes, relative latent vectors can also be used to structure them into a combination of local shapes, which greatly improves the scalability of model reconstruction.

的范数与输入点

对于确定查询点

的占用的重要性相关，由此可以用它来推断相对潜变量

的重要性权重。由此引入了一个注意力机制，相对潜在向量

经过由权重向量

参数化的线性层，产生相对权重

，

表示点乘操作，这些权重通过softmax函数在

内进行归一化，得到各自的插值权重

，这些插值权重总和为1。实际上特征加权是通过学习

个独立的线性层，由

个相应的权重向量

参数化，产生

个相对权重

，最后通过softmax将

变为权重

，并平均为

。查询点

处的特征向量

由相邻点

的相对潜在向量

加权求和得到：

。(3) Feature weighting. Relative latent vector

The norm and input point

To determine the query point

The importance of occupancy is related to the relative latent variable

The importance weight of . This introduces an attention mechanism, relative to the potential vector

Through the weight vector

Parameterized linear layer, generating relative weights

,

Represents the dot product operation, these weights are passed through the softmax function in

Normalize them to get their respective interpolation weights

, the sum of these interpolation weights is 1. In fact, feature weighting is learned by

independent linear layers, consisting of

The corresponding weight vector

Parameterization, generating

Relative weight

, and finally through softmax

Becomes weight

, and the average is

Query point

The eigenvector at

By adjacent points

The relative potential vector

The weighted summation yields:

.

将特征向量

解码为占据得分

，这是一个维度为2的全连接层输出，然后通过softmax转换为占用概率

，标签值为0表示这个点在物体形状的外部，可以把它当作背景或者噪声，标签值为1表示这个点在物体形状的内部，不参与物体的表面重构。(4) Decoded into occupancy probability. Linear layer

The feature vector

Decoded as occupancy score

, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax

, a label value of 0 means that the point is outside the shape of the object and can be regarded as background or noise. A label value of 1 means that the point is inside the shape of the object and does not participate in the surface reconstruction of the object.

后，把点云当作一个体素格子进行重构，首先初始化一个空三角形网格，然后执行以下操作：(5) Voxel surface reconstruction. Obtain the occupancy probability of each point

Finally, reconstruct the point cloud as a voxel grid. First, initialize an empty triangle mesh and then do the following:

1. Get the cube of the current voxel at (x, y, z), which contains 8 vertices.

2. Get the boundary set of the current cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau].

3. For each point in the set, do the following:

i. Select vertices v1 and v2, where p1 and p2 are the occupancy probabilities of vertices v1 and v2 respectively, p1<0.5, p2>0.5.

ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5.

iii. Add the interpolation point vertex to the triangle mesh M.

4. Construct the triangle face list faces of the current cube. This list contains the points with an occupation probability of 0.5 in the current cube and the vertex points that have just been generated.

5. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends.

The pseudo code is as follows:

Input: 3D voxel grid V (size nx, ny, nz), occupancy probability threshold tau

Output: triangular mesh M

M = Empty grid

for x in range(nx-1):

for y in range(ny-1):

for z in range(nz-1):

cube = Get Cube(x, y, z, V)

set = Get cube set (cube, tau)

for point in set:

if p1<0.5 and p2>0.5

vertex = interpolate(v1, v2, p1, p2)

M. Adding vertices

faces = build triangle face list (cube, set)

for face in faces:

M. Add triangle (face)

When all triangle faces have been added to the triangle mesh, the algorithm ends.

Returns M, where the cube get function "Get Cube (x, y, z, V)" returns a cube with eight vertices according to the voxel coordinates, the cube boundary get function "Get Cube Set (cube, tau)" returns a set of all points in the occupation probability interval [0.5-tau,0.5+tau], "Interpolation (v1, v2, p1, p2)" returns an interpolation point with an occupation probability equal to 0.5, and the "Build Triangle Function Build Triangle (cube, set)" returns a list of all triangular faces with an occupation probability equal to 0.5.

The present invention also provides a point cloud data 3D reconstruction system based on deep learning, comprising the following modules:

The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned;

The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position;

The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image;

The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering.

The specific implementation method of each module corresponds to each step and is not described in detail in the present invention.

The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims (10)

1. A method for three-dimensional reconstruction of point cloud data based on deep learning, characterized in that it comprises the following steps: Step 1: Scan the object to be scanned multiple times from multiple different angles using a three-dimensional laser scanner to obtain multiple point cloud data for the same object to be scanned; Step 2: preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position; Step 3: For any two point cloud data, a transformation relationship is calculated based on the source point cloud and the target point cloud, and the source point cloud is aligned to the coordinate system of the target point cloud, so as to realize the registration of the point cloud and finally obtain a complete point cloud image; Step 4: reconstruct the surface of the unstructured point cloud image to obtain a triangular mesh model of the point cloud data, and then optimize and render it to obtain a three-dimensional model that is close to the real object. 2. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 1, characterized in that: in step 2, firstly, local division is performed to divide each point cloud data into a plurality of local patches, each local patch is centered on a central point and includes all points within a range r from the center; Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch. Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. 3. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 2, characterized in that: the number of channels in the last fully connected layer is changed to achieve the removal of discrete points and the recovery of noise points, and the determination of outliers: the number of channels in the last fully connected layer is set to 1, and the process description function

,

,

Indicates the probability that the point is an outlier, and sets the threshold by yourself

,Right now

When

is an outlier, remove it, and the set of all outliers is recorded as

; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function

,

,

Represents the offset vector of each learned point, and the corrected point is

,

represents the original noisy point cloud,

Represents a point in the original noisy point cloud. 4. The method for 3D reconstruction of point cloud data based on deep learning according to claim 1, characterized in that: the specific implementation method of registering any two pieces of point cloud data in step 3 is as follows; First, feature extraction is performed on the source point cloud and the target point cloud respectively to obtain two global features. Then, the two global features are spliced to obtain the merged features of the two point clouds. The splicing operation is to associate the source point cloud and the target point cloud so that they can be better matched and aligned in subsequent processing. Next, the merged features are used as input, and the transformation relationship between point clouds is calculated through five fully connected layers. The final output is a seven-dimensional vector, of which the first three are translation vectors and the last four are unit quaternions. The unit quaternion is converted into a rotation matrix. The source point cloud is translated and rotated according to the output translation vector and rotation matrix, that is, the source point cloud is aligned to the coordinate system of the target point cloud. Finally, the aligned source point cloud and target point cloud are fused. 5. The method for 3D reconstruction of point cloud data based on deep learning as claimed in claim 1, characterized in that: in step 4, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then voxel surface reconstruction is performed on each frame of point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows; First initialize an empty triangle mesh, then do the following: 41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices; 42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold; 43. For each point in the set, do the following: i. Select vertices v1 and v2, where p1 and p2 are the occupancy probabilities of vertices v1 and v2, respectively, p1<0.5, p2>0.5; ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5; iii. Add the interpolation point vertex to the triangle mesh M; 44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points; 45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends. 6. The method for three-dimensional reconstruction of point cloud data based on deep learning as claimed in claim 5, characterized in that: the neural network learning process is as follows; (1) Latent vector encoding: Use the point convolution method to first encode each input point

Generate a latent vector

, encoder

Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector

The dimensions of

is the point in the registered point cloud, and n represents the number of input points; (2) Relative latent vector encoding: Given an arbitrary query point

, query point

From the input point, construct a set of point set neighbors

, which includes

The nearest k points, let the points in the point set neighbors be

, and then use the query point

Relative to

The local coordinates of

, for each point set neighbor

Points in

The latent vector

These enhanced latent vectors are enhanced by

Processing to obtain the relative potential vector

,in

It is a cascade operation; through the above operation, a query is completed for each input point; (3) Feature weighting: through learning

independent linear layers, consisting of

The corresponding weight vector

Parameterization, generating

Relative weight

,

represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to

Becomes weight

, and the average is

, query point

The eigenvector at

By adjacent points

The relative potential vector

The weighted summation yields:

; (4) Decoding into occupancy probability: linear layer

The feature vector

Decoded as occupancy score

, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax

; (5) Voxel surface reconstruction: obtain the occupancy probability of each point

After that, the occupation probability

0.5

The point cloud of tau is reconstructed as a voxel grid. 7. The method for three-dimensional reconstruction of point cloud data based on deep learning as described in claim 1 is characterized in that the three-dimensional laser scanner used in step 1 includes a laser pulse scanner and a phase ranging scanner. 8. A point cloud data 3D reconstruction system based on deep learning, characterized by comprising the following modules: The point cloud acquisition module uses a 3D laser scanner to perform multiple scans from multiple different angles on the object to be scanned, thereby obtaining multiple point cloud data for the same object to be scanned; The preprocessing module is used to preprocess the collected multiple point cloud data, including removing discrete points and restoring noise points to the correct position; The registration module calculates a transformation relationship between the source point cloud and the target point cloud for any two point cloud data, aligns the source point cloud to the coordinate system of the target point cloud, thereby realizing point cloud registration and finally obtaining a complete point cloud image; The reconstruction module is used to reconstruct the surface of unstructured point cloud images to obtain a triangular mesh model of the point cloud data, and then obtain a three-dimensional model that is close to the real object through optimization and rendering. 9. The point cloud data 3D reconstruction system based on deep learning as claimed in claim 7, characterized in that: in the preprocessing module, local division is first performed to divide each point cloud data into multiple local patches, each local patch is centered on a central point and includes all points within a range r from the center; Secondly, each local patch is input into the quaternion space transfer network to learn a four-dimensional vector. Through this four-dimensional vector, the original point cloud is transferred to a new posture state, and then the dimension of the posture state is changed through a fully connected layer. The multi-layer perceptron network is used to extract the features of each point cloud, and the feature vector of each point cloud is inversely transformed to restore it to the original posture. After obtaining the processed point-by-point feature vector, the feature vector of each point in a patch is fused to obtain the feature vector of a local patch. Finally, the fused feature vector is regressed using several fully connected layers to obtain the final result. The specific implementation method is as follows: By changing the number of channels in the last fully connected layer, the discrete points can be removed and the noise points can be restored. For the determination of outliers, the number of channels in the last fully connected layer is set to 1. The process description function

,

,

Indicates the probability that the point is an outlier, and sets the threshold by yourself

,Right now

When

is an outlier, remove it, and the set of all outliers is recorded as

; For noise point determination: the number of channels in the last fully connected layer is set to 3, and the process description function

,

,

Represents the offset vector of each learned point, and the corrected point is

,

represents the original noisy point cloud,

Represents a point in the original noisy point cloud. 10. The point cloud data 3D reconstruction system based on deep learning as claimed in claim 7, characterized in that: in the reconstruction module, the surface shape points, shape external points and shape internal points of the point cloud are distinguished by a neural network, and then the voxel surface is reconstructed for each frame of the point cloud on the surface shape points to obtain a triangular mesh model of the point cloud data, and the shape external points and shape internal points do not participate in the reconstruction; the specific implementation method of the surface reconstruction is as follows; First initialize an empty triangle mesh, then do the following: 41. Get the cube of the current voxel at (x, y, z), which contains 8 vertices; 42. Get the boundary set of the current cube cube, which is the set of all points whose occupancy probability is in the interval [0.5-tau, 0.5+tau], where tau is the set threshold; 43. For each point in the set, do the following: i. Select vertices v1 and v2, where p1<0.5, p2>0.5; ii. Use the interpolation function to interpolate (v1, v2, p1, p2) to obtain a new interpolation point vertex with an occupation probability equal to 0.5; iii. Add the interpolation point vertex to the triangle mesh M; 44. Build a triangular face list of the current cube, which contains points with an occupation probability of 0.5 in the current cube and the newly generated vertex points; 45. For each triangular face, add it to the triangular mesh M. When all triangular faces have been added to the triangular mesh, the algorithm ends. The neural network learning process is as follows: (1) Latent vector encoding: Use the point convolution method to first encode each input point

Generate a latent vector

, encoder

Implemented by the point cloud convolutional network, only the number of channels in the last layer needs to be changed to control the vector

The dimensions of

is the point in the registered point cloud, and n represents the number of input points; (2) Relative latent vector encoding: Given an arbitrary query point

, query point

From the input point, construct a set of point set neighbors

, which includes

The nearest k points, let the points in the point set neighbors be

, and then use the query point

Relative to

The local coordinates of

, for each point set neighbor

Points in

The latent vector

These enhanced latent vectors are enhanced by

Processing to obtain the relative potential vector

,in

It is a cascade operation; through the above operation, a query is completed for each input point; (3) Feature weighting: through learning

independent linear layers, consisting of

The corresponding weight vector

Parameterization, generating

Relative weight

,

represents the dot multiplication operation, the sum of the weights is 1; finally, the relative weights are converted to

Becomes weight

, and the average is

, query point

The eigenvector at

By adjacent points

The relative potential vector

The weighted summation yields:

; (4) Decoding into occupancy probability: linear layer

The feature vector

Decoded as occupancy score

, which is a fully connected layer output with a dimension of 2, and then converted to occupancy probability through softmax

; (5) Voxel surface reconstruction: obtain the occupancy probability of each point

After that, the occupation probability

0.5

The point cloud of tau is reconstructed as a voxel grid.

Images