WO2023197601A1

WO2023197601A1 - Gradient field-based point cloud repair method

Info

Publication number: WO2023197601A1
Application number: PCT/CN2022/132439
Authority: WO
Inventors: 胡玮; 陈浩澜; 杜毕安; 罗世通
Original assignee: 北京大学
Priority date: 2022-04-14
Filing date: 2022-11-17
Publication date: 2023-10-19
Also published as: CN116977187A

Abstract

Disclosed in the present invention is a gradient field-based point cloud repair method. Firstly, a distributed global gradient field is estimated from an input degenerated point cloud; then gradient ascent is carried out by using the estimated gradient field, and points are converged to a potential surface, so as to complete point cloud repair.

Description

A point cloud repair method based on gradient field

Technical field

The invention belongs to the technical field of computer software and relates to point cloud repair, and in particular to a point cloud repair method based on a gradient field.

Background technique

The increasingly mature depth sensing, laser scanning and image processing technologies can make it more convenient for people to obtain three-dimensional point clouds from real-world scenes. 3D point clouds consist of discrete 3D points irregularly sampled from a continuous surface. They have attracted increasing attention as an effective representation method of 3D shapes and are widely used in autonomous driving, robotics and immersive interactive telepresence. middle. However, point clouds are often corrupted by noise or suffer from low density due to inherent limitations of scanning equipment, or matching ambiguities when reconstructed from images. Therefore, point cloud restoration, such as denoising and upsampling, is crucial for related 3D vision applications.

Point cloud repair methods can be divided into two types: optimization-based repair methods and deep learning-based repair methods. Optimization-based methods rely heavily on geometric prior knowledge, and it is sometimes difficult to strike a balance between detail preservation and repair effects. Recently, due to the emergence of neural network architectures specifically designed for point clouds, deep learning-based methods have emerged and achieved good inpainting performance. For point cloud denoising, most deep learning-based denoising models predict the displacement of noise points from the underlying surface, and then move the point displacement back to the corresponding latent surface. Such methods mainly face two problems, namely point cloud shrinkage or outliers, which come from overestimation or underestimation of displacement. For point cloud upsampling tasks, complex regularization terms or fine-tuning operations are usually required to prevent the trivial result of point clouds clustering together.

Contents of the invention

In view of the problems existing in the prior art, the purpose of the present invention is to provide a point cloud repair method based on gradient fields.

The steps of the gradient field-based depth point set resampling method of the present invention include:

A neural network is trained using a training data set; wherein the training data set includes degraded point clouds and corresponding clean point clouds; the neural network includes a contextual feature extraction network and a gradient field estimation network; the method for training the neural network is :

The degraded point cloud is input into the context feature extraction network, and the context feature extraction network obtains each point in the degraded point cloud.

The corresponding feature h _i is input to the gradient field estimation network; the gradient field estimation network is based on the point

Contextual point cloud, point

and its characteristics h _i to get the point

The corresponding gradient

according to

Corresponding true gradients in clean point clouds

Calculate the loss function

Where, S represents the point cloud distribution of the degraded point cloud X; the neural network is trained by minimizing the loss function L. When the loss function converges or reaches the set number of training cycles, the training is completed;

Input the degraded point cloud X to be sampled into the trained neural network to obtain the gradient field g(x) corresponding to each point x in the degraded point cloud X; then according to the gradient of each point in the degraded point cloud X field, the points in the degraded point cloud X are iteratively updated through gradient ascent until reaching the set upper limit or convergence, and the repair of the degraded point cloud X is completed.

Further, the method of iteratively updating the points in the degraded point cloud to be sampled through gradient ascent is:

Among them, T is the set total number of iteration cycles, α _t is the set hyperparameter;

is the coordinate value of the i-th collection point x _i in the degraded point cloud X after updating in the t-th iteration cycle,

It is the gradient field of the corresponding point calculated based on the coordinates updated in the t-1th loop iteration.

Further, regularization terms are added to the iterative update, that is, gradient ascent and optimization based on regularization terms are alternately performed; the iterative process is

Among them, I represents the identity matrix, λ is the hyperparameter, and L is the Laplacian matrix of the k-nearest neighbor graph generated based on the input point cloud.

Further, get points

The corresponding gradient

The method is: the gradient field estimation network first extracts each point in the degraded point cloud

each neighbor point in the context point cloud

relative characteristics of

Then according to the nearest neighbor points

distance point

The distance is a relative feature

Give the corresponding weight and get the points

aggregated features

Then

Input the global multi-layer perceptron and estimate the points

The corresponding gradient

in,

for the point

is the set of points in the neighborhood with center radius r.

Further, the nearest neighbor point

distance point

The farther the distance, the relative characteristics

The smaller the weight.

Further, the cosine annealing method is used to determine the relative characteristics

the weight of.

The present invention also provides a server, including a memory and a processor. The memory stores a computer program. The computer program is configured to be executed by the processor. The computer program includes instructions for executing each step in the above method. .

The present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

As shown in Figure 1, the present invention first estimates the distributed global gradient field from the input degraded point cloud; then uses the estimated gradient field to perform gradient rise, converge the points to the potential surface, and complete point cloud repair.

Three-dimensional point clouds obtained by scanning real-world objects or scenes have been widely used in recent years, including immersive interactive telepresence, autonomous driving, monitoring, etc. However, sampled point clouds often encounter noise or low density. This invention proposes a new point cloud repair paradigm (Deep Point Set Resampler, DeepRS), which makes points approach their corresponding potential surfaces by learning the continuous gradient field of point clouds. In particular, the present invention represents the point cloud through the gradient field of the point cloud, that is, the gradient of the logarithmic probability density function, and makes this gradient field continuous, thereby ensuring the continuity of the solvable optimization model. The present invention uses a neural network to fit this gradient field. Based on this, a gradient-based Markov chain Monte Carlo method (MCMC) can be performed on the input noisy or sparse point cloud. In addition, the present invention further proposes to introduce regularization into the MCMC process during the point cloud repair process. This is essentially an iterative improvement of the intermediate resampled point cloud, and introduces various prior knowledge during the resampling process. The present invention demonstrates through extensive experiments that the proposed point cloud resampling method achieves state-of-the-art performance in representative restoration tasks including point cloud denoising and upsampling.

The advantages of the present invention are as follows:

1) This invention unifies issues such as denoising and upsampling into point cloud resampling, and proposes an integrated point cloud repair solution paradigm.

2) Compared with previous methods, the present invention analyzes the continuity of distribution modeling and proposes a continuous model by using the cosine annealing method, thereby ensuring that gradient-based optimization is solvable.

3) The present invention introduces regularization into the point set resampling process, and can repeatedly enhance the point cloud in the intermediate process with a specific regularization method during the sampling process.

Description of the drawings

Figure 1 is a schematic diagram of the point cloud repair method proposed in the present invention.

Figure 2 is a schematic diagram of the DeepRS network.

Figure 3 shows the network structure of DeepRS.

Detailed ways

The present invention will be described in further detail below with reference to the accompanying drawings. The examples cited are only used to explain the present invention and are not intended to limit the scope of the present invention.

(1) Algorithm framework:

First, the present invention treats the non-degraded point cloud Y as sampling from the three-dimensional distribution p(y). Considering the input and collected degraded point cloud, the present invention records the degraded point cloud as

Where H is the corresponding degradation function, such as downsampling, blurring, etc.; N is the additional noise from a certain noise distribution (such as Gaussian distribution, etc.);

It represents the convolution operation. Assume that the distribution q(X) corresponding to value. Therefore, reconstruct the point cloud

It is equivalent to maximizing ∑ _i log q(xi ₎ , where M is the number of points in the point cloud. This can be done by gradient ascent until convergence to q(x) mode. This gradient ascent process is only related to

That is, it is related to the first derivative of the logarithmic density function. Therefore the gradient field

Always point to a clean surface. And because, q(x) is unknown during the test process. Rather than estimating q(x) from degenerate observations, the present invention chooses to estimate its gradient as this is easier to operate. To sum up, the model of the present invention aims to learn the gradient field g(x) so that ∑ _i log q(xi ₎ is maximized, that is, max _g(x) ∑ _i log q(xi ₎ .

It can be seen that repairing the point cloud is equivalent to solving the equation g(x)=0, so the model must have continuity to ensure that this equation can be solved through gradient iteration. The present invention uses the cosine annealing method to make the gradient field estimation continuous to the center point. Specifically, since the present invention estimates the gradient of a certain point x from the local neighborhood N _r (x) with radius r, when the position of x changes during the resampling process, other points may suddenly enter or leave the neighborhood. Domain N _r (x), this will lead to discontinuity. Therefore, before aggregating the features of nearby points, the present invention assigns a corresponding weight to each point, which decays as the distance from x becomes larger. Formally, the aggregate characteristic of x is

Among them, x _j ∈N _r (x) means that x _j is in the neighborhood of x with radius r, and f _j (x) is the feature of x _j calculated relative to x. Essentially, this formula ensures that the feature weight decreases as the distance from x increases, and finally drops to 0 when the distance is equal to or exceeds r.

The depth point set resampling method proposed by the present invention first learns the gradient field from the training data, and then performs point cloud repair through gradient ascent. This framework allows the present invention to introduce regularization into the gradient ascent process for further fine-tuning based on prior knowledge. Regularization in existing work can only be considered during the training phase, usually by incorporating it into the loss function. The framework of the present invention introduces regularization in the repair process, and is therefore more flexible for designing various prior parameters for different downstream tasks. Adding a regularization term can make the recovered point cloud have specific properties according to the corresponding prior knowledge, and its formula can be written as:

X and Z represent the input degraded point cloud and repaired point cloud respectively, H(·) represents the degradation operator defined on Z, and P(Z) represents the regularization term. Specifically, the invention of the present invention mainly uses the graph Laplacian regular operator (GLR) and the weighted graph Laplacian regular operator (RGLR), two commonly used regular terms in optimization-based algorithms, to perform point cloud repair. , this invention will specifically introduce the use of these two algorithms in the network model part.

Graph provides a structurally adaptable, accurate and compact representation method for point clouds. Therefore, the present invention represents each point in the point cloud as a node in the graph G, and connects the points that are neighbors to each other to construct a graph, for example, a k-nearest neighbor (kNN) graph, each point is related to it The nearest k neighbor nodes are connected. The mathematical formula of GLR is generally written as:

where, L is the graph Laplacian matrix, encoding the connectivity of the graph and the degree of each node. i～j means that points i and j are connected, which means that these two points are highly correlated in the point cloud. If the GLR is small, it means that the graph signal is smooth, because if the weight is large, z _i and z _j should be similar accordingly. In GLR, the graph Laplacian matrix L is fixed. The present invention can also regard the Laplacian matrix as a learnable function of the graph signal Z and extend it to RGLR:

Among them, w _ij (x _i ,z _j ) can be learned adaptively during the optimization process. RGLR helps promote flaky smoothness of point clouds, allowing inpainted point clouds to have this better property.

Next, the present invention will introduce how to add regularization terms to the process of the present invention. This invention mainly focuses on differentiable regularization terms. In the above formula, for the sake of simplicity, we might as well assume that H is the identity matrix. Derivative of Z and let the derivative be 0, we get 2(XZ)+λP′(Z)=0. Therefore, the present invention can easily solve Z. For example, when GLR is selected as the prior, Z=(I+λL ^-1 )X is obtained. The reconstruction process is completed by alternating gradient ascent and regularization-based optimization during the resampling process.

(2)Network model

Given a collected degraded point cloud containing M points

The goal of the present invention is to repair it through the above point set resampling method, such as denoising, upsampling, etc. To achieve this goal, the present invention designs a neural network for gradient field training and point cloud resampling. The network consists of a context feature extraction network and a gradient field estimation network. The overall structure is shown in Figure 2, and the specific network composition is shown in Figure 3.

a) Context feature extraction network

In order to estimate the global gradient field, a context point cloud is required for auxiliary operations. This point cloud can be the same as the degraded point cloud, or a more appropriate point cloud can be selected. Given a context point cloud

The present invention first learns the features corresponding to each point through the context feature extraction network. The network is based on dynamic graph convolutional neural network (DGCNN; Reference Wang Y, Sun Y, Liu Z, et al.Dynamic graph cnn for learning on point clouds[J].Acm Transactions On Graphics(tog),2019,38 (5):1-12) can extract multi-scale and local and non-local features for each point, and further obtain features with richer background information by densely connecting the above convolutional neural network. Specifically, the context point cloud is input into the feature extraction network. First, the feature extraction network constructs a k-nearest neighbor graph for this point cloud. Each point is regarded as a vertex of the graph and is related to its k nearest neighbors. Neighbors are connected. Then, it uses the dynamic graph convolution network to extract the features corresponding to each point and record the points

The extracted features are h _i (including multi-scale information, that is, local features and non-local features), and the extracted features are saved and input into the next-level gradient field estimation network.

b) Gradient field estimation network

After obtaining the above h _i , the present invention attempts to use the gradient field estimation network to estimate the gradient field from a global perspective. The network consists of multiple multi-layer perceptrons (MLP). In this part, the input point cloud is the context point cloud and its features and degraded point cloud. For each point in the degraded point cloud

The gradient field estimation network first extracts each point in the degraded point cloud

K-nearest neighbors in context point cloud

relative characteristics of

Among them, F is implemented using multi-layer perceptron (MLP) and the edge convolution method proposed in dynamic graph convolutional neural network, h _j is the starting point

extracted features,

The meaning of has been mentioned in the chapter of algorithm framework. Then, the extracted relative features are passed through the cosine annealing method introduced in the second paragraph of the algorithm framework section, and each point is given a corresponding weight to ensure the continuity of the model, and we get

Finally, add

Input the global multi-layer perceptron G and estimate the gradient corresponding to each point.

The overall learning process can be written as:

(3)Network training and application

(a)Training process

During the training process, the present invention first obtains a training data set, including degraded (such as sparse or noisy) point clouds and corresponding clean point clouds. First, the present invention calculates the gradient of the degraded point cloud according to the method described in Section (2). Generally, due to the limitation of the amount of data during the training process, and for convenience, the present invention simply takes the context point cloud as the degraded point cloud itself. This method is sufficient to obtain good experimental results. Remember the right points

The final calculated gradient is

We define the true gradient of each point

Among them, Y represents the corresponding clean point cloud,

Represents distance points in a clean point cloud

nearest point. Then the optimization objective (loss function) defined by the present invention at this time

where S represents the point in the point cloud at

Distribution in space (three-dimensional vector space). The present invention trains the model by minimizing the loss function. When the loss function converges or reaches a certain number of training cycles, the training is completed.

(b) Application process

In the application stage, since the input data only has degraded point clouds, the degraded point clouds themselves must also serve as context point clouds. Input it into the network model of the present invention to obtain the gradient field g(x) corresponding to each point x. According to the obtained gradient field, without regularization, the present invention performs a gradient-based Markov chain Monte Carlo method (MCMC), and iteratively updates the points through simple gradient rise until reaching the upper limit or convergence. Through this The resampling method obtains the repaired point cloud. Right now:

Among them, t is the number of iteration cycles, α _t is an artificially set hyperparameter, which can be changed as the number of cycles changes.

It is the coordinate value updated in the t-th iteration cycle of the collection point x _i of the degraded point cloud.

Then it is the gradient field of the corresponding point calculated based on the coordinates updated in the t-1th loop iteration. If you want to add a regularization term, you only need to simply modify the gradient ascent process, that is, alternate gradient ascent and optimization based on regularization terms. Taking the graph Laplacian operator as an example, the iterative process at this time is

Represents the intermediate result obtained in the t-th cycle, and the remaining marks have the same meaning as the marks of the formula without regularization. Tables 1, 2, and 3 show the effects of the present invention. It can be seen that the experimental results of the present invention are better than the previous methods in denoising and upsampling tasks.

Table 1: Denoising results for Gaussian noise on PUNet and PCNet datasets

Table 2: Denoising results for other noise on the PUNet dataset

Table 3: Upsampling results on PU-GAN and MPU datasets

Note: Ours(Gen) and Ours

refers to the method in the present invention.

Although specific embodiments of the present invention have been disclosed for illustrative purposes, the purpose is to assist in understanding the content of the invention and practicing it therein. Those skilled in the art will understand that the invention can be practiced without departing from the spirit and scope of the invention and the appended claims. Various substitutions, changes and modifications are possible. Therefore, the present invention should not be limited to the contents disclosed in the preferred embodiments, and the scope of protection claimed by the present invention shall be subject to the scope defined by the claims.

Claims

A point cloud repair method based on gradient field, the steps include:

A neural network is trained using a training data set; wherein the training data set includes degraded point clouds and corresponding clean point clouds; the neural network includes a contextual feature extraction network and a gradient field estimation network; the method for training the neural network is :

The degraded point cloud is input into the context feature extraction network, and the context feature extraction network obtains each point in the degraded point cloud.
The corresponding feature h i is input to the gradient field estimation network; the gradient field estimation network is based on the point
Contextual point cloud, point
and its characteristics h i to get the point
The corresponding gradient
according to
Corresponding true gradients in clean point clouds
Calculate the loss function
Where, S represents the point cloud distribution of the degraded point cloud X; the neural network is trained by minimizing the loss function L. When the loss function converges or reaches the set number of training cycles, the training is completed;

Input the degraded point cloud X to be repaired into the trained neural network to obtain the gradient field g(x) corresponding to each point x in the degraded point cloud X; then according to the gradient of each point in the degraded point cloud X field, the points in the degraded point cloud X are iteratively updated through gradient ascent until reaching the set upper limit or convergence, and the repair of the degraded point cloud X is completed.
The method according to claim 1, characterized in that the method for iteratively updating the points in the degraded point cloud to be repaired through gradient ascent is:

Among them, T is the set total number of iteration cycles, α t is the set hyperparameter;
is the coordinate value of the i-th collection point x i in the degraded point cloud X after updating in the t-th iteration cycle,
It is the gradient field of the corresponding point calculated based on the coordinates updated in the t-1th loop iteration.
The method according to claim 2, characterized in that a regularization term is added to the iterative update, and gradient ascent and optimization based on the regularization term are alternately performed.
The method according to claim 3, characterized in that the iterative process is

Among them, I represents the identity matrix, λ is the hyperparameter, and L is the Laplacian matrix of the k-nearest neighbor graph generated based on the input point cloud.
Represents the intermediate result obtained in the t-th cycle.
The method according to claim 1, characterized in that, obtaining points
The corresponding gradient
The method is: the gradient field estimation network first extracts each point in the degraded point cloud
each neighbor point in the context point cloud
relative characteristics of
Then according to the nearest neighbor points
distance point
The distance is a relative feature
Give the corresponding weight and get the points
aggregated features
Then
Input the global multi-layer perceptron and estimate the points
The corresponding gradient
in,
for the point
is the set of points in the neighborhood with center radius r.
The method according to claim 5, characterized in that the nearest neighbor point
distance point
The farther the distance, the relative characteristics
The smaller the weight.
The method according to claim 5 or 6, characterized in that the cosine annealing method is used to determine the relative characteristics
the weight of.
A server, characterized in that it includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, the computer program includes a component for executing any one of claims 1 to 7 Instructions for each step in the method.
A computer-readable storage medium on which a computer program is stored, characterized in that when the computer program is executed by a processor, the steps of the method of any one of claims 1 to 7 are implemented.