CN114764746A - Super-resolution method and device for laser radar, electronic device and storage medium - Google Patents

Abstract

The present invention provides a laser radar super-resolution method and device, electronic equipment and storage medium, wherein the method includes: inputting an original point cloud into a discriminant model to obtain the shape of a point cloud block corresponding to the original point cloud Embedding; the original point cloud is obtained by constraining the scanned lidar point cloud data; the three-dimensional coordinates of the target grid points at the target resolution and their corresponding shape embedding values are spliced together and input into the generative model, The signed distance function value of the target grid point is obtained; the shape embedding value corresponding to the target grid point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to obtain the super-resolution reconstruction result. The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data. The invention can realize the fast super-resolution reconstruction process of new laser radar data at any resolution.

Description

Super-resolution method and device for lidar, electronic device and storage medium

technical field

The present invention relates to the field of optics, and in particular, to a super-resolution method and device of a laser radar, an electronic device and a storage medium.

Background technique

Lidar is an optical remote sensing technology that measures parameters such as the distance of the target by emitting pulsed laser light at the target. Super-resolution is the process of increasing the resolution of the original image by means of hardware or software. The signed distance function can be expressed as the signed distance value of a point in a metric space to the boundary of a subset of the space: points are positive inside the boundary of the area, negative outside, and 0 on the boundary.

There are many existing technologies for processing lidar data alone and for super-resolution reconstruction of images, while there are few studies on super-resolution reconstruction technology around lidar data. And in these technologies, most of them use the method of combining multiple frames of images or combining 3D lidar data and 2D image data to achieve super-resolution reconstruction. There is no technology that directly performs super-resolution reconstruction on single-frame 3D lidar data, and there are The defects of high complexity, slow processing speed and high equipment cost. More importantly, the super-resolution reconstruction in the prior art is limited to reconstruction to a specified high resolution, and cannot achieve the goal of reconstruction to any resolution.

SUMMARY OF THE INVENTION

The present invention provides a super-resolution method and device of a laser radar, an electronic device and a storage medium, so as to solve the technical defects existing in the prior art.

The present invention provides a super-resolution method for laser radar, including:

Input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is obtained by constraining the laser radar point cloud data obtained by scanning;

The three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value are spliced together and input into the generative model to obtain the signed distance function value of the target grid point; the target grid point corresponds to The shape embedding value of is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

According to the super-resolution method of lidar according to the present invention, wherein the three-dimensional coordinates of the target grid point and their corresponding shape embedded values are spliced together and input into a generative model to obtain the target grid After the point's signed distance function value, include:

Based on the comparison result between the absolute value of the signed distance function value of the target grid point and the preset threshold, the grid point for optimization and completion is selected to form a point cloud result after super-resolution reconstruction.

According to the super-resolution method of lidar according to the present invention, wherein, based on the comparison result of the absolute value of the signed distance function value of the target grid point and the preset threshold, the grid for optimization and completion is selected. Grid points, including:

Comparing the absolute value of the symbol distance function value of the target grid point with a preset threshold to obtain a comparison result;

If the comparison result is that the absolute value of the signed distance function value of the target grid point is smaller than the preset threshold, the target grid point is used as the grid point for optimization completion.

The super-resolution method for lidar according to the present invention, wherein the method further comprises:

Input the original point cloud into the discriminant model, and obtain the corresponding point cloud with the category probability value after completion under the same resolution;

The discriminant model is obtained after training based on point cloud sample data and corresponding reconstructed point cloud category samples.

为：The super-resolution method for lidar according to the present invention, wherein the loss function of the co-training of the discriminative model and the generative model

for:

表示判别式模型的第一项损失函数，为二分类交叉熵损失函数：i表示判别式模型的每一层，一共m层；j表示第i层得到的网格中第j个点，每一层一共n_i个点；y_i，j表示第i层第j个点存在的真实概率，p_i，j表示第i层第j个点存在的预测概率；in,

Represents the first loss function of the discriminant model, which is a two-category cross-entropy loss function: i represents each layer of the discriminant model, a total of m layers; j represents the jth point in the grid obtained by the i-th layer, each There are a total of n _i points in the layer; y _{i, j} represent the real probability of the existence of the j-th point in the i-th layer, and p _{i, j} represent the predicted probability of the existence of the j-th point in the i-th layer;

表示判别式模型的第二项损失函数，为多分类交叉熵损失函数：n指补全后的点云中的n个点，k为类别的总数，y_i,c指第i个点属于第c类的真实概率，p_i,c为预测出的第i个点属于第c类的预测概率；

Represents the second loss function of the discriminant model, which is a multi-class cross-entropy loss function: n refers to n points in the completed point cloud, k is the total number of categories, y _{i, c} refers to the i-th point belonging to the The true probability of class c, p _i,c is the predicted probability that the predicted i-th point belongs to the c-th class;

表示生成式模型的损失函数，Ω为整个三维空间，Ω₀为物体表面，函数φ为生成式模型表征的符号距离函数，x为三维坐标点，

为符号距离函数的梯度，n(x)为坐标点x的法向量，ψ(φ (x))＝exp(-α|φ(x)|)，α>>1。

Represents the loss function of the generative model, Ω is the entire three-dimensional space, Ω ₀ is the surface of the object, the function φ is the signed distance function represented by the generative model, x is the three-dimensional coordinate point,

is the gradient of the signed distance function, n(x) is the normal vector of the coordinate point x, ψ(φ(x))=exp(-α|φ(x)|), α>>1.

The present invention also provides a super-resolution device for lidar, comprising:

The shape embedding determination module is used to input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is the laser radar point cloud obtained by scanning After the data is constrained, it is obtained;

The signed distance function value determination module is used for splicing together the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value and inputting them into the generative model to obtain the signed distance function of the target grid point. value; the shape embedding value corresponding to the target grid point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

The super-resolution device for lidar according to the present invention, wherein the super-resolution device for lidar further comprises:

a grid point screening module for selecting grid points for optimization and completion based on the comparison result of the absolute value of the signed distance function value of the target grid point and a preset threshold;

A super-resolution reconstruction module, configured to perform super-resolution reconstruction on the target point cloud based on the grid points used for optimization and completion to obtain an optimized target point cloud.

The super-resolution device for lidar according to the present invention, wherein the super-resolution device for lidar further includes a classification module, and the classification module is used for:

Input the original point cloud into the discriminant model, and obtain the corresponding point cloud with the category probability value after completion under the same resolution;

The discriminant model is obtained after training based on point cloud sample data and corresponding reconstructed point cloud category samples.

The present invention also provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and running on the processor, when the processor executes the program, the laser radar as described above can be implemented by the processor. Steps of a super-resolution method.

The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of any one of the above-mentioned super-resolution methods for lidar.

The invention combines the generative model and the discriminative model, and realizes the fast super-resolution reconstruction process of the new laser radar data by learning the existing high-resolution data. Since the present invention models the symbolic distance function of the scene object, it can predict the symbolic distance function used to reconstruct the result to any resolution, and realizes the continuous representation of the object surface, so it can realize the detection of the lidar point at any resolution. Super-resolution reconstruction of cloud data; by learning the information in large batches of scene data, the completion process further utilizes the prior knowledge of big data on the basis of using geometric principles, thus achieving better performance than the existing technology. Better super-resolution reconstruction results; the target point cloud data scanned by lidar can be quickly completed and processed to improve resolution, thereby realizing the effect of real-time feedback of the optimized and perfected point cloud results; the present invention does not require complicated With the support of hardware equipment, super-resolution reconstruction can be realized by using the simple electronic equipment and storage medium and executing the program to process the point cloud data, so it can be directly applied to the existing system using lidar equipment. medium, with strong portability.

Description of drawings

In order to explain the present invention or the technical solutions in the prior art more clearly, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the For some embodiments of the invention, for those of ordinary skill in the art, other drawings can also be obtained according to these drawings without any creative effort.

1 is a schematic flowchart of a super-resolution method of a laser radar provided by the present invention;

2 is a schematic structural diagram of a super-resolution device for a laser radar provided by the present invention;

FIG. 3 is a schematic structural diagram of an electronic device provided by the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention. , not all examples. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

A super-resolution method of a laser radar of the present invention is described below in conjunction with FIG. 1, and the method includes:

S1. Input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is obtained by constraining the laser radar point cloud data obtained by scanning ;

The original point cloud is used to generate the shape embedding of the point cloud block corresponding to the original point cloud. By constraining the scanned lidar point cloud data, the original point cloud containing several point cloud blocks can be obtained. Constraints can specifically be: constrain the point cloud data scanned by the lidar to a specified cuboid range, which is the part that needs attention in practical application scenarios.

In the discriminative model: the first half uses the U-Net structure and connects several convolutional neural network layers, completes the sparse point cloud and classifies each point, and outputs the classification results; the second half connects Several convolutional neural network layers, and finally output the shape embedding of the point cloud block. At this time, the discriminant model has the following input and output forms: input "constraint lidar point cloud"; output "corresponding completed point cloud with class probability value at the same resolution", "point cloud" Shape Embedding of Cloud Blocks".

S2, the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value are spliced together and input into the generative model to obtain the signed distance function value of the target grid point; the target grid The shape embedding value corresponding to the point is obtained by linear interpolation based on the shape embedding of the point cloud block; the generative model is a neural network composed of several fully connected layers; the signed distance function value is used to form a super-resolution The reconstructed point cloud results;

When these points are input into the generative model, not only the three-dimensional coordinates (x, y, z values) of the points are input, but the coordinates are spliced together "The shape embedding of the point cloud block is predicted from the discriminative model, The shape embedding of each target grid point obtained by linear interpolation", that is, the input has the form: (S ₁ ,S ₂ ,...,S _n ,x,y,z), where (S ₁ ,S ₂ ,...,S _n ) is the shape embedding vector corresponding to the (x, y, z) point, with a total of n dimensions.

The final output of the training process is the value of the signed distance function for each point input to the generative model.

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data. During the training process, a combination of self-supervised training and supervised training is used to optimize the discriminative model for predicting shape embeddings and the generative model for fitting signed distance functions through gradient descent and backpropagation.

Specifically, in the training phase: first, the constrained original point cloud is input into a discriminative model, and the network maps the original sparse point cloud scanned by each frame of lidar to its corresponding uniform distribution and low degree of sparsity. Some shape embeddings on the point cloud data. Then, the points in the optimized and reconstructed point cloud corresponding to the frame, the points randomly obtained in the constraint space, and the shape embedding corresponding to each point are input into the generative model, and the generative model is fitted and trained. In the generative model, multiple frames of point cloud data are trained at the same time, and all point clouds during the training process share the same generative model. In order to distinguish the signed distance functions of different point clouds, the three-dimensional coordinates of the point cloud are spliced in the input with the shape embedding value representing the scene and location of the point. Since there are many similar scenes in each frame of point cloud, each scene is evenly divided into several point cloud blocks when training the discriminative model for predicting shape embedding, and each block is assigned a shape embedding value. For each point in the optimized and reconstructed point cloud provided by the training data, the shape embedding value corresponding to the point is obtained by linear interpolation, and then input into the generative model for training. Since the entire calculation process is steerable, it can be trained in the above-mentioned end-to-end manner.

The 3D coordinates entered in the generative model can be improved by coding. That is, the three-dimensional coordinates are mapped to the m-dimension. In this way, the input to the generative model becomes (m+n) dimensional data. Among them, there are m-dimensional encoded coordinates and n-dimensional shape embeddings. That is, encode x, y, and z into:

(sin(πx), cos(πx), sin(2πx), cos(2πx),…)

(sin(πy), cos(πy), sin(2πy), cos(2πy),…)

(sin(πz), cos(πz), sin(2πz), cos(2πz),…)

Each is m/3 dimension.

The invention combines the generative model and the discriminative model, and realizes the fast super-resolution reconstruction process of the new laser radar data by learning the existing high-resolution data. In order to continuously characterize the surface of the object, it can realize super-resolution reconstruction of lidar point cloud data at any resolution; this feature allows users to independently select the reconstructed resolution according to the application scenario, without the need for complex reconstruction. Algorithm modification and hardware device replacement. At the same time, modeling the signed distance function of the object makes it possible to render the object and meets more application requirements.

The invention learns the information in the large batch of scene data, so that the completion process further utilizes the prior knowledge of the big data on the basis of using the geometrical principle, so as to obtain better super-resolution than the prior art. Reconstruction results; the target point cloud data obtained by the laser radar scanning can be quickly completed and processed to improve the resolution, thereby realizing the effect of real-time feedback of the optimized and perfected point cloud results; the present invention does not require the support of complex hardware equipment, only It is necessary to use the simple electronic equipment and storage medium, and execute the program to process the point cloud data to achieve super-resolution reconstruction, so it can be directly applied to existing systems using lidar equipment, with strong portability .

According to the super-resolution method of lidar according to the present invention, wherein the three-dimensional coordinates of the target grid point and their corresponding shape embedded values are spliced together and input into a generative model to obtain the target grid After the point's signed distance function value, include:

Based on the comparison result between the absolute value of the signed distance function value of the target grid point and the preset threshold, a grid point for optimal completion is selected. All the obtained grid points for optimal completion can constitute the point cloud after super-resolution reconstruction.

That is to say, after obtaining the symbol distance function value, a smaller preset threshold value can be defined (the preset threshold value can be set to a number close to zero, such as 0.001, 0.01, etc.), and the absolute value of the symbol distance function value is smaller than the threshold value. A point is regarded as a point whose signed distance function value is zero, that is, a point on the surface of the object, and other points are regarded as a point not on the surface. In this way, the results obtained by optimizing and improving the input point cloud and super-resolution reconstruction are predicted.

According to the super-resolution method of lidar according to the present invention, wherein, based on the comparison result of the absolute value of the signed distance function value of the target grid point and the preset threshold, the grid for optimization and completion is selected. Grid points, including:

Comparing the absolute value of the symbol distance function value of the target grid point with a preset threshold to obtain a comparison result;

If the comparison result is that the absolute value of the signed distance function value of the target grid point is smaller than the preset threshold, the target grid point is used as the grid point for optimization completion.

If the comparison result is that the absolute value of the signed distance function value of the target grid point is not less than the preset threshold, the target grid point is regarded as a grid point not on the surface of the object, and the grid point not on the surface of the object can be regarded as a grid point not on the surface of the object. Grid points are discarded without consideration.

The super-resolution method for lidar according to the present invention, wherein the method further comprises:

Input the original point cloud into the discriminant model, and obtain the corresponding point cloud with the category probability value after completion under the same resolution;

The discriminant model is obtained after training based on point cloud sample data and corresponding reconstructed point cloud category samples.

During super-resolution reconstruction, we input the sparse point cloud into the super-resolution reconstruction network to obtain the final predicted reconstructed point cloud A at any resolution, and the reconstructed point with category information at the resolution corresponding to the input point cloud. cloud B. Next, for each point a in A, find the point b closest to a through nearest neighbor search in B, and assign the category information of point b to point a. In this way, each point in A has class information. That is, a super-resolution point cloud reconstruction result with category information at any resolution is finally obtained.

为：Since the output of the discriminative model is connected to the generative model, the entire calculation process can be guided, so the simultaneous training of the discriminative model and the generative model can be realized through the optimization process of the following loss function. The super-resolution method for lidar according to the present invention, wherein the loss function of the co-training of the discriminative model and the generative model

for:

表示判别式模型的第一项损失函数，为二分类交叉熵损失函数：(对应的训练过程是：输出点云块的形状嵌入的训练过程)， i表示判别式模型的每一层，一共m层；j表示第i层得到的网格中第j个点，每一层一共n_i个点；y_i，j表示第i层第j个点存在的真实概率， p_i，j表示第i层第j个点存在的预测概率；in,

Represents the first loss function of the discriminant model, which is the two-category cross-entropy loss function: (the corresponding training process is: the training process of the shape embedding of the output point cloud block), i represents each layer of the discriminant model, a total of m layer; j represents the j-th point in the grid obtained by the i-th layer, and each layer has a total of n _i points; y _{i, j} represent the real probability of the j-th point in the i-th layer, p _{i, j} represent the i-th point The predicted probability of the existence of the jth point in the layer;

表示判别式模型的第二项损失函数，为多分类交叉熵损失函数(对应的训练过程是：输出点云块对应的相同分辨率下补全后的、带有类别概率值的点云的训练过程)，n指补全后的点云中的n个点， k为类别的总数，y_i，c指第i个点属于第c类的真实概率(对于每个点i，有且只有一类下的真实概率为1，其他类下的真实概率为0，即该点的真实类别只能为某一类)，p_i，c为预测出的第i个点属于第c类的预测概率；

Represents the second loss function of the discriminant model, which is the multi-class cross-entropy loss function (the corresponding training process is: the training of the point cloud with the category probability value after completion at the same resolution corresponding to the output point cloud block process), n refers to n points in the completed point cloud, k is the total number of categories, y _{i, c} refers to the true probability that the i-th point belongs to the c-th category (for each point i, there is one and only one The true probability under the class is 1, the true probability under other classes is 0, that is, the true class of the point can only be a certain class), p _{i, c} is the predicted probability that the i-th point belongs to the c-th class ;

In the discriminative model: the first half uses the U-Net structure and connects several convolutional neural network layers, completes the sparse point cloud and classifies each point, and outputs the classification results; the second half connects several A convolutional neural network layer that finally outputs the shape embedding of the point cloud patch. At this time, our discriminant model has the following input and output forms: input "constraint lidar point cloud"; output "corresponding completed point cloud with class probability value at the same resolution", "point cloud" Shape Embedding of Cloud Blocks". The above process of classifying and generating the shape embedding of point cloud blocks is carried out synchronously, and is also co-trained in the training phase.

表示生成式模型的损失函数，Ω为整个三维空间，Ω₀为物体表面，函数φ为生成式模型表征的符号距离函数，x为三维坐标点，

为符号距离函数的梯度，n(x)为坐标点x的法向量，ψ(φ (x))＝exp(-α|φ(x)|)，α>>1。在最小化该loss时：公式中第一个积分项使得符号距离函数在整个空间中梯度的模长趋于1(物理上反映了符号距离函数在空间场中的特性)；第二个积分项里第一个加项使得在表面上的点的符号距离函数值趋于0(物理上反映了表面上的点到表面的距离为0)，第二个加项使得表面上的点的梯度和该点的法向量趋于同向(物理上，梯度表征了函数值变化最快的方向，对于符号距离函数，表面法向量方向恰好为函数值变化最快的方向)；第三个积分项利用exp指数函数，使得不在表面上的点的符号距离函数值趋于无穷大。

Represents the loss function of the generative model, Ω is the entire three-dimensional space, Ω ₀ is the surface of the object, the function φ is the signed distance function represented by the generative model, x is the three-dimensional coordinate point,

is the gradient of the signed distance function, n(x) is the normal vector of the coordinate point x, ψ(φ(x))=exp(-α|φ(x)|), α>>1. When minimizing the loss: the first integral term in the formula makes the modulo length of the gradient of the signed distance function in the entire space tend to 1 (physically reflecting the characteristics of the signed distance function in the space field); the second integral term The first addition term makes the value of the signed distance function of the point on the surface tend to 0 (physically reflecting that the distance from the point on the surface to the surface is 0), and the second addition term makes the gradient of the point on the surface and The normal vectors of the point tend to be in the same direction (physically, the gradient represents the direction in which the function value changes the fastest, and for the signed distance function, the surface normal vector direction happens to be the direction in which the function value changes the fastest); the third integral term uses exp is the exponential function such that the signed distance function values for points not on the surface tend to infinity.

In the generative model, when training with this loss function: for the points in the optimized reconstructed point cloud (on the surface of the object) corresponding to each frame, optimize their predicted signed distance function values to zero; Randomly obtained points in space (not on the surface of the object) can optimize their predicted signed distance function values to their true signed distance function values, but since the object surface is concerned in the super-resolution reconstruction process, That is, the point with the signed distance function value is zero, so the predicted signed distance function value of these points not on the surface is chosen to be optimized to infinity, thereby enhancing the discrimination of the surface.

For lidar data, the point cloud has large sparsity. Although it is possible to directly perform super-resolution reconstruction on the original point cloud data using only a single degenerate generative model, the obtained results will have problems such as uneven distribution, unsatisfactory reconstruction results, and large data noise. Therefore, in order to achieve super-resolution reconstruction with better results, it is necessary to make up for this sparsity, that is, to complete the sparse area. The combination of the generative model and the discriminative model in the present invention enables the completion process to be performed along with the super-resolution reconstruction process.

Referring to FIG. 2 , the super-resolution device of lidar provided by the present invention is described below. The super-resolution device of lidar described below and the super-resolution method of lidar described above can be referred to each other correspondingly. The lidar The super-resolution devices include:

The shape embedding determination module 10 is used to input the original point cloud into the discriminant model to obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is the laser radar point obtained by scanning Obtained after the cloud data is constrained;

The original point cloud is used to generate the shape embedding of the point cloud block corresponding to the original point cloud, and the original point cloud containing several point cloud blocks can be obtained by constraining the scanned lidar point cloud data. Constraints can be specifically: first, constrain the point cloud data scanned by the lidar to a specified cuboid range, which is the part that needs attention in the actual application scenario.

The signed distance function value determination module 20 is used for splicing together the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value and inputting them into the generative model to obtain the signed distance of the target grid point function value; the shape embedding value corresponding to the target grid point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

When these points are input into the generative model, not only the three-dimensional coordinates (x, y, z values) of the points are input, but the coordinates are spliced together "The shape embedding of the point cloud block is predicted from the discriminative model, The shape embedding value (that is, the shape embedding vector) of each target grid point obtained by linear interpolation", that is, the input has the form: (S ₁ , S ₂ , ..., _Sn , x, y, z) , where (S ₁ , S ₂ ,..., S _n ) is the shape embedding vector corresponding to the point (x, y, z), with a total of n dimensions.

The final output of the training process is the value of the signed distance function for each point input to the generative model.

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data. During the training process, a combination of self-supervised training and supervised training is used to optimize the discriminative model for predicting shape embeddings and the generative model for fitting signed distance functions through gradient descent and backpropagation.

Specifically, in the training phase: first, the constrained original point cloud is input into a discriminative model, and the network maps the original sparse point cloud scanned by each frame of lidar to its corresponding uniform distribution and low degree of sparsity. Some shape embeddings on the point cloud data. Then, the points in the optimized and reconstructed point cloud corresponding to the frame, the points randomly obtained in the constraint space, and the shape embedding corresponding to each point are input into the generative model, and the generative model is fitted and trained. In the generative model, multiple frames of point cloud data are trained at the same time, and all point clouds during the training process share the same generative model. In order to distinguish the signed distance functions of different point clouds, the three-dimensional coordinates of the point cloud are spliced in the input with the shape embedding value representing the scene and location of the point. Since there are many similar scenes in each frame of point cloud, each scene is evenly divided into several point cloud blocks when training the discriminative model for predicting shape embedding, and each block is assigned a shape embedding value. For each point in the optimized and reconstructed point cloud provided by the training data, the shape embedding value corresponding to the point is obtained by linear interpolation, and then input into the generative model for training. Since the entire calculation process is steerable, it can be trained in the above-mentioned end-to-end manner.

The super-resolution device for lidar according to the present invention, wherein the super-resolution device for lidar further comprises:

The super-resolution reconstruction module is used to select the grid points for optimization and completion based on the comparison result of the absolute value of the symbol distance function value of the target grid point and the preset threshold, so as to form the point after the super-resolution reconstruction Cloud results.

That is to say, after obtaining the symbol distance function value, a smaller preset threshold value can be defined (the preset threshold value can be set to a number close to zero, such as 0.001, 0.01, etc.), and the absolute value of the symbol distance function value is smaller than the threshold value. A point is regarded as a point whose signed distance function value is zero, that is, a point on the surface of the object, and other points are regarded as a point not on the surface. In this way, the results obtained by optimizing and improving the input point cloud and super-resolution reconstruction are predicted.

According to the super-resolution device for lidar according to the present invention, the super-resolution reconstruction module is specifically used for:

Comparing the absolute value of the symbol distance function value of the target grid point with a preset threshold to obtain a comparison result;

If the comparison result is that the absolute value of the signed distance function value of the target grid point is smaller than the preset threshold, the target grid point is used as the grid point for optimization completion.

If the comparison result is that the absolute value of the signed distance function value of the target grid point is not less than the preset threshold, the target grid point is regarded as a grid point not on the surface of the object, and the grid point not on the surface of the object can be regarded as a grid point not on the surface of the object. Grid points are discarded without consideration.

Since the output of the discriminative model is connected to the generative model, the entire calculation process can be guided, so the simultaneous training of the discriminative model and the generative model can be realized by optimizing the following loss function. According to the super-resolution method of lidar according to the present invention, wherein,

为：Loss function for co-training of the discriminative model and generative model

for:

表示判别式模型的第一项损失函数(对应的训练过程是：输出点云块的形状嵌入的训练过程)，i表示判别式模型的每一层，一共m层；j表示第i层得到的网格中第j个点，每一层一共n_i个点； y_i，j表示第i层第j个点存在的真实概率，p_i，j表示第i层第j个点存在的预测概率；in,

Represents the first loss function of the discriminant model (the corresponding training process is: the training process of the shape embedding of the output point cloud block), i represents each layer of the discriminant model, a total of m layers; j represents the i-th layer obtained The jth point in the grid, there are a total of n _i points in each layer; y _{i, j} represent the real probability of the existence of the jth point in the i-th layer, p _{i, j} represent the predicted probability of the existence of the j-th point in the i-th layer ;

表示判别式模型的第二项损失函数(对应的训练过程是：输出点云块对应的相同分辨率下补全后的、带有类别概率值的点云的训练过程)，n指补全后的点云中的n个点，k为类别的总数，y_i,c指第i 个点属于第c类的真实概率(对于每个点i，有且只有一类下的真实概率为1，其他类下的真实概率为0，即该点的真实类别只能为某一类)，p_i,c为预测出的第i个点属于第c类的预测概率；

Indicates the second loss function of the discriminant model (the corresponding training process is: the training process of the completed point cloud with the category probability value corresponding to the output point cloud block at the same resolution), n refers to the completed point cloud n points in the point cloud of , k is the total number of categories, y _i,c refers to the true probability that the i-th point belongs to the c-th category (for each point i, the true probability under one and only one category is 1, The true probability of other classes is 0, that is, the true class of the point can only be a certain class), p _i,c is the predicted probability that the i-th point belongs to the c-th class;

In the discriminative model: the first half uses the U-Net structure and connects several convolutional neural network layers, completes the sparse point cloud and classifies each point, and outputs the classification results; the second half connects several A convolutional neural network layer that finally outputs the shape embedding of the point cloud patch. At this time, our discriminant model has the following input and output forms: input "constraint lidar point cloud"; output "corresponding completed point cloud with class probability value at the same resolution", "point cloud" Shape Embedding of Cloud Blocks". The above process of classifying and generating the shape embedding of point cloud blocks is carried out synchronously, and is also co-trained in the training phase.

表示生成式模型的损失函数，Ω为整个三维空间，Ω₀为物体表面，函数φ为生成式模型表征的符号距离函数，x为三维坐标点，

为符号距离函数的梯度，n(x)为坐标点x的法向量，ψ(φ (x))＝exp(-α|φ(x)|)，α>>1。在最小化该loss时：公式中第一个积分项使得符号距离函数在整个空间中梯度模长趋于1(物理上反映了符号距离函数在空间场中的特性)；第二个积分项里第一个加项使得在表面上的点的符号距离函数值趋于0(物理上反映了表面上的点到表面的距离为0)，第二个加项使得表面上的点的梯度和该点的法向量趋于同向(物理上，梯度表征了函数值变化最快的方向，对于符号距离函数，表面法向量方向恰好为函数值变化最快的方向)；第三个积分项利用exp指数函数，使得不在表面上的点的符号距离函数值趋于无穷大。

Represents the loss function of the generative model, Ω is the entire three-dimensional space, Ω ₀ is the surface of the object, the function φ is the signed distance function represented by the generative model, x is the three-dimensional coordinate point,

is the gradient of the signed distance function, n(x) is the normal vector of the coordinate point x, ψ(φ(x))=exp(-α|φ(x)|), α>>1. When minimizing the loss: the first integral term in the formula makes the gradient modulus length of the signed distance function tend to 1 in the entire space (physically reflecting the characteristics of the signed distance function in the space field); the second integral term The first addition term makes the signed distance function value of the point on the surface tend to 0 (physically reflects the distance from the point on the surface to the surface is 0), and the second addition term makes the gradient of the point on the surface and the The normal vectors of the points tend to be in the same direction (physically, the gradient represents the direction in which the value of the function changes the fastest. For the signed distance function, the direction of the surface normal vector is exactly the direction in which the value of the function changes the fastest); the third integral term uses exp Exponential function such that the value of the signed distance function for points not on the surface tends to infinity.

In the generative model, when training with this loss function: for the points in the optimized reconstructed point cloud (on the surface of the object) corresponding to each frame, optimize their predicted signed distance function values to zero; Randomly obtained points in space (not on the surface of the object) can optimize their predicted signed distance function values to their true signed distance function values, but since the object surface is concerned in the super-resolution reconstruction process, That is, the point with the signed distance function value is zero, so the predicted signed distance function value of these points not on the surface is chosen to be optimized to infinity, thereby enhancing the discrimination of the surface.

For lidar data, the point cloud has large sparsity. Although it is possible to directly perform super-resolution reconstruction on the original point cloud data using only a single degenerate generative model, the obtained results will have problems such as uneven distribution, unsatisfactory reconstruction results, and large data noise. Therefore, in order to achieve super-resolution reconstruction with better results, it is necessary to make up for this sparsity, that is, to complete the sparse area. The combination of the generative model and the discriminative model in the present invention enables the completion process to be performed along with the super-resolution reconstruction process.

The super-resolution device for lidar according to the present invention, wherein the super-resolution device for lidar further includes a classification module, and the classification module is used for:

Input the original point cloud into the discriminant model, and obtain the corresponding point cloud with the category probability value after completion under the same resolution;

The discriminant model is obtained after training based on point cloud sample data and corresponding reconstructed point cloud category samples.

3 illustrates a schematic diagram of the physical structure of an electronic device, the electronic device may include: a processor (processor) 310, a communication interface (Communications Interface) 320, a memory (memory) 330 and a communication bus 340, wherein the processor 310, The communication interface 320 and the memory 330 communicate with each other through the communication bus 340 . The processor 310 can invoke the logic instructions in the memory 330 to execute the super-resolution method of the lidar, the method comprising:

S1. Input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is obtained by constraining the laser radar point cloud data obtained by scanning ;

S2, the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value are spliced together and input into the generative model to obtain the signed distance function value of the target grid point; the target grid The shape embedding value corresponding to the point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

In addition, the above-mentioned logic instructions in the memory 330 may be implemented in the form of software functional units and may be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

In another aspect, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, when the program instructions are executed by a computer When executed, the computer can execute the super-resolution method of the lidar provided by the above methods, and the method includes:

S1. Input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is obtained by constraining the laser radar point cloud data obtained by scanning ;

S2, the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value are spliced together and input into the generative model to obtain the signed distance function value of the target grid point; the target grid The shape embedding value corresponding to the point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored, and the computer program is implemented when executed by a processor to execute the above-mentioned super-resolution methods for lidars, the Methods include:

S1. Input the original point cloud into the discriminant model, and obtain the shape embedding of the point cloud block corresponding to the original point cloud; wherein, the original point cloud is obtained by constraining the laser radar point cloud data obtained by scanning ;

S2, the three-dimensional coordinates of the target grid point of the target resolution and the corresponding shape embedded value are spliced together and input into the generative model to obtain the signed distance function value of the target grid point; the target grid The shape embedding value corresponding to the point is obtained by linear interpolation based on the shape embedding of the point cloud block; the signed distance function value is used to form the point cloud result after super-resolution reconstruction;

The discriminative model and the generative model are obtained after co-training based on the point cloud sample data and the corresponding reconstructed point cloud sample data.

The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution in this embodiment. Those of ordinary skill in the art can understand and implement it without creative effort.

From the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and certainly can also be implemented by hardware. Based on this understanding, the above-mentioned technical solutions can be embodied in the form of software products in essence or the parts that make contributions to the prior art, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic A disc, an optical disc, etc., includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be The technical solutions described in the foregoing embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A super-resolution method of a laser radar is characterized by comprising the following steps:

inputting the original point cloud into a discriminant model to obtain the shape embedding of a point cloud block corresponding to the original point cloud; the original point cloud is obtained by constraining point cloud data of the laser radar obtained by scanning;

splicing the three-dimensional coordinates of target grid points of a target resolution and corresponding shape embedding values thereof together and inputting the spliced three-dimensional coordinates into a generative model to obtain a symbol distance function value of the target grid points; the shape embedding value corresponding to the target grid point is obtained through linear interpolation based on the shape embedding of the point cloud block; the symbol distance function value is used for forming a point cloud result after super-resolution reconstruction;

and the discriminant model and the generative model are obtained by performing collaborative training on the point cloud sample data and the corresponding reconstructed point cloud sample data.

2. The super-resolution method for lidar according to claim 1, wherein the stitching together the three-dimensional coordinates of the target grid points and the corresponding shape embedding values and inputting the stitched coordinates into the generative model to obtain the symbol distance function values of the target grid points comprises:

and selecting the grid points for optimizing completion based on the comparison result of the absolute value of the symbol distance function value of the target grid point and a preset threshold value so as to form a point cloud result after super-resolution reconstruction.

3. The super-resolution method for lidar according to claim 2, wherein the selecting a grid point for optimal completion based on a comparison result between an absolute value of the symbol distance function of the target grid point and a preset threshold comprises:

comparing the absolute value of the symbol distance function value of the target grid point with a preset threshold value to obtain a comparison result;

and if the comparison result is that the absolute value of the symbol distance function value of the target grid point is smaller than a preset threshold value, taking the target grid point as a grid point for optimizing completion.

4. The super resolution method for lidar according to claim 1, wherein the method further comprises:

inputting the original point cloud into the discriminant model to obtain a point cloud which is completed under the same corresponding resolution and has a category probability value;

and the discriminant model is obtained by training point cloud sample data and the corresponding reconstructed point cloud class sample.

5. The super-resolution method for lidar according to claim 4, wherein the discriminant model and the generative model are loss functions trained cooperatively

Comprises the following steps:

wherein,

the first term loss function representing the discriminant model is a two-class cross entropy loss function: i represents each layer of the discriminant model, for a total of m layers; j represents j points in the grid obtained from the ith layer, and n is the total of each layer_iPoint; y is_i，jRepresenting the true probability, p, of the existence of the jth point of the ith layer_i，jRepresenting the prediction probability of the j point of the ith layer;

the second term loss function representing the discriminant model is a multi-class cross entropy loss function: n refers to n points in the completed point cloud, k is the total number of categories, y_i,cMeans the true probability, p, that the ith point belongs to class c_i,cPredicting probability that the predicted ith point belongs to the class c;

representing the loss function of the generative model, omega being the entire three-dimensional space, omega₀Is the surface of an object, boxPhi is a symbol distance function represented by the generative model, x is a three-dimensional coordinate point,

n (x) is the normal vector to coordinate point x, ψ (φ (x)) -exp (- α | φ (x) |), α>>1。

6. A super-resolution device for a laser radar, comprising:

the shape embedding determining module is used for inputting the original point cloud into the discriminant model to obtain the shape embedding of the point cloud block corresponding to the original point cloud; the original point cloud is obtained by constraining point cloud data of the laser radar obtained by scanning;

the system comprises a symbol distance function value determining module, a shape embedding module and a generating model, wherein the symbol distance function value determining module is used for splicing the three-dimensional coordinates of target grid points of target resolution and corresponding shape embedding values and inputting the three-dimensional coordinates into the generating model to obtain the symbol distance function value of the target grid points; the shape embedding value corresponding to the target grid point is obtained through linear interpolation based on the shape embedding of the point cloud block; the symbol distance function value is used for forming a point cloud result after super-resolution reconstruction;

and the discriminant model and the generative model are obtained by performing collaborative training on the point cloud sample data and the corresponding reconstructed point cloud sample data.

7. The super-resolution device for lidar according to claim 6, wherein the super-resolution device for lidar further comprises:

and the super-resolution reconstruction module is used for selecting the grid points for optimizing completion based on the comparison result of the absolute value of the symbol distance function value of the target grid point and a preset threshold value so as to form a point cloud result after super-resolution reconstruction.

8. The lidar super-resolution device according to claim 6, further comprising a classification module configured to:

inputting the original point cloud into the discriminant model to obtain a point cloud which is completed under the same corresponding resolution and has a category probability value;

and the discriminant model is obtained by training based on the point cloud sample data and the corresponding reconstructed point cloud class sample.

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor, when executing the program, carries out the steps of the super resolution method of lidar according to any of claims 1 to 5.

10. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the super-resolution method for lidar according to any one of claims 1 to 5.

Images