WO2022000522A1

WO2022000522A1 - Saliency characteristics-based simulation incomplete point cloud mask generation method

Info

Publication number: WO2022000522A1
Application number: PCT/CN2020/100817
Authority: WO
Inventors: 李革; 张若楠; 晏玮
Original assignee: 北京大学深圳研究生院
Priority date: 2020-07-01
Filing date: 2020-07-08
Publication date: 2022-01-06
Also published as: CN111783798B; CN111783798A

Abstract

A saliency characteristics-based simulation incomplete point cloud mask generation method. The method comprises: step 1, obtaining the number of current point cloud points, a loss function measurement mode, a discard rate and the cycle number of times; step 2, when being not within the cycle number of times, jumping out of the cycle and outputting a point cloud mask M; step 3, when being within the cycle number of times, calculating coordinates x_c of a sphere center (S3), wherein the position of the sphere center of the point cloud can be roughly measured by using the median or average value of all the coordinates; step 4, calculating a gradient g of the current point cloud according to the loss function (S4); step 5, calculating a change rate δ of each point in each current point cloud with respect to the sphere center (S5); step 6, calculating scores s=-w*δ (S6) of each point in the current point cloud; and step 7, sorting all the scores from low to high, and deleting preceding [pN/T] points from the sorted points in the point cloud (S7). According to the method, a multi-region missing point cloud condition can be generated, and more flexible, changeable and complex masks can be generated.

Description

A mask generation method for simulating residual defect clouds based on saliency features

technical field

The invention relates to the field of point cloud and 3D data preprocessing, and more particularly to a mask generation method for simulating residual defect clouds based on saliency features.

Background technique

See reference appendices [1], [2], [3] and [4] for prior art related to the present invention.

The steps of the PointCloud Saliency Map method in the prior art are: 1. Input loss function L(X, y; θ), point cloud input X, label y, model weight θ, hyperparameter α, discarded points n, and cycle times T; 2. In 0-T cycles, perform steps 3-8; 3. Calculate the gradient; 4. Calculate the center coordinate point through the median value; 5. Calculate the inner product; 6. Calculate the saliency map score; 7. If If it is a high loss rate: sort from low to high, and lose the first n/T; 8. If it is a low loss rate: sort from high to low, and lose the first n/T; 9. Output the point cloud after pruning data Y.

Reference algorithm diagram

Similar existing technologies, such as the PointCloud Saliency Map method, are mainly used for the construction of feature maps of point clouds, please refer to the reference appendix [4].

The existing mask generation methods all use random sampling, that is, randomly select a point from the original data, and then remove the fixed number of points within a certain range of the point, so as to simulate the point cloud with the corresponding missing rate; therefore, the disadvantages are: 1. The generated incomplete mask is generally a single area, as shown in the figure below, the blue part is the incomplete part, a single area, not multiple areas, so it cannot well describe the diversity of the incomplete situation, such as missing multiple areas. Therefore, it cannot well reflect the lack of point clouds in real life. 2. For the training of subsequent generation tasks such as point cloud completion, the diversity of input data cannot be well guaranteed, which is not conducive to the robustness of training.

The point cloud data obtained by using the existing point cloud acquisition device or depth perception may exist due to the limited scanning area, incomplete angle, physical environment light influence, limitations of the laser scanner itself, or the collection structure of objects is too complex and changeable, etc. , which inevitably lead to the presence of missing areas in the scan results. However, due to the limited scale of point cloud datasets at this stage, there are too few such missing point cloud datasets, most of which are modeled or complete point clouds, and there is a lack of ways to simulate residual defect clouds. At the same time, the existing simulation method is simple and practical with random sampling mode, which cannot measure the importance of the point in the point cloud to the whole point cloud, and cannot satisfy the diversity of the residual defect clouds that appear in various real scenes. condition. Therefore, it is particularly important how to simulate diverse residual defect clouds and generate residual defect clouds according to the attributes of the midpoints of the point cloud.

SUMMARY OF THE INVENTION

In view of the above problems and the defects of related methods, the present invention proposes a mask generation method for simulating residual defect clouds based on saliency features, which can well simulate and represent point clouds with different missing rates in real scenes. It is flexible, convenient, and widely used to measure the salient features of each point in the cloud, and to judge whether it is proposed by scoring. It provides a basic guarantee for intelligent processing tasks such as intelligent point cloud completion and repair generation in the later stage.

In order to solve the above-mentioned technical problems, the present invention adopts the following technical solutions to achieve the purpose:

A mask generation method for simulating residual defect clouds based on saliency features. The described method includes the following steps:

Step 1. Obtain the current number of point cloud points, the measurement method of the loss function, the discard rate and the number of cycles;

Step 2. Outside the number of cycles, jump out of the cycle and output the point cloud mask M;

Step 3. Within the number of cycles, calculate the coordinates x _{c of the} center of the point cloud. You can use the median or average of all coordinates to roughly measure the center of the sphere, or find the coordinates of the center of the sphere through finer grid division. ;

Step 4: Calculate the current gradient g of each point cloud according to the loss function;

Step 5. Calculate the rate of change δ of each point in the current point cloud relative to the position of the center of the sphere;

Step 6: Calculate the saliency score s=-w*δ of each point in the current point cloud;

Step 7: Sort all the points from high to low, delete the first [pN/T] points, that is, these points in the M mask are set to 0.

This method proposes a mask generation method for simulating residual defect clouds based on saliency features. By scoring each point in the point cloud to measure the degree of its influence on the shape of the entire point cloud, some fixed proportion point sets are then eliminated. , forming the required mask that can simulate residual defect clouds. This method provides pre-order diversified residual defect cloud data for point cloud completion, repair and reconstruction tasks, and can generate various proportions of missing point clouds in simulated real scenes, making up for the fact that the existing random sampling cannot be based on the importance of points. degree of susceptibility to mask defects.

Compared with the prior art, the present invention has the following advantages and effects: Diversity: it can generate missing point clouds in multiple areas, a simpler random downsampling method, more diversity, and more flexible and complex masks can be generated. .

It should be understood that the matters described in this Summary are not intended to limit key or critical features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Description of drawings

Fig. 1 is the flow chart of the present invention;

FIG. 2 is an effect diagram of the present invention.

detailed description

To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings and through specific embodiments. Obviously, the described embodiments are some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the scope of protection in this aspect.

Step 1 S1, obtain the current number of point cloud points, the measurement method of the loss function, the discard rate and the number of cycles;

In the original residual defect cloud expression, only random sampling method is used, which cannot well express the defect situation in the real scene. Therefore, the present invention measures the saliency score by estimating each point based on the saliency feature, and then inputs the loss function in the input part to further control the position of the points to be eliminated.

Step 2 S2, outside the number of cycles, jump out of the cycle, and output the point cloud mask M;

This step is used to control whether the loop is set to end, if not, continue to S3-S7, otherwise, output the final point cloud masking result. The mask consists of 0 and 1, 1 means the point is retained, and 0 means the point is eliminated;

Step 3 S3, within the number of cycles, calculate the point cloud sphere center position coordinate x _c , use the median or average value of all coordinates to roughly measure the sphere center position, or find the sphere center through finer grid division coordinate.

Since our assumption comes from the observation that for the shape of the point cloud, the more central points do not have a great contribution to the shape relative to the edge shape, so we use the "peeling onion" rule to eliminate points, so we need to know its core. Where the point is located, that is, the coordinates of the spherical center of the point cloud here, can be measured using the spherical coordinate system, or the median or average value in the Cartesian coordinate system to roughly measure, or it can be measured through a finer mesh. Grid division to find the coordinates of the center of the sphere.

Step 4 S4, according to the loss function, calculate the current gradient change g of each point cloud;

This step is to measure the rate of change of the loss function affected by each point cloud. The loss function here can be the classification cross-entropy loss function of the point cloud, or it can be a segmentation XOR loss function calculated based on the point cloud features; it can be set depending on the task requirements.

Step 5 S5, calculate the rate of change δ of each point in the current point cloud relative to the position of the center of the sphere

Step 6 S6, calculate the saliency score of each point in the current point cloud s=-w*δ

This step provides pre-order support for calculating the saliency score in the next step. We assume that the negative value of a saliency score obtained at each point is proportional to the gradient of the loss function. The w weight value depends on the user settings, which can be simply set to all 1. The w weight value range is [0, 1]. The larger the value, the more significant its gradient density is, and the smaller the value, the less affected by the gradient. . This part is conducive to the consistent value of the region and the preservation of locality. It can also be obtained or learned by learning the adjacency matrix in the graph structure.

Step 7 S7: Sort all the points from high to low, and delete the [pN/T] points before the sorting, that is, these points in the M mask are set to 0.

It can be seen from step 1 S1 that the points that need to be eliminated in each cycle, the points that will have a greater impact on the shape are preferentially eliminated. On the one hand, it increases the recognition difficulty of the output mask, and on the other hand provides the diversity of missing point clouds. Express.

Example:

This embodiment discloses a mask generation method for simulating residual defect clouds based on salient features, which specifically includes the following steps:

Step S1, obtain the current number of point cloud points, the loss function measurement method, the discard rate and the number of cycles;

Assuming that the point cloud is X, the current number of point clouds is N, the loss function is L(x,y), the discard rate is p, and the number of cycles is T.

The loss function can be the loss function of point cloud classification, such as the loss function of the point cloud classification model obtained through point cloud network training such as PointNet[1], PointNet++[2], DGCNN[3].

Step S2, outside the number of cycles, jump out of the cycle, and output the point cloud mask M;

Here M is a label containing 0 or 1, 0 means that the point cloud has been eliminated, 1 means retained; it can also be a collection of point clouds after elimination; output storage is performed according to different tasks;

Step S3 , within the number of cycles, calculate the point cloud sphere center position coordinate x _c , and use the median or average value of all the coordinates to roughly measure the sphere center position.

It's worth noting that computing gradients directly in Cartesian coordinates is problematic because there is no view/angle invariance for points in Cartesian coordinates. In order to solve this problem, we choose to calculate in spherical coordinate system. In spherical coordinates, the coordinates of a point are expressed as (r, ψ, φ), where r is the radius. If a point is offset by δ in the direction of r, it will increase the loss function change and gradient

This change can be used as the contribution of each point to the recognition result. To calculate the gradient

It can be measured using the geometric median point xc of the point cloud, where (x _i1 , x _i2 , x _i3 ) represent the three-dimensional coordinates of _{point x i:}

x _cj =median({x _ij |x _i ∈X})(j=1, 2, 3), (2)

Step S4: Calculate the current gradient g of each point cloud according to the loss function, and measure the contribution of each point in the point cloud to the shape of the entire point cloud through the gradient change;

in,

Step S5: Calculate the rate of change δ of each point in the current point cloud relative to the position of the center of the sphere, that is, the distance of each point cloud relative to the core position is measured. The physical meaning of this value is expressed as, it is relative to the center of the sphere. The scale of the importance of the current point in terms of the degree to which the cloud shape affects. The assumption considered here is that the point in the edge region has a greater impact on the collision, while the presence or absence of the center point has little impact on the integrity of the shape. It can be understood as a "peeling onion"-style level of change. The farther the outer layer is from the center, the greater the rate of change.

δ=g _i , (4)

Step S6, calculate the saliency score of each point in the current point cloud s=-w*δ

The salient feature score expresses the degree of influence of each point on the overall shape, so it is a measure of the product of the change in position relative to the center of the sphere and the weight. w can be simply set to 1, or a custom empirical value can also be used. The value range is generally [0,1]. The larger the value, the more significant its gradient density is, and the smaller the value, the less affected by the gradient. , this part is conducive to the consistent value of the region and preserves locality, and can also be obtained or learned by learning the adjacency matrix in the graph structure;

s _i = -wδ, (5)

This means that there may be user requirements to have other weighted reference parts for each point.

Step S7, sort all the point scores from high to low, delete [pN/T] points before the sorting, that is, these points in the M mask are set to 0.

The scores of all points can be sorted by sorting algorithms, such as quick sort, heap sort, etc., and the proposed points or masks can be obtained by proposing [pN/T] points before sorting.

Fig. 2 is the effect drawing of the present invention, mask: namely the light-colored part in Fig. 2, the darker color is the original image, in order to simulate the situation of the realistic residual defect cloud, a mask needs to be generated to obtain the image in the pink image; the mask can be Talk about different types of disabilities. Different defect areas and shapes will have different effects on subsequent point cloud reconstruction and other processing.

The darker part of Figure 2 is the missing area B, the remaining part is the final mask part, and the percentage is the missing area:

The four pictures in the first row are: random seed sampling, the missing part of random seed is a continuous area, because it is a point within a certain radius of a seed area, so you can see that the blue part is an area, not a different area ;

Four pictures in the second row: Using the method of the present invention, the missing area can be several pieces, which increases the difficulty of repairing to a certain extent.

To sum up, this embodiment discloses a mask generation method for simulating residual defect clouds based on saliency features. Most of the existing simulation methods are only based on random sampling, which cannot well represent their contribution to the difficulty of the task, and cannot meet the missing situation in the real point cloud scene. There are few datasets, which cannot provide a large number of real cases for further in-depth learning and exploration. Therefore, the mask generation method for simulating residual defect clouds based on saliency features proposed by the present invention measures its contribution to the shape of the entire point cloud by scoring the midpoint of each point cloud by introducing a scoring mode of salient features. degree, to eliminate or retain according to the proportion, and can simulate the incomplete situation under different deletion proportions. It provides data sources and diverse simulation masks for subsequent completion and reconstruction tasks.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, and combinations made without departing from the spirit and principle of the present invention , simplification, etc. shall be equivalent replacement manners, which are all included in the protection scope of the present invention.

Reference appendix:

[1] Qi, CR, Su, H., Mo, K., & Guibas, LJ "Pointnet: Deep learning on point sets for 3d classification and segmentation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp.652-660.

[2] Charles R Qi, Li Yi, et al. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. arXiv preprint arXiv: 1706.02413, 2017.

[3]Yue Wang,Yongbin Sun,etal.Dynamic graph cnn for learning on point clouds.ACM Transactions on Graphics,38(5):1–12,October 2019.

[4] Zheng, Tianhang and Chen, Changyou and Yuan, Junsong and Li, Bo and Ren, Kui. PointCloud Saliency Maps. The IEEE International Conference on Computer Vision (ICCV), October, 2019

Claims

A mask generation method for simulating residual defect clouds based on saliency features, characterized in that the method comprises the following steps:

Step 1. Obtain the current number of point cloud points, the measurement method of the loss function, the discard rate and the number of cycles;

Step 2. Outside the number of cycles, jump out of the cycle and output the point cloud mask M;

Step 3. Within the number of cycles, calculate the coordinates x c of the center of the point cloud. You can use the median or average of all coordinates to roughly measure the center of the sphere, or find the coordinates of the center of the sphere through finer grid division. ;

Step 4: Calculate the current gradient g of each point cloud according to the loss function;

Step 5. Calculate the rate of change δ of each point in the current point cloud relative to the position of the center of the sphere;

Step 6: Calculate the saliency score s=-w*δ of each point in the current point cloud;

Step 7: Sort all the points from high to low, delete the first [pN/T] points, that is, these points in the M mask are set to 0.
The mask generation method for simulating residual defect clouds based on saliency features according to claim 1, characterized in that, in the first step, a measurable loss function is determined, and the loss function can be a point cloud Classification network model function, point cloud segmentation network model function, point cloud generation task model function and other related loss functions.
The mask generation method for simulating residual defect clouds based on salient features according to claim 1, characterized in that, in the second step, it is the basis for judging whether to end, and if the loop ends, the point cloud is output Mask M; the M is a label containing 0 or 1, 0 indicates that the point cloud has been eliminated, and 1 indicates that it is retained; it can also be a collection of point clouds after elimination; output storage is performed according to different tasks.
The method for generating a mask for simulating residual defect clouds based on saliency features according to claim 1, wherein in the third step, the position coordinates x c of the center of the point cloud are calculated, and all coordinates can be used. The median or average value is used to roughly measure the position of the center of the sphere, or to calculate the position of the center of the sphere precisely according to the gridding method; in the spherical coordinate system, the coordinates of a point are expressed as (r, ψ, φ), where r is the radius . If a point is offset by δ in the direction of r, it will increase the loss function change and gradient
It is measured using the geometric median point xc of the point cloud, where (x i1 , x i2 , x i3 ) represent the three-dimensional coordinates of the point x i:

x cj =median({x ij |x i ∈X})(j=1, 2, 3), (2)
The mask generation method for simulating residual defect clouds based on saliency features according to claim 1, characterized in that, in the step 4, the gradient g of each point cloud is calculated; the points can be measured by gradient changes The contribution of each point in the cloud to the shape of the entire point cloud;

in,
The mask generation method for simulating residual defect clouds based on saliency features according to claim 1, characterized in that, in the step 5, the rate of change δ of each point relative to the position of the center of the sphere is calculated; wherein , the farther the outer layer is from the center, the greater the rate of change;

δ = gr i, (4) .
The mask generation method for simulating residual defect clouds based on saliency features according to claim 1, characterized in that, in the step 6, the saliency score of each point is calculated; w can be simply set as full 1, or through custom experience points

s i = -wδ, (5)

Its value range is generally [0, 1]. The larger the value, the more significance it refers to its gradient density, and the smaller the value is, the less it is affected by the gradient.