CN113298952B - Incomplete point cloud classification method based on data expansion and similarity measurement - Google Patents

Abstract

The invention belongs to the field of three-dimensional computer vision, and particularly relates to a defect point cloud classification network based on data expansion and similarity measurement. The invention aims to solve the problem of how to expand a complete point cloud classification network to incomplete point clouds, thereby providing an excellent solution for the classification problem of the incomplete point clouds. Therefore, the invention provides a new classification network IPC-Net aiming at incomplete point cloud on the basis of the traditional point-based classification network, can solve the problems of low classification precision and poor network robustness of the incomplete point cloud based on data expansion and similarity measurement, and in addition, utilizes an auxiliary loss function to combine with an attention mechanism to help solve the related problems.

Description

A cloud classification method based on data expansion and similarity measurement

technical field

The invention belongs to the field of three-dimensional computer vision, and specifically relates to a residual defect-oriented cloud classification method based on data expansion and similarity measurement.

Background technique

Point clouds are different from the regular grid structure of 2D images, but a set of disordered 3D points, so point cloud classification is challenging. In recent years, there have been many deep learning methods applied to 3D point clouds. The accurate classification of point clouds by these methods usually depends on the richness and availability of data. Compared with MS COCO and ImageNet datasets, which are commonly used in the field of 2D images, the scale of 3D point cloud datasets is usually smaller. Many, and the richness of the sample is insufficient. The commonly used point cloud classification datasets have complete structures and clear definitions, and there are two main ways to source point cloud data in practical applications: the first is to use lidar to measure the distance between the sensor and the object to be measured. Discrete point cloud data; the second is to use an RGB-D camera to obtain point cloud data by measuring the distance between the camera and the object. No matter which method is used to collect data, due to the limitation of occlusion between objects in the scene, light reflection, sensor resolution and viewing angle, geometric and semantic information will be lost, resulting in different degrees of incomplete point cloud. In the identification and classification, the lack of regionality will cause the point cloud to lose some features and affect the accuracy of shape classification. For example, in the VKITT dataset, some houses that are truncated by the middle are easily misidentified as trucks by the network.

SUMMARY OF THE INVENTION

The purpose of the present invention is how to extend the complete point cloud classification network to the residual defect cloud, so as to provide an excellent solution for the classification problem oriented to the residual defect cloud. Classification.

The object of the present invention is achieved in this way:

A cloud classification method based on data expansion and similarity measurement, comprising the following steps:

S1: Preprocess the original point cloud;

This step first normalizes the unnormalized data, so that the data range is always kept in [-1, 1] for the convenience of subsequent tasks;

S2: Propose a data expansion method for 3D point cloud - random area erasure;

This step can take into account the semantic information of the complete point cloud and the residual defect cloud. For the complete point cloud, the point cloud in some areas is randomly removed, and the local information is stripped off; this process simulates the occlusion and lack of actual point cloud collection. It increases the complexity and richness of the data, and enhances the robustness and generalization performance of the network; the specific steps are as follows:

S21: During the training process, after the original point cloud data is input into the network, the copy itself obtains the same "replica" as the input data of the random area erasing module;

S22: The method of random area erasing will randomly select a point in the "replica" point set as the erased area center;

S23: The random area erasing module sets an erasing range, and randomly selects any size within the range as the erasing radius to form a sphere for erasing;

S24: Clear the point cloud data in the erasing sphere. At the same time, the module needs to determine whether the number of points retained by the erased point cloud meets the lower limit of the subsequent classification network. The input points of the classification network are 1024, so when erasing When the number of remaining points is less than 1024, the network will abandon the random area erasure of the point cloud data;

S3: Design a point similarity measurement module, introduce an attention mechanism, and complete the design of the auxiliary loss function;

The similarity measurement module in this step takes the complete point cloud and the residual defect cloud output by S2 as the input, and sets the evaluation index to judge the similarity between the two; applies the attention mechanism to the complete point cloud to obtain the difference between the different points. The weight value, which is used as a parameter to participate in the construction of the similarity measurement loss; the similarity loss function is introduced into the overall loss function in the form of auxiliary loss, which is introduced in the training process of the network model and removed in the testing process of the network; the specific steps are as follows:

S31: Downsampling the complete point cloud and the residual defect cloud with different amounts of data together, the downsampling method is the farthest point sampling,

The complete point cloud and the residual defect cloud sampled by the farthest point each retain 1024 sampling points;

S32: Apply the attention mechanism to the downsampled complete point cloud to obtain the weight between each point. The formula for obtaining the attention is:

A=g(Q _i ,W)

where Q _i ={q _j |1≤j≤i} represents the complete point cloud, and W is the learnable weight of the shared multilayer perceptron;

S33: Use the evaluation indicators in point cloud completion to evaluate the relationship between the residual defect cloud and the complete point cloud. This part mainly includes the following steps:

S331: Use the chamfering distance (CD) in point cloud completion to evaluate the relationship between the complete point cloud Q and the residual defect cloud P, CD represents each point in one point cloud to the nearest point in the other point cloud. The average distance between neighbor points, which constitutes the CD loss, and its specific formula is:

S332: Use the Earth Movement Distance (EMD) in point cloud completion to evaluate the relationship between the complete point cloud Q and the residual defect cloud P. The EMD distance represents the optimal point cloud "moving" into another point cloud. The minimum consumption under the planned path constitutes the EMD loss, and its specific formula is:

where φ is a bijection that minimizes the average distance between corresponding points, and must have the same magnitude;

S333: Combining CD loss and EMD loss, CD calculation efficiency is high and time complexity is low, EMD has a better discrimination degree for the individual characteristics and local details of a single point cloud, and the two are combined to jointly construct the similarity measure loss, The loss function formula is:

S34: Take the weight obtained by the complete point cloud through the attention module as a parameter, and introduce it into the calculation of the distance d _ij between the complete point cloud and the residual defect cloud, d _ij is the core part of the similarity measurement loss, and its formula is as follows:

d _ij =A·||QP||

S4: Classify complete point cloud and residual defect cloud;

In this step, the expanded data obtained by S2 is used as the input of the classification network. The classification network part continues the encoder-decoder classification structure of PointNet++; the multi-level feature extraction structure captures local and global features, and the classification loss and similarity measurement loss are A specific ratio is combined to form the overall loss function of the network; its specific steps are as follows:

S41: Input the expanded complete point cloud and the residual defect cloud together into the classification network to achieve residual defect cloud classification; the specific steps are as follows:

S411: Use the SA layer in the coding part of PointNet++ to achieve feature extraction. The SA layer specifically includes three modules: Sampling, Grouping, and PointNet. The specific steps are as follows:

S4111: The Sampling module in the SA layer first downsamples the point cloud of the input module, the sampling method is the farthest point sampling, and the number of sampling points is 1024 points;

S4112: The Grouping module in the SA layer takes the selected point as the centroid, selects an appropriate number of points in the fixed area through the neighborhood algorithm to form a local point set, and the number of points selected in each neighborhood is 32;

S4113: The PointNet part of the SA layer uses max-pooling as a symmetric function to achieve the invariance of the arrangement of point clouds, and uses the MLP module to extract local features. The formula for this process is as follows:

f({x ₁ ,...,x _n })≈g(h(x ₁ ),...,h(x _n ))

In this implementation process, h represents MLP, g represents the max-pooling function, and the max-pooling function represents the maximum value in the feature and satisfies the permutation invariance. The formula is:

max(x ₁ ,x ₂ ,x ₃ ,...)≡max(x _α1 ,x _α2 ,x _α3 ,...), _xi ∈R ^D

S412: Repeat the process of S411 twice to gradually obtain local features and global features at different levels;

S413: The decoding part introduces the global feature obtained by the downsampling obtained from the encoding part as an input into the fully connected network, and completes the classification through Softmax;

S42: Combine the loss function obtained by the classification network and the similarity measure loss obtained by S3 to form the loss function of IPC-Net. The loss function L _CLS of the classification network part is NLLLoss; The form of is applied to the training process of the model and removed in the testing process; the hyperparameter alpha is used to balance the two losses, the alpha value is determined to be 0.05 through experiments, and the loss function of IPC-Net is:

L _CON = 0.95L _CLS + 0.05L _SM .

Compared with the prior art, the beneficial effects of the present invention are:

1. The present invention proposes a residual defect cloud classification network IPC-Net. It can extend the complete point cloud classification network to incomplete point clouds, thus ensuring the robustness of point cloud classification with different completeness.

2. The present invention designs a data expansion method for 3D point cloud, which pays attention to data expansion at the semantic level and has practical significance. We propose a point similarity measurement module in which an attention mechanism is introduced and refined to design an auxiliary loss function.

3. The present invention does not depend on a specific data set, is universal on different input data sets, and achieves an excellent classification effect on the ModelNet40 data set.

Description of drawings

Fig. 1 is the flow chart of the residual defect cloud classification network IPC-Net based on data expansion and similarity measure proposed by the present invention;

Fig. 2 is each module structure diagram of IPC-Net network;

Fig. 3 is the flow chart of random area erasing;

Figure 4 is the visualization result of random erasure of the ModelNet40 dataset;

Figure 5 is a visualization of the homemade dataset used to judge the generalization performance of IPC-Net.

Detailed ways

The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

The point cloud mentioned in this application is a massive set of points that express the spatial distribution of the target and the characteristics of the target surface under the same spatial reference system. After obtaining the spatial coordinates of each sampling point on the surface of the object, a set of points is obtained.

The point cloud classification mentioned in this application is to assign a semantic label to each point. The classification of point clouds is to classify point clouds into different point cloud sets, and the same point cloud set has similar or the same properties.

The point cloud completion mentioned in this application is a method widely used in the field of three-dimensional object detection and recognition for repairing missing point clouds. The complete point cloud is estimated from the missing point cloud, so as to obtain higher quality point cloud. The classification network for the residual defect cloud is a further promotion of the point cloud classification network. The common analysis method of the residual defect cloud is mainly the point cloud completion network. Point cloud completion is a priori task to achieve residual and defect cloud classification. After obtaining complete point cloud data, shape classification is performed. However, in point cloud classification tasks, point cloud completion operations are redundant. Because the network can achieve classification by learning the semantic information of residual defect clouds without generating local details with indistinct features. In addition, adding the point cloud completion network before the classification network will also increase the complexity of the model. At the same time, in the research of deep learning analysis of 3D point cloud, training data plays a pivotal role. However, the actual training data usually has the problems of small size, low complexity and poor richness. Therefore, data scaling is a commonly used training method. Data augmentation is an explicit form of regularization that aims to expand the number and richness of the training set using data only through human manipulation. Due to the operational transformations performed in the process, data augmentation maximizes the semantic information that the network can learn, which also becomes a key part of preventing overfitting of convolutional neural network models.

The content of the present invention mainly includes a data expansion module and a point similarity measurement module. The data expansion part is mainly realized by erasing random areas. The random region erasing process simulates the occlusion of point clouds in practical applications, thereby improving the generalization performance of the network. The point similarity measure module combines an attention mechanism with a similarity measure to guide the learning of incomplete point features towards the original point features. At the same time, the present invention uses attention to obtain the weights between different points of the original point cloud, so that the network pays more attention to the learning features with high information content. The present invention also uses the hierarchical classification structure of PointNet++ as a reference to demonstrate IPC-Net and demonstrate its effectiveness on geometric tasks.

The present invention will be further described in detail below with reference to the accompanying drawings.

FIG. 1 is a flowchart of the residual defect cloud classification network IPC-Net based on data expansion and similarity measurement proposed by the present invention, and FIG. 2 is a specific structural diagram of each module of the IPC-Net network. Specifically include the following steps:

S1, preprocess input data;

S2, perform data expansion on the original point cloud;

S3, use the similarity measurement module to evaluate the difference between the complete point cloud and the residual defect cloud, and output the similarity measurement loss;

S4, use the classification network to achieve cloud classification of residual defects.

The following describes the residual defect cloud classification network based on data expansion and point cloud similarity measurement in more detail with reference to the accompanying drawings.

In step S1, the original point cloud is normalized, and the point cloud data of different sizes are normalized to the range of [-1, 1]. The visualization result of the original point cloud is shown in Figure 5.

In step S2, data expansion is performed on the original point cloud with complete structure. The data expansion process is mainly realized by erasing random regions. The process of erasing regions is shown in Figure 3. The specific steps are as follows:

S2.1, in the training process, after the original point cloud data is input into the network, the copy itself obtains the same "replica" as the input data of the random area erasing module.

S2.2, the method of random area erasing will randomly select a point in the "replica" point set as the center of the area to be erased.

S2.3, the random area erasing module sets an erasing range, and randomly selects any size within the range as the erasing radius to form a sphere for erasing.

S2.4, clear the point cloud data in the erased sphere. At the same time, the module needs to judge whether the number of points retained by the erased point cloud meets the lower limit of the requirements of the subsequent classification network. The number of input points of the classification network is 1024, so when When the number of remaining points after erasing is less than 1024, the network will give up the random area erasing of the point cloud data.

The data expansion module of the present invention ensures the principle of randomness to the greatest extent. First, the position of the center point of the erasing area is random, secondly, the radius of the erasing area is also random. Finally, there may be points that are highly concentrated in the erasing area. Therefore, for a point cloud data, whether the erasing operation can be performed is also random. Figure 4 visually shows the complete process of random erasing.

Step S3, introduce the input point cloud sampled by the farthest point into the similarity measurement module, use the evaluation index to connect the complete point cloud and the residual defect cloud, and obtain the point cloud similarity measurement loss by evaluating the difference between the two, The specific steps of this process are as follows:

S3.1, down-sampling the complete point cloud and residual defect cloud with different amounts of data together. The down-sampling method is the farthest point sampling, and the complete point cloud and residual defect cloud sampled by the farthest point each retain 1024 sampling points.

S3.2, the attention mechanism is applied to the downsampled complete point cloud to obtain the weight between each point. The formula for obtaining the attention is:

A=g(Q _i ,W)

where Q _i ={q _j |1≤j≤i} represents the complete point cloud, and W is the learnable weight of the shared multilayer perceptron.

S3.3, use the evaluation indicators in point cloud completion to evaluate the relationship between the residual defect cloud and the complete point cloud. This part mainly includes the following steps:

S3.3.1, use the chamfering distance (CD) in point cloud completion to evaluate the relationship between the complete point cloud Q and the residual defect cloud P, CD represents each point in one point cloud to another point cloud with it The average distance between the nearest neighbor points, which constitutes the CD loss, and its specific formula is:

S3.3.2, the Earth Movement Distance (EMD) in point cloud completion is used to evaluate the relationship between the complete point cloud Q and the residual defect cloud P. The EMD distance represents the "movement" of one point cloud into another point cloud. The minimum consumption under the optimal planning path constitutes the EMD loss, and its specific formula is:

where φ is a bijection that minimizes the average distance between corresponding points, and must have the same magnitude.

S3.3.3, Combining CD loss and EMD loss, CD calculation efficiency is high and time complexity is low, EMD has a better degree of discrimination for the individual characteristics and local details of a single point cloud, and the two are combined to jointly construct a similarity measure loss, the loss function formula is:

S3.4, take the weight obtained by the complete point cloud through the attention module as a parameter, and introduce it into the calculation of the distance d _ij between the complete point cloud and the residual defect cloud, d _ij is the core part of the similarity measure loss, and its formula is as follows:

d _ij =A·||QP||

S4.1, input the expanded complete point cloud and residual defect cloud into the classification network to realize residual defect cloud classification. The specific steps are as follows:

S4.1.1, use the SA layer in the coding part of PointNet++ to achieve feature extraction. The SA layer specifically includes three modules: Sampling, Grouping, and PointNet. The specific steps are as follows:

S4.1.1.1, the Sampling module in the SA layer first downsamples the point cloud of the input module, the sampling method is the farthest point sampling, and the number of sampling points is 1024 points.

S4.1.1.2, the Grouping module in the SA layer takes the selected point as the centroid, and selects an appropriate number of points in the fixed area through the neighborhood algorithm to form a local point set, and the number of points selected in each neighborhood is 32.

In S4.1.1.3, the PointNet part of the SA layer uses max-pooling as a symmetric function to achieve the invariance of the arrangement of point clouds, and uses the MLP module to extract local features. The formula for this process is as follows:

f({x ₁ ,...,x _n })≈g(h(x ₁ ),...,h(x _n ))

In this process, h represents the MLP, g represents the max-pooling function, and the max-pooling function represents the maximum value in the feature and satisfies the permutation invariance. The formula is:

max(x ₁ ,x ₂ ,x ₃ ,...)≡max(x _α1 ,x _α2 ,x _α3 ,...), _xi ∈R ^D

S4.1.2, repeat the process of S411 twice to gradually obtain local features and global features at different levels.

S4.1.3, the decoding part takes the global features obtained by down-sampling obtained from the encoding part as input into the fully connected network, and completes the classification through Softmax.

S4.2, the loss function obtained by the classification network and the similarity measure loss obtained in S3 are combined to form the loss function of IPC-Net, and the loss function L _CLS of the classification network part is NLLLoss. The similarity measure loss is obtained by S333, and is applied to the training process of the model in the form of an auxiliary function, and is removed in the testing process. The hyperparameter alpha is used to balance the two losses. The alpha value in the present invention is determined to be 0.05 through experiments, and the loss function of IPC-Net is:

L _CON = 0.95L _CLS +0.05L _SM

Combining the above-described classification method of residual defect cloud, when the input point cloud is a complete point cloud or a residual defect cloud with different degrees of defect, the ICP-Net network proposed by the present invention can achieve a better classification effect, and The network has strong robustness.

Although the present invention has been disclosed above with preferred embodiments, it is not intended to limit the present invention. Those skilled in the art to which the present invention pertains can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention should be defined by the claims.

Claims (1)

1. A residual defect cloud classification method based on data expansion and similarity measure, is characterized in that: comprise the following steps: S1: Preprocess the original point cloud; This step first normalizes the unnormalized data, so that the data range is always kept in [-1, 1] for the convenience of subsequent tasks; S2: Propose a data expansion method for 3D point cloud - random area erasure; This step can take into account the semantic information of the complete point cloud and the residual defect cloud. For the complete point cloud, the point cloud in some areas is randomly removed, and the local information is stripped; this process simulates the occlusion and missing problems in the actual collection of point clouds. , and increases the complexity and richness of the data, and enhances the robustness and generalization performance of the network; the specific steps are as follows: S21: During the training process, after the original point cloud data is input into the network, the copy itself obtains the same "replica" as the input data of the random area erasing module; S22: The method of random area erasing will randomly select a point in the "replica" point set as the erased area center; S23: The random area erasing module sets an erasing range, and randomly selects any size within the range as the erasing radius to form a sphere for erasing; S24: Clear the point cloud data in the erasing sphere. At the same time, the module needs to determine whether the number of points retained by the erased point cloud meets the lower limit of the subsequent classification network. The input points of the classification network are 1024, so when erasing When the number of remaining points is less than 1024, the network will give up the random area erasure of the point cloud data; S3: Design a point similarity measurement module, introduce an attention mechanism, and complete the design of the auxiliary loss function; The similarity measurement module in this step takes the complete point cloud and the residual defect cloud output by S2 as the input, and sets the evaluation index to judge the similarity between the two; applies the attention mechanism to the complete point cloud to obtain the difference between the different points. The weight value, which is used as a parameter to participate in the construction of the similarity measurement loss; the similarity loss function is introduced into the overall loss function in the form of auxiliary loss, which is introduced in the training process of the network model and removed in the testing process of the network; the specific steps are as follows: S31: down-sampling the complete point cloud and the residual defect cloud with different amounts of data together, the down-sampling method is the farthest point sampling, and the complete point cloud and the residual defect cloud sampled by the farthest point each retain 1024 sampling points; S32: Apply the attention mechanism to the downsampled complete point cloud to obtain the weight between each point. The formula for obtaining the attention is: A=g(Q _i ,W) where Q _i ={q _j |1≤j≤i} represents the complete point cloud, and W is the learnable weight of the shared multilayer perceptron; S33: Use the evaluation indicators in point cloud completion to evaluate the relationship between the residual defect cloud and the complete point cloud. This part mainly includes the following steps: S331: Use the chamfering distance CD in point cloud completion to evaluate the relationship between the complete point cloud Q and the residual defect cloud P, CD represents each point in one point cloud to the nearest neighbor point in another point cloud The average distance between , which constitutes the CD loss, and its specific formula is:

S332: Use the earth movement distance EMD in point cloud completion to evaluate the relationship between the complete point cloud Q and the residual defect cloud P. The EMD distance represents the optimal planning path for one point cloud to "move" into another point cloud The minimum consumption under , thus constitutes the EMD loss, and its specific formula is:

where φ is a bijection that minimizes the average distance between corresponding points, and must have the same magnitude; S333: Combining CD loss and EMD loss, CD calculation efficiency is high and time complexity is low, EMD has a better discrimination degree for the individual characteristics and local details of a single point cloud, and the two are combined to jointly construct the similarity measure loss, The loss function formula is:

S34: Take the weight obtained by the complete point cloud through the attention module as a parameter, and introduce it into the calculation of the distance d _ij between the complete point cloud and the residual defect cloud, d _ij is the core part of the similarity measurement loss, and its formula is as follows: d _ij =A·||QP|| S4: Classify complete point cloud and residual defect cloud; In this step, the expanded data obtained by S2 is used as the input of the classification network. The classification network part continues the encoder-decoder classification structure of PointNet++; the multi-level feature extraction structure captures local and global features, and the classification loss and similarity measurement loss are A specific ratio is combined to form the overall loss function of the network; its specific steps are as follows: S41: Input the expanded complete point cloud and the residual defect cloud together into the classification network to achieve residual defect cloud classification; the specific steps are as follows: S411: Use the SA layer in the coding part of PointNet++ to achieve feature extraction. The SA layer specifically includes three modules: Sampling, Grouping, and PointNet. The specific steps are as follows: S4111: The Sampling module in the SA layer first downsamples the point cloud of the input module, the sampling method is the farthest point sampling, and the number of sampling points is 1024 points; S4112: The Grouping module in the SA layer takes the selected point as the centroid, selects an appropriate number of points in the fixed area through the neighborhood algorithm to form a local point set, and the number of points selected in each neighborhood is 32; S4113: The PointNet part of the SA layer uses the maximum pooling max-pooling as a symmetric function to achieve the invariance of the arrangement of the point cloud, and uses the MLP module to extract local features. The formula for this process is as follows: f({x ₁ ,...,x _n })≈g(h(x ₁ ),...,h(x _n )) In this process, h represents the MLP, g represents the max-pooling function, and the max-pooling function represents the maximum value in the feature and satisfies the permutation invariance. The formula is: max(x ₁ ,x ₂ ,x ₃ ,...)≡max(x _α1 ,x _α2 ,x _α3 ,...), _xi ∈R ^D S412: Repeat the process of S411 twice to gradually obtain local features and global features at different levels; S413: The decoding part introduces the global feature obtained by the downsampling obtained from the encoding part as an input into the fully connected network, and completes the classification through Softmax; S42: Combine the loss function obtained by the classification network and the similarity measure loss obtained by S3 to form the loss function of IPC-Net. The loss function L _CLS of the classification network part is NLLLoss; The form of is applied to the training process of the model and removed in the testing process; the hyperparameter alpha is used to balance the two losses, the alpha value is determined to be 0.05 through experiments, and the loss function of IPC-Net is: L _CON = 0.95L _CLS + 0.05L _SM .

Images