CN117572457A - A cross-scene multispectral point cloud classification method based on pseudo-label learning - Google Patents

Abstract

The invention relates to a cross-scene multispectral point cloud classification method based on pseudo tag learning, and belongs to the technical field of multispectral laser radar point clouds. The method comprises the following steps: 1) Respectively characterizing the multispectral laser radar point clouds of the source domain scene and the target domain scenePre-alignment of row features; 2) Respectively extracting graph features of two scenes; 3) Calculating loss; 4) Iteratively performing 3) updating the source domain-target domain alignment network parameters until the model converges to obtain a pseudo tag of the target domain and the confidence coefficient thereof; 5) Setting threshold value for pseudo tags in descending orderαBefore selectingα% of pseudo tags; 6) Splicing the adjacent matrix and the feature matrix in the target domain to obtain a new feature matrix; 7) Calculating loss according to the pseudo tag obtained in the step 5) and the feature matrix obtained in the step 6); 8) And 7) updating parameters in the step until the model converges, and finally obtaining a target domain multispectral point cloud data classification result. The method can realize high-precision classification of multi-spectrum point cloud of the cross-scene.

Description

A cross-scene multispectral point cloud classification method based on pseudo-label learning

Technical field

The invention relates to a cross-scene multispectral point cloud classification method based on pseudo-label learning, and belongs to the technical field of multispectral laser radar point cloud.

Background technique

The multispectral LiDAR system can simultaneously obtain the three-dimensional spatial distribution information and spectral information in the scene, and can provide richer feature information for remote sensing scene interpretation tasks. In multispectral LiDAR-related processing tasks, most current classification methods, especially those based on deep learning, require a large number of training data sets to achieve optimal performance. However, collecting and labeling large amounts of point clouds is often laborious and time-consuming. On the other hand, they only apply to fixed scenarios, where training and test samples are independent and identically distributed. Performance drops significantly when applied to unfamiliar scenarios. Therefore, these methods are not directly transferable to other scenarios, nor can they be tested on unlabeled data collected in real time. This has become a major constraint in the interpretation of multispectral LiDAR data.

When multispectral LiDAR collects data from remote sensing scenes, various factors such as laser pulse emission angle, spatial distribution of ground objects, seasonal and weather changes will affect the intensity of the received laser pulse, which causes spectral drift. In addition, both traditional methods and methods based on deep learning have poor scene adaptability, and their performance will drop significantly when there is a distribution difference between training samples and test samples. Obviously, multispectral point clouds contain both spatial geometric information and spectral information of ground objects. By learning the spatial geometry-spectral consistency information that represents the essential attributes of ground objects from the multispectral point cloud of the source domain scene, it can guide the multispectral points of the target domain scene. The high-precision generation of cloud pseudo-labels and the use of target domain pseudo-labels to train the network can improve the performance of the multispectral point cloud feature classification network in target domain scenes and improve the network's scene adaptability. Therefore, how to generate high-precision target domain scene point cloud pseudo-labels when the multispectral point cloud spectrum drifts and the distribution of ground objects is inconsistent in different scenes, etc., and how to achieve cross-scene multispectral point cloud without real labels of the target domain scene. High-precision classification is a technical problem that needs to be solved urgently.

Contents of the invention

The technical problem to be solved by this invention is to provide a cross-scene multi-spectral point cloud classification method based on pseudo-label learning to cope with the spectral drift phenomenon of multi-spectral lidar point clouds between different scenes and alleviate the problems caused by the spectral drift phenomenon. Cross-scene multispectral lidar point cloud classification is difficult and other problems. High-precision classification of cross-scene multispectral point clouds is achieved without real labels of target domain scenes.

The technical solution of the present invention is: a cross-scene multispectral point cloud classification method based on pseudo-label learning, which includes the following steps:

Step1: Pre-align the multispectral lidar point cloud features of the labeled source domain scene and the unlabeled target domain scene according to the _L2 norm and Laplacian matrix;

Step2: Based on the pre-aligned features, use Graph Convolution Neural Networks (GCN) to extract graph features of the two scenes respectively;

Step3: Calculate the source domain classification loss, Maximum Mean Discrepancy (MMD) loss, and target domain Shannon entropy loss based on the extracted graph features and source domain labels of the two scenes;

Step4: Perform Step3 iteratively, update the source domain-target domain alignment network parameters, determine whether the model has converged, if so, end, and then proceed to Step5, otherwise repeat Step3 to obtain the pseudo label and its confidence of the target domain;

Step5: Arrange the pseudo-labels in descending order according to the confidence level, set the threshold α, and select the top α% of the pseudo-labels of the target domain as the true value input of the target domain classification network;

Step6: Splice the adjacency matrix and feature matrix in the target domain to obtain a new feature matrix as the feature input of the target domain classification network;

Step7: Calculate the target domain classification loss based on the pseudo labels selected in Step5 and the new feature matrix obtained in Step6;

Step8: Perform Step7 iteratively, update the target domain classification network parameters, and determine whether the model has converged. If so, end it. Otherwise, repeat Step7, and finally obtain the target domain multispectral point cloud data classification results.

Specifically, in Step 1, the multispectral lidar point cloud data of the labeled source domain scene is recorded as (P _s , Y), and the unlabeled target domain scene is recorded as (P _t , ),in Indicates that the source domain scene contains N _s labeled multispectral points, Represents the i-th labeled multispectral point in the source domain scene, Respectively indicating that the target domain scene contains N _t unlabeled multispectral points, Represents the i-th unlabeled multispectral point in the target domain scene, Represents the ground truth labels corresponding to all source domain scene multispectral points, Represents the true value label corresponding to the i-th multispectral point in the source domain scene.

Specifically, in Step 1, the specific steps for feature pre-alignment based on L ₂ norm and Laplacian matrix are:

(1) Perform feature transformation on the source domain and target domain features through L ₂ norm. The specific feature transformation formula is:

where x is the source domain and target domain features, are the source domain and target domain features after feature transformation, is 2 norm.

(2) According to the formula in step (1), obtain the M-dimensional source domain features and M-dimensional target domain features ,Will and Splicing to obtain the overall feature matrix of M dimensions , calculate the overall feature matrix according to the K nearest neighbor algorithm The adjacency matrix W, further calculates the diagonal matrix D, the elements in the diagonal matrix D , is an element in the adjacency matrix W, then the Laplacian matrix L=DW, so the final overall feature matrix X is updated according to the following formula:

in, is the updated feature matrix, ^T is the matrix transposition operation, N _s is the number of labeled multispectral points in the source domain scene, and N _t is the number of unlabeled multispectral points in the target domain scene.

Specifically, the Step 3 is:

The source domain scene and target domain scene graph features extracted in Step 2 are recorded as and , the source domain classification loss calculation formula is:

in, is the label of the i-th point in the source domain scene, is the predicted label of the i-th point in the source domain scene, is the source domain scene label set, N _s is the number of labeled multispectral points in the source domain scene;

In order to measure the difference between the extracted features, the Maximum Mean Discrepancy (MMD) loss is used to calculate the feature deviation of the two scenes to promote GCN to extract domain-invariant features:

in, is a mapping function that maps original variables to high-dimensional space, is the graph feature of the i-th source domain multispectral point, is the graph feature of the jth target domain multispectral point, and N _t is the number of unlabeled multispectral points in the target domain scene;

Shannon entropy loss is used to constrain the network to obtain higher confidence target domain scene pseudo-labels. The specific Shannon entropy loss formula is:

Among them, H is the Shannon entropy matrix, is an element in H, specifically Calculated as follows:

Among them, P is the network’s prediction probability matrix for the multispectral lidar point cloud in the target domain, is the prediction probability, l is the number of characteristic channels of the multispectral point cloud, Pre-aligned features for target domain nodes.

Specifically, the source domain-target domain alignment network parameters are updated in Step 4, specifically as follows:

(1) All parameters are optimized using the standard backpropagation algorithm;

(2) During training, the overall loss is a combination of source domain classification loss, Maximum Mean Discrepancy (MMD) loss, and target domain Shannon entropy loss. The overall loss of training is:

in, and is the balance coefficient to balance the loss.

Specifically, the adjacency matrix and feature matrix in the target domain are spliced in Step 6, specifically:

Denote the adjacency matrix of the target domain as , then the M-dimensional target domain features and Splicing to obtain updated target domain features .

Specifically, the specific formula for calculating the target domain classification loss in Step 7 is as follows:

in, is the pseudo label of the i-th point in the target domain scene, is the predicted label of the i-th point in the target domain scene, N _t is the number of unlabeled multispectral points in the target domain scene, is the target domain scene graph feature extracted in Step 2, is the pseudo-label set of the target domain.

Specifically, the target domain classification network parameters are updated in Step 8, specifically as follows:

(1) All parameters are optimized using standard backpropagation algorithm.

(2) In training, the target domain classification loss in Step 7 is used as the training loss.

Multispectral lidar often has the same objects with different spectra or different objects with the same spectrum in different scenes. This will lead to the use of only the network trained with source domain point cloud labels to train the target domain when there are no labels for training in the target domain scene. Point cloud classification accuracy is low. This invention aligns the features of the source domain and target domain scenes by designing feature pre-alignment operations, and uses Maximum Mean Discrepancy (MMD) loss and Shannon entropy loss to promote GCN to extract domain-invariant features and obtain high-quality targets. Domain point cloud pseudo-labels. The target domain features are enhanced according to the target domain adjacency matrix, and the labeled source domain scene multispectral point cloud is used to train the graph neural network to perform high-precision classification of the unlabeled target domain multispectral point cloud.

The beneficial effects of the present invention are: compared with the existing technology, the present invention alleviates the negative impact caused by the spectral drift of multispectral point clouds between different scenes. The feature domain alignment operation helps GCN extract domain invariant features, and maximum mean difference (MMD) loss and Shannon entropy loss are used to ensure the accuracy of pseudo-labeling of target domain point clouds. The target domain features are further enhanced based on the adjacency matrix. In situations such as spectral drift of multispectral point clouds and inconsistent distribution of ground objects in different scenes, effective and reliable information transfer is achieved to classify ground objects in unlabeled target domain scenes. Achieve high-precision classification of cross-scene multispectral point clouds without real labels of target domain scenes.

Description of the drawings

Figure 1 is the framework of the present invention's cross-scene multispectral point cloud classification method based on pseudo-label learning;

Figure 2 is a distribution map of real ground objects in the data set in the embodiment, (a) is a source scene visualization map, (b) is a target scene visualization map.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

Embodiment 1: As shown in Figure 1, a cross-scene multispectral point cloud classification method based on pseudo-label learning includes the following steps:

Step1: Pre-align the multispectral lidar point cloud features of the labeled source domain scene and the unlabeled target domain scene according to the _L2 norm and Laplacian matrix;

In Step 1, the multispectral lidar point cloud data of the labeled source domain scene is recorded as (P _s , Y), and the unlabeled target domain scene is recorded as (P _t , ),in Indicates that the source domain scene contains N _s labeled multispectral points, Represents the i-th labeled multispectral point in the source domain scene, Respectively indicating that the target domain scene contains N _t unlabeled multispectral points, Represents the i-th unlabeled multispectral point in the target domain scene, Represents the ground truth labels corresponding to all source domain scene multispectral points, Represents the true value label corresponding to the i-th multispectral point in the source domain scene.

In Step 1, the specific steps for feature pre-alignment based on L ₂ norm and Laplacian matrix are:

(1) Perform feature transformation on the source domain and target domain features through L ₂ norm. The specific feature transformation formula is:

where x is the source domain and target domain features, are the source domain and target domain features after feature transformation, is 2 norm.

(2) According to the formula in step (1), obtain the M-dimensional source domain features and M-dimensional target domain features ,Will and Splicing to obtain the overall feature matrix of M dimensions , calculate the overall feature matrix according to the K nearest neighbor algorithm The adjacency matrix W, further calculates the diagonal matrix D, the elements in the diagonal matrix D , is an element in the adjacency matrix W, then the Laplacian matrix L=DW, so the final overall feature matrix X is updated according to the following formula:

in, is the updated feature matrix, ^T is the matrix transposition operation, N _s is the number of labeled multispectral points in the source domain scene, and N _t is the number of unlabeled multispectral points in the target domain scene.

Step2: Based on the pre-aligned features, use Graph Convolution Neural Networks (GCN) to extract graph features of the two scenes respectively;

Step3: Calculate the source domain classification loss, Maximum Mean Discrepancy (MMD) loss, and target domain Shannon entropy loss based on the extracted graph features and source domain labels of the two scenes;

The source domain scene and target domain scene graph features extracted in Step 2 are recorded as and , the source domain classification loss calculation formula is:

in, is the label of the i-th point in the source domain scene, is the predicted label of the i-th point in the source domain scene, is the source domain scene label set, N _s is the number of labeled multispectral points in the source domain scene;

In order to measure the difference between the extracted features, the Maximum Mean Discrepancy (MMD) loss is used to calculate the feature deviation of the two scenes to promote GCN to extract domain-invariant features:

in, is a mapping function that maps original variables to high-dimensional space, is the graph feature of the i-th source domain multispectral point, is the graph feature of the jth target domain multispectral point, and N _t is the number of unlabeled multispectral points in the target domain scene;

Shannon entropy loss is used to constrain the network to obtain higher confidence target domain scene pseudo-labels. The specific Shannon entropy loss formula is:

Among them, H is the Shannon entropy matrix, is an element in H, specifically Calculated as follows:

Among them, P is the network’s prediction probability matrix for the multispectral lidar point cloud in the target domain, is the prediction probability, l is the number of characteristic channels of the multispectral point cloud, Pre-aligned features for target domain nodes.

Step4: Perform Step3 iteratively, update the source domain-target domain alignment network parameters, determine whether the model has converged, if so, end, and then proceed to Step5, otherwise repeat Step3 to obtain the pseudo label and its confidence of the target domain;

In Step 4, the source domain-target domain alignment network parameters are updated, specifically:

(1) All parameters are optimized using the standard backpropagation algorithm;

(2) During training, the overall loss is a combination of source domain classification loss, Maximum Mean Discrepancy (MMD) loss, and target domain Shannon entropy loss. The overall loss of training is:

in, and is the balance coefficient to balance the loss. In the present invention, and The value is 1.

Use the standard backpropagation algorithm to update the source domain-target domain alignment network parameters and determine whether the model has converged. If so, end. Otherwise, repeat step S3 until the model converges.

Step5: Arrange the pseudo labels in descending order according to the confidence level, set the threshold α, and select the top α% of the pseudo labels in the target domain as the true value input of the target domain classification network. In the present invention, the value of α is 50;

Step6: Splice the adjacency matrix and feature matrix in the target domain to obtain a new feature matrix as the feature input of the target domain classification network;

The adjacency matrix and feature matrix in the splicing target domain in Step 6 are specifically:

Denote the adjacency matrix of the target domain as , then the M-dimensional target domain features and Splicing to obtain updated target domain features .

Step7: Calculate the target domain classification loss based on the pseudo labels selected in Step5 and the new feature matrix obtained in Step6;

The specific formula for calculating the target domain classification loss in Step 7 is as follows:

in, is the pseudo label of the i-th point in the target domain scene, is the predicted label of the i-th point in the target domain scene, N _t is the number of unlabeled multispectral points in the target domain scene, is the target domain scene graph feature extracted in Step 2, is the pseudo-label set of the target domain.

Step8: Use the target classification loss in step S7 as the training loss, use the standard backpropagation algorithm to update the target domain classification network parameters, and determine whether the model has converged. If so, end it. Otherwise, repeat step S7 until the model converges.

On the basis of the specific implementation records, the feasibility of the present invention will be demonstrated through experiments below:

1. Experimental data

Harbor of Tobermory data set: The scene of this data set is a small seaport located in Tobermory, UK. The three-band point cloud data collected by Optech Titan lidar, the wavelengths are 1550nm, 1064nm and 532nm respectively. The visualization effect of the data set is as follows 2, where (a) is the source scene visualization diagram and (b) is the target scene visualization diagram. The study area is divided into seven categories according to the height, material and semantic information of land cover, namely bare land, grassland, roads, buildings, trees, power lines and cars.

University of Houston data set: This data set scene is a part of the Houston campus. It is a three-band point cloud data collected by the Optech Titan lidar. The wavelengths are 1550nm, 1064nm and 532nm respectively. The study area is divided into seven categories according to the height, material and semantic information of land cover, namely bare land, cars, grassland, roads, power lines, buildings and trees. The F score is used as the evaluation index. The visualization of the two data sets is shown in Figure 2.

2. Experimental content

In the experiment, the method of the present invention and the traditional GCN method were used to classify and verify the above data set. The Harborof Tobermory data set is used as the source domain scene, and the University of Houston data set is used as the target domain scene. In order to save computing resources, the super-point segmentation method is used to divide the two scenes into 8000 super-points as input. The method of the present invention is used for point cloud classification, and the classification results are evaluated using the evaluation indicators in the following formula. Table 1 shows the average intersection over union (MIoU) ratio of the method of the present invention in different ground objects.

Among them, TP is the number of positive class points divided into positive class points, FP is the number of negative class points divided into positive class points, and FN is the number of positive class points divided into negative class points.

Table 1

This invention can effectively deal with the spectral drift phenomenon of multi-spectral lidar point clouds between different scenes, alleviate problems such as difficulty in classifying cross-scenario multi-spectral lidar point clouds caused by the spectral drift phenomenon, and solve the problem of no real label of the target domain scene. High-precision classification of cross-scenario multispectral point clouds is achieved.

The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above-described embodiments. Within the scope of knowledge possessed by those of ordinary skill in the art, other modifications can be made without departing from the spirit of the present invention. various changes.

Claims (8)

1. A cross-scene multispectral point cloud classification method based on pseudo tag learning is characterized in that: the method comprises the following steps:

step1: respectively enabling the multispectral laser radar point cloud characteristics of the tagged source domain scene and the untagged target domain scene to be according to L ₂ Performing feature pre-alignment on the norms and the Laplace matrix;

step2: respectively extracting graph features of two scenes by adopting a graph convolution neural network (GCN) according to the pre-aligned features;

step3: calculating source domain classification loss, maximum mean difference MMD loss and target domain shannon entropy loss according to the extracted graph characteristics and source domain labels of the two scenes;

step4: iteratively performing Step3, updating the source domain-target domain alignment network parameters, judging whether the model is converged, if yes, ending, then performing Step5, otherwise repeating Step3 to obtain a pseudo tag of the target domain and the confidence coefficient thereof;

step5: according to the confidence level, the pseudo labels are arranged in a descending order, a threshold value alpha is set, and the target domain pseudo labels with the alpha percent before are selected to be used as the true value input of the target domain classification network;

step6: splicing the adjacent matrix and the feature matrix in the target domain to obtain a new feature matrix as the feature input of the target domain classification network;

step7: calculating the classification loss of the target domain according to the pseudo tag selected by Step5 and the new feature matrix obtained by Step 6;

step8: and (3) iteratively performing Step7, updating the target domain classification network parameters, judging whether the model is converged, if yes, ending, otherwise repeating Step7, and finally obtaining a target domain multispectral point cloud data classification result.

2. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 1, wherein the method comprises the following steps of: in Step1, the labeled source domain scene multispectral lidar point cloud data is denoted as (P _s Y), the unlabeled target domain scene is denoted (P _t ，) Wherein->Representing a source domain scene contains N _s Multiple spectral spots with labels->Representing the i-th labeled multispectral point in the source domain scene,>respectively representing that the target domain scene contains N _t Label-free multispectral spots->Representing the i-th unlabeled multispectral point in the target domain scene,>truth-value label corresponding to multispectral points of all source field scenes>And (5) representing a truth value label corresponding to the ith multispectral point in the source domain scene.

3. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 1, wherein the method comprises the following steps of: in Step1, the process is described in terms of L ₂ The specific steps of feature pre-alignment of the norm and the Laplace matrix are as follows:

(1) Through L ₂ The norm carries out characteristic transformation on the characteristics of the source domain and the target domain, and a specific characteristic transformation formula is as follows:

；

where x is the source domain, target domain characteristics,for the source domain, target domain characteristics after characteristic transformation,/->Is 2 norms;

(2) Obtaining source domain features of M dimensions according to the formula of the step (1)And M-dimensional target domain featuresWill->And->Splicing to obtain an overall feature matrix of M dimension +.>Calculating an overall feature matrix according to K nearest neighbor algorithm>Further calculating a diagonal matrix D, the elements in the diagonal matrix D being +.>，For elements in the adjacency matrix W, then the laplace matrix l=d-W, so the final overall feature matrix X is updated according to the following formula:

；

wherein,in order to update the feature matrix after the update, ^T for matrix transposition operationsMake N _s The number of the multispectral points with labels for the source domain scene is N _t The number of unlabeled multispectral points in the target domain scene.

4. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 1, wherein the method comprises the following steps of: the Step3 specifically comprises the following steps:

respectively marking the source domain scene and the target domain scene map features extracted from Step2 asAnd->The source domain classification loss calculation formula is:

；

wherein,is the label of the i-th point in the source domain scene,/->Is the predictive label of the i-th point in the source domain scene,>is a source domain scene tag set, N _s The number of the multi-spectrum points is the number of the labeled multi-spectrum points for the source field scene;

in order to measure the difference between the extracted features, the feature deviation of two scenes is calculated by adopting the maximum mean difference MMD loss, so as to promote the GCN extraction domain invariant feature:

；

wherein,is a mapping function mapping the original variable to a high-dimensional space,/->Is a graph characteristic of the ith source domain multispectral point,is the graph characteristic of the j-th target domain multispectral point, N _t The number of unlabeled multispectral points in the target domain scene;

the shannon entropy loss constraint network is adopted to obtain a target domain scene pseudo tag with higher confidence, and a specific shannon entropy loss formula is as follows:

；

wherein H is shannon entropy matrix,is an element in H, in particular +.>The calculation formula is as follows:

；

wherein P is a prediction probability matrix of the network to the target domain multispectral laser radar point cloud,for predicting probability, l is the number of characteristic channels of the multispectral point cloud, +.>Pre-aligned features for the target domain node.

5. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 4, wherein the method comprises the following steps: the Step4 updates the source domain-target domain alignment network parameters specifically as follows:

(1) All parameters are optimized using a standard back propagation algorithm;

(2) In training, the overall loss is a combination of source domain classification loss, maximum mean difference MMD loss and target domain shannon entropy loss, and the overall loss of training is as follows:

；

wherein,and->Is the balance coefficient of the balance loss.

6. A cross-scene multispectral point cloud classification method based on pseudo tag learning as claimed in claim 3, wherein: the adjacent matrix and the feature matrix in the splicing target domain in Step6 specifically are:

marking the target domain adjacency matrix asThe target domain feature of M dimension is +.>And->Splicing to obtain updated target domain characteristics +.>。

7. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 1, wherein the method comprises the following steps of: the specific formula for calculating the target domain classification loss in Step7 is as follows:

；

wherein,pseudo tag which is the i-th point in the target domain scene,>is the predictive label of the ith point in the target domain scene, N _t For the number of unlabeled multispectral points in the target domain scene, < >>For the target domain scene graph feature extracted from Step2,/a>Is a set of target domain pseudo tags.

8. The cross-scene multispectral point cloud classification method based on pseudo tag learning of claim 1, wherein the method comprises the following steps of: the Step8 updates the target domain classification network parameters specifically as follows:

(1) All parameters are optimized using a standard back propagation algorithm;

(2) In training, the target domain classification loss in Step7 is used as a training loss.