CN114694022A - Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method - Google Patents

Abstract

A semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood, the method comprising: registering the acquired point cloud data with the remote sensing image to generate point cloud data fusing RGB information; selecting a spherical neighborhood to obtain local neighborhood characteristics of point cloud data fused with RGB information, and extracting multi-scale point cloud characteristics by changing the radius of the spherical neighborhood; and combining the extracted basic features, the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud, inputting the combination into an improved model MSMF-PointNet based on PointNet for semantic segmentation, and outputting a classification result. The method can obtain the classification precision far better than PointNet in the outdoor scene point cloud data obtained by airborne LiDAR scanning, and has the advantages of better classification of building facades, fences and the like due to the addition of characteristics such as linearity and verticality, better classification results of trees and shrubs due to the addition of roughness and total variance, better classification results of roofs and impervious grounds due to the addition of flatness.

Description

A Semantic Segmentation Method Based on Spherical Neighborhood Multi-scale and Multi-feature Algorithm

technical field

The invention belongs to the technical field of remote sensing and photogrammetry, and in particular relates to a semantic segmentation method based on a spherical neighborhood-based multi-scale and multi-feature algorithm.

Background technique

Deep learning is an emerging technology that can automatically learn to extract advanced features of input data through deep network structure. It is the most influential and fastest-growing cutting-edge technology in current pattern recognition, computer vision and data analysis. Before being applied to 3D data, deep learning has become an effective force for various tasks in 2D computer vision and image processing. After winning the championship with a score of more than ten percent in the second place, the deep neural network structure dominated by CNN has made major breakthroughs in the fields of image classification, segmentation and recognition. However, due to the high-density, massive and unstructured characteristics of 3D laser point clouds, traditional deep learning methods cannot be directly applied to the segmentation of 3D point clouds.

A large number of scholars have done a lot of work on the deep learning of 3D point cloud data. In order to apply the existing neural network structure to the 3D laser point cloud data, it is necessary to perform data pre-processing before inputting the point cloud data into the neural network. There are three commonly used methods for processing: (1) using traditional convolutional neural networks after projecting 3D point cloud data into multi-view 2D images; (2) using 3D convolutional neural networks after converting point clouds into grid voxels network; (3) use a 3D graph convolutional neural network after converting the point cloud into a graph (Graph) structure.

However, the above methods will cause the loss of data information in the process of preprocessing. In response to this shortcoming, in 2017, scholars from Stanford University, Qi et al., published a pioneering research work and proposed a deep learning model PointNet, which can be directly applied to On the 3D point cloud data, the accuracy of semantic segmentation is further improved. However, the PointNet network structure mainly extracts global features between point clouds, ignoring the extraction of local features associated with points in point clouds, and the lack of local feature extraction ability often leads to insufficient segmentation accuracy and poor object detail segmentation. And other issues.

On the one hand, due to the high-density, massive and unstructured characteristics of 3D laser point clouds, it is of great theoretical value to study fast and effective 3D scene semantic segmentation algorithms. On the other hand, due to the complexity of real natural scenes, overlapping, occlusion and other phenomena between 3D objects, the research is combined with other fields of technology, and a highly robust, automated and intelligent semantic segmentation method for complex 3D scene objects is proposed for further research. Point cloud semantic segmentation and its application in various fields have important practical significance.

SUMMARY OF THE INVENTION

To solve the above problems, a semantic segmentation method based on spherical neighborhood multi-scale and multi-feature algorithm is provided.

The purpose of this invention is to realize in the following way:

A semantic segmentation method based on a spherical neighborhood-based multi-scale and multi-feature algorithm, the method comprising:

S1: Register the acquired point cloud data with the remote sensing image to generate point cloud data fused with RGB information;

S2: Multi-scale neighborhood design and feature extraction for the point cloud data fused with RGB information: By studying the point cloud spatial index structure, the spherical neighborhood is selected to obtain the local neighborhood features of the point cloud data fused with RGB information, and By changing the radius of the spherical neighborhood, multi-scale point cloud features are extracted; the point cloud features include basic features and covariance-based multi-features; the basic features include xyz coordinate information and RGB information of the point cloud; The multi-feature of covariance includes 5-dimensional neighborhood features, that is, the covariance-based total variance, roughness, flatness, linearity and verticality information;

S3: Combine the extracted basic features, 5-dimensional neighborhood features of at least two scales, and the xyz coordinate information of the point cloud into an improved PointNet-based model MSMF-PointNet for semantic segmentation, and output the classification result.

The improved PointNet-based model MSMF-PointNet includes an improved PointNet network and at least two Mini-pointnet networks; the xyz coordinate information and RGB information of the point cloud are input into the improved PointNet network, and the output of the improved PointNet network is 64 dimensional point features and 1024-dimensional global features; the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud are combined into the Mini-pointnet network; the output of Mini-PointNet is two 256-dimensional feature vectors , the two parts of the output data are fully connected and input into the softmax classifier for classification.

The improved PointNet network includes six layers, from input to output, the first T-Net point cloud rotation transformation, the first perceptron mlp, the second T-Net, the second perceptron mlp, the third perceptron mlp and Max pooling network.

The Mini-pointnet network includes 4 layers, which are T-Net point cloud rotation transformation, two-layer perceptron mlp and Max pooling network in order from input to output.

The xyz coordinate information of the point cloud is calculated according to the data recorded by the GPS, INS and laser rangefinder of the system; the RGB information is acquired by the imaging device.

The registration in S1 mainly assigns the spectral information of different pixel points in each band of the image to the point cloud data through the function of "value extraction to points" in Arcigs.

An electronic device comprising:

at least one processor;

as well as

a memory communicatively coupled to the at least one processor; wherein,

The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

A non-transitory computer-readable storage medium storing computer instructions, wherein,

The computer instructions are for causing the computer to perform the above-described method.

A computer program product comprising a computer program that, when executed by a processor, implements the above-described method.

The beneficial effects of the present invention are as follows: (1) The proposed method can obtain far better classification accuracy than PointNet in the outdoor scene point cloud data obtained by airborne LiDAR scanning. , fences, etc. are better classified, adding roughness, full variance, trees and shrubs are better classified, adding flatness, roofs and impervious ground classification results are better.

(2) The method of the present invention adds spectral information and other geometric features, and conducts training based on deep learning, which can effectively make up for the lack of spatial geometric features of point clouds, improve the classification accuracy of point clouds, and improve the classification accuracy of roofs, impervious ground and trees. The classification accuracy is higher.

Description of drawings

FIG. 1 is a schematic diagram of index establishment based on a virtual regular grid.

Figure 2 is a comparison chart of the classification effects of different radii.

Figure 3 is the MSMF-PointNet network model architecture.

Figure 4 shows the classification accuracy of various ground objects and overall classification at different scales.

Figure 5 is a flowchart of the semantic segmentation model of the spherical neighborhood-based multi-scale and multi-feature algorithm.

Detailed ways

The present invention will be described in further detail below with reference to Fig. 1 to Fig. 5 and specific embodiments.

It should be noted that the following detailed description is exemplary and intended to provide further explanation of the application. Unless otherwise defined, all technical and scientific terms used herein have the same technical meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

A semantic segmentation method based on a spherical neighborhood-based multi-scale and multi-feature algorithm, the method comprising:

S1: Register the acquired point cloud data with the remote sensing image to generate point cloud data fused with RGB information;

S2: Multi-scale neighborhood design and feature extraction for the point cloud data fused with RGB information: By studying the point cloud spatial index structure, the spherical neighborhood is selected to obtain the local neighborhood features of the point cloud data fused with RGB information, and By changing the radius of the spherical neighborhood, multi-scale point cloud features are extracted; the point cloud features include basic features and covariance-based multi-features; the basic features include xyz coordinate information and RGB information of the point cloud; The multi-feature of covariance includes 5-dimensional neighborhood features, that is, the covariance-based total variance, roughness, flatness, linearity and verticality information;

S3: Combine the extracted basic features, 5-dimensional neighborhood features of at least two scales, and the xyz coordinate information of the point cloud into an improved PointNet-based model MSMF-PointNet for semantic segmentation, and output the classification result.

As shown in Figure 3, the improved PointNet-based model MSMF-PointNet includes an improved PointNet network and at least two Mini-pointnet networks; the xyz coordinate information and RGB information of the point cloud are input into the improved PointNet network, and the improved The PointNet network outputs 64-dimensional point features and 1024-dimensional global features; the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud are combined into the Mini-pointnet network; Mini-PointNet outputs two A 256-dimensional feature vector, the two parts of the output data are fully connected, and input into the softmax classifier for classification.

The improved PointNet network includes six layers, which are T-Net point cloud rotation transformation, first perceptron mlp, T-Net, second perceptron mlp, third perceptron mlp and Max pooling network in order from input to output.

The Mini-pointnet network includes 4 layers, which are T-Net point cloud rotation transformation, two-layer perceptron mlp and Max pooling network in order from input to output.

Further, research on point cloud feature extraction method based on spherical neighborhood:

In order to improve the efficiency of query and retrieval of point cloud data, it is necessary to build a spatial index of point cloud data. Commonly used index methods include quadtree, octree, k-d tree, etc. Due to the high index efficiency of k-d tree, it has been widely used in point cloud data, and different spatial indexes are used for specific data processing requirements. In the present invention, for the design of subsequent filtering algorithms, k-d tree is selected to construct the spatial index of point cloud and make space Partition and nearest neighbor search.

After the K-d tree is established, it is necessary to further select the query method of neighborhood points. The scale difference between different objects in the outdoor environment is large, and the method of neighborhood selection in the process of feature extraction determines the ability of features to describe different objects. The single-point classification of the point cloud is very dependent on the extraction of local features, and the local features are extracted from the neighborhood point set of the selected point, so the single-point classification is closely related to the selected local neighborhood area.

Radius query is to give the target point and the threshold of the query distance (with the target point as the center of the circle, and the query distance as the radius R), and find all the data (data within the radius) whose distance from the query point is less than the threshold from the dataset. Common local There are three types of neighborhood selection: K-nearest neighborhood, cylindrical neighborhood and spherical neighborhood. The spherical neighborhood method is also called the radius query method. The schematic diagram is shown in Figure 1. In the figure, (a) is the spherical radius neighborhood, (b) is the columnar neighborhood, and (c) is the K-nearest neighborhood.

Due to the spatial heterogeneity of 3D urban scenes, the classification effect under spherical neighborhood is better. Compared with the K-nearest neighbor method, the spherical neighborhood corresponds to a fixed part in space and is relatively less affected by the density of the point cloud, so the present invention selects the spherical neighborhood method.

Multi-scale point cloud features are obtained by changing the size of the spherical neighborhood radius R.

The choice of scale will directly affect the classification accuracy of the point cloud, so it is necessary to select an appropriate scale according to the features in the scene, that is, the radius value R in the spherical neighborhood method. If the radius is too large, the spherical neighborhood contains too many points, which will greatly increase the calculation time and reduce the efficiency. Generally speaking, the minimum value of the radius is greater than the average density of the point cloud and increases regularly. In order to select the most suitable radius parameter, and to verify that the segmentation effect of the multi-scale fusion point cloud feature is better, select a part of the Vaihingen data set, select The radius parameter R=0.4m, 0.8m, 1.2m, 2.0m and their four combinations 0.4+0.8, 0.8+1.2, 0.8+2.0, 1.2+2.0 The calculated point cloud features are input into the PointNet network respectively , except the input is different, the rest are the same. The histogram of the classification effect is shown in Figure 2: from the comparison and analysis of the histogram in Figure 2, the combination of multi-scale R=0.8+1.2 has the highest classification accuracy, and single-scale R=0.8 The accuracy is the highest, because it is only to determine the most suitable radius parameter, and the network has not been upgraded, so there is no specific accuracy, only the accuracy is relatively high. Therefore, the present invention comprehensively considers that the radius R of the final spherical neighborhood is 0.8m and 1.2m.

Basic and common features of point cloud and selection of multi-features based on covariance

The basic features of the point cloud are the xyz coordinate information and RGB information of the point cloud. The xyz can be calculated according to the data recorded by the GPS, INS and laser rangefinder of the system. The RGB information is obtained by the imaging device. Therefore, in order to improve the classification accuracy, the airborne LiDAR point cloud and multi-spectral aerial imagery are usually fused to generate point cloud data with spectral information, so as to realize the spectral information supplement of the point cloud data.

When the internal and external azimuth elements of the aerial image are known, the three-dimensional coordinates of the LiDAR point cloud are substituted into the collinear condition equation, and the pixel position of the corresponding three-dimensional point on the image can be calculated, and then the near-infrared (NIR), Grayscale values for the red (R) and green (G) channels. The collinear condition equation can be expressed as:

In the formula, f is the image focal length, XYZ is the three-dimensional coordinates of the ground point, (X _S , Y _S , Z _S ) are the three line elements of the outer orientation element, (a ₁ , b ₁ , c ₁ , a ₂ , b ₂ , c ₂ , a ₃ , b ₃ , c ₃ ) are the rotation matrix parameters calculated from the three corner elements in the outer orientation element. After calculating the spectral information corresponding to each laser foot point (NIR, R, G three-channel gray value), the point cloud three-dimensional coordinates and spectral information are combined to obtain enhanced point cloud data, which is used as the input data for subsequent point cloud classification. Fusion of point clouds and images.

Covariance features are common features in point clouds, which can characterize the shape of objects or the distribution state of local point clouds, and have important applications in point cloud data processing. For three-dimensional point cloud data, the point cloud can be represented by D={P _i , i=1, . . . , N}, where N is the number of points in the point cloud D. For this, first calculate the mean and covariance matrix of the current point cloud:

Among them, the formula for the unbiased estimate of C is:

Secondly, since the covariance matrix C is a positive definite matrix, it can be known from the characteristics of the symmetric positive definite matrix that the eigenvalues of the matrix C: λ ₁ ≥λ ₂ ≥λ ₃ ≥ 0, and the corresponding eigenvectors e ₁ , e ₂ , e ₃ among They are perpendicular to each other and can form an orthogonal coordinate system, that is, C can be expressed as:

According to the size of the eigenvalues, the morphological features of the local neighborhood point cloud can be obtained. When λ ₁ >>λ ₂ ≈λ ₃ , the local neighborhood point cloud is linearly distributed. When λ ₁ ≈ λ ₂ >>λ ₃ , the point cloud is distributed in a plane. When λ ₁ >>λ ₂ ≈λ ₃ , the point cloud is distributed offline in three dimensions. And use linearity L _λ , flatness P _λ and dispersion S _λ to represent one-dimensional, two-dimensional and three-dimensional features:

L _λ =(λ ₁ -λ ₂ )/λ ₁ (6)

P _λ =(λ ₂ -λ ₃ )/λ ₁ (7)

S _λ =λ ₃ /λ ₁ (8)

The sum of the three is 1. According to formula (3), the covariance matrix constructed by the neighborhood of the selected point can be obtained, and then the eigenvalues of the covariance matrix and the corresponding eigenvectors can be obtained.

In addition to selecting linearity and flatness, the present invention also selects total variance:

Verticality:

V _λ =1-|Z·N| (10)

Where Z is the vertical unit vector, and N is the normal vector of the point.

A piece of Vaihingen experimental data set is intercepted, and the characteristics of each neighborhood when R=0.8 is selected. The estimation results of each eigenvalue are visualized, and it can be clearly seen that the eigenvalues of different objects are obviously different, which is beneficial to distinguish.

Research on point cloud segmentation method based on improved PointNet:

Based on the improved PointNet network, the invention proposes a MSMF-PointNet ground object classification and segmentation algorithm that integrates multi-scale and multi-neighborhood features, which makes up for the shortcomings of the original PointNet network for insufficient use of point cloud features and lack of local features, and draws on PointNet and PointNet++ for reference. The network parameter settings of the 16-dimensional spherical neighborhood-based multi-scale local features (xyz, RGB, R = 0.8 and 1.2 corresponding to R, O _λ , P _λ , L _λ , V _λ ) replace the original single point The xyz information is used as a new data source for the training classification of the classifier.

The specific improvement ideas are as follows: In view of the problem of increasing the dimension of the fusion point cloud feature space, the number of channels is increased by adjusting the dimension of the input transformation matrix, so that the matrix changes from the original processing of three-dimensional feature vectors to the six-dimensional and eight-dimensional feature vectors after fusion; Due to the increase in the amount of data caused by the expansion of the feature space of cloud data, the depth features of the point cloud are fully extracted by deepening the number of network layers in the MLP layer; for the problem of lack of local neighborhood features, the point cloud features calculated in the spherical neighborhood are used as input to build Two mini-PointNet feature extraction networks extract multi-neighbor point cloud features at different scales. The MSMF-PointNet network model architecture is shown in Figure 3.

Experimental verification and analysis

Test area data: The test data set of the German city of Vaihingen provided by ISPRS is used. The dataset is a collection of ALS (Airborne Laser Scanning) point clouds, consisting of 10 strips captured by the Leica ALS50 system, with an average overlap of around 30% between two adjacent strips. The ground resolution of the multispectral aerial image is 8cm, the size of each image is 7680pixel×13824pixel, and the inner and outer orientation elements of the image are provided. At present, the point cloud marked by the data is divided into 9 categories as the algorithm evaluation criteria, which contain rich geographical environment, urban environment and building types, which can fully verify the algorithm. The aerial image data and the corresponding airborne LiDAR point cloud data are provided in the Vaihingen dataset. The average point distance of the LiDAR point cloud data is 4 points/m2. The classification categories of LiDAR point cloud data have been marked in the reference data provided by the Vaihingen dataset. After extracting the relevant building features, the classifier can be directly trained to classify the LiDAR point cloud test data, and finally achieve accurate building points. extract. Using the PointNet network and the proposed algorithm for training and testing, the semantic labels of points in the dataset contain 9 categories (power lines, vehicles, low plants, impervious surfaces, guardrails, roofs, walls, shrubs, trees). The training dataset contains a total of 753,876 points, and the test dataset has 411,722 points.

Table 1 Types and numbers of point clouds in ISPRS Vaihingen dataset

Table 2 Vaihingen data parameters

In order to verify the establishment of the algorithm based on the multi-scale and multi-neighborhood features of PointNet proposed in the present invention, firstly, the same data are used for single-scale single feature (SS) (only contains xyz information), single-scale multi-feature (SM), multi-scale multi-feature (SM), multi-scale The features (MM) are verified separately. A single-scale single feature is a PointNet network that only contains xyz information; a single-scale multi-feature is only a mini-pointnet added to the original PointNet, and the neighborhood features when R=0.8 are calculated. Enter the mini network together with xyz, and at the same time combine the xyzRGB information after point cloud and image fusion as the input of the original PointNet network; multi-scale and multi-feature is the MSMF-PointNet algorithm proposed in this paper, respectively calculating R=0.8 and R= The point cloud features at 1.2 are input into the mini network together with xyz, and at the same time, the xyzRGB information is input into the PointNet network instead of the single feature of the point cloud. The classification accuracy is shown in Figure 4.

It can be seen from Figure 4 that the proposed PointNet-based multi-scale and multi-neighbor algorithm (MM) has the highest overall classification accuracy and the best classification effect, with an overall accuracy of 88.1%. From the above quantitative evaluation results, the fusion After spectral information and covariance feature (SM)-based, the point cloud classification accuracy of all ground objects has been greatly improved. Among them, the overall accuracy of SM is improved by 19.2 percentage points compared with SS; after fusing spectral information and covariance features (MM) of two scales, the accuracy is further improved, which is 8.4 percentage points higher than that of single-scale SM. Among them, the accuracy of impervious ground, trees and roofs has been improved the most, by 19.7 percentage points, 30 percentage points and 31.3 percentage points respectively. It can be seen that the attribute information of the point cloud of multi-scale and multi-neighborhood can be effectively enhanced, thereby realizing more accurate classification of various ground objects. The specific classification results are compared as shown in Figure 4, which are the final classification results of the true value, SS, SM, and MM in order. On the whole, the final classification result of MM is almost the same as the real value, especially on the roof, tree and impervious ground, the classification effect is particularly outstanding. It can be seen from SS that the ground objects are misclassified as roofs and trees in most cases. When local features based on spherical neighborhoods are added to SM, the classification accuracy of houses and trees is greatly improved. In SS, only terrain and elevation are simple. The ground is classified into trees, roofs and shrubs, while in SM, the classification is refined into roofs, trees, shrubs, impervious ground, fences and vehicles. Adding multi-scale MM is more accurate and refined than SM. In the upper area circled in the figure, it can be clearly seen that there is a bulge in the middle. In the field, it is a narrow groove with a large difference in height between the left and right sides. R When R = 0.8, the ground on the right side with a higher height of the narrow ditch is classified as a roof. When the local feature of R = 1.2 is added, the impervious ground is accurately classified.

The above are only the preferred embodiments of the present invention. It should be pointed out that for those skilled in the art, some changes and improvements can be made without departing from the overall concept of the present invention, and these should also be regarded as the present invention. scope of protection.

Claims (9)

1. a semantic segmentation method based on the multi-scale multi-feature algorithm of spherical neighborhood, is characterized in that: described method comprises: S1: Register the acquired point cloud data with the remote sensing image to generate point cloud data fused with RGB information; S2: Multi-scale neighborhood design and feature extraction for the point cloud data fused with RGB information: By studying the point cloud spatial index structure, the spherical neighborhood is selected to obtain the local neighborhood features of the point cloud data fused with RGB information, and By changing the radius of the spherical neighborhood, multi-scale point cloud features are extracted; the point cloud features include basic features and covariance-based multi-features; the basic features include xyz coordinate information and RGB information of the point cloud; The multi-feature of covariance includes 5-dimensional neighborhood features, that is, the covariance-based total variance, roughness, flatness, linearity and verticality information; S3: Combine the extracted basic features, 5-dimensional neighborhood features of at least two scales, and the xyz coordinate information of the point cloud into an improved PointNet-based model MSMF-PointNet for semantic segmentation, and output the classification result. 2. the semantic segmentation method based on the multi-scale multi-feature algorithm of spherical neighborhood as claimed in claim 1, is characterized in that: described PointNet-based improved model MSMF-PointNet comprises improved PointNet network and at least two Mini- pointnet network; the xyz coordinate information and RGB information of the point cloud are input into the improved PointNet network, and the improved PointNet network outputs 64-dimensional point features and 1024-dimensional global features; the 5-dimensional neighborhood of the at least two scales The combination of the feature and the xyz coordinate information of the point cloud is input to the Mini-pointnet network; the output of the Mini-PointNet is two 256-dimensional feature vectors, and the two parts of the output data are fully connected and input to the softmax classifier for classification. 3. the semantic segmentation method based on the multi-scale multi-feature algorithm of spherical neighborhood as claimed in claim 2, it is characterized in that: described improved PointNet network comprises six layers, is the first T-Net point sequentially from input to output Cloud rotation transform, first perceptron mlp, second T-Net, second perceptron mlp, third perceptron mlp and Max pooling network. 4. The semantic segmentation method based on the multi-scale multi-feature algorithm of spherical neighborhood as claimed in claim 2, it is characterized in that: described Mini-pointnet network comprises 4 layers, from input to output sequentially is T-Net point cloud rotation Transform, two-layer perceptron mlp and Max pooling network. 5. the semantic segmentation method based on the multi-scale multi-feature algorithm of spherical neighborhood as claimed in claim 1, it is characterized in that: the xyz coordinate information of described point cloud is according to the data recorded by GPS, INS and laser rangefinder of the system Solved; RGB information is acquired by the imaging device. 6. The semantic segmentation method based on the spherical neighborhood multi-scale and multi-feature algorithm according to claim 1, wherein the registration in the S1 mainly uses the function of "value extraction to points" in Arcigs The spectral information of different pixel points in the band is assigned to the point cloud data. 7. An electronic device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the execution of any of claims 1-6 Methods. 8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any of claims 1-6. 9. A computer program product comprising a computer program which, when executed by a processor, implements the method of any of claims 1-6.

Images