CN114694022A - Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method - Google Patents
Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method Download PDFInfo
- Publication number
- CN114694022A CN114694022A CN202210244408.8A CN202210244408A CN114694022A CN 114694022 A CN114694022 A CN 114694022A CN 202210244408 A CN202210244408 A CN 202210244408A CN 114694022 A CN114694022 A CN 114694022A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- neighborhood
- pointnet
- scale
- features
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 43
- 230000011218 segmentation Effects 0.000 title claims abstract description 28
- 238000004422 calculation algorithm Methods 0.000 title claims abstract description 23
- 238000000605 extraction Methods 0.000 claims description 10
- 230000003595 spectral effect Effects 0.000 claims description 9
- 230000009466 transformation Effects 0.000 claims description 9
- 239000013598 vector Substances 0.000 claims description 8
- 238000004590 computer program Methods 0.000 claims description 4
- 238000013461 design Methods 0.000 claims description 4
- 238000003384 imaging method Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 2
- 239000011159 matrix material Substances 0.000 description 10
- 230000000694 effects Effects 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 4
- 238000012360 testing method Methods 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000012952 Resampling Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000003909 pattern recognition Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011158 quantitative evaluation Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000010200 validation analysis Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Processing (AREA)
Abstract
A semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood, the method comprising: registering the acquired point cloud data with the remote sensing image to generate point cloud data fusing RGB information; selecting a spherical neighborhood to obtain local neighborhood characteristics of point cloud data fused with RGB information, and extracting multi-scale point cloud characteristics by changing the radius of the spherical neighborhood; and combining the extracted basic features, the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud, inputting the combination into an improved model MSMF-PointNet based on PointNet for semantic segmentation, and outputting a classification result. The method can obtain the classification precision far better than PointNet in the outdoor scene point cloud data obtained by airborne LiDAR scanning, and has the advantages of better classification of building facades, fences and the like due to the addition of characteristics such as linearity and verticality, better classification results of trees and shrubs due to the addition of roughness and total variance, better classification results of roofs and impervious grounds due to the addition of flatness.
Description
Technical Field
The invention belongs to the technical field of remote sensing and photogrammetry, and particularly relates to a semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood.
Background
Deep learning is a new technology capable of automatically learning and extracting advanced features of input data through a deep network structure, and is a leading-edge technology which is most influential and fastest developed in current pattern recognition, computer vision and data analysis. Before being applied to 3D data, deep learning has become an effective power for various tasks in 2D computer vision and image processing, and particularly, in 2012 AlexNe has captured the second scores of ten and a few percent higher by applying Convolutional Neural Networks (CNNs) to image recognition competitions of ImageNet, and then the deep Neural network structure mainly based on CNNs has made a major breakthrough in the fields of image classification, segmentation, recognition, and the like. However, due to the characteristics of high density, mass and no structure of the three-dimensional laser point cloud, the traditional deep learning method cannot be directly applied to the segmentation of the three-dimensional point cloud.
A large number of scholars do a large amount of work on deep learning of three-dimensional point cloud data, and in order to apply the existing neural network structure to the three-dimensional laser point cloud data, data preprocessing is required to be performed on the point cloud data before the point cloud data is input into a neural network, three methods are commonly used at present, (1) the 3D point cloud data is projected into a multi-view 2D image and then a traditional convolutional neural network is used; (2) converting the point cloud into grid voxels and then using a 3D convolutional neural network; (3) the point cloud is converted into a Graph (Graph) structure and then a neural network is convolved with a 3D Graph.
However, the data information is lost in the preprocessing process, and aiming at the defect that the data information is lost in 2017, a pioneer study work is published by a student Qi of Stanford university and the like, a deep learning model PointNet is provided and can be directly applied to three-dimensional point cloud data, so that the semantic segmentation precision is further improved. However, the PointNet network structure mainly extracts global features among point clouds, omits the extraction of associated local features between points in the point clouds, and often causes the problems of insufficient segmentation precision, poor object detail segmentation effect and the like due to insufficient extraction capability of the local features.
On one hand, due to the high density, the mass and the non-structural characteristics of the three-dimensional laser point cloud, the method has important theoretical value in researching the rapid and effective three-dimensional scene semantic segmentation algorithm. On the other hand, due to the complexity of a real natural scene and the phenomena of overlapping, shielding and the like among three-dimensional targets, research is combined with technologies in other fields, and the high-robustness, automatic and intelligent method for segmenting the target semantics of the complex three-dimensional scene is provided, so that the method has important practical significance for further researching point cloud semantics segmentation and application of the point cloud semantics in various fields.
Disclosure of Invention
In order to solve the problems, a semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood is provided.
The object of the invention is achieved in the following way:
a semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood, the method comprising:
s1: registering the acquired point cloud data with the remote sensing image to generate point cloud data fusing RGB information;
s2, carrying out multi-scale neighborhood design and feature extraction on the point cloud data fused with the RGB information: the method comprises the steps of obtaining local neighborhood characteristics of point cloud data fusing RGB information by researching a point cloud space index structure and selecting a spherical neighborhood, and extracting multi-scale point cloud characteristics by changing the radius of the spherical neighborhood; the point cloud features comprise basic features and covariance-based multi-features; the basic features comprise xyz coordinate information and RGB information of the point cloud; the covariance-based multi-feature comprises a 5-dimensional neighborhood feature, namely covariance-based full-variance, roughness, flatness, linearity and verticality information;
s3: and combining the extracted basic features, the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud, inputting the combination into an improved model MSMF-PointNet based on PointNet for semantic segmentation, and outputting a classification result.
The improved model MSMF-PointNet based on PointNet comprises an improved PointNet network and at least two Mini-PointNet networks; inputting xyz coordinate information and RGB information of the point cloud into an improved PointNet network, wherein 64-dimensional point features and 1024-dimensional global features are output by the improved PointNet network; combining the 5-dimensional neighborhood characteristics of at least two scales with the xyz coordinate information of the point cloud and inputting the combination into a Mini-pointet network; the Mini-PointNet outputs two 256-dimensional characteristic vectors, the two parts of output data are fully connected, and the data are input into a softmax classifier for classification.
The improved PointNet network comprises six layers, namely a first T-Net point cloud rotation transformation, a first sensor mlp, a second T-Net point cloud rotation transformation, a second sensor mlp, a third sensor mlp and a Max pooling network from input to output.
The Mini-pointet network comprises 4 layers, namely T-Net point cloud rotation transformation, two layers of perceptrons mlp and a Max poiling network from input to output.
The xyz coordinate information of the point cloud is calculated according to the data recorded by the GPS, the INS and the laser range finder of the system; the RGB information is acquired by the imaging device.
In the step S1, the registration is performed by assigning spectral information of different pixel points of each band of the image to the point cloud data mainly through the function of "value to point" in arcs.
An electronic device, comprising:
at least one processor;
and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method described above.
A non-transitory computer readable storage medium storing computer instructions, wherein,
the computer instructions are for causing the computer to perform the method described above.
A computer program product comprising a computer program which, when executed by a processor, implements the method described above.
The invention has the beneficial effects that: (1) the method can obtain the classification precision far better than PointNet in the outdoor scene point cloud data obtained by airborne LiDAR scanning, and has the advantages of better classification of building facades, fences and the like due to the addition of characteristics such as linearity and verticality, better classification results of trees and shrubs due to the addition of roughness and total variance, better classification results of roofs and impervious grounds due to the addition of flatness.
(2) The method of the invention adds spectral information and other geometric characteristics, and trains based on deep learning, thereby effectively making up the defects of the spatial geometric characteristics of the point cloud, improving the point cloud classification precision, and having higher classification precision for roofs, impervious grounds and trees.
Drawings
Fig. 1 is a schematic diagram of index creation based on a virtual regular grid.
Fig. 2 is a comparison graph of classification effect of different radii.
Figure 3 is a MSMF-PointNet network model architecture.
FIG. 4 shows the classification accuracy of various types of ground features and the overall classification accuracy at different scales.
FIG. 5 is a flow chart of a semantic segmentation model of a multi-scale multi-feature algorithm based on spherical neighborhood.
Detailed Description
The present invention will be described in further detail with reference to fig. 1 to 5 and the following detailed description.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure as claimed. Unless otherwise defined, all technical and scientific terms used herein have the same technical meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
A semantic segmentation method of a multi-scale and multi-feature algorithm based on a spherical neighborhood, the method comprising:
s1: registering the acquired point cloud data with the remote sensing image to generate point cloud data fusing RGB information;
s2, performing multi-scale neighborhood design and feature extraction on the point cloud data fused with the RGB information: the method comprises the steps of obtaining local neighborhood characteristics of point cloud data fusing RGB information by researching a point cloud space index structure and selecting a spherical neighborhood, and extracting multi-scale point cloud characteristics by changing the radius of the spherical neighborhood; the point cloud features comprise basic features and multi-features based on covariance; the basic features comprise xyz coordinate information and RGB information of the point cloud; the covariance-based multi-feature comprises a 5-dimensional neighborhood feature, namely covariance-based full variance, roughness, flatness, linearity and verticality information;
s3: and combining the extracted basic features, the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud, inputting the combination into an improved model MSMF-PointNet based on PointNet for semantic segmentation, and outputting a classification result.
As shown in fig. 3, the improved model MSMF-PointNet based on PointNet comprises an improved PointNet network and at least two Mini-PointNet networks; inputting xyz coordinate information and RGB information of the point cloud into an improved PointNet network, wherein 64-dimensional point features and 1024-dimensional global features are output by the improved PointNet network; combining the 5-dimensional neighborhood characteristics of at least two scales with the xyz coordinate information of the point cloud and inputting the combination into a Mini-pointet network; the Mini-PointNet outputs two 256-dimensional characteristic vectors, the two parts of output data are fully connected, and the data are input into a softmax classifier for classification.
The improved PointNet network comprises six layers, namely a T-Net point cloud rotation transformation, a first sensor mlp, a T-Net, a second sensor mlp, a third sensor mlp and a Max Pooling network from input to output in sequence.
The Mini-pointet network comprises 4 layers, namely T-Net point cloud rotation transformation, two layers of perceptrons mlp and a Max poiling network from input to output.
Further, a point cloud feature extraction method based on spherical neighborhood is researched:
in order to improve the efficiency of query and retrieval of point cloud data, a spatial index of the point cloud data needs to be constructed, and common indexing methods include a quadtree, an octree, a k-d tree and the like. Because the k-d tree indexing efficiency is high, the k-d tree indexing method is widely applied to point cloud data, different space indexes are adopted according to specific data processing requirements, and for the subsequent filtering algorithm design, the k-d tree is selected to construct the space index of the point cloud, and space division and neighbor search are carried out.
After the K-d tree is established, a neighborhood point query mode needs to be further selected, the scale difference between different objects in an outdoor environment is large, and the description capacity of the features on the different objects is determined by the neighborhood selection mode in the feature extraction process. The single-point classification of the point cloud depends on the extraction of local features, and the local features are extracted from a neighborhood point set of the selected points, so the single-point classification is closely related to the selected local neighborhood region.
Radius query is to give a target point and a threshold of a query distance (with the target point as a circle center and the query distance as a radius R), find out all data (data within the radius) whose distance from the query point is smaller than the threshold from the data set, and there are three common local neighborhood choices: the schematic diagram is shown in fig. 1, wherein (a) is a spherical radius neighborhood, (b) is a columnar neighborhood, and (c) is a K neighbor neighborhood.
Due to the spatial anisotropy of the three-dimensional urban scene, the classification effect under the spherical neighborhood is good. Compared with the K nearest neighbor method, the spherical neighborhood corresponds to a fixed part in the space and is relatively low influenced by the density of the point cloud, so the method selects the spherical neighborhood method.
And obtaining a multi-scale point cloud characteristic by changing the size of the spherical neighborhood radius R.
The selection of the scale directly affects the classification accuracy of the point cloud, so that a proper scale, namely the radius value R in the spherical neighborhood method, needs to be selected according to the ground features in the scene. Too large radius, too many points contained in the spherical neighborhood can greatly increase the calculation time and reduce the efficiency. Generally, the minimum value of the radius is greater than the average density of the point cloud and increases upwards regularly, in order to select the most suitable radius parameter and verify that the segmentation effect of the multi-scale fused point cloud features is better, a part of the Vaihingen dataset is selected, point cloud features calculated by the radius parameter R being 0.4m,0.8m,1.2m,2.0m and four combinations of the point cloud features 0.4+0.8, 0.8+1.2, 0.8+2.0 and 1.2+2.0 are selected and respectively input into the PointNet network, the rest are the same except that the input is different, and a histogram of the classification effect is shown in fig. 2: as shown by comparing and analyzing the histogram in fig. 2, the combined classification accuracy of the multi-scale R-0.8 +1.2 is the highest, and the accuracy of the single-scale R-0.8 is the highest, and the network has not been upgraded only to determine the most suitable radius parameter, so that there is no specific precision, but only the precision is relatively high or low. Therefore, the invention considers comprehensively, and the radius R of the final spherical neighborhood is 0.8m and 1.2 m.
Selection of basic and common features of point clouds and covariance-based multi-features
The basic characteristics of the point cloud are xyz coordinate information and RGB information of the point cloud, xyz can be calculated according to data recorded by a GPS, an INS and a laser range finder of the system, the RGB information is acquired by an imaging device, and the spectral information is beneficial to distinguishing vegetation and other ground features, so that in order to improve the classification accuracy, airborne LiDAR point cloud and multispectral aerial images are generally fused to generate point cloud data with spectral information so as to realize spectral information supplement of the point cloud data.
Under the condition that the inside and outside orientation elements of the aerial photography film are known, the three-dimensional coordinates of the LiDAR point cloud are substituted into a collinear condition equation, the pixel position of the corresponding three-dimensional point on the image can be calculated, and then the gray values of Near Infrared (NIR), red (R) and green (G) channels are obtained through resampling. The collinearity condition equation can be expressed as:
wherein f is an image focal length, XYZ is a three-dimensional coordinate of a ground point, and (X)S,YS,ZS) Three line elements which are external orientation elements (a)1,b1,c1,a2,b2,c2,a3,b3,c3) Is a rotation matrix parameter calculated from three angular elements in the exterior orientation elements. And after calculating spectral information (NIR, R and G three-channel gray values) corresponding to each laser foot point, combining the three-dimensional coordinates and the spectral information of the point cloud to obtain enhanced point cloud data which is used as input data of subsequent point cloud classification, and fusing the point cloud and the image.
The covariance characteristic is a common characteristic in the point cloud, can represent the shape of an object or the distribution state of local point cloud, and has important application in point cloud data processing. For three-dimensional point cloud data, D ═ P can be usediN represents the point cloud, where N is the number of points in the point cloud D. To this end, first the mean and covariance matrix of the current point cloud are calculated:
wherein, the formula of the unbiased estimation of C is:
secondly, since the covariance matrix C is a positive definite matrix, the characteristic value of the matrix C is known from the characteristics of the symmetric positive definite matrix: lambda [ alpha ]1≥λ2≥λ3Not less than 0, and the corresponding feature vector e1,e2,e3Perpendicular to each other, an orthogonal coordinate system can be formed, i.e. C can be expressed as:
and according to the size condition of the characteristic value, the morphological characteristics of the local neighborhood point cloud can be obtained. At λ1>>λ2≈λ3And then, the local neighborhood point clouds are distributed in a linear way. At λ1≈λ2>>λ3When the point cloud is distributed in a surface shape. Lambda [ alpha ]1>>λ2≈λ3And the point clouds are distributed in a three-dimensional off-line manner. Using a combined linearity LλFlatness PλAnd dispersion SλTo represent one-, two-, and three-dimensional features:
Lλ＝(λ1-λ2)/λ1 (6)
Pλ＝(λ2-λ3)/λ1 (7)
Sλ＝λ3/λ1 (8)
the sum of the three is 1. And (4) obtaining a covariance matrix constructed by the neighborhood of the selected point according to the formula (3), and further obtaining an eigenvalue of the covariance matrix and a corresponding eigenvector.
The invention selects the whole range difference besides the linearity and the flatness:
perpendicularity:
Vλ＝1-|Z·N| (10)
wherein Z is a unit vector in the vertical direction, and N is a normal vector of the point.
And intercepting a Vaihingen experimental data set, displaying each neighborhood characteristic when R is 0.8, visualizing the estimation result of each characteristic value, clearly seeing that the characteristic values of different ground objects are obviously different, and beneficially distinguishing.
The point cloud segmentation method based on the improved PointNet is researched:
the invention provides an MSMF-PointNet terrain classification segmentation algorithm fusing multi-scale and multi-neighborhood characteristics based on an improved PointNet network, overcomes the defects of insufficient utilization of point-to-point cloud characteristics and lack of local characteristics of the original PointNet network, draws the network parameter settings of PointNet and PointNet + +, and simultaneously sets corresponding R, O and multi-scale local characteristics (xyz, RGB, R is 0.8 and 1.2) based on 16-dimensional spherical neighborhoodλ,Pλ,Lλ,Vλ) And replacing the original single-point xyz information as a new data source for training classification of the classifier.
The specific improvement idea is as follows: aiming at the problem of the increase of the dimensionality of the feature space of the fused point cloud, the number of channels is increased by adjusting the dimensionality of an input transformation matrix, so that the matrix is changed from the original processing of three-dimensional feature vectors into the processing of six-dimensional and eight-dimensional feature vectors after fusion; aiming at the increase of data volume caused by the expansion of the feature space of the fused point cloud data, the depth features of the point cloud are fully extracted by deepening the number of network layers of an MLP layer; aiming at the problem of lack of local neighborhood characteristics, point cloud characteristics calculated by spherical neighborhood are used as input, two mini-PointNet characteristic extraction networks are built, and multi-neighborhood point cloud characteristics with different scales are extracted. The MSMF-PointNet network model architecture is shown in FIG. 3.
Experimental validation and analysis
And (3) measuring area data: the city test data set of Vaihingen, germany, provided by isps was used. The dataset is a collection of ALS (aerodynamic Laser scanning) point clouds, consisting of 10 strips captured by the Leica ALS50 system, with an average overlap between two adjacent strips of around 30%. The ground resolution of the multispectral aerial image is 8cm, the size of each image is 7680 pixels × 13824 pixels, and the medial and lateral orientation elements of the image are provided. At present, the point cloud marked by the data is divided into 9 categories as algorithm evaluation standards, which contain abundant geographic environments, urban environments and building types, and can fully verify the algorithm. Aerial image data and airborne LiDAR point cloud data corresponding to the aerial image data are provided in the Vaihingen dataset, and the average point distance of the LiDAR point cloud data is 4points/m 2. The classification categories of LiDAR point cloud data are labeled in reference data provided by the Vaihingen data set, after relevant building features are extracted, a classifier can be directly trained to classify LiDAR point cloud test data, and finally accurate extraction of building points is achieved. Training and testing was performed using the PointNet network and proposed algorithm, the semantic labels of the data concentration points contained 9 classes (power line, vehicle, low plant, impervious surface, fence, roof, wall, shrub, tree). The training data set contained 753,876 points in total, and the test data set contained 411,722 points.
TABLE 1 ISPRS Vaihingen dataset Point cloud categories and numbers
TABLE 2 Vaihingen data parameters
In order to verify the establishment of the multi-scale and multi-neighborhood feature algorithm based on PointNet, the single-scale single feature (SS) (only containing xyz information), the single-scale multi-feature (SM) and the multi-scale multi-feature (MM) are verified respectively by using the same data, and the single-scale single feature is a PointNet network only containing the xyz information; the single-scale multi-feature method comprises the steps that only one mini-PointNet is added on the basis of the original PointNet, the neighborhood feature when R is 0.8 is calculated and is input into a mini network together with xyz, and meanwhile point cloud and image fused xyz RGB information are combined to serve as input of the original PointNet network; the multi-scale multi-feature is the MSMF-PointNet algorithm proposed herein, which calculates the point cloud features when R is 0.8 and R is 1.2, respectively, and outputs them together with xyz into the mini network, and at the same time, outputs the xyz RGB information into the PointNet network instead of the point cloud single feature. The classification accuracy is shown in fig. 4.
As can be seen from fig. 4, the overall classification accuracy of the proposed PointNet-based multi-scale multi-neighborhood algorithm (MM) is the highest, the classification effect is the best, the overall accuracy reaches 88.1%, and from the quantitative evaluation result, the point cloud classification accuracy of all terrain features is greatly improved after the spectral information and the covariance-based features (SM) are fused. Wherein, the overall precision of SM is improved by 19.2 percentage points compared with SS; after the spectrum information and the covariance characteristics (MM) of two scales are fused, the precision is further improved, and compared with the SM of a single scale, the precision is improved by 8.4 percentage points. Wherein the precision of the impervious ground, the tree and the roof is improved most obviously by 19.7 percent, 30 percent and 31.3 percent respectively. Therefore, the attribute information of the point cloud of the multi-scale and multi-neighborhood can be effectively enhanced, and therefore, the more accurate classification of various ground object targets is realized. The specific classification result pair is a final classification result graph of true value, SS, SM and MM in sequence as shown in FIG. 4. On the whole, the final classification result of the MM is almost the same as the true value, and the classification effect is particularly prominent particularly on roofs, trees and impervious ground. It can be seen from SS that the ground features are classified into roofs and trees by mistake, when local features based on spherical neighborhood are added into SM, the classification precision of houses and trees is greatly improved, SS is simply classified into trees, roofs and shrubs according to terrain and elevation, and SM is classified into roofs, trees, shrubs, impervious ground, fences and cars by refining classification. The MM with multiple scales is more accurate and finer than SM, as the upper area circled on the graph, a bulge is obviously seen in the middle, a narrow ditch with larger height difference at the left side and the right side is formed on the spot, when R is equal to 0.8, the right-side ground with the higher height of the narrow ditch is classified as a roof, and when the local characteristic that R is equal to 1.2 is added, the waterproof ground is accurately classified.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various changes and modifications can be made without departing from the overall concept of the present invention, and these should also be considered as the protection scope of the present invention.
Claims (9)
1. A semantic segmentation method of a multi-scale and multi-feature algorithm based on spherical neighborhood is characterized by comprising the following steps: the method comprises the following steps:
s1: registering the acquired point cloud data with the remote sensing image to generate point cloud data fusing RGB information;
s2, performing multi-scale neighborhood design and feature extraction on the point cloud data fused with the RGB information: the method comprises the steps of obtaining local neighborhood characteristics of point cloud data fusing RGB information by researching a point cloud space index structure and selecting a spherical neighborhood, and extracting multi-scale point cloud characteristics by changing the radius of the spherical neighborhood; the point cloud features comprise basic features and multi-features based on covariance; the basic features comprise xyz coordinate information and RGB information of the point cloud; the covariance-based multi-feature comprises a 5-dimensional neighborhood feature, namely covariance-based full variance, roughness, flatness, linearity and verticality information;
s3: and combining the extracted basic features, the 5-dimensional neighborhood features of at least two scales and the xyz coordinate information of the point cloud, inputting the combination into an improved model MSMF-PointNet based on PointNet for semantic segmentation, and outputting a classification result.
2. The semantic segmentation method based on the spherical neighborhood multi-scale multi-feature algorithm of claim 1, characterized in that: the improved model MSMF-PointNet based on PointNet comprises an improved PointNet network and at least two Mini-PointNet networks; inputting xyz coordinate information and RGB information of the point cloud into an improved PointNet network, wherein 64-dimensional point features and 1024-dimensional global features are output by the improved PointNet network; the 5-dimensional neighborhood characteristics of at least two scales and the xyz coordinate information of the point cloud are combined and input into a Mini-pointet network; the Mini-PointNet outputs two 256-dimensional feature vectors, and the output data of the two parts are fully connected and input into a softmax classifier for classification.
3. The semantic segmentation method based on the spherical neighborhood multi-scale multi-feature algorithm of claim 2, characterized in that: the improved PointNet network comprises six layers, namely a first T-Net point cloud rotation transformation, a first sensor mlp, a second T-Net point cloud rotation transformation, a second sensor mlp, a third sensor mlp and a Max pooling network from input to output.
4. The semantic segmentation method based on the spherical neighborhood multi-scale multi-feature algorithm of claim 2, characterized in that: the Mini-pointenet network comprises 4 layers, namely T-Net point cloud rotation transformation, two layers of sensors mlp and a Max poiling network from input to output.
5. The semantic segmentation method based on the spherical neighborhood multi-scale multi-feature algorithm of claim 1, characterized in that: the xyz coordinate information of the point cloud is calculated according to the GPS, INS and the data recorded by the laser range finder of the system; the RGB information is acquired by the imaging device.
6. The semantic segmentation method based on the spherical neighborhood multi-scale multi-feature algorithm of claim 1, characterized in that: in the S1, the registration mainly assigns the spectral information of different pixel points of each wave band of the image to the point cloud data through the function of 'extracting values to points' in Arcigs.
7. An electronic device, comprising:
at least one processor;
and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
8. A non-transitory computer readable storage medium storing computer instructions, wherein,
the computer instructions are for causing the computer to perform the method of any one of claims 1-6.
9. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210244408.8A CN114694022A (en) | 2022-03-11 | 2022-03-11 | Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210244408.8A CN114694022A (en) | 2022-03-11 | 2022-03-11 | Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114694022A true CN114694022A (en) | 2022-07-01 |
Family
ID=82138380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210244408.8A Pending CN114694022A (en) | 2022-03-11 | 2022-03-11 | Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114694022A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116977572A (en) * | 2023-09-15 | 2023-10-31 | 南京信息工程大学 | Building elevation structure extraction method for multi-scale dynamic graph convolution |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241871A (en) * | 2017-12-27 | 2018-07-03 | 华北水利水电大学 | Laser point cloud and visual fusion data classification method based on multiple features |
-
2022
- 2022-03-11 CN CN202210244408.8A patent/CN114694022A/en active Pending
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108241871A (en) * | 2017-12-27 | 2018-07-03 | 华北水利水电大学 | Laser point cloud and visual fusion data classification method based on multiple features |
Non-Patent Citations (1)
| Title |
|---|
| 费美琪: "基于PointNet的机载LiDAR点云语义分割方法研究", 《万方学位论文》, 19 December 2021 (2021-12-19) * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116977572A (en) * | 2023-09-15 | 2023-10-31 | 南京信息工程大学 | Building elevation structure extraction method for multi-scale dynamic graph convolution |
| CN116977572B (en) * | 2023-09-15 | 2023-12-08 | 南京信息工程大学 | Building elevation structure extraction method for multi-scale dynamic graph convolution |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Xia et al. | Geometric primitives in LiDAR point clouds: A review | |
| CN109410321B (en) | Three-dimensional reconstruction method based on convolutional neural network | |
| CN110415342A (en) | A kind of three-dimensional point cloud reconstructing device and method based on more merge sensors | |
| CN109883401B (en) | Method and system for measuring visual field of city mountain watching | |
| CN103337052B (en) | Automatic geometric correcting method towards wide cut remote sensing image | |
| CN110866079A (en) | Intelligent scenic spot real scene semantic map generating and auxiliary positioning method | |
| CN113989797A (en) | Three-dimensional dynamic target detection method and device based on voxel point cloud fusion | |
| Hu et al. | Research on a single-tree point cloud segmentation method based on UAV tilt photography and deep learning algorithm | |
| Karsli et al. | Automatic building extraction from very high-resolution image and LiDAR data with SVM algorithm | |
| CN113379738A (en) | Method and system for detecting and positioning epidemic trees based on images | |
| CN114120067A (en) | Object identification method, device, equipment and medium | |
| CN111487643A (en) | Building detection method based on laser radar point cloud and near-infrared image | |
| CN112906719A (en) | Standing tree factor measuring method based on consumption-level depth camera | |
| CN113963259A (en) | Street view ground object multi-dimensional extraction method and system based on point cloud data | |
| CN112767459A (en) | Unmanned aerial vehicle laser point cloud and sequence image registration method based on 2D-3D conversion | |
| Yao et al. | Automated detection of 3D individual trees along urban road corridors by mobile laser scanning systems | |
| CN118411507A (en) | Semantic map construction method and system for scene with dynamic target | |
| CN114694022A (en) | Spherical neighborhood based multi-scale multi-feature algorithm semantic segmentation method | |
| Liu et al. | Comparison of 2D image models in segmentation performance for 3D laser point clouds | |
| CN116994029A (en) | Fusion classification method and system for multi-source data | |
| CN112146647B (en) | Binocular vision positioning method and chip for ground texture | |
| CN113362458B (en) | Three-dimensional model interpretation method for simulating multi-view imaging, terminal and storage medium | |
| Cui et al. | A Review of Indoor Automation Modeling Based on Light Detection and Ranging Point Clouds. | |
| Bai et al. | Application of unmanned aerial vehicle multi-vision image 3D modeling in geological disasters | |
| Wen et al. | Comparing Deep Learning and MCWST Approaches for Individual Tree Crown Segmentation |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |