CN115830375A - Point cloud classification method and device - Google Patents
Point cloud classification method and device Download PDFInfo
- Publication number
- CN115830375A CN115830375A CN202211494418.3A CN202211494418A CN115830375A CN 115830375 A CN115830375 A CN 115830375A CN 202211494418 A CN202211494418 A CN 202211494418A CN 115830375 A CN115830375 A CN 115830375A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- features
- frequency
- cloud data
- local
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 94
- 238000012545 processing Methods 0.000 claims abstract description 109
- 238000000605 extraction Methods 0.000 claims abstract description 45
- 239000013598 vector Substances 0.000 claims abstract description 33
- 238000005070 sampling Methods 0.000 claims description 63
- 238000011176 pooling Methods 0.000 claims description 46
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000012549 training Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010606 normalization Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002776 aggregation Effects 0.000 description 1
- 238000004220 aggregation Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Landscapes
- Image Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a point cloud classification method and a point cloud classification device, wherein the method comprises the following steps: extracting the characteristics of the original point cloud data to obtain the local characteristics of the original point cloud data; inputting the local features into the first processing branch and the second processing branch respectively to obtain high-frequency features and low-frequency features of the local features; and processing the high-frequency characteristic and the low-frequency characteristic to obtain a target characteristic, and inputting the target characteristic into a classifier to obtain a classification result of the original point cloud data. According to the point cloud classification method and device, after the local features of the original point cloud data are obtained, the obtained local features are subjected to high-frequency feature extraction and low-frequency feature extraction by adopting two parallel branches, and the obtained target features after the high-frequency features and the low-frequency features are spliced are used for point cloud classification, so that the calculation complexity is reduced. When the low-frequency features are extracted, the long-distance dependency relationship between point cloud coordinates can be established based on vector attention processing, and the complexity of subsequent classification is further reduced.
Description
Technical Field
The invention relates to the technical field of artificial intelligence, in particular to a point cloud classification method and device.
Background
The complexity of the acquired point cloud image is higher and higher due to the rapid development of the existing data acquisition hardware and computational power. Three-dimensional point cloud data containing rich spatial information is applied to various scenes of computer vision, such as automatic driving, virtual reality, three-dimensional modeling and robot operation.
In the existing method, the efficiency of point cloud classification is lower and lower due to the fact that the complexity of a model for processing point cloud is higher and higher.
Disclosure of Invention
The invention provides a point cloud classification method and a point cloud classification device, which are used for solving the technical problems that in the prior art, the complexity of a model for processing point clouds is higher and higher, and the efficiency of point cloud classification is lower and lower.
The invention provides a point cloud classification method, which comprises the following steps:
extracting the characteristics of original point cloud data to obtain the local characteristics of the original point cloud data;
inputting the local features into a first processing branch to perform high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch to perform low-frequency feature extraction to obtain low-frequency features of the local features;
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and inputting the target features into a classifier to obtain a classification result of the original point cloud data, wherein the classifier is obtained by training based on a point cloud data sample and a class label corresponding to the point cloud data sample.
According to the point cloud classification method provided by the invention, the local features are input into a second processing branch for low-frequency feature extraction, and the low-frequency features of the local features are obtained, and the method comprises the following steps:
carrying out average pooling on the local features to obtain feature vectors after the average pooling;
performing hierarchical dichotomy clustering on the feature vectors to obtain a plurality of feature clusters of the feature vectors;
and respectively determining the feature relation in each feature cluster based on a vector attention algorithm, and combining the obtained feature relations to obtain the low-frequency features of the local features.
According to the point cloud classification method provided by the invention, the local feature is input into a first processing branch for high-frequency feature extraction, so that the high-frequency feature of the local feature is obtained, and the method comprises the following steps:
performing maximum pooling on the local features to obtain local features subjected to maximum pooling;
and carrying out residual error processing on the local features subjected to the maximum pooling processing to obtain the high-frequency features of the local features.
According to the point cloud classification method provided by the invention, the step of extracting the characteristics of the original point cloud data to obtain the local characteristics of the original point cloud data comprises the following steps:
the method comprises the steps of performing down-sampling on original point cloud data, obtaining a sampling central point of the original point cloud data, and determining a neighbor point corresponding to the sampling central point based on a K neighbor algorithm;
obtaining the relative position coordinates of the adjacent point and the sampling central point based on the coordinates of the adjacent point and the coordinates of the sampling central point;
splicing the relative position coordinates and the sampling center point according to coordinate dimensions to obtain spliced six-dimensional coordinates;
and mapping the six-dimensional coordinates to a high-dimensional space based on a multilayer perceptron to obtain local features of the original point cloud data.
According to the point cloud classification method provided by the invention, the down-sampling of the original point cloud data is carried out to obtain the sampling center point of the original point cloud data, and the method comprises the following steps:
normalizing the original point cloud data into a unit sphere, and uniformly sampling the normalized original point cloud data to obtain point cloud samples with a preset number;
and based on a farthest point sampling method, carrying out downsampling on the point cloud sample to obtain a sampling central point of the point cloud sample.
According to the point cloud classification method provided by the invention, the high-frequency feature and the low-frequency feature are spliced, and the feature obtained after splicing is subjected to dimension-increasing processing to obtain the target feature of the original point cloud data, and the method comprises the following steps:
splicing the high-frequency characteristic and the low-frequency characteristic to obtain the characteristic of a sampling central point;
and performing dimension increasing processing on the characteristics of the sampling central point for preset times, and taking the characteristics obtained after the dimension increasing processing as target characteristics of the original point cloud data.
The invention also provides a point cloud classification device, comprising:
the characteristic extraction module is used for extracting the characteristics of the original point cloud data to obtain the local characteristics of the original point cloud data;
the processing module is used for inputting the local features into a first processing branch for high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch for low-frequency feature extraction to obtain low-frequency features of the local features;
the splicing module is used for splicing the high-frequency features and the low-frequency features and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and the classification module is used for inputting the target characteristics into a classifier to obtain a classification result of the original point cloud data, and the classifier is obtained by training based on the point cloud data sample and the corresponding class label.
The invention also provides an electronic device, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor executes the computer program to realize any one of the point cloud classification methods.
The present invention also provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements any of the point cloud classification methods described above.
The invention also provides a computer program product comprising a computer program which, when executed by a processor, implements a point cloud classification method as described in any one of the above.
According to the point cloud classification method and device, after the local features of the original point cloud data are obtained, the obtained local features are subjected to high-frequency feature extraction and low-frequency feature extraction by adopting two parallel branches, and the obtained target features after the high-frequency features and the low-frequency features are spliced are used for point cloud classification, so that the calculation complexity is reduced. Meanwhile, when the low-frequency features are extracted, the long-distance dependency relationship between the point cloud coordinates can be established based on vector attention processing, the complexity of subsequent classification is further reduced, and the recognition rate of point cloud classification is improved.
Drawings
In order to more clearly illustrate the technical solutions of the present invention or the prior art, the drawings needed to be used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on the drawings without creative efforts.
FIG. 1 is a schematic flow chart of a point cloud classification method provided by the present invention;
FIG. 2 is a schematic flow chart of a point cloud classification method according to the present invention;
FIG. 3 is a schematic view of a partial feature processing flow provided by the present invention;
FIG. 4 is a schematic structural diagram of a point cloud classifying device according to the present invention;
fig. 5 is a schematic structural diagram of an electronic device provided in the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is obvious that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
With the rapid development of data acquisition hardware and computational power, 3D data containing rich spatial information is applied to various scenes of computer vision, such as automatic driving, virtual reality, three-dimensional modeling, and robotic operations. Representations of 3D data include depth maps, point clouds, meshes, and voxels. The point cloud data is a group of points sampled from the surface of the object, and can retain the most original information of the environment and the object in the three-dimensional space, such as position coordinates, colors, reflecting surface intensity and the like. Compared with dense regular 2D image data, the 3D point cloud data is more sparse and has the characteristics of storage disorder and distribution nonuniformity, so that the classical convolutional neural network cannot directly process the point cloud. Meanwhile, the three-dimensional point cloud of the real scene has many noises (background and occlusion), which brings many challenges to the point cloud classification task.
Point cloud classification methods based on deep learning in related methods are classified into a multi-view geometric method, a three-dimensional voxel method and a point-based method according to different point cloud data processing modes. The multi-view geometry method and the three-dimensional voxel method firstly need to project or grid point clouds into a regular and ordered grid, such as a two-dimensional image and a three-dimensional voxel, and then use a two-dimensional or three-dimensional convolution neural network for learning, and the data conversion can cause information loss to a certain degree. However, as the complexity of the acquired point cloud image is higher and higher, the efficiency of the point cloud classification by the correlation method is lower and lower.
Aiming at defects in a related method, the invention provides a point cloud classification method, and fig. 1 is a flow schematic diagram of the point cloud classification method provided by the invention. Referring to fig. 1, the point cloud classification method provided by the present invention may include:
The executing body of the point cloud classification method provided by the invention can be electronic equipment, a component in the electronic equipment, an integrated circuit or a chip. The electronic device may be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palmtop computer, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), and the like, and the present invention is not limited in particular.
The technical scheme of the invention is explained in detail below by taking a computer as an example to execute the point cloud classification method provided by the invention.
In step 110, original point cloud data is obtained, and feature extraction is performed on the obtained original point cloud data. The raw point cloud data is point cloud data containing objects for subsequent classification processes. And extracting the characteristics of the obtained original point cloud data to obtain the local characteristics of the original point cloud data.
The MLP (multi layer Perceptron) performs feature extraction on the acquired original point cloud data based on a multi-layer Perceptron to obtain local features of the original point cloud.
MLP＝ReLU(BatchNorm(Conv2d(X)))；
Wherein, X is the input original point cloud data, reLU is the active layer, batchNorm is the normalization process, and Conv2d is the convolution process.
Optionally, the acquired original point cloud data may be preprocessed before the original point cloud data is acquired and feature extraction is performed on the original point cloud data. The preprocessing comprises operations of random scale transformation, displacement, sample data mixing enhancement and the like.
In step 120, after the local feature is acquired, the acquired local feature is input into a first processing branch. And based on the first processing branch, the obtained local features are processed in parallel by adopting the first processing branch and the second processing branch. And performing high-frequency feature extraction on the local features to obtain the high-frequency features of the local features output by the first processing branch. And inputting the acquired local features into a second processing branch. And based on the second processing branch, carrying out low-frequency feature extraction on the local features to obtain the low-frequency features of the local features output by the second processing branch.
In particular, the first processing branch may be composed of a max pooling layer MaxPool and a residual MLP (ResMLP) function in series. After the local feature is acquired, the acquired local feature f is subjected to g Inputting a maximum pooling layer MaxPool for processing, then processing a characteristic input residual MLP (ResMLP) function obtained after the maximum pooling layer MaxPool is processed, and finally obtaining the high-frequency characteristic of the local characteristic.
f mp ＝MaxPool(f g )；
f h ＝ResMLP(f mp )；
Wherein f is g For local features, f mp Features obtained after treatment for the maximum pooling layer Maxpool, f h Is a high frequency characteristic of a local feature.
After the local feature is acquired, the acquired local feature is input to the second processing branch. And processing the local features based on the second processing branch to obtain the high-frequency features of the local features output by the second processing branch.
In particular, the second processing branch may be composed of an average pooling layer and a local attention module in series. First local feature f g Inputting the average pooling layer AvgPool to obtain the characteristic f of the average pooling layer after treatment ap 。
f ap ＝AvgPool(f g )；
Average pooling layer processed feature f ap Based on the local attention module, the average pooled layer processed feature f may be averaged ap The points with similar features in (b) are clustered to obtain the low frequency features of the local features.
The local attention module is processed based on the features processed by the average pooling layer, so that the long-distance dependency relationship between point cloud coordinates can be established, and the complexity of subsequent classification is reduced.
It can be understood that the obtained local features are subjected to feature extraction by adopting two parallel branches, wherein one branch can be formed by connecting a maximum pooling layer MaxPool and a residual error layer MLP (ResMLP) in series and is used for extracting high-frequency features; the other branch can be formed by connecting an average pooling layer AvgPool and a local attention module in series, and aims to extract low-frequency features, so that the high-frequency features and the low-frequency features of the local features are respectively extracted.
In step 130, after the high frequency feature of the local feature and the low frequency feature of the local feature are obtained, the obtained high frequency feature and the obtained low frequency feature are spliced to obtain the target feature of the original point cloud data.
And splicing the obtained high-frequency features and low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data for a subsequent point cloud classification process.
And splicing the obtained high-frequency characteristic and the low-frequency characteristic to obtain the characteristic of the sampling central point. Performing dimension-increasing processing on the characteristics of the sampling central point, wherein the dimension-increasing processing process can be as follows: acquiring the neighbor characteristics of the spliced characteristics, and subtracting the spliced characteristics from the acquired neighbor characteristics to obtain relative characteristics; splicing the obtained relative features with the spliced features, and using a multilayer sensor to perform dimension increasing on the obtained features; and then extracting corresponding high-frequency features and low-frequency features of the obtained ascending-dimension features, splicing the extracted high-frequency features and low-frequency features, and taking the obtained features as target features of the original point cloud data.
In step 140, after the target feature is obtained, the obtained target feature is input into a classifier. The classifier is obtained based on point cloud data samples and class labels corresponding to the point cloud data samples through training.
And inputting the obtained target into a classifier for classification to obtain a classification result output by the classifier. Wherein, the classifier is a pre-trained supervised classifier.
According to the point cloud classification method provided by the embodiment of the invention, after the local features of the original point cloud data are obtained, the obtained local features are extracted by adopting two parallel branches to obtain the high-frequency features and the low-frequency features, and the obtained target features after the high-frequency features and the low-frequency features are spliced are used for point cloud classification, so that the calculation complexity is reduced. Meanwhile, when the low-frequency features are extracted, the long-distance dependency relationship between the point cloud coordinates can be established based on vector attention processing, the complexity of subsequent classification is further reduced, and the recognition rate of point cloud classification is improved.
In one embodiment, inputting the local feature into a second processing branch for low-frequency feature extraction, to obtain a low-frequency feature of the local feature, includes: carrying out average pooling on the local features to obtain feature vectors after the average pooling; performing hierarchical dichotomy clustering on the feature vectors to obtain a plurality of feature clusters of the feature vectors; and respectively determining the feature relation in each feature cluster based on a vector attention algorithm, and combining the obtained feature relations to obtain the low-frequency features of the local features.
The second processing branch may be composed of an average pooling layer and a local attention module in series. The local features are subjected to average pooling, and the local features f can be g Inputting into an average pooling layer AvgPool to obtain average pooled layerCharacteristic vector f of ap 。
f ap ＝AvgPool(f g )；
Obtaining a feature vector f ap Then, f can be ap Inputting a local attention module comprising a vector attention algorithm, and acquiring low-frequency features f of the local features l 。
f l ＝LocalAttention(f ap )；
LocalAttention denotes the local attention module, i.e., for the feature vector
(S × D is dimension), and Queries, keys, values, abbreviated as Q, K and V are obtained by linear mapping. Then to
And performing hierarchical dichotomy clustering in a high-dimensional feature space according to the content (features) of the data to ensure that the sizes of all the clusters are the same. With the standard multi-head configuration in the Transformer (4 heads are set), each head independently performs the generation and clustering operations of Q, K, V.
The specific process of hierarchical dichotomy clustering is expressed by a formula as follows:
wherein c is 1 ,c 2 The center of the original cluster is represented,
and representing the feature cluster obtained after hierarchical dichotomy clustering. dist denotes the euclidean distance. N passes (n = log) 2 L) binary clustering iteration to finally obtain L feature clusters with equal size
(
May be set to 16). K and V are also divided into corresponding subsets according to the same index
And
within each feature cluster, feature relationships within the feature cluster are computed using a Vector Attention algorithm (VA):
Y i ＝VA(Q i ,K i ,V i )；
finally, the characteristics of each cluster
Are combined into
Thereby obtaining a low frequency feature f of the local feature l ；
f l ＝Linear(Y)+f ap ；
According to the point cloud classification method provided by the embodiment of the invention, the points with similar characteristics are gathered in the same class in the characteristic space, the attention is calculated in each class, the long-distance dependency relationship is established, and the complexity of subsequent point cloud classification calculation is obviously reduced. Meanwhile, the local part is divided based on hierarchical dichotomy clustering, and the calculation amount of vector attention is reduced.
In one embodiment, inputting the local feature into a first processing branch for high-frequency feature extraction, to obtain a high-frequency feature of the local feature, includes: performing maximum pooling on the local features to obtain local features subjected to maximum pooling; and carrying out residual error processing on the local features subjected to the maximum pooling processing to obtain the high-frequency features of the local features.
The first processing branch may consist of a maximum pooling layer MaxPool and a residual MLP (ResMLP) function in series. After the local feature is acquired, the acquired local feature f is subjected to g Firstly inputting a maximum pooling layer MaxPool for processing to obtain a local characteristic f after maximum pooling processing mp Then processing the maximum pooling layer Maxpool to obtain the characteristic f mp And inputting a residual MLP (ResMLP) function for processing to finally obtain the high-frequency characteristics of the local characteristics.
f mp ＝MaxPool(f g )；
f h ＝ResMLP(f mp )；
Wherein f is g For local features, f mp Features obtained after treatment for the maximum pooling layer Maxpool, f h Is a high frequency characteristic of a local feature.
ResMLP(f mp )＝LeakyReLU(BatchNorm(Conv1d(f mp )))+f mp
Here, leakyreu is the active layer, batchNorm is the normalization process, and Conv1d is the convolution process.
According to the point cloud classification method provided by the embodiment of the invention, the local features are input into the first processing branch, so that the high-frequency features of the local features output by the first processing branch are obtained, and the extraction of the high-frequency features in the local features is realized.
In one embodiment, performing feature extraction on original point cloud data to obtain local features of the original point cloud data includes: carrying out down-sampling on original point cloud data to obtain a sampling central point of the original point cloud data; determining a neighbor point corresponding to the sampling center point based on a K neighbor algorithm; obtaining the relative position coordinates of the adjacent point and the sampling central point based on the coordinates of the adjacent point and the coordinates of the sampling central point; splicing the relative position coordinates and the sampling center point according to coordinate dimensions to obtain spliced six-dimensional coordinates; and mapping the six-dimensional coordinates to a high-dimensional space based on a multilayer perceptron to obtain local features of the original point cloud data.
The method comprises the steps of carrying out down-sampling on original point cloud data to obtain a sampling central point p of the original point cloud data i Then for each sampled center point p i Finding its neighbor point p using k-nearest neighbor algorithm (kNN) j ；
p i ＝FPS(p)；
p j ＝kNN(p i ),k＝16；
To the nearest point p j And a central point p i Differencing to obtain relative position coordinates p j -p i Then with the center point p i Splicing according to coordinate dimensions, mapping the spatial relationship between a sampling center point and a neighborhood point to a high-dimensional spatial dimension of 64 and recording as D by using a multilayer perceptron (MLP), thereby extracting deep local features:
f g ＝MLP(||p i ,p j -p i ||)；
MLP＝ReLU(BatchNorm(Conv2d(X)))；
wherein, X is the input original point cloud data, reLU is the active layer, batchNorm is the normalization process, and Conv2d is the convolution process.
According to the point cloud classification method provided by the embodiment of the invention, the local features of the original point cloud data are obtained by extracting the features of the original point cloud data, and the original point cloud data is screened after being downsampled, so that the data volume of the point cloud data is reduced, and the subsequent processing process is facilitated.
In one embodiment, downsampling original point cloud data to obtain a sampling center point of the original point cloud data comprises: normalizing the original point cloud data into a unit sphere, and uniformly sampling the normalized original point cloud data to obtain a preset number of point cloud samples; and based on a farthest point sampling method, carrying out down-sampling on the point cloud sample to obtain a sampling central point of the point cloud sample.
After acquiring original point cloud data, firstly normalizing the three-dimensional coordinate data of the point cloud into a unit sphere; then, the normalized original point cloud data is uniformly sampled, for example, 1024 points can be uniformly sampled to obtain
Namely, the point cloud samples with the preset number are obtained.
And (3) carrying out down-sampling on the point cloud sample p by 2 multiplying power based on a furthest point sampling method (FPS) to obtain a sampling central point of the point cloud sample.
According to the point cloud classification method provided by the embodiment of the invention, original point cloud data are normalized into a unit sphere, the normalized original point cloud data are uniformly sampled to obtain a preset number of point cloud samples, the point cloud samples are downsampled based on a farthest point sampling method to obtain the sampling central point of the point cloud samples, and a foundation is provided for extracting the local characteristics of the original point cloud data based on the sampling central point.
In one embodiment, the splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data, includes: splicing the high-frequency characteristic and the low-frequency characteristic to obtain the characteristic of a sampling central point; and performing dimension increasing processing on the characteristics of the sampling central point for preset times, and taking the characteristics obtained after the dimension increasing processing as target characteristics of the original point cloud data.
After the high-frequency characteristic of the local characteristic and the low-frequency characteristic of the local characteristic are obtained, the obtained high-frequency characteristic and the obtained low-frequency characteristic are spliced to obtain the characteristic of the sampling central point. The characteristic of the sampling central point is the core characteristic of the original point cloud data.
After the characteristics of the sampling center point are obtained, in order to facilitate the subsequent point cloud classification process, the characteristics of the sampling center point are subjected to dimension increasing processing for preset times, the characteristics obtained after the dimension increasing processing are used as target characteristics of the original point cloud data, and the target characteristics are used for subsequent point cloud classification.
The performing the upscaling processing on the feature of the sampling central point may include: obtaining the neighbor characteristic of the sampling central point, and subtracting the characteristic of the sampling central point from the obtained neighbor characteristic to obtain a relative characteristic; splicing the obtained relative features with the features of the sampling central point, and using a multilayer perceptron to carry out dimension increasing on the spliced features; then extracting corresponding high-frequency features and low-frequency features of the obtained dimension-increasing features, and splicing the extracted high-frequency features and low-frequency features; and finally, taking the obtained spliced features as the features obtained after the dimension increasing treatment.
According to the point cloud classification method provided by the embodiment of the invention, after the high-frequency characteristic of the local characteristic and the low-frequency characteristic of the local characteristic are obtained, the obtained high-frequency characteristic and low-frequency characteristic are spliced to obtain the characteristic of the sampling central point. And the feature of the central point is subjected to dimension increasing processing to obtain the target feature of the original point cloud data, thereby providing a basis for subsequent point cloud classification.
The following describes a technical solution provided by the present invention, taking a schematic flow chart of a point cloud classification method provided by the present invention as an example, fig. 2:
in step 210, after acquiring the original point cloud data, firstly normalizing the three-dimensional coordinate data of the point cloud into a unit sphere; then, the sample is uniformly sampled for 1024 points to obtain
In step 220, the coordinates p of the input point are down-sampled by 2-fold using the farthest point sampling method (FPS), and then the center point p of each sample is down-sampled i Finding its neighbor point p using k-nearest neighbor algorithm (kNN) j ；
p i ＝FPS(p)；
p j ＝kNN(p i )，k＝16；
In step 230, the neighboring point p is examined j And a central point p i Differencing to obtain relative position coordinates p j -p i Then with the center point p i Splicing according to coordinate dimensions, mapping the spatial relation between a sampling center point and a neighborhood point to a high-dimensional space dimension of 64 and recording as D by using a multilayer perceptron (MLP), thereby extracting a deep local feature f g ：
f g ＝MLP(||p i ，p j -p i ||)；
MLP＝ReLU(BatchNorm(Conv2d(X)))；
Wherein, X is the input original point cloud data, reLU is the active layer, batchNorm is the normalization process, and Conv2d is the convolution process.
In step 240, the high frequency feature of the local feature and the low frequency feature of the local feature are extracted, the extraction process is as shown in the schematic diagram of the local feature processing process provided in fig. 3, and the obtained local feature f g Two parallel branches are adopted for feature aggregation, and the first processing branch is formed by connecting a maximum pooling layer MaxPool and a residual error function MLP (ResMLP) in series and is used for acquiring high-frequency features f of local features h The second processing branch is composed of an average pooling layer AvgPool and a local attention module which are connected in series and aims at extracting the low-frequency feature f of the local feature l 。
The first processing branch is composed of a max pooling layer MaxPool and a residual MLP (ResMLP) function in series. After the local feature is acquired, the acquired local feature f is subjected to g Firstly inputting a maximum pooling layer MaxPool for processing to obtain a local characteristic f after maximum pooling processing mp Then processing the maximum pooling layer MaxPool to obtain the characteristic f mp Inputting residual MLP (ResMLP) function for processing to finally obtain high-frequency characteristic f of local characteristic h 。
f mp ＝MaxPool(f g )；
f h ＝ResMLP(f mp )；
Wherein f is g For local features, f mp Features obtained after treatment for the maximum pooling layer Maxpool, f h Is a high frequency characteristic of a local feature.
The second processing branch is to firstly process the local characteristic f g Inputting the average pooling layer AvgPool to obtain the feature vector f after the average pooling layer is processed ap . Then f is put ap Inputting a local attention module containing a vector attention algorithm to obtain a low-frequency feature f of the local feature l 。
f ap ＝AvgPool(f g )；
f l ＝LocalAttention(f ap )；
Wherein:
ResMLP(f mp )＝LeakyReLU(BatchNorm(Conv1d(f mp )))+f mp
LocalAttention denotes the local attention module, i.e., for the feature vector
(S is the number of sampling center points in the step 220), and Queries, keys, values, abbreviated as Q, K and V, are obtained through linear mapping. Then to
And performing hierarchical dichotomy clustering in a high-dimensional feature space according to the content (features) of the data to ensure that the sizes of all the clusters are the same. With the standard multi-head configuration in the Transformer (4 heads are set), each head independently performs the generation and clustering operations of Q, K, V. The specific process of hierarchical dichotomy clustering is expressed by a formula as follows:
wherein c is 1 ,c 2 The center of the original cluster is represented,
and representing the characteristic cluster obtained after binary clustering. dist denotes the euclidean distance. N passes (n = log) 2 L) binary clustering iteration to finally obtain L feature clusters with equal size
(
Set to 16). K and V are also divided into corresponding subsets according to the same index
And
within each feature cluster, the intra-feature cluster feature relationships are computed using Vector Attention (VA):
Y i ＝VA(Q i ,K i ,V i )；
finally, the characteristics of each characteristic cluster are clustered
Are combined into
f l ＝Linear(Y)+f ap
f i ＝LeakyReLU(BatchNorm(Conv1d(||f h ,f l ||)))；
MLP＝ReLU(BatchNorm(Conv2d(·)))；
then, repeating the steps 240 and 250;
and 270, repeating the step 260 for three times, finally obtaining a feature vector of 32 multiplied by 1024, obtaining the feature vector of 1 multiplied by 1024 through the global maximum pooling layer as the final feature representation of the point cloud, and finally transmitting the feature vector into a classifier to obtain a classification result.
Fig. 4 is a schematic structural diagram of a point cloud classifying device provided by the present invention, as shown in fig. 4, the device includes:
a feature extraction module 410, configured to perform feature extraction on original point cloud data to obtain local features of the original point cloud data;
the processing module 420 is configured to input the local features into a first processing branch to perform high-frequency feature extraction, so as to obtain high-frequency features of the local features, and input the local features into a second processing branch to perform low-frequency feature extraction, so as to obtain low-frequency features of the local features;
a splicing module 430, configured to splice the high-frequency features and the low-frequency features, and perform dimension-increasing processing on the features obtained after splicing to obtain target features of the original point cloud data;
the classification module 440 is configured to input the target features into a classifier, which is trained based on point cloud data samples and corresponding class labels, to obtain a classification result of the original point cloud data.
According to the point cloud classification device provided by the embodiment of the invention, after the local features of the original point cloud data are obtained, the obtained local features are extracted by adopting two parallel branches to obtain the high-frequency features and the low-frequency features, and the obtained target features after the high-frequency features and the low-frequency features are spliced are used for point cloud classification, so that the calculation complexity is reduced. Meanwhile, when the low-frequency features are extracted, the long-distance dependency relationship between the point cloud coordinates can be established based on vector attention processing, the complexity of subsequent classification is further reduced, and the recognition rate of point cloud classification is improved.
In one embodiment, the processing module 420 is specifically configured to:
inputting the local features into a second processing branch for low-frequency feature extraction to obtain the low-frequency features of the local features, wherein the low-frequency features comprise:
carrying out average pooling on the local features to obtain feature vectors after average pooling;
performing hierarchical dichotomy clustering on the feature vectors to obtain a plurality of feature clusters of the feature vectors;
and respectively determining the feature relation in each feature cluster based on a vector attention algorithm, and combining the obtained feature relations to obtain the low-frequency features of the local features.
In one embodiment, the feature extraction module 410 is specifically configured to:
performing feature extraction on original point cloud data to obtain local features of the original point cloud data, wherein the method comprises the following steps:
the method comprises the steps of carrying out down-sampling on original point cloud data, obtaining a sampling central point of the original point cloud data, and determining a neighbor point corresponding to the sampling central point based on a K neighbor algorithm;
obtaining the relative position coordinates of the adjacent point and the sampling central point based on the coordinates of the adjacent point and the coordinates of the sampling central point;
splicing the relative position coordinates and the sampling center point according to coordinate dimensions to obtain spliced six-dimensional coordinates;
and mapping the six-dimensional coordinates to a high-dimensional space based on a multilayer perceptron to obtain local features of the original point cloud data.
In one embodiment, the feature extraction module 410 is specifically configured to:
the method for sampling the original point cloud data to obtain the sampling center point of the original point cloud data comprises the following steps:
normalizing the original point cloud data into a unit sphere, and uniformly sampling the normalized original point cloud data to obtain a preset number of point cloud samples;
and based on a farthest point sampling method, carrying out downsampling on the point cloud sample to obtain a sampling central point of the point cloud sample.
In one embodiment, the splicing module 430 is specifically configured to:
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data, wherein the steps of:
splicing the high-frequency features and the low-frequency features to obtain the features of the sampling central point;
and performing dimension increasing processing on the characteristics of the sampling central point for preset times, and taking the characteristics obtained after the dimension increasing processing as target characteristics of the original point cloud data.
Fig. 5 illustrates a physical structure diagram of an electronic device, which may include, as shown in fig. 5: a processor (processor) 510, a communication Interface (Communications Interface) 520, a memory (memory) 530 and a communication bus 540, wherein the processor 510, the communication Interface 520 and the memory 530 communicate with each other via the communication bus 540. Processor 510 may invoke logic instructions in memory 530 to perform a point cloud classification method comprising:
extracting the characteristics of original point cloud data to obtain the local characteristics of the original point cloud data;
inputting the local features into a first processing branch to perform high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch to perform low-frequency feature extraction to obtain low-frequency features of the local features;
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and inputting the target features into a classifier to obtain a classification result of the original point cloud data, wherein the classifier is obtained by training based on a point cloud data sample and a class label corresponding to the point cloud data sample.
In addition, the logic instructions in the memory 530 may be implemented in the form of software functional units and stored in a computer readable storage medium when the logic instructions are sold or used as a stand-alone product. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In another aspect, the present invention also provides a computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions which, when executed by a computer, enable the computer to perform the point cloud classification method provided by the above methods, the method comprising:
extracting the characteristics of original point cloud data to obtain the local characteristics of the original point cloud data;
inputting the local features into a first processing branch to perform high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch to perform low-frequency feature extraction to obtain low-frequency features of the local features;
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and inputting the target features into a classifier to obtain a classification result of the original point cloud data, wherein the classifier is obtained by training based on a point cloud data sample and a class label corresponding to the point cloud data sample.
In yet another aspect, the present invention also provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, is implemented to perform the point cloud classification methods provided above, the method comprising:
extracting the characteristics of original point cloud data to obtain the local characteristics of the original point cloud data;
inputting the local features into a first processing branch to perform high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch to perform low-frequency feature extraction to obtain low-frequency features of the local features;
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and inputting the target features into a classifier to obtain a classification result of the original point cloud data, wherein the classifier is obtained by training based on a point cloud data sample and a class label corresponding to the point cloud data sample.
The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present embodiment. One of ordinary skill in the art can understand and implement it without inventive effort.
Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware. With this understanding in mind, the above-described technical solutions may be embodied in the form of a software product, which can be stored in a computer-readable storage medium such as ROM/RAM, magnetic disk, optical disk, etc., and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in the embodiments or some parts of the embodiments.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A method of point cloud classification, the method comprising:
extracting the characteristics of original point cloud data to obtain the local characteristics of the original point cloud data;
inputting the local features into a first processing branch to perform high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch to perform low-frequency feature extraction to obtain low-frequency features of the local features;
splicing the high-frequency features and the low-frequency features, and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and inputting the target features into a classifier to obtain a classification result of the original point cloud data, wherein the classifier is obtained by training based on a point cloud data sample and a class label corresponding to the point cloud data sample.
2. The point cloud classification method of claim 1, wherein the inputting the local features into a second processing branch for low-frequency feature extraction to obtain the low-frequency features of the local features comprises:
carrying out average pooling on the local features to obtain feature vectors after average pooling;
performing hierarchical dichotomy clustering on the feature vectors to obtain a plurality of feature clusters of the feature vectors;
and respectively determining the feature relation in each feature cluster based on a vector attention algorithm, and combining the obtained feature relations to obtain the low-frequency features of the local features.
3. The point cloud classification method according to claim 1, wherein the step of inputting the local features into a first processing branch for high-frequency feature extraction to obtain the high-frequency features of the local features comprises:
performing maximum pooling on the local features to obtain local features subjected to maximum pooling;
and carrying out residual error processing on the local features subjected to the maximum pooling processing to obtain the high-frequency features of the local features.
4. The point cloud classification method of claim 1, wherein the extracting features of the original point cloud data to obtain local features of the original point cloud data comprises:
the method comprises the steps of carrying out down-sampling on original point cloud data, obtaining a sampling central point of the original point cloud data, and determining a neighbor point corresponding to the sampling central point based on a K neighbor algorithm;
obtaining the relative position coordinates of the adjacent point and the sampling central point based on the coordinates of the adjacent point and the coordinates of the sampling central point;
splicing the relative position coordinates and the sampling center point according to coordinate dimensions to obtain spliced six-dimensional coordinates;
and mapping the six-dimensional coordinates to a high-dimensional space based on a multilayer perceptron to obtain local features of the original point cloud data.
5. The point cloud classification method of claim 4, wherein the down-sampling of the original point cloud data to obtain a sampling center point of the original point cloud data comprises:
normalizing the original point cloud data into a unit sphere, and uniformly sampling the normalized original point cloud data to obtain a preset number of point cloud samples;
and based on a farthest point sampling method, carrying out down-sampling on the point cloud sample to obtain a sampling central point of the point cloud sample.
6. The point cloud classification method according to claim 1, wherein the step of splicing the high-frequency features and the low-frequency features and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data comprises the steps of:
splicing the high-frequency characteristic and the low-frequency characteristic to obtain the characteristic of a sampling central point;
and performing dimension increasing processing on the characteristics of the sampling central point for preset times, and taking the characteristics obtained after the dimension increasing processing as target characteristics of the original point cloud data.
7. A point cloud classification device, comprising:
the characteristic extraction module is used for extracting the characteristics of the original point cloud data to obtain the local characteristics of the original point cloud data;
the processing module is used for inputting the local features into a first processing branch for high-frequency feature extraction to obtain high-frequency features of the local features, and inputting the local features into a second processing branch for low-frequency feature extraction to obtain low-frequency features of the local features;
the splicing module is used for splicing the high-frequency features and the low-frequency features and performing dimension-increasing processing on the spliced features to obtain target features of the original point cloud data;
and the classification module is used for inputting the target characteristics into a classifier to obtain a classification result of the original point cloud data, and the classifier is obtained by training based on the point cloud data sample and the corresponding class label.
8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the point cloud classification method of any of claims 1 to 6 when executing the computer program.
9. A non-transitory computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, implements the point cloud classification method according to any one of claims 1 to 6.
10. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the point cloud classification method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211494418.3A CN115830375B (en) | 2022-11-25 | 2022-11-25 | Point cloud classification method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211494418.3A CN115830375B (en) | 2022-11-25 | 2022-11-25 | Point cloud classification method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115830375A true CN115830375A (en) | 2023-03-21 |
| CN115830375B CN115830375B (en) | 2024-09-24 |
Family
ID=85531771
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211494418.3A Active CN115830375B (en) | 2022-11-25 | 2022-11-25 | Point cloud classification method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115830375B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117422823A (en) * | 2023-09-21 | 2024-01-19 | 北京智源人工智能研究院 | Three-dimensional point cloud characterization model construction method and device, electronic equipment and storage medium |
| CN117496161A (en) * | 2023-12-29 | 2024-02-02 | 武汉理工大学 | Point cloud segmentation method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111915592A (en) * | 2020-08-04 | 2020-11-10 | 西安电子科技大学 | Remote sensing image cloud detection method based on deep learning |
| US20210158161A1 (en) * | 2019-11-22 | 2021-05-27 | Fraud.net, Inc. | Methods and Systems for Detecting Spurious Data Patterns |
| CN112949647A (en) * | 2021-02-26 | 2021-06-11 | 中国科学院自动化研究所 | Three-dimensional scene description method and device, electronic equipment and storage medium |
| CN114091628A (en) * | 2022-01-20 | 2022-02-25 | 山东大学 | Three-dimensional point cloud up-sampling method and system based on double branch network |
| CN114373099A (en) * | 2022-01-05 | 2022-04-19 | 上海交通大学 | Three-dimensional point cloud classification method based on sparse graph convolution |
| CN115147703A (en) * | 2022-07-28 | 2022-10-04 | 广东小白龙环保科技有限公司 | GinTrans network-based garbage segmentation method and system |
| CN115375877A (en) * | 2022-09-20 | 2022-11-22 | 广东工业大学 | Three-dimensional point cloud classification method and device based on channel attention mechanism |
-
2022
- 2022-11-25 CN CN202211494418.3A patent/CN115830375B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210158161A1 (en) * | 2019-11-22 | 2021-05-27 | Fraud.net, Inc. | Methods and Systems for Detecting Spurious Data Patterns |
| CN111915592A (en) * | 2020-08-04 | 2020-11-10 | 西安电子科技大学 | Remote sensing image cloud detection method based on deep learning |
| CN112949647A (en) * | 2021-02-26 | 2021-06-11 | 中国科学院自动化研究所 | Three-dimensional scene description method and device, electronic equipment and storage medium |
| CN114373099A (en) * | 2022-01-05 | 2022-04-19 | 上海交通大学 | Three-dimensional point cloud classification method based on sparse graph convolution |
| CN114091628A (en) * | 2022-01-20 | 2022-02-25 | 山东大学 | Three-dimensional point cloud up-sampling method and system based on double branch network |
| CN115147703A (en) * | 2022-07-28 | 2022-10-04 | 广东小白龙环保科技有限公司 | GinTrans network-based garbage segmentation method and system |
| CN115375877A (en) * | 2022-09-20 | 2022-11-22 | 广东工业大学 | Three-dimensional point cloud classification method and device based on channel attention mechanism |
Non-Patent Citations (1)
| Title |
|---|
| Y. LIU, B. TIAN, Y. LV, L. LI AND F. -Y. WANG: "Point Cloud Classification Using Content-Based Transformer via Clustering in Feature Space", IEEE, vol. 11, no. 1, 31 January 2024 (2024-01-31), pages 231 - 239, XP011958291, DOI: 10.1109/JAS.2023.123432 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117422823A (en) * | 2023-09-21 | 2024-01-19 | 北京智源人工智能研究院 | Three-dimensional point cloud characterization model construction method and device, electronic equipment and storage medium |
| CN117496161A (en) * | 2023-12-29 | 2024-02-02 | 武汉理工大学 | Point cloud segmentation method and device |
| CN117496161B (en) * | 2023-12-29 | 2024-04-05 | 武汉理工大学 | Point cloud segmentation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115830375B (en) | 2024-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112529015B (en) | Three-dimensional point cloud processing method, device and equipment based on geometric unwrapping | |
| CN106529447B (en) | Method for identifying face of thumbnail | |
| CN112288011B (en) | Image matching method based on self-attention deep neural network | |
| CN111862289B (en) | Point cloud up-sampling method based on GAN network | |
| CN115830375B (en) | Point cloud classification method and device | |
| CN113989890A (en) | Face expression recognition method based on multi-channel fusion and lightweight neural network | |
| CN111507409B (en) | Hyperspectral image classification method and device based on depth multi-view learning | |
| CN112634149B (en) | Point cloud denoising method based on graph convolution network | |
| CN107301643B (en) | Well-marked target detection method based on robust rarefaction representation Yu Laplce's regular terms | |
| CN112785611B (en) | 3D point cloud weak supervision semantic segmentation method and system | |
| CN111414953A (en) | Point cloud classification method and device | |
| US11682166B2 (en) | Fitting 3D primitives to a high-resolution point cloud | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| CN115294563A (en) | 3D point cloud analysis method and device based on Transformer and capable of enhancing local semantic learning ability | |
| Bogacz et al. | Period classification of 3D cuneiform tablets with geometric neural networks | |
| CN112801945A (en) | Depth Gaussian mixture model skull registration method based on dual attention mechanism feature extraction | |
| CN118351320B (en) | Instance segmentation method based on three-dimensional point cloud | |
| Elmoogy et al. | Surfcnn: A descriptor accelerated convolutional neural network for image-based indoor localization | |
| CN114693923A (en) | Three-dimensional point cloud semantic segmentation method based on context and attention | |
| CN114549757A (en) | Three-dimensional point cloud up-sampling method based on attention mechanism | |
| CN114170465A (en) | Attention mechanism-based 3D point cloud classification method, terminal device and storage medium | |
| Zhou et al. | Retrieval and localization with observation constraints | |
| CN116740498B (en) | Model pre-training method, model training method, object processing method and device | |
| CN111860668A (en) | Point cloud identification method of deep convolution network for original 3D point cloud processing | |
| CN116912486A (en) | Target segmentation method based on edge convolution and multidimensional feature fusion and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |