CN113888748A - Point cloud data processing method and device - Google Patents
Point cloud data processing method and device Download PDFInfo
- Publication number
- CN113888748A CN113888748A CN202111133410.XA CN202111133410A CN113888748A CN 113888748 A CN113888748 A CN 113888748A CN 202111133410 A CN202111133410 A CN 202111133410A CN 113888748 A CN113888748 A CN 113888748A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- clusters
- cluster
- clustering
- matrix
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
Abstract
The embodiment of the invention discloses a point cloud data processing method and a point cloud data processing device, wherein the method comprises the following steps: fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information; based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced; and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object. By adopting the technical scheme, the problem of over-segmentation in the point cloud clustering process is solved, and the effectiveness of a clustering result is improved.
Description
Technical Field
The embodiment of the invention relates to the technical field of vehicle-mounted laser radar point cloud processing, in particular to a point cloud data processing method and device.
Background
Due to the data characteristics of the laser radar point cloud, such as dense points, various distribution, large data volume and no distribution rule, and the disorder of the arrangement of the original point cloud data, most of the current common processing based on the laser radar point cloud has the step of point cloud clustering.
The clustering of the traditional method usually depends on the geometric distance between the cloud points and the prior hypothesis of specific objects, such as that pedestrians and trunks are cylindrical, vehicles are rectangular and the like, and the clustering method which strongly depends on the geometric relationship and the prior characteristics usually causes the problem of segmentation, namely that the same object is clustered into a plurality of cloud point clusters.
At present, in order to reduce such over-segmentation, the judgment between different point cloud clusters is generally performed according to the geometric features of the object and other artificially set features. Because the judgment mode has the limitation of manually setting the characteristics, the judgment result of which object the point cloud cluster belongs to is poor in applicability, and the over-segmentation condition cannot be effectively solved.
Disclosure of Invention
The embodiment of the invention provides a point cloud data processing method and device, which are used for overcoming the problem of over-segmentation in a point cloud clustering process and improving the effectiveness of a clustering result.
In a first aspect, the present invention provides a point cloud data processing method, including:
fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information;
based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced;
and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object.
Optionally, the attention mechanism network model is obtained by training in the following manner:
taking a feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein object categories to which the point cloud clusters belong are marked in the training sample;
training an initial attention mechanism network model by using the training sample to obtain a weighted matrix after weighting;
for each point cloud cluster in the weighting matrix, determining a predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters;
and in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining a trained attention mechanism network model.
Optionally, the attention mechanism network model is a preamble codec predictor transform network model.
Optionally, the fusing the feature information of each point cloud cluster includes:
splicing the characteristic information of each point cloud cluster, and combining the characteristic information of each point cloud cluster after splicing to form a splicing matrix;
and performing feature fusion processing on the spliced matrix based on the multi-layer perceptron MLP to obtain a feature matrix.
Optionally, based on the multi-layer perceptron MLP, performing feature fusion processing on the mosaic matrix to obtain a feature matrix, including:
for any point cloud cluster, performing feature fusion processing on a splicing matrix corresponding to the point cloud cluster based on a multilayer perceptron MLP to obtain comprehensive feature information of the point cloud cluster;
and performing the feature fusion processing on all the point cloud clusters according to the number of the point cloud clusters to obtain a feature matrix corresponding to all the point cloud clusters, wherein the feature matrix comprises the number and feature dimensions of the point cloud clusters.
Optionally, clustering the point cloud clusters after weighting processing to determine point cloud clusters belonging to the same object, including:
and for each cloud cluster after weighting, sequentially selecting one point cloud cluster as an initial clustering center, and determining target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.
Optionally, determining a target point cloud cluster belonging to the same object according to a distance relationship between other point cloud clusters and the initial clustering center includes:
calculating Euclidean distances between other point cloud clusters and the initial clustering center;
if the Euclidean distance value is smaller than a preset clustering threshold value, all point cloud clusters in the preset clustering threshold value range are used as candidate point cloud clusters which belong to the same object with the initial clustering center;
and taking the average value of the distances of the candidate point cloud clusters as a new clustering center, returning to execute the operation of calculating the Euclidean distances between other point cloud clusters and the new clustering center, and taking all candidate point cloud clusters which belong to the same object with the new clustering center as target point cloud clusters when the determined new clustering center is not changed.
Optionally, the clustering process on the original point cloud data includes:
preprocessing original point cloud data, wherein the preprocessing comprises ground point elimination and down-sampling;
clustering the preprocessed point clouds to obtain a plurality of point cloud clusters;
the original point cloud data is obtained by analyzing the laser radar original data, and the original point cloud data comprises the number of all point clouds of a current frame, three-dimensional coordinates of the point clouds and reflection intensity.
In a second aspect, an embodiment of the present invention further provides a point cloud data processing apparatus, where the apparatus includes:
a feature fusion module configured to: fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information;
a weighting processing module configured to: based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced;
a quadratic clustering module configured to: and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object.
Optionally, the attention mechanism network model is obtained by training in the following manner:
taking a feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein object categories to which the point cloud clusters belong are marked in the training sample;
training an initial attention mechanism network model by using the training sample to obtain a weighted matrix after weighting;
for each point cloud cluster in the weighting matrix, determining a predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters;
and in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining a trained attention mechanism network model.
Optionally, the attention mechanism network model is a forward-sequence coding predictor transformer network model.
Optionally, the feature fusion module includes:
a stitching unit configured to: splicing the characteristic information of each point cloud cluster, and combining the characteristic information of each point cloud cluster after splicing to form a splicing matrix;
a feature fusion unit configured to: and performing feature fusion processing on the spliced matrix based on the multi-layer perceptron MLP to obtain a feature matrix.
Optionally, the feature fusion unit is specifically configured to:
for any point cloud cluster, performing feature fusion processing on a splicing matrix corresponding to the point cloud cluster based on a multilayer perceptron MLP to obtain comprehensive feature information of the point cloud cluster;
and performing the feature fusion processing on all the point cloud clusters according to the number of the point cloud clusters to obtain a feature matrix corresponding to all the point cloud clusters, wherein the feature matrix comprises the number and feature dimensions of the point cloud clusters.
Optionally, the secondary clustering module includes:
a target point cloud cluster determination unit configured to: and for each cloud cluster after weighting, sequentially selecting one point cloud cluster as an initial clustering center, and determining target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.
Optionally, the target point cloud cluster determining unit is specifically configured to:
calculating Euclidean distances between other point cloud clusters and the initial clustering center;
if the Euclidean distance value is smaller than a preset clustering threshold value, all point cloud clusters in the preset clustering threshold value range are used as candidate point cloud clusters which belong to the same object with the initial clustering center;
and taking the average value of the distances of the candidate point cloud clusters as a new clustering center, returning to execute the operation of calculating the Euclidean distances between other point cloud clusters and the new clustering center, and taking all candidate point cloud clusters which belong to the same object with the new clustering center as target point cloud clusters when the determined new clustering center is not changed.
Optionally, the clustering process on the original point cloud data includes:
preprocessing original point cloud data, wherein the preprocessing comprises ground point elimination and down-sampling;
clustering the preprocessed point clouds to obtain a plurality of point cloud clusters;
the original point cloud data is obtained by analyzing the laser radar original data, and the original point cloud data comprises the number of all point clouds of a current frame, three-dimensional coordinates of the point clouds and reflection intensity.
In a third aspect, an embodiment of the present invention further provides a computing device, including:
a memory storing executable program code;
a processor coupled with the memory;
the processor calls the executable program codes stored in the memory to execute the point cloud data processing method provided by any embodiment of the invention.
In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the point cloud data processing method provided in any embodiment of the present invention.
According to the technical scheme provided by the embodiment of the invention, the characteristic matrix can be obtained by fusing the characteristic information of the cloud clusters of each point on the basis of the first clustering result. By weighting the point cloud clusters in the feature matrix based on the attention mechanism network model, a single point cloud cluster can contain feature information of other point cloud clusters, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced. By clustering the point cloud clusters after weighting processing again, the target point cloud clusters belonging to the same object can be determined, the problem of over-segmentation in the point cloud clustering process is effectively solved on the premise of avoiding under-segmentation, and the effectiveness of the point cloud clustering result is effectively improved.
The technical effects of the embodiment of the invention comprise:
1. through the attention mechanism network model, each point cloud can acquire the information of other point cloud clusters, so that the similarity between different point cloud cluster characteristics belonging to the same object can be increased, the similarity between the point cloud clusters belonging to different object points can be reduced, the secondary clustering effect can be improved, the condition that the application range possibly caused by artificial design characteristics is limited can be avoided, and the robustness and the practicability of the model can be improved.
2. By fusing the characteristic information of the point cloud cluster, the characteristic matrix obtained after the characteristic fusion can simultaneously reflect the comprehensive characteristics of the position and the shape of the point cloud cluster, and the characteristic dimension of the point cloud cluster can be reduced, so that the calculation amount in the subsequent clustering process is reduced.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
FIG. 1 is a schematic diagram of a point cloud quadratic clustering method according to an embodiment of the present invention;
fig. 2 is a flowchart of a training method of an attention mechanism network model according to an embodiment of the present invention;
fig. 3 is a flowchart of a point cloud data processing method according to an embodiment of the present invention;
fig. 4 is a flowchart of a point cloud data processing method according to a second embodiment of the present invention;
fig. 5 is a block diagram of a point cloud data processing apparatus according to a fourth embodiment of the present invention;
fig. 6 is a schematic structural diagram of a computing device according to a fifth embodiment of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without inventive effort based on the embodiments of the present invention, are within the scope of the present invention.
It is to be noted that the terms "comprises" and "comprising" and any variations thereof in the embodiments and drawings of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those steps or elements listed, but may alternatively include other steps or elements not listed, or inherent to such process, method, article, or apparatus.
The embodiment of the invention discloses a point cloud data processing method and device. In order to clearly and clearly explain the contents of the embodiments of the present invention, the following briefly introduces the basic operation principle of the embodiments of the present invention.
Fig. 1 is a schematic diagram of a point cloud quadratic clustering method according to an embodiment of the present invention. As shown in fig. 1. And the input object of the secondary clustering is a point cloud cluster obtained after the primary clustering. Point cloud clusters tend to have more information, such as shape, size, average reflection intensity, curvature, and intra-cluster point cloud distribution density, than single point cloud points. In reality, different point cloud clusters of the same object have similar features, such as reflection intensity, point cloud density, and the like, and meanwhile, if an object is clustered into several clusters, the several clusters are often close to each other or have similar projection areas in a certain direction, and the point cloud clusters of different objects generally have different features. Therefore, by combining the point cloud cluster characteristics, secondary clustering can be performed on the basis of the point cloud cluster, and over-segmented objects are combined.
The quadratic clustering method provided by the embodiment of the invention comprises two parts of point cloud cluster feature extraction and point cloud cluster clustering. Different from the prior art, the feature extraction in the embodiment of the invention adopts a feature extraction network based on deep learning. Through the feature extraction network, the shape feature of each point cloud cluster can be calculated according to the relative position relationship of the point cloud clusters in the point cloud, and the position feature can be calculated according to the position information of the point cloud clusters in the whole frame of point cloud. By combining, i.e., stitching, these two features, a comprehensive feature including the shape and location of the point cloud cluster can be obtained.
After the comprehensive characteristics of the point cloud cluster are obtained, the technical scheme of the embodiment of the invention adopts a neural network or a deep learning model to carry out secondary clustering. In particular, quadratic clustering may be performed using a transform (forward-coded predictor) attention network model in natural language processing. The model takes all the point cloud cluster characteristics of each frame into consideration, so that the similarity of the characteristics of point cloud clusters which belong to the same object but are clustered into multiple clusters is increased, and the similarity of the characteristics of point cloud clusters which do not belong to the same object is reduced. The application of the model can avoid the situation of under-segmentation, and can also solve the over-segmentation situation in the primary clustering process, thereby improving the effectiveness of subsequent secondary clustering.
In the prior art, the secondary clustering process usually judges different point cloud clusters according to the geometric characteristics of an object and other artificially set characteristics, and does not combine the global information of the whole frame of point cloud.
In the following, the training process and the application process of the attention mechanism network model in the point cloud secondary clustering process are described in detail.
Example one
Fig. 2 is a flowchart of a training method for an attention mechanism network model according to an embodiment of the present invention, where the method may be applied to a secondary clustering process of point cloud data, and may be applied in application scenarios such as target recognition and tracking sensing. The method may be performed by a training apparatus of the attention mechanism network model, which may be implemented by means of software and/or hardware. As shown in fig. 2, the method provided by this embodiment includes:
s100, fusing the feature information of the cloud clusters of each point to obtain a feature matrix.
The point cloud cluster is obtained by clustering original point cloud data. The specific clustering process may be:
analyzing the laser radar point cloud data to obtain the original point cloud data P of the current frame0In which P is0Can be represented as [ N0，M]Wherein N is0And M is the three-dimensional coordinate and the reflection intensity of the point cloud. By aligning the original point cloud data P0Preprocessing, such as ground point elimination and down-sampling, can remove noise in the original point cloud dataAnd (4) information.
After the original point cloud data is preprocessed, the original point cloud data can be clustered to obtain a point cloud cluster. The clustering method can determine point cloud clusters belonging to the same object according to the similarity of the point cloud cluster characteristics. The input of the clustering process is the point cloud after pretreatment, and the output is the point cloud cluster. Each cluster of point clouds obtained by clustering comprises one or more point clouds and can be represented by the following formula:
(c0,c1,c2…,cF)＝f(P1)
wherein, ciRepresenting the ith cluster of point clouds including (p)i0,pi1,pi2…)，pijIs the j point cloud point in the i cluster. The number of the point clouds contained in each cluster of point clouds is closely related to the adopted clustering algorithm and a specific scene, and the secondary clustering processing flow provided by the embodiment of the invention does not have requirements on the number of the clusters and the number of the point clouds contained in each cluster of point clouds. Compared with the cloud point of a single point, the clustered point cloud cluster has more information, such as the shape, the size, the average reflection intensity, the curvature, the distribution density of the point cloud in the cluster and the like.
In this embodiment, the feature information of each point cloud cluster includes shape feature information and position feature information. The extraction of the shape feature information may be implemented by a feature extraction network, for example, a pointenet (a deep network framework using original point cloud data that does not need voxelization and rendering as input), and through the feature extraction network, each point cloud cluster may calculate its shape feature according to the relative position relationship between the point clouds. The extraction of the positional feature information may be realized by MLP (multi layer Perceptron). The extraction of the above feature information can be expressed by the following formula:
Flocal_i＝netlocal(ci),ci＝(pi0,pi1…,pij)
Fglobal_i＝netglobal(Ci)
wherein, Flocal_iAnd Fglobal_iRespectively representing shape feature information extracted from ith cluster point cloudAnd position feature information, ciAs the ith cluster of point clouds, pijIs the jth point of the ith cluster of point cloud, and Ci is the central coordinate of the ith cluster of point cloud, netlocalAnd netglobalExtraction networks for extracting shape features and position features, respectively.
In this embodiment, for the extracted shape feature information and position feature information, the shape feature information and the position feature information may be fused, which is convenient for subsequent feature processing and clustering operation. The feature fusion can be realized by adopting MLP, and the specific fusion process can comprise the step of fusing the shape feature and the position feature of each point cloud cluster, so that the comprehensive feature information obtained after feature fusion can reflect the comprehensive features of the position and the shape of the point cloud cluster at the same time. In addition, the characteristic fusion process also comprises the step of fusing the characteristic information of each point cloud cluster to reduce the characteristic dimensionality of the point cloud cluster, so that the calculation amount in the subsequent clustering process is reduced. For example, the process of the feature fusion process can be represented by the following formula:
Di＝MLP(cat[Flocal_i,Fglobal_i])
in the above formula, DiAnd representing the comprehensive characteristic information of the ith cluster of point cloud. The shape feature information and the position feature information are effectively merged through MLP, namely MLP (cat [ F)local_i,Fglobal_i]) And obtaining comprehensive characteristic information capable of reflecting the position and the shape of the point cloud cluster.
By carrying out the same processing on all the point cloud clusters, a feature matrix M corresponding to all the point cloud clusters can be obtained: [ F, D ], wherein F is the number of point cloud clusters, and D is a characteristic dimension.
And S110, taking the feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein the training sample is marked with the object class to which each point cloud cluster belongs.
In this embodiment, the training of the attention mechanism network model adopts a supervised learning training mode, and the class of the object to which each point cloud cluster belongs is marked in the training sample.
And S120, training the initial attention mechanism network model by using the training sample to obtain a weighting matrix subjected to weighting processing.
In this embodiment, the reason why the attention mechanism network model is adopted is as follows: the effectiveness of the attention mechanism network model is based on a basic premise that point cloud clusters of the same object have more similar characteristics than point cloud clusters of different objects, and the attention mechanism network model can realize information sharing among different point cloud clusters, namely, a certain point cloud can obtain information of other point cloud clusters, so that the point cloud clusters which are over-segmented have more similar characteristics. For example, in the attention mechanism network model processing process, the feature value of each point cloud cluster can be a weighted average value of other cluster features, and by performing such weighting processing on different point cloud clusters, the similarity between the point cloud cluster features corresponding to the same object can be increased, and the similarity between the point cloud cluster features corresponding to different objects can be decreased.
The preset neural network model is trained by using the training samples, and the obtained weighting matrix subjected to weighting processing can be represented by the following formula:
M’＝Transformer(M)
wherein, M is an input transform network model used for representing a characteristic matrix where all point cloud clusters are located, and M' is a weighting matrix obtained after weighting processing. In the existing quadratic clustering method, a transformer network model is not adopted to carry out weighting processing on the point cloud cluster characteristics, but characteristic calculation is carried out according to the point cloud related position and the absolute position of the cluster. Therefore, the existing method is difficult to effectively increase or reduce the similarity of the characteristics between different clusters by simply using the self information of the point cloud clusters, and the secondary clustering can be carried out only according to the self characteristics of the point cloud of each cluster which is calculated at the beginning, and the characteristic information and the global information of other clusters are lacked, which can seriously affect the effect of the subsequent secondary clustering. In the embodiment of the invention, the attention mechanism network model can effectively combine the information of the whole frame of point cloud and enable each point cloud to acquire the information of other point cloud clusters, thereby being more beneficial to increasing the similarity between the characteristics of different point cloud clusters belonging to the same object and reducing the similarity between the point cloud clusters belonging to different object points.
Specifically, taking an over-divided long truck as an example, the long truck is divided into a head and a tail. By adopting the attention mechanism network in the embodiment, a single point cloud cluster can contain the feature information of other point cloud clusters, and compared with a mode of continuously increasing the similarity of the features of the train head and the train tail only through the constraint of a loss function, the attention mechanism network can be used for increasing the similarity of the features of the train head and the train tail in the network, so that the effectiveness of secondary clustering is increased.
And S130, determining a predicted point cloud cluster belonging to the same object with each point cloud cluster in the weighting matrix according to the distance relationship between the point cloud cluster and other point cloud clusters.
And determining the point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.
And S140, in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining the trained attention mechanism network model.
In the training process of the model, after each batch of training samples are sent into the model, a predicted value, namely a predicted point cloud cluster belonging to the same object, is output through forward propagation. Then, the difference value between the predicted value and the actual value, namely the loss function value, is calculated through the loss function. After the loss function value is obtained, the model updates each parameter through back propagation to reduce the loss between the real value and the predicted value, so that the predicted value generated by the model is close to the real value direction, and the model training is completed until the loss function value is converged.
According to the technical scheme, the attention mechanism network model is trained, in the secondary clustering process of point cloud data, the trained attention mechanism network model is utilized to perform weighting processing on each point cloud cluster, the similarity between the point cloud cluster characteristics corresponding to the same object is increased, the similarity between the point cloud cluster characteristics corresponding to different objects is reduced, under the premise of reducing under-segmentation, the over-segmentation condition is reduced, accurate clustering of the point cloud clusters is achieved, the condition that the application range possibly caused by artificial feature design is limited is avoided, and the robustness and the practicability of the model are improved.
After the training of the attention mechanism network model is completed, the method can be applied to the secondary clustering process of the point cloud data, and refer to the contents of the following embodiments.
Example two
Fig. 3 is a flowchart of a point cloud data processing method according to an embodiment of the present invention, where the method is applicable to application scenarios such as target recognition and tracking sensing, and implements secondary clustering on a vehicle-mounted lidar point cloud. The method can be carried out by a point cloud data processing device, which can be implemented by means of software and/or hardware. As shown in fig. 3, the method provided by this embodiment includes:
and S210, fusing the characteristic information of the cloud clusters of each point to obtain a characteristic matrix.
In this embodiment, the process of fusing the feature information of each point cloud cluster may refer to the description of the above embodiments, and is not described herein again.
The quadratic clustering method provided by this embodiment is performed on the result of the primary clustering, and therefore, the number of point clouds is already smaller than the number of original point clouds by at least 2-3 orders of magnitude. Specifically, taking a mechanical laser radar with 32 lines and a horizontal resolution of 0.2 degrees as an example, the magnitude of the clustered point cloud cluster is about 300 clusters, and compared with the original point cloud, the number of the clustered point cloud cluster is reduced by nearly 200 times. Compared with a mode of directly clustering the original point cloud data by using a neural network model, the quadratic clustering method provided by the embodiment has smaller calculation amount.
It should be noted that the secondary clustering process flow provided by the embodiment of the present invention does not have requirements on the number of clusters and the number of point clouds included in each cluster of point clouds.
S220, based on the trained attention mechanism network model, weighting is carried out on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced.
In this embodiment, after the attention mechanism network model is trained, each point cloud cluster can acquire information of other point cloud clusters, so that the similarity between the characteristics of different point cloud clusters belonging to the same object can be increased, and the similarity between the point cloud clusters belonging to different object points can be reduced. The training process of the attention mechanism network model may refer to the description of the above embodiments, and will not be described herein again.
And S230, clustering the point cloud clusters subjected to weighting processing, and determining the point cloud clusters belonging to the same object.
In this embodiment, a mean-shift clustering method is preferably used, and the specific process is as follows:
and for each cloud cluster after weighting, one point cloud cluster is sequentially selected as an initial clustering center, and point cloud clusters belonging to the same object are determined according to the distance relationship between other point cloud clusters and the initial clustering center. Wherein, the distance relationship may be a euclidean distance.
In this embodiment, on the basis of the first clustering result, the feature matrix can be obtained by fusing the feature information of the cloud clusters of each point. By weighting the point cloud clusters in the feature matrix based on the attention mechanism network model, a single point cloud cluster can contain feature information of other point cloud clusters, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced. By clustering the point cloud clusters after weighting processing again, the target point cloud clusters belonging to the same object can be determined, the problem of over-segmentation in the point cloud clustering process is effectively solved on the premise of avoiding under-segmentation, and the effectiveness of the point cloud clustering result is effectively improved.
EXAMPLE III
Fig. 4 is a flowchart of a point cloud data processing method according to a second embodiment of the present invention, where on the basis of the second embodiment of the present invention, the steps in the foregoing embodiment of the present invention, that is, "fusing feature information of cloud clusters of each point to obtain a feature matrix" and "performing clustering processing on point cloud clusters after weighting processing to determine point cloud clusters belonging to the same object" are refined, as shown in fig. 4, the method according to the present embodiment includes:
s310, splicing the characteristic information of each point cloud cluster, and combining the characteristic information of each point cloud cluster after splicing to form a splicing matrix.
In this embodiment, before the feature information of each point cloud cluster is fused, the shape feature and the position feature of each point cloud cluster can be spliced, that is, the shape feature and the position feature are combined into the same feature space, so as to facilitate subsequent feature fusion operation.
Specifically, the shape feature and the position feature correspond to one matrix before the stitching, and the stitching process is to stitch the two matrices into the same matrix, for example, if there are 5 clusters of point clouds, the position feature corresponds to a 5 × 4 matrix, the shape feature corresponds to a 5 × 10 matrix, and the stitched matrix is a 5 × 14 matrix.
And S320, performing feature fusion processing on the spliced matrix based on the multilayer perceptron MLP to obtain a feature matrix.
For example, the process of obtaining the feature matrix may be implemented as follows:
for any point cloud cluster, performing feature fusion processing on a splicing matrix corresponding to the point cloud cluster based on a multilayer perceptron MLP to obtain comprehensive feature information of the point cloud cluster;
and performing the characteristic fusion processing on all the point cloud clusters according to the number of the point cloud clusters to obtain characteristic matrixes corresponding to all the point cloud clusters.
The process of performing feature fusion processing by MLP can be expressed by the following formula:
Di＝MLP(cat[Flocal_i,Fglobal_i])
in the above formula, DiAnd representing the comprehensive characteristic information of the ith cluster of point cloud. The shape feature information and the position feature information are effectively merged through MLP, namely MLP (cat [ F)local_i,Fglobal_i]) To obtain energyAnd reflecting the comprehensive characteristic information of the position and the shape of the point cloud cluster.
By carrying out the same processing on all the point cloud clusters, a feature matrix M corresponding to all the point cloud clusters can be obtained: [ F, D ], wherein F is the number of point cloud clusters, and D is a characteristic dimension.
S330, based on the trained attention mechanism network model, weighting the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced.
And S340, sequentially selecting one point cloud cluster from the weighted cloud clusters as an initial clustering center, and calculating Euclidean distances between other point cloud clusters and the initial clustering center.
And S350, if the Euclidean distance value is smaller than a preset clustering threshold value, all point cloud clusters in the preset clustering threshold value range are used as candidate point cloud clusters which belong to the same object with the initial clustering center.
The preset clustering threshold is a parameter value obtained in the training process of the attention mechanism network model, and is fixed after the training of the attention mechanism network model is completed.
And S360, taking the average value of the distances of the candidate point cloud clusters as a new clustering center, returning to execute the operation of calculating the Euclidean distances between other point cloud clusters and the new clustering center, and taking all candidate point cloud clusters which belong to the same object with the new clustering center as target point cloud clusters when the determined new clustering center is not changed.
In the embodiment, the position of the clustering center can be updated by using the average value of the distance of each candidate point cloud cluster as a new clustering center, so that the accuracy of determining the clustering center is improved, and the accuracy of determining the target point cloud cluster can be improved.
According to the technical scheme, the characteristic information of the point cloud cluster is spliced, and the splicing matrix is subjected to characteristic fusion, so that the obtained characteristic matrix can reflect the comprehensive characteristics of the position and the shape of the point cloud cluster at the same time, the characteristic dimension of the point cloud cluster can be reduced, and the calculation amount in the subsequent clustering process is reduced. In the secondary clustering process, the accuracy of determining the target point cloud cluster can be improved by updating the position of the clustering center.
Example four
Fig. 5 is a block diagram of a point cloud data processing apparatus according to a fourth embodiment of the present invention, as shown in fig. 5, the apparatus includes: a feature fusion module 410, a weighting processing module 420 and a secondary clustering module 430; wherein the content of the first and second substances,
a feature fusion module 410 configured to: fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information;
a weighting processing module 420 configured to: based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced;
a quadratic clustering module 430 configured to: and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object.
Optionally, the attention mechanism network model is obtained by training in the following manner:
taking a feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein object categories to which the point cloud clusters belong are marked in the training sample;
training an initial attention mechanism network model by using the training sample to obtain a weighted matrix subjected to weighting processing;
for each point cloud cluster in the weighting matrix, determining a predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters;
and in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining a trained attention mechanism network model.
Optionally, the attention mechanism network model is a preamble codec predictor transform network model.
Optionally, the feature fusion module 410 includes:
a stitching unit configured to: splicing the characteristic information of each point cloud cluster, and combining the characteristic information of each point cloud cluster after splicing to form a splicing matrix;
a feature fusion unit configured to: and performing feature fusion processing on the spliced matrix based on the multi-layer perceptron MLP to obtain a feature matrix.
Optionally, the feature fusion unit is specifically configured to:
for any point cloud cluster, performing feature fusion processing on a splicing matrix corresponding to the point cloud cluster based on a multilayer perceptron MLP to obtain comprehensive feature information of the point cloud cluster;
and performing the feature fusion processing on all the point cloud clusters according to the number of the point cloud clusters to obtain a feature matrix corresponding to all the point cloud clusters, wherein the feature matrix comprises the number and feature dimensions of the point cloud clusters.
Optionally, the secondary clustering module 430 includes:
a target point cloud cluster determination unit configured to: and for each cloud cluster after weighting, sequentially selecting one point cloud cluster as an initial clustering center, and determining target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.
Optionally, the target point cloud cluster determining unit is specifically configured to:
calculating Euclidean distances between other point cloud clusters and the initial clustering center;
if the Euclidean distance value is smaller than a preset clustering threshold value, all point cloud clusters in the preset clustering threshold value range are used as candidate point cloud clusters which belong to the same object with the initial clustering center;
and taking the average value of the distances of the candidate point cloud clusters as a new clustering center, returning to execute the operation of calculating the Euclidean distances between other point cloud clusters and the new clustering center, and taking all candidate point cloud clusters which belong to the same object with the new clustering center as target point cloud clusters when the determined new clustering center is not changed.
Optionally, the clustering process on the original point cloud data includes:
preprocessing original point cloud data, wherein the preprocessing comprises ground point elimination and down-sampling;
clustering the preprocessed point clouds to obtain a plurality of point cloud clusters;
the original point cloud data is obtained by analyzing the laser radar original data, and the original point cloud data comprises the number of all point clouds of a current frame, three-dimensional coordinates of the point clouds and reflection intensity.
The point cloud data processing device provided by the embodiment of the invention can execute the point cloud data processing method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method. For details of the point cloud data processing method provided in any embodiment of the present invention, reference may be made to the above-mentioned embodiments.
EXAMPLE five
Referring to fig. 6, fig. 6 is a schematic structural diagram of a computing device according to a fifth embodiment of the present invention. As shown in fig. 6, the computing device may include:
a memory 701 in which executable program code is stored;
a processor 702 coupled to the memory 701;
the processor 702 calls the executable program code stored in the memory 701 to execute the point cloud data processing method provided by any embodiment of the present invention.
The embodiment of the invention discloses a computer-readable storage medium which stores a computer program, wherein the computer program enables a computer to execute a point cloud data processing method provided by any embodiment of the invention.
In various embodiments of the present invention, it should be understood that the sequence numbers of the above-mentioned processes do not imply an inevitable order of execution, and the execution order of the processes should be determined by their functions and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
In the embodiments provided herein, it should be understood that "B corresponding to A" means that B is associated with A from which B can be determined. It should also be understood, however, that determining B from a does not mean determining B from a alone, but may also be determined from a and/or other information.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated units, if implemented as software functional units and sold or used as a stand-alone product, may be stored in a computer accessible memory. Based on such understanding, the technical solution of the present invention, which is a part of or contributes to the prior art in essence, or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a memory and includes several requests for causing a computer device (which may be a personal computer, a server, a network device, or the like, and may specifically be a processor in the computer device) to execute part or all of the steps of the above-described method of each embodiment of the present invention.
It will be understood by those skilled in the art that all or part of the steps in the methods of the embodiments described above may be implemented by hardware instructions of a program, and the program may be stored in a computer-readable storage medium, where the storage medium includes Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM), or other Memory, such as a magnetic disk, or a combination thereof, A tape memory, or any other medium readable by a computer that can be used to carry or store data.
Those of ordinary skill in the art will understand that: the figures are merely schematic representations of one embodiment, and the blocks or flow diagrams in the figures are not necessarily required to practice the present invention.
Those of ordinary skill in the art will understand that: modules in the devices in the embodiments may be distributed in the devices in the embodiments according to the description of the embodiments, or may be located in one or more devices different from the embodiments with corresponding changes. The modules of the above embodiments may be combined into one module, or further split into multiple sub-modules.
Finally, it should be noted that: the above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A point cloud data processing method is characterized by comprising the following steps:
fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information;
based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced;
and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object.
2. The method of claim 1, wherein the attention mechanism network model is trained by:
taking a feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein the training sample is marked with the category of an object to which each point cloud cluster belongs;
training an initial attention mechanism network model by using the training sample to obtain a weighted matrix after weighting;
for each point cloud cluster in the weighting matrix, determining a predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters;
and in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining a trained attention mechanism network model.
3. The method of claim 1 or 2, wherein the attention mechanism network model is a forward-coded predictor transform network model.
4. The method according to claim 1, wherein the fusing the feature information of the cloud clusters of the points to obtain a feature matrix comprises:
splicing the characteristic information of each point cloud cluster, and combining the characteristic information of each point cloud cluster after splicing to form a splicing matrix;
and performing feature fusion processing on the spliced matrix based on the multi-layer perceptron MLP to obtain a feature matrix.
5. The method according to claim 4, wherein the performing feature fusion processing on the mosaic matrix based on multi-layer perceptron MLP to obtain a feature matrix comprises:
for any point cloud cluster, performing feature fusion processing on a splicing matrix corresponding to the point cloud cluster based on a multilayer perceptron MLP to obtain comprehensive feature information of the point cloud cluster;
and performing the feature fusion processing on all the point cloud clusters according to the number of the point cloud clusters to obtain a feature matrix corresponding to all the point cloud clusters, wherein the feature matrix comprises the number and feature dimensions of the point cloud clusters.
6. The method of claim 1, wherein the clustering process is performed on the point cloud clusters after the weighting process, and determining the point cloud clusters belonging to the same object comprises:
and for each cloud cluster after weighting, sequentially selecting one point cloud cluster as an initial clustering center, and determining target point cloud clusters belonging to the same object according to the distance relationship between other point cloud clusters and the initial clustering center.
7. The method of claim 6, wherein determining the target point cloud clusters belonging to the same object according to the distance relationship between the other point cloud clusters and the initial clustering center comprises:
calculating Euclidean distances between other point cloud clusters and the initial clustering center;
if the Euclidean distance value is smaller than a preset clustering threshold value, all point cloud clusters in the preset clustering threshold value range are used as candidate point cloud clusters which belong to the same object with the initial clustering center;
and taking the average value of the distances of the candidate point cloud clusters as a new clustering center, returning to execute the operation of calculating the Euclidean distances between other point cloud clusters and the new clustering center, and taking all candidate point cloud clusters which belong to the same object with the new clustering center as target point cloud clusters when the determined new clustering center is not changed.
8. The method of claim 1, wherein clustering the raw point cloud data comprises:
preprocessing original point cloud data, wherein the preprocessing comprises ground point elimination and down-sampling;
clustering the preprocessed point clouds to obtain a plurality of point cloud clusters;
the original point cloud data is obtained by analyzing the laser radar original data, and the original point cloud data comprises the number of all point clouds of a current frame, three-dimensional coordinates of the point clouds and reflection intensity.
9. A point cloud data processing apparatus, comprising:
a feature fusion module configured to: fusing the characteristic information of each point cloud cluster to obtain a characteristic matrix; the method comprises the steps that point cloud clusters are obtained by clustering original point cloud data, and the characteristic information comprises shape characteristic information and position characteristic information;
a weighting processing module configured to: based on the trained attention mechanism network model, carrying out weighting processing on the point cloud clusters in the feature matrix, so that the similarity between the point cloud cluster features corresponding to the same object is increased, and the similarity between the point cloud cluster features corresponding to different objects is reduced;
a quadratic clustering module configured to: and clustering the point cloud clusters subjected to weighting processing to determine the point cloud clusters belonging to the same object.
10. The apparatus of claim 9, wherein the attention mechanism network model is trained by:
taking a feature matrix of the point cloud clusters obtained after feature fusion as a training sample, wherein object categories to which the point cloud clusters belong are marked in the training sample;
training an initial attention mechanism network model by using the training sample to obtain a weighted matrix subjected to weighting processing;
for each point cloud cluster in the weighting matrix, determining a predicted point cloud cluster belonging to the same object as the point cloud cluster according to the distance relationship between the point cloud cluster and other point cloud clusters;
and in the training process, when the loss function value of the similarity between the predicted point cloud cluster and the corresponding object real point cloud cluster is converged, obtaining a trained attention mechanism network model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111133410.XA CN113888748A (en) | 2021-09-27 | 2021-09-27 | Point cloud data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111133410.XA CN113888748A (en) | 2021-09-27 | 2021-09-27 | Point cloud data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113888748A true CN113888748A (en) | 2022-01-04 |
Family
ID=79006966
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111133410.XA Pending CN113888748A (en) | 2021-09-27 | 2021-09-27 | Point cloud data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113888748A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114442101A (en) * | 2022-01-28 | 2022-05-06 | 南京慧尔视智能科技有限公司 | Vehicle navigation method, device, equipment and medium based on imaging millimeter wave radar |
| CN114596555A (en) * | 2022-05-09 | 2022-06-07 | 新石器慧通（北京）科技有限公司 | Obstacle point cloud data screening method and device, electronic equipment and storage medium |
| WO2023165290A1 (en) * | 2022-03-04 | 2023-09-07 | 华为技术有限公司 | Data processing method and apparatus, and electronic device and storage medium |
| CN117475170A (en) * | 2023-12-22 | 2024-01-30 | 南京理工大学 | FPP-based high-precision point cloud registration method guided by local-global structure |
-
2021
- 2021-09-27 CN CN202111133410.XA patent/CN113888748A/en active Pending
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114442101A (en) * | 2022-01-28 | 2022-05-06 | 南京慧尔视智能科技有限公司 | Vehicle navigation method, device, equipment and medium based on imaging millimeter wave radar |
| CN114442101B (en) * | 2022-01-28 | 2023-11-14 | 南京慧尔视智能科技有限公司 | Vehicle navigation method, device, equipment and medium based on imaging millimeter wave radar |
| WO2023165290A1 (en) * | 2022-03-04 | 2023-09-07 | 华为技术有限公司 | Data processing method and apparatus, and electronic device and storage medium |
| CN114596555A (en) * | 2022-05-09 | 2022-06-07 | 新石器慧通（北京）科技有限公司 | Obstacle point cloud data screening method and device, electronic equipment and storage medium |
| CN114596555B (en) * | 2022-05-09 | 2022-08-30 | 新石器慧通（北京）科技有限公司 | Obstacle point cloud data screening method and device, electronic equipment and storage medium |
| CN117475170A (en) * | 2023-12-22 | 2024-01-30 | 南京理工大学 | FPP-based high-precision point cloud registration method guided by local-global structure |
| CN117475170B (en) * | 2023-12-22 | 2024-03-22 | 南京理工大学 | FPP-based high-precision point cloud registration method guided by local-global structure |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113888748A (en) | Point cloud data processing method and device | |
| Chen et al. | Deep learning for autonomous ship-oriented small ship detection | |
| CN113076871A (en) | Fish shoal automatic detection method based on target shielding compensation | |
| CN113761999A (en) | Target detection method and device, electronic equipment and storage medium | |
| CN115797736B (en) | Training method, device, equipment and medium for target detection model and target detection method, device, equipment and medium | |
| CN109993753A (en) | The dividing method and device of urban function region in remote sensing image | |
| Li et al. | Gated auxiliary edge detection task for road extraction with weight-balanced loss | |
| CN111311611A (en) | Real-time three-dimensional large-scene multi-object instance segmentation method | |
| CN115457492A (en) | Target detection method and device, computer equipment and storage medium | |
| CN116433903A (en) | Instance segmentation model construction method, system, electronic equipment and storage medium | |
| Yao et al. | Mask guided GAN for density estimation and crowd counting | |
| Idicula et al. | A novel sarnede method for real-time ship detection from synthetic aperture radar image | |
| CN115810115B (en) | Fusion method of image and multi-frame millimeter wave radar target based on image characteristics | |
| CN116861262A (en) | Perception model training method and device, electronic equipment and storage medium | |
| CN116310832A (en) | Remote sensing image processing method, device, equipment, medium and product | |
| CN115240168A (en) | Perception result obtaining method and device, computer equipment and storage medium | |
| CN114155524A (en) | Single-stage 3D point cloud target detection method and device, computer equipment and medium | |
| CN112749293A (en) | Image classification method and device and storage medium | |
| CN113569600A (en) | Method and device for identifying weight of object, electronic equipment and storage medium | |
| CN116821699B (en) | Perception model training method and device, electronic equipment and storage medium | |
| WO2024045942A1 (en) | Ambient information sensing method, apparatus, and system, computer device, and storage medium | |
| CN114444597B (en) | Visual tracking method and device based on progressive fusion network | |
| CN113822375B (en) | Improved traffic image target detection method | |
| CN116246128B (en) | Training method and device of detection model crossing data sets and electronic equipment | |
| CN116704264B (en) | Animal classification method, classification model training method, storage medium, and electronic device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |