CN112396068A - Point cloud data processing method and device and electronic equipment - Google Patents
Point cloud data processing method and device and electronic equipment Download PDFInfo
- Publication number
- CN112396068A CN112396068A CN202110066231.2A CN202110066231A CN112396068A CN 112396068 A CN112396068 A CN 112396068A CN 202110066231 A CN202110066231 A CN 202110066231A CN 112396068 A CN112396068 A CN 112396068A
- Authority
- CN
- China
- Prior art keywords
- point
- cloud data
- point cloud
- points
- data set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method and a device for processing point cloud data and electronic equipment, and relates to the technical field of data processing, wherein the method comprises the following steps: acquiring an original point cloud data set, extracting characteristic information of each point in the point cloud data set, and sampling the original point cloud data set to generate a key point set of the original point cloud data set; for each key point in the key point set, calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point sets to generate feature points corresponding to the key points; and counting the characteristic points corresponding to each key point to generate a characteristic point set of the original point cloud data set. According to the point cloud data processing method and device and the electronic equipment, the manual intervention process is reduced, so that the possibility of wrong feature extraction can be reduced, the feature robustness is enhanced, and the final perception performance is improved.
Description
Technical Field
The invention belongs to the technical field of data processing, and particularly relates to a point cloud data processing method and device and electronic equipment.
Background
With the rapid development and cost reduction of 3D sensor technologies (laser radar, RGB-D cameras, multi-view cameras, etc.), more and more autonomous driving apparatuses start to use 3D sensors as indispensable perception sensors. Compare traditional 2D sensor (like color camera), abundant geometric position information in the traffic environment can be acquireed to the 3D sensor, promotes unmanned vehicles's perception performance, and then makes the security and the high efficiency of autopilot equipment all obtain guaranteeing.
Generally, when a 3D sensor detects a target, three-dimensional point cloud data is obtained, and due to the characteristics of the point cloud data such as sparsity, disorder, and rotational invariance, the existing sensing method based on the point cloud data mainly converts the point cloud data into dense, ordered, and structured data (e.g., rasterization and voxelization), and then uses the conventional 2D convolution method to perform feature extraction on the structured point cloud data and complete the sensing task. In the method, loss of original point cloud information is inevitably caused in the point cloud structuring process, extracted features are not robust enough when the 2D convolution method is used on 3D data, and the convolution method in the prior art mostly needs more manual intervention and is difficult to guarantee the accuracy and universal applicability of calculation, so that the final perception performance is influenced.
Disclosure of Invention
In view of the above, the present invention provides a method, an apparatus and an electronic device for processing point cloud data to alleviate the above technical problems.
In a first aspect, an embodiment of the present invention provides a method for processing point cloud data, including: acquiring an original point cloud data set, wherein the original point cloud data set is generated after a 3D sensor senses the surrounding environment; performing the following convolution operations on the original point cloud data set: extracting feature information of each point in the point cloud data set, wherein the feature information comprises a category of each point, an offset vector of each point relative to a target center, and an uncertain parameter of each point to the offset vector; sampling the original point cloud data set to generate a key point set of the original point cloud data set; for each keypoint of the set of keypoints, performing the following process: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point set to generate feature points corresponding to the key points; and counting the characteristic points corresponding to each key point to generate a characteristic point set of the original point cloud data set.
Preferably, in a possible embodiment, the method further comprises: determining the set of feature points as a set of input points for the convolution operation; circularly executing the convolution operation until the execution times of the convolution operation reach a preset time threshold value; and determining the corresponding characteristic point set as a target point set when the convolution operation is finished so as to detect the target.
Preferably, in a possible implementation, the step of determining the neighboring point set of the keypoint based on the gaussian distance includes: determining the point with the Gaussian distance larger than a preset distance threshold value as an adjacent point of the key point; adding the neighboring points to the set of neighboring points for the keypoint to generate a set of neighboring points for the keypoint.
Preferably, in a possible implementation, the step of extracting feature information of each point in the point cloud data set includes: and inputting the original point cloud data set to a pre-trained feature extraction neural network, and outputting feature information of each point in the point cloud data set through the feature extraction neural network.
Preferably, in a possible implementation, the step of sampling the original point cloud data set to generate the key point set of the original point cloud data set includes: acquiring a preset sampling algorithm; extracting a plurality of key points included in the original point cloud data set through the sampling algorithm; determining a plurality of the keypoints as points in the set of keypoints to generate the set of keypoints; and the key point is a point corresponding to the convolution center position of the convolution operation.
Preferably, in a possible implementation manner, the step of performing feature aggregation on the neighboring point set to generate a feature point corresponding to the key point includes: inputting the adjacent point set into a pre-trained feature aggregation network; and performing feature aggregation on the adjacent point sets through the feature aggregation network to output feature points corresponding to the key points.
In a second aspect, an embodiment of the present invention further provides a device for processing point cloud data, including: the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring an original point cloud data set, and the original point cloud data set is generated after a 3D sensor senses the surrounding environment; a convolution module to perform a convolution operation on the original point cloud dataset: wherein the convolution module comprises: an extraction unit, configured to extract feature information of each point in the point cloud data set, where the feature information includes a category of each point, an offset vector of each point with respect to a target center, and an uncertainty parameter of each point to the offset vector; the sampling unit is used for sampling the original point cloud data set to generate a key point set of the original point cloud data set; a feature aggregation unit, configured to perform, for each keypoint in the set of keypoints, the following processes: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point set to generate feature points corresponding to the key points; and the statistical unit is used for counting the characteristic points corresponding to each key point and generating a characteristic point set of the original point cloud data set.
Preferably, in a possible implementation, the above apparatus further comprises: a loop module for determining the feature point set as an input point set of the convolution operation; circularly executing the convolution operation until the execution times of the convolution operation reach a preset time threshold value; and determining the corresponding characteristic point set as a target point set when the convolution operation is finished so as to detect the target.
In a third aspect, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the method according to the first aspect.
In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the method according to the first aspect.
The embodiment of the invention has the following beneficial effects:
according to the point cloud data processing method, the point cloud data processing device and the electronic equipment, after an original point cloud data set is obtained, convolution operation can be conducted on the original point cloud data set, specifically, the convolution operation comprises the steps of extracting feature information of each point in the point cloud data set, sampling processing is conducted on the original point cloud data set, a key point set of the original point cloud data set is generated, and then for each key point in the key point set, the Gaussian distance from all the points in the original point cloud data set to the key point is calculated according to the feature information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point sets to generate feature points corresponding to the key points; and finally, counting the characteristic points corresponding to each key point to generate a characteristic point set of the original point cloud data set, wherein the mode of determining the adjacent point sets of the key points based on the Gaussian distance can automatically screen out points with similar characteristics to extract the characteristics, so that different key points can have a self-adaptive characteristic extraction range, and manual intervention is not needed to set the range of convolution operation.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
Fig. 1 is a flowchart of a method for processing point cloud data according to an embodiment of the present invention;
fig. 2 is a flowchart of another method for processing point cloud data according to an embodiment of the present invention;
fig. 3 is a flow chart of processing point cloud data according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a device for processing point cloud data according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of another apparatus for processing point cloud data according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, a machine learning (neural network) method can realize target detection, tracking, segmentation and the like on a traffic environment, so that the machine learning method is an important component of current automatic driving environment perception.
While a common point-based convolution operation typically includes the following: for example, the Point internet method, the Point sift method, the Point cloud core Point Convolution method (Kernel Point Convolution, KPConv), the RS-CNN (relationship-Shape CNN, RS-CNN) neural network, and the RandLA-Net neural network, when performing Convolution operation, are mostly divided into three steps: 1) selecting key points; 2) selecting a certain number of adjacent points in a certain range of the key points, and extracting and aggregating the point characteristics of the local areas; 3) repeating 1) and 2) multiple times, thereby covering the convolution operation to the entire point cloud range.
However, the method directly extracts the features on the point cloud data, so that the defects of information loss, insufficient robustness of the features and the like are overcome. However, such methods require a lot of manual intervention, for example, in step 2), a certain range and a certain number of points are required to be set manually. Different targets to be detected have different geometric characteristics, and the artificially set uniform parameters cannot ensure the effective perception range and can cause wrong feature extraction, thereby influencing the final perception performance.
Based on this, the embodiments of the present invention provide a method and an apparatus for processing point cloud data, and an electronic device, so as to alleviate the above technical problems.
For the convenience of understanding the embodiment, a detailed description will be given to a method for processing point cloud data disclosed in the embodiment of the present invention.
In a possible implementation manner, an embodiment of the present invention provides a method for processing point cloud data, and in particular, the method for processing point cloud data provided by the embodiment of the present invention may be applied to a vehicle controller, or a background server in communication connection with the vehicle controller, and may be used to process point cloud data acquired by a 3D sensor on a vehicle, so as to perform target detection, tracking, segmentation, and the like on a traffic environment, thereby achieving an automatic driving purpose of the vehicle.
Specifically, as shown in fig. 1, a flow chart of a method for processing point cloud data includes the following steps:
step S102, acquiring an original point cloud data set;
in step S102, the original point cloud data set is typically point cloud data generated after a 3D sensor disposed on the vehicle senses the surrounding environment, such as a laser radar sensor, an RGB-D camera sensor, a multi-view camera, and the like.
The following convolution operations may be further performed on the original point cloud data set:
step S104, extracting characteristic information of each point in the point cloud data set;
the characteristic information comprises the category of each point, an offset vector of each point relative to the center of the target, and uncertain parameters of each point to the offset vector;
step S106, sampling the original point cloud data set to generate a key point set of the original point cloud data set;
step S108, for each key point in the key point set, executing the following processes: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point sets to generate feature points corresponding to the key points;
specifically, the step S108 is actually a traversal process, i.e., traversing each keypoint in the set of keypoints, and the process of the step S108 is performed for each keypoint.
And step S110, counting the characteristic points corresponding to each key point, and generating a characteristic point set of the original point cloud data set.
In the step S108, the process of performing feature aggregation on the adjacent point sets is actually a process of compressing the adjacent point sets of the key points in a feature aggregation manner, and finally aggregating the adjacent point sets into one feature point of the key point, and in the process of the step S110, when the feature points corresponding to each key point are counted, a feature point set including a plurality of feature points can be obtained, and at this time, the number of the feature points in the feature point set is smaller than that in the original point cloud data set, and the feature points include features of the target.
Further, the processes of the steps S104 to S110 may be repeated for multiple times, that is, each feature point set may be used as an original point cloud data set for the next convolution operation, after the steps are repeated for multiple times, the key points may be gradually reduced, the features of the key points may be continuously abstracted and aggregated from the feature point set of the previous cycle, and finally, the features may be used for tasks such as target detection, segmentation, tracking, and the like as needed.
Therefore, the above method provided by the embodiment of the present invention further includes the following steps: determining the characteristic point set as an input point set of convolution operation; circularly executing the convolution operation, namely the process from the step S104 to the step S110, until the execution times of the convolution operation reach a preset time threshold value; and determining the corresponding characteristic point set at the end of the convolution operation as a target point set so as to detect the target.
Therefore, the method for processing point cloud data provided by the embodiment of the invention can perform convolution operation on an original point cloud data set after the original point cloud data set is obtained, specifically, the convolution operation includes extracting feature information of each point in the point cloud data set, sampling the original point cloud data set, generating a key point set of the original point cloud data set, and then calculating the gaussian distance from all the points in the original point cloud data set to the key point according to the feature information for each key point in the key point set; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point sets to generate feature points corresponding to the key points; and finally, counting the characteristic points corresponding to each key point to generate a characteristic point set of the original point cloud data set, wherein the mode of determining the adjacent point sets of the key points based on the Gaussian distance can automatically screen out points with similar characteristics to extract the characteristics, so that different key points can have a self-adaptive characteristic extraction range, and manual intervention is not needed to set the range of convolution operation.
In practical use, taking vehicle driving as an example, in the vehicle driving process, a 3D sensor arranged on a vehicle can detect data in a sensing range in real time to obtain geometric position information of rich targets in a traffic environment. In general, the raw point cloud data set obtained from the lidar sensor includes a large amount of point cloud data generated by the lidar sensor after sensing the surrounding environment, such as data including a normally detected target and data including a mark of a lane line, and therefore, the raw point cloud data set is usually represented by a set including a plurality of points, for example, by a set P, that is, the raw point cloud data set is represented by the set PP={p 1，p 2，…，p nN = {1,2, …, N }, where N is the number of points in the point cloud, and a certain point in the set Pp i={x i，y i，z iAnd represents coordinate values in a cartesian coordinate system.
After the original point cloud data set P is obtained, the method for processing point cloud data according to the embodiment of the present invention may be executed. For easy understanding, on the basis of fig. 1, fig. 2 further illustrates a flowchart of another method for processing point cloud data according to an embodiment of the present invention, and as shown in fig. 2, the method includes the following steps:
step S202, acquiring an original point cloud data set;
in step S202, the point cloud data of the lidar sensor is obtained as an example.
Step S204, inputting the original point cloud data set to a pre-trained feature extraction neural network, and outputting feature information of each point in the point cloud data set through the feature extraction neural network;
specifically, the pre-trained feature extraction neural network is obtained by training through a machine learning method, and supervised learning can be performed on each point in the original point cloud data set, so that the category of each point, the offset vector of each point relative to the target center, and uncertain parameters of the offset vector are output.
Step S206, acquiring a preset sampling algorithm; extracting a plurality of key points included in the original point cloud data set through the sampling algorithm;
step S208, determining a plurality of key points as points in a key point set to generate a key point set;
and the key point is a point corresponding to the convolution center position of the convolution operation.
The preset sampling algorithm includes, but is not limited to, a conventional sampling algorithm fps (farmest Point sampling), a random sampling algorithm, rasterization sampling, a sampling algorithm based on machine learning, and the like, and may be specifically set according to an actual use situation, which is not limited in this embodiment of the present invention.
Step S210, for each keypoint in the set of keypoints, performing the following process: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point sets to generate feature points corresponding to the key points;
step S212, counting the characteristic points corresponding to each key point, and generating a characteristic point set of the original point cloud data set.
In step S210, the step of determining the neighboring point set of the key point based on the gaussian distance includes: determining points with the Gaussian distance larger than a preset distance threshold value as adjacent points of the key point; adding neighboring points to the set of neighboring points for the keypoint to generate a set of neighboring points for the keypoint.
Further, in step S210, the step of performing feature aggregation on the neighboring point sets to generate feature points corresponding to the key points includes: inputting the adjacent point set into a pre-trained feature aggregation network; and performing feature aggregation on the adjacent point sets through a feature aggregation network to output feature points corresponding to the key points.
For convenience of understanding, based on the point cloud data processing method shown in fig. 1 and fig. 2, fig. 3 shows a point cloud data processing flow chart, which describes in detail a processing procedure of an original point cloud data set obtained in a lidar sensor, and specifically, as shown in fig. 3, the original point cloud data set is represented as:P={p 1，p 2，…，p nn = {1,2, …, N }, where N is the number of points in the point cloud, and a certain point in the set Pp i={x i，y i，z iIndicating coordinate values under a cartesian coordinate system, the original point cloud data set includes three targets to be sensed, two vehicles and a pedestrian, and the center of each target is represented by a cross symbol, as shown in (31) in fig. 3, specifically, the processing flow of the point cloud data in the embodiment of the present invention includes the following processes:
(1) inputting an original point cloud data set into a feature extraction neural network;
by means of machine learning, each point is supervised learned, so that the class (class) of each point, the offset vector (offset) of each point relative to the target center and the uncertainty parameter (sigma) of each point to the offset vector are output. As shown in (32), (33), (34) of fig. 3, wherein the determined classes are pedestrians, vehicles, the ground, and the like through the feature extraction neural network, and the offset vector is shown by an arrow in (33).
As shown in fig. 3. At this time, each point in the point cloud data set is not only corresponding to the coordinate value in the spacep i={x i，y i，z iBesides, the intermediate m-dimensional features acquired by a machine learning method and the learned feature information are also attached, and are expressed as:
P={p 1，p 2，…，p n}，N={1,2，…，n}；
Pi={x i ，y i，zi，f i1…f im，offset i ，class i ，σ i }，i∈N。
wherein the content of the first and second substances,p i indicating the way the ith point is represented after machine learning,f i1 …f im representing m-dimensional semantic features learned through a feature extraction neural network,offset i ，class i ，σ i representing the offset vectors, classes and uncertainty parameters learned by the feature extraction neural network.
Specifically, the m-dimensional semantic features are output in the middle layer of the feature extraction neural network,offset i ，class i ，σ i is output at the last layer of the feature extraction neural network. Wherein the content of the first and second substances,offset i andσ i as a parameter for calculating the Gaussian distance, the method helps the key points to find corresponding adjacent points,class i the method is used for supervised learning when the pre-trained network is acquired. Further, in the following steps, when finding out neighboring points for feature aggregation, the m-dimensional semantic features of the neighboring points and the key points are input to the feature aggregation network, and the specific value of m may be set according to actual calculation requirements, which is not limited in the embodiment of the present invention.
(2) And extracting key points from the original point cloud data set to serve as convolution center positions of convolution operation. After the oversampling algorithm is performed, the obtained sampling points are shown in (35) of fig. 3, where the dark color points are the key points sampled by the sampling algorithm. Here, taking FPS sampling algorithm as an example, the sampling logic is briefly described:
(a) the original point cloud data set is represented as
. At this time, the key point set is empty, denoted as S = { }.
(b) Randomly selecting a pointp k={x k，y k，z kAdd the set of key points, denoted S = { k }, compute all other points top kDistance D ofk(ii) a Wherein D iskExpressed as:
and k is any point in the original point cloud data set, and m is other points which are not k in the original point cloud data set.
(c) Selecting a point corresponding to the maximum distance and adding the point into a sampling set S = { k, m };
(d) at this time, since there is more than one point in the sampling set, when sampling from the remaining points in the original point cloud data set, a certain point is consideredp r （r∉S) The distance to the sample set S is:
so the point with the largest distance to the sample set is added to the sample set:
(e) and (d) repeating until the number of the sampling points meets the requirement.
In addition, the sampling algorithm may also adopt other sampling algorithms besides the FPS sampling algorithm, and the embodiment of the present invention is not limited to this.
(3) Acquiring the key point set, and searching for a point with a Gaussian distance larger than a preset distance threshold value for each key point in the key point set; points with gaussian distances greater than a preset distance threshold are determined as neighboring points and added to the set of neighboring points for the keypoint, specifically, as shown at (36) in fig. 3.
Specifically, by combining the above (1) and (2), the gaussian distance from each point to the key point can be obtained first, and then the point with the gaussian distance greater than the preset distance threshold is searched, taking the preset distance threshold as 0.5 as an example, this step can be expressed as,
wherein the content of the first and second substances,
；
wherein dij represents a Gaussian distance, sigma is an uncertain parameter extracted by the extracted neural network,P ja set of neighboring points representing keypoints.
(4) For each key pointp k Adjacent point set ofP jAnd performing feature aggregation by using a pre-trained feature aggregation network to obtain feature points.
As shown in (37) of fig. 3, a plurality of points within the circular area are aggregated into one post point.
Is shown as
Wherein the content of the first and second substances,
representing key pointsp k The characteristic points of (a) are set,g(x) The feature aggregation network used for feature aggregation, including but not limited to a feature aggregation network of a pointet structure, may be specifically set according to an actual use situation, and the embodiment of the present invention does not limit the use of the feature aggregation network.
(5) After feature aggregation is carried out on the adjacent point set of each key point, a new feature point set is obtained;
as shown at (38) in fig. 3, it is expressed as:
wherein l is a characteristic dimension obtained by characteristic polymerization in the above (4). Will be provided with
And (3) serving as a new feature point set as an input of the step (1).
(6) After repeating the steps (1) to (5) for multiple times, the key points are gradually reduced, the features of the key points are continuously abstracted and aggregated from the feature point set of the previous cycle, and the features can be used for tasks such as target detection, segmentation, tracking and the like according to needs. Furthermore, the number of times of using the convolution operation can be freely set according to the requirement of a perception task and the abstraction degree of the characteristics. For example, for target detection, a target object needs to be abstracted to one point for output, so the number of convolution operations should be increased appropriately according to a sensing range, and the like, which is based on an actual use case, and the embodiment of the present invention does not limit this.
In summary, compared with other point-based convolution operation modes, the point cloud data processing method provided by the embodiment of the invention enables the acquired features to be more robust and is beneficial to an automatic driving perception task due to the Gaussian distance, and the convolution operation mode does not need any manual intervention, so that the efficiency is improved, and meanwhile, the errors caused by the manual intervention are reduced.
Further, on the basis of the above embodiment, an embodiment of the present invention further provides a processing apparatus for point cloud data, specifically, a schematic structural diagram of the processing apparatus for point cloud data as shown in fig. 4, where the apparatus includes:
an obtaining module 40, configured to obtain an original point cloud data set, where the original point cloud data set is generated after a 3D sensor senses an ambient environment;
a convolution module 50 for performing a convolution operation on the original point cloud data set:
wherein the convolution module 50 comprises:
an extracting unit 502, configured to extract feature information of each point in the point cloud data set, where the feature information includes a category of each point, an offset vector of each point with respect to a target center, and an uncertainty parameter of each point to the offset vector;
a sampling unit 504, configured to perform sampling processing on the original point cloud data set to generate a key point set of the original point cloud data set;
a feature aggregation unit 506, configured to perform the following process for each keypoint in the set of keypoints: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point set to generate feature points corresponding to the key points;
a statistic unit 508, configured to count feature points corresponding to each key point, and generate a feature point set of the original point cloud data set.
Further, as shown in fig. 5, another schematic structural diagram of a processing apparatus for point cloud data includes, in addition to the structure shown in fig. 4:
a rotation module 60 for determining the feature point set as an input point set of the convolution operation; circularly executing the convolution operation until the execution times of the convolution operation reach a preset time threshold value; and determining the corresponding characteristic point set as a target point set when the convolution operation is finished so as to detect the target.
The processing device of point cloud data provided by the embodiment of the invention has the same technical characteristics as the processing method of point cloud data provided by the embodiment, so that the same technical problems can be solved, and the same technical effects can be achieved.
Further, an embodiment of the present invention further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor executes the computer program to implement the method shown in fig. 1 or fig. 2.
Further, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, where the computer program is executed by a processor to perform the method shown in fig. 1 or fig. 2.
Further, an embodiment of the present invention further provides a schematic structural diagram of an electronic device, as shown in fig. 6, which is the schematic structural diagram of the electronic device, wherein the electronic device includes a processor 101 and a memory 100, the memory 100 stores computer-executable instructions capable of being executed by the processor 101, and the processor 101 executes the computer-executable instructions to implement the method for processing the point cloud data.
In the embodiment shown in fig. 6, the electronic device further comprises a bus 102 and a communication interface 103, wherein the processor 101, the communication interface 103 and the memory 100 are connected by the bus 102.
The Memory 100 may include a high-speed Random Access Memory (RAM) and may further include a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The communication connection between the network element of the system and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the internet, a wide area network, a local network, a metropolitan area network, and the like can be used. The bus 102 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, or the like. The bus 102 may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one double-headed arrow is shown in FIG. 6, but that does not indicate only one bus or one type of bus.
The processor 101 may be an integrated circuit chip having signal processing capabilities. In implementation, the steps of the above method may be performed by integrated logic circuits of hardware or instructions in the form of software in the processor 101. The Processor 101 may be a general-purpose Processor, and includes a Central Processing Unit (CPU), a Network Processor (NP), and the like; the device can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, or a discrete hardware component. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. The steps of the method disclosed in connection with the embodiments of the present invention may be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module may be located in ram, flash memory, rom, prom, or eprom, registers, etc. storage media as is well known in the art. The storage medium is located in the memory, and the processor 101 reads information in the memory, and completes the processing method of the point cloud data of the foregoing embodiment in combination with hardware thereof.
The method and the apparatus for processing point cloud data and the computer program product of the electronic device provided by the embodiments of the present invention include a computer-readable storage medium storing a program code, where instructions included in the program code may be used to execute the method described in the foregoing method embodiments, and specific implementation may refer to the method embodiments, and will not be described herein again.
In addition, in the description of the embodiments of the present invention, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood in specific cases for those skilled in the art.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate orientations or positional relationships based on the orientations or positional relationships shown in the drawings, and are only for convenience of description and simplicity of description, but do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus, should not be construed as limiting the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art will understand that the following embodiments are merely illustrative of the present invention, and not restrictive, and the scope of the present invention is not limited thereto: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (10)
1. A method for processing point cloud data is characterized by comprising the following steps:
acquiring an original point cloud data set, wherein the original point cloud data set is generated after a 3D sensor senses the surrounding environment;
performing the following convolution operations on the original point cloud data set:
extracting feature information of each point in the point cloud data set, wherein the feature information comprises a category of each point, an offset vector of each point relative to a target center, and an uncertain parameter of each point to the offset vector;
sampling the original point cloud data set to generate a key point set of the original point cloud data set;
for each keypoint of the set of keypoints, performing the following process: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point set to generate feature points corresponding to the key points;
and counting the characteristic points corresponding to each key point to generate a characteristic point set of the original point cloud data set.
2. The method of claim 1, further comprising:
determining the set of feature points as a set of input points for the convolution operation;
circularly executing the convolution operation until the execution times of the convolution operation reach a preset time threshold value;
and determining the corresponding characteristic point set as a target point set when the convolution operation is finished so as to detect the target.
3. The method of claim 1, wherein the step of determining the set of neighboring points for the keypoint based on the gaussian distance comprises:
determining the point with the Gaussian distance larger than a preset distance threshold value as an adjacent point of the key point;
adding the neighboring points to the set of neighboring points for the keypoint to generate a set of neighboring points for the keypoint.
4. The method of claim 1, wherein the step of extracting feature information for each point in the point cloud dataset comprises:
and inputting the original point cloud data set to a pre-trained feature extraction neural network, and outputting feature information of each point in the point cloud data set through the feature extraction neural network.
5. The method of claim 1, wherein sampling the raw point cloud dataset to generate a set of keypoints for the raw point cloud dataset comprises:
acquiring a preset sampling algorithm;
extracting a plurality of key points included in the original point cloud data set through the sampling algorithm;
determining a plurality of the keypoints as points in the set of keypoints to generate the set of keypoints;
and the key point is a point corresponding to the convolution center position of the convolution operation.
6. The method according to claim 1, wherein the step of performing feature aggregation on the neighboring point set to generate the feature point corresponding to the key point comprises:
inputting the adjacent point set into a pre-trained feature aggregation network;
and performing feature aggregation on the adjacent point sets through the feature aggregation network to output feature points corresponding to the key points.
7. An apparatus for processing point cloud data, comprising:
the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring an original point cloud data set, and the original point cloud data set is generated after a 3D sensor senses the surrounding environment;
a convolution module to perform a convolution operation on the original point cloud dataset:
wherein the convolution module comprises:
an extraction unit, configured to extract feature information of each point in the point cloud data set, where the feature information includes a category of each point, an offset vector of each point with respect to a target center, and an uncertainty parameter of each point to the offset vector;
the sampling unit is used for sampling the original point cloud data set to generate a key point set of the original point cloud data set;
a feature aggregation unit, configured to perform, for each keypoint in the set of keypoints, the following processes: calculating the Gaussian distance from all points in the original point cloud data set to the key point according to the characteristic information; determining a set of neighboring points of the keypoint based on the Gaussian distance; performing feature aggregation on the adjacent point set to generate feature points corresponding to the key points;
and the statistical unit is used for counting the characteristic points corresponding to each key point and generating a characteristic point set of the original point cloud data set.
8. The apparatus of claim 7, further comprising:
a loop module for determining the feature point set as an input point set of the convolution operation; circularly executing the convolution operation until the execution times of the convolution operation reach a preset time threshold value; and determining the corresponding characteristic point set as a target point set when the convolution operation is finished so as to detect the target.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any of claims 1-6 when executing the computer program.
10. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, which computer program, when being executed by a processor, is adapted to carry out the method of any of the preceding claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110066231.2A CN112396068B (en) | 2021-01-19 | 2021-01-19 | Point cloud data processing method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110066231.2A CN112396068B (en) | 2021-01-19 | 2021-01-19 | Point cloud data processing method and device and electronic equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112396068A true CN112396068A (en) | 2021-02-23 |
| CN112396068B CN112396068B (en) | 2021-04-16 |
Family
ID=74624896
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110066231.2A Active CN112396068B (en) | 2021-01-19 | 2021-01-19 | Point cloud data processing method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112396068B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095172A (en) * | 2021-03-29 | 2021-07-09 | 天津大学 | Point cloud three-dimensional object detection method based on deep learning |
| CN113569877A (en) * | 2021-09-26 | 2021-10-29 | 苏州挚途科技有限公司 | Point cloud data processing method and device and electronic equipment |
| CN113723468A (en) * | 2021-08-06 | 2021-11-30 | 西南科技大学 | Object detection method of three-dimensional point cloud |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991468A (en) * | 2019-12-13 | 2020-04-10 | 深圳市商汤科技有限公司 | Three-dimensional target detection and intelligent driving method, device and equipment |
| CN111414953A (en) * | 2020-03-17 | 2020-07-14 | 集美大学 | Point cloud classification method and device |
-
2021
- 2021-01-19 CN CN202110066231.2A patent/CN112396068B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110991468A (en) * | 2019-12-13 | 2020-04-10 | 深圳市商汤科技有限公司 | Three-dimensional target detection and intelligent driving method, device and equipment |
| CN111414953A (en) * | 2020-03-17 | 2020-07-14 | 集美大学 | Point cloud classification method and device |
Non-Patent Citations (2)
| Title |
|---|
| YONG LI 等: "A Single Point-Based Multilevel Features Fusion and Pyramid Neighborhood Optimization Method for ALS Point Cloud Classification", 《APPLIED SCIENCES》 * |
| 朱慧 等: "基于层级边缘卷积的三维点云分类", 《中国科技论文》 * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095172A (en) * | 2021-03-29 | 2021-07-09 | 天津大学 | Point cloud three-dimensional object detection method based on deep learning |
| CN113095172B (en) * | 2021-03-29 | 2022-08-05 | 天津大学 | Point cloud three-dimensional object detection method based on deep learning |
| CN113723468A (en) * | 2021-08-06 | 2021-11-30 | 西南科技大学 | Object detection method of three-dimensional point cloud |
| CN113723468B (en) * | 2021-08-06 | 2023-08-04 | 西南科技大学 | Object detection method of three-dimensional point cloud |
| CN113569877A (en) * | 2021-09-26 | 2021-10-29 | 苏州挚途科技有限公司 | Point cloud data processing method and device and electronic equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112396068B (en) | 2021-04-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112396068B (en) | Point cloud data processing method and device and electronic equipment | |
| CN108470354B (en) | Video target tracking method and device and implementation device | |
| CN109948497B (en) | Object detection method and device and electronic equipment | |
| CN108256404B (en) | Pedestrian detection method and device | |
| CN109117825B (en) | Lane line processing method and device | |
| CN108009466B (en) | Pedestrian detection method and device | |
| WO2022111069A1 (en) | Image processing method and apparatus, electronic device and storage medium | |
| CN107622489B (en) | Image tampering detection method and device | |
| CN108399627B (en) | Video inter-frame target motion estimation method and device and implementation device | |
| CN109063776B (en) | Image re-recognition network training method and device and image re-recognition method and device | |
| CN111767819A (en) | Image identification method and device, electronic equipment and computer readable medium | |
| CN109815823B (en) | Data processing method and related product | |
| CN107578011A (en) | The decision method and device of key frame of video | |
| CN112396067B (en) | Point cloud data sampling method and device and electronic equipment | |
| CN110807461B (en) | Target position detection method | |
| CN113449690A (en) | Method and system for detecting image scene change and electronic equipment | |
| CN111680183A (en) | Object retrieval method and device, storage medium and electronic equipment | |
| CN114419599A (en) | Obstacle identification method and device and electronic equipment | |
| CN111382638A (en) | Image detection method, device, equipment and storage medium | |
| EP3764335A1 (en) | Vehicle parking availability map systems and methods | |
| CN112990009A (en) | End-to-end-based lane line detection method, device, equipment and storage medium | |
| CN112784494A (en) | Training method of false positive recognition model, target recognition method and device | |
| CN111652034A (en) | Ship retrieval method and device based on SIFT algorithm | |
| CN111932545A (en) | Image processing method, target counting method and related device thereof | |
| CN115272493B (en) | Abnormal target detection method and device based on continuous time sequence point cloud superposition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |