WO2023087526A1

WO2023087526A1 - Point cloud denoising method, electronic device, and storage medium

Info

Publication number: WO2023087526A1
Application number: PCT/CN2022/071296
Authority: WO
Inventors: 黄超; 孟泽楠
Original assignee: 上海仙途智能科技有限公司
Priority date: 2021-11-18
Filing date: 2022-01-11
Publication date: 2023-05-25
Also published as: CN115249349A; CN115249349B

Abstract

A point cloud denoising method, an electronic device, and a storage medium. The method comprises: acquiring a three-dimensional point cloud to be denoised (S101); performing down-sampling on first point cloud features extracted from the three-dimensional point cloud (S102); collecting shape feature statistics according to the down-sampled first point cloud features, and determining first point cloud features belonging to a same obstacle (S103); performing up-sampling on the first point cloud features belonging to the same obstacle to obtain second point cloud features, the dimension of the second point cloud features being the same as the dimension of the first point cloud features (S104); fusing the first point cloud features and the second point cloud features, and identifying noise points by using the fused point cloud features (S105); and removing the noise points in the three-dimensional point cloud according to the noise point identification result (S106). The method allows for accurate identification and removal of noise points.

Description

Method, electronic device and storage medium for point cloud denoising

technical field

The present application relates to the field of point cloud processing, and in particular to a method, electronic equipment and storage medium for point cloud denoising.

Background technique

With the wide application of laser radar, depth camera, millimeter-wave radar and other equipment, 3D point cloud is widely used in different fields, such as the field of automatic driving, security inspection and security, patrol inspection or disaster rescue.

Exemplarily, 3D point cloud has become another data form widely used in the field of autonomous driving besides image data. Obstacle detection can be performed through the 3D point cloud collected by a point cloud acquisition device such as lidar, so as to assist the vehicle in good path planning or driving control based on the obstacle detection result.

However, in some scenarios, there may be noise points in the 3D point cloud collected by point cloud collection devices such as lidar, resulting in inaccurate obstacle detection results. For example, in the scene where the sanitation vehicle performs watering operations, the 3D point related to the water mist in the 3D point cloud is a noise point that needs to be removed. For another example, in haze or dust weather, the 3D point in the 3D point cloud related to fine particles such as haze and dust is a noise point that needs to be removed.

Therefore, in order to improve the accuracy of obstacle detection, it is necessary to effectively remove the noise points in the 3D point cloud.

Contents of the invention

In view of this, the present application provides a point cloud denoising method, electronic equipment and a storage medium.

Specifically, the present application is achieved through the following technical solutions: In the first aspect, the embodiment of the present application provides a point cloud denoising method, the method includes: obtaining a 3D point cloud to be denoised; The first point cloud feature extracted in the point cloud is subjected to down-sampling processing; according to the first point cloud feature after down-sampling, the shape feature statistics are carried out to determine the first point cloud feature belonging to the same obstacle; The first point cloud feature performs upsampling processing to obtain the second point cloud feature; the dimension of the second point cloud feature is the same as the dimension of the first point cloud feature; the first point cloud feature and the first point cloud feature are combined The two point cloud features are fused, and the fused point cloud features are used to identify noise points; and the noise points in the three-dimensional point cloud are removed according to the noise point identification results.

In the second aspect, the embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the program, the first aspect is implemented. the method described.

In a third aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores executable instructions, and when the executable instructions are executed by a processor, the method as described in the first aspect is implemented .

In the point cloud denoising method, electronic equipment, and storage medium provided by the embodiments of the present application, in the process of identifying noise points, in order to save computing power resources of the electronic equipment, in the process of performing shape feature statistics, firstly, the three-dimensional The first point cloud feature extracted from the point cloud is down-sampled, which saves the calculation amount of the shape feature statistical process and improves the statistical efficiency. Considering that the down-sampling process will cause the loss of low-precision information, the first point cloud feature and the second point cloud feature obtained by up-sampling are fused, and the first point cloud feature is used to make up for the low-precision information lost in the down-sampling process, and then the fused point cloud feature is used to identify noise points, which is beneficial to improve The accuracy of noise point identification enables good removal of the noise points in the three-dimensional point cloud.

Description of drawings

Fig. 1 is a schematic flowchart of a point cloud denoising method shown in an exemplary embodiment of the present application.

Fig. 2 is a structural diagram of a noise point recognition model shown in an exemplary embodiment of the present application.

Fig. 3 is a schematic diagram of a running track of frame-selected noise points in a manual labeling process according to an exemplary embodiment of the present application.

Fig. 4 is a schematic structural diagram of an electronic device shown in an exemplary embodiment of the present application.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present application as recited in the appended claims.

The terminology used in this application is for the purpose of describing particular embodiments only, and is not intended to limit the application. As used in this application and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present application, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

In view of the problems in the related technologies, the embodiment of the present application provides a point cloud denoising method, which realizes effective noise point identification and noise point removal on the 3D point cloud to be denoised.

The point cloud denoising method provided in the embodiment of the present application can be applied to electronic equipment. Exemplarily, the electronic device may include a program for executing the point cloud denoising method. Exemplarily, the electronic device includes at least a memory and a processor, the memory stores executable instructions of the point cloud denoising method, and the processor can be configured to execute the executable instructions.

In an exemplary application scenario, 3D point clouds are widely used in the field of vehicle driving. Obstacle detection can be performed on the 3D point clouds collected by a point cloud collection device such as lidar, so as to assist the vehicle to perform good driving based on the obstacle detection results. Path planning or driving control. Considering that there are noise points in the 3D point cloud collected by the point cloud collection device, which lead to inaccurate obstacle detection results, the noise points include but are not limited to 3D points corresponding to water mist, gravel, dust or other fine particles, therefore, The electronic device may be a vehicle-mounted terminal, and the vehicle-mounted terminal may use the point cloud denoising method provided in the embodiment of the present application to remove noise points in the 3D point cloud after receiving the 3D point cloud collected by the point cloud collection device, thereby improving The accuracy of subsequent obstacle detection.

In another exemplary application scenario, the 3D point cloud is also widely used in the field of inspection. For example, in warehouses or large factories, the inspection robot can be equipped with a point cloud collection device. Based on the obstacle detection results, it can assist the inspection robot in planning a good inspection path or avoiding obstacles. Considering that there are noise points in the three-dimensional point cloud collected by the point cloud collection device, which lead to inaccurate obstacle detection results, the noise points include but are not limited to three-dimensional points corresponding to fine particles such as dust or dust, therefore, the electronic device It can be an inspection robot or a terminal installed on the inspection robot. The inspection robot or the terminal installed on the inspection robot can use the point cloud provided by the embodiment of the present application after receiving the three-dimensional point cloud collected by the point cloud acquisition device. The denoising method removes noise points in the three-dimensional point cloud, thereby improving the accuracy of subsequent obstacle detection.

Of course, the electronic device can also be other types of devices, and this embodiment does not impose any restrictions on this. For example, the electronic device can be a server that can receive 3D points to be denoised from other devices (such as self-driving vehicles, inspection robots) cloud, use the point cloud denoising method provided by the embodiment of the present application to remove noise points in the 3D point cloud, and return the denoised 3D point cloud to other devices.

Next, the point cloud denoising method provided by the embodiment of the present application is exemplified. Please refer to FIG. 1. FIG. 1 is a schematic flow chart of a point cloud denoising method, which can be applied to electronic devices. The method Including: in step S101, acquiring the 3D point cloud to be denoised.

In step S102, downsampling is performed on the first point cloud features extracted from the three-dimensional point cloud.

In step S103, the shape feature statistics are performed according to the down-sampled first point cloud features, and the first point cloud features belonging to the same obstacle are determined.

In step S104, the first point cloud feature belonging to the same obstacle is subjected to upsampling processing to obtain a second point cloud feature; the dimension of the second point cloud feature is the same as the dimension of the first point cloud feature .

In step S105, the first point cloud feature and the second point cloud feature are fused, and noise point recognition is performed using the fused point cloud feature.

In step S106, the noise points in the 3D point cloud are removed according to the noise point identification result.

In this embodiment, in order to save computing power resources of the electronic device during the noise point recognition process, the first point cloud feature extracted from the 3D point cloud is first down-sampled during the shape feature statistics process processing, thereby saving the calculation amount of the statistical process of shape features and improving statistical efficiency, and considering the loss of low-precision information caused by the downsampling process, the first point cloud feature and the second point cloud feature obtained by upsampling are fused processing, using the first point cloud feature to make up for the low-precision information lost in the down-sampling process, and then using the fused point cloud feature to identify noise points, which is conducive to improving the accuracy of noise point identification and realizing the three-dimensional point The noise points in the cloud are well removed.

In some embodiments, the 3D point cloud to be denoised may be collected by a point cloud collection device. Currently, mainstream point cloud collection devices are divided into two categories: active and passive. Active sensors can be divided into two types based on TOF (Time of Flight) systems and triangulation systems. The TOF system determines the true distance from the sensor to the surface of the object by measuring the time interval between the transmitted signal reaching the surface of the object and returning to the receiver. The triangulation system calculates the spatial position of a point by measuring the relationship between two sensors at different locations on the same point of the object. Passive sensors rely on image pairs or image sequences to recover 3D data from 2D image data based on camera parameters. Typical active point cloud collection devices include, but are not limited to, lidar, depth cameras (such as RGB-D cameras), millimeter-wave radars, or binocular vision sensors, etc.; typical passive point cloud collection devices include, but are not limited to, stereo Camera, SFM (structure from motion) system, SFS (shape from shading) system, etc.

Exemplarily, the electronic device is a vehicle-mounted terminal installed on a vehicle, and the vehicle-mounted terminal communicates with a point cloud acquisition device (such as a laser radar installed on the vehicle). During the process of collecting 3D point clouds, the vehicle-mounted terminal can obtain the 3D point clouds to be denoised collected by the point cloud collection device, and use the methods such as steps S101 to S106 to remove noise points. Exemplarily, the electronic device is an inspection robot, the inspection robot is equipped with a point cloud collection device, and the inspection robot can obtain the 3D point cloud to be denoised collected by the point cloud collection device, and use steps S101-S106 method for noise point removal.

Wherein, the noise points in the 3D point cloud include, but are not limited to, 3D points corresponding to at least one fine particle: water mist, gravel, dust or dust, and the like. It can be understood that this application does not impose any limitation on the type of fine particles.

In some embodiments, after the three-dimensional point cloud to be denoised is acquired, first point cloud features may be extracted from the three-dimensional point cloud to be denoised for subsequent processing.

As a possible implementation, the first point cloud features include features of several 3D points in the 3D point cloud, and the features of the 3D points include but not limited to coordinates of 3D points, reflection intensity, and corresponding The identification of the light pulse sequence (such as the radar beam ID), depth information (such as the distance between the 3D point and the point cloud collection device), height information or angle information (such as the deflection angle of the line connecting the 3D point and the point cloud collection device).

As another possible implementation method, considering that the data volume of the first point cloud feature obtained directly for the feature of the 3D point is relatively large, and it needs to consume more computing resources, so in order to improve the recognition efficiency of the noise point, it can be treated as Perform three-dimensional grid processing on the denoised three-dimensional point cloud, segment the three-dimensional point cloud according to a preset distance, and obtain a three-dimensional gridded three-dimensional point cloud; it can be understood that the preset distance can be determined according to the actual application scene The specific setting is not limited in this embodiment. For example, the preset distance includes but is not limited to 5cm, 20cm, or 1m. After obtaining the 3D gridded 3D point cloud, feature extraction can be performed on each 3D grid in the 3D gridded 3D point cloud, for example, according to the statistics of the features of the 3D points in the 3D grid The value determines the feature of the 3D mesh, so as to obtain the feature of the first point cloud based on the features of all the 3D mesh in the 3D point cloud. Wherein, the characteristics of the three-dimensional point include at least one of the following: coordinates of the three-dimensional point, reflection intensity, identification of the light pulse sequence corresponding to the three-dimensional point, depth information, height information or angle information. The statistical value includes but not limited to average value, median, maximum or minimum value, etc., for example, the characteristics of the three-dimensional grid can be determined according to the average value of the characteristics of the three-dimensional points in the three-dimensional grid, and can be determined according to the actual Specific calculations are performed in application scenarios, which is not limited in this embodiment. In this embodiment, the three-dimensional point cloud to be denoised is subjected to three-dimensional grid processing, and the features of the three-dimensional grid are extracted as the first point cloud feature, which is beneficial to reduce the amount of data involved in the calculation and improve the recognition efficiency of noise points .

In some embodiments, after acquiring the first point cloud features, in order to save computing resources for the subsequent shape feature statistics process, and on the other hand, considering the scene of limited computing power of electronic devices, the first point cloud features can be calculated first Perform downsampling processing. For example, the dimension of the first point cloud feature is 1024*1024*1024 dimensions, and the dimension of the first point cloud feature after downsampling is 256*256*256 dimensions, and then use the first point cloud feature after downsampling The shape feature statistics of point cloud features is beneficial to save computing resources. The shape feature statistics process is to detect the relative relationship between 3D points (or 3D grids) to determine whether they belong to the same obstacle. The first point after downsampling Cloud features can determine the first point cloud features belonging to each obstacle (or each 3D point or 3D grid belonging to each obstacle) by dividing and counting the shape features of cloud features, and then perform up-sampling processing to obtain the second point cloud features, so The dimension of the second point cloud feature is the same as the dimension of the first point cloud feature, such as the second point cloud feature is also 1024*1024*1024 dimensions, and finally utilizes the fusion of the first point cloud feature and the second The point cloud feature of the point cloud feature is used for noise point recognition, which is beneficial to improve the accuracy of noise point recognition.

In some embodiments, a noise point recognition model can be pre-built to identify noise points in the 3D point cloud to be denoised, for example, after obtaining the 3D point cloud to be denoised, the 3D point cloud can be input In the pre-established noise point recognition model, the noise point recognition result is obtained after the three-dimensional point cloud is processed by the noise point recognition model.

Please refer to Fig. 2, Fig. 2 has shown the architectural diagram of noise point recognition model, described noise point recognition model at least comprises feature extraction layer 22, shape recognition network 23 and noise point recognition network 24 etc., and described noise point recognition model An input layer 21 and an output layer 25 may also be included.

The input layer 21 is used to acquire the input 3D point cloud to be denoised.

The feature extraction layer 22 is used to extract features from the 3D point cloud, for example, the 3D point cloud can be segmented according to a preset distance to obtain a 3D gridded 3D point cloud, and then the 3D gridded 3D point cloud can be obtained. Each three-dimensional grid in the three-dimensional point cloud performs feature extraction to obtain the first point cloud features; wherein, the feature of each three-dimensional grid is the statistical value (such as the average value) of the characteristics of the three-dimensional points in the three-dimensional grid wait). Extracting grid features in this embodiment is beneficial to reducing the amount of data involved in subsequent calculations and improving calculation efficiency.

The shape recognition network 23 includes a coefficient convolution layer 231, a first multi-layer perceptron network 232 and an upsampling layer 233; the coefficient convolution layer 231 is used for downsampling the first point cloud features; The first multi-layer perceptron network 232 is used to perform shape feature statistics according to the downsampled first point cloud features, such as performing statistical analysis of shape features according to the positional relationship between three-dimensional grids to determine the first point cloud belonging to the same obstacle. Point cloud features, that is, divide and count the first point cloud features according to obstacles; the upsampling layer 233 is used to upsample the first point cloud features belonging to the same obstacle to obtain a second point cloud feature. In the shape recognition network 23, the first point cloud features are down-sampled, thereby saving the calculation amount of the first multi-layer perceptron network 232 in the shape feature statistics process, and improving the statistical efficiency.

The noise point recognition network 24 includes a feature fusion layer 241 and a second multilayer perceptron network 242; the feature fusion layer 241 is used to fuse the first point cloud feature and the second point cloud feature; the second point cloud feature The two-layer multi-layer perceptron network 242 is used to perform noise point recognition on the fused point cloud features to obtain noise point recognition results. In the noise point recognition network 24, considering that the downsampling process will cause the loss of low-precision information, the first point cloud feature and the second point cloud feature obtained by upsampling are fused, and the first point cloud feature is used The feature makes up for the low-precision information lost in the downsampling process, and then uses the fused point cloud features to identify noise points, which is conducive to improving the accuracy of noise point recognition.

The output layer 25 is used to output the noise point identification result. In this embodiment, when the noise point recognition model performs shape feature statistics, in order to further reduce the amount of computation and save computing resources, the first point cloud feature is down-sampled through the coefficient convolution layer 231, and then it can be applied to the calculation In devices with limited resources, the noise point recognition model is suitable for more devices and has wide applicability; and considering that the downsampling process will cause the loss of low-precision information, the noise point recognition model is used in the shape recognition network23 In the second half, an upsampling structure is designed to restore the point cloud features to the size before downsampling, and fuse the second point cloud features obtained by upsampling with the first point cloud features to make up for the low precision lost in the downsampling process. Information, and then the second multi-layer perceptron network 242 performs noise point recognition on the fused point cloud features, which is conducive to improving the accuracy of noise point recognition and achieving a trade-off between computing resources and accuracy.

Next, the training process of the noise point recognition model is exemplified, using a supervised learning method as an example. The noise point recognition model is trained based on several 3D point cloud samples marked with noise points.

First, for the acquisition of training samples, multiple single-frame 3D point cloud samples can be fused into the first dense point cloud, wherein the multiple single-frame 3D point cloud samples are collected by a point cloud mounted on a mobile platform The device is collected during the moving process, and the movable platform may be a vehicle or a mobile robot. Taking a vehicle as an example, the electronic device applying the point cloud denoising method of the embodiment of the present application may be a vehicle mounted on the vehicle. In the terminal, the multiple single-frame three-dimensional point cloud samples are collected by a point cloud collection device mounted on the vehicle during driving of the vehicle.

Exemplarily, in the fusion process, the pose of the point cloud acquisition device when collecting each frame of three-dimensional point cloud samples can be determined first through point cloud registration, and then based on the point cloud acquisition device when collecting each frame of three-dimensional point cloud samples The pose transforms multiple single-frame 3D point cloud samples into the same 3D coordinate system, and realizes the fusion process of multiple single-frame 3D point cloud samples in this 3D coordinate system.

Among them, when multiple single-frame 3D point cloud samples are fused into the first dense point cloud, static obstacles (such as trees, stopped cars) will appear in a denser state in the first dense point cloud (because more 3D points of the frame), while dynamic obstacles (such as pedestrians, moving cars) will present a moving trajectory in the first dense point cloud, corresponding to their own dynamic motion direction. Fine particles such as water mist, gravel, dust or dust, due to their small size and weight, will present different motion states according to the driving state of the movable platform (such as a vehicle), wind direction and wind force, etc. In the dense point cloud, fine particles such as water mist, gravel, dust or dust will also present a movement trajectory, corresponding to the direction of its dynamic movement relative to itself.

In a possible implementation manner, after obtaining the first dense point cloud, the 3D points belonging to the ground and the 3D points belonging to the movable platform (such as a vehicle) in the first dense point cloud may be removed to obtain The second dense point cloud; wherein, the three-dimensional point belonging to the ground is a three-dimensional point with a height of 0, and since the point cloud collection device is mounted on the movable platform (such as a vehicle), the position of the point cloud collection device is known, then it belongs to The three-dimensional points of the movable platform (such as a vehicle) can also be determined; after removing the three-dimensional points belonging to the ground and the three-dimensional points belonging to the movable platform (such as a vehicle), then in the second dense point cloud The three-dimensional points with motion tracks around the movable platform (such as vehicles) can be determined as three-dimensional points corresponding to fine particles such as water mist, gravel, dust or dust, and these three-dimensional points can be determined as noise points and marked , and then the first dense point cloud may be split according to the marked noise points to obtain a plurality of single-frame three-dimensional point cloud samples marked with noise points. In this embodiment, the automatic labeling process of noise points is realized, and multiple single-frame three-dimensional point cloud samples are fused into the first dense point cloud for one-time labeling, which is beneficial to improve labeling efficiency.

Of course, in another possible implementation, in order to improve the accuracy of noise point labeling, the user can also manually label, such as projecting the second dense point cloud into a two-dimensional space to obtain the labeled image, and the user can use experience to Mark the trajectory of fine particles such as water mist, gravel, dust or dust in the marked image. For example, you can easily use the mouse to click or use other tools such as a stylus to frame the trajectory of fine particles, such as the figure 3, the black box in Figure 3 shows the running track of the noise points framed by the manual labeling process; and a fixed height value is set, and then the electronic device can back-project the framed area in the labeled image to the three-dimensional space, to obtain marked noise points, and finally disassemble the first dense point cloud according to the marked noise points to obtain a plurality of single-frame three-dimensional point cloud samples marked with noise points. In this embodiment, the manual labeling process is conducive to improving the accuracy of noise point labeling, and multiple single-frame 3D point cloud samples are fused into the first dense point cloud for labeling, which only needs to be selected simply, without the need for each All points are labeled, which is beneficial to improve the efficiency of manual labeling.

Further, in another possible implementation manner, after obtaining the first dense point cloud, in addition to removing the 3D points belonging to the ground and the 3D points belonging to the mobile platform (such as a vehicle), it can also be based on the Obstacle identification is performed on the first dense point cloud to obtain an obstacle identification result. For example, the first dense point cloud can be input into an existing obstacle detection model, and the existing obstacle detection model is used to perform obstacle identification, and obtain The obstacle recognition result output by the obstacle detection model; then according to the obstacle recognition result, the three-dimensional points belonging to obstacles in the first dense point cloud are removed to obtain the second dense point cloud. In this embodiment, except In addition to removing ground points and 3D points of the vehicle, 3D points belonging to other obstacles (such as other vehicles, pedestrians, motorcycles, trees) will be removed to further eliminate interference factors, and the movement of mobile platforms (such as vehicles) After the obstacles on the path (such as the road) are removed, the remaining three-dimensional points with motion trajectories on the road can be determined as three-dimensional points corresponding to fine particles such as water mist, gravel, dust or dust, so the described In the second dense point cloud, the three-dimensional points with motion tracks on the moving path (such as a road) are marked as noise points, and finally the first dense point cloud is split according to the marked noise points to obtain a plurality of points marked with A single-frame 3D point cloud sample of noisy points. In this embodiment, the irrelevant factors are removed to eliminate the interference of obstacles that also have moving trajectories, which is conducive to further improving the accuracy of noise point labeling.

In some embodiments, considering that the number of samples is also related to the accuracy of model training, in order to improve the accuracy of model training, the number of samples for training can be increased. In addition to marking noise points in the above-mentioned way, also The training sample set can be enriched by at least one of the following data enhancement methods: in the first data enhancement method, the noise points in the single-frame 3D point cloud sample marked with noise points can be superimposed on other 3D point clouds In the sample, additional single-frame 3D point cloud samples labeled with noise points are obtained. For example, in a vehicle driving scene, considering that the distances of fine particles such as water mist, gravel, dust or dust to the vehicle body are similar, the marked noise points can be superimposed on the corresponding 3D points of different vehicles running in different scenes In the cloud, to expand the training data set and enhance the diversity.

In the second data enhancement manner, the positions of the noise points in the single-frame 3D point cloud samples marked with noise points may be moved to obtain additional single-frame 3D point cloud samples marked with noise points. For example, in a vehicle driving scene, considering that the distances of fine particles such as water mist, gravel, dust or dust to the vehicle body are similar, after determining the 3D points belonging to the vehicle, the marked noise points can be moved to a distance relative to the vehicle body. For example, through symmetrical inversion, or adjusting the data of noise points on a certain coordinate axis, the noise points can be symmetrically or translated to other positions relative to the vehicle, so that some vehicles are on the left, or in the The front of the car, or in the middle of the car body, there are different situations such as water mist, gravel, dust or dust.

In the third data enhancement method, all the 3D points in the single-frame 3D point cloud samples marked with noise points can be rotated by a preset angle to obtain additional single-frame 3D point cloud samples marked with noise points. For example, in a vehicle driving scene, in order to prevent the data from being fixed in a fixed direction of east, west, north, south, and causing possible bias in the model training results, all the 3D points in the single frame 3D point cloud sample marked with noise points can be Rotated by preset angles clockwise or counterclockwise to expand the training data set and enhance diversity.

In the fourth data enhancement method, a set of 3D points representing obstacles can be added to the vicinity of the noise points in the single-frame 3D point cloud sample marked with noise points to obtain additional single-frame 3D point cloud samples marked with noise points. Point cloud sample. For example, in a vehicle driving scene, a set of three-dimensional points representing obstacles such as motorcycles or pedestrians can be added near noise points, so that the ability of the noise point recognition model to distinguish when noise points intersect with other objects can be verified.

In the fifth data enhancement method, all the three-dimensional points in the single-frame three-dimensional point cloud sample marked with noise points can be randomly offset, and the three-dimensional points can be offset in any direction to realize the expansion of training data Set, enhance the diversity, and the increase of training samples can improve the robustness of the model.

In some embodiments, after acquiring several 3D point cloud samples marked with noise points, model training may be performed based on the several 3D point cloud samples marked with noise points to obtain the noise point recognition model. For example, please refer to the model structure shown in Figure 2, after the electronic device acquires several 3D point cloud samples marked with noise points, it can input several 3D point cloud samples marked with noise points as shown in Figure 2 In the default model of:

Then feature extraction is performed through the feature extraction layer of the preset model, for example, the 3D point cloud sample is subjected to 3D grid processing, the point cloud space is divided into 3D grids according to the preset distance, and the features of the 3D grid are extracted to obtain The first point cloud feature, for example, the feature of the 3D grid may be the average value of the features of the 3D points in the 3D grid.

Then, the shape recognition network in the preset model processes the first point cloud features, the sparse convolution layer downsamples the first point cloud features, and the first multi-layer perceptron network according to the downsampled first The point cloud features perform shape feature statistics to determine the first point cloud features belonging to the same obstacle, and the up-sampling layer performs up-sampling processing on the first point cloud features belonging to the same obstacle to obtain the second point cloud features.

Then, it is processed by the noise point recognition network in the preset model, the first point cloud feature and the second point cloud feature are fused by the feature fusion layer, and the fused point cloud is paired by the second multi-layer perceptron network The features are used to identify noise points to obtain noise point prediction results.

Finally, determine the difference between the noise point prediction result and the marked noise point of the 3D point cloud sample, reversely adjust the parameters used to build the preset model according to the difference, and obtain the trained noise point recognition Model. For example, the loss value corresponding to the difference between the noise point prediction result and the marked noise point of the 3D point cloud sample can be calculated based on a preset loss function, and the loss value used to construct the specified The parameters of the model.

In this embodiment, the first point cloud feature used is the grid feature extracted from the gridded point cloud, which is beneficial to reduce the amount of calculation; when the noise point recognition model performs shape feature statistics, in order to further reduce Calculation, saving computing resources, downsampling the first point cloud features through the sparse convolution layer, and then applicable to devices with limited computing resources, making the noise point recognition model applicable to more devices, with a wide range of and considering the loss of low-precision information caused by the downsampling process, the noise point recognition model designed an upsampling structure in the second half of the shape recognition network to restore the point cloud features to the size before downsampling, and the upsampling The second point cloud feature obtained by sampling is fused with the first point cloud feature to make up for the low-precision information lost in the downsampling process, and then the second multi-layer perceptron network performs noise point recognition on the fused point cloud feature, which has It is beneficial to improve the accuracy of noise point identification, and realizes the trade-off between computing resources and accuracy.

It can be understood that the training process and the application process of the noise point recognition model may be performed by the same electronic device, or may be performed by different electronic devices, which is not limited in this embodiment.

Corresponding to the above point cloud denoising method, please refer to FIG. 4 , the embodiment of the present application also provides an electronic device 30, including a memory 32, a processor 31, and an A computer program 33, used for executing the above method when the processor 31 executes the program.

The processor 31 executes the executable instructions included in the memory 32, and the processor 31 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuit (Application Specific Integrated Circuit, ASIC), off-the-shelf programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

The memory 32 stores the executable instructions of the point cloud denoising method, and the memory 32 can include at least one type of storage medium, and the storage medium includes a flash memory, a hard disk, a multimedia card, a card memory (for example, SD or DX memory, etc. etc.), Random Access Memory (RAM), Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Diskettes, CDs, etc. Also, the device may cooperate with a network storage device performing a storage function of the memory through a network connection. The storage 32 may be an internal storage unit of the device 30 , such as a hard disk or a memory of the device 30 . Memory 32 also can be the external storage device of equipment 30, for example equipped with plug-in type hard disk on equipment 30, smart memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card, flash memory card (Flash Card) wait. Further, the memory 32 may also include both an internal storage unit of the device 30 and an external storage device. The memory 32 is used to store the computer program 33 and other programs and data required by the device. The memory 32 can also be used to temporarily store data that has been output or will be output.

Exemplarily, the electronic equipment includes a vehicle-mounted terminal, the vehicle-mounted terminal is installed on a vehicle, and the vehicle is also equipped with a point cloud collection device, and the vehicle-mounted terminal is communicatively connected with the point cloud collection device (such as a laser radar installed on the vehicle), The point cloud acquisition device mounted on the vehicle collects the three-dimensional point cloud during the driving process of the vehicle, and the vehicle-mounted terminal can obtain the three-dimensional point cloud to be denoised collected by the point cloud acquisition device, and based on the point cloud denoising method provided in the embodiment of the present application Perform denoising processing.

Exemplarily, when the processor executes the program, it is used to perform the following steps: acquire the 3D point cloud to be denoised; perform downsampling processing on the first point cloud features extracted from the 3D point cloud; The first point cloud feature after sampling is subjected to shape feature statistics, and the first point cloud feature belonging to the same obstacle is determined; the first point cloud feature belonging to the same obstacle is upsampled to obtain the second point cloud feature; The dimension of the second point cloud feature is the same as the dimension of the first point cloud feature; the first point cloud feature and the second point cloud feature are fused, and the fused point cloud feature is used to perform Identifying noise points: removing the noise points in the 3D point cloud according to the noise point identification results.

Optionally, after the acquisition of the 3D point cloud to be denoised, the processor is further configured to: segment the 3D point cloud according to a preset distance to obtain a 3D gridded 3D point cloud; Each 3D grid in the gridded 3D point cloud is subjected to feature extraction to obtain the features of the first point cloud.

Optionally, the first point cloud features include features of all 3D meshes in the 3D point cloud. The characteristics of each grid are determined by statistical values of the characteristics of the three-dimensional points within the three-dimensional grid. The characteristics of the three-dimensional point include at least one of the following: coordinates of the three-dimensional point, reflection intensity, identification of the light pulse sequence corresponding to the three-dimensional point, depth information, height information or angle information.

Optionally, the noise point recognition result is obtained by inputting the 3D point cloud into a pre-established noise point recognition model, and processing the 3D point cloud through the noise point recognition model. Wherein, the noise point recognition model includes a feature extraction layer, a shape recognition network and a noise point recognition network. The first point cloud feature is obtained by feature extraction of the three-dimensional point cloud by the feature extraction layer. The shape recognition network includes a sparse convolution layer for downsampling the first point cloud features, a first multi-layer perceptron network for performing shape feature statistics according to the downsampled first point cloud features, and An upsampling layer for performing upsampling processing on the first point cloud features belonging to the same obstacle. The noise point recognition network includes a feature fusion layer for fusing the first point cloud features and the second point cloud features, and a second multilayer perception layer for performing noise point recognition on the fused point cloud features machine network.

Optionally, the noise point recognition model is trained based on several three-dimensional point cloud samples labeled with noise points.

Optionally, the processor is further configured to: fuse a plurality of single-frame three-dimensional point cloud samples into a first dense point cloud; wherein, the plurality of single-frame three-dimensional point cloud samples are collected by a point cloud mounted on a vehicle The device is collected during the driving of the vehicle; removing the 3D points belonging to the ground and the 3D points belonging to the vehicle in the first dense point cloud to obtain a second dense point cloud; The three-dimensional points with motion tracks around the vehicle are marked as noise points; the first dense point cloud is split according to the marked noise points to obtain a plurality of single-frame three-dimensional point cloud samples marked with noise points.

Optionally, the processor is further configured to: perform obstacle recognition according to the first dense point cloud to obtain an obstacle recognition result; The 3D point removal of the second dense point cloud is obtained; and the 3D point with a motion track on the road in the second dense point cloud is marked as a noise point.

Optionally, the processor is further configured to: after obtaining a plurality of single-frame 3D point cloud samples marked with noise points, obtain the 3D point cloud samples marked with noise points in at least one of the following ways: The noise points in the single-frame three-dimensional point cloud samples marked with noise points are superimposed on other three-dimensional point cloud samples; or, the position of the noise points in the single-frame three-dimensional point cloud samples marked with noise points is moved; or, Rotating all 3D points in the single-frame 3D point cloud sample marked with noise points by a preset angle; or, adding a set of 3D points representing obstacles to the single-frame 3D point cloud sample marked with noise points Near the noise point; or, randomly offset all three-dimensional points in the single-frame three-dimensional point cloud sample marked with noise points.

Optionally, the 3D point cloud to be denoised is collected by a point cloud collection device mounted on the vehicle during vehicle driving; the noise points include 3D points corresponding to at least one of the following fine particles: water mist, grit, dirt or dust.

It can be understood that more or less components than those shown in FIG. 3 may be included, some components may be combined, or different components may be included. For example, the device may also include input and output devices, network access devices, buses, and so on.

Correspondingly, in an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium including instructions, such as a memory including instructions, the instructions can be executed by a processor of the device to complete the above method. For example, the non-transitory computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

A non-transitory computer-readable storage medium, enabling the terminal to execute the above method when instructions in the storage medium are executed by a processor of the terminal.

Correspondingly, an embodiment of the present application further provides a computer program product, including the computer program according to any one of the methods described above.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or in A combination of one or more of . Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of data processing apparatus. Multiple modules. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for transmission by the data The processing means executes. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

Computers suitable for the execution of a computer program include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory and/or a random access memory. The essential components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic or magneto-optical disks, or optical disks, to receive data therefrom or to It transmits data, or both. However, a computer is not required to have such a device. Additionally, a computer can be embedded in another device, such as an autonomous vehicle, mobile robot, etc., just to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM, and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as primarily describing features of particular embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function in certain combinations as described above and even be initially so claimed, one or more features from a claimed combination may in some cases be removed from that combination and the claimed A protected combination can point to a subcombination or a variant of a subcombination.

Similarly, while operations are depicted in the figures in a particular order, this should not be construed as requiring that those operations be performed in the particular order shown, or sequentially, or that all illustrated operations be performed, to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can often be integrated together in a single software product in, or packaged into multiple software products.

Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above is only a preferred embodiment of the application, and is not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application should be included in the application. within the scope of protection.

Claims

A point cloud denoising method, comprising:

Obtain the 3D point cloud to be denoised;

The first point cloud feature extracted from the three-dimensional point cloud is subjected to down-sampling processing;

Perform shape feature statistics according to the first point cloud feature after downsampling, and determine the first point cloud feature belonging to the same obstacle;

Carrying out upsampling processing on the first point cloud feature belonging to the same obstacle to obtain a second point cloud feature; the dimension of the second point cloud feature is the same as the dimension of the first point cloud feature;

performing fusion processing on the first point cloud feature and the second point cloud feature;

Use the fused point cloud features for noise point recognition;

The noise points in the three-dimensional point cloud are removed according to the noise point identification result.
The method according to claim 1, characterized in that, after said acquiring the 3D point cloud to be denoised, further comprising:

Segmenting the 3D point cloud according to a preset distance to obtain a 3D gridded 3D point cloud;

Perform feature extraction on each 3D grid in the 3D gridded 3D point cloud to obtain the first point cloud feature.
The method according to claim 2, characterized in that,

The first point cloud features include features of all 3D meshes in the 3D point cloud;

The characteristics of the three-dimensional grid are determined by statistical values of the characteristics of the three-dimensional points in the three-dimensional grid;

The characteristics of the three-dimensional point include at least one of the following: coordinates of the three-dimensional point, reflection intensity, identification of the light pulse sequence corresponding to the three-dimensional point, depth information, height information or angle information.
The method according to claim 1, characterized in that,

The noise point recognition result is obtained by inputting the 3D point cloud into a pre-established noise point recognition model, and processing the 3D point cloud through the noise point recognition model;

Wherein, the noise point recognition model includes a feature extraction layer, a shape recognition network and a noise point recognition network;

The first point cloud feature is obtained by performing feature extraction on the three-dimensional point cloud by the feature extraction layer;

The shape recognition network includes a sparse convolution layer for downsampling the first point cloud features, a first multi-layer perceptron network for performing shape feature statistics according to the downsampled first point cloud features, and An upsampling layer for performing upsampling processing on the first point cloud features belonging to the same obstacle;

The noise point recognition network includes a feature fusion layer for fusing the first point cloud features and the second point cloud features, and a second multilayer perception layer for performing noise point recognition on the fused point cloud features machine network.
The method according to claim 4, wherein the noise point recognition model is trained based on several three-dimensional point cloud samples labeled with noise points.
The method according to claim 5, further comprising:

Fusing multiple single-frame three-dimensional point cloud samples into a first dense point cloud; wherein, the multiple single-frame three-dimensional point cloud samples are collected by a point cloud collection device mounted on the vehicle during vehicle driving;

removing the 3D points belonging to the ground and the 3D points belonging to the vehicle in the first dense point cloud to obtain a second dense point cloud;

Marking three-dimensional points with motion trajectories around the vehicle in the second dense point cloud as noise points;

The first dense point cloud is split according to the marked noise points to obtain a plurality of single-frame three-dimensional point cloud samples marked with noise points.
The method according to claim 6, further comprising:

performing obstacle identification according to the first dense point cloud to obtain an obstacle identification result;

removing the 3D points belonging to obstacles in the first dense point cloud according to the obstacle identification result to obtain the second dense point cloud;

The marking the three-dimensional points with motion tracks around the vehicle in the second dense point cloud as noise points includes:

Marking the 3D points with motion tracks on the road in the second dense point cloud as noise points.
The method according to claim 6, wherein, after obtaining a plurality of single-frame 3D point cloud samples marked with noise points, the 3D point cloud samples marked with noise points are obtained by at least one of the following methods:

Superimposing noise points in the single-frame three-dimensional point cloud sample marked with noise points into other three-dimensional point cloud samples;

Move the position of the noise point in the single-frame three-dimensional point cloud sample marked with the noise point;

Rotating all three-dimensional points in the single-frame three-dimensional point cloud sample marked with noise points by a preset angle;

Adding a set of three-dimensional points representing obstacles to the vicinity of noise points in the single-frame three-dimensional point cloud sample marked with noise points;

All the three-dimensional points in the single-frame three-dimensional point cloud sample marked with noise points are randomly offset.
The method according to any one of claims 1 to 8, characterized in that,

The three-dimensional point cloud to be denoised is collected by a point cloud collection device mounted on the vehicle during driving of the vehicle;

The noise points include three-dimensional points corresponding to at least one of the following fine particles: water mist, gravel, dust or dust.
An electronic device comprising:

memory;

processor; and

A computer program stored on the memory and operable on the processor, the processor implements the method according to any one of claims 1 to 9 when executing the program.
The device according to claim 10, wherein the electronic device comprises a vehicle-mounted terminal.
A computer-readable storage medium, on which executable instructions are stored, and when the executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is realized.