CN115131246A

CN115131246A - Method and device for denoising point cloud data, computer equipment and storage medium

Info

Publication number: CN115131246A
Application number: CN202210773245.2A
Authority: CN
Inventors: 宋潇; 王哲
Original assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Lingang Intelligent Technology Co Ltd
Priority date: 2022-07-01
Filing date: 2022-07-01
Publication date: 2022-09-30

Abstract

The disclosure provides a point cloud data denoising method, a point cloud data denoising device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring point cloud data; rasterizing the point cloud data to obtain local point cloud data contained in at least one target grid; determining the probability that the target grids comprise noise point cloud points based on the local point cloud data contained in each target grid; and updating the point cloud data based on the probability that each target grid comprises noise point cloud points to obtain updated point cloud data.

Description

Method and device for denoising point cloud data, computer equipment and storage medium

Technical Field

The disclosure relates to the technical field of computer vision, in particular to a point cloud data denoising method, a point cloud data denoising device, a computer device and a storage medium.

Background

Point cloud data acquired with a lidar can typically be used for object detection. However, in point cloud data acquired under different environments, there are usually some noise point cloud points related to the environment. For example, in point cloud data acquired in a heavy rain environment, there are usually noise point cloud points of a rain type and a splash type.

The existence of noise point cloud points will affect the accuracy of the object detection result. Therefore, how to identify and remove the noise point cloud points in the point cloud data becomes an urgent problem to be solved.

Disclosure of Invention

The embodiment of the disclosure at least provides a method and a device for denoising point cloud data, computer equipment and a storage medium.

In a first aspect, an embodiment of the present disclosure provides a method for denoising point cloud data, including:

acquiring point cloud data;

rasterizing the point cloud data to obtain local point cloud data contained in at least one target grid;

determining a probability that the target grid includes noise point cloud points based on the local point cloud data contained by each of the target grids;

and updating the point cloud data based on the probability that each target grid comprises noise point cloud points to obtain updated point cloud data.

In the embodiment, local point cloud data which correspond to each target grid and have smaller orders of magnitude can be obtained by rasterizing the acquired point cloud data; by processing the local point cloud data contained in the target grid, the calculation pressure can be reduced, the calculation speed can be improved, and the probability that the target grid comprises the noise point cloud points can be accurately determined. Based on the probability that each target grid comprises the noise point cloud points, each target grid comprising the noise point cloud points can be accurately screened out. And based on the processing of the local point cloud data corresponding to each target grid including the noise point cloud points, deleting the noise point cloud points in the point cloud data to obtain the updated point cloud data without the noise point cloud points, and completing the denoising processing of the point cloud data.

In a possible implementation manner, the updating the point cloud data based on the probability that each target grid includes a noise point cloud point to obtain updated point cloud data includes:

and under the condition that the probability that the target grid comprises the noise point cloud points is greater than the preset probability, deleting local point cloud data contained in the target grid to obtain updated point cloud data.

In this embodiment, the higher the probability that a point cloud point in the target grid is a noise point cloud point. By utilizing the preset probability and the probability that the target grid comprises the noise point cloud points, the target grid comprising the noise point cloud points can be accurately screened out. The denoising processing of the point cloud data can be realized by deleting the local point cloud data contained in the screened target grid.

In one possible embodiment, the determining the probability that the target grid includes noise point cloud points based on the local point cloud data included in each of the target grids comprises:

aiming at the local point cloud data contained in each target grid, performing noise feature extraction on point cloud information of each point cloud in the local point cloud data to obtain intermediate feature information of each point cloud;

performing feature fusion on the intermediate feature information of each point cloud point to obtain fusion feature information of the target grid;

determining a probability that the target grid includes noise point cloud points based on the fusion feature information.

According to the embodiment, noise feature extraction is performed on the point cloud information of each point cloud point, intermediate feature information related to noise can be extracted from the point cloud information of each point cloud point, and then fusion feature information with stronger noise correlation can be obtained on the basis of reducing feature quantity and redundancy of the feature information through fusion of the intermediate feature information. The probability prediction is carried out by utilizing the fusion characteristic information with stronger noise correlation, so that the accuracy of the output probability can be improved.

In a possible embodiment, after obtaining the updated point cloud data, the method further includes:

determining object information of each object to be detected based on the updated point cloud data;

and controlling a driving device to drive based on the object information of each object to be detected.

According to the embodiment, the object detection is carried out based on the updated point cloud data, so that the influence of noise point cloud points on the detection accuracy can be avoided, and the object information of each object to be detected can be accurately obtained; and then the running device is controlled to run based on the object information of each object to be detected, so that the running safety of the running device and the safety of the object to be detected can be improved.

In one possible embodiment, the method for denoising the point cloud data is performed by a denoising neural network.

According to the embodiment, the trained denoising neural network has reliable prediction precision, and the point cloud data denoising method provided by the embodiment of the disclosure is executed by using the trained denoising neural network, so that the point cloud data can be accurately denoised, and the accuracy of the obtained updated point cloud data is improved.

In one possible implementation, the denoised neural network is trained according to the following steps:

acquiring sample point cloud data;

rasterizing the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; determining labeling label information of each sample grid;

inputting the local sample point cloud data contained in each sample grid in at least one sample grid into a neural network to be trained, and generating a prediction probability that the sample grid comprises noise point cloud points;

and carrying out iterative training on the neural network to be trained on the basis of the prediction probability of each sample grid including noise point cloud points and the label information of each sample grid until a training cut-off condition is met, so as to obtain the de-noising neural network.

In the embodiment, iterative training is performed on the neural network to be trained based on the sample grid including the prediction probability of the noise point cloud point and the label information of the sample grid, so that the consistency of the prediction probability output by the network to be trained and the label information can be improved, and the denoising neural network with reliable prediction precision is obtained.

In a possible embodiment, the determining the label tag information of each sample grid includes:

for each sample point cloud point in the sample point cloud data, determining a point type of the sample point cloud point; wherein the point types include a noise type and a non-noise type: the noise type comprises at least one of a rainwater type, a fog type, a dust type, an exhaust gas type and a splash type;

determining the target number of sample point cloud points belonging to the noise type in the local sample point cloud data respectively contained in each sample grid;

and determining labeling label information of the sample grids based on the target quantity and a preset quantity threshold value corresponding to each sample grid.

According to the embodiment, the cloud points of the sample noise points which possibly appear are clearly classified by presetting a plurality of noise types. The point type of the cloud point of the sample point is determined based on various set noise types, omission of the cloud point of the sample point belonging to the noise can be avoided, and accuracy of the determined point type is improved. The labeling information is determined based on the target number and the preset number threshold of the sample point cloud points belonging to the noise type in the local sample point cloud data corresponding to the sample grid, and the accuracy of the determined labeling information can be improved.

In a second aspect, an embodiment of the present disclosure further provides a device for denoising point cloud data, including:

the acquisition module is used for acquiring point cloud data;

the division module is used for carrying out rasterization division on the point cloud data to obtain local point cloud data contained in at least one target grid;

a determining module, configured to determine, based on the local point cloud data included in each of the target grids, a probability that the target grid includes noise point cloud points;

and the updating module is used for updating the point cloud data based on the probability that each target grid comprises the noise point cloud points to obtain the updated point cloud data.

In a possible implementation manner, the updating module, when updating the point cloud data based on the probability that each target grid includes noise point cloud points to obtain updated point cloud data, is configured to delete local point cloud data included in the target grid to obtain updated point cloud data when the probability that the target grid includes noise point cloud points is greater than a preset probability.

In a possible implementation manner, when the determining module determines the probability that the target grid includes noise point cloud points based on the local point cloud data included in each target grid, the determining module is configured to perform noise feature extraction on point cloud information of each point cloud point in the local point cloud data to obtain intermediate feature information of each point cloud point, for the local point cloud data included in each target grid;

determining a probability that the target grid includes noise point cloud points based on the fused feature information.

In a possible embodiment, the apparatus further comprises:

the control module is used for determining the object information of each object to be detected based on the updated point cloud data after the updated point cloud data is obtained;

In one possible embodiment, the apparatus further comprises:

the training module is used for training to obtain the denoising neural network according to the following steps:

acquiring sample point cloud data;

and carrying out iterative training on the neural network to be trained based on the prediction probability that each sample grid comprises noise point cloud points and the label information of each sample grid until a training cut-off condition is met, and obtaining the de-noising neural network.

In a possible embodiment, the training module, when determining the label labeling information of each sample grid, is configured to determine, for each sample point cloud point in the sample point cloud data, a point type of the sample point cloud point; wherein the point types include a noise type and a non-noise type: the noise type comprises at least one of a rainwater type, a fog type, a dust type, an exhaust gas type and a splash type;

In a third aspect, this disclosure provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the machine-readable instructions stored in the memory, and when the machine-readable instructions are executed by the processor, the machine-readable instructions, when executed by the processor, perform the steps of the first aspect or any one of the possible implementations of the first aspect.

In a fourth aspect, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed to perform the steps in the first aspect or any one of the possible implementation manners of the first aspect.

For the description of the effect of the above denoising device, computer device, and computer readable storage medium for point cloud data, reference is made to the description of the denoising method for point cloud data, and details are not repeated here.

In order to make the aforementioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with figures are described in detail below.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly described below, and the drawings herein incorporated in and forming a part of the specification illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the technical solutions of the present disclosure. It is appreciated that the following drawings depict only certain embodiments of the disclosure and are therefore not to be considered limiting of its scope, for those skilled in the art will be able to derive additional related drawings therefrom without the benefit of the inventive faculty.

Fig. 1 illustrates a flowchart of a method for denoising point cloud data according to an embodiment of the present disclosure;

FIG. 2 illustrates a flow chart of a method of training a neural network to be trained provided by an embodiment of the present disclosure;

fig. 3 is a schematic diagram illustrating a point cloud data denoising apparatus according to an embodiment of the present disclosure;

fig. 4 shows a schematic structural diagram of a computer device provided in an embodiment of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only a part of the embodiments of the present disclosure, not all of the embodiments. The components of embodiments of the present disclosure, as generally described and illustrated herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the disclosure without making creative efforts, shall fall within the protection scope of the disclosure.

Furthermore, the terms "first," "second," and the like in the description and in the claims, and in the drawings described above, in the embodiments of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be implemented in other sequences than those illustrated or described herein.

Reference herein to "a plurality or a number" means two or more. "and/or" describes the association relationship of the associated objects, meaning that there may be three relationships, e.g., a and/or B, which may mean: a exists alone, A and B exist simultaneously, and B exists alone. The character "/" generally indicates that the former and latter associated objects are in an "or" relationship.

Research shows that various types of noise point cloud points exist in the point cloud data acquired by the laser radar in special weather environments or special road conditions. For example, cloud point data captured in rainy days often includes noise point cloud points of a rain type and a splash type. For another example, in point cloud data captured in a soil road, a dust type noise point cloud point is often included. The existence of noise point cloud points affects the detection accuracy of object detection on point cloud data. Therefore, how to denoise the point cloud data becomes an urgent problem to be solved.

Based on the research, the method provides a denoising scheme for point cloud data, and local point cloud data which correspond to each target grid and are smaller in magnitude order can be obtained by performing rasterization division on the acquired point cloud data; by processing the local point cloud data contained in the target grid, the calculation pressure can be reduced, the calculation speed can be improved, and the probability that the target grid comprises the noise point cloud points can be accurately determined. Based on the probability that each target grid comprises the noise point cloud points, each target grid comprising the noise point cloud points can be accurately screened out. And then based on the processing of the local point cloud data corresponding to each target grid including the noise point cloud points, the noise point cloud points in the point cloud data can be deleted, the updated point cloud data without the noise point cloud points is obtained, and the denoising processing of the point cloud data is completed.

The above-mentioned drawbacks are the results of the inventor after practical and careful study, and therefore, the discovery process of the above-mentioned problems and the solutions proposed by the present disclosure to the above-mentioned problems should be the contribution of the inventor in the process of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.

In order to facilitate understanding of the embodiment, a detailed description is first given to a point cloud data denoising method disclosed in the embodiment of the present disclosure, an execution main body of the point cloud data denoising method provided in the embodiment of the present disclosure is generally a terminal device or other processing device with certain computing capability, where the terminal device may be a User Equipment (UE), a mobile device, a User terminal, a Personal Digital Assistant (PDA), a handheld device, a computer device, or the like; in some possible implementations, the method for denoising the point cloud data may be implemented by a processor calling computer readable instructions stored in a memory.

The method for denoising point cloud data provided by the embodiment of the present disclosure is described below by taking an execution subject as a computer device as an example.

As shown in fig. 1, a flowchart of a method for denoising point cloud data provided in an embodiment of the present disclosure may include the following steps:

s101: and acquiring point cloud data.

Here, the point cloud data may be acquired using a laser radar installed on the traveling apparatus, and the point cloud data may be a point cloud vector set including point cloud information of a plurality of point cloud points. The point cloud information may include coordinates of point cloud points in a three-dimensional world coordinate system, color information of the point cloud points, reflection intensity information, distance information, and the like. Noise point cloud points may exist among the point cloud points included in the point cloud data. For example, the noise point cloud point may be a rain type point cloud point or an exhaust type point cloud point.

For example, the point cloud data may be road point cloud data acquired by the laser radar in a normal weather environment or a normal road condition, or may also be road point cloud data acquired by the laser radar in a special weather environment and/or a special road condition, which is not limited herein. The special weather environment may include, for example, rainy days, foggy days, windy days, sandstorm days, etc., and the special road conditions may include, for example, dirt roads, construction roads, muddy roads, etc.

In an exemplary process that the driving device drives in a special weather environment or a special road condition, the installed laser radar can be used for collecting point cloud data of a driving road. Further, the point cloud data can be obtained.

S102: and rasterizing the point cloud data to obtain local point cloud data contained in at least one target grid.

Here, the point cloud data is rasterized and divided to obtain a plurality of grids, where the obtained plurality of grids may include an empty grid and a non-empty target grid. Each target grid may contain local point cloud data.

The local point cloud data is part of the point cloud data in the point cloud data and comprises at least one point cloud point in the point cloud data and point cloud information of each point cloud point in the at least one point cloud point. The local point cloud data contained in each target grid can constitute the point cloud data.

For example, a horizontal axis direction (i.e., an x-axis direction) in the world coordinate system may be a direction of a road, a vertical axis direction (i.e., a y-axis direction) in the world coordinate system may be a direction perpendicular to the road, and a vertical axis direction (i.e., a z-axis direction) in the world coordinate system may be a direction perpendicular to the road and pointing to the sky (or the ground).

After the point cloud data is obtained, the point cloud data can be rasterized and divided according to a preset raster size. For example, the predetermined grid size may be L meters by M meters by N meters, where L is the length in the x-axis direction, M is the length in the y-axis direction, and N is the length in the z-axis direction. For example, the predetermined grid size may be 0.1 meters by 10 meters.

In a specific application, L, M, N may be determined according to a parameter of a laser radar that is actually used, and the embodiment of the present disclosure is not limited specifically. For example, the maximum x value, the maximum y value, and the maximum z value in the coordinates of the point cloud point in the point cloud data acquired by the laser radar may be determined.

In specific implementation, the number of the divided target grids and the target grid in which each cloud point is located can be determined according to the preset grid size and the coordinates of each cloud point in the point cloud data in a world coordinate system, and each cloud point in the same target grid is used as local point cloud data contained in the target grid. At least one target grid and local point cloud data contained in each target grid in the at least one target grid can be obtained by rasterizing the point cloud data.

S103: and determining the probability that the target grid comprises the noise point cloud points based on the local point cloud data contained in each target grid.

Here, the noise point cloud point may be a point cloud point belonging to a noise type in the local point cloud data. The noise type may include, but is not limited to, a rain type, a fog type, a dirt type, an exhaust type, a splash type, among others. The specific noise type may be determined according to the actual denoising requirement, and the embodiment of the present disclosure is not particularly limited. For example, when the point cloud point of the tree type needs to be denoised, the noise type may be the tree type; when the point cloud points of the road and pole type need to be denoised, the noise type can be the road and pole type.

The probability is used for representing the probability that the local point cloud data contained in the target grid comprises the noise point cloud points.

In specific implementation, for each target grid, feature information related to each point cloud point and a noise type may be determined based on point cloud information of each point cloud point in local point cloud data included in the target grid, and then, based on the feature information related to each point cloud point and the noise type, whether each point cloud point belongs to a noise point cloud point of the noise type is determined.

Then, a probability that the target grid includes noise point cloud points may be determined based on the number of noise point cloud points in the local point cloud data. For example, in the case where the number of noise point cloud points in the local point cloud data is greater than a preset value, the probability that the target grid includes the noise point cloud points is determined to be 1, and in the case where the number of noise point cloud points in the local point cloud data is determined not to be greater than the preset value, the probability that the target grid includes the noise point cloud points is determined to be 0. For another example, the probability that the target grid includes the noise point cloud points may be determined according to a ratio of the number of noise point cloud points in the local point cloud data to the number of non-noise point cloud points in the local point cloud data. For another example, after determining the feature information related to the noise type of each cloud point in the local point cloud data, the probability that the target grid includes the noise point cloud point may be determined directly based on the feature information of each cloud point.

S104: and updating the point cloud data based on the probability that each target grid comprises noise point cloud points to obtain updated point cloud data.

Here, the updated point cloud data may be point cloud data from which noise point cloud points are removed.

For example, after the probabilities that the target grids respectively include the noise point cloud points are obtained, the target grids may be sorted according to the probability that each target grid includes the noise point cloud points in the descending order of the probabilities, so as to obtain the sorting values of the target grids. For example, the probability that the target grid 1 includes noise point cloud points is 0.3, the probability that the target grid 2 includes noise point cloud points is 0.9, and the probability that the target grid 3 includes noise point cloud points is 0.8, then after the target grids are sorted in the order of probability from high to low, the sorting value of the target grid 2 is 1, the sorting value of the target grid 3 is 2, and the sorting value of the target grid 1 is 3.

Then, each target grid with the ranking value smaller than the preset ranking value may be determined as a noise grid including noise point cloud points. And (3) taking each point cloud point in the local point cloud data contained in the noise grid as a noise point cloud point, deleting all point cloud points in the local point cloud data contained in each noise grid, updating the point cloud data, and obtaining the updated point cloud data.

In an embodiment, regarding the above S104, the following steps may be performed: and under the condition that the probability that the target grid comprises the noise point cloud points is greater than the preset probability, deleting local point cloud data contained in the target grid to obtain updated point cloud data.

Here, the preset probability may be a preset minimum probability value. Under the condition that the probability that the target comprises the noise point cloud points is larger than the preset probability, determining that the target grid belongs to the noise grid, and all the cloud points in the local point cloud data contained in the target grid belong to the noise point cloud points; otherwise, the target grid is determined not to belong to the noise grid, and each point cloud point in the local point cloud data contained in the target grid does not belong to the noise point cloud point.

In specific implementation, for each target grid, the probability that the target grid includes the noise point cloud points can be compared with a preset probability, whether the probability that the target grid includes the noise point cloud points is greater than the preset probability is determined, if yes, the target grid is determined to belong to the noise grid, and local point cloud data contained in the target grid are deleted from the point cloud data; if not, determining that the target grid does not belong to the noise grid, and keeping the local point cloud data contained in the target grid in the point cloud data.

Based on the step, each noise grid with the probability of the noise point cloud points being greater than the preset probability can be screened out, and the local point cloud data contained in each noise grid is deleted, so that the point cloud data are denoised, and the updated point cloud data are obtained.

Thus, local point cloud data which correspond to each target grid and are smaller in magnitude order can be obtained by rasterizing the acquired point cloud data; by processing the local point cloud data contained in the target grid, the calculation pressure can be reduced, the calculation speed can be improved, and the probability that the target grid comprises the noise point cloud points can be accurately determined. Based on the probability that each target grid comprises the noise point cloud points, each target grid comprising the noise point cloud points can be accurately screened out. And then based on the processing of the local point cloud data corresponding to each target grid including the noise point cloud points, the noise point cloud points in the point cloud data can be deleted, the updated point cloud data without the noise point cloud points is obtained, and the denoising processing of the point cloud data is completed.

In an embodiment, the method for denoising point cloud data provided by the embodiment of the present disclosure may be executed by a trained denoising neural network, and specifically, the acquired point cloud data may be input to the denoising neural network, and the point cloud data is rasterized and divided by the denoising neural network to obtain local point cloud data included in at least one target grid; then, based on local point cloud data contained in each target grid, determining the probability that the target grid comprises noise point cloud points; and finally, updating the point cloud data based on the probability that each target grid comprises noise point cloud points, and outputting the updated point cloud data.

Here, since the trained denoising neural network has reliable prediction accuracy, the point cloud data can be accurately denoised by executing the point cloud data denoising method provided by the embodiment of the present disclosure using the trained denoising neural network, so as to obtain accurate and updated point cloud data.

In an embodiment, regarding the above S103, the following steps may be performed:

s103-1: and aiming at the local point cloud data contained in each target grid, performing noise feature extraction on the point cloud information of each point cloud point in the local point cloud data to obtain intermediate feature information of each point cloud point.

Here, the intermediate feature information is feature information related to a noise type in the point cloud information.

In specific implementation, for local point cloud data contained in each target grid, information related to the noise type in the point cloud information of each point cloud can be determined based on the point cloud information of each point cloud in the local point cloud data, and the information related to the noise type in the point cloud information of each point cloud is extracted to obtain intermediate characteristic information of each point cloud.

S103-2: and performing feature fusion on the intermediate feature information of each point cloud point to obtain fusion feature information of the target grid.

Here, the fused feature information is feature information of a high dimension, for example, 64 dimensions, 128 dimensions, or the like. The fusion characteristic information is characteristic information which can represent whether the target grid is related to the noise type or not.

For example, for each target grid, target feature information related to a noise type may be further extracted from intermediate feature information of each cloud point in local point cloud data included in the target grid, feature fusion operation is performed on the target feature information of each cloud point to obtain high-dimensional fusion feature information, and the high-dimensional fusion feature information is used as the fusion feature information of the target grid.

S103-3: and determining the probability that the target grid comprises the noise point cloud points based on the fusion characteristic information.

In specific implementation, after the fusion feature information of the target grid is obtained, convolution processing may be performed on the fusion feature information, upsampling processing may be performed on a result of the convolution processing, and a probability that the target grid includes the noise point cloud point is determined according to a result of the upsampling processing.

As can be seen from the foregoing embodiments, the point cloud data denoising method provided by the embodiments of the present disclosure may be implemented by using a denoising neural network. Therefore, S103-1 through S103-3, above, may also be performed using a de-noising neural network.

Specifically, after the local point cloud data corresponding to each target grid is obtained, the denoising neural network may sequentially process the local point cloud data included in each target grid in a serial processing manner, so as to sequentially output the probability that each target grid includes the noise point cloud point. In this way, the processing pressure of the target neural network can be reduced by means of the series processing, so that the target neural network can be lighter in weight.

Optionally, after the local point cloud data included in each target grid is acquired, the local point cloud data included in each target grid may also be processed in a parallel processing manner, so as to output a probability that each target grid includes noise point cloud points.

Illustratively, the denoised neural network may include a plurality of network layers. Aiming at local point cloud data contained in each target grid, full connection processing can be performed on point cloud information of each point cloud in the local point cloud data by using a full connection layer in a denoising neural network to obtain first characteristic information of each point cloud; converting the range of the characteristic value corresponding to each first characteristic information by using a batch normalization layer in the de-noising neural network to obtain second characteristic information of each cloud point; then, a Linear rectification function (ReLU) function layer in the denoising neural network is utilized to perform numerical conversion on a characteristic value corresponding to the second characteristic information of each point cloud, the characteristic value smaller than 0 is set to be 0, and the characteristic value larger than 0 is reserved, so that intermediate characteristic information of each point cloud is obtained; and then, performing feature fusion on the intermediate feature information of each point cloud point by utilizing a maximum pooling layer in the denoising neural network to obtain high-dimensional fusion feature information of the target grid.

Further, 2D convolution can be performed on the road characteristic information by using a 2D convolution layer in the denoising neural network to obtain third characteristic information; converting the range of the characteristic value corresponding to the third characteristic information by using another batch normalization layer in the de-noising neural network to obtain fourth characteristic information; then, carrying out numerical transformation on the characteristic value corresponding to the fourth characteristic information by utilizing a ReLU function in another ReLU function layer in the de-noising neural network to obtain fifth characteristic information; then, utilizing an upsampling layer in the denoising neural network to perform upsampling processing on the fifth characteristic information to obtain sixth characteristic information; and finally, performing feature processing on the sixth feature information by using a sigmoid activation function layer in the denoising neural network, and outputting a probability map of the target grid including the noise point cloud points. Wherein, the probability interval corresponding to the probability map is (0, 1). According to the probability map, the probability that the target grid includes the noise point cloud point can be obtained.

Therefore, by utilizing different network layers, the feature information concerned by each network layer can be extracted from each network layer, and finally, the probability that the target grid comprises the noise point cloud points can be accurately determined based on the sigmoid activation function layer.

After the probability that each target grid respectively comprises the noise point cloud points is obtained, the denoising neural network can screen the noise grids from the target grids by using the preset probability, and delete the local point cloud data contained in each noise grid from the obtained point cloud data, so as to obtain the updated point cloud data. Since the objects corresponding to the noise point cloud points of various noise types are objects that do not affect the driving of the driving device, the objects corresponding to the noise point cloud points of various noise types can be prevented from being recognized as obstacles by deleting the noise point cloud points of various noise types, thereby affecting the driving of the driving device. For example, in the case of identifying rainwater corresponding to a noise point cloud point of a rainwater type as an obstacle, a situation such as sudden braking or sudden turning of a running device may occur, and running safety is affected.

In an embodiment, after the updated point cloud data is obtained, the updated point cloud data may be further input to a downstream lidar sensing algorithm module, object detection is performed on the updated point cloud data, and each object to be detected corresponding to the updated point cloud data and object information of each object to be detected are determined.

The object to be detected may include, for example, a lane line, a building, a pedestrian, a vehicle, a road pole, and the like. The object information may include, for example, an object position, an object type, a distance between an object distance and a traveling device of the point cloud data, whether or not the object to be detected is a movable object, a moving direction of the movable object, a moving speed, a safe distance from the movable object, and the like.

Then, the travel device may be controlled to travel based on the object information of each object to be detected.

Here, the traveling device may be a device to which a laser radar that photographs the point cloud data is mounted. For example, the traveling device may include any device that can travel on a road, such as an autonomous vehicle, a manually-driven vehicle, and a robot

For example, after the object information of each object to be detected is obtained, a safe driving route of the autonomous vehicle may be planned according to the object information (e.g., an object position, a moving speed, etc.) of each object to be detected, and the autonomous vehicle may be controlled to drive according to the safe driving route.

For another example, when it is determined that the object to be detected includes a pedestrian, the distance between the manually-driven vehicle and the pedestrian can be determined according to the position of the pedestrian and the position of the manually-driven vehicle, and when the distance is smaller than the preset safety distance, voice alarm can be performed to prompt the pedestrian to avoid the vehicle and remind the driver of timely decelerating to timely avoid the pedestrian.

For another example, when it is determined that another vehicle is included in the object to be detected, the movement speed, the movement direction, the movement trajectory, and the like of the other vehicle may be determined, the movement trajectory of the other vehicle may be predicted, and the travel device may be controlled to travel based on the predicted trajectory of the other vehicle.

In an embodiment, in a case that the point cloud data denoising method provided by the embodiment of the present disclosure is executed by a denoising neural network, the embodiment of the present disclosure further provides a method for training a neural network to be trained, as shown in fig. 2, a flowchart of the method for training the neural network to be trained provided by the embodiment of the present disclosure may include the following steps:

s201: and acquiring sample point cloud data.

Here, the sample point cloud data may be point cloud data acquired with any laser radar. The sample point cloud data can be a sample point cloud vector set, and the sample point cloud vector set comprises point cloud information of a plurality of sample point cloud points. The point cloud information may include coordinates of the sample point cloud points in a three-dimensional world coordinate system, color information of the sample point cloud points, reflection intensity information, distance information, and the like. Sample noise point cloud points may exist among the sample point cloud points included in the sample point cloud data. For example, a rain type sample point cloud point, an exhaust type sample point cloud point, a dust type sample point cloud point.

For example, the point cloud data collected by the laser radar under a special weather environment and/or a special road condition may be used as the sample point cloud data.

S202: rasterizing the sample point cloud data to obtain local sample point cloud data contained in at least one sample grid; and determining labeling label information for each sample grid.

Here, the label information may specifically be a label value corresponding to the sample grid. Under the condition that the target number of the cloud points of the sample noise points included in the sample grid is greater than a preset number threshold, the labeling label information can be a first label value; the tagging information may be a second tag value in a case that the target number of the sample noise point cloud points included in the sample grid is not greater than a preset number threshold. The second tag value may also be a tag value corresponding to other types of tags, where different types of tags may correspond to different tag values. .

In specific implementation, the number of the divided sample grids and the sample grid in which each sample point cloud point is located can be determined according to the preset grid size and the coordinates of each sample point cloud point in the sample point cloud data in the world coordinate system, and each sample point cloud point located in the same sample grid is used as the local sample point cloud data contained in the sample grid. Meanwhile, for each sample grid, the labeling label information of the sample grid can be predetermined.

In one embodiment, the labeling label information for each sample grid may be determined as follows:

the method comprises the steps of firstly, determining the point type of a sample point cloud point aiming at each sample point cloud point in sample point cloud data; wherein the point types include a noise type and a non-noise type: the noise type may include at least one of a rain type, a fog type, a dirt type, an exhaust type, a splash type.

Here, the non-noise type means that the cloud point of the sample point is of another type. Such as lane line type, building type, vehicle type, etc. The noise type may include, but is not limited to, rain type, fog type, dirt type, tail gas type, splash type. The specific noise type may be set according to the actual denoising requirement, and is not specifically limited herein.

In specific implementation, after the sample point cloud data is obtained, the sample point cloud data may be subjected to area division by using a 3-dimensional (3D) marking frame with a preset size and in a sliding window manner according to a preset step length to obtain a plurality of 3D marking frames and at least one sample point cloud point included in each 3D marking frame.

Then, for each 3D labeling frame, a point type of each sample point cloud point included in the 3D labeling frame may be determined based on point cloud information of each sample point cloud point included in the 3D labeling frame in a manual labeling manner.

And step two, determining the target number of the sample point cloud points belonging to the noise type in the local sample point cloud data respectively contained in each sample grid.

In specific implementation, for each sample grid obtained based on the division in S202, the target number of sample point cloud points belonging to the noise type in the local sample point cloud data may be determined according to the point type of each sample point cloud point in the local sample point cloud data included in the sample grid.

And step three, determining the labeling label information of the sample grids based on the target quantity and the preset quantity threshold value corresponding to each sample grid.

Here, the target number corresponding to the sample grid is the target number of sample point cloud points belonging to the noise type in the local sample point cloud data contained in the sample grid; or, the target number corresponding to the sample grid is the number of noise-type sample point cloud points included in the sample grid. The preset number threshold may be a preset minimum number threshold. Determining that a sample grid is a noise grid if the target number of the cloud points of the sample points including the noise type in the sample grid is not less than a preset number threshold; conversely, if the target number of cloud points of sample points including noise type in a sample grid is less than the preset number threshold, the sample grid may be determined to be a non-noise grid, such as a background grid or any preset type of grid.

For example, for each sample grid, the target number corresponding to the sample grid may be compared with a preset number threshold, and when the target number corresponding to the sample grid is not less than the preset number (for example, 1), it is determined that the label value corresponding to the label information of the sample grid is the first label value, that is, the sample grid is represented as a noise grid. And under the condition that the target number corresponding to the sample grid is smaller than the preset number, determining that the label value corresponding to the label information of the sample grid is a second label value, namely, indicating that the sample grid is a non-noise grid.

In a possible implementation manner, after the point type of each sample point cloud point included in each 3D labeling box is determined, a corresponding semantic label may also be assigned to each sample point cloud point according to the point type of each sample point cloud point. The semantic label of the rainwater type sample point cloud point is a rainwater label, the semantic label of the fog type sample point cloud point is a fog label, the semantic label of the dust type sample point cloud point is a dust label, the semantic label of the tail gas type sample point cloud point is a tail gas label, the semantic label of the splash type sample point cloud point is a splash label, and the semantic label of the non-noise type sample point cloud point is other labels.

Then, according to the semantic label of each sample point cloud point, each sample point cloud point belonging to the noise type is determined from the multiple sample point cloud points included in the sample point data, and the label value of each sample point cloud point belonging to the noise type is set to be a first preset value, that is, the sample point cloud point is represented as a sample noise point cloud point. Meanwhile, the label value of each sample point cloud point which is included in the sample point data and does not belong to the noise type can be set as a second preset value, that is, the sample point cloud point is represented as other types of cloud point.

Then, for each sample grid, the target number of sample point cloud points having the first label value in the local sample point cloud data, that is, the target number of sample noise point cloud points in the local sample point cloud data, may be determined according to the label value of each sample point cloud point in the local sample point cloud data corresponding to the sample grid. And comparing the target quantity corresponding to the sample grid with a preset quantity threshold, and determining that the label value corresponding to the label information of the sample grid is a first label value under the condition that the target quantity corresponding to the sample grid is not less than the preset quantity, namely indicating that the sample grid is a noise grid. And under the condition that the target number corresponding to the sample grid is smaller than the preset number threshold, determining that the label value corresponding to the label marking information of the sample grid is a second label value, namely, the sample grid is represented as a non-noise grid.

S203: and inputting local sample point cloud data contained in each sample grid in at least one sample grid into the neural network to be trained, and generating the prediction probability that the sample grid comprises the noise point cloud points.

Here, the neural network to be trained is the denoising neural network to be trained. The prediction probability is the probability that the sample grid comprises the noise point cloud points and is output by the neural network to be trained.

In specific implementation, the neural network to be trained can be utilized to sequentially process the local sample point cloud data of each sample grid, and the prediction probability that each sample grid comprises the noise point cloud points is output.

S204: and carrying out iterative training on the neural network to be trained based on the prediction probability of each sample grid including the cloud point of the noise point and the label information of each sample grid until a training cut-off condition is met, thereby obtaining the de-noising neural network.

Here, the training cutoff condition may include that the number of rounds of iterative training reaches a preset number of rounds, and/or the prediction accuracy of the trained neural network reaches a preset accuracy.

In specific implementation, a binary cross entropy function can be used to determine the binary cross entropy loss of the neural network to be trained based on the prediction probability of the sample grid including the noise point cloud point and the label value corresponding to the label marking information of the sample grid. For example, in the case that the label value corresponding to the label information of the sample grid is the first label value, the first loss may be determined based on the prediction probability that the sample grid includes the noise point cloud point and the first label value by using a binary cross entropy function. And under the condition that the label value corresponding to the labeling label information of the sample grid is the second label value, determining the second loss based on the prediction probability of the sample grid including the noise point cloud point and the second label value by using a binary cross entropy function.

And then, iteratively training the neural network to be trained by using the binary cross entropy loss until a training cut-off condition is met, thereby obtaining the denoising neural network.

It will be understood by those skilled in the art that in the method of the present invention, the order of writing the steps does not imply a strict order of execution and any limitations on the implementation, and the specific order of execution of the steps should be determined by their function and possible inherent logic.

Based on the same inventive concept, the embodiment of the present disclosure further provides a point cloud data denoising device corresponding to the point cloud data denoising method, and since the principle of solving the problem of the device in the embodiment of the present disclosure is similar to that of the point cloud data denoising method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and repeated details are not repeated.

As shown in fig. 3, a schematic diagram of a point cloud data denoising device provided in the embodiment of the present disclosure includes:

an obtaining module 301, configured to obtain point cloud data;

a partitioning module 302, configured to perform rasterization partitioning on the point cloud data to obtain local point cloud data included in at least one target grid;

a determining module 303, configured to determine, based on the local point cloud data included in each of the target grids, a probability that the target grid includes a noise point cloud point;

an updating module 304, configured to update the point cloud data based on a probability that each target grid includes a noise point cloud point, so as to obtain updated point cloud data.

In a possible implementation manner, the updating module 304, when the probability that each target grid includes noise point cloud points is obtained by updating the point cloud data based on the probability that each target grid includes noise point cloud points, is configured to delete local point cloud data included in the target grid when the probability that each target grid includes noise point cloud points is greater than a preset probability, so as to obtain updated point cloud data.

In a possible implementation manner, when the determining module 303 determines the probability that the target grid includes noise point cloud points based on the local point cloud data included in each target grid, the determining module is configured to perform noise feature extraction on point cloud information of each point cloud point in the local point cloud data to obtain intermediate feature information of each point cloud point, for the local point cloud data included in each target grid;

In a possible embodiment, the apparatus further comprises:

a control module 305, configured to determine object information of each object to be detected based on the updated point cloud data after obtaining the updated point cloud data;

In a possible implementation manner, the method for denoising point cloud data provided by the above embodiments is performed by a denoising neural network.

In a possible embodiment, the apparatus further comprises:

a training module 306, configured to train to obtain the denoised neural network according to the following steps:

acquiring sample point cloud data;

In a possible implementation, the training module 306, when determining the label labeling information of each sample grid, is configured to determine, for each sample point cloud point in the sample point cloud data, a point type of the sample point cloud point; wherein the point types include a noise type and a non-noise type: the noise type comprises at least one of a rainwater type, a fog type, a dust type, an exhaust gas type and a splash type;

The description of the processing flow of each module in the device and the interaction flow between the modules may refer to the related description in the above method embodiments, and will not be described in detail here.

Based on the same technical concept, the embodiment of the application also provides computer equipment. Referring to fig. 4, a schematic structural diagram of a computer device provided in an embodiment of the present application includes:

a processor 41, a memory 42, and a bus 43. Wherein the memory 42 stores machine-readable instructions executable by the processor 41, the processor 41 is configured to execute the machine-readable instructions stored in the memory 42, and when the machine-readable instructions are executed by the processor 41, the processor 41 performs the following steps: s101: acquiring point cloud data; s102: rasterizing the point cloud data to obtain local point cloud data contained in at least one target grid; s103: determining the probability that the target grid comprises the noise point cloud points based on the local point cloud data contained in each target grid, and S104: and updating the point cloud data based on the probability that each target grid comprises noise point cloud points to obtain updated point cloud data.

The storage 42 includes a memory 421 and an external storage 422; the memory 421 is also referred to as an internal memory, and is used for temporarily storing the operation data in the processor 41 and the data exchanged with the external storage 422 such as a hard disk, the processor 41 exchanges data with the external storage 422 through the memory 421, and when the computer device is operated, the processor 41 communicates with the storage 42 through the bus 43, so that the processor 41 executes the execution instructions mentioned in the above method embodiments.

The embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the computer program performs the steps of the point cloud data denoising method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

The computer program product of the point cloud data denoising method provided in the embodiment of the present disclosure includes a computer readable storage medium storing a program code, where instructions included in the program code may be used to execute the steps of the point cloud data denoising method described in the above method embodiments, which may be specifically referred to the above method embodiments, and are not described herein again.

The computer program product may be embodied in hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied in a computer storage medium, and in another alternative embodiment, the computer program product is embodied in a Software product, such as a Software Development Kit (SDK) or the like.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the apparatus described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one logical division, and there may be other divisions when actually implementing, and for example, a plurality of units or components may be combined, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some communication interfaces, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one position, or may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer-readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present disclosure. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

If the technical scheme of the application relates to personal information, a product applying the technical scheme of the application clearly informs personal information processing rules before processing the personal information, and obtains personal independent consent. If the technical scheme of the application relates to sensitive personal information, a product applying the technical scheme of the application obtains individual consent before processing the sensitive personal information, and simultaneously meets the requirement of 'express consent'. For example, at a personal information collection device such as a camera, a clear and significant identifier is set to inform that the personal information collection range is entered, the personal information is collected, and if the person voluntarily enters the collection range, the person is regarded as agreeing to collect the personal information; or on the device for processing the personal information, under the condition of informing the personal information processing rule by using obvious identification/information, obtaining personal authorization in the modes of pop-up window information or asking the person to upload personal information thereof and the like; the personal information processing rule may include information such as a personal information processor, a personal information processing purpose, a processing method, and a type of personal information to be processed.

Finally, it should be noted that: the above-mentioned embodiments are merely specific embodiments of the present disclosure, which are used for illustrating the technical solutions of the present disclosure and not for limiting the same, and the scope of the present disclosure is not limited thereto, and although the present disclosure is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive of the technical solutions described in the foregoing embodiments or equivalent technical features thereof within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present disclosure, and should be construed as being included therein. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

Claims

1. A method for denoising point cloud data, comprising:

acquiring point cloud data;

2. The method of claim 1, wherein the updating the point cloud data based on the probability that each of the target grids includes noise point cloud points comprises:

3. The method of claim 1 or 2, wherein determining the probability that the target grid includes noise point cloud points based on the local point cloud data included in each of the target grids comprises:

4. The method of any one of claims 1 to 3, further comprising, after obtaining the updated point cloud data:

5. The method of any one of claims 1 to 4, wherein the method of denoising the point cloud data is performed by a denoising neural network.

6. The method of claim 5, wherein the denoised neural network is trained according to the following steps:

acquiring sample point cloud data;

7. The method of claim 6, wherein the determining label information for each sample grid comprises:

determining a point type of each sample point cloud point in the sample point cloud data; wherein the point types include a noise type and a non-noise type: the noise type comprises at least one of a rainwater type, a fog type, a dust type, an exhaust gas type and a splash type;

8. A device for denoising point cloud data, comprising:

the acquisition module is used for acquiring point cloud data;

the division module is used for performing rasterization division on the point cloud data to obtain local point cloud data contained in at least one target grid;

9. A computer device, comprising: a processor, a memory storing machine readable instructions executable by the processor, the processor for executing the machine readable instructions stored in the memory, the processor performing the steps of the method of denoising point cloud data of any of claims 1-7 when the machine readable instructions are executed by the processor.

10. A computer-readable storage medium, characterized in that a computer program is stored thereon, which, when being executed by a computer device, performs the steps of the method of denoising point cloud data according to any one of claims 1 to 7.