WO2021056516A1

WO2021056516A1 - Method and device for target detection, and movable platform

Info

Publication number: WO2021056516A1
Application number: PCT/CN2019/108897
Authority: WO
Inventors: 徐斌; 陈晓智
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2019-09-29
Filing date: 2019-09-29
Publication date: 2021-04-01
Also published as: CN112154448A

Abstract

A method and device for target detection, and a movable platform. The method: determining a point cloud vector on the basis of the number of points in a point cloud to be processed and attributes of each point; then utilizing a three-dimensional convolutional neural network to extract a target three-dimensional point cloud feature of the point cloud vector, retaining height information, thus retaining three dimensional structure information of the point cloud to the greatest extent, and determining target information in said point cloud on the basis of the extracted target three-dimensional point cloud feature, where the target information comprises geometric location information of a target in each dimension of a three-dimensional coordinate system. This allows a three-dimensional target in a point cloud to be found accurately, solves the existing problem of a high error rate with target detection results followed by inability to guide a plane, vehicle, or robot to perform obstacle avoidance and route planning.

Description

Target detection method, equipment and movable platform

Technical field

The embodiments of the present application relate to detection technology, and in particular, to a target detection method, device, and movable platform.

Background technique

With the continuous development of economy and technology, airplanes, automobiles, robots, etc. are applied to many aspects of people's lives. Accordingly, how to guide airplanes, automobiles, robots, etc. to avoid obstacles and plan paths has become the focus of attention.

In related technologies, point clouds are usually used to detect targets to guide airplanes, cars, robots, etc. to avoid obstacles and plan paths. Especially for intelligent driving, a lidar sensor is usually equipped, and the target detection is performed through the laser point cloud output by the lidar sensor.

However, the clustering algorithm is usually used when using the point cloud for target detection. Clustering is the collection of similar elements. In this way, the accuracy of the clustering results is often low, resulting in a high error rate in the above-mentioned target detection results, and subsequent failure to guide airplanes, cars, robots, etc. to avoid obstacles and path planning.

Summary of the invention

The embodiments of the present application provide a target detection method and device to overcome at least one of the above-mentioned problems.

In the first aspect, an embodiment of the present application provides a target detection method, including:

Determine a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, the point cloud vector including geometric position information of the points in each dimension in the three-dimensional coordinate system;

Processing the point cloud vector by using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector;

The target information in the point cloud to be processed is determined according to the characteristics of the target three-dimensional point cloud, and the target information includes geometric position information of the target in each dimension in the three-dimensional coordinate system.

In a second aspect, an embodiment of the present application provides a target detection device, including a memory, a processor, and computer-executable instructions stored in the memory and running on the processor, and the processor executes the computer-executed The following steps are implemented when ordering:

In the third aspect, an embodiment of the present application provides a movable platform, including:

Mobile device platform; and

As for the target detection device described in the above second aspect and various possible designs of the second aspect, the target detection device is installed on the movable platform body.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium that stores computer-executable instructions. When the processor executes the computer-executable instructions, the first aspect and the first aspect described above are implemented. In terms of various possible designs, the target detection method described.

The target detection method, device, and movable platform provided by the embodiments of the present application determine the point cloud vector based on the number of points in the point cloud to be processed and the attributes of each point; and then use a three-dimensional convolutional neural network to calculate the point cloud vector Perform target 3D point cloud feature extraction, retain height information, and then retain the maximum 3D structure information of the point cloud, and then determine the target information in the point cloud to be processed according to the extracted target 3D point cloud features, where the target information includes the target The geometric position information of each dimension in the three-dimensional coordinate system accurately finds the three-dimensional target in the point cloud, and solves the problem that the existing target detection results have a high error rate and the subsequent failure to guide aircraft, cars, robots, etc. to avoid obstacles and path planning.

Description of the drawings

The drawings here are incorporated into the specification and constitute a part of the specification, show embodiments that conform to the application, and are used together with the specification to explain the principle of the application.

FIG. 1 is a schematic diagram of the architecture of a target detection system provided by an embodiment of the application;

2 is a schematic flowchart of a target detection method provided by an embodiment of the application;

FIG. 3 is a schematic diagram of a three-dimensional convolutional neural network provided by an embodiment of the application;

4 is a schematic flowchart of another target detection method provided by an embodiment of the application;

FIG. 5 is a schematic structural diagram of a target detection device provided by an embodiment of this application;

FIG. 6 is a schematic structural diagram of another target detection device provided by an embodiment of the application;

FIG. 7 is a schematic diagram of the hardware structure of a target detection device provided by an embodiment of the application;

FIG. 8 is a schematic structural diagram of a movable platform provided by an embodiment of the application.

Through the above drawings, the specific embodiments of the present application have been shown, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the concept of the present application in any way, but to explain the concept of the present application for those skilled in the art by referring to specific embodiments.

detailed description

The exemplary embodiments will be described in detail here, and examples thereof are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the application as detailed in the appended claims.

In order to solve this technical problem, this embodiment provides a target detection method. The point cloud vector is determined by the number of points in the point cloud to be processed and the attributes of each point; then a three-dimensional convolutional neural network is used to target the point cloud vector The three-dimensional point cloud feature is extracted, and the target information in the point cloud to be processed is determined according to the extracted target three-dimensional point cloud feature, where the target information includes the geometric position information of the target in each dimension in the three-dimensional coordinate system.

Since the three-dimensional convolutional neural network is used to extract the target three-dimensional point cloud feature from the point cloud vector, when performing the three-dimensional point cloud feature extraction, three-dimensional convolution is used to ensure the retention of height information, and then the three-dimensional structure information of the point cloud is retained to the greatest extent. Determine the target information in the point cloud to be processed according to the extracted target three-dimensional point cloud features, which can accurately find the three-dimensional target in the point cloud, solve the problem of high error rate of existing target detection results, and fail to guide aircraft, cars, robots, etc. The problem of obstacle avoidance and path planning.

FIG. 1 is a schematic structural diagram of a target detection system provided by an embodiment of the application. As shown in FIG. 1, it includes: a sensor 101, a first processor 102 and a second processor 103. Taking a vehicle as an example, the sensor 101 can generate a point cloud to be processed in real time, where the point cloud is used to identify the ground surrounding the vehicle. The first processor 102 can determine the point cloud vector based on the number of points in the point cloud generated by the sensor 101 and the attributes of each point. The point cloud vector includes the geometric position information of the points in each dimension in the three-dimensional coordinate system, and uses the three-dimensional The convolutional neural network processes the point cloud vector to extract the target three-dimensional point cloud feature of the point cloud vector, and determine the target information in the above-mentioned point cloud according to the target three-dimensional point cloud feature. The target information includes the target's various dimensions in the three-dimensional coordinate system. The geometric position information and the target information are sent to the second processor 103 for subsequent driving planning and use.

Here, the first processor 102 and the second processor 103 may be a vehicle computing platform, an unmanned aerial vehicle processor, or the like. This embodiment does not particularly limit the implementation of the first processor 102 and the second processor 103, as long as the first processor 102 and the second processor 103 can perform the above-mentioned corresponding functions.

It should be understood that the foregoing architecture is only an exemplary system architecture block diagram. During specific implementation, it can be set according to application requirements. For example, the first processor 102 and the second processor 103 can be set separately or combined to meet different application requirements. .

In addition, the above-mentioned target detection system may also include a receiving device, a display device, and the like.

In the specific implementation process, the receiving device can be an input/output interface or a communication interface. The receiving device may receive a user's instruction, for example, the receiving device may be an input interface connected to a mouse.

The display device can be used to display the above-mentioned target information. The display device may also be a touch screen, which is used to receive user instructions while displaying the above-mentioned target information, so as to realize interaction with the user.

The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the drawings.

FIG. 2 is a schematic flowchart of a target detection method provided by an embodiment of this application. The execution subject of this embodiment may be the first processor in the embodiment shown in FIG. 1. As shown in FIG. 2, the method includes:

S201: Determine a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, where the point cloud vector includes geometric position information of the points in each dimension in the three-dimensional coordinate system.

The above-mentioned point cloud may be an image point cloud, a radar point cloud, a laser point cloud, etc., and one or more of the above-mentioned point clouds may be used in the subsequent processing according to actual conditions.

Here, as mentioned above, before determining the point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, the point cloud to be processed can be acquired by the sensor. Specifically, the acquisition range can be limited in the three-dimensional space. For example, get F meters in front of the sensor, B meters behind the sensor, L and R meters on the left and right sides of the sensor, U meters on the sensor and D meters on the bottom. In this way, the point cloud processing range of the entire three-dimensional space is limited to the range of (F+B)x(L+R)x(U+D). Among them, the values of F, B, L, R, U, and D can be set according to actual conditions.

After obtaining the point cloud to be processed, the point cloud vector can be determined according to the number of points in the point cloud to be processed and the attributes of each point.

Optionally, the determining the point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point includes:

Meshing the point cloud to be processed;

According to the number of points in each grid after division and the attributes of each point, determine the vector of each grid after division, where the attributes of each point include the three-dimensional coordinates and reflectivity of the point in the three-dimensional coordinate system;

The point cloud vector is determined according to the vector of each grid after the division.

Specifically, the determining the vector of each grid after division according to the number of points in each grid after division and the attributes of each point includes:

Adjust the number of points in each grid after division according to the preset number of points;

According to the product of the number of points in each grid after adjustment and the attributes of each point, the vector of each grid after division is determined.

Exemplarily, the grid is divided along the direction of each coordinate axis of the point cloud coordinate system, the front-view direction (X axis) is divided every resx meter, and the left and right direction (Y axis) is divided every resy meter, upward The direction (Z axis) is divided every resz meter. In this way, for the entire three-dimensional space, a total of T=((F+B)/resx)*((L+R)/resy)*((U+D)/resz) small grids are divided, each small The grid is a small cuboid of resx*resy*resz. Among them, the values of resx, resy, and resz can be set according to actual conditions.

Moreover, for all the above points, divide them according to the range of the grid, and determine the grid that each point belongs to. For any grid, the number of laser points contained in it can be limited to N. When the number of points is greater than N , Random sampling can be performed to obtain N points among them, and the redundant laser points can be discarded. When the number of points is less than N, random copying can be performed to reach N points, so that all small grids contain the same number of points. Among them, the value of the aforementioned number N can be set according to actual conditions.

In this way, through the above processing, the entire point cloud is represented by a K*N*4 vector, where K represents the number of grids with a non-zero number of points. N is the maximum number of points in each grid. 4 means that each point has 4-dimensional attributes, which are xyz coordinates and reflectivity. It should be understood that in addition to using the K*N*4 vector to represent the above-mentioned point cloud, other information can also be added to the point cloud vector, such as density, height, and so on.

As mentioned above, the entire three-dimensional space is divided into quantitative small grids, and then the feature vector of each grid is determined, that is, point cloud preprocessing (point cloud coding) is carried out through a certain structured method, and the original point cloud is preserved to the greatest extent Structure information.

S202: Use a three-dimensional convolutional neural network to process the point cloud vector to extract a target three-dimensional point cloud feature of the point cloud vector.

Here, for feature extraction, in order to retain three-dimensional structure information, the embodiment of the present application adopts three-dimensional convolution to extract features from three-dimensional space and retain spatial structure information.

Optionally, the three-dimensional convolutional neural network includes a third convolutional neural network.

The processing the point cloud vector using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector includes:

The third convolutional neural network is used to extract the three-dimensional grid feature of each grid vector after division.

Wherein, the third convolutional neural network is obtained through three-dimensional grid vector and three-dimensional grid feature training.

Here, after the above-mentioned preprocessing of the point cloud through a certain structured method, the feature extraction is performed on each small grid using a three-dimensional convolutional neural network. This part can be realized by a convolutional layer, an up-sampling layer, and a fully connected layer. , That is, input the above K*N*4 vector and pass through a series of convolutional layers, up-sampling layers and fully connected layers, and the feature vector obtained is K*C, and the feature corresponding to each small grid is C-dimensional . For all grids in the three-dimensional space, the feature corresponding to each grid is C-dimensional. If the grid has no points, the feature is a C-dimensional zero vector. If the total number of small grids is X1*Y1*Z1, then the feature vector obtained is X1*Y1*Z1*C. Once the structured data is obtained, the feature can be extracted directly using the convolution operation.

Optionally, the three-dimensional convolutional neural network further includes a fourth convolutional neural network.

After the third convolutional neural network is used to extract the three-dimensional grid features of the divided vectors of each grid, the method further includes:

The fourth convolutional neural network is used to extract features of the target three-dimensional point cloud from the extracted three-dimensional grid features.

Wherein, the fourth convolutional neural network is obtained through three-dimensional grid feature and three-dimensional point cloud feature training.

Exemplarily, if two-dimensional convolution is used, the feature vector needs to be reshaped as X1*Y1*(Z1*C), where Z1*C is recorded as the dimension of the feature. When two-dimensional convolution is used, the height dimension ( Z direction) and the characteristic channel dimension are combined, at this time, a certain height information will be lost. The embodiment of the present application uses three-dimensional convolution, which can maintain structural information in various directions and maximize the preservation of the spatial information of the original point cloud. After a series of three-dimensional convolution and other operations, the final feature vector will be obtained, denoted as X2*Y2*Z2*C2.

In addition, the above-mentioned three-dimensional convolutional neural network may include:

Multiple convolutional layers for performing convolution operations on the point cloud vector to output three-dimensional point cloud features;

The up-sampling layer is connected to at least one convolutional layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by at least one of the multiple convolutional layers, and to compare the obtained 3D point cloud features are processed to output the processed 3D point cloud features;

The fully connected layer is connected to the convolutional layer and the upsampling layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by a convolutional layer and the processed three-dimensional point cloud features, And perform feature fusion on the acquired three-dimensional point cloud features and the processed three-dimensional point cloud features to generate a fused three-dimensional point cloud feature, and input the fused three-dimensional point cloud feature into another convolutional layer to After the convolution operation of the other convolution layer, the target three-dimensional point cloud feature is determined.

Optionally, the depths of the convolutional layers in the multiple convolutional layers are different;

The fully connected layer is connected to the first convolutional layer, the second convolutional layer, and the up-sampling layer among the plurality of convolutional layers to obtain the three-dimensional point cloud features output by the first convolutional layer and The processed three-dimensional point cloud feature, and feature fusion is performed on the three-dimensional point cloud feature output by the first convolution layer and the processed three-dimensional point cloud feature to generate the fused three-dimensional point cloud feature , Input the fused three-dimensional point cloud feature into the second convolutional layer, and after the convolution operation of the second convolutional layer, determine the target three-dimensional point cloud feature, wherein the second The depth of the convolutional layer is greater than the depth of the first convolutional layer.

Optionally, the number of the fully connected layers is multiple.

Optionally, the number of up-sampling layers is multiple.

Exemplarily, as shown in FIG. 3, the above-mentioned three-dimensional convolutional neural network includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a first upsampling layer, and a first full convolutional layer. The connection layer, the fifth convolutional layer, the second upsampling layer, the second fully connected layer, and the sixth convolutional layer. Among them, the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer The activation functions of the build-up layer, the fifth convolutional layer, and the sixth convolutional layer all use Relu.

The first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer sequentially perform three-dimensional feature extraction on the extracted three-dimensional point cloud features. The first up-sampling layer is used for spatial resolution according to the first preset Upsampling the three-dimensional features extracted by the fourth convolutional layer at a high rate. The first fully connected layer is used to perform feature fusion on the three-dimensional features sampled on the first upsampling layer and the three-dimensional features extracted by the third convolutional layer. Volume 5 The buildup layer is used to extract 3D features from the fused features of the first fully connected layer, the second upsampling layer is used to upsample the 3D features extracted by the fifth convolutional layer according to the second preset spatial resolution, and the second The fully connected layer is used to perform feature fusion on the three-dimensional features sampled on the second upsampling layer and the three-dimensional features extracted by the second convolution layer, and the sixth convolution layer is used to perform three-dimensional features on the fused features of the second fully connected layer Extract and determine the three-dimensional point cloud features of the target.

Wherein, the depths of the convolutional layers in the above-mentioned multiple convolutional layers are different. The depths of the first convolutional layer, the second convolutional layer, the third convolutional layer, and the fourth convolutional layer increase in order.

The above-mentioned up-sampling layer can enlarge the image and increase the information of the image. For example, the convolutional layer reduces the resolution of the image. After up-sampling by the up-sampling layer, the quality of the up-sampled image can exceed the quality of the original image, which facilitates subsequent processing.

The above-mentioned fully connected layer realizes the splicing and fusion of data, and the data output by multiple convolutional layers (some of which are upsampled by the upsampling layer) are input into different fully connected layers respectively, and the deep and shallow features are merged.

S203: Determine target information in the to-be-processed point cloud according to the characteristics of the target three-dimensional point cloud, where the target information includes geometric position information of the target in each dimension in a three-dimensional coordinate system.

The point cloud to be processed includes the point cloud to be processed corresponding to the target. Here, the point cloud can be coded and used as the input of the neural network, and the characteristics of the point cloud can be learned through the neural network, which can be directly used for the prediction of the three-dimensional target. The prediction is a dense prediction on the feature map of the neural network, and the final detection result can be obtained through end-to-end learning.

Optionally, the determining the target information in the to-be-processed point cloud according to the characteristics of the target three-dimensional point cloud includes:

The first convolutional neural network is used to determine the center point coordinates, the three-dimensional size, and the yaw angle of the target in the to-be-processed point cloud in the three-dimensional coordinate system based on the characteristics of the target three-dimensional point cloud.

Wherein, the first convolutional neural network is obtained by training of point cloud features and the center point coordinates, three-dimensional size, and yaw angle of the target in a three-dimensional coordinate system.

Here, for each scene point cloud, first extract its features, and then use the convolutional neural network to determine the target information based on the above features, which can be represented by 7 parameters, (x, y, z) represents the coordinates of its center point, (l, h, w) represents its three-dimensional size, and r represents its yaw angle. Among them, the above-mentioned target information can be expressed by selecting the center point, the three-dimensional size, and the yaw angle, as well as the coordinates of the corner points, the coordinates of the bottom rectangle, the height, and the like.

Specifically, the determining the target information in the to-be-processed point cloud according to the characteristics of the target three-dimensional point cloud includes:

The target information is determined according to the effective points in the target three-dimensional point cloud feature.

Optionally, before the determining the target information according to the effective points in the target three-dimensional point cloud feature, the method further includes:

Obtaining the pixel Euclidean distance between each point in the target three-dimensional point cloud feature and the center point of the grid where it is located;

According to the Euclidean distance of the pixel and a preset distance threshold, an effective point in the target three-dimensional point cloud feature is determined.

Here, after the point cloud to be processed is divided into grids, each grid is mapped to the feature map to obtain Ri. In this way, for each voxel point on the above-mentioned target 3D point cloud feature, first find the nearest Ri, the judgment method can use the pixel Euclidean distance L between this point and the center point. At this time, a distance threshold T will be set at the same time. If L is less than T, this point is a valid point, otherwise it is an invalid point. According to all the effective points, determine the above-mentioned target information, namely the coordinates of the center point, the three-dimensional size and the yaw angle. The distance threshold T can be set according to actual conditions. For example, taking the target as a vehicle as an example, the above threshold can be set according to the length of the vehicle body.

In addition, after the target information in the to-be-processed point cloud is determined based on the target three-dimensional point cloud feature, a second convolutional neural network can be used to determine the target category probability based on the target three-dimensional point cloud feature ；

Error target removal is performed on the target according to the category probability of the target.

Wherein, the second convolutional neural network is obtained through point cloud feature and target category probability training.

Here, the target information in the point cloud to be processed determined above may be a large amount (obtained by intensive prediction of each 3D voxel point). In order to obtain accurate results, non-maximum value suppression and corresponding score thresholds can be set, Get the final test result. Specifically, the category probability of the target can be determined based on the above-mentioned three-dimensional point cloud characteristics of the target, for example, the probability of being a vehicle is A%. Finally, the corresponding relationship between the target information and the category probability of the target can be determined. For example, the category probability of the target corresponding to the first target information: the probability of being a vehicle is 99%; the category probability of the target corresponding to the second target information: the probability of being a vehicle is 10%. According to the corresponding relationship between the target information and the class probability of the target, the wrong target is removed, for example, the above-mentioned second target is removed, and so on, to obtain the final detection result.

In the target detection method provided in this embodiment, the point cloud vector is determined by the number of points in the point cloud to be processed and the attributes of each point; then the point cloud vector is used to extract features of the target three-dimensional point cloud from the point cloud vector and retain Height information, which retains the three-dimensional structure information of the point cloud to the greatest extent, and then determines the target information in the point cloud to be processed according to the extracted target three-dimensional point cloud features, where the target information includes the geometric position of the target in each dimension in the three-dimensional coordinate system Information, accurately find out the three-dimensional target in the point cloud.

Among them, the use of convolutional neural network to perform three-dimensional target object detection on point cloud data, including detecting the coordinates of the object relative to the sensor, three-dimensional size, the yaw angle of the object in the real world, etc., so that the point cloud data can be used to detect Dynamic obstacles, guide airplanes, cars, robots, etc. to avoid obstacles and plan paths.

Especially for self-driving cars, they are usually equipped with lidar sensors. Obstacle detection through laser point clouds is a very important part of the entire technical link.

FIG. 4 is a schematic flowchart of another target detection method provided by an embodiment of the application. This embodiment, on the basis of the embodiment in FIG. 2, describes in detail the specific implementation process of this embodiment. As shown in Figure 4, the method includes:

S401: Perform grid division on the point cloud to be processed.

S402: Adjust the number of points in each grid after division according to the preset number of points.

S403. Determine the vector of each grid after division according to the product of the number of points in each grid after adjustment and the attributes of each point, where the attributes of each point include the three-dimensional coordinates of the point in the three-dimensional coordinate system and Reflectivity.

Here, the entire three-dimensional space is divided into quantitative small grids, and then the feature vector of each grid is determined, that is, the point cloud is preprocessed (point cloud coding) through a certain structured method to maximize the preservation of the point cloud Original structure information.

S404: Use the third convolutional neural network to extract the three-dimensional grid features of the divided vectors of each grid, where the third convolutional neural network is obtained by training the three-dimensional grid vector and the three-dimensional grid feature.

S405: Perform target 3D point cloud feature extraction on the extracted 3D grid features using the fourth convolutional neural network, where the fourth convolutional neural network is obtained by training on the 3D grid feature and the 3D point cloud feature.

Among them, the third convolutional neural network and the fourth convolutional neural network are both three-dimensional convolutional neural networks.

For feature extraction, in order to retain three-dimensional structure information, the embodiment of the present application adopts three-dimensional convolution to extract features from three-dimensional space and retain spatial structure information.

S406. Use the first convolutional neural network to determine the center point coordinates, the three-dimensional size, and the yaw angle of the target in the three-dimensional coordinate system of the target in the point cloud to be processed based on the above-mentioned target three-dimensional point cloud feature, where the first convolutional neural network Through point cloud features and the target's center point coordinates, three-dimensional size and yaw angle training in the three-dimensional coordinate system.

After the point cloud is encoded above, it is used as the input of the neural network, and the characteristics of the point cloud are learned through the neural network, which is directly used for the prediction of the three-dimensional target. The prediction is a dense prediction on the feature map of the neural network, and the final detection result can be obtained through end-to-end learning.

S407. Use the second convolutional neural network to determine the category probability of the target based on the target three-dimensional point cloud feature, where the second convolutional neural network is obtained through point cloud feature and target category probability training.

S408: Perform wrong target removal on the target according to the class probability of the target.

Since the target information in the point cloud to be processed determined above may be a large amount (obtained by intensive prediction of each three-dimensional voxel point), in order to obtain an accurate result, the wrong target is removed to obtain the final detection result.

In the target detection method provided in this embodiment, the point cloud vector is determined by the number of points in the point cloud to be processed and the attributes of each point; then the point cloud vector is used to extract features of the target three-dimensional point cloud from the point cloud vector and retain Height information, which retains the three-dimensional structure information of the point cloud to the greatest extent, and then determines the target information in the point cloud to be processed according to the extracted target three-dimensional point cloud features, where the target information includes the geometric position of the target in each dimension in the three-dimensional coordinate system Information, accurately find the three-dimensional target in the point cloud, and solve the problem that the existing target detection results have a high error rate, and the subsequent failure to guide airplanes, cars, robots, etc. to avoid obstacles and path planning.

FIG. 5 is a schematic structural diagram of a target detection device provided by an embodiment of the application. For ease of description, only the parts related to the embodiments of the present application are shown. As shown in FIG. 5, the target detection device 50 includes: a first determination module 501, an extraction module 502, and a second determination module 503.

The first determining module 501 is configured to determine a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, and the point cloud vector includes the geometric position information of the points in each dimension in the three-dimensional coordinate system.

The extraction module 502 is configured to process the point cloud vector by using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector.

The second determining module 503 is configured to determine target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud, where the target information includes geometric position information of the target in each dimension in a three-dimensional coordinate system.

The device provided in this embodiment can be used to implement the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and details are not described herein again in this embodiment.

Fig. 6 is a schematic structural diagram of another target detection device provided by an embodiment of the present invention. As shown in FIG. 6, this embodiment is based on the embodiment in FIG. 5, and the foregoing target detection device further includes: a third determination module 504 and a removal module 505.

In a possible design, the second determining module 503 is specifically configured to:

In a possible design, the first convolutional neural network is obtained by training of point cloud features and the center point coordinates, three-dimensional size, and yaw angle of the target in a three-dimensional coordinate system.

In a possible design, the third determining module 504 is configured to use the second determining module 503 to determine the target information in the to-be-processed point cloud according to the target three-dimensional point cloud characteristics. The convolutional neural network determines the class probability of the target based on the characteristics of the target three-dimensional point cloud.

The removal module 505 is configured to remove erroneous targets for the target according to the class probability of the target.

In a possible design, the second convolutional neural network is obtained through point cloud feature and target category probability training.

In a possible design, the first determining module 501 is specifically configured to:

Meshing the point cloud to be processed;

In a possible design, the first determining module 501 determines the vector of each grid after division according to the number of points in each grid after division and the attributes of each point, including:

In a possible design, the three-dimensional convolutional neural network includes a third convolutional neural network.

The extraction module 502 is specifically used for:

In a possible design, the third convolutional neural network is obtained through training of three-dimensional grid vectors and three-dimensional grid features.

In a possible design, the three-dimensional convolutional neural network further includes a fourth convolutional neural network.

After the extraction module 502 uses the third convolutional neural network to extract the three-dimensional grid features of the divided vectors of each grid, it is also used to:

In a possible design, the fourth convolutional neural network is obtained by training on three-dimensional grid features and three-dimensional point cloud features

In a possible design, before determining the target information according to the effective points in the target three-dimensional point cloud feature, the second determining module 503 is further configured to:

In a possible design, the three-dimensional convolutional neural network includes:

In a possible design, the depths of the convolutional layers in the multiple convolutional layers are different;

In a possible design, the number of the fully connected layers is multiple.

In a possible design, the number of the up-sampling layers is multiple.

The device provided in the embodiment of the present application can be used to implement the technical solutions of the foregoing method embodiments, and its implementation principles and technical effects are similar, and the details of the embodiments of the present application are not repeated here.

FIG. 7 is a schematic diagram of the hardware structure of a target detection device provided by an embodiment of the application. As shown in FIG. 7, the target detection device 70 of this embodiment includes: a memory 701 and a processor 702; wherein

The memory 701 is used to store program instructions;

The processor 702 is configured to execute program instructions stored in the memory. When the program instructions are executed, the processor executes the following steps:

In a possible design, the determining the target information in the to-be-processed point cloud according to the characteristics of the target three-dimensional point cloud includes:

In a possible design, after the target information in the point cloud to be processed is determined according to the characteristics of the target three-dimensional point cloud, the method further includes:

Using a second convolutional neural network to determine the class probability of the target based on the characteristics of the target three-dimensional point cloud;

In a possible design, the second convolutional neural network is trained by point cloud features and target category probabilities.

In a possible design, the determining the point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point includes:

Meshing the point cloud to be processed;

In a possible design, the determination of the vector of each grid after division according to the number of points in each grid after division and the attributes of each point includes:

In a possible design, the three-dimensional convolutional neural network includes a third convolutional neural network;

In a possible design, the three-dimensional convolutional neural network further includes a fourth convolutional neural network;

In a possible design, the fourth convolutional neural network is obtained through training of three-dimensional grid features and three-dimensional point cloud features.

In a possible design, before the determining the target information according to the effective points in the target three-dimensional point cloud feature, the method further includes:

In a possible design, the number of the fully connected layers is multiple.

In a possible design, the number of the up-sampling layers is multiple.

In a possible design, the memory 701 may be independent or integrated with the processor 702.

When the memory 701 is set independently, the target detection device further includes a bus 703 for connecting the memory 701 and the processor 702.

In a possible design, the target detection device 70 may be a single device, and the device includes a complete set of the foregoing memory 701, processor 702, and so on. In addition, taking a vehicle as an example, the components of the target detection device 70 may be distributed and integrated on the vehicle, that is, the memory 701, the processor 702, etc. may be respectively arranged in different positions of the vehicle.

FIG. 8 is a schematic structural diagram of a movable platform provided by an embodiment of the application. As shown in FIG. 8, the movable platform 80 of this embodiment includes: a movable platform body 801 and a target detection device 802; the target detection device 802 is provided on the movable platform body 801, the movable platform body 801 and the The target detection device 802 is connected wirelessly or wiredly.

The target detection device 802 determines a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, and the point cloud vector includes geometric position information of the points in each dimension in a three-dimensional coordinate system;

Meshing the point cloud to be processed;

In a possible design, the number of the fully connected layers is multiple.

In a possible design, the number of the up-sampling layers is multiple.

The movable platform provided in this embodiment includes: a movable platform body and a target detection device. The target detection device is set on the movable platform body. The target detection device passes through the number of points in the point cloud to be processed and the number of points in each point. Attribute, determine the point cloud vector; then use the 3D convolutional neural network to extract the target 3D point cloud feature from the point cloud vector, retain the height information, and then retain the maximum 3D structure information of the point cloud, and then according to the extracted target 3D point cloud feature Determine the target information in the point cloud to be processed, where the target information includes the geometric position information of the target in each dimension in the three-dimensional coordinate system, accurately find the three-dimensional target in the point cloud, and solve the high error rate of the existing target detection results. The follow-up can not guide aircraft, cars, robots and other issues in obstacle avoidance and path planning.

The embodiment of the present application provides a computer-readable storage medium having program instructions stored in the computer-readable storage medium, and when a processor executes the program instructions, the target detection method as described above is implemented.

In the several embodiments provided by the present invention, it should be understood that the disclosed device and method may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the modules is only a logical function division, and there may be other divisions in actual implementation, for example, multiple modules can be combined or integrated. To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or modules, and may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional modules in the various embodiments of the present invention may be integrated into one processing unit, or each module may exist alone physically, or two or more modules may be integrated into one unit. The units formed by the above modules can be implemented in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated modules implemented in the form of software functional modules may be stored in a computer readable storage medium. The above-mentioned software function module is stored in a storage medium and includes a number of instructions to make a computer device (which may be a personal computer, a server, or a network device, etc.) or a processor (English: processor) execute the various embodiments of the present application Part of the method.

It should be understood that the foregoing processor may be a central processing unit (Central Processing Unit, CPU for short), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), and application specific integrated circuits (Application Specific Integrated Circuits). Referred to as ASIC) and so on. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor or the like. The steps of the method disclosed in combination with the invention can be directly embodied as executed and completed by a hardware processor, or executed and completed by a combination of hardware and software modules in the processor.

The memory may include a high-speed RAM memory, or may also include a non-volatile storage NVM, such as at least one disk storage, and may also be a U disk, a mobile hard disk, a read-only memory, a magnetic disk, or an optical disk.

The bus may be an Industry Standard Architecture (ISA) bus, Peripheral Component (PCI) bus, or Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus and so on. For ease of representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

The above-mentioned storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Except programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium may be any available medium that can be accessed by a general-purpose or special-purpose computer.

An exemplary storage medium is coupled to the processor, so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium may also be an integral part of the processor. The processor and the storage medium may be located in Application Specific Integrated Circuits (ASIC for short). Of course, the processor and the storage medium may also exist as discrete components in the electronic device or the main control device.

A person of ordinary skill in the art can understand that all or part of the steps in the foregoing method embodiments can be implemented by a program instructing relevant hardware. The aforementioned program can be stored in a computer readable storage medium. When the program is executed, it executes the steps including the foregoing method embodiments; and the foregoing storage medium includes: ROM, RAM, magnetic disk, or optical disk and other media that can store program codes.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The technical solutions recorded in the foregoing embodiments can still be modified, or some or all of the technical features can be equivalently replaced; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the technical solutions of the embodiments of the present invention. range.

Claims

A target detection method is characterized in that it comprises:

Determine a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, the point cloud vector including geometric position information of the points in each dimension in the three-dimensional coordinate system;

Processing the point cloud vector by using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector;

The target information in the point cloud to be processed is determined according to the characteristics of the target three-dimensional point cloud, and the target information includes geometric position information of the target in each dimension in the three-dimensional coordinate system.
The method according to claim 1, wherein the determining the target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud comprises:

The first convolutional neural network is used to determine the center point coordinates, the three-dimensional size, and the yaw angle of the target in the to-be-processed point cloud in the three-dimensional coordinate system based on the characteristics of the target three-dimensional point cloud.
The method according to claim 2, wherein the first convolutional neural network is obtained by training of point cloud features and the center point coordinates, three-dimensional size, and yaw angle of the target in a three-dimensional coordinate system.
The method according to any one of claims 1 to 3, wherein after the determining the target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud, the method further comprises:

Using a second convolutional neural network to determine the class probability of the target based on the characteristics of the target three-dimensional point cloud;

Error target removal is performed on the target according to the category probability of the target.
The method according to claim 4, wherein the second convolutional neural network is obtained through point cloud feature and target category probability training.
The method according to any one of claims 1 to 5, wherein the determining a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point comprises:

Meshing the point cloud to be processed;

According to the number of points in each grid after division and the attributes of each point, determine the vector of each grid after division, where the attributes of each point include the three-dimensional coordinates and reflectivity of the point in the three-dimensional coordinate system;

The point cloud vector is determined according to the vector of each grid after the division.
The method according to claim 6, wherein the determining the vector of each grid after division according to the number of points in each grid after division and the attributes of each point comprises:

Adjust the number of points in each grid after division according to the preset number of points;

According to the product of the number of points in each grid after adjustment and the attributes of each point, the vector of each grid after division is determined.
The method according to claim 6 or 7, wherein the three-dimensional convolutional neural network comprises a third convolutional neural network;

The processing the point cloud vector using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector includes:

The third convolutional neural network is used to extract the three-dimensional grid feature of each grid vector after division.
The method according to claim 8, wherein the third convolutional neural network is obtained through training of three-dimensional grid vectors and three-dimensional grid features.
The method according to claim 8 or 9, wherein the three-dimensional convolutional neural network further comprises a fourth convolutional neural network;

After the third convolutional neural network is used to extract the three-dimensional grid features of the divided vectors of each grid, the method further includes:

The fourth convolutional neural network is used to extract features of the target three-dimensional point cloud from the extracted three-dimensional grid features.
The method according to claim 10, wherein the fourth convolutional neural network is obtained through training of three-dimensional grid features and three-dimensional point cloud features.
The method according to any one of claims 1 to 11, wherein the determining the target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud comprises:

The target information is determined according to the effective points in the target three-dimensional point cloud feature.
The method according to claim 12, wherein before the determining the target information according to the effective points in the target three-dimensional point cloud feature, the method further comprises:

Obtaining the pixel Euclidean distance between each point in the target three-dimensional point cloud feature and the center point of the grid where it is located;

According to the Euclidean distance of the pixel and a preset distance threshold, an effective point in the target three-dimensional point cloud feature is determined.
The method according to any one of claims 1 to 13, wherein the three-dimensional convolutional neural network comprises:

Multiple convolutional layers for performing convolution operations on the point cloud vector to output three-dimensional point cloud features;

The up-sampling layer is connected to at least one convolutional layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by at least one of the multiple convolutional layers, and to compare the obtained 3D point cloud features are processed to output the processed 3D point cloud features;

The fully connected layer is connected to the convolutional layer and the upsampling layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by a convolutional layer and the processed three-dimensional point cloud features, And perform feature fusion on the acquired three-dimensional point cloud features and the processed three-dimensional point cloud features to generate a fused three-dimensional point cloud feature, and input the fused three-dimensional point cloud feature into another convolutional layer to After the convolution operation of the other convolution layer, the target three-dimensional point cloud feature is determined.
The method according to claim 14, wherein the depths of the convolutional layers in the plurality of convolutional layers are different;

The fully connected layer is connected to the first convolutional layer, the second convolutional layer, and the up-sampling layer among the plurality of convolutional layers to obtain the three-dimensional point cloud features output by the first convolutional layer and The processed three-dimensional point cloud feature, and feature fusion is performed on the three-dimensional point cloud feature output by the first convolution layer and the processed three-dimensional point cloud feature to generate the fused three-dimensional point cloud feature , Input the fused three-dimensional point cloud feature into the second convolutional layer, and after the convolution operation of the second convolutional layer, determine the target three-dimensional point cloud feature, wherein the second The depth of the convolutional layer is greater than the depth of the first convolutional layer.
The method according to claim 14 or 15, wherein the number of the fully connected layers is multiple.
The method according to any one of claims 14 to 16, wherein the number of the up-sampling layer is multiple.
A target detection device, which is characterized by comprising a memory, a processor, and computer-executable instructions stored in the memory and running on the processor, and the processor implements the following steps when the processor executes the computer-executable instructions :

Determine a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point, the point cloud vector including geometric position information of the points in each dimension in the three-dimensional coordinate system;

Processing the point cloud vector by using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector;

The target information in the point cloud to be processed is determined according to the characteristics of the target three-dimensional point cloud, and the target information includes geometric position information of the target in each dimension in the three-dimensional coordinate system.
The device according to claim 18, wherein the determining the target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud comprises:

The first convolutional neural network is used to determine the center point coordinates, the three-dimensional size, and the yaw angle of the target in the to-be-processed point cloud in the three-dimensional coordinate system based on the characteristics of the target three-dimensional point cloud.
The device according to claim 19, wherein the first convolutional neural network is obtained by training of point cloud features and center point coordinates, three-dimensional size and yaw angle of the target in a three-dimensional coordinate system.
The device according to any one of claims 18 to 20, characterized in that, after determining the target information in the to-be-processed point cloud according to the characteristics of the target three-dimensional point cloud, it further comprises:

Using a second convolutional neural network to determine the class probability of the target based on the characteristics of the target three-dimensional point cloud;

Error target removal is performed on the target according to the category probability of the target.
The device according to claim 21, wherein the second convolutional neural network is obtained through point cloud feature and target category probability training.
The device according to any one of claims 18 to 22, wherein the determining a point cloud vector according to the number of points in the point cloud to be processed and the attributes of each point comprises:

Meshing the point cloud to be processed;

According to the number of points in each grid after division and the attributes of each point, determine the vector of each grid after division, where the attributes of each point include the three-dimensional coordinates and reflectivity of the point in the three-dimensional coordinate system;

The point cloud vector is determined according to the vector of each grid after the division.
The device according to claim 23, wherein the determining the vector of each grid after division according to the number of points in each grid after division and the attributes of each point comprises:

Adjust the number of points in each grid after division according to the preset number of points;

According to the product of the number of points in each grid after adjustment and the attributes of each point, the vector of each grid after division is determined.
The device according to claim 23 or 24, wherein the three-dimensional convolutional neural network comprises a third convolutional neural network;

The processing the point cloud vector using a three-dimensional convolutional neural network to extract the target three-dimensional point cloud feature of the point cloud vector includes:

The third convolutional neural network is used to extract the three-dimensional grid feature of each grid vector after division.
The device according to claim 25, wherein the third convolutional neural network is obtained by training with a three-dimensional grid vector and a three-dimensional grid feature.
The device according to claim 25 or 26, wherein the three-dimensional convolutional neural network further comprises a fourth convolutional neural network;

After the third convolutional neural network is used to extract the three-dimensional grid features of the divided vectors of each grid, the method further includes:

The fourth convolutional neural network is used to extract features of the target three-dimensional point cloud from the extracted three-dimensional grid features.
The device according to claim 27, wherein the fourth convolutional neural network is obtained through training of three-dimensional grid features and three-dimensional point cloud features.
The device according to any one of claims 18 to 28, wherein the determining the target information in the point cloud to be processed according to the characteristics of the target three-dimensional point cloud comprises:

The target information is determined according to the effective points in the target three-dimensional point cloud feature.
The device according to claim 29, characterized in that, before the determining the target information according to the effective points in the target three-dimensional point cloud feature, the method further comprises:

Obtaining the pixel Euclidean distance between each point in the target three-dimensional point cloud feature and the center point of the grid where it is located;

According to the Euclidean distance of the pixel and a preset distance threshold, an effective point in the target three-dimensional point cloud feature is determined.
The device according to any one of claims 18 to 30, wherein the three-dimensional convolutional neural network comprises:

Multiple convolutional layers for performing convolution operations on the point cloud vector to output three-dimensional point cloud features;

The up-sampling layer is connected to at least one convolutional layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by at least one of the multiple convolutional layers, and to compare the obtained 3D point cloud features are processed to output the processed 3D point cloud features;

The fully connected layer is connected to the convolutional layer and the upsampling layer of the multiple convolutional layers, and is used to obtain the three-dimensional point cloud features output by a convolutional layer and the processed three-dimensional point cloud features, And perform feature fusion on the acquired three-dimensional point cloud features and the processed three-dimensional point cloud features to generate a fused three-dimensional point cloud feature, and input the fused three-dimensional point cloud feature into another convolutional layer to After the convolution operation of the other convolution layer, the target three-dimensional point cloud feature is determined.
The device according to claim 31, wherein the depths of the convolutional layers in the plurality of convolutional layers are different;

The fully connected layer is connected to the first convolutional layer, the second convolutional layer, and the up-sampling layer among the plurality of convolutional layers to obtain the three-dimensional point cloud features output by the first convolutional layer and The processed three-dimensional point cloud feature, and feature fusion is performed on the three-dimensional point cloud feature output by the first convolution layer and the processed three-dimensional point cloud feature to generate the fused three-dimensional point cloud feature , Input the fused three-dimensional point cloud feature into the second convolutional layer, and after the convolution operation of the second convolutional layer, determine the target three-dimensional point cloud feature, wherein the second The depth of the convolutional layer is greater than the depth of the first convolutional layer.
The device according to claim 31 or 32, wherein the number of the fully connected layers is multiple.
The device according to any one of claims 31 to 33, wherein the number of the upsampling layers is multiple.
A movable platform, characterized in that it comprises:

The movable platform body; and

The target detection device according to any one of claims 18 to 34, wherein the target detection device is installed on the movable platform body.
A computer-readable storage medium, wherein a computer-executable instruction is stored in the computer-readable storage medium, and when the processor executes the computer-executable instruction, the computer-executable instruction is implemented as described in any one of claims 1 to 17 Target detection method.