CN111681212B - Three-dimensional target detection method based on laser radar point cloud data - Google Patents

Abstract

The invention discloses a three-dimensional target detection method based on laser radar point cloud data, which adopts a dense data expression form according to the data characteristics of laser radar point cloud so as to obtain dense characteristics and convert three-dimensional characteristics into two-dimensional characteristics, thereby effectively improving the operation efficiency and the operation precision.

Description

A three-dimensional target detection method based on lidar point cloud data

technical field

The invention relates to the technical field of three-dimensional target detection in automatic driving, in particular to a three-dimensional target detection method based on laser radar point cloud data.

Background technique

Lidar can obtain object information in three-dimensional space, and calculate the position information of the object in space through the reflection time of the surface of the object.

In the process of vehicle driving, the detection of three-dimensional objects located around the vehicle is a basic part of automatic driving. Current autonomous vehicles generally detect objects by fusing image RGB information with lidar point cloud information. This patent only uses lidar point cloud data as input for the detection of objects of interest. Although significant progress has been made in 2D target detection in 2D images and a very high detection accuracy has been achieved, the detection effect of 3D LiDAR point cloud in scenarios such as unmanned driving is still poor, and this is mainly Caused by the sparsity of the lidar point cloud.

Further, Apple proposed VoxelNet in 2018 to perform object detection on lidar point cloud input data. It voxelizes the lidar point cloud data in the space, divides the space into independent voxels, and uses the PointNet-like network VFELayer (feature learning network) to extract features from the point cloud data in the voxels. Finally, object detection is performed using the 3D volume and stitching the features in the top view direction.

However, VoxelNet divides the three-dimensional space into three-dimensional space, and extracts features from the point cloud in each segmented voxel to form a four-dimensional feature map (three-dimensional space plus one-dimensional features), which requires the use of three-dimensional convolution operations. deal with. Its operation speed is an order of magnitude slower than the two-dimensional convolution operation. At the same time, due to the sparseness of the point cloud, most of the voxels are empty, so the three-dimensional convolution operation has a huge operation and is a useless convolution operation, but it still needs to waste computing resources.

Furthermore, PointPillars is also a network based on spatial voxel segmentation. Different from VoxelNet, it cuts the space into strips of cuboid columns in the direction of the top view, and extracts features from the point cloud in the column. The feature map processed in this way is a three-dimensional feature map (two-dimensional space plus one-dimensional features). To process this feature map, only two-dimensional convolution operations are required, which is the same as the feature map of general RGB image object detection. Therefore, the subsequent framework of 2D RGB image detection can be directly used for processing. The speed and accuracy performance is much higher than that of VoxelNet.

In the process of feature extraction in the column, PointPillars directly fuses the point clouds in the same vertical direction into one feature. This fusion method is relatively rough, and the feature distribution in the vertical direction is not obvious. At the same time, the features in the bird's-eye feature map It is also very sparse, resulting in a lot of wasted computing power in the two-dimensional convolution operation.

SUMMARY OF THE INVENTION

In view of the deficiencies of the prior art, the present invention discusses a three-dimensional target detection method based on lidar point cloud data, aiming at solving the feature sparse problem of VoxelNet and PointPillars, and constructing a dense feature expression form. Compared with VoxelNet, it has the operational efficiency of two-dimensional convolution. Compared with Pointpillars, the point cloud features in the vertical direction are not forcibly compressed together, and more features in the vertical direction of the object are retained, which is better. Represents a target whose features are more important in the vertical direction.

Furthermore, the present invention adopts a dense data expression form according to the data characteristics of the lidar point cloud, so as to obtain dense features and convert three-dimensional features into two-dimensional features, thereby effectively improving computing efficiency and improving computing accuracy.

In order to achieve the above object, the present invention adopts the following technical solutions:

A three-dimensional target detection method based on lidar point cloud data, the method comprises the following steps:

The point cloud is represented as a dense surface map, and the number of rows in the figure is K, where K is the number of channels of the lidar; given a lidar point p=(x,y,z,r,)l, where (x, y, z), r and l∈{0,...,K-1} represent the position, reflectivity and layer number of the point respectively; point p is located in the grid (h,w) of the surface map S ^h×w , where h=l,

(h,w)网格内的深度的计算如下：The surface map projects 3D points into a 2D grid according to the surface of the scene, and for each grid (h,w) of the surface map, obtains the centroid point by averaging all points in the grid

The depth within the (h,w) grid is calculated as follows:

The surface depth map D _map ={d}∈R ^H×W can then be obtained; the surface depth map stores depth information in each grid;

若网格没有任何点，则使用零填充；且网格特征编码器不执行体素特征编码层中的随机采样；A grid feature encoder based on a voxel feature encoding layer that processes each grid of the surface map to generate features for that grid, resulting in a regular 2D surface surface feature map

If the grid does not have any points, zero padding is used; and the grid feature encoder does not perform random sampling in the voxel feature encoding layer;

由网格特征编码器对其分别进行独立处理，生成N个表面特征图，即S^H×W,

然后，由特征串接得到一个多尺度表面特征F∈R^3C×H×W：Surface maps with N different resolutions, namely S ^H×W ,

They are processed independently by the grid feature encoder to generate N surface feature maps, namely S ^H×W ,

Then, a multi-scale surface feature F∈R ^3C×H×W is obtained by feature concatenation:

This multi-scale surface feature is used as the initial input for subsequent modules;

中的前视图特征具有与其输入表面特征F相同的分辨率，但特征的维度不同；has a surface feature convolution module, and obtains a full-resolution output by adding a network deconvolution layer with a low-resolution output; the surface feature convolution module generates

The front view feature in F has the same resolution as its input surface feature F, but the dimension of the feature is different;

Front view features based on depth surface map View conversion module from front view to bird's-eye view, the depth of different objects is different, but the absolute depth obtained from the 2d front-view pseudo-image is not equal; the depth of the object is obtained from the top-view feature , and regresses the height after the view transform.

The points derived from the heatmap ^HO represent the position of the center of the detected object in the top view, i.e., x, z, while the parameter map ^PO contains the parameters of the object, the detection network consists of a common feature extractor and two branches, the heatmap branch and parameter branches.

It should be noted that the view conversion module has two steps: expansion and compression;

In the expansion step, the feature f at each (h, w) position in the FV feature will be mapped to the corresponding position (d, h, w) of the augmented feature map E according to the depth information D, where

Here R is the maximum depth range, if D _map (h, w)>R, set d=D;

In the compression step, a 2D feature map is obtained by extruding the expanded feature map on its H axis by randomly selecting, its size is D × W, dimension c'; finally, the output is processed using M consecutive 2D convolutional layers, Thus, the final top-view feature map is obtained.

The beneficial effect of the present invention is that a dense feature expression form is constructed. Not only has the operational efficiency of two-dimensional convolution, but also the point cloud features in the vertical direction are not forcibly compressed together, retaining more features in the vertical direction of the object, so as to better express the more important objects in the vertical direction. .

Description of drawings

1 is a schematic diagram of the overall framework of three-dimensional target detection of the present invention;

Fig. 2 is the rectangular parallelepiped voxel and surface map of the present invention;

FIG. 3 is a schematic diagram of a comparison diagram of the evaluation results of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. It should be noted that the following examples are based on the technical solution, and provide detailed implementations and specific operation processes, but the protection scope of the present invention is not limited to the present invention. Example.

The present invention is a three-dimensional target detection method based on lidar point cloud data, and the method comprises the following steps:

The point cloud is represented as a dense surface map, and the number of rows in the figure is K, where K is the number of channels of the lidar; given a lidar point p=(x,y,z,r,)l, where (x, y, z), r and l∈{0,...,K-1} represent the position, reflectivity and layer number of the point respectively; point p is located in the grid (h,w) of the surface map S ^h×w , where h=l,

(h,w)网格内的深度的计算如下：The surface map projects 3D points into a 2D grid according to the surface of the scene, and for each grid (h,w) of the surface map, obtains the centroid point by averaging all points in the grid

The depth within the (h,w) grid is calculated as follows:

The surface depth map D _map ={d}∈R ^H×W can then be obtained; the surface depth map stores depth information in each grid;

若网格没有任何点，则使用零填充；且网格特征编码器不执行体素特征编码层中的随机采样；A grid feature encoder based on a voxel feature encoding layer that processes each grid of the surface map to generate features for that grid, resulting in a regular 2D surface surface feature map

If the grid does not have any points, zero padding is used; and the grid feature encoder does not perform random sampling in the voxel feature encoding layer;

由网格特征编码器对其分别进行独立处理，生成N个表面特征图，即S^H×W,

然后，由特征串接得到一个多尺度表面特征F∈R^3C×H×W：Surface maps with N different resolutions, namely S ^H×W ,

They are processed independently by the grid feature encoder to generate N surface feature maps, namely S ^H×W ,

Then, a multi-scale surface feature F∈R ^3C×H×W is obtained by feature concatenation:

This multi-scale surface feature is used as the initial input for subsequent modules;

中的前视图特征具有与其输入表面特征F相同的分辨率，但特征的维度不同；has a surface feature convolution module, and obtains a full-resolution output by adding a network deconvolution layer with a low-resolution output; the surface feature convolution module generates

The front view feature in F has the same resolution as its input surface feature F, but the dimension of the feature is different;

Front view features based on depth surface map View conversion module from front view to bird's-eye view, the depth of different objects is different, but the absolute depth obtained from the 2d front-view pseudo-image is not equal; the depth of the object is obtained from the top-view feature , and regresses the height after the view transform.

The points derived from the heatmap ^HO represent the position of the center of the detected object in the top view, i.e., x, z, while the parameter map ^PO contains the parameters of the object, the detection network consists of a common feature extractor and two branches, the heatmap branch and parameter branches.

It should be noted that the view conversion module has two steps: expansion and compression;

In the expansion step, the feature f at each (h, w) position in the FV feature will be mapped to the corresponding position (d, h, w) of the augmented feature map E according to the depth information D, where

Here R is the maximum depth range, if D _map (h, w)>R, set d=D;

In the compression step, a 2D feature map is obtained by extruding the expanded feature map on its H axis by randomly selecting, its size is D × W, dimension c'; finally, the output is processed using M consecutive 2D convolutional layers, Thus, the final top-view feature map is obtained.

Example

Surface map

Lidar is a commonly used sensor in autonomous driving. For example, the Velodyne HDL-64E lidar records 64 lines of points in the order of the laser beam, while the point distribution between adjacent scan lines is uniform. Based on the observation of the scanning mechanism of the lidar, the present invention expresses the point cloud in a dense form, that is, a surface map. The surface map is a 2D pseudo-image with K rows, where K is the number of lidar channels. Points along the scan direction (usually horizontal) are placed in a row of the surface map, while points along a column of the surface map correspond to points obtained at different channels (ie laser beams) during a single scan. (with the same horizontal but different vertical angles).

表面图根据场景的表面，将三维点投影到二维网格中。Given a lidar point p = (x, y, z, r, l), where (x, y, z), r and l∈{0,...,K-1} represent the position, reflectivity, respectively and the number of layers where the point is located. The point p is located in the grid (h,w) of the surface map ^Sh×w , where h=l,

Surface maps project 3D points onto a 2D grid based on the surface of the scene.

(h,w)网格内的深度的计算如下：For each grid (h,w) of the surface map, obtain the centroid point by averaging all points within the grid

The depth within the (h,w) grid is calculated as follows:

The surface depth _{map Dmap} ={d}∈R ^H×W can then be obtained. The surface depth map stores depth information in each grid and will be used later in the view conversion module.

SurfaceNet

SurfaceNet proposes an accurate detection box method for predicting objects using surface map representations. It consists of four modules (shown in Figure 1): 1) a grid feature encoder, which can process any number of points within each grid; 2) a surface feature convolution module, which uses a 2D backbone network to extract high-level features; 3) a view conversion module, which converts features from Front View (FV) to Bird's Eye View (BEV); and 4) 3D detection box parameter prediction and anchor-free 3D center heatmap prediction.

Grid Feature Encoder

Due to the irregularity of the point cloud, the number of points in the grid is arbitrary. The trellis feature encoder is designed to encode an arbitrary number of points into dense features with a fixed dimension C, as shown in Fig. 1(a).

如果网格没有任何点，则使用零填充。此外，本发明的网格特征编码器不执行VFE中的随机采样。这里有两个原因：1)每个网格点的数量很少；2)每个网格之间点的分布是近似均匀的，并且不需要减少点的不均衡性。The encoder of the present invention is based on a Voxel Feature Encoder layer. The VFE layer processes each mesh of the Surface map to generate features for that mesh, resulting in a regular 2D surface feature map

If the grid does not have any points, zero padding is used. Furthermore, the trellis feature encoder of the present invention does not perform random sampling in the VFE. There are two reasons for this: 1) the number of points per grid is small; 2) the distribution of points between each grid is approximately uniform and there is no need to reduce the unevenness of points.

)。由网格特征编码器对其分别进行独立处理，生成三个表面特征图(即S^H×W,

)。然后，由特征串接得到一个多尺度表面特征F∈R^3C×H×W：In order to facilitate multi-scale three-dimensional target detection, the present invention adopts N surface maps with different resolutions, (ie S ^H×W ,

). They are processed independently by the grid feature encoder to generate three surface feature maps (i.e. S ^H×W ,

). Then, a multi-scale surface feature F∈R ^3C×H×W is obtained by feature concatenation:

This multiscale surface feature is used as the initial input for subsequent modules. For clearer representation, the present invention omits "multi-scale" and uses surface features to represent multi-scale surface features.

Surface Feature Convolutional Module (SFCM)

Since the receptive field of a surface feature is very limited (ie, only within its underlying grid), the present invention uses a 2D convolutional neural network (see Fig. 1(b)) to more efficiently progressively enlarge the receptive field.

中的前视图特征具有与其输入表面特征F相同的分辨率，但特征的维度不同。In general, feature maps generated by 2D convolutional neural networks have lower resolution than their input images due to computational power. In order to avoid the performance degradation caused by low-resolution features (especially in small target detection), the present invention designs a surface feature convolution module SFCM, and obtains full resolution by adding a low-resolution output network deconvolution layer rate output. Therefore, the module generates

The front-view features in F have the same resolution as their input surface features F, but the dimensions of the features are different.

view conversion module

The Front View feature embeds the local surface information of a single mesh and its adjacencies into the front view. However, it is difficult to directly predict absolute depth information from front view features. However, based on the location of the front view feature, the present invention can discriminate height and width information. Therefore, the present invention proposes a view conversion module from the front view to the bird's-eye view based on the front view features of the depth surface map, as shown in Fig. 1(c).

The reasons for using the view transformation module are: 1) the depths of different objects are different, but the absolute depths obtained from the 2d front-view pseudo-image are not equal; 2) the heights of different objects are similar because they always stand on the ground. Therefore, the present invention can easily obtain the depth of an object from top view (BEV) features and regress the height after view transformation.

Specifically, the view transformation module has two steps: expansion and compression. In the expansion step, the feature f at each (h, w) position in the FV feature will be mapped to the corresponding position (d, h, w) of the augmented feature map E according to the depth information D, where

Here R is the maximum depth range. If _Dmap (h,w)>R, then set d=D.

In the compression step, the present invention obtains a 2D feature map by randomly selecting and extruding the expanded feature map on its H axis, the size of which is D×W and the dimension c′.

Finally, M consecutive 2D convolutional layers are used to process the above output, resulting in the final top-view feature map.

3D object detection without anchor boxes

As shown in Figure 2, the present invention treats 3D objects as some points with attributes. The points derived from the heatmap ^HO represent the position (i.e., x, z) of the center of the detected object in the top view, while the parameter map PO contains the parameters of the object, such as height y, size (h, w, ^l ) and rotation angle θ . The detection network of the present invention consists of a common feature extractor and two branches (ie, the heatmap branch and the parameter branch). The present invention uses a common feature extraction module similar to RPN in VoxelNet. The difference is that the present invention uses a two-dimensional convolutional layer to directly process the two-dimensional features output by the view conversion module instead of a three-dimensional convolutional layer. The heatmap and parameter map branches have the same topology and consist of M consecutive 2D convolutional layers.

As shown in Figure 3, the present invention evaluates SurfaceNet on the KITTI 3D object detection dataset, which contains 7481 training and 7518 testing point clouds. The assessment uses three difficulty levels: easy, medium and hard. Due to the limited number of visits to the KITTI test server, the present invention evaluates the method of the present invention by dividing the official training set into 3712 point clouds for training and 3769 point clouds for validation. The 3D bounding box intersection over union (IoU) threshold is set to 0.25% for pedestrian detection. Furthermore, the offline evaluation code of PointRCNN is used to obtain the metrics of the method of the present invention. Further, it can be seen from Figure 3 that SurfaceNet is 66.17%, outperforming the state-of-the-art methods (such as AVOD-FP and PointPillars) by more than 7%. Also, the method of the present invention only uses LiDAR point cloud, while AVOD-FPN uses point cloud and RGB image.

loss function

The SurfaceNet of the present invention predicts the center heatmap H ^O ∈ R ^D×W and the parameter map P ^O ∈ R ^5×D×W of the 3D prediction frame. HO is used to determine the center of the object in the (x, z) plane, while PO is used to ^regress height (y), size (w, h, ^l ) and rotation angle θ.

For the regression of each center heatmap, the present invention uses a mean squared error loss:

L _hm =MSE(H ^gt -H ^o )

H ^gt is generated through a Gaussian heatmap through the position (x, z) of the object's ground-truth center.

For each parameter, the present invention uses the sum of the smoothed L1 losses for each parameter as the total loss function.

where Δy, Δw, Δh, Δl and Δθ are the losses of the corresponding properties

The height loss Δy is defined using the error between the true and predicted values:

Δy=y ^gt -y ^o

The loss of predicted box size {Δw,Δh,Δl} uses a logarithmic loss:

Spin loss definition:

Δθ=sin(θ ^gt -θ ^o )

During training, y ^o , w ^o , h ^o , l ^o and θ ^o are obtained from the parameter prediction map P ^o according to the center position of the ground-truth 3D bounding box, while y ^gt , w ^gt , h ^gt , l ^gt and θ ^gt are parameters corresponding to the true value of the object.

Finally, the overall loss function is defined as follows:

L=L _hm +βL _loc

where β is the balance parameter used to adjust the balance between the two loss terms. Of course, the total loss function described in the present invention is only one of them, and the functional deformation based on this should fall within the protection scope of the present invention.

For those skilled in the art, various corresponding changes and deformations can be given according to the above technical solutions and concepts, and all these changes and deformations should be included within the protection scope of the claims of the present invention.

Claims (2)

1. A three-dimensional target detection method based on laser radar point cloud data is characterized by comprising the following steps:

representing the point cloud into a dense surface map, wherein the number of lines in the map is K, and K is the number of channels of the laser radar; giving a lidar point p ═ (x, y, z, r, l), where (x, y, z), r, and l ∈ {0,..., K-1} denote position, reflectivity, and the number of layers where the point is located, respectively; point p is located on surface map S^h×wIn the grid (h, w) of (a), wherein h ═ l,

surface map three-dimensional points are projected into a two-dimensional grid according to the surface of the scene, for each of the surface mapsGrid (h, w) by averaging all points within the grid, centroid points are obtained

The depth within the (h, w) grid is calculated as follows:

wherein a surface depth map D may then be obtained_map＝{d}∈R^H×W(ii) a The surface depth map stores depth information in each mesh;

grid feature encoder based on voxel feature encoding layer, voxel feature encoding layer processes each grid of surface map to generate features of the grid, thereby generating regular 2D table surface feature map

If the grid has no points, zero padding is used; the grid characteristic encoder does not execute random sampling in the voxel characteristic encoding layer;

surface maps with N different resolutions, i.e. S^H×W,

The three surface feature maps are generated by independently processing the three surface feature maps by a grid feature encoder respectively, namely

Then, a multiscale surface feature F epsilon R is obtained by feature concatenation^3C×H×W：

This multi-scale surface feature is used as an initial input for subsequent modules;

having surface feature convolution modules and byAdding a network deconvolution layer of low resolution output to obtain full resolution output; generated by a surface feature convolution module

The front view features in (a) have the same resolution as their input surface features F, but the dimensions of the features are different;

a view transformation module having front view features based on a depth surface map from a front view to a bird's eye view, the depths of different objects being different but the absolute depths obtained from the 2d front view pseudo-image being unequal; obtaining the depth of the object from the top view characteristics, and regressing the height after view transformation;

from heatmap H^ODerived points represent the position of the center of the detected object in top view, i.e., x, z, while the parameter map P^OContaining the parameters of the objects, the detection network consists of one common feature extractor and two branches, namely a hot map branch and a parameter branch.

2. The lidar point cloud data-based three-dimensional target detection method of claim 1, wherein the view conversion module comprises two steps: expanding and compressing;

in the expansion step, the feature f of each (h, w) position in the FV feature is mapped to the corresponding position (D, h, w) of the augmented feature map E according to the depth information D, wherein

Where R is the maximum depth range, if D_map(h,w)>R, setting D as D;

in the compression step, a 2D characteristic diagram is obtained by randomly selecting a characteristic diagram extruded and expanded on an H axis, wherein the size of the characteristic diagram is DxW and the dimension c'; finally, the output is processed using M consecutive 2D convolutional layers, resulting in the final top view feature map.

Images