CN117437404B - Multi-mode target detection method based on virtual point cloud - Google Patents

Abstract

The present invention relates to the technical field of multimodal target detection, and specifically to a multimodal target detection method based on a virtual point cloud, comprising the following detection steps: inputting a picture into a neural network, extracting features from the picture to obtain key points of the picture; constructing a virtual point cloud through key point information in a virtual point cloud construction network; voxelizing the virtual point cloud and the real point cloud of the picture to obtain voxel organization; inputting the voxelized organization into a target detection network to obtain a detection result; jointly updating parameters in a neural network, a virtual point cloud construction network and a target detection network to obtain a multimodal target detection model composed of the neural network, the virtual point cloud construction network and the target detection network; and inputting a picture to be classified into the multimodal target detection model to obtain a category of the picture. The present invention can effectively improve the accuracy of target detection.

Description

A multimodal target detection method based on virtual point cloud

Technical Field

The present invention relates to the technical field of multimodal target detection, and in particular to a multimodal target detection method based on virtual point cloud.

Background technique

Multimodal object detection refers to a technology that uses multiple different types of sensors or data sources, such as lidar, camera, radar, etc., to fuse information for object detection and positioning. Its purpose is to improve the accuracy and robustness of object detection, while also making the understanding of complex scenes more comprehensive.

At present, there are three main methods for multimodal environment perception: 1. Use multiple sensors to obtain data from each modality, and superimpose and fuse the modal data before perception, which is also called pre-fusion; 2. Design a neural network for each modal data, use the neural network to extract features, obtain the required local features and global features, and superimpose and fuse the modal features corresponding to each modal data at the feature level, which is also called feature fusion; 3. Use the perception results of each modal data to make logical choices and comprehensively obtain the final result, which is also called post-fusion.

In the actual target detection process, it is found that the point cloud data is relatively sparse and the position of the point cloud is disordered, which leads to the problems of missed detection and false detection when using the above-mentioned existing technologies, greatly affecting the accuracy of target detection.

Summary of the invention

In order to avoid and overcome the technical problems existing in the prior art, the present invention provides a multimodal target detection method based on virtual point cloud. The present invention can effectively improve the accuracy of target detection.

To achieve the above object, the present invention provides the following technical solutions:

A multimodal target detection method based on virtual point cloud includes the following detection steps:

S1. Input the image into the neural network and extract features of the image to obtain the key points of the image;

S2, constructing a virtual point cloud using key point information in a virtual point cloud construction network;

The specific steps of step S2 are as follows:

S21, inputting the Gaussian graph of the key point into the coordinate prediction network to obtain the predicted value of the Gaussian graph offset;

S22. Based on the Smoke algorithm, the mean and variance of the depth of all key points are calculated by mathematical statistics, and the three-dimensional coordinates of the key points are calculated by combining the predicted value of the Gaussian graph offset. The coordinate conversion formula is as follows:

z _t = μ _z + δ _z σ _z

S23, inputting the key points into the confidence network to obtain the confidence corresponding to each key point;

S24, selecting key points whose confidence levels are within a set range, and calculating a set number of virtual point clouds located in the point cloud space and the coordinates of these virtual point clouds through the depth values of these key points combined with the intrinsic parameter matrix of the camera;

S3, voxelizing the virtual point cloud and the real point cloud of the image to obtain voxel organization;

S4, inputting the voxelized tissue into the target detection network to obtain a detection result of the voxelized tissue, where the detection result is the image category corresponding to the voxelized tissue;

S5, jointly updating the parameters in the neural network, the virtual point cloud construction network and the target detection network to obtain a multimodal target detection model consisting of the neural network, the virtual point cloud construction network and the target detection network;

The key point loss function in the joint coordinate prediction network and the target loss function in the target detection network form a joint loss function; the parameters in the multimodal target detection model composed of the neural network, the virtual point cloud construction network and the target detection network are updated through the joint loss function to obtain the optimal multimodal target detection model; the number of 3D key points that need to expand the virtual uniform point cloud is recorded to indirectly reflect the accuracy of the monocular network. When the number is small, a large loss weight μ _vp is given to further improve the training efficiency of the first part of the monocular network; the loss optimization calculation formula is as follows:

Where: ΔLoss _i and ΔLoss _i-1 are the loss values of this round and the previous round, n is the number of training rounds that have participated in the training, N is the number of virtual point clouds that conform to the 3D space range constructed in this round of training, N _max is the number of 3D key points selected by the key point network setting, and β is an adjustable minimum value;

The total loss is the sum of the two losses, as follows:

Loss＝μ ₁ *L ₁ +(1-μ ₂ )*L ₂

Where: _L1 is the positioning loss of the 3D key point, _L2 is the loss of the final prediction result;

S6. Input the image to be classified into the multimodal object detection model to obtain the category of the image.

As a further solution of the present invention: the specific steps of step S1 are as follows:

S11, input the image into the neural network as the DLA-34 network to extract features and obtain a corresponding feature map;

S12. Based on CenterNet, obtain the camera coordinates of each point cloud on the feature map;

S13, converting the camera coordinates of the point cloud into projection points on the XY plane in the camera coordinates through a conversion formula;

S14, for each projection point, a two-dimensional Gaussian probability distribution centered at the projection point position is calculated, thereby generating a Gaussian graph;

S15. Add the Gaussian graphs generated by all projection points to form a heat map, where the Gaussian graph generation formula is as follows:

S16. Select the pixel with the largest two-dimensional Gaussian probability in the heat map as the key point.

As a further solution of the present invention: the specific steps of step S3 are as follows: the obtained virtual point cloud and the real point cloud corresponding to the image are voxelized to obtain the processed voxel organization; then the voxel organization is equally divided into individual voxel blocks; and then the point cloud in each voxel block is feature encoded.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention proposes a two-stage multimodal target detection method based on virtual point cloud, that is, using image detection target information to construct a virtual point cloud to assist point cloud-based target detection. This method first constructs a virtual point cloud using image detection target information to increase the density of the point cloud, thereby improving the performance of target features. Secondly, the feature dimension of the point cloud is increased to distinguish between real and virtual point clouds, and voxels with confidence encoding are used to enhance the correlation of the point cloud. Finally, the loss function is designed using the scale coefficient of the virtual point cloud, image detection supervised training is added, the efficiency of two-stage network training is improved, the model error accumulation problem existing in the two-stage end-to-end network model is avoided, and the accuracy and robustness of the target detection system are effectively improved.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of the main detection steps of the present invention.

FIG. 2 is a diagram showing the overall structure of the model of the present invention.

FIG. 3 is a schematic diagram of the construction position of the virtual point cloud within a voxel in the present invention.

Detailed ways

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Please refer to Figures 1 to 3. In an embodiment of the present invention, a multimodal target detection method based on virtual point cloud is mainly composed of multimodal data input of sensors, neural networks, virtual point cloud construction networks, target detection networks and loss joint training. First, the image is sent to the backbone network DLA-34 for feature extraction, and then the coordinate prediction values and target confidence of a certain number of target 3D key points are obtained through the regression network; then, according to the generated key point information, the corresponding virtual point cloud is constructed in the lidar point cloud, and the point cloud feature dimension is increased to distinguish between virtual and real point clouds, and the output target confidence is simultaneously included in the feature encoding, and then sent to the voxel-based 3D target detection together with the real point cloud. At the same time, in order to avoid the model error accumulation problem existing in the two-stage end-to-end series network, the proportional relationship of the virtual point cloud is used to design the loss function, and the image detection is increased. Supervised training, thereby improving the training efficiency of the image processing module.

The detection network proposed in the present invention also performs data expansion on the voxel block where the virtual point cloud is located, as shown in Figure 3. The method is as follows: According to the position information of the virtual point cloud, the corresponding voxel block position can be determined, and the position is marked by verifying whether there is a voxel at the position. If it exists, the voxel at the position is marked by adding a confidence value after the echo. If there is no real point cloud at the position, the spatial distribution of the entire voxel block is considered and added according to the uniform distribution strategy. When designing a single voxel, it is stipulated that a maximum of 5 point clouds are taken. Since the voxel 3D target detection method does not consider much in the height direction, a rectangular section is selected, and four points are evenly constructed, which are added to the entire point cloud data together with the virtual points.

The main contents of the present invention are as follows:

a. Image-based key point detection. After passing the monocular image through the feature extraction network, a feature map of corresponding size is obtained. Based on the idea of CenterNet, the projection point of the target 3D key point on the 2D image is directly predicted. The true value of the 3D key point in the point cloud data is converted to the camera plane projection through the camera formula, and then encoded into a 2D Gaussian map. The Gaussian map is a two-dimensional probability distribution function that assigns higher probability values to pixels near the center of the object and lower probability values to pixels farther from the center. For each key point, a Gaussian map is generated by calculating the two-dimensional Gaussian probability distribution centered on the key point position. The standard deviation of the Gaussian is usually set to a fixed value, which determines the distribution of probability values around the key point. The Gaussian maps of all key points are then added together to generate the final heat map, which represents the probability that each pixel belongs to a specific object category. The pixel with the highest probability value in each heat map is regarded as the position of the corresponding key point. The image feature map passes through a series of networks, and the prediction head outputs a prediction of the Gaussian map offset K is the camera intrinsic parameter. Then, drawing on the idea of SMOKE, we first use mathematical statistics to calculate the mean and variance of the 3D key point depth, and then use the prediction head to predict the depth offset, thereby obtaining the 3D coordinate prediction value [x _p , y _p , z _p ] of the key point.

b. Construct a virtual point cloud. Select N key points with the highest confidence from the final feature map of the neural network; take the predicted depth z, combined with the camera intrinsic parameter transformation matrix, to obtain N virtual 3D points [x _vp ,y _vp ,z _vp ] in the point cloud space; in order to prevent exceeding the front view range of the real point cloud, these virtual point clouds are filtered to obtain N' points, and added to the point cloud data, and their reflection intensity is replaced by the average value of the entire point cloud.

c. Object detection based on point cloud voxelization. The point cloud 3D object detection network is based on voxel features. The main idea is to divide the entire 3D space into voxel blocks of the same size along the x, y, and z axes. The point cloud in each voxel block is feature encoded, and the voxel features are obtained after fully considering its global and local features, and then the object detection is performed using 3D convolution.

Specific implementation method: resize all images to a uniform size (1280*384*3), input them into the network, first extract features through the DLA-34 backbone network to obtain a feature layer, and then use the prediction head and regression head to obtain the parameters required for 3D position prediction. The prediction head predicts the 2D center and category of the target by generating a heat map, and the regression head regresses the 2D center to convert it into the offset required for 3D coordinates. Take some points with the highest eigenvalues in the heat map as key points, use mathematical statistics to calculate the mean and variance of the 3D key point depth, and use the prediction head to predict the depth offset to obtain the 3D coordinate prediction value of the key point. Key points are not all center points. There is only one center point for a target. Key points include the center point and some surrounding points. At this time, we have the coordinates of the key points and the eigenvalues of the key points in the heat map, that is, the confidence size. We then construct a virtual point cloud based on this information and combine it with the camera intrinsic parameter transformation matrix to obtain N virtual 3D points in the point cloud space. Each virtual point increases the confidence data dimension and is then sent to the voxel-based point cloud 3D object detection network together with the real point cloud.

The proposed multimodal detection model was experimented with the KITTI dataset, and the results were compared with several LiDAR and multimodal 3D object detection methods. For vehicle detection, the proposed network performed well, with detection accuracy better than the classic 3D point cloud detection network and some multi-sensor information fusion networks, and the vehicle detection accuracy reached 86.9%.

The 3D detection network proposed in the present invention performs well in barrier-free target detection, and can detect them well even if they are blocked. It also has good results in long-distance target detection. The improvement in accuracy is mainly due to the simultaneous processing of image and laser point cloud information. By obtaining key points from the image to construct a virtual point cloud, the point cloud of the long-distance target in the point cloud space is no longer sparse, so it has a better detection effect on long-distance and small objects.

Different methods were tried during the network training process, including adding the loss bias weight and directly adding the two parts of the loss, and compared through the training convergence process. After introducing the bias weight into the loss function, the convergence speed of the model was significantly accelerated, and the detection effect was also partially improved. This method can not only better balance the two parts of the loss, but also better express the importance of different detection modalities, improving the performance of the model.

The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims (3)

1. A multimodal target detection method based on virtual point cloud, characterized in that it includes the following detection steps: S1. Input the image into the neural network and extract features of the image to obtain the key points of the image; S2, constructing a virtual point cloud using key point information in a virtual point cloud construction network; The specific steps of step S2 are as follows: S21, inputting the Gaussian graph of the key point into the coordinate prediction network to obtain the predicted value of the Gaussian graph offset; S22. Based on the Smoke algorithm, the mean and variance of the depth of all key points are calculated by mathematical statistics, and the three-dimensional coordinates of the key points are calculated by combining the predicted value of the Gaussian graph offset. The coordinate conversion formula is as follows: z _t = μ _z + δ _z σ _z S23, inputting the key points into the confidence network to obtain the confidence corresponding to each key point; S24, selecting key points whose confidence levels are within a set range, and calculating a set number of virtual point clouds located in the point cloud space and the coordinates of these virtual point clouds through the depth values of these key points combined with the intrinsic parameter matrix of the camera; S3, voxelizing the virtual point cloud and the real point cloud of the image to obtain voxel organization; S4, inputting the voxelized tissue into the target detection network to obtain a detection result of the voxelized tissue, where the detection result is the image category corresponding to the voxelized tissue; S5, jointly updating the parameters in the neural network, the virtual point cloud construction network and the target detection network to obtain a multimodal target detection model consisting of the neural network, the virtual point cloud construction network and the target detection network; The key point loss function in the joint coordinate prediction network and the target loss function in the target detection network constitute a joint loss function; the parameters in the multimodal target detection model composed of the neural network, the virtual point cloud construction network and the target detection network are updated through the joint loss function to obtain the optimal multimodal target detection model; the number of 3D key points that need to expand the virtual uniform point cloud is recorded to reflect the accuracy of the monocular network; the loss function calculation formula is as follows: Where: ΔLoss _i and ΔLoss _i-1 are the loss values of this round and the previous round, n is the number of training rounds that have participated in the training, N is the number of virtual point clouds that conform to the 3D space range constructed in this round of training, N _max is the number of 3D key points selected by the key point network setting, and β is an adjustable minimum value; The total loss is the sum of the two losses, as follows: Loss＝μ ₁ *L ₁ +(1-μ ₂ )*L ₂ Where: _L1 is the positioning loss of the 3D key point, _L2 is the loss of the final prediction result; S6. Input the image to be classified into the multimodal object detection model to obtain the category of the image. 2. According to the multimodal target detection method based on virtual point cloud according to claim 1, it is characterized in that the specific steps of step S1 are as follows: S11, input the image into the neural network as the DLA-34 network to extract features and obtain a corresponding feature map; S12. Based on CenterNet, obtain the camera coordinates of each point cloud on the feature map; S13, converting the camera coordinates of the point cloud into projection points on the XY plane in the camera coordinates through a conversion formula; S14, for each projection point, a two-dimensional Gaussian probability distribution centered at the projection point position is calculated, thereby generating a Gaussian graph; S15. Add the Gaussian graphs generated by all projection points to form a heat map, where the Gaussian graph generation formula is as follows: S16. Select the pixel with the largest two-dimensional Gaussian probability in the heat map as the key point. 3. According to claim 2, a multimodal target detection method based on virtual point cloud is characterized in that the specific steps of step S3 are as follows: the virtual point cloud and the real point cloud corresponding to the image are voxelized to obtain the processed voxel organization; then the voxel organization is equally divided into individual voxel blocks; and then the point cloud in each voxel block is feature encoded.