CN116682105A - Millimeter wave radar and visual feature attention fusion target detection method - Google Patents

Abstract

The present invention provides a target detection method for the fusion of millimeter-wave radar and visual feature attention, comprising the steps of: acquiring millimeter-wave radar point cloud data and visual image information; preprocessing point cloud data and visual image fusion in the data layer; Preliminary detection of images to extract image features; target association between radar point cloud and image information to extract radar features; input image features and radar features to feature attention fusion network for fusion; use 3Dbox decoder to decode target detection results and output. The present invention uses the intensity of the millimeter-wave radar scattering cross-section to self-adaptively adjust the size of the spatial information projected from the point cloud onto the image, solving the problem that the size of the existing radar point cloud projected onto the image is fixed; a feature attention fusion of radar and image is proposed Network, which solves the problem of uneven weight distribution in the feature fusion of millimeter-wave radar and visual images, and has the advantages of improving the accuracy and robustness of target detection.

Description

A target detection method based on millimeter-wave radar and visual feature attention fusion

technical field

The invention relates to the technical field of sensor feature fusion for target detection, in particular to a target detection method for millimeter wave radar and visual feature attention fusion.

Background technique

At present, the sensors used for target detection mainly include various sensors such as visual cameras, infrared sensors, laser radars, and millimeter-wave radars. Among them, the camera has been widely used because of its relatively large detection range, ability to obtain more original target information and strong classification ability, but the camera cannot obtain the depth information of the target and is easily affected by different weather and light. In recent years, although the camera has greatly improved in the two-dimensional detection of the target, and the detection accuracy is also high, there is still a big gap in the three-dimensional target detection using only visual image input. Millimeter-wave radar can not only achieve all-weather and all-weather work, but also obtain information such as the relative distance, velocity and scattering cross-section strength of the target from the echo signal. It has relatively strong anti-interference ability, but the point cloud of millimeter-wave radar is very sparse. Susceptible to external clutter and noise. Therefore, for the defect of a single sensor, a method of fusion of multiple sensors is generally used to solve it. Relatively speaking, there are many researches on the fusion of laser radar and camera for target detection. However, due to the narrow detection range and high price of laser radar, more and more researches turn to the fusion of millimeter wave radar and camera. . Combining the advantages of millimeter-wave radar and visual camera, the fusion of the two can make good use of the information provided by the other party to complement each other, making the fused information more abundant and comprehensive, and at the same time avoiding the huge amount of calculation and cost when using lidar high question.

Generally, the fusion methods of millimeter-wave radar and visual camera are divided into three types: data-level fusion, decision-level fusion and feature-level fusion. Among them, feature-level fusion has a large research space. However, due to the sparseness of millimeter-wave radar point cloud data, its representation ability is very weak, and there is still a problem of uneven distribution of feature weights in the fusion of radar features and image features. Most of them use radar features to assist image features as targets. detection. Therefore, how to make full use of the point cloud data to enhance its spatial information, improve the representation ability, and realize the reasonable distribution of radar features and image feature weights has become a key issue when processing millimeter-wave radar point cloud data.

Contents of the invention

In view of the above shortcomings of the prior art, the present invention provides a target detection method for the fusion of millimeter-wave radar and visual feature attention, which is used to solve the problem of sparse point cloud data of millimeter-wave radar, weak representation ability and radar feature and There is also the problem of uneven distribution of feature weights when image features are fused.

In order to achieve the above purpose and other related purposes, the present invention provides a target detection method for the fusion of millimeter wave radar and visual feature attention, the method comprising the following steps:

Step 1: Obtain the millimeter-wave radar point cloud data and the frame image of the visual camera;

Step 2: Preprocess the radar point cloud, and project the radar point onto the image plane to obtain the radar image, and then use the intensity of the radar scattering cross section to control the size of the spatial information projected from the radar point cloud to the image, and the visual image information in the data layer carry out fusion;

Step 3: Carry out preliminary detection on the image and extract image features;

Step 4: Associate the radar point cloud with the image and extract the radar features;

Step 5: Input the image features and radar features into the feature attention fusion module, and reasonably allocate the weights of the two features;

Step 6: Combining the results of the first regression of the network detection head in step 3 and the second regression of step 5, use the 3Dbox decoder to decode the detection result of the target and output it.

Preferably, in the second step of the present invention, the radar point cloud is preprocessed, and the specific steps of projecting the radar point onto the image plane to obtain the radar image are as follows:

Scan the radar detection point multiple times and accumulate its data to increase the density of the point cloud;

Load the point cloud data, filter points by distance by setting the threshold of the appropriate distance and add z offset to the radar point cloud to remove the radar point cloud with abnormal speed;

The radar point is expressed as a 3D point detected by the radar in the egocentric coordinate system, and it is parameterized as P _radar (x, y, z, v _x , v _y , σ), where (x, y, z) represents the target position, v _x , v _y are the radial velocity of the target in the x and y directions respectively, and σ represents the scattering cross section intensity of the target;

Based on the Nuscenes dataset, the millimeter-wave radar points are projected to the image plane through the given camera internal and external parameters in the dataset. After coordinate transformation, the millimeter-wave radar point is converted from the self-coordinate system to the image coordinate system to generate a radar image with the same size as the visual image.

For the radar point cloud data projected onto the image plane, the point cloud data is preliminarily expanded to a height range of 0.5-2.0 meters in the vertical direction and a width range of 0.2-1.5 meters in the horizontal direction, providing radar information and cameras Preliminary correlation between the pixels;

The specific height values of the radar point pixels on the radar image in the horizontal width and vertical height are adaptively adjusted through the scattering cross-section intensity σ of the radar point to obtain a variable-sized two-dimensional point cloud pixel area.

Preferably, in step 4 of the present invention, the radar point cloud is associated with the target of the 2D image, and the specific steps of extracting radar features are as follows:

Use the depth of the 3D bounding box obtained in the preliminary regression in step 4, the observation angle, the 3D size and the camera calibration matrix to generate a 3D viewing frustum;

Expand the radar point cloud to a fixed-size 3D cylinder to increase the correlation rate of the point cloud, and project the cylinder to the pixel coordinate system to match the 2D bounding box, and at the same time project the cylinder to the camera coordinate system to match the constructed 3D The depth value matching of the visual frustum;

Extraction of radar features, for each object-related radar detection, generate 3 radar heatmap channels, located in the center and inside of the 2D bounding box of the object, where the width and height of the heatmap are proportional to the size of the 2D bounding box , the value of the heatmap is determined by the normalized object depth and the x, y components of Vx, Vy in the egocentric coordinate system.

Preferably, in step 5 of the present invention, the image features and radar features are input into the feature attention fusion module to carry out weight learning, and the specific steps for rationally allocating the weights of the two features are as follows:

The feature attention fusion network mainly consists of channel attention module CAM and convolutional layers of different sizes. The first one is a convolutional layer Conv1×1 with a convolution kernel size of 1×1, a step size of (1,1), and a filling of (0,0); the second one is a convolution kernel size of 3×3, The step size is (1, 1), the convolution layer Conv3×3 is filled with (1, 1); the third is the convolution kernel size is 7×7, the step size is (1, 1), and the filling is (3 , 3) the convolutional layer Conv7×7;

The radar features are mainly extracted by one Conv1×1 and two Conv3×3 respectively, and then the attention weight matrix obtained by the three branches is added element by element, and then the feature weight is further extracted by a 7×7 convolutional layer to generate Spatial attention information of radar features; image features are sequentially processed by Conv1×1, Conv3×3 and Conv1×1 to extract image feature weights, and then multiplied with the original image features after being processed by the channel attention module to generate image features channel attention information;

The spatial attention information of radar features and the channel attention information of image features are spliced and fused to generate image radar feature tensors, and then the detection head is used to perform secondary regression to obtain the depth, speed, rotation angle and attributes of the target;

As mentioned above, the characteristics and beneficial effects of the target detection method of a millimeter-wave radar and visual feature attention fusion of the present invention are:

(1) A method is proposed to adaptively adjust the spatial information size of the radar point cloud projected onto the image by using the intensity of the radar scattering cross section of the millimeter wave radar. Height and width size, improve the spatial information of the radar point;

(2) Compared with the state-of-the-art, an attention learning mechanism is added to the feature fusion module. Due to the sparsity of the millimeter-wave radar point cloud, the radar features are processed to generate spatial attention information, and the image features are processed to generate channel attention information, and then the two-way attention information is fused to redistribute the millimeter-wave radar features and visual image features reasonably. The weight of the radar point cloud is enhanced to improve the accuracy and robustness of 3D target detection.

Description of drawings

Fig. 1 is a schematic flow chart of the algorithm of the present invention.

Fig. 2 is a schematic diagram of the target detection network structure of the millimeter-wave radar and visual feature attention fusion of the present invention.

Fig. 3 is a schematic diagram of the structure of the feature attention fusion network proposed by the present invention.

Detailed ways

In order to make the purpose and technical solution of the present invention clearer and easier to understand, the technical solution of the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the following specific examples are only used to explain the present invention, and are not intended to limit the scope of the present invention.

As shown in Figure 2, it is a target detection network structure for the fusion of millimeter-wave radar and visual feature attention. The network structure is mainly composed of five modules: data preprocessing, data layer fusion, feature extraction, feature attention fusion, and target detection.

Data preprocessing module: the preprocessing operation of millimeter wave radar radar point cloud data, and generate radar images.

The detailed process is as follows:

1. Download the Nuscenes data set, from which to read the millimeter-wave radar point cloud data and the frame image information of the visual camera. The millimeter-wave radar point cloud data is mainly the information read from the radar echo signal, including radar Various information such as the distance to the target, the position, the target speed, and the intensity of the target's scattering cross section;

2. Perform preprocessing operations on the radar point cloud. The specific implementation methods include:

(1) Scan the radar detection point multiple times and accumulate its data to increase the density of point cloud data;

(2) Load the point cloud data, calculate the distance from the radar to the target, that is, the depth information, filter points by distance by setting an appropriate threshold and add a z offset to the radar point cloud; then according to the distance of the point cloud according to the rule from small to large The filtered point clouds are sorted. The expression to filter points by distance is:

1m≤d≤100m

Among them, d represents the distance from the radar to the target point.

(3) Express the radar point as a 3D point detected by the radar in the egocentric coordinate system, and parameterize it as P _radar (x, y, z, v _x , v _y , σ), where (x, y, z) represents the position of the target, v _x and v _y represent the radial velocity of the target in the x and y directions respectively, and σ represents the scattering cross-section intensity of the target;

3. Based on the Nuscenes dataset, the millimeter-wave radar points are projected to the image plane through the given camera internal and external parameters in the dataset. Using the coordinate transformation formula, the millimeter-wave radar point is transformed from the self-coordinate system to the image coordinate system, and a radar image with the same size as the visual image is generated. The specific operation of the coordinate conversion is to firstly convert the radar point from the self-coordinate system to the global coordinate system, then convert the global coordinate system to the camera coordinate system, and finally convert the camera coordinate system to the image coordinate system.

4. Due to the sparsity of the millimeter-wave radar point cloud, for the radar point cloud data projected onto the image plane, the point cloud data is preliminarily expanded to a height range of 0.5-2.0 meters in the vertical direction and 0.2 meters in the horizontal direction. -1.5m width range. This step establishes a preliminary association between the radar information and the camera pixels, and at the same time expands the information of the radar point cloud in space; and then adaptively adjusts the lateral direction of the radar point pixels on the radar image through the scattering cross-section intensity σ of the radar point The concrete value of width w _p and vertical height h _p , its expression is:

Among them, σ( ) represents the function of the intensity of the scattering section, w′ _p and h′ _p represent the specific point cloud width and height values respectively; S _p represents the point cloud pixel area.

Through the above operations, the pixel area of the two-dimensional point cloud with variable size can be obtained, which solves the existing problem of projecting the radar point cloud onto the image with a fixed size.

Data layer fusion module: use the depth information obtained from millimeter-wave radar echoes, and the velocity components v _x and v _y on the X and Y axes as pixel values to project into the visual image channel; at positions without radar echoes, the corresponding The pixel values of the radar channel are all set to zero; finally, the visual image added with radar information is converted into a three-channel image to realize the fusion of the millimeter-wave radar point cloud data and the visual image at the data layer.

Feature extraction module: including the extraction of visual image features and the extraction of millimeter-wave radar features.

1. The extraction of visual image features, the detailed process is as follows:

(1) The fusion result of the millimeter-wave radar point cloud data and the visual image in the data layer is used as the input of the CneterNet central point detection network with the improved DLA-34 as the backbone network. The fusion result of the data layer is the three-dimensional image with radar information Channel visual image information I _i+r ∈ ^{R W×H×C} , where W and H represent the height and width of the image, respectively. Generate predicted key point heatmap after downsampling R represents the downsampling rate, C represents the number of object categories; for the generated heat map, the Focal Loss function is established, and its expression is:

Among them, N represents the number of key points in the image; α, β represent hyperparameters, usually α=2, β=4; Represents the ground truth heatmap values of an object generated by a Gaussian kernel.

(2) Predict the center point, 2D size (W and H), center bias, and This provides a rough 3DBox and a precise 2DBox for each detected target.

2. The extraction of millimeter-wave radar features, the detailed process is as follows:

(1) According to the data preprocessing module, the depth information obtained from the millimeter-wave radar echo and the velocity components v _x and v _y on the X and Y axes are converted into pixel values, and a three-dimensional image with the same size as the visual image is generated. channel raw radar point cloud image. According to the rigid transformation formula, the radar point is converted into the camera coordinate system, the formula is as follows:

X _image = RX _radar +T

Among them, X _image and X _radar represent the coordinates of the radar point cloud in the camera and millimeter-wave radar coordinate system, respectively. R and T represent the rotation matrix and translation matrix, respectively.

(2) For the generated radar point cloud image, in order to supplement the height information of the radar, the radar point is expanded into a cylinder with a size of (1.5, 0.2, 0.2) in the [x, y, z] direction to enhance the space of the point cloud Information, and project the cylinder to the pixel coordinate system to associate with the 2D bounding box.

(3) Combining the 2D bounding box, estimated depth information, and camera calibration matrix output in the above visual image feature extraction to create a 3D ROI viewing frustum for the object, and project the cylinder to the camera coordinate system and the constructed 3D viewing frustum The depth value matching of the frustum is carried out, and the points outside the frustum are ignored, and the size of the frustum area is controlled by controlling the parameter δ.

(4) After associating the radar point with the corresponding object, use the depth and velocity information of the radar point cloud to establish complementary features for the image, and generate 3 radar heat map channels (d,v _x ,v _y ) and the radar heat map The width and height of the map are proportional to the 2DBox of the target, and its heat map value is determined by the depth value d of the target and the X and Y components of the radial velocity (v _x and v _y ) in the self-coordinate system, which is expressed as The expression of h _j is as follows:

In the formula, M _i represents the normalization factor; i=1,2,3 represents the feature channel; f _i represents the feature value of the three channels (d,v _x ,v _y ); c _x,j , c _y,j respectively represent the The x and y axis coordinates of the center point of the j target on the image; w _j , h _j represent the width and height of the jth target 2DBox; α represents the hyperparameter, which is used to control the width and height of the 2D Box. If two targets have overlapping heatmap regions, the region with the smaller heatmap value is usually selected.

Feature attention fusion module: Input visual image features and millimeter-wave radar features into the feature attention fusion module. The structural diagram of the feature attention fusion network is shown in Figure 3. The feature attention fusion network includes a channel attention The force module CAM and three different attention weight generation units. The first one is a convolutional layer Conv1×1 with a convolution kernel size of 1×1, a step size of (1,1), and a filling of (0,0); the second one is a convolution kernel size of 3×3, The step size is (1, 1), the convolution layer Conv3×3 is filled with (1, 1); the third is the convolution kernel size is 7×7, the step size is (1, 1), and the filling is (3 , 3) the convolutional layer Conv7×7.

The specific implementation steps of the feature attention fusion network are as follows:

(1) Use one Conv1×1 and two Conv3×3 to extract the weights of the radar features obtained in step 4, and add the attention weight matrix obtained by the three branches by elements, and then go through 7×7 The convolutional layer further extracts the feature weights to generate the spatial attention information of the radar features;

(2) Process the generated image features sequentially through Conv1×1, Conv3×3 and Conv1×1 to extract the image feature weights, and then multiply the original image features with the result of the channel attention module to generate image features. Channel attention information;

(3) Use the spatial attention information of the radar features in (1) and the channel attention information of the image features in (2) to perform splicing and fusion operations to generate image radar feature tensors;

Specifically, the image features are processed by CAM, and the method of establishing a channel attention mechanism includes:

The input visual image feature I _img ∈ ^{R W×H×3} is processed by maximum layering and average pooling respectively to obtain two feature maps of I′ _img ∈ R ^1×1×3 , and then the two feature maps are respectively Passed into the two-layer perceptron network, after the output features are added to each tensor corresponding element, the image channel feature can be generated by using the activation function, and finally multiplied with the original visual image feature by the tensor corresponding element, the The expression of the process is as follows:

M _c (I _img )=Sigmoid(MLP(avgpool(I _img ))+MLP(maxpool(I _img )))

In the formula, Sigmoid( ) is the activation function; MLP is the perceptron network, which is used as a matrix operation; avgpool( ) and maxpool( ) represent mean pooling calculation and maximum pooling calculation respectively; M _c (I _img ) is the output of the image channel attention mechanism module.

Target detection module: The output result of feature attention fusion is used as the input of the secondary regression of the center point detection network detection head of CenterNet, and the depth, speed, rotation angle and attribute information of the target are recalculated. The secondary regression head consists of 3 Conv3×3 The convolutional layer is composed of a Conv1×1 convolutional layer output. Combining the information extracted from the image feature and the depth, speed, rotation angle and attribute information obtained by the regression of the secondary regression head, the detection result of the 3D target is restored through the 3D bounding box decoder, where the depth and pose are included in the two regression heads. Only the results of the quadratic regression with higher accuracy are used.

It should be noted that the above descriptions are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Within the scope of knowledge of those skilled in the art, they can also be used without departing from the gist of the present invention. Make various changes.

Claims (8)

1. a target detection method of millimeter-wave radar and visual feature attention fusion, it is characterized in that, the method comprises the following steps: Step 1: Obtain the millimeter-wave radar point cloud data and the frame image of the visual camera; Step 2: performing a preprocessing operation on the radar point cloud data, and then fusing the preprocessed radar point cloud data with the frame image at the data layer; Step 3: Preliminary detection is performed on the frame image, and image features are extracted; Step 4: Carry out target association between the radar point cloud data and the frame image, and extract radar features; Step 5: Input the image feature and the radar feature into the feature attention fusion module, and reasonably allocate the weights of the two features; Step 6: Use the 3Dbox decoder to decode the detection result of the target and output it. 2. the target detection method of millimeter-wave radar according to claim 1 and visual feature attention fusion, it is characterized in that, in described step 2, the concrete steps that described radar point cloud data are carried out preprocessing operation are: Scan the radar detection point multiple times and accumulate its data to increase the density of the point cloud; Load the point cloud data, filter points by distance by setting the threshold of the appropriate distance and add z offset to the radar point cloud to remove the radar point cloud with abnormal speed; The radar point is expressed as a 3D point detected by the radar in the egocentric coordinate system, and it is parameterized as P _radar (x, y, z, v _x , v _y , σ), where (x, y, z) represents the target position, v _x , v _y are the radial velocity of the target in the x and y directions respectively, and σ represents the scattering cross section intensity of the target; Project the millimeter-wave radar points to the image plane with the camera internal and external parameters given in the dataset. 3. the target detection method of millimeter-wave radar and visual feature attention fusion according to claim 2, is characterized in that, the concrete steps of projecting millimeter-wave radar point cloud to image plane are: After coordinate conversion, the millimeter-wave radar point is converted from the self-coordinate system to the image coordinate system to generate a radar image with the same size as the visual image; The point cloud data is preliminarily expanded into a height range of 0.5-2.0 meters in the vertical direction and a width range of 0.2-1.5 meters in the horizontal direction, and the initial correlation is made between the radar information and the pixels of the camera; The specific height values of the radar point pixels on the radar image in the horizontal width and vertical height are adaptively adjusted through the scattering cross-section intensity σ of the radar point to obtain a variable-sized two-dimensional point cloud pixel area. 4. the target detection method of millimeter-wave radar and visual feature attention fusion according to claim 1, is characterized in that, in described step 2, the described radar point cloud data after preprocessing and described frame image are in The specific steps for data layer fusion are as follows: Firstly, using the depth information obtained from the millimeter-wave radar echo, the velocity components v _x and v _y on the X and Y axes are projected into the visual image channel as pixel values. At the pixel position without radar echo, all the corresponding radar channels are Set to zero; finally, convert the visual image with radar information into a three-channel image to realize the fusion of the millimeter-wave radar point cloud data and the frame image at the data layer. 5. the target detection method of millimeter-wave radar according to claim 4 and visual feature attention fusion, it is characterized in that, in described step 3, described frame image is carried out preliminary detection, the specific step of extracting image feature is : The fusion result of the millimeter-wave radar point cloud data and the frame image in the data layer is used as the input of the CneterNet center point detection network with DLA-43 as the backbone network, and the image is initially detected, and the target is obtained by using the detection head regression. The feature information of the image includes rough 3D bounding box, depth, viewing angle, 2D size, and velocity. 6. the target detection method of millimeter-wave radar and visual feature attention fusion according to claim 5, is characterized in that, in described step 4, carry out target association with described radar point cloud data and described frame image, extract The specific steps of the radar feature are: Use the image feature information of the target obtained by regression and the camera calibration matrix to generate a 3D viewing frustum, and associate radar detection with the target in the viewing frustum area; Expand the radar point cloud data into a fixed-size 3D cylinder to increase the correlation rate of the point cloud, and project the cylinder to the pixel coordinate system to associate with the 2D bounding box, and at the same time project the cylinder to the camera coordinate system and The constructed 3D viewing frustum performs depth value matching and ignores points outside the viewing frustum; Use the depth and velocity information of the radar point cloud data to build complementary features for the image and generate a 3-channel (d, v _x , v _y ) radar feature. 7. the target detection method of millimeter-wave radar and visual feature attention fusion according to claim 6, is characterized in that, in described step 5, described image feature and described radar feature input feature attention fusion module The specific steps are: Use one Conv1×1 and two Conv3×3 to extract the weights of the radar features obtained in step 4, and add the attention weight matrix obtained by the three branches by elements, and then go through the 7×7 convolutional layer After further extracting the feature weights, the spatial attention information of the radar features is generated; The generated image features are sequentially processed by Conv1×1, Conv3×3 and Conv1×1 to extract the image feature weights, and then multiplied with the original image features after being processed by the channel attention module element-wise to generate the channel attention information of the image features ; Using the spatial attention information of radar features and the channel attention information of image features to perform splicing and fusion operations to generate image radar feature tensors; Using the detection head to perform quadratic regression on the generated image radar feature tensor to obtain the depth, speed, rotation angle and attribute information of the target. 8. the target detection method of millimeter-wave radar and visual feature attention fusion according to claim 7, it is characterized in that, in described step 6, utilize 3Dbox decoder to decode the detection result of target and output the specific steps as: Combining the feature information of the image with multiple pieces of information obtained by regression after the feature attention fusion module, the 3D bounding box decoder restores the detection result of the 3D object, and outputs it to the visual image for display.