CN112731436B - Multi-mode data fusion travelable region detection method based on point cloud up-sampling - Google Patents

Abstract

The invention discloses a multi-modal data fusion drivable area detection method based on point cloud upsampling, which mainly includes two parts: spatial point cloud adaptive upsampling and multi-modal data fusion drivable area detection. The camera and lidar are registered through a joint calibration algorithm, the point cloud is projected onto the image plane to obtain a sparse point cloud image, the pixel local window is used to calculate edge intensity information, and the point cloud upsampling scheme is adaptively selected to obtain a dense point cloud image; the obtained Dense point cloud images and RGB images are used for feature extraction and cross-fusion to achieve rapid detection of drivable areas. The detection method of the present invention can realize rapid and accurate detection and segmentation of drivable areas.

Description

Multimodal data fusion drivable area detection method based on point cloud upsampling

Technical field

The invention relates to a multi-modal data fusion drivable area detection method based on point cloud upsampling, which mainly includes adaptive upsampling of spatial point clouds and multi-modal data fusion drivable area detection.

Background technique

Depending on the type of sensor selected, the current drivable area detection algorithms mainly include two solutions: camera-based and lidar-based. Among them, the camera has many advantages such as low cost, high frame rate and high resolution, but it is easily interfered by factors such as weather and has low robustness. On the other hand, lidar mainly obtains data from three-dimensional point clouds. Although it is insufficient in resolution and cost, it has high three-dimensional measurement accuracy and strong anti-interference ability, so it is widely used in unmanned systems. universally applied. Regarding the sparsity of point clouds, some existing methods use methods such as joint bilateral filtering upsampling to perform weighted estimation within a local window to obtain dense spatial information. However, most of them suffer from blurred edge recovery and poor detail retention. Not enough and other issues.

As the accuracy requirements for drivable area detection algorithms continue to increase, although using a single sensor to detect drivable areas can achieve more reliable detection in some scenarios, there are still certain limitations. In order to obtain better detection results, fusion methods based on images and point clouds have also begun to appear.

Zhang Y et al. proposed a traditional camera and lidar fusion in the literature ["Fusion of LiDAR and Camera by Scanning in LiDARImagery and Image-Guided Diffusion for Urban Road Detection,"[J].2018:579-584.] method. This method uses the idea of row and column scanning to determine the discrete point cloud of the drivable area based on the preliminary screening of point clouds, and uses the image as a guide to achieve pixel-level segmentation of the road area. The disadvantage of this method is that the detection process does not fully utilize the image information, and is not suitable for some poorly structured road scenes.

Contents of the invention

In order to overcome the above defects, the technical problem to be solved by the present invention is to provide a spatial point cloud upsampling method based on edge intensity adaptation to enhance the retention of edge and detail information.

Correspondingly, another technical problem to be solved by the present invention is to provide a drivable area detection framework that can fully integrate point cloud and image characteristics.

For smart car drivable area detection, the present invention solves the above technical problems and mainly includes the following steps: completing adaptive upsampling of sparse point clouds based on pixel edge intensity, and then using the synchronized RGB image and dense point cloud image as input to perform Feature extraction, fusion, and output detection results.

The present invention is specifically implemented by adopting the following technical solutions:

A multi-modal data fusion drivable area detection method based on point cloud upsampling. This method uses a joint calibration algorithm to calibrate the camera and lidar, project the point cloud to the image plane to obtain a sparse point cloud map, and use pixel local windows to calculate Based on edge intensity information, the point cloud upsampling scheme is adaptively selected to obtain a dense point cloud image; feature extraction and cross-fusion of the obtained dense point cloud image and RGB image are performed to achieve rapid detection of drivable areas.

In the above technical solution, further, based on the sparse point cloud image, the edge intensity information can be calculated using the pixel local window, and then the pixels are divided into two categories: non-edge areas and edge areas, and adaptive upsampling is completed accordingly. Use the pixel local window to calculate the edge intensity information, specifically: for each pixel, use the point cloud distance within the pixel local window to calculate the edge intensity information according to the following formula. When the edge intensity information is greater than the specified threshold τ, the pixel is considered to be on the edge. area, otherwise the pixel is considered to be in a non-edge area:

Among them, σ represents the standard deviation calculation, Represents the average distance of the point cloud within the window, and λ is a fixed parameter. The pixel local window refers to the neighborhood window centered on the pixel, and the edge intensity information is used to represent the possibility that the pixel is on the edge.

Furthermore, the adaptive selection point cloud upsampling scheme described is as follows: for non-edge area pixels, the calculation can be completed well using only the spatial Gaussian kernel in its neighborhood window; while for edge pixels, In other words, relying only on spatial position will tend to blur the edge recovery. Therefore, color information is introduced. First, the initial weight of each point is calculated based on the color and spatial position Gaussian kernel. On this basis, each point in the local window is calculated based on the average depth of the point cloud. Points are divided into two categories: foreground points and background points. The number and weight sum of the two types of points are counted, and the weight of each point is adjusted accordingly, and finally the spatial position information estimation of the pixel to be calculated is completed; among them, the foreground point is a point with less than the average depth information, and the background point is a point with greater than or equal to the average depth information.

As another improvement of the present invention, feature extraction and cross-fusion are performed on the obtained dense point cloud image and RGB image. Specifically, the synchronized dense point cloud image and RGB image are used as input, and feature extraction and intersection are performed through a multi-layer convolution network. Fusion, a multi-layer convolutional network combined with atrous convolution and pyramid pooling modules can quickly increase the receptive field and aggregate multi-scale contextual information. A loss function that focuses on detection results in difficult-to-detect areas and non-road areas is used to improve detection accuracy while ensuring vehicle driving safety.

The beneficial effects of the present invention are:

Compared with the traditional joint bilateral filtering upsampling algorithm, the adaptive point cloud upsampling method based on edge strength used in the present invention can more reliably restore the detailed information of the scene and improve the accuracy; at the same time, the RGB image used in the present invention and dense point cloud image fusion method can effectively fuse the characteristics of multi-modal data, combine the advantages of the two sensors, and achieve fast and accurate detection and segmentation of drivable areas. The multi-layer convolution network of the present invention can achieve rapid growth of the receptive field and multi-scale aggregation of information. At the same time, the present invention also adopts a loss function that focuses on difficult-to-detect areas and non-road areas, and can accurately and reliably output the detection of drivable areas. As a result, rapid detection and segmentation of road areas are achieved.

Description of the drawings

The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

Figure 1 is a flow chart of the multi-modal data fusion drivable area detection method based on point cloud upsampling;

Figure 2(a) is a sparse point cloud image, and (b) is the corresponding edge area representation of the scene;

Figure 3(a) is the joint bilateral filtering upsampling result, (b) is the upsampling result of the method of the present invention;

Figure 4 is a comparison of the drivable area detection results of the three networks, as well as the corresponding scene graph Image and result truth map Label. The three networks are: the detection network RGB that only inputs images, the detection network Lidar that only inputs dense point clouds, Multi-modal data fusion detection network Fusion of the present invention;

Figure 5 is a structural diagram of the multi-layer convolutional network of the present invention.

Detailed ways

As shown in Figure 1, the present invention provides a multi-modal data fusion drivable area detection method based on point cloud upsampling. The specific implementation is as follows:

1. Camera internal parameter calibration, camera and lidar external parameter joint calibration, as shown below.

1.1 Fixed the positions of the camera and lidar, and synchronously collected point cloud and image data based on the hard trigger mechanism;

1.2 Obtain camera internal parameter information based on single-object calibration, and at the same time obtain the plane equation of the calibration plate in the camera and lidar coordinate systems of each frame, which are recorded as a _c,i and a _l,i respectively, where i represents the number of frames and c represents the camera Coordinate system, l represents the lidar coordinate system. Use θ to represent the normal vector of the calibration plate plane, X to represent the space point on the plane, d to represent the distance from the origin of the coordinate system to the plane, and the plane constraint is

a _{c, i} : θ _{c, i} X+d _{c, i} = 0

a _{l, i} : θ _{l, i} X+d _{l, i} = 0

1.3 Construct the following optimization equation to solve for the rotation matrix R and the translation vector t, where L represents the number of points on the plane of each frame, and num is the total number of frames.

2. According to the joint calibration results, project the laser point cloud to the image plane to obtain the initial sparse point cloud image. For each pixel, use the point cloud distance within the local window to calculate the edge intensity information T according to the following formula. When the edge intensity is greater than the specified threshold τ (the threshold τ can be selected as needed, such as 1.1), the pixel is considered to be in the edge area. .

where σ represents the standard deviation calculation, Represents the average distance of the point cloud within the window, λ is a fixed parameter, and the value here is 3. (a) and (b) in Figure 2 are sparse point cloud images and corresponding edge image representations respectively.

3. Based on the above edge intensity information, divide each pixel in the image into two types: non-edge area and edge area, and complete the corresponding point cloud upsampling accordingly to achieve densification of the sparse point cloud and obtain a dense point cloud image.

3.1 If pixel q is in a non-edge area, directly use the spatial Gaussian kernel to calculate the weighted result in its neighborhood N(q) to avoid uneven point cloud reconstruction caused by excessive color differences.

3.2 If q is in the edge area, in order to avoid the edge recovery being too blurry, refer to the joint bilateral filtering upsampling method for processing. First, use the similarity of color and spatial position to assign an initial weight g(p) to each point, and s represents the summation calculation. The purpose It is to balance the differences in space and color. I represents the pixel value of the RGB image, specifically as

On this basis, considering the spatial distribution correlation of the point cloud in the local window, it is divided into two categories: foreground points and background points according to depth information, which are recorded as F and B respectively. Among them, the foreground points are points that are smaller than the average depth information. Background points are points that are greater than or equal to the average depth information. c represents the category F or B of the neighborhood point cloud. m and n represent the number and weight sum of the two categories. t _q represents the edge strength of the current pixel. Each point is calculated by category. The weight adjustment factor of

m _c =|c|,

According to the calculated weight, the spatial position information corresponding to the current pixel is calculated as follows:

In this step, represents the spatial position information of the pixel to be calculated, d _p represents the known spatial point in the neighborhood, K represents the normalization factor, σ _r and σ _I represent the standard deviation of the spatial domain and color domain respectively.

4. Use RGB and the dense point cloud image obtained in step 3 as two three-channel data inputs at the same time to construct a multi-modal data fusion drivable area detection network (i.e., a multi-layer convolutional network). As shown in Figure 5, the multi-layer convolutional network uses a dual encoder (the dual encoder structure is the same, but does not share parameters) and a single decoder structure, using RGB images and dense point cloud images as original inputs respectively. The feature map is cross-fused through 1×1 convolution, and the result is used as the input of the next layer of convolutional network; the output result of the encoder is used as the input of the pyramid pooling module to obtain the final feature map output; the output result of the pyramid pooling module is decoded The device restores the resolution and uses the Sigmoid function to calculate the probability that each pixel belongs to the drivable area. When the probability is greater than the set threshold, it is judged that the pixel belongs to the drivable area. The multi-layer convolutional network of the present invention combines atrous convolution and pyramid pooling modules, which can quickly increase the receptive field and aggregate multi-scale contextual information.

In the process of supervised learning, the present invention designs the loss function as follows, focusing on the detection results of difficult-to-detect areas and non-road areas to improve detection accuracy while ensuring the safety of vehicle driving.

Among them, y=1 and y=0 represent positive and negative samples respectively. Positive samples are road areas, and negative samples are non-road areas. Difficult detection areas refer to areas where detection is more difficult. For positive samples, the detection results tend to be Non-road; for negative samples, the detection results tend to be roads. y′ represents the detection probability, α and γ are fixed constants, both of which take the value 2 here.

The resolution of the feature map is restored through the decoder, and the Sigmoid layer is used to calculate the probability that each pixel belongs to the road. When the probability is greater than the set threshold (such as 0.5), the pixel is judged to belong to the drivable area.

Example 1

This embodiment mainly compares the performance indicators of the joint bilateral filtering upsampling algorithm JBU and the adaptive upsampling method based on edge intensity information in the present invention. In this embodiment, a sparse point cloud image is obtained by downsampling the depth truth map by 5 times, and the upsampling effects of the two methods are compared. (a) and (b) of Figure 3 respectively represent the JBU upsampling results and the upsampling results of the method of the present invention. It can be found that the method of the present invention can better prevent edge blur while reducing reconstruction errors.

Example 2

This example mainly uses the KITTI data set to compare the drivable area detection performance of a single image data network, a single point cloud data network, and the multi-modal data fusion network in the present invention. The three network detection results are shown in Figure 4, which can be viewed intuitively It can be seen that the multi-modal data fusion drivable area detection method in the present invention can further improve the accuracy of road detection, avoid false detection of vehicles to a large extent, and at the same time improve the reliability of boundary detection.

The above-described specific embodiments describe in detail the technical solutions and beneficial effects of the present invention. It should be understood that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, equivalent substitutions, etc. made within shall be included in the protection scope of the present invention.

Claims (3)

1. A multi-modal data fusion drivable area detection method based on point cloud upsampling, which is characterized in that the method uses a joint calibration algorithm to calibrate the camera and lidar, and projects the point cloud to the image plane to obtain a sparse point cloud map , use the pixel local window to calculate edge intensity information, adaptively select the point cloud upsampling scheme, and obtain a dense point cloud image; perform feature extraction and cross-fusion of the obtained dense point cloud image and RGB image to achieve rapid detection of drivable areas; The described calculation of edge intensity information using a pixel local window is specifically as follows: for each pixel, the point cloud distance within the pixel local window is used to calculate the edge intensity information according to the following formula. When the edge intensity information is greater than the specified threshold τ, it is considered to be the edge intensity information. The pixel is in the edge area, otherwise the pixel is considered to be in the non-edge area: Among them, σ represents the standard deviation calculation, Represents the average distance of the point cloud within the window, λ is a fixed parameter; The described adaptive selection point cloud upsampling scheme is specifically: for non-edge area pixels, the spatial Gaussian kernel is directly used to calculate the weighted result within its local window; for edge area pixels, the spatial and color Gaussian kernels are first jointly calculated separately. The weight of each point in the local window; secondly, the point cloud is divided into two categories: foreground points and background points according to the average depth of the point cloud in the window, the number and weight sum of the two types of points in the local window are counted, and the weight of each point is calculated accordingly. The weights are adjusted; finally, the updated weights are used to weight each point within the local window to complete the calculation of the spatial position information of the pixels to be calculated. 2. The multi-modal data fusion drivable area detection method based on point cloud upsampling according to claim 1, characterized in that feature extraction and cross-fusion are performed on the obtained dense point cloud image and RGB image, specifically: RGB Images and dense point cloud images are used as input at the same time, and multi-layer convolutional networks are used for feature extraction and cross-fusion. The loss function focuses on the detection results of difficult-to-detect areas and non-road areas, and outputs the detection probability of drivable areas; The loss function is as follows: Among them, y=1 and y=0 represent positive and negative samples respectively. Positive samples are road areas, and negative samples are non-road areas. Difficult detection areas refer to areas where detection is more difficult. For positive samples, the detection results tend to be Non-road; for negative samples, the detection results tend to be roads; y′ represents the probability of being judged to be a road area, and α and γ are fixed constants. 3. The multi-modal data fusion drivable area detection method based on point cloud upsampling according to claim 2, characterized in that the multi-layer convolution network structure adopts a dual encoder and a single decoder structure. The two encoders have the same structure, but do not share parameters. They use RGB images and dense point cloud images as original inputs respectively. For the two feature maps output by the same layer encoder, 1×1 convolution is used for cross-fusion, and the fusion result is used as the next The input of layer convolution, and the downsampled feature map obtained by the dual encoder is input into the pyramid pooling module to obtain the final feature map output; The output result of the pyramid pooling module is restored to resolution through the decoder, and the Sigmoid function is used to calculate the probability that each pixel belongs to the drivable area. When the probability is greater than the set threshold, the pixel is judged to belong to the drivable area.