CN117934831A - A 3D semantic segmentation method based on camera and laser fusion - Google Patents

Abstract

The invention discloses a three-dimensional semantic segmentation method based on camera and laser fusion, which comprises the following steps: inputting camera image and laser point cloud data into a camera module and a laser module respectively to extract image and laser point cloud data characteristics to obtain a camera image characteristic image and a laser point cloud data characteristic image, inputting the camera image characteristic image and the laser point cloud data characteristic image into a fusion module to perform characteristic fusion to obtain fused camera image characteristics and laser point cloud data characteristics, inputting the camera image characteristic image and the laser point cloud data characteristic image into the camera module and the laser module respectively to obtain a camera image characteristic image and a laser point cloud data characteristic image, inputting the camera image characteristic image and the laser point cloud data characteristic image into a monitoring module to calculate a loss function, updating parameter weights of a three-dimensional semantic segmentation network, and obtaining a trained three-dimensional semantic segmentation network; acquiring camera image and laser point cloud data, and inputting a trained three-dimensional semantic segmentation network to obtain semantic segmentation results of the laser point cloud data and the camera image; the method effectively combines the texture information of the image and the distance information of the laser, and improves the accuracy of semantic segmentation.

Description

A 3D semantic segmentation method based on camera and laser fusion

Technical Field

The present invention relates to the technical field of autonomous driving and semantic segmentation, and in particular to a three-dimensional semantic segmentation method based on camera and laser fusion.

Background technique

In the field of autonomous driving, semantic segmentation is very important for scene understanding. The semantic segmentation task is to assign a corresponding semantic label to each camera pixel and laser point cloud input. There are currently two main methods: camera-based and lidar-based methods.

Camera images contain color data from three channels, so they have richer appearance information, such as color and texture. However, as a passive sensor, the camera is easily affected by lighting conditions and weather. In addition, since the camera is a 2D sensor that lacks depth information, it is usually difficult to obtain accurate distance information of the surrounding environment. LiDAR is an active sensor that calculates the exact distance by emitting lasers to the outside and receiving reflected lasers. Its performance is almost unaffected under different lighting conditions. However, due to the sparse point cloud, irregular distribution, and lack of texture, the segmentation effect is poor in scenes with small objects, long distances, and similar structures.

At present, the camera image and laser point cloud fusion solution combines the advantages of the camera-based and laser radar-based methods, and achieves the purpose of 3D semantic segmentation by considering the texture of the image and the distance of the laser. However, this method has the problem that the laser radar segmentation lacks texture features and the image segmentation lacks distance.

Summary of the invention

In view of the above-mentioned deficiencies in the prior art, the present invention provides a three-dimensional semantic segmentation method based on camera and laser fusion.

In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

A three-dimensional semantic segmentation method based on camera and laser fusion includes the following steps:

S1, input the camera image into the camera module of the 3D semantic segmentation network to extract image features and obtain a camera image feature map of the original size;

S2, inputting the laser point cloud data into the laser module of the 3D semantic segmentation network to extract the features of the laser point cloud data, and obtaining a laser point cloud data feature map of the original size;

S3, inputting the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into a fusion module of a three-dimensional semantic segmentation network for feature fusion, to obtain fused camera image features and laser point cloud data features;

S4, inputting the fused image features and laser point cloud data features obtained in step S3 into the camera module and the laser module respectively to obtain a camera image feature map and a laser point cloud data feature map;

S5, inputting the camera image feature map and the laser point cloud data feature map obtained in step S4 into the supervision module of the 3D semantic segmentation network, and calculating the loss function in a self-supervised mode or a supervised mode;

S6. According to the loss function calculated in step S5, the gradients of the camera module, laser module, fusion module and supervision module of the 3D semantic segmentation network are calculated, and the parameter weights of the 3D semantic segmentation network are updated by using the gradient descent method to obtain a trained 3D semantic segmentation network.

S7, obtaining camera images and laser point cloud data, inputting the three-dimensional semantic segmentation network trained in step S6, and obtaining semantic segmentation results of the laser point cloud data and the camera images.

Furthermore, the camera module and the laser module are composed of an encoder and a decoder, wherein the size of the feature map in the encoder decreases layer by layer, the size of the feature map in the decoder increases layer by layer, and a jump connection structure is added between the encoder layer and the decoder layer with the same image size.

Furthermore, step S1 specifically includes:

S11, obtaining a camera image captured by a camera, inputting the camera image into an encoder, and using a convolutional neural network to extract local features of the camera image to obtain a camera image feature map;

S12, according to the camera image feature map obtained in step S11, using a pooling layer to reduce the size of the camera image feature map layer by layer;

S13. Input the reduced-size camera image feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the camera image feature map layer by layer to obtain the camera image feature map of the original size.

Furthermore, step S2 specifically includes:

S21, acquiring laser point cloud data collected by the laser radar, and projecting the laser point cloud data onto a camera plane to obtain two-dimensional laser point cloud data;

S22, inputting the two-dimensional laser point cloud data obtained in step S21 into an encoder, and using a convolutional neural network to extract local features of the two-dimensional laser point cloud data to obtain a laser point cloud data feature map;

S23, according to the laser point cloud data feature map obtained in step S22, using a pooling layer to reduce the size of the laser point cloud data feature map layer by layer;

S24. Input the reduced-size laser point cloud data feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the laser point cloud data feature map layer by layer to obtain the laser point cloud data feature map of the original size.

Furthermore, the calculation formula for projecting the laser point cloud data onto the camera plane is:

[x′ _i , y′ _i , z′ _i ] ^T = K × T _r × [x _i , y _i , z _i , 1] ^T

M _l [u _i ][v _i ]＝1

Among them, x′ _i , y′ _i , z′ _i respectively represent the position of the i-th laser point cloud data in the camera coordinate system, T represents transpose, K represents the intrinsic parameter of the camera, _Tr represents the transfer matrix from laser to camera, x _i , y _i , _zi represent the position of the i-th laser point cloud data on the x, y and z axes, _ui , _vi represent the index of the i-th laser point cloud in the vertical and horizontal directions of the camera plane, respectively, and M _l represents the lidar mask.

Furthermore, the fusion module is composed of a splicing module, a convolutional layer, and a sliding window attention module, wherein the sliding window attention module is composed of a first sliding window attention layer and a second sliding window attention layer, the first sliding window attention layer is composed of a layer normalization module, a W-MSA module, and a multilayer perceptron module, and the second sliding window attention layer is composed of a layer normalization module, a SW-MSA module, and a multilayer perceptron module.

Furthermore, step S3 specifically includes:

S31, inputting the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into a splicing module of a fusion module to obtain splicing features of the camera and the laser radar;

S32, inputting the splicing features of the camera and the laser radar obtained in step S31 into the convolution layer to obtain the fusion features of the camera and the laser radar;

S33, inputting the fusion features of the camera and the laser radar obtained in step S32 into the sliding window attention module to obtain the fusion attention features of the camera and the laser radar;

S34. Integrate the fused attention features of the camera and the laser radar obtained in step S33 and the fused features of the camera and the laser radar obtained in step S32 into the image feature map in step S1 and the laser point cloud data feature map in step S2 in proportion to obtain fused camera image features and laser point cloud data features.

Furthermore, the calculation formula of the fused image features and laser point cloud data features in step S34 is:

C _fusion = C _origin + a ₁ × SelfAttension × FusionFeature

L _fusion = L _origin + a ₂ × SelfAttension × FusionFeature

Among them, C _fusion represents the fused camera image features, C _origin represents the camera image features of the original size, a ₁ and a ₂ represent the fusion scale factors respectively, SelfAttension represents the fusion attention features of the camera and lidar, FusionFeature represents the fusion features of the camera and lidar, L _fusion represents the fused laser point cloud data features, and L _origin represents the laser point cloud data features of the original size.

Furthermore, the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of calculating the loss function using the self-supervision mode is as follows:

The supervision module generates pseudo labels by adding the confidence PIDNet network, while retaining high-confidence pixels and laser point cloud data. By setting the camera mask and lidar mask, the loss function of the self-supervised mode is obtained, namely:

L _{self-supervised} = L _foc1 + L _lov1 + L _fov2 + L _lov2 + L _kl

Among them, L _{self-supervised} represents the loss function of the self-supervised mode, L _kl represents the one-way KL divergence, L _foc1 , L _lov1 represent the focal loss and Lovalz loss between the prediction result of the camera branch and the pseudo label, respectively, L _fov2 , L _lov2 represent the focal loss and Lovalz loss between the prediction result of the lidar branch and the pseudo label, u, v represent the length and width of the prediction result feature map, respectively, C represents the confidence, focalloss(·) represents the focal loss function calculation formula, Pred _camera represents the prediction result of the camera branch, Pred _Lidar represents the prediction result of the lidar branch, label represents the pseudo label value, M _θ1 represents the camera mask, M _θ2 represents the lidar mask, and M _l represents the lidar mask.

Furthermore, the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of calculating the loss function in the supervised mode is as follows:

The supervision module uses focus loss and Lovalz loss to adjust the parameter weights to obtain the loss function of the supervised mode, namely:

L _supervised =L _foc1 +L _lov1 +L _fov2 +L _lov2

Among them, L _supervised represents the loss function of the supervised mode, L _foc1 and L _lov1 represent the focus loss and Lovalz loss between the prediction results of the camera branch and the true value label, respectively, and L _fov2 and L _lov2 represent the focus loss and Lovalz loss between the prediction results of the lidar branch and the true value label, respectively.

The present invention has the following beneficial effects:

1. The three-dimensional semantic segmentation method based on camera and laser fusion proposed in the present invention improves the accuracy of semantic segmentation by effectively combining the texture information of camera images and the distance information of laser radar. At the same time, in view of the sparse laser point cloud of small objects, the prediction result is more accurate by introducing camera image information;

2. The sliding window attention mechanism is used in the fusion module, which makes the 3D semantic segmentation network more robust under drastic changes in lighting and color;

3. The PIDNet network with confidence is used to generate pseudo labels. No manual annotation of point cloud labels is required. The 3D semantic segmentation network can still be trained in a cross-data and cross-modal manner, which improves the prediction accuracy of the 3D semantic segmentation network.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a schematic diagram of a flow chart of a 3D semantic segmentation method based on camera and laser fusion proposed in the present invention;

Figure 2 is a schematic diagram of the three-dimensional semantic segmentation network structure;

Fig. 3 is a schematic diagram of the fusion module structure;

FIG4 is a schematic diagram of the structure of the PIDNet network with confidence added to the supervision module;

Figure 5 is a schematic diagram of the segmentation results of the three-dimensional semantic segmentation network.

Detailed ways

The specific implementation modes of the present invention are described below to facilitate those skilled in the art to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

As shown in FIG1 , a 3D semantic segmentation method based on camera and laser fusion includes the following steps S1-S7:

As shown in FIG. 2 , FIG. 2 is a schematic diagram of the structure of a three-dimensional semantic segmentation network; the three-dimensional semantic segmentation network in FIG. 2 includes a camera module, a laser module, a fusion module, and a supervision module. The three-dimensional semantic segmentation network as a whole adopts a dual-stream architecture, that is, two input ends, the camera image is input into the camera module, and the laser point cloud data of the laser radar is input into the laser module. Among them, the camera module and the laser module have similar structures, each consisting of an encoder whose feature map size decreases layer by layer and a decoder whose feature map size increases layer by layer. In the encoder stage, a convolutional neural network is used to extract local features of the camera image or laser point cloud data, and the size of the feature map is reduced by a pooling layer; in the decoder stage, a convolutional neural network and a bilinear upsampling method are used to restore the size of the feature map layer by layer until it is restored to the size of the original input camera image or laser point cloud data image. At the same time, this embodiment adds a jump connection structure between the encoder and the decoder of the same size feature map of the camera module or laser module, as shown in FIG. 2 in the camera module or laser module, and the feature is directly propagated from the encoder to the decoder through the path shown by the dotted line.

S1. Input the camera image into the camera module of the 3D semantic segmentation network to extract image features and obtain a camera image feature map of the original size.

S2. Input the laser point cloud data into the laser module of the 3D semantic segmentation network to extract the features of the laser point cloud data and obtain a feature map of the laser point cloud data of the original size.

Specifically, the camera module and the laser module are composed of an encoder and a decoder, wherein the size of the feature map in the encoder decreases layer by layer, the size of the feature map in the decoder increases layer by layer, and a jump connection structure is added between the encoder layer and the decoder layer with the same image size.

In this embodiment, the convolutional perception field of view is expanded layer by layer, the segmentation performance of objects of different sizes is improved, and the edge position information of the original image features is retained by using the jump connection structure, making the semantic segmentation more accurate. The number of layers of the encoder and decoder can be freely set and fine-tuned according to the input image size, usually 4 layers.

Specifically, step S1 includes S11-S13:

S11. Obtain a camera image captured by a camera, input the camera image into an encoder, and use a convolutional neural network to extract local features of the camera image to obtain a camera image feature map.

S12. According to the camera image feature map obtained in step S11, a size of the camera image feature map is reduced layer by layer using a pooling layer.

S13. Input the reduced-size camera image feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the camera image feature map layer by layer to obtain the camera image feature map of the original size.

Specifically, step S2 specifically includes S21-S24:

S21. Obtain laser point cloud data collected by the laser radar, and project the laser point cloud data onto the camera plane to obtain two-dimensional laser point cloud data.

Specifically, the calculation formula for projecting the laser point cloud data onto the camera plane is:

[x′ _i , y′ _i , z′ _i ] ^T = K × T _r × [x _i , y _i , z _i , 1] ^T

M _l [u _i ][v _i ]＝1

Among them, x′ _i , y′ _i , z′ _i respectively represent the position of the i-th laser point cloud data in the camera coordinate system, T represents transpose, K represents the intrinsic parameter of the camera, _Tr represents the transfer matrix from laser to camera, x _i , y _i , _zi represent the position of the i-th laser point cloud data on the x, y and z axes, _ui , _vi represent the index of the i-th laser point cloud in the vertical and horizontal directions of the camera plane, respectively, and M _l represents the lidar mask.

In this embodiment, since the camera data is two-dimensional data and the laser point cloud data is three-dimensional data, there is a problem of spatial inconsistency. Usually, the viewing angle range of a mechanical rotating laser radar is larger than the viewing angle range of a camera. Therefore, in this embodiment, the laser point cloud data is projected onto the camera plane and converted into two-dimensional laser point cloud data. Wherein, the formula [x′ _i , y′ _i , z′ _i ] ^T ＝K× _Tr ×[x _i , y _i , z _i , 1] ^T is the formula for projecting the laser point cloud data onto the camera image. The laser point cloud is P _i ＝{x _i , y _i , z _i } ^T , x _i , y _i , z _i respectively represent the coordinates of the laser point cloud in three-dimensional space; _Tr represents the transfer matrix from the laser to the camera, that is, it describes the physical distance between the two sensors of the laser radar and the camera, K represents the internal parameters of the camera, that is, the relationship between the three-dimensional space mapping to the two-dimensional photosensitive plane, [x′ _i , y′ _i , z′ _i ] ^T represents the three-dimensional point in the camera coordinate system; the formula By scaling z′ _i, we get [u _i , _vi , 1] ^T , which is the two-dimensional coordinates of the laser point cloud on the camera plane. Since the laser point cloud data is usually sparse, not every image pixel has a corresponding laser point cloud. Therefore, the formula M _l [u _i ][ _vi ] = 1 is used to indicate the position where the laser point cloud is mapped.

S22, inputting the two-dimensional laser point cloud data obtained in step S21 into an encoder, using a convolutional neural network to extract local features of the two-dimensional laser point cloud data, and obtaining a laser point cloud data feature map.

S23. According to the laser point cloud data feature map obtained in step S22, a size of the laser point cloud data feature map is reduced layer by layer using a pooling layer.

S24. Input the reduced-size laser point cloud data feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the laser point cloud data feature map layer by layer to obtain the laser point cloud data feature map of the original size.

S3, inputting the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into the fusion module of the three-dimensional semantic segmentation network for feature fusion, so as to obtain fused camera image features and laser point cloud data features.

As shown in FIG3 , FIG3 is a schematic diagram of the fusion module structure; in FIG3 (a), the fusion module includes a splicing module (C), a convolution layer (Conv), and a sliding window attention module (Swin-transformer). In this embodiment, the original size image feature (C _origin ) in the camera image feature map and the original size laser point cloud data feature (L _origin ) in the laser point cloud data feature map are input into the splicing module to obtain a splicing feature (Concat Feature), and then the splicing feature is input into the convolution layer to obtain a fusion feature (Fusion Feature), and then the fusion feature is input into the sliding window attention module to obtain a fusion attention feature (Self-Attention Feature), and finally the fusion attention feature is fused according to the ratio with the original size image feature and the original size laser point cloud data feature to obtain a fused camera image feature (C _fusion ) and a fused laser point cloud data feature (L _fusion ). In Figure 3(b), the first sliding window attention layer and the second sliding window attention layer are set in the sliding window attention module. The first sliding window attention layer is composed of a layer normalization module (LN), a W-MSA module (Windows Multi-Head Self-Attention) and a multi-layer perceptron module MLP), and the second sliding window attention layer is composed of a layer normalization module (LN), a SW-MSA module (Shifted Windows Multi-Head Self-Attention) and a multi-layer perceptron module (MLP); first, the fusion feature is flattened to obtain a patch feature (PatchFeature), and then passes through the first sliding window attention layer and the second sliding window attention layer. The difference between the first sliding window attention layer and the second sliding window attention layer is that the first sliding window attention layer uses the W-MSA structure. The purpose of the W-MSA structure is to reduce the amount of calculation and only calculate self-attention in each window. The second sliding window attention layer uses the SW-MSA structure. The purpose of the SW-MSA structure is to provide information exchange between windows by shifting windows.

Specifically, the fusion module is composed of a splicing module, a convolutional layer, and a sliding window attention module, wherein the sliding window attention module is composed of a first sliding window attention layer and a second sliding window attention layer, the first sliding window attention layer is composed of a layer normalization module, a W-MSA module, and a multilayer perceptron module, and the second sliding window attention layer is composed of a layer normalization module, a SW-MSA module, and a multilayer perceptron module.

Specifically, step S3 specifically includes S31-S34:

S31, input the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into the splicing module of the fusion module to obtain the splicing features of the camera and the laser radar.

S32: Input the concatenated features of the camera and the lidar obtained in step S31 into the convolution layer to obtain the fusion features of the camera and the lidar.

S33. Input the fusion features of the camera and the laser radar obtained in step S32 into the sliding window attention module to obtain the fusion attention features of the camera and the laser radar.

S34. Integrate the fused attention features of the camera and the laser radar obtained in step S33 and the fused features of the camera and the laser radar obtained in step S32 into the image feature map in step S1 and the laser point cloud data feature map in step S2 in proportion to obtain fused camera image features and laser point cloud data features.

Specifically, the calculation formula of the fused image features and laser point cloud data features in step S34 is:

C _fusion = C _origin + a ₁ × SelfAttension × FusionFeature

L _fusion = L _or i _g in + a ₂ × SelfAttension × FusionFeature

Among them, C _fusion represents the fused camera image features, C _origin represents the camera image features of the original size, a ₁ and a ₂ represent the fusion scale factors respectively, SelfAttension represents the fusion attention features of the camera and lidar, FusionFeature represents the fusion features of the camera and lidar, L _fusion represents the fused laser point cloud data features, and L _origin represents the laser point cloud data features of the original size.

S4. Input the fused image features and laser point cloud data features obtained in step S3 into the camera module and the laser module respectively to obtain a camera image feature map and a laser point cloud data feature map.

In this embodiment, the image features and laser point cloud data features fused in the fusion module are input into the camera module and the laser module respectively, and feature extraction is continued respectively. The purpose of setting the fusion module is to effectively interact with data. The fusion module in this embodiment adopts a sliding window attention structure. Compared with the convolution method, it can not only better select features through the global attention mechanism, but also weight the fused image features and laser point cloud data features to the original features of the image and laser through the attention mechanism, so as to perform the next feature extraction.

S5. Input the camera image feature map and the laser point cloud data feature map obtained in step S4 into the supervision module of the three-dimensional semantic segmentation network, and calculate the loss function in a self-supervised mode or a supervised mode.

In this embodiment, two modes are adopted in the supervision module, one is a supervised mode and the other is a self-supervised mode; the supervised mode adopts the real label method, that is, the real label of the laser point cloud in the original data set is used for training to supervise the camera's prediction value and the lidar's prediction value; the self-supervised mode uses a pseudo-label method, that is, there is no real label of the laser point cloud, and the image is inferred by a 2D image segmentation network pre-trained in other training sets, while retaining the pseudo-label results and confidence, and using the pseudo-label and confidence method to jointly supervise the network convergence, and using the pseudo-label to supervise the camera's prediction value and the lidar's prediction value.

Specifically, the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of calculating the loss function using the self-supervision mode is as follows:

The supervision module generates pseudo labels by adding the confidence PIDNet network, while retaining high-confidence pixels and laser point cloud data. By setting the camera mask and lidar mask, the loss function of the self-supervised mode is obtained, namely:

L _{self-supervised} = L _foc1 + L _lov1 + L _fov2 + L _lov2 + L _kl

Among them, L _{self-supervised} represents the loss function of the self-supervised mode, L _kl represents the one-way KL divergence, L _foc1 , L _lov1 represent the focal loss and Lovalz loss between the prediction result of the camera branch and the pseudo label, respectively, L _fov2 , L _lov2 represent the focal loss and Lovalz loss between the prediction result of the lidar branch and the pseudo label, u, v represent the length and width of the prediction result feature map, respectively, C represents the confidence, focalloss(·) represents the focal loss function calculation formula, Pred _camera represents the prediction result of the camera branch, Pred _Lidar represents the prediction result of the lidar branch, label represents the pseudo label value, M _θ1 represents the camera mask, M _θ2 represents the lidar mask, and M _l represents the lidar mask.

In this embodiment, the camera mask or the lidar mask is activated only when the confidence is greater than θ1/θ2.

As shown in Figure 4, Figure 4 is a schematic diagram of the structure of the PIDNet network with confidence added to the supervision module; in this embodiment, the self-supervised mode is used to generate pseudo labels to train the network, and the pseudo labels are used to predict the camera's prediction value and the lidar's prediction value. Figure 4 is a PIDNet network with confidence added. The network can be used to calculate confidence and pseudo labels. The network is only used to generate pseudo labels and does not participate in the training process of the entire three-dimensional semantic segmentation network. Among them, the confidence C is defined according to the entropy E, and the entropy E is calculated by the probability p. If the output of the three-dimensional semantic segmentation network is more concentrated in a certain category, the smaller the entropy E, and the confidence C is closer to 1. The calculation formula for confidence is: n represents the number of categories for semantic segmentation.

Specifically, the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of calculating the loss function in the supervised mode is as follows:

The supervision module uses focus loss and Lovalz loss to adjust the parameter weights to obtain the loss function of the supervised mode, namely:

L _supervised =L _foc1 +L _lov1 +L _fov2 +L _lov2

Among them, L _supervised represents the loss function of the supervised mode, L _foc1 and L _lov1 represent the focus loss and Lovalz loss between the prediction results of the camera branch and the true value label, respectively, and L _fov2 and L _lov2 represent the focus loss and Lovalz loss between the prediction results of the lidar branch and the true value label, respectively.

S6. According to the loss function calculated in step S5, the gradients of the camera module, laser module, fusion module and supervision module of the three-dimensional semantic segmentation network are calculated, and the parameter weights of the three-dimensional semantic segmentation network are updated by the gradient descent method to obtain a trained three-dimensional semantic segmentation network.

In this embodiment, the calculated loss function is used to perform back propagation layer by layer by calculating the gradients of the camera module, laser module, fusion module and supervision module, and the parameter weights of the three-dimensional semantic segmentation network are updated, so that the three-dimensional semantic segmentation network is finally converged to obtain a trained three-dimensional semantic segmentation network.

S7, obtaining camera images and laser point cloud data, inputting the three-dimensional semantic segmentation network trained in step S6, and obtaining semantic segmentation results of the laser point cloud data and the camera images.

In this embodiment, the three-dimensional semantic segmentation method proposed in the present invention is compared with the pure laser radar segmentation method. It is found that the performance of the three-dimensional semantic segmentation method proposed in the present invention is improved by 2.6% compared with the pure laser radar segmentation method. At the same time, the introduction of images has more advantages in the segmentation of small objects. The specific segmentation results are shown in Table 1:

Table 1 Results on the SemanticKITTI dataset

^* Implemented by us, forward-looking lidar data comes from here, ⁺ other results come from benchmarks ^* , bold is the highest result, underline is the second highest result

As shown in FIG5 , FIG5 is a schematic diagram of the segmentation result of the 3D semantic segmentation network; FIG5 is a 3D semantic segmentation result of the camera image and the laser point cloud data obtained by segmenting the camera image and the laser point cloud data using the 3D semantic segmentation method based on camera and laser fusion proposed by the present invention. As can be seen from FIG5 , due to the shadow of the trees on the left side of the road, the ground becomes difficult to distinguish, but the 3D semantic segmentation method proposed by the present invention can still accurately identify the road boundary, and at the same time, although the reflection point cloud of the cyclist in the distance is sparse, it can still be accurately predicted by the 3D semantic segmentation network.

The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims (10)

1. A three-dimensional semantic segmentation method based on camera and laser fusion, characterized in that it includes the following steps: S1, input the camera image into the camera module of the 3D semantic segmentation network to extract image features and obtain a camera image feature map of the original size; S2, inputting the laser point cloud data into the laser module of the 3D semantic segmentation network to extract the features of the laser point cloud data, and obtaining a laser point cloud data feature map of the original size; S3, inputting the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into a fusion module of a three-dimensional semantic segmentation network for feature fusion, to obtain fused camera image features and laser point cloud data features; S4, inputting the fused image features and laser point cloud data features obtained in step S3 into the camera module and the laser module respectively to obtain a camera image feature map and a laser point cloud data feature map; S5, inputting the camera image feature map and the laser point cloud data feature map obtained in step S4 into the supervision module of the 3D semantic segmentation network, and calculating the loss function in a self-supervised mode or a supervised mode; S6. According to the loss function calculated in step S5, the gradients of the camera module, laser module, fusion module and supervision module of the 3D semantic segmentation network are calculated, and the parameter weights of the 3D semantic segmentation network are updated by using the gradient descent method to obtain a trained 3D semantic segmentation network. S7, obtaining camera images and laser point cloud data, inputting the three-dimensional semantic segmentation network trained in step S6, and obtaining semantic segmentation results of the laser point cloud data and the camera images. 2. According to claim 1, a three-dimensional semantic segmentation method based on camera and laser fusion is characterized in that the camera module and the laser module are composed of an encoder and a decoder, wherein the size of the feature map in the encoder decreases layer by layer, the size of the feature map in the decoder increases layer by layer, and a jump connection structure is added between the encoder layer and the decoder layer with the same image size. 3. According to the three-dimensional semantic segmentation method based on camera and laser fusion according to claim 2, it is characterized in that step S1 specifically comprises: S11, obtaining a camera image captured by a camera, inputting the camera image into an encoder, and using a convolutional neural network to extract local features of the camera image to obtain a camera image feature map; S12, according to the camera image feature map obtained in step S11, using a pooling layer to reduce the size of the camera image feature map layer by layer; S13. Input the reduced-size camera image feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the camera image feature map layer by layer to obtain the camera image feature map of the original size. 4. According to the method of claim 2, the three-dimensional semantic segmentation method based on camera and laser fusion is characterized in that step S2 specifically comprises: S21, acquiring laser point cloud data collected by the laser radar, and projecting the laser point cloud data onto a camera plane to obtain two-dimensional laser point cloud data; S22, inputting the two-dimensional laser point cloud data obtained in step S21 into an encoder, and using a convolutional neural network to extract local features of the two-dimensional laser point cloud data to obtain a laser point cloud data feature map; S23, according to the laser point cloud data feature map obtained in step S22, using a pooling layer to reduce the size of the laser point cloud data feature map layer by layer; S24. Input the reduced-size laser point cloud data feature map into the decoder, and use a convolutional neural network and a bilinear upsampling method to restore the size of the laser point cloud data feature map layer by layer to obtain the laser point cloud data feature map of the original size. 5. According to the three-dimensional semantic segmentation method based on camera and laser fusion according to claim 4, it is characterized in that the calculation formula for projecting the laser point cloud data on the camera plane is: [ ^xi′ _, yi _′ ,zi _′ ] ^{T =} K×T _r ×[ _{xi,yi ,} ^zi _, 1] ^T M _l [u _i ][v _i ]＝1 Among them, x _i ^′ , y _i ^′ , z _i ^′ respectively represent the position of the i-th laser point cloud data in the camera coordinate system, T represents transpose, K represents the intrinsic parameter of the camera, _Tr represents the transfer matrix from laser to camera, x _i , y _i , z _i represent the position of the i-th laser point cloud data on the x, y and z axes, _ui , _vi represent the index of the i-th laser point cloud in the vertical and horizontal directions of the camera plane, respectively, and M _l represents the lidar mask. 6. According to a three-dimensional semantic segmentation method based on camera and laser fusion according to claim 1, it is characterized in that the fusion module is composed of a splicing module, a convolutional layer, and a sliding window attention module, wherein the sliding window attention module is composed of a first sliding window attention layer and a second sliding window attention layer, the first sliding window attention layer is composed of a layer normalization module, a W-MSA module and a multi-layer perceptron module, and the second sliding window attention layer is composed of a layer normalization module, a SW-MSA module and a multi-layer perceptron module. 7. The three-dimensional semantic segmentation method based on camera and laser fusion according to claim 6, characterized in that step S3 specifically comprises: S31, inputting the camera image feature map in step S1 and the laser point cloud data feature map in step S2 into a splicing module of a fusion module to obtain splicing features of the camera and the laser radar; S32, inputting the splicing features of the camera and the laser radar obtained in step S31 into the convolution layer to obtain the fusion features of the camera and the laser radar; S33, inputting the fusion features of the camera and the laser radar obtained in step S32 into the sliding window attention module to obtain the fusion attention features of the camera and the laser radar; S34. Integrate the fused attention features of the camera and the laser radar obtained in step S33 and the fused features of the camera and the laser radar obtained in step S32 into the image feature map in step S1 and the laser point cloud data feature map in step S2 in proportion to obtain fused camera image features and laser point cloud data features. 8. The three-dimensional semantic segmentation method based on camera and laser fusion according to claim 7, characterized in that the calculation formula of the fused image features and laser point cloud data features in step S34 is: C _fusion = C _origin + a ₁ × SelfAttension × FusionFeature L _fusion = L _origin + a ₂ × SelfAttension × FusionFeature Among them, C _fusion represents the fused camera image features, C _origin represents the camera image features of the original size, a ₁ and a ₂ represent the fusion scale factors respectively, SelfAttension represents the fusion attention features of the camera and lidar, FusionFeature represents the fusion features of the camera and lidar, L _fusion represents the fused laser point cloud data features, and L _origin represents the laser point cloud data features of the original size. 9. The three-dimensional semantic segmentation method based on camera and laser fusion according to claim 1 is characterized in that the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of calculating the loss function in the self-supervision mode is as follows: The supervision module generates pseudo labels by adding the confidence PIDNet network, while retaining high-confidence pixels and laser point cloud data. By setting the camera mask and lidar mask, the loss function of the self-supervised mode is obtained, namely: L _{self-supervised} = L _foc1 + L _lov1 + L _fov2 + L _lov2 + L _kl Among them, L _{self-supervised} represents the loss function of the self-supervised mode, L _kl represents the one-way KL divergence, L _foc1 , L _lov1 represent the focal loss and Lovalz loss between the prediction result of the camera branch and the pseudo label, respectively, L _fov2 , L _lov2 represent the focal loss and Lovalz loss between the prediction result of the lidar branch and the pseudo label, u, v represent the length and width of the prediction result feature map, respectively, C represents the confidence, focalloss(·) represents the focal loss function calculation formula, Pred _camera represents the prediction result of the camera branch, Pred _Lidar represents the prediction result of the lidar branch, label represents the pseudo label value, M _θ1 represents the camera mask, M _θ2 represents the lidar mask, and M _l represents the lidar mask. 10. The three-dimensional semantic segmentation method based on camera and laser fusion according to claim 1 is characterized in that the camera image feature map and the laser point cloud data feature map obtained in step S4 are input into the supervision module, and the specific process of using the supervised mode to calculate the loss function is: The supervision module uses focus loss and Lovalz loss to adjust the parameter weights to obtain the loss function of the supervised mode, namely: L _supervised =L _foc1 +L _lov1 +L _fov2 +L _lov2 Among them, L _supervised represents the loss function of the supervised mode, L _foc1 and L _lov1 represent the focus loss and Lovalz loss between the prediction results of the camera branch and the true value label, respectively, and L _fov2 and L _lov2 represent the focus loss and Lovalz loss between the prediction results of the lidar branch and the true value label, respectively.