CN112598675A - Indoor scene semantic segmentation method based on improved full convolution neural network - Google Patents

Abstract

The invention discloses an indoor scene semantic segmentation method based on an improved full convolutional neural network. First, a convolutional neural network is constructed, and its hidden layer includes 5 neural network blocks, 5 feature re-extraction convolutional layer blocks, 5 block attention convolutional blocks, 12 fusion layers, and 4 upsampling layers; using the original The indoor scene image is input into the convolutional neural network for training, and the corresponding semantic segmentation prediction map is obtained; then the set of the semantic segmentation prediction map corresponding to the original indoor scene image is calculated and the corresponding real semantic segmentation image is processed into a single hot Encode the loss function value between sets of images to obtain the optimal weight vector and bias term of the convolutional neural network classification training model; input the indoor scene images to be semantically segmented into the trained convolutional neural network classification training In the model, the predicted semantic segmentation image is obtained. The advantages of the present invention are that the efficiency and accuracy of semantic segmentation of indoor scene images are improved.

Description

Indoor scene semantic segmentation method based on improved fully convolutional neural network

technical field

The invention relates to a deep learning semantic segmentation method, in particular to an indoor scene semantic segmentation method based on an improved full convolutional neural network.

Background technique

Image semantic segmentation is one of the most challenging tasks in computer vision and plays a key role in applications such as autonomous driving, medical image analysis, virtual reality, and human-computer interaction. The core purpose of semantic segmentation is to give a category label to each pixel in a picture, and to determine which category the pixel belongs to. Since the dataset of semantic segmentation generally involves indoor scenes or outdoor scenes, there are many segmentation objects, so in the It is essentially a multi-classification problem.

From the perspective of supervised learning, image semantic segmentation can be divided into three types: fully supervised, semi-supervised and unsupervised. However, from the perspective of operability and theoretical application, the current mainstream models mostly use fully supervised types, and a few use semi-supervised types. , because it is easier to implement and the model is easier to train.

In terms of model application, based on the emergence and development of fully convolutional neural networks, the application of fully convolutional neural networks has achieved superior performance and segmentation effects in image semantic segmentation tasks, but there are still many shortcomings and defects, such as a large number of parameters. , there is a lot of redundant information, and the feature expression extraction is insufficient. Therefore, there is still a lot of room for improvement in the image semantic segmentation model based on the fully convolutional neural network. According to the characteristics of the image itself, the structure of the model, and the operation principle of the biological human visual system, it has been proposed and trained. It has become the current and future development goal in the field of image semantic segmentation.

SUMMARY OF THE INVENTION

In order to solve the problems in the background art, the present invention provides an indoor scene semantic segmentation method based on an improved fully convolutional neural network, which obtains the model design direction and ideas from the expression of semantic features, the operation principle of the human visual system, etc., and improves Compared with the traditional fully convolutional neural network, the image segmentation performance is effectively improved.

The technical scheme adopted in the present invention comprises the following steps:

Step 1: Select Q pairs of original indoor scene images and corresponding real semantic segmentation images, and form a training set of all original indoor scene images and their corresponding real semantic segmentation images; each pair of original indoor scene images includes the original indoor scene color image and the original Depth images of indoor scenes, using one-hot encoding technology (one-hot) to process the real semantic segmentation images in the training set into 41 one-hot encoded images;

Step 2: Build a convolutional neural network classification training model: the convolutional neural network classification training model includes an input layer, a hidden layer and an output layer; the input layer includes a color map input layer and a depth map input layer; the hidden layer includes a color image processing module and an output layer. The depth image processing module; the color image processing module and the depth image processing module are symmetrical in structure, including five neural network blocks, five feature re-extraction convolution blocks and ten fusion layers; the hidden layer also includes five block attention Convolutional blocks, four upsampling layers and two fusion layers;

Step 3: Input the training set into the convolutional neural network classification training model of Step 2 for training. During the training process, 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images are obtained by each iteration of training processing, and 41 images are calculated. The loss function value between the set of semantic segmentation prediction images and the set of 41 one-hot encoded images corresponding to the real semantic segmentation images;

The loss function value is obtained by categorical cross entropy;

Step 4: Repeat step 3 for a total of V times, and obtain a total of Q×V loss function values; then find the smallest loss function value from the Q×V loss function values, and assign the weight corresponding to the smallest loss function value The vector and the bias term are used as the optimal weight vector and the optimal bias term of the convolutional neural network classification training model, so as to complete the training of the convolutional neural network classification training model;

Step 5: Use the convolutional neural network classification training model obtained after training to perform prediction processing on the indoor scene image to be predicted, and output the corresponding predicted semantic segmentation image to realize the semantic segmentation of the indoor scene image.

Described step 2) is specifically:

The color map input layer and the depth map input layer respectively input the first neural network block in the color image processing module and the depth image processing module;

The structure of the color image processing module and the depth image processing module is the same, specifically:

One of the outputs of the first neural network block is extracted by the first feature and then the convolution block is input to the first fusion layer, the other output of the first neural network block is input to the first fusion layer; one of the second neural network block The output is extracted by the second feature and the convolution block is input to the third fusion layer, and the other output of the second neural network block is input to the third fusion layer; one of the outputs of the third neural network block is extracted by the third feature. The convolution block is input to the fifth fusion layer, and the other output of the third neural network block is input to the fifth fusion layer; one of the outputs of the fourth neural network block is extracted by the fourth feature, and the convolution block is input to the seventh fusion. layer, the other output of the fourth neural network block is input to the seventh fusion layer; one of the outputs of the fifth neural network block is extracted by the fifth feature and then the convolution block is input to the ninth fusion layer, the fifth neural network block The other output is input to the ninth fusion layer; the two inputs of each fusion layer are fused by element-wise addition;

The output of the first fusion layer is input to the first block attention convolution block and the corresponding second fusion layer, respectively, and the output of the third fusion layer is input to the second block attention convolution block and the corresponding Four fusion layers, the output of the fifth fusion layer is respectively input to the third block attention convolution block and the corresponding sixth fusion layer, and the output of the seventh fusion layer is input to the fourth block attention convolution block respectively and the corresponding eighth fusion layer, the output of the ninth fusion layer is respectively input into the fifth block attention convolution block and the corresponding tenth fusion layer;

The two outputs of the first block attention convolution block are input to the second fusion layer of the color image processing module and the depth image processing block, respectively, and the two outputs of the second block attention convolution block are input to the color image, respectively The fourth fusion layer of the processing module and the depth image processing module, the two outputs of the third block attention convolution block are respectively input to the sixth fusion layer of the color image processing module and the depth image processing module, and the fourth The two outputs of the block attention convolution block are input to the eighth fusion layer of the color image processing module and the depth image processing module, respectively, and the two outputs of the fifth block attention convolution block are input to the color image processing module and the depth image processing module, respectively. The tenth fusion layer of the image processing module;

The two inputs of the second fusion layer are fused by element-by-element addition and then input to the eleventh fusion layer and the corresponding second neural network block respectively, and the two inputs of the fourth fusion layer are added element-by-element. After fusion, the first upsampling layer and the corresponding third neural network block are input respectively, and the two inputs of the sixth fusion layer are fused by element-by-element addition and then input to the second upsampling respectively. layer and the corresponding fourth neural network block, the two inputs of the eighth fusion layer are fused by element-by-element addition and then input to the third upsampling layer and the corresponding fifth neural network block; the tenth The output of each fusion layer is input to the fourth upsampling layer;

The two inputs of the eleventh fusion layer, the first upsampling layer, the second upsampling layer, the third upsampling layer and the fourth upsampling layer are fused by element-by-element addition and input into the first Twelve fusion layers;

The twelfth fusion layer concatenates all the inputs and outputs them through the output layer. The output layer is mainly composed of a convolutional layer and a fifth upsampling layer that are connected in sequence.

The corresponding representation indicates that the former input and the latter output are located in the same color image processing module or the same depth image processing module.

The five neural network blocks use the MobileNet V2 network structure. The first neural network block uses layers 1 to 4 in MobileNetV2 (the number of repetitions n is 1, 1, and 2, a total of four layers), and the second neural network block uses Layers 5 to 7 in MobileNetV2 (the number of repetitions n is 3 and a total of three layers), the third neural network block uses layers 8 to 11 in MobileNetV2 (the number of repetitions n is 4 and a total of four layers), and the fourth neural network block uses The 12-14 layers in MobileNetV2 (the number of repetitions n is 3, a total of three layers), and the fifth neural network block uses the 15-17 layers in MobileNetV2 (the number of repetitions n is 3, a total of three layers).

Each of the feature re-extraction convolution blocks is composed of four re-extraction modules connected in sequence, and each re-extraction module includes a convolution layer, a normalization layer and an activation layer connected in sequence; the activation methods of all the activation layers are ReLU6; the stride of all convolutional layers is 1; the number of convolution kernels of the convolutional layer in each re-extraction module is the same as the normalization parameters of the normalization layer; the first re-extraction module, the third re-extraction module and the The convolution kernel size of the convolution layer in the fourth re-extraction module is the same, and the convolution kernel size of the convolution layer in the second re-extraction module is half of the convolution kernel size of the convolution layer in the first re-extraction module. ; The convolution kernel size of the convolution layer in the first re-extraction module and the third re-extraction module is 3*3, and the convolution kernel of the convolution layer in the second re-extraction module and the fourth re-extraction module. The size is 1*1; the expansion factor of the convolutional layer in the first re-extraction module is 1, and the expansion factor of the convolutional layer in the third re-extraction module is 2; the expansion factor of the convolutional layer in the first re-extraction module The zero-padding parameter is 1, the zero-padding parameter of the convolutional layer in the third re-extraction module is 2, and the zero-padding parameter of the convolutional layer in the second and fourth re-extraction modules is both 0.

The convolution kernel size of the convolution layer of the first re-extraction module in the first feature re-extraction convolution block is 24, and the volume of the convolution layer of the first re-extraction module in the second feature re-extraction convolution block The kernel size is 32, the convolution kernel size of the convolution layer of the first re-extraction module in the third feature re-extraction convolution block is 64, and the fourth feature re-extraction convolution block The first re-extraction module in the convolution block The convolution kernel size of the convolutional layer is 96, and the convolution kernel size of the convolutional layer of the first re-extraction module in the fifth feature re-extraction convolution block is 160;

The grouping parameters of the four sequentially set convolutional layers in the first feature re-extraction convolution block are 24, 12, 24, and 24, respectively, and the second feature re-extracts the four sequentially set convolutional layers in the convolutional block. The grouping parameters are 32, 16, 16, and 32, respectively. The grouping parameters of the four successively set convolutional layers in the third feature re-extraction convolution block are 64, 32, 32, and 64, respectively. The fourth feature re-extracts the volume. The grouping parameters of the four sequentially set convolutional layers in the accumulation block are 64, 48, 48, and 99, respectively, and the fifth feature extraction convolutional block The grouping parameters of the four sequentially set convolutional layers are 96, 80 , 80, 160.

Each of the block attention convolution blocks includes a block layer, a channel attention layer, and a spatial size attention layer. For the size attention layer, the input of the block attention convolution block is multiplied by the output of the channel attention layer and the output of the spatial size attention layer, and then the multiplied results are added together as the block attention convolution block. Output.

The block layer adopts the split function that comes with pytorch, and the parameter is half of the number of channels corresponding to the input feature map of the block attention convolution block;

The channel attention layer includes an adaptive max pooling layer, a first convolutional layer of channel attention, a second convolutional layer of channel attention, and a channel attention activation layer connected in sequence; the max pooling layer of the adaptive max pooling layer The parameterization parameter is 1; the convolution kernel size of the first convolutional layer of channel attention and the second convolutional layer of channel attention are both 1x1, the stride is 1, the bias term is False, the grouping parameters are the same, and the channel attention is The number of convolution kernels in the first convolution layer is half of the number of convolution kernels in the second convolution layer of channel attention;

The number of convolution kernels in the first convolutional layer is 12 and the grouping parameter is 12; the channel attention in the second block attention convolution block The channel attention of the attention layer The number of convolution kernels of the first convolution layer is 16, and the grouping parameter is 16; the channel attention of the channel attention layer in the third block attention convolution block is the first convolution layer. The number of convolution kernels is 32, and the grouping parameter is 32; the channel attention of the channel attention layer in the fourth block attention convolution block The number of convolution kernels in the first convolutional layer is 48, and the grouping parameter is 48; The channel attention of the channel attention layer in the fifth block attention convolution block The number of convolution kernels in the first convolution layer is 80, and the grouping parameter is 80;

The spatial size attention layer consists of a channel-wise maximization layer, a spatial-scale attention convolutional layer, and a spatial-scale attention activation layer; the channel-wise maximization layer uses the max function that comes with pytorch; the spatial size attention convolutional layer The size of the convolution kernel is 3 × 3, the number of convolution kernels is 1, the expansion factor is 2, the zero-padding parameter is 2, the grouping parameter is 1, and the bias term is False;

The activation method used by the channel attention activation layer and the spatial dimension attention activation layer is the sigmoid function.

The upsampling layer adopts the UpsamlingBilinear2d function that comes with pytorch. The function parameter of the first upsampling layer is 2, the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer. The function parameter of the upsampling layer is 4; the convolution kernel size of the convolution layer in the output layer is 1×1, the number of convolution kernels is 41, the step size is 1, and the bias term is False.

The input end of the color map input layer receives the color image of the indoor scene, the input end of the depth map input layer receives the depth image of the indoor scene, and the output of the output layer is 41 semantic segmentation prediction images corresponding to the indoor scene images input by the input layer.

Beneficial effects of the present invention:

1) The method of the present invention starts from the effective extraction of image channel and spatial semantic information, and reduces the information loss during the gradient propagation process as much as possible, and at the same time does not significantly increase the number of model parameters, a module called feature re-extraction module is designed. The module structure is roughly uniform columnar, which contains a 1*1 convolution and two 3*3 convolutional neural networks, and a 3*3 convolutional neural network adopts the setting of dilated convolution. To facilitate the effective propagation of gradients, two 3*3 convolutional neural networks are located at both ends of the module, and 1*1 convolution is located inside the module.

2) The method of the present invention starts from the operation principle of the human visual system, combines the attention mechanism of human vision and block convolution, and designs a module called block attention convolution block. This module includes a channel-based and space-based attention mechanism module. Using the principle of block convolution, the input convolution feature map is first divided into two parts by channel, one part is used for feature screening based on the channel attention mechanism, and the other part is used for feature screening based on the channel attention mechanism. It is used for feature screening based on the spatial attention mechanism, and finally the two filtered features are multiplied with the input convolution feature map, which effectively reduces redundant features.

3) The method of the present invention combines the designs of the above two modules, and uses MobileNetV2 as the main frame to design the network model MSCNet of the present invention. Experiments show that this model has fewer parameters and faster speed. Each training round only takes about 43 seconds, while maintaining high accuracy. It is a lightweight network suitable for mobile terminals.

Description of drawings

Fig. 1 is the overall realization block diagram of the method of the present invention;

Figure 2a is the first original indoor scene color image of the same scene;

Figure 2b is the first original indoor scene depth image of the same scene;

Figure 2c is a predicted semantic segmentation image obtained by using the method of the present invention to predict the original indoor scene image shown in Figures 2a and 2b;

Figure 3a is the second original indoor scene color image of the same scene;

Figure 3b is the second original indoor scene depth image of the same scene;

Figure 3c is a predicted semantic segmentation image obtained by using the method of the present invention to predict the original indoor scene images shown in Figures 3a and 3b;

Fig. 4a is the 3rd original indoor scene color image of the same scene;

Figure 4b is the third original indoor scene depth image of the same scene;

Figure 4c is a predicted semantic segmentation image obtained by using the method of the present invention to predict the original indoor scene images shown in Figures 4a and 4b;

Fig. 5a is the 4th original indoor scene color image of the same scene;

Figure 5b is the fourth original indoor scene depth image of the same scene;

Fig. 5c is a predicted semantic segmentation image obtained by using the method of the present invention to predict the original indoor scene images shown in Figs. 5a and 5b.

FIG. 6 is a structural block diagram of the block attention convolution block of the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

An indoor scene semantic segmentation method based on an improved fully convolutional neural network proposed by the present invention, the overall implementation block diagram of which is shown in Figure 1, which includes two processes: a training phase and a testing phase;

The specific steps of the training phase process are:

然后采用现有的独热编码技术(one-hot)将训练集中的每对原始的室内场景图像对应的真实语义分割图像处理成41幅独热编码图像，将

处理成的41幅独热编码图像构成的集合记为

其中，室内场景图像包括RGB彩色图像和Depth深度图，Q为正整数，Q≥200，如取Q＝794，q为正整数，1≤q≤Q，1≤i≤W，1≤j≤H，W表示{RGB^q(i,j)，Depth^q(i,j)}中彩色图RGB和深度图Dept的宽度，H表示{RGB^q(i,j),Depth^q(i j,)}中彩色图RGB和深度图Depth的高度，如取W＝480、H＝640，RGB^q(i,j)，Depth^q(i,j)分别表示{RGB^q(i,j),Depth^q(i,j)}中坐标位置为(i,j)的像素点的像素值，

表示

中坐标位置为(i,j)的像素点的像素值；在此，原始的室内场景图像直接选用室内场景图像数据库NYUv2训练集中的1448幅图像，为了更方便训练，将每幅图像尺寸都缩减到宽度480，高度640。此外，为了有效缓解模型过拟合问题，采用了随机裁剪、随机水平翻转、随机缩放三种数据增强方法来扩充训练集中的数据。Step 1_1: Select Q pairs of original indoor scene RGB color images and Depth map images and the real semantic segmentation images corresponding to each pair of original indoor scene images respectively, and form a training set, and combine the qth pair of original indoor scene images in the training set. Denote it as {RGB ^q (i,j),Depth ^q (i,j)}, and denote the training set {RGB ^q (i,j),Depth ^q (i,j)} and the corresponding real semantic segmentation image as

Then, the existing one-hot encoding technology (one-hot) is used to process the real semantic segmentation images corresponding to each pair of original indoor scene images in the training set into 41 one-hot encoded images, and the

The set of processed 41 one-hot encoded images is denoted as

Among them, the indoor scene image includes RGB color image and Depth depth map, Q is a positive integer, Q≥200, if Q=794, q is a positive integer, 1≤q≤Q, 1≤i≤W, 1≤j≤ H, W represent the width of the color map RGB and depth map Dept in {RGB ^q (i, j), Depth ^q (i, j)}, H represents {RGB ^q (i, j), Depth ^q (ij,)} The height of the color map RGB and depth map Depth, such as W=480, H=640, RGB ^q (i, j), Depth ^q (i, j) represent {RGB ^q (i, j), Depth ^q ( The pixel value of the pixel whose coordinate position is (i,j) in i,j)},

express

The pixel value of the pixel whose median coordinate position is (i, j); here, the original indoor scene image is directly selected from the 1448 images in the indoor scene image database NYUv2 training set. In order to facilitate training, the size of each image is reduced to a width of 480 and a height of 640. In addition, in order to effectively alleviate the problem of model overfitting, three data augmentation methods, random cropping, random horizontal flipping, and random scaling, are used to expand the data in the training set.

Step 1_2: Build a convolutional neural network classification training model: the convolutional neural network includes an input layer, a hidden layer and an output layer; the hidden layer includes the first neural network block, the second neural network block, and the third neural network set in sequence. Network block, fourth neural network block, fifth neural network block, first feature extraction convolution block, second feature extraction convolution block, third feature extraction convolution block, fourth feature extraction Feature re-extraction convolution block, 5th feature re-extraction convolution block, 1st block attention convolution block, 2nd block attention convolution block, 3rd block attention convolution block, 4th block attention convolution block, 5th block attention convolution block, 1st fusion layer, 2nd fusion layer, 3rd fusion layer, 4th fusion layer, 5th fusion layer, 6th fusion layer, 7th fusion layer, 8th fusion layer, 9th fusion layer, 10th fusion layer, 11th fusion layer, 12th fusion layer, 1st upsampling layer , the second upsampling layer, the third upsampling layer, and the fourth upsampling layer.

For the input layer, the present invention has two inputs, namely the color map RGB input layer and the depth map Depth input layer. The color map RGB input end of the input layer receives the R channel components, G channel components, B channel components of an original RGB input image. Channel component, the depth map Depth input of the input layer receives a single-channel component of the original Depth input image, and the color map RGB output of the input layer outputs the R channel component, G channel component and B channel component of the original input image to the hidden layer. , the depth map Depth output terminal of the input layer outputs the single-channel component of the original input image to the hidden layer; wherein, the width of the original input image received by the input terminal of the input layer is required to be 480 and the height is 640; in addition, the input is a color map RGB The convolutional neural network structure is symmetric with the convolutional neural network structure whose input is the depth map Depth.

For the five neural network blocks, the five neural network blocks use the MobileNet V2 network structure, and the first neural network block uses layers 1 to 4 in MobileNetV2 (the number of repetitions n is 1, 1, and 2, a total of four layers), and the second The neural network block adopts 5 to 7 layers in MobileNetV2 (the number of repetitions n is 3 in total), the third neural network block adopts 8 to 11 layers in MobileNetV2 (the number of repetitions n is 4 in total), and the fourth The neural network block adopts layers 12 to 14 in MobileNetV2 (the number of repetitions n is 3, a total of three layers), and the fifth neural network block adopts layers 15 to 17 in MobileNetV2 (the number of repetitions n is 3, a total of three layers).

高度为

第2个神经网络块的彩色图RGB输出端输出32幅特征图，其构成的集合记为R2；第2个神经网络块的深度图Depth输入端接收DF1中的所有特征图，第2个神经网络块的深度图Depth输出端输出32幅特征图，其构成的集合记为D2；R2和D2中的每幅特征图的宽度均为

高度均为

第3个神经网络块的彩色图RGB输入端接收RF2中的所有特征图，第3个神经网络块的彩色图RGB输出端输出64幅特征图，将64幅特征图构成的集合记为R3；第3个神经网络块的深度图Depth输入端接收DF2中的所有特征图，第3个神经网络块的深度图Depth输出端输出64幅特征图，将64幅特征图构成的集合记为D3；R3和D3中每幅特征图宽度均为

高度均为

第4个神经网络块的彩色图RGB输入端接收RF3中的所有特征图，第4个神经网络块的彩色图RGB输出端输出96幅特征图，将96幅特征图构成的集合记为R4；第4个神经网络块的深度图Depth输入端接收DF3中的所有特征图，第4个神经网络块的深度图Depth输出端输出96幅特征图，将其构成的集合记为D4；R4和D4中的每幅特征图的宽度均为

高度均为

第5个神经网络块的彩色图RGB输入端接收RF4中的所有特征图，第5个神经网络块的彩色图RGB输出端输出160幅特征图，将160幅特征图构成的集合记为R5；第5个神经网络块的深度图Depth输入端接收DF4中的所有特征图，第5个神经网络块的深度图Depth输出端输出160幅特征图，将160幅特征图构成的集合记为D5；R5和D5中的每幅特征图的宽度为

高度为

The color map RGB input terminal of the first neural network block receives the R channel components, G channel components and B channel components of the original input image output by the output terminal of the input layer, and the depth map Depth input terminal receives the output terminal of the input layer. The single-channel component of the input image, the color map RGB output of the first neural network block outputs 24 feature maps, and the set composed of them is recorded as R1, and the depth map Depth output of the first neural network block outputs 24 features. Figure, denote the set formed by it as D1; the width of each feature map in R1 and D1 is

height is

The color map RGB output terminal of the second neural network block outputs 32 feature maps, and the set formed by them is denoted as R2; the depth map Depth input terminal of the second neural network block receives all the feature maps in DF1, and the second neural network block receives all the feature maps in DF1. The depth map Depth output of the network block outputs 32 feature maps, and the set formed by them is denoted as D2; the width of each feature map in R2 and D2 is

height is

The color map RGB input terminal of the third neural network block receives all the feature maps in RF2, and the color map RGB output terminal of the third neural network block outputs 64 feature maps, and the set of 64 feature maps is recorded as R3; The depth map Depth input of the third neural network block receives all the feature maps in DF2, and the depth map Depth output of the third neural network block outputs 64 feature maps, and the set of 64 feature maps is recorded as D3; The width of each feature map in R3 and D3 is

height is

The color map RGB input end of the fourth neural network block receives all the feature maps in RF3, and the color map RGB output end of the fourth neural network block outputs 96 feature maps, and the set composed of 96 feature maps is recorded as R4; The depth map Depth input of the fourth neural network block receives all feature maps in DF3, and the depth map Depth output of the fourth neural network block outputs 96 feature maps, and the set formed by them is recorded as D4; R4 and D4 The width of each feature map in is

height is

The color map RGB input terminal of the fifth neural network block receives all the feature maps in RF4, and the color map RGB output terminal of the fifth neural network block outputs 160 feature maps, and the set of 160 feature maps is recorded as R5; The depth map Depth input of the fifth neural network block receives all feature maps in DF4, and the depth map Depth output of the fifth neural network block outputs 160 feature maps, and the set of 160 feature maps is recorded as D5; The width of each feature map in R5 and D5 is

height is

高度为

For the first feature re-extraction convolution block, it consists of the fifty-first convolutional layer, the fifty-first normalization layer, the fifty-first activation layer, the fifty-second convolutional layer, and the fifty-second convolutional layer. Normalization layer, fifty-second activation layer, fifty-third convolutional layer, fifty-third normalization layer, fifty-third activation layer composition, fifty-fourth convolutional layer, fifty-fourth normalization layer, fifth Fourteen activation layers; the input of the first feature extraction convolution block receives all the feature maps in R1 and D1, and the color map RGB output of the first feature extraction block outputs 24 feature maps, which are composed of The set is denoted as RS1, the output of the depth map Depth outputs 24 feature maps, and the set composed of them is denoted as DS1; among them, the size of the convolution kernel of the fifty-first convolutional layer is 3 × 3, and the number of convolution kernels is 24. The stride is 1, the dilation factor is 1, the padding parameter is 1, the groups parameter is 24, and the standardization parameter of the fifty-first layer is 24; Volume 52 The convolution kernel size of the product layer is 1x1, the number of convolution kernels is 12, the stride is 1, the padding parameter is 0, the groups parameter is 12, and the fifty-second layer standardization parameter is 12; the size of the convolution kernel of the fifty-third convolutional layer is 3×3, the number of convolution kernels is 24, the stride is 1, the dilation factor is 2, and the padding parameter is is 2, the groups parameter is 24, and the normalization parameter of the fifty-third layer is 24; the convolution kernel size of the fifty-fourth convolution layer is 1x1, the number of convolution kernels is 24, and the stride is 1. The padding parameter is 0, the groups parameter is 24, and the fifty-fourth layer normalization parameter is 24. The activation mode of all activation layers is "ReLU6", and the width of each feature map in RS1 and DS1 is

height is

高度均为

For the first fusion layer, the color map RGB input terminal of the first fusion layer receives all feature maps in R1 and all feature maps in RS1, and fuses R1 and RS1 through the existing add (element-by-element addition) method to obtain Set RA1, the color map RGB output terminal of the first fusion layer outputs RA1; the depth map Depth input terminal of the first fusion layer receives all feature maps in D1 and all feature maps in DS1, and the first connection layer Some add (element-by-element addition) methods fuse D1 and DS1 to obtain a set DA1, and the depth map Depth output of the first fusion layer outputs DA1; among them, the total number of feature maps contained in RA1 and DA1 is 24, The width of each feature map in RA1 and DA1 is

height is

高度均为

For the first block attention convolution block, the structure is shown in Figure 6, which consists of the first block layer, the first channel attention layer, and the first spatial size attention layer set in sequence; the first block The color map RGB input of the attention convolution block receives all feature maps in RA1, the depth map Depth input of the first block attention convolution block receives all feature maps in DA1, and all features in RA1 and DA1 The graphs are respectively input into the first block attention convolution block. First, all the input feature maps are divided into two parts by channel through the first block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the spatial dimension attention layer. Input, and finally multiply the results of the attention processing on both sides with the original input feature map and add the multiplied results. The color map RGB output of the first block attention convolution block outputs 24 feature maps, which are The set is recorded as RT1, the depth map Depth output of the first block attention convolution block outputs 24 feature maps, and the set is recorded as DT1; among them, the first block layer uses the split function that comes with pytorch, and the parameters is half of the original input feature map and the number of corresponding channels; the first channel attention layer includes the first adaptive maximum pooling, the first channel attention first convolution layer, the first channel attention second convolution layer and activation layer, where the first adaptive maximum pooling parameter is 1, the first channel attention The first convolutional layer convolution kernel size is 1x1, the number of convolution kernels is 12, the stride is 1, and the groups are ) parameter is 12, the bias term (bias) is False; the size of the convolution kernel of the first channel attention second convolution layer is 1x1, the number of convolution kernels is 24, the stride is 1, and the groups are ) parameter is 12, the bias term (bias) is False; the activation mode of the first channel attention activation layer is "Sigmoid"; the first space size attention layer includes the first channel-wise maximization layer, the first space size attention layer The convolution layer and activation function, in which the first channel maximization layer adopts the max function that comes with pytorch; the first spatial dimension attention convolution layer convolution kernel size is 3 × 3, the number of convolution kernels is 1, and the expansion The factor (dilation) is 2, the padding parameter is 2, the grouping (groups) parameter is 1, the bias term (bias) is False, and the activation mode of the first spatial size attention activation layer is "Sigmoid". The width of each feature map in RT1 and DT1 is

height is

高度均为

For the second fusion layer, the color map RGB input terminal of the second fusion layer receives all the feature maps in DT1, and fuses RA1 and DT1 through the existing add (element-by-element addition) method to obtain the set RF1, and the second fusion layer The color map RGB output of the layer outputs RF1; the depth map Depth input of the second fusion layer receives all feature maps in RT1 and all feature maps in DA1, and the second fusion passes the existing add (element-by-element addition). ) way to fuse RT1 and DA1 to obtain a set DF1, and the depth map Depth output of the second fusion layer outputs DF1; among them, the total number of feature maps contained in RF1 and DF1 is 24, and each feature in RF1 and DF1 The width of the graph is

height is

高度为

For the second feature re-extraction convolution block, it consists of the fifty-fifth convolution layer, the fifty-fifth normalization layer, the fifty-fifth activation layer, the fifty-sixth convolution layer, and the fifty-sixth convolution layer. Normalization layer, fifty-sixth activation layer, fifty-seventh convolutional layer, fifty-seventh normalization layer, fifty-seventh activation layer, fifty-eighth convolutional layer, fifty-eighth normalization layer, fifty It consists of eight activation layers; the color map RGB input end of the second feature re-extraction convolution block receives all the feature maps in R2, and the color map RGB output end of the second feature re-extraction convolution block outputs 32 feature maps. The set formed by it is denoted as RS2; the depth map Depth input of the second feature re-extraction convolution block receives all the feature maps in D2, and the depth map Depth output of the second feature re-extraction convolution block outputs 32 features. Figure, the set composed of it is recorded as DS2; among them, the size of the convolution kernel of the fifty-fifth convolution layer is 3 × 3, the number of convolution kernels is 32, the stride is 1, and the dilation factor is 1. ) is 1, the padding parameter is 1, the groups parameter is 32, and the normalization parameter of the fifty-fifth layer is 32; the convolution kernel size of the fifty-sixth convolutional layer is 1x1, and the convolution kernel The number is 16, the stride is 1, the padding parameter is 0, the groups parameter is 16, and the normalization parameter of the fifty-sixth layer is 16; the size of the convolution kernel of the fifty-seventh convolutional layer It is 3×3, the number of convolution kernels is 32, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 16, and the fifty-seventh The layer normalization parameter is 32; the convolution kernel size of the fifty-eighth convolution layer is 1x1, the number of convolution kernels is 32, the stride is 1, the padding parameter is 0, and the groups are The parameter is 32, and the normalization parameter of the fifty-eighth layer is 32. The activation method of all activation layers is "ReLU6", and the width of each feature map in RS2 and DS2 is

height is

高度均为

For the third fusion layer, the color map RGB input terminal of the third fusion layer receives all feature maps in R2 and all feature maps in RS2, and the third fusion layer uses the existing add (element-by-element addition) method Fuse R2 and RS2 to get the set RA2, the color map RGB output of the third fusion layer outputs RA2; the depth map Depth input of the third fusion layer receives all feature maps in D2 and all feature maps in DS2, the second Each connection layer fuses D2 and DS2 through the existing add (element-by-element addition) method to obtain a set DA2, and the depth map Depth output of the third fusion layer outputs DA2; among them, the total width of the feature maps contained in RA2 and DA2 The number is 32, and the width of each feature map in RA2 and DA2 is

height is

高度均为

For the second block attention convolution block, it consists of the second block layer, the second channel attention layer, and the second spatial dimension attention layer set in sequence; the color of the second block attention convolution block Figure RGB input receives all feature maps in RA2, all feature maps in RA2 are input into the second block attention convolution block, first all input feature maps are divided into two parts by channel through the second block layer, one part As the input of the channel attention layer, the other part is used as the input of the spatial dimension attention layer. Finally, the results of the attention processing on both sides are multiplied by the original input feature map respectively, and the multiplication results are added together. The second block attention The color map RGB output of the convolution block outputs 32 feature maps, and the set composed of them is recorded as RT2; the depth map Depth input of the second block attention convolution block receives all the feature maps in DA2. All feature maps are input into the second block attention convolution block. First, all input feature maps are divided into two parts by channel through the second block layer, one part is used as the input of the channel attention layer, and the other part is used as the spatial dimension attention layer. The input of the force layer, and finally the results of the attention processing on both sides are multiplied by the original input feature maps and then the multiplication results are added together. The depth map Depth output of the second block attention convolution block outputs 32 feature maps. , and the set formed by it is recorded as DT2; among them, the second block layer adopts the split function that comes with pytorch, and the parameters are half of the original input feature map and the number of corresponding channels; the second channel attention layer contains the second largest adaptability Pooling, second channel attention to the first convolutional layer, second channel attention to the second convolutional layer and activation layer, where the second adaptive maximum pooling parameter is 1, the second channel attention to the first convolutional layer The convolution kernel size is 1x1, the number of convolution kernels is 16, the stride is 1, the groups parameter is 16, and the bias term is False; the second channel pays attention to the second convolution layer The size of the convolution kernel is 1x1, the number of convolution kernels is 32, the stride is 1, the groups parameter is 16, and the bias term is False; the activation method of the second channel attention activation layer is "Sigmoid"; the second spatial size attention layer includes the second channel-wise maximization layer, the second spatial-scale attention convolutional layer and the activation function, wherein the second channel-wise maximization layer adopts the max function that comes with pytorch; The size of the convolution kernel of the two-dimensional attention convolution layer is 3×3, the number of convolution kernels is 1, the dilation factor is 2, the padding parameter is 2, and the groups parameter is 1. The bias term (bias) is False, and the activation mode of the second spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT2 and DT2 is

height is

高度均为

For the fourth fusion layer, the color map RGB input terminal of the fourth fusion layer receives all the feature maps in RA2 and DT2, and the fourth fusion layer is obtained by fusing RA2 and DT2 through the existing add (element-by-element addition) method. Set RF2, the color map RGB output of the fourth fusion layer outputs RF2; the depth map Depth input of the fourth fusion layer receives all feature maps in RT2 and all feature maps in DA2, and the fourth fusion passes existing The add (element-by-element addition) method fuses RT2 and DA2 to obtain a set DF2, and the depth map Depth output of the fourth fusion layer outputs DF2; among them, the total number of feature maps contained in RF2 and DF2 is 32, RF2 and the width of each feature map in DF2 is

height is

高度为

For the third feature re-extraction convolution block, it consists of the fifty-ninth convolutional layer, the fifty-ninth normalization layer, the fifty-ninth activation layer, the sixtieth convolutional layer, and the sixtieth normalization layer. , 60th activation layer, 61st convolution layer, 61st normalization layer, 61st activation layer, 62nd convolution layer, 62nd normalization layer, 62nd activation layer Composition; the color map RGB input end of the third feature extraction convolution block receives all the feature maps in R3, and the color map RGB output end of the third feature extraction convolution block outputs 64 feature maps, which are composed of The set is denoted as RS3; the depth map Depth input of the third feature re-extraction convolution block receives all feature maps in D3, and the depth map Depth output of the third feature re-extraction convolution block outputs 64 feature maps. The set composed of them is denoted as DS3; among them, the convolution kernel size of the fifty-ninth convolutional layer is 3×3, the number of convolution kernels is 64, the stride is 1, and the dilation factor is 1. , the padding parameter is 1, the groups parameter is 64, and the normalization parameter of the fifty-ninth layer is 64; the convolution kernel size of the sixtieth convolution layer is 1x1, the number of convolution kernels is 32, and the step The stride is 1, the padding parameter is 0, the groups parameter is 32, and the standardization parameter of the sixtieth layer is 32; the size of the convolution kernel of the sixty-first convolutional layer is 3×3, The number of convolution kernels is 64, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 32, and the standardization parameter of the sixty-first layer is 64. ; The convolution kernel size of the 62nd convolution layer is 1x1, the number of convolution kernels is 64, the stride is 1, the padding parameter is 0, the groups parameter is 64, and the The sixty-two layer normalization parameter is 64. The activation method of all activation layers is "ReLU6", and the width of each feature map in RS3 and DS3 is

height is

高度均为

For the fifth fusion layer, the color map RGB input of the fifth fusion layer receives all feature maps in R3 and all feature maps in RS3, and the fifth fusion layer uses the existing add (element-by-element addition) method Fusing R3 and RS3 to get the set RA3, the color map RGB output of the fifth fusion layer outputs RA3; the depth map Depth input of the fifth fusion layer receives all feature maps in D3 and all feature maps in DS3, the third Each fusion layer fuses D3 and DS3 through the existing add (element-by-element addition) method to obtain a set DA3, and the depth map Depth output of the fifth fusion layer outputs DA3; among them, the total width of the feature maps contained in RA3 and DA3 The number is 64, and the width of each feature map in RA3 and DA3 is

height is

高度均为

For the third block attention convolution block, it consists of the third block layer, the third channel attention layer, and the third spatial dimension attention layer set in sequence; the color of the third block attention convolution block Figure RGB input receives all feature maps of RA3, all feature maps in RA3 are input into the third block attention convolution block, first all input feature maps are divided into two parts by channel through the third block layer, one part As the input of the channel attention layer, the other part is used as the input of the spatial dimension attention layer. Finally, the results of the attention processing on both sides are multiplied by the original input feature map respectively, and the multiplication results are added together. The third block attention The color map RGB output of the convolution block outputs 64 feature maps, and the set of 64 feature maps is recorded as RT3; the depth map Depth input of the third block attention convolution block receives all the feature maps in DA3 , All feature maps in DA3 are input into the third block attention convolution block, first all input feature maps are divided into two parts by channel through the third block layer, one part is used as the input of the channel attention layer, and the other part is used as the input of the channel attention layer. The input of the spatial size attention layer, and finally the results of the attention processing on both sides are multiplied by the original input feature map respectively, and then the multiplication results are added together. The depth map Depth output of the third block attention convolution block outputs 64 The set of 64 feature maps is recorded as DT3; among them, the third block layer adopts the split function that comes with pytorch, and the parameters are half of the original input feature map and the number of corresponding channels; the third channel attention layer Contains the third adaptive max pooling, the third channel attention first convolution layer, the third channel attention second convolution layer and the activation layer, where the third adaptive max pooling parameter is 1, the third channel attention The size of the convolution kernel of the first convolutional layer is 1x1, the number of convolution kernels is 32, the stride is 1, the groups parameter is 32, and the bias term is False; the third channel pays attention to The size of the convolution kernel of the second convolution layer is 1x1, the number of convolution kernels is 64, the stride is 1, the groups parameter is 32, and the bias term is False; the third channel pays attention to The activation method of the force activation layer is "Sigmoid"; the third spatial dimension attention layer includes the third channel-based maximization layer, the third spatial-scale attention convolutional layer and the activation function, of which the third channel-based maximization layer adopts pytorch automatic layer. The max function of the belt; the third spatial dimension attention convolution layer convolution kernel size is 3 × 3, the number of convolution kernels is 1, the expansion factor (dilation) is 2, the padding parameter is 2, and the grouping ( groups) parameter is 1, the bias term (bias) is False, and the activation mode of the third spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT3 and DT3 is

height is

高度均为

For the sixth fusion layer, the color map RGB input terminal of the sixth fusion layer receives all the feature maps in RA3 and DT3, and the sixth fusion layer fuses RA3 and DT3 through the existing add (element-by-element addition) method to obtain Set RF3, the color map RGB output of the sixth fusion layer outputs RF3; the depth map Depth input of the sixth fusion layer receives all feature maps in RT3 and all feature maps in DA3, and the sixth fusion passes existing The add (element-by-element addition) method fuses RT3 and DA3 to obtain a set DF3, and the depth map Depth output of the sixth fusion layer outputs DF3; among them, the total number of feature maps contained in RF3 and DF3 is 64, RF3 and the width of each feature map in DF3 is

height is

高度为

For the 4th feature re-extraction convolution block, it consists of the sixty-third convolutional layer, the sixty-third normalization layer, the sixty-third activation layer, the sixty-fourth convolutional layer, and the sixty-fourth convolutional layer. Normalization layer, 64th activation layer, 65th convolution layer, 65th normalization layer, 65th activation layer, 66th convolution layer, 66th normalization layer, 60th Six activation layers; the color map RGB input of the fourth feature re-extraction convolution block receives all the feature maps in R4, the output of the fourth feature re-extraction convolution block outputs 96 feature maps, and the 96 feature maps are The formed set is denoted as RS4; the depth map Depth input of the fourth feature re-extraction convolution block receives all the feature maps in D4, and the output of the fourth feature re-extraction convolution block outputs 96 feature maps. The set of feature maps is denoted as DS4; the size of the convolution kernel of the sixty-third convolutional layer is 3 × 3, the number of convolution kernels is 96, the stride is 1, and the dilation factor is 1. The padding parameter is 1, the groups parameter is 64, and the standardization parameter of the 63rd layer is 96; the size of the convolution kernel of the 64th convolution layer is 1x1, and the number of convolution kernels is 48 , the stride is 1, the padding parameter is 0, the groups parameter is 48, the standardization parameter of the sixty-fourth layer is 48; the size of the convolution kernel of the sixty-fifth convolutional layer is 3 ×3, the number of convolution kernels is 96, the stride is 1, the dilation factor is 2, the padding parameter is 2, the groups parameter is 48, and the 65th layer is normalized The parameter is 96; the size of the convolution kernel of the sixty-sixth convolutional layer is 1x1, the number of convolution kernels is 96, the stride is 1, the padding parameter is 0, and the groups parameter is 96, the standardization parameter of the sixty-sixth layer is 96. The activation method of all activation layers is "ReLU6", and the width of each feature map in RS4 and DS4 is

height is

高度均为

For the seventh fusion layer, the color map RGB input terminal of the seventh fusion layer receives all feature maps in R4 and all feature maps in RS4, and the seventh fusion layer uses the existing add (element-by-element addition) method. Fusing R4 and RS4 to get the set RA4, the color map RGB output of the seventh fusion layer outputs RA4; the depth map Depth input of the seventh fusion layer receives all feature maps in D4 and all feature maps in DS4, the seventh Each fusion layer fuses D4 and DS4 through the existing add (element-by-element addition) method to obtain a set DA4, and the depth map Depth output of the seventh fusion layer outputs DA4; among them, the total width of the feature maps contained in RA4 and DA4 The number is 96, and the width of each feature map in RA4 and DA4 is

height is

高度为

For the fourth block attention convolution block, it consists of the fourth block layer, the fourth channel attention layer, and the fourth spatial dimension attention layer set in sequence; the color of the fourth block attention convolution block Figure RGB input receives all feature maps of RA4, all feature maps in RA4 are input into the fourth block attention convolution block, first all input feature maps are divided into two parts by channel through the fourth block layer, one part As the input of the channel attention layer, the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied with the original input feature map respectively, and the multiplication results are added together. The fourth block attention The output of the convolution block outputs 96 feature maps, and the set of 96 feature maps is recorded as RT4; the depth map Depth input of the fourth block attention convolution block receives all the feature maps of DA4 in DA4. All feature maps of the input are input to the fourth block attention convolution block. First, all input feature maps are divided into two parts by channel through the fourth block layer, one part is used as the input of the channel attention layer, and the other part is used as the spatial dimension attention. The input of the force layer, and finally the results of the attention processing on both sides are multiplied by the original input feature maps and then the multiplication results are added together. The depth map Depth output of the fourth block attention convolution block outputs 96 feature maps. , the set composed of 96 feature maps is recorded as DT4; among them, the fourth block layer adopts the split function that comes with pytorch, and the parameters are half of the original input feature map and the number of corresponding channels; the fourth channel attention layer contains the fourth Adaptive max pooling, the first convolutional layer of the fourth channel attention, the second convolutional layer of the fourth channel attention, and the activation layer, where the fourth adaptive max pooling parameter is 1, and the fourth channel attention is the first The convolutional layer convolution kernel size is 1x1, the number of convolution kernels is 48, the stride is 1, the groups parameter is 48, and the bias term is False; the fourth channel attention is the second The convolutional layer convolution kernel size is 1x1, the number of convolution kernels is 96, the stride is 1, the groups parameter is 48, and the bias term is False; the fourth channel attention activation layer The activation method is "Sigmoid"; the fourth spatial size attention layer includes the fourth channel maximization layer, the fourth spatial dimension attention convolution layer and the activation function, of which the fourth channel maximization layer uses the max that comes with pytorch Function; the fourth spatial dimension of the attention convolution layer convolution kernel size is 3 × 3, the number of convolution kernels is 1, the expansion factor (dilation) is 2, the padding parameter is 2, and the groups parameter is 2. is 1, the bias term (bias) is False, and the activation mode of the fourth spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT4 and DT4 is

height is

高度均为

For the eighth fusion layer, the color map RGB input terminal of the eighth fusion layer receives all the feature maps in RA4 and DT4, and the eighth fusion layer fuses RA4 and DT4 through the existing add (element-by-element addition) method to obtain Set RF4, the color map RGB output terminal of the eighth fusion layer outputs RF4; the depth map Depth input terminal of the eighth fusion layer receives all feature maps in RT4 and all feature maps in DA4, and the eighth fusion layer Some add (element-by-element addition) methods fuse RT4 and DA4 to obtain a set DF4, and the depth map Depth output of the eighth fusion layer outputs DF4; among them, the total number of feature maps contained in RF4 and DF4 is 96, The width of each feature map in RF4 and DF4 is

height is

高度为

For the fifth feature re-extraction convolution block, it consists of the sixty-seventh convolutional layer, the sixty-seventh normalization layer, the sixty-seventh activation layer, the sixty-eighth convolutional layer, and the sixty-eighth convolutional layer. Normalization layer, 68th activation layer, 69th convolution layer, 69th normalization layer, 69th activation layer, 70th convolution layer, 70th normalization layer, 70th activation layer Composition; the color map RGB input end of the fifth feature extraction convolution block receives all the feature maps in R5, the color map RGB output end of the fifth feature extraction convolution block outputs 160 feature maps, and the 160 feature maps are The set of images is denoted as RS5; the depth map Depth input of the fifth feature re-extraction convolution block receives all feature maps in D5, and the fifth feature re-extraction depth map Depth output of the convolution block outputs 160 features Figure, the set composed of 160 feature maps is marked as DS5; among them, the size of the convolution kernel of the sixty-seventh convolution layer is 3×3, the number of convolution kernels is 160, the stride is 1, and the dilation is 1. The factor (dilation) is 1, the padding parameter is 1, the grouping (groups) parameter is 96, and the standardization parameter of the sixty-third layer is 160; the convolution kernel size of the sixty-eighth convolution layer is 1x1, the volume The number of product kernels is 80, the stride is 1, the padding parameter is 0, the groups parameter is 80, and the standardization parameter of the sixty-eighth layer is 80; the volume of the sixty-ninth convolutional layer The kernel size is 3×3, the number of convolution kernels is 160, the stride is 1, the dilation factor is 2, the padding parameter is 2, and the groups parameter is 80. The standardization parameter of the sixty-ninth layer is 160; the convolution kernel size of the seventieth convolution layer is 1x1, the number of convolution kernels is 160, the stride is 1, the padding parameter is 0, and the grouping ( groups) parameter is 160, and the seventieth layer normalization parameter is 160. The activation method of all activation layers is "ReLU6", and the width of each feature map in RS5 and DS5 is

height is

高度均为

For the ninth fusion layer, the color map RGB input of the ninth fusion layer receives all feature maps in R5 and all feature maps in RS5, and the ninth fusion layer uses the existing add (element-by-element addition) method Fusing R5 and RS5 to get the set RA5, the color map RGB output of the ninth fusion layer outputs RA5; the depth map Depth input of the ninth fusion layer receives all feature maps in D5 and all feature maps in DS5, the ninth Each fusion layer fuses D5 and DS5 through the existing add (element-by-element addition) method to obtain a set DA5, and the depth map Depth output of the ninth fusion layer outputs DA5; among them, the total width of the feature maps contained in RA5 and DA5 The number is 160, and the width of each feature map in RA5 and DA5 is

height is

高度为

For the fifth block attention convolution block, it consists of the fifth block layer, the fifth channel attention layer, and the fifth spatial dimension attention layer set in sequence; the color of the fifth block attention convolution block Figure RGB input receives all feature maps of RA5, all feature maps in RA5 are input into the fifth block attention convolution block, first all input feature maps are divided into two parts by channel through the fifth block layer, one part As the input of the channel attention layer, the other part is used as the input of the spatial size attention layer. Finally, the results of the attention processing on both sides are multiplied by the original input feature map respectively, and the multiplication results are added together. The fifth block attention The color map RGB output of the convolution block outputs 160 feature maps, and the set composed of them is recorded as RT5; the depth map Depth input of the fifth block attention convolution block receives all the feature maps of DA5 in DA5. All feature maps are input into the fifth block attention convolution block. First, all input feature maps are divided into two parts by channel through the fifth block layer, one part is used as the input of the channel attention layer, and the other part is used as the spatial dimension attention. The input of the force layer, and finally the results of the attention processing on both sides are multiplied by the original input feature maps and then the multiplication results are added together. The depth map Depth output of the fifth block attention convolution block outputs 160 feature maps. , the set composed of it is recorded as DT5. The fifth block layer adopts the split function that comes with pytorch, and the parameters are half of the original input feature map and the number of corresponding channels; the fifth channel attention layer contains the fifth adaptive maximum pool , the fifth channel pays attention to the first convolutional layer, the fifth channel pays attention to the second convolutional layer and the activation layer, where the fifth adaptive maximum pooling parameter is 1, and the fifth channel pays attention to the first convolutional layer volume The kernel size is 1x1, the number of convolution kernels is 80, the stride is 1, the groups parameter is 80, and the bias term is False; the fifth channel pays attention to the second convolutional layer volume The kernel size is 1x1, the number of convolution kernels is 160, the stride is 1, the groups parameter is 80, and the bias term is False; the activation method of the fifth channel attention activation layer is "Sigmoid"; the fifth spatial size attention layer includes the fifth channel maximization layer, the fifth spatial dimension attention convolution layer and the activation function, of which the fifth channel maximization layer uses the max function that comes with pytorch; the fifth The size of the convolution kernel of the attention convolution layer is 3 × 3, the number of convolution kernels is 1, the dilation factor (dilation) is 2, the padding parameter is 2, the grouping (groups) parameter is 1, and the partial Set the item (bias) to False, and the activation mode of the fifth spatial dimension attention activation layer is "Sigmoid". The width of each feature map in RT5 and DT5 is

height is

高度均为

For the tenth fusion layer, the color map RGB input terminal of the tenth fusion layer receives all the feature maps in RA5 and DT5, and the tenth fusion layer is obtained by fusing RA5 and DT5 through the existing add (element-by-element addition) method. Set RF5, the color map RGB output terminal of the 10th fusion layer outputs RF5; the depth map Depth input terminal of the 10th fusion layer receives all feature maps in RT5 and all feature maps in DA5, and the 10th fusion layer Some add (element-by-element addition) methods fuse RT5 and DA5 to obtain a set DF5, and the depth map Depth output of the 10th fusion layer outputs DF5; among them, the total number of feature maps contained in RF5 and DF5 is 160, The width of each feature map in RF5 and DF5 is

height is

高度为

For the first upsampling layer, the input of the first upsampling layer receives all feature maps in RF2 and all feature maps in DF2, and first fuses RF2 and DF2 through the existing add (element-by-element addition) method to obtain Set RD2, the first upsampling layer obtains the twice-enlarged RD2 feature map (height and width) through the UpsamlingBilinear2d function that comes with pytorch. The function parameter is 2, and the set formed by it is recorded as RD2x2. The output of the sampling layer outputs RD2x2; the total number of feature maps (channels) contained in RD2x2 is 32, and the width of each feature map in RD2x2 is

height is

高度为

For the second upsampling layer, the input of the second upsampling layer receives all feature maps in RF3 and all feature maps in DF3, and first fuses RF3 and DF3 through the existing add (element-by-element addition) method to obtain Set RD3, the second upsampling layer obtains the quadrupled RD3 feature map through the UpsamlingBilinear2d function that comes with pytorch, the function parameter is 4, the set formed by it is recorded as RD3x4, the output of the second upsampling layer output RD3x4; the total number of feature maps contained in RD3x4 is 64, and the width of each feature map in RD3x4 is

height is

高度为

For the third upsampling layer, the input end of the third upsampling layer receives all feature maps in RF4 and all feature maps in DF4, and first fuses RF4 and DF4 through the existing add (element-by-element addition) method to obtain Set RD4, the third upsampling layer obtains the quadrupled RD4 feature map through the UpsamlingBilinear2d function that comes with pytorch, the function parameter is 4, the set formed by it is recorded as RD4x4, the output of the third upsampling layer output RD4x4; the total number of feature maps contained in RD4x4 is 96, and the width of each feature map in RD4x4 is

height is

高度为

For the third upsampling layer, the input end of the third upsampling layer receives all feature maps in RF4 and all feature maps in DF4, and first fuses RF4 and DF4 through the existing add (element-by-element addition) method to obtain Set RD4, the third upsampling layer obtains the quadrupled RD4 feature map through the UpsamlingBilinear2d function that comes with pytorch, the function parameter is 4, the set formed by it is recorded as RD4x4, the output of the third upsampling layer output RD4x4; the total number of feature maps contained in RD4x4 is 96, and the width of each feature map in RD4x4 is

height is

高度为

For the fourth upsampling layer, the input end of the fourth upsampling layer receives all feature maps in RF5 and all feature maps in DF5, and first fuses RF5 and DF5 through the existing add (element-by-element addition) method to obtain Set RD5, the fourth upsampling layer obtains the quadrupled RD5 feature map through the UpsamlingBilinear2d function that comes with pytorch, the function parameter is 4, the set formed by it is recorded as RD5x4, the output of the fourth upsampling layer output RD5x4; the total number of feature maps contained in RD5x4 is 160, and the width of each feature map in RD5x4 is

height is

高度为

For the eleventh fusion layer, the input of the eleventh fusion layer receives all feature maps in RF1 and all feature maps in DF1, and fuses RF1 and DF1 through the existing add (element-by-element addition) method to obtain a set RD1, where the total number of feature maps contained in RD1 is 24, the width of each feature map in RD1 is , and the height is

height is

For the twelfth fusion layer, the input of the twelfth fusion layer receives all feature maps in RD1, RD2x2, RD3x4, RD4x4 and RD5x4, and the twelfth fusion layer connects RD1, RD2x2, RD3x4 through the existing concatenate method , RD4x4 and RD5x4 to obtain the set RDM, and the output terminal of the twelfth fusion layer outputs the RDM; among them, the total number of feature maps contained in the RDM is 376 (24+32+64+96+160), and each feature in the RDM The width of the graph is

For the output layer, it consists of connecting the seventy-first convolutional layer and the fifth upsampling layer in sequence, wherein the convolution kernel size of the seventy-first convolutional layer is 1×1, the number of convolution kernels is 41, The step size (stride) is 1, and the bias term (bias) is False; the fifth upsampling layer obtains the quadrupled feature map through the UpsamlingBilinear2d function that comes with pytorch, and the function parameter is 4; the input of the output layer receives For all feature maps in RDM, the output of the output layer outputs 41 semantic segmentation prediction maps corresponding to the original input image.

Step 1_3: Take each pair of original indoor scene RGB images and Depth images in the training set as the original input image, input them into the convolutional neural network for training, and obtain 41 semantic segmentations corresponding to each pair of original indoor scene images in the training set Prediction map, the set of 41 semantic segmentation prediction maps corresponding to {RGB ^q (i,j),Depth ^q (i,j)} is recorded as

与

之间的损失函数值记为

采用分类交叉熵(categorical crossentropy)获得。Step 1_4: Calculate the loss function value between the set of 41 semantic segmentation prediction images corresponding to each pair of original indoor scene images in the training set and the set of 41 one-hot encoded images processed from the corresponding real semantic segmentation images. ,Will

and

The loss function value between is denoted as

Obtained using categorical crossentropy.

Step 1_5: Repeat steps 1_3 and 1_4 for a total of V times to obtain a convolutional neural network classification training model, and obtain a total of Q×V loss function values; then find the loss with the smallest value from the Q×V loss function values. function value; then the weight vector and bias term corresponding to the loss function value with the smallest value are corresponding to the optimal weight vector and optimal bias term of the convolutional neural network classification training model, which are correspondingly recorded as W ^best and b ^best ; Wherein, V>1, in this embodiment, take V=500.

The specific steps of the test phase process are:

Step 2_1: Separate P pairs of color maps and depth maps from the original data set as a test set, where P is a positive integer, and in the present invention, P is 699, let {RGB ^p (i', j'), Depth ^p (i',j')} represents the indoor scene image to be semantically segmented in the test set, 1≤p≤P; among them, 1≤i'≤W', 1≤j'≤H', W' denotes {RGB ^p (i', j'), Depth ^p (i', j')} the width of the color map and depth map, H' represents {RGB ^p (i', j'), Depth ^p (i', j)'} color map and depth The height of the image, RGB ^p (i', j'), Depth ^p (i', j') represent the color image RGB in {RGB ^p (i', j'), Depth ^p (i', j')} respectively And the pixel value of the pixel whose depth map Depth coordinate position is (i, j).

其中，

表示

中坐标位置为(i',j')的像素点的像素值。Step 2_2: Input the color map RGB and depth map Depth in {RGB ^p (i',j'),Depth ^p (i',j')} into the trained improved fully convolutional neural network semantic segmentation model, And use the convolution kernel optimal weight W ^best and the optimal bias term b ^best obtained in the training to predict, and get the prediction corresponding to {RGB ^p (i',j'),Depth ^p (i',j')} Semantic segmentation of images, denoted as

in,

express

The pixel value of the pixel whose middle coordinate position is (i', j').

In order to further verify the effect of the present invention, the following is verified through specific experiments.

The present invention uses the structure of the improved full convolutional neural network semantic segmentation model built by the deep learning library Pytorch1.1.0 based on python3.6. The indoor scene image database NYUv2 test set is used to analyze the segmentation effect of the indoor scene images (take 699 pairs of indoor scene images) predicted by the method of the present invention. Here, three commonly used objective parameters for evaluating semantic segmentation methods are used as evaluation indicators, namely Class Acuracy, Mean Pixel Accuracy (MPA), and the ratio of the intersection and union of the segmented image and the label image (Mean Pixel Accuracy). Intersection over Union, MIoU) to evaluate the segmentation performance of predicted semantically segmented images. The higher the index value of the three types, the better the performance of the model in the present invention.

The method of the present invention is used to predict each pair of indoor scene images in the NYUv2 test set of the indoor scene image database, and the predicted semantic segmentation image corresponding to each pair of indoor images is obtained, which reflects the class accuracy CA and the average pixel accuracy of the semantic segmentation effect of the method of the present invention. The rate MPA, the ratio of the intersection and union of the segmented image and the label image, MIoU, are listed in Table 1. As can be seen from the data in Table 1, the method of the present invention has achieved good results in three categories of indicators: class accuracy CA, average pixel accuracy MPA, and ratio MIoU of the intersection and union of segmented images and label images. effectiveness of the method of the present invention.

Table 1. Evaluation results on the test set using the method of the present invention

CA 61.01% MPA 74.34% MIoU 47.92%

Figure 2a shows the first original indoor scene color RGB image of the same scene, Figure 2b shows the first original indoor scene depth Depth image of the same scene, and Figure 2c shows the method of the present invention. Predict with the original indoor scene image shown in 2b, and obtain the predicted semantic segmentation image; Figure 3a shows the second original indoor scene color RGB image of the same scene, and Figure 3b shows the second image of the same scene The original indoor scene depth Depth image, Figure 3c shows the predicted semantic segmentation image obtained by using the method of the present invention to predict the original indoor scene image shown in Figures 3a and 3b; Figure 4a shows the third image of the same scene. An original indoor scene color RGB image, Figure 4b shows the third original indoor scene depth Depth image of the same scene, Figure 4c shows the original indoor scene image shown in Figures 4a and 4b using the method of the present invention Make predictions and get the predicted semantic segmentation images; Figure 5a shows the 4th original indoor scene color RGB image of the same scene, Figure 5b shows the 4th original indoor scene depth Depth image of the same scene, Figure 5c The predicted semantic segmentation images obtained by using the method of the present invention to predict the original indoor scene images shown in Figures 5a and 5b are given. Comparing Fig. 2a, 2b and Fig. 2c, comparing Fig. 3a, 3b and Fig. 3c, comparing Fig. 4a, 4b and Fig. 4c, comparing Fig. 5a, 5b and Fig. 5c, it can be seen that the predicted semantic segmentation image obtained by the method of the present invention The segmentation accuracy is high.

Claims (9)

1. The indoor scene semantic segmentation method based on the improved full convolution neural network is characterized by comprising the following steps of:

step 1: selecting Q pairs of original indoor scene images and corresponding real semantic segmentation images, and forming a training set by all the original indoor scene images and the corresponding real semantic segmentation images; each pair of original indoor scene images comprises an original indoor scene color image and an original indoor scene depth image, and the real semantic segmentation images in the training set are processed into 41 independent thermal coding images by adopting an independent thermal coding technology;

step 2: constructing a convolutional neural network classification training model: the convolutional neural network classification training model comprises an input layer, a hidden layer and an output layer; the input layer comprises a color image input layer and a depth image input layer; the hidden layer comprises a color image processing module and a depth image processing module; the color image processing module and the depth image processing module are symmetrical in structure and respectively comprise five neural network blocks, five feature re-extraction volume blocks and ten fusion layers; the hidden layer also comprises five block attention volume blocks, four upper sampling layers and two fusion layers;

and step 3: inputting the training set into the convolutional neural network classification training model in the step 2 for training, in the training process, performing iterative training processing each time to obtain 41 semantic segmentation predicted images corresponding to each pair of original indoor scene images, and calculating a loss function value between a set formed by the 41 semantic segmentation predicted images and a set formed by 41 one-hot coded images corresponding to real semantic segmentation images;

and 4, step 4: repeating the step 3 for a total of V times to obtain Q multiplied by V loss function values; then finding out the minimum loss function value from the Q multiplied by V loss function values, and taking the weight vector and the bias item corresponding to the minimum loss function value as the optimal weight vector and the optimal bias item of the convolutional neural network classification training model, thereby completing the training of the convolutional neural network classification training model;

and 5: and performing prediction processing on an indoor scene image to be predicted by using the convolutional neural network classification training model obtained after training, outputting and obtaining a corresponding prediction semantic segmentation image, and realizing indoor scene image semantic segmentation.

2. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 1, wherein: the step 2) is specifically as follows:

the color image input layer and the depth image input layer are respectively input into a first neural network block in the color image processing module and the depth image processing module;

the color image processing module and the depth image processing module have the same structure, and specifically comprise:

one output of the first neural network block is input into a first fusion layer through a first feature re-extraction convolution block, and the other output of the first neural network block is input into a first fusion layer; one output of the second neural network block is input into a third fusion layer through a second feature re-extraction convolution block, and the other output of the second neural network block is input into the third fusion layer; one output of the third neural network block is input into a fifth fusion layer through a third feature re-extraction convolution block, and the other output of the third neural network block is input into the fifth fusion layer; one output of the fourth neural network block is input into the seventh fusion layer through the fourth feature re-extraction convolution block, and the other output of the fourth neural network block is input into the seventh fusion layer; one output of the fifth neural network block is input into a ninth fusion layer through a fifth feature re-extraction convolution block, and the other output of the fifth neural network block is input into the ninth fusion layer; the two inputs of each fusion layer are fused in an element-by-element addition mode;

the output of the first fusion layer is respectively input into a first block attention volume block and a corresponding second fusion layer, the output of the third fusion layer is respectively input into a second block attention volume block and a corresponding fourth fusion layer, the output of the fifth fusion layer is respectively input into a third block attention volume block and a corresponding sixth fusion layer, the output of the seventh fusion layer is respectively input into a fourth block attention volume block and a corresponding eighth fusion layer, and the output of the ninth fusion layer is respectively input into a fifth block attention volume block and a corresponding tenth fusion layer;

two outputs of the first block attention volume block are respectively input into a second fusion layer of the color image processing module and a second fusion layer of the depth image processing module, two outputs of the second block attention volume block are respectively input into a fourth fusion layer of the color image processing module and a fourth fusion layer of the depth image processing module, two outputs of the third block attention volume block are respectively input into a sixth fusion layer of the color image processing module and a sixth fusion layer of the depth image processing module, two outputs of the fourth block attention volume block are respectively input into an eighth fusion layer of the color image processing module and a eighth fusion layer of the depth image processing module, and two outputs of the fifth block attention volume block are respectively input into a tenth fusion layer of the color image processing module and the tenth fusion layer of the depth image processing module;

the two inputs of the second fusion layer are fused in an element-by-element addition mode and then respectively input into an eleventh fusion layer and a corresponding second neural network block, the two inputs of the fourth fusion layer are fused in an element-by-element addition mode and then respectively input into a first up-sampling layer and a corresponding third neural network block, the two inputs of the sixth fusion layer are fused in an element-by-element addition mode and then respectively input into a second up-sampling layer and a corresponding fourth neural network block, and the two inputs of the eighth fusion layer are fused in an element-by-element addition mode and then respectively input into a third up-sampling layer and a corresponding fifth neural network block; the output of the tenth fusion layer is input into the fourth upsampling layer;

the two inputs of the eleventh fusion layer, the first up-sampling layer, the second up-sampling layer, the third up-sampling layer and the fourth up-sampling layer are fused in an element-by-element addition mode and then are all input into the twelfth fusion layer;

and all the inputs of the twelfth fusion layer are connected in a concatenate mode and then output through the output layer, and the output layer mainly comprises a convolution layer and a fifth upper sampling layer which are sequentially connected.

3. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 1, wherein: the five neural network blocks adopt a MobileNet V2 network structure, the first neural network block adopts 1-4 layers in MobileNet V2, the second neural network block adopts 5-7 layers in MobileNet V2, the third neural network block adopts 8-11 layers in MobileNet V2, the fourth neural network block adopts 12-14 layers in MobileNet V2, and the fifth neural network block adopts 15-17 layers in MobileNet V2.

4. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 2, wherein: each feature re-extraction convolution block consists of four re-extraction modules which are connected in sequence, and each re-extraction module comprises a convolution layer, a normalization layer and an activation layer which are connected in sequence;

the activation mode of all the activation layers adopts ReLU 6; the step length of all the convolution layers is 1; the number of convolution kernels of the convolution layer in each re-extraction module is the same as the standardized parameters of the standardized layer;

the convolution kernels of the convolution layers in the first re-extraction module, the third re-extraction module and the fourth re-extraction module are the same in size, and the convolution kernel of the convolution layer in the second re-extraction module is half of the convolution kernel of the convolution layer in the first re-extraction module;

the convolution kernels of the convolution layers in the first re-extraction module and the third re-extraction module are both 3x 3, and the convolution kernels of the convolution layers in the second re-extraction module and the fourth re-extraction module are both 1x 1; the expansion factor of the convolution layer in the first re-extraction module is 1, and the expansion factor of the convolution layer in the third re-extraction module is 2; the zero padding parameter of the convolution layer in the first re-extraction module is 1, the zero padding parameter of the convolution layer in the third re-extraction module is 2, and the zero padding parameters of the convolution layers in the second re-extraction module and the fourth re-extraction module are both 0.

5. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 4, wherein: the convolution kernel size of the convolution layer of the first re-extraction module in the first feature re-extraction volume block is 24, the convolution kernel size of the convolution layer of the first re-extraction module in the second feature re-extraction volume block is 32, the convolution kernel size of the convolution layer of the first re-extraction module in the third feature re-extraction volume block is 64, the convolution kernel size of the convolution layer of the first re-extraction module in the fourth feature re-extraction volume block is 96, and the convolution kernel size of the convolution layer of the first re-extraction module in the fifth feature re-extraction volume block is 160;

grouping parameters of four sequentially arranged convolutional layers in the first feature re-extraction convolutional block are respectively 24, 12, 24 and 24, grouping parameters of four sequentially arranged convolutional layers in the second feature re-extraction convolutional block are respectively 32, 16 and 32, grouping parameters of four sequentially arranged convolutional layers in the third feature re-extraction convolutional block are respectively 64, 32 and 64, grouping parameters of four sequentially arranged convolutional layers in the fourth feature re-extraction convolutional block are respectively 64, 48 and 99, and grouping parameters of four sequentially arranged convolutional layers in the fifth feature re-extraction convolutional block are respectively 96, 80 and 160.

6. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 2, wherein: each of the block attention convolution blocks comprises a block layer, a channel attention layer and a space size attention layer, wherein the input of the block attention convolution block is input into the channel attention layer and the space size attention layer after passing through the block layer, the input of the block attention convolution block is multiplied by the output of the channel attention layer and the output of the space size attention layer, and the multiplied results are added to be used as the output of the block attention convolution block.

7. The method for indoor scene semantic segmentation based on the improved full convolution neural network as claimed in claim 6, wherein: the blocking layer adopts a split function carried by the pytorech, and the parameter is half of the number of channels corresponding to the input characteristic diagram of the blocking attention volume block;

the channel attention layer comprises an adaptive maximum pooling layer, a channel attention first convolution layer, a channel attention second convolution layer and a channel attention activation layer which are sequentially connected; the maximum pooling parameter of the adaptive maximum pooling layer is 1; the convolution kernels of the channel attention first convolution layer and the channel attention second convolution layer are both 1x1 in size, the step length is 1, the bias terms are False, the grouping parameters are the same, and the number of convolution kernels in the channel attention first convolution layer is half of that of the channel attention second convolution layer;

the number of convolution kernels of the channel attention first convolution layer of the channel attention layer in the first block attention convolution block is 12, and the grouping parameter is 12; the number of convolution kernels of the channel attention first convolution layer of the channel attention layer in the second block attention convolution block is 16, and the grouping parameter is 16; the number of convolution kernels of the first convolution layer of the channel attention layer in the third block attention convolution block is 32, and the grouping parameter is 32; the number of convolution kernels of the first convolution layer of the channel attention layer in the fourth block attention convolution block is 48, and the grouping parameter is 48; the number of convolution kernels of the first convolution layer of the channel attention layer in the fifth block attention convolution block is 80, and the grouping parameter is 80;

the spatial dimension attention layer comprises a per-channel maximization layer, a spatial dimension attention convolution layer and a spatial dimension attention activation layer which are sequentially connected; adopting a max function carried by the pytorch according to a channel maximization layer; the convolution kernel size of the spatial dimension attention convolution layer is 3 multiplied by 3, the number of the convolution kernels is 1, the expansion factor is 2, the zero padding parameter is 2, the grouping parameter is 1, and the bias term is False;

the channel attention layer and the space size attention layer adopt an activation mode which is a Sigmoid function.

8. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 2, wherein: the upsampling layer adopts an upsampling Biliner 2d function carried in the pytorech, the function parameter of the first upsampling layer is 2, and the function parameters of the second upsampling layer, the third upsampling layer, the fourth upsampling layer and the fifth upsampling layer are 4; the convolution kernel size of the convolution layer in the output layer is 1 multiplied by 1, the number of the convolution kernels is 41, the step length is 1, and the bias term is False.

9. The indoor scene semantic segmentation method based on the improved full convolution neural network as claimed in claim 4, wherein: the input end of the color image input layer receives an indoor scene color image, the input end of the depth image input layer receives an indoor scene depth image, and the output of the output layer is 41 semantic segmentation predicted images corresponding to the indoor scene image input by the input layer.

Images