CN116704174A - RGB-D image salient object detection method based on deep learning - Google Patents

Abstract

The invention discloses a deep learning-based RGB-D image salient target detection method, comprising the following steps: S1, constructing an encoder, obtaining multi-level features, specifically including RGB branch and depth branch feature extraction and building an interactive attention module; S2. Construct a decoder module, specifically including constructing a RGB branch and a depth branch cross-level feature fusion module and constructing a cross-modal feature fusion module; S3. Based on an encoder and a decoder, construct a RGB-D image based on deep learning. A target detection model; S4. Train the established model and save the parameters. The present invention improves the detection ability of the model by comprehensively exploring cross-modal feature fusion.

Description

A salient target detection method for RGB-D images based on deep learning

technical field

The invention relates to the technical field of computer vision, in particular to a method for detecting salient objects in RGB-D images based on deep learning.

Background technique

RGB-D images include RGB images and depth images. RGB images provide people with rich appearance and color information, while depth images provide additional spatial cues. Each pixel value of the depth image represents the actual distance between the sensor and the object, and there is generally a one-to-one correspondence between the pixels of the RGB image and the depth image.

Image salient object detection is to imitate the human visual system to detect salient objects or regions. Previous work on salient object detection mainly deals with RGB images. Although algorithms for image salient object detection are being continuously improved and innovated to achieve a processing mechanism comparable to that of the human visual system, there are still many problems in dealing with image salient object detection tasks in complex scenes, such as when the object and the background The colors are similar, and the target is smaller than the background. It is difficult to accurately detect the salient target only by relying on the RGB image. With the development of three-dimensional perception and sensing technology, people can not only obtain the shape and color information of objects, but also obtain the spatial position information of objects, and further improve the ability to perceive the scene. Depth information is a supplement to RGB image information, which is of great significance to image salient target detection, and can effectively improve the accuracy of detection and recognition results. For the same scene, different data sources can provide additional information of different modalities, making the expression of the scene richer and more comprehensive. Therefore, comprehensively considering the fusion of RGB image information and depth image information can obtain better salient object detection results.

According to different feature extraction strategies, salient object detection in RGB-D images can be roughly divided into research on salient object detection in RGB-D images based on traditional models and research on salient object detection in RGB-D images based on deep learning models. Traditional RGB-D image salient target detection methods rely on prior knowledge to set target features, and mainly extract manual features. Although good results have been achieved, it relies heavily on the designer's prior knowledge and requires manual adjustments in different situations, making its generalization ability poor in complex scenes. Considering the shortcomings of traditional methods based on manually designed features, more and more researchers have begun to apply neural networks to the detection of salient objects in RGB-D images.

Contents of the invention

The purpose of the present invention is to address the deficiencies in the prior art and propose a method for detecting salient objects in RGB-D images based on deep learning.

To achieve the above object, the technical solution of the present invention is:

A method for detecting salient objects in RGB-D images based on deep learning, comprising the following steps:

S1. Obtain an image data set, and preprocess the image data set;

S2. Construct and train a deep learning-based RGB-D image salient target detection model, the deep learning-based RGB-D image salient target detection model including an encoder and a decoder,

The encoder includes an RGB encoder and a depth encoder, and the end interaction attention module of the RGB encoder and the depth encoder;

The decoder includes a cross-level feature fusion module, a cross-modal fusion module, and a convolution layer with a convolution kernel size of 3×3;

S3. Perform salient target detection through the trained deep learning-based RGB-D image salient target detection model

S3-1. Extract the corresponding hierarchical encoder features through the RGB encoder and the depth encoder respectively. The multi-level encoder features of the RGB image and the depth image are expressed as and /> Among them, r represents the RGB image, d represents the depth image, and i represents the level of the feature;

S3-2. Strengthen the relationship between the multi-level encoder features of the RGB image and the depth image through the interactive attention module in the encoder, and obtain the fusion feature

S3-3. Merge features As the input of the cross-level feature fusion module, the RGB branch and depth branch decoding features are obtained;

S3-4. Input the RGB branch and depth branch decoding features of each layer into the cross-modal feature fusion module of the corresponding level to obtain f _i ^rgbd ; finally, output f ₁ ^rgbd from the last layer of the decoder through the convolution kernel size The final saliency prediction map S ^rgbd is obtained for the 3×3 convolutional layer, and the corresponding saliency maps predicted by the RGB branch and the depth branch, S ^r and S ^d , can also be obtained.

Preferably, in the step S1, the preprocessing method of the image data set is: performing data expansion through random flipping, rotation and multi-scale input.

Preferably, the multi-scale input method randomly adjusts the image size to 128×128, 256×256 and 352×352.

Preferably, the RGB encoder and depth encoder use a ResNet50 backbone network to extract multi-level encoder features of RGB images and depth images.

Preferably, the interactive attention module includes a self-attention mechanism, a cross-attention layer and a feed-forward network.

As a preference, in the step S2, the method of training the RGB-D image salient object detection model based on deep learning is: initialize the network through the pre-training parameters on ImageNet, and use the Adam algorithm to optimize, the batch size is 8, The initial learning rate is 1e-4, and the learning rate is adjusted every 40 rounds, with a total of 150 rounds of training.

When training the network, the binary cross-entropy loss function is used for optimization, and the final loss function is defined as:

Loss＝l _bce (S ^r ,G)+l _bce (S ^d ,G)+l _bce (S ^rgbd ,G)

Wherein, G is a truth graph, l _bce =-[Glog(S)+(1-G)log(1-S)].

Preferably, the fusion feature is obtained through the interactive attention module The method is as follows: the input fifth-layer RGB encoder features and depth encoder features are divided into several 1×1×128 tokens; secondly, the self-attention mechanism is applied to the RGB encoder features for feature self-enhancement; in An inter-feature cross-attention layer is introduced between the self-enhanced RGB encoder features and the associated deep encoder features; a feed-forward network is used to obtain the fused features/> The expression is as follows:

Among them, SA is a self-attention mechanism within a feature, CA is a cross-attention mechanism between features, and FF is a feed-forward network.

Preferably, in the cross-level feature fusion module, two convolutional layers are used to process the corresponding level encoder features and the output of the previous level of cascade fusion, so as to obtain decoder features, and the cross-level feature fusion The module includes RGB branch and depth branch, so as to obtain RGB branch and depth branch decoding features respectively.

Preferably, in the cross-modal fusion module, a convolutional layer is used to fuse the RGB decoder features and corresponding encoder features to obtain fused RGB features f _i ^r and depth decoder features f _i ^d ; then Connect the output of the cross-modal fusion module at the previous level with the fused RGB features and the fused depth features to obtain f _i ^r ′ and f _i ^d ′; for the fifth-level cross-modal fusion module, The output after refining with the interactive attention module feature to replace the output of the previous level; finally, the channel attention mechanism is used to enhance the fused f _i ^r ′ and f _i ^d ′ to obtain f _i ^rgbd , as follows:

Among them, CA represents the channel attention mechanism, [·,·] represents the channel connection.

The present invention has following characteristics and beneficial effect:

(1) Add the interactive attention module in the encoder stage, use the RGB feature itself and the depth feature to strengthen the RGB feature, and then input it to the subsequent decoder module, realizing the effective fusion of cross-modal information and to a certain extent Reduce the amount of calculation.

(2) In the decoder stage, a dual-branch structure is adopted to decode the RGB encoder features and depth information encoder features separately, and input the decoded information into the cross-modal fusion module of the corresponding level to explore more comprehensively and deeply. cross-modal fusion. A channel attention mechanism is also introduced in the cross-modal fusion module to enhance the fused features.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained according to these drawings without any creative effort.

Fig. 1 is the overall frame diagram of the network of the embodiment of the present invention;

Fig. 2 is a structural diagram of an interactive attention module in an embodiment of the present invention.

Fig. 3 is a structural diagram of a cross-modal fusion module in an embodiment of the present invention.

Fig. 4 is a result graph of an embodiment of the present invention.

Detailed ways

The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

The present invention provides a RGB-D image salient target detection method based on deep learning, as shown in Figure 1, comprising the following steps:

S1. Build an encoder to obtain multi-level features;

S1-1, RGB branch and depth branch feature extraction. The ResNet50 backbone network is used to extract multi-level features of RGB images and depth images, which are expressed as and />

S1-2. Build an interactive attention module. As shown in Fig. 2, first, the input fifth-layer RGB encoder features and depth encoder features are divided into several 1×1×128 tokens. Second, a self-attention mechanism is applied to the RGB encoder features to fully mine the relationships within the RGB encoder features. An inter-feature cross-attention layer is introduced between the self-enhanced RGB encoder features and the associated deep encoder features to further explore the relationship between different modality features. Finally, a feed-forward layer is employed. The whole process is described as follows:

Among them, SA is a self-attention mechanism within a feature, CA is a cross-attention mechanism between features, and FF is a feed-forward network. Both the self-attention mechanism and the cross-attention mechanism use efficient attention to reduce memory and computational costs. After each attention mechanism there is a residual connection and normalization operation. In the intra-feature self-attention mechanism, input RGB encoder features as query, key and value, and output self-augmented RGB encoder features. In the inter-feature cross-attention mechanism, the self-enhanced RGB encoder features are input as queries, and the depth encoder features are used as keys and values. Meanwhile, a positional encoding is introduced to avoid learning the positional errors between attention weight tokens.

By stacking L=2 of the above architectures, the final RGB encoder features not only strengthen intra-feature relationships, but also further achieve refinement with deep encoder features.

S2, building a decoder module;

S2-1. Construct the cross-level feature fusion module of the RGB branch and the depth branch, and use two convolutional layers to process the corresponding level encoder features and the output of the previous level of cascade fusion, so as to obtain the decoder features. Build a cross-level feature fusion module.

S2-2. Construct a cross-modal feature fusion module. As shown in Figure 3, first, a convolutional layer is used to fuse the RGB decoder features and the corresponding encoder features to obtain the fused RGB features f _i ^r , and the depth decoder features f _i ^d are the same. Then the output of the cross-modal fusion module at the previous level is concatenated with the fused RGB features and the fused depth features respectively to obtain f _i ^r ′ and f _i ^d ′. For the cross-modal fusion module of the fifth level, due to the lack of the output of the previous level, the present invention will use the interactive attention module to refine the output feature to replace the output of the previous layer. Finally, the channel attention mechanism is used to enhance the fused f _i ^r ′ and f _i ^d ′ to obtain f _i ^rgbd . details as follows:

Among them, CA represents the channel attention mechanism, [·,·] represents the channel connection.

S3. Based on the encoder and decoder, construct a RGB-D image salient target detection model based on deep learning. Extract RGB features hierarchically from the input image according to the encoder and deep features /> Get fusion features through interactive attention module/> Input to the decoder together; the decoder obtains the RGB branch and depth branch decoding features through the cross-level feature fusion module in S2-1, and then inputs the decoder features of each layer to the cross-modal features mentioned in S2-2 In the fusion module, f _i ^rgbd is obtained; finally, the output f ₁ ^rgbd of the last layer of the decoder is passed through a convolutional layer with a convolution kernel size of 3×3 to obtain the final saliency prediction map S ^rgbd . Corresponding saliency maps, S ^r and S ^d , are also obtained for RGB branch and depth branch predictions. When training the network, a binary cross-entropy loss function (BCE) is used to simultaneously optimize the RGB branch, the depth branch and the RGB-D branch. The final loss function is defined as:

Loss＝l _bce (S ^r ,G)+l _bce (S ^d ,G)+l _bce (S ^rgbd ,G)

Wherein, G is a truth graph, l _bce =-[Glog(S)+(1-G)log(1-S)].

S4. Train the established model and save the parameters. The present invention is realized based on Pytorch, and the GPU of 2080Ti model is used for training. Data augmentation is performed by random flipping, rotation, and multi-scale input. The multi-scale input randomly adjusts the image size to 128×128, 256×256 and 352×352. During the training process, use ResNet50 as the backbone network of the encoder stage, initialize the network through the pre-training parameters on ImageNet, and use the Adam algorithm for optimization, the batch size is 8, the initial learning rate is 1e-4, and every 40 rounds The learning rate was adjusted and a total of 150 epochs were trained.

S5. Input the image data to be detected into the trained RGB-D image salient object detection model based on deep learning, so as to output the final saliency prediction map S ^rgbd of the image data to be detected.

Fig. 4 is a comparison chart of the results of the method of the present invention, the first column is the RGB image, the second column is the depth image, the third column is the truth map, and the fourth column is the result map of the method of the present invention. It can be seen from the comparison that the final saliency prediction map S ^rgbd finally obtained by the scheme provided in this embodiment is closest to the truth map up to the third column.

The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, without departing from the principle and spirit of the present invention, various changes, modifications, substitutions and modifications to these embodiments still fall within the protection scope of the present invention.

Claims (9)

1. a RGB-D image salient target detection method based on depth learning, is characterized in that, comprises the steps: S1. Obtain an image data set, and preprocess the image data set; S2. Construct and train a deep learning-based RGB-D image salient target detection model, the deep learning-based RGB-D image salient target detection model including an encoder and a decoder, The encoder includes an RGB encoder and a depth encoder, and the end interaction attention module of the RGB encoder and the depth encoder; The decoder includes a cross-level feature fusion module, a cross-modal fusion module, and a convolution layer with a convolution kernel size of 3×3; S3. Perform salient target detection through the trained deep learning-based RGB-D image salient target detection model S3-1. Extract the corresponding hierarchical encoder features through the RGB encoder and the depth encoder respectively. The multi-level encoder features of the RGB image and the depth image are expressed as and /> Among them, r represents the RGB image, d represents the depth image, and i represents the level of the feature; S3-2. Strengthen the relationship between the multi-level encoder features of the RGB image and the depth image through the interactive attention module in the encoder, and obtain the fusion feature S3-3. Merge features As the input of the cross-level feature fusion module, the RGB branch and depth branch decoding features are obtained; S3-4. Input the decoding features of the RGB branch and the depth branch of each layer into the cross-modal feature fusion module of the corresponding layer, and obtain Finally, output the last layer of the decoder /> The final saliency prediction map S ^rgbd is obtained through the convolution layer with a convolution kernel size of 3×3, and the corresponding saliency maps predicted by the RGB branch and the depth branch, S ^r and S ^d , can also be obtained. 2. a kind of RGB-D image salient object detection method based on deep learning according to claim 1, is characterized in that, in described step S1, the pretreatment method of image data set is: through random flipping, rotation and multiple Scale input method for data augmentation. 3. A method for detecting salient objects in RGB-D images based on deep learning according to claim 2, wherein the multi-scale input method randomly adjusts the image size to 128×128, 256×256 and 352× 352. 4. a kind of RGB-D image salient target detection method based on deep learning according to claim 1, is characterized in that, described RGB coder and depth coder adopt ResNet50 backbone network to extract the multiplicity of RGB image and depth image Hierarchical encoder features. 5. A deep learning-based RGB-D image salient object detection method according to claim 1, wherein the interactive attention module includes a self-attention mechanism, a cross-attention layer and a feed-forward network. 6. a kind of RGB-D image salient target detection method based on deep learning according to claim 1, is characterized in that, in described step S2, the method for training the RGB-D image salient target detection model based on deep learning is : The network is initialized by pre-training parameters on ImageNet and optimized using the Adam algorithm. The batch size is 8, the initial learning rate is 1e-4, and the learning rate is adjusted every 40 rounds. A total of 150 rounds of training are performed. When training the network, the binary cross-entropy loss function is used for optimization, and the final loss function is defined as: Loss＝l _bce (S ^r ,G)+l _bce (S ^d ,G)+l _bce (S ^rgbd ,G) Wherein, G is a truth graph, l _bce =-[Glog(S)+(1-G)log(1-S)]. 7. A kind of RGB-D image salient target detection method based on deep learning according to claim 5, is characterized in that, obtain fusion feature by described interactive attention module The method is as follows: the input fifth-layer RGB encoder features and depth encoder features are divided into several 1×1×128 tokens; secondly, the self-attention mechanism is applied to the RGB encoder features for feature self-enhancement; in An inter-feature cross-attention layer is introduced between the self-enhanced RGB encoder features and the associated deep encoder features; a feed-forward network is used to obtain the fused features/> The expression is as follows: Among them, SA is a self-attention mechanism within a feature, CA is a cross-attention mechanism between features, and FF is a feed-forward network. 8. A kind of RGB-D image salient target detection method based on deep learning according to claim 4, is characterized in that, in described cross-level feature fusion module, adopts two convolutional layers, to the cascade fusion Corresponding layer encoder features and previous layer output are processed to obtain decoder features, and the cross-level feature fusion module includes RGB branch and depth branch, thereby obtaining RGB branch and depth branch decoding features respectively. 9. A kind of RGB-D image salient target detection method based on deep learning according to claim 8, is characterized in that, in described cross-modal fusion module, utilizes a convolutional layer to combine RGB decoder feature and corresponding The encoder features are fused to obtain the fused RGB features f _i ^r and the depth decoder features f _i ^d ; then the output of the cross-modal fusion module at the previous level is connected to the fused RGB features and the fused depth features respectively Together, f _i ^r′ and f _i ^d′ are obtained; for the fifth-level cross-modal fusion module, the output after refinement with the interactive attention module feature to replace the output of the previous level; finally, the channel attention mechanism is used to enhance the fused f _i ^r′ and f _i ^d′ to obtain f _i ^rgbd , as follows: f _i ^rgbd ＝CA(conv([f _i ^r′ ,f _i ^d′ ])) Among them, CA represents the channel attention mechanism, [·,·] represents the channel connection.