CN116452937A - Multimodal Feature Object Detection Method Based on Dynamic Convolution and Attention Mechanism - Google Patents

Abstract

The invention relates to a multi-mode characteristic target detection method based on a dynamic convolution and attention mechanism, and belongs to the field of image recognition. In the method, two data streams are provided at the beginning stage of a backlight of YOLOv5, respectively input visible light images and infrared light images, and a dynamic convolution module ODConv, a multispectral convolution attention feature fusion module MS-CBAM and a residual error network are used for carrying out feature extraction operation. The invention has the advantages that the characteristics of the visible light image and the infrared image are combined, and the target detection precision of multi-mode and small targets is greatly improved by combining various attention mechanisms and architectures, so that the problem of weak target detection performance in a dim environment is solved. Compared with other multi-mode fusion target detection, the method has the advantages of high training speed and low hardware resource consumption.

Description

Multimodal Feature Object Detection Method Based on Dynamic Convolution and Attention Mechanism

technical field

The invention belongs to the field of image recognition, and relates to a multimodal feature target detection method based on a dynamic convolution and attention mechanism.

Background technique

Object detection is a very important technology in computer vision tasks, and its performance directly affects the detection accuracy and computing efficiency of related tasks. Therefore, this field has been concerned by academia, industry and other aspects. The object detection discussed in this invention aims to improve the overall network performance by utilizing new modality data and new modality fusion methods. For example, at night, the traffic system is likely to face insufficient light sources for surveillance video, and it is difficult to realize functions such as taking pictures of most violations, monitoring pedestrians and vehicles, and automatically alerting traffic accidents from a single spectral data source. The infrared image captured by the infrared camera enhances the visible light image features of objects such as vehicles and pedestrians at night, which can greatly improve the detection accuracy of night targets. Therefore, how to use a large amount of multispectral image data to improve the performance of target recognition and detection models is a task of great research value and challenge. The multimodal feature fusion dual-stream neural network integrates the information of these two different modalities into the deep learning neural network, which greatly improves the training accuracy and accuracy of the above-mentioned problems in the field of target detection. However, the convolutional receptive field of the existing CNN can only perform information fusion in a local area, and the two-stream convolutional neural network cannot make good use of the complementarity between different modalities. Simply superimposing feature maps will increase the learning of the neural network. Difficulty, exacerbating the modal imbalance, leading to performance degradation. The present invention transforms the existing YOLOv5 neural network model, and introduces improved channel attention, spatial attention and dynamic convolution to form a mode fusion module, so that it can more fully understand the above two modes under multiple attention Perform cross-modal fusion, learning and prediction. Meanwhile, the NWD localization loss function is used to enhance the small object detection accuracy.

Contents of the invention

In view of this, the object of the present invention is to provide a multimodal feature target detection method based on a dynamic convolution and attention mechanism.

To achieve the above object, the present invention provides the following technical solutions:

A multimodal feature target detection method based on dynamic convolution and attention mechanism, the method includes the following steps:

S1: Establish a neural network model based on YOLOv5's dual-stream convolutional detection network, in which Backbone uses convolution operations and feature fusion modules for modality fusion and feature learning;

S2: Use channel attention and spatial attention to form a multispectral module MS-CBAM, use channel attention to weight the feature maps of visible light and infrared light images, and then stack infrared and visible light images together and use spatial attention to pair The feature map is used for feature weighting, and then the residual network is used to refine the features;

S3: Introduce a multi-head attention mechanism to the convolution structure, and establish a dynamic convolution ODConv module by assigning different attention coefficient matrices to the four dimensions of input channel dimension, output channel dimension, spatial dimension and convolution kernel;

S4: Set the MS-CBAM module to output as the larger position of the 80×80×256 feature map, and the ODConv module to output as the medium and small position of the feature map of 40×40×512 and 20×20×1024; output Three feature maps of different sizes enter the Neck layer, which is the feature pyramid, for feature extraction, predict the output features, and output the prediction results;

S5: In the training phase, the visible light and infrared light data enter the dual-stream neural network training after specific Mosaic data enhancement, adaptive anchor frame calculation, and adaptive image scaling process; YOLO v5l pre-trained weights are used for initialization, and random gradients are used Descent algorithm to learn the parameters of the network;

In the prediction stage, use the softmax classifier to obtain the final classification probability of the category;

In the optimization stage, the joint optimization of positioning loss, classification loss, and confidence loss is used to reduce the error between the real value and the predicted value, and NWD is introduced into the positioning loss to improve the accuracy of small target detection; repeat S5 until iteration When the number of iterations reaches the set number of iterations, the model training is completed and the target detection task is performed.

Optionally, in said S1, the input of the YOLO v5-based dual-stream convolutional target detection network framework is image pairs of different modalities, Backbone is a dual-stream convolutional network, and the dual-stream neural network model includes Backbone, Neck, and a prediction layer;

Let the input visible light feature map be X _V , the input infrared light feature map be X _T , and the length, width, and number of channels of the feature map be H, W, and C respectively;

The feature extraction network structure uses three feature fusion modules and the residual network to form three feature extraction loops and a thinning structure. The i-th feature fusion calculation process is expressed as:

Among them, σ is the feature fusion function, the visible light image input feature map is X _V , the infrared light image input feature map is X _T , and F is the feature fusion module, which performs batch normalization operation; fusion feature map The length, width, and number of channels are H, W, and 2C respectively; after that, the residual network will be constructed by fusing features and original features:

Obtain new feature maps f _t ⁱ and

Optionally, in said S2, for visible light and infrared light input images, the channel attention mechanism calculation is performed on the two, and then the feature map is superimposed according to the channel dimension superimposition method, and then input to the spatial attention power to operate;

The calculation of the MS-CBAM module is expressed as:

X＝M _S [concat[M _C (X _V ),M _C (X _T )]]

Among them, M _C represents the channel attention mechanism, M _S represents the spatial attention mechanism; Concat represents the stacking of feature maps in the channel dimension;

Afterwards, the residual network of X is refined, and the process is expressed as:

_X'V ＝ _XV +X

X' _T = X _T + X

The finally obtained feature maps are X' _V ∈ ^{V B×C×H×W} , X' _T ∈ T ^B×C×H×W , representing the final output of the MS-CBAM module.

Optionally, in the S3, a multi-head self-attention mechanism is introduced in the convolution process, and different attention coefficient matrices ODConv are given to convolution in the four dimensions of input channel dimension, output channel dimension, spatial dimension and convolution kernel. , to improve the ability of feature extraction; the overall operation of the ODConv module is expressed as:

X'＝ODConv(concat(X _V ,X _T ))

Among them, X _V and X _T are the feature map inputs of visible light and infrared light modes respectively, concat means that the two inputs are superimposed along the channel number dimension, and ODConv means dynamic convolution operation;

Its dynamic convolution formula that combines four dimensions is expressed as:

y＝(α _w1 ⊙α _f1 ⊙α _c1 ⊙α _s1 ⊙W ₁ +...+α _wn ⊙α _fn ⊙α _cn ⊙α _sn ⊙W _n )*x

is the attention coefficient matrix of the convolution kernel dimension W _i , /> and /> Respectively represent the dynamic convolution attention coefficient matrix along the spatial dimension, input channel dimension, and output channel dimension of the convolution kernel W _i , and ⊙ represents the multiplication along different dimensions of the kernel space.

Optionally, the input and output of the MS-CBAM module and the ODConv module are both visible light and infrared light feature maps, and the output will form a residual network with the input;

The loss functions of the positioning loss, classification loss, and confidence loss are expressed as:

L _total ＝L _box +L _cls +L _conf

Among them, the positioning loss uses the NWD loss function; the NWD loss function introduces the NormalizedWasserstein Distance calculation method, and calculates the similarity through the corresponding Gaussian distribution.

The beneficial effect of the present invention is that: the present invention can well optimize target detection under the condition of insufficient brightness of the whole or part of the image, and has more advantages in terms of prediction accuracy and reliability when applied to a target detection system.

Other advantages, objects and features of the present invention will be set forth in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the investigation and research below, or can be obtained from It is taught in the practice of the present invention. The objects and other advantages of the invention may be realized and attained by the following specification.

Description of drawings

In order to make the purpose of the present invention, technical solutions and advantages clearer, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flow chart of the overall architecture of the present invention;

Fig. 2 is Backbone flowchart of the present invention;

Fig. 3 is a structural diagram of a dynamic convolution feature fusion module;

Figure 4 is a structural diagram of the MS-CBAM feature fusion module.

Detailed ways

Embodiments of the present invention are described below through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific implementation modes, and various modifications or changes can be made to the details in this specification based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the diagrams provided in the following embodiments are only schematically illustrating the basic concept of the present invention, and the following embodiments and the features in the embodiments can be combined with each other in the case of no conflict.

Wherein, the accompanying drawings are for illustrative purposes only, and represent only schematic diagrams, rather than physical drawings, and should not be construed as limiting the present invention; in order to better illustrate the embodiments of the present invention, some parts of the accompanying drawings may be omitted, Enlargement or reduction does not represent the size of the actual product; for those skilled in the art, it is understandable that certain known structures and their descriptions in the drawings may be omitted.

In the drawings of the embodiments of the present invention, the same or similar symbols correspond to the same or similar components; , "front", "rear" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the drawings, which are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the referred devices or elements must It has a specific orientation, is constructed and operated in a specific orientation, so the terms describing the positional relationship in the drawings are for illustrative purposes only, and should not be construed as limiting the present invention. For those of ordinary skill in the art, the understanding of the specific meaning of the above terms.

A kind of target detection method based on dynamic convolution and attention mechanism and YOLO v5 dual-stream network provided by the present invention, as shown in Figure 1, the method comprises the following steps:

Step 1: Build the basic two-stream neural network model in step 1, as shown in Figure 2, YOLO v5 includes data processing, Backbone, Neck, prediction layer, and the present invention designs a kind of fusion module based on dynamic convolution feature, MS- The idea of feature extraction and fusion of CBAM multimodal cyclic fusion and refinement repeats the fusion operation multiple times and performs residual processing to increase the consistency of multispectral features.

Step 2: In step 2, construct a dynamic convolution ODConv based on the four dimensions of input channel, output channel, space and convolution kernel to give convolution different attention coefficient matrices, as shown in Figure 3, using a multi-head attention mechanism and parallel strategies to learn modality-complementary attention in four dimensions in convolutional kernels.

Step 3: Construct the MS-CBAM module based on channel attention and spatial attention, as shown in Figure 4, weight the feature map in the channel dimension and spatial dimension respectively, and use the residual network for feature refinement.

Step 4: Set the MS-CBAM module to output as the larger position of the feature map of 80×80×256, and the ODConv module to output as the position of the medium and small size of the feature map of 40×40×512 and 20×20×1024 . Input the feature map to the feature pyramid of YOLO v5, and continue the feature fusion and prediction of YOLO v5;

Step 5: Use the training samples to train the neural network after the parameters are determined until the training conditions are met, and use the test set to test the trained neural network;

In step 1, the present invention predicts based on the Neck and Head layers of YOLO v5, and establishes a baseline network based on a two-stream convolutional network, which is characterized in that the convolutional network is first used to extract the respective local features of the dual modes of visible light and infrared light, Then use the feature fusion module to perform the feature weighted fusion operation.

First, the visible light and infrared light images processed in step 1 are respectively subjected to three convolution operations, and the convolved visible light and infrared light feature maps are denoted as X _V , X _T .

The present invention designs a feature fusion operation using the MS-CBAM module and the ODConv module to form a residual network to perform feature fusion. As shown in Figure 2, the module and the residual network jointly construct a feature extraction and fusion idea of feature loop fusion and refinement. In the present invention, the feature fusion operation is respectively performed in three places of 80×80×256, 40×40×512, and 20×20×1024 in the YOLO v5 network, that is, the large, medium and small three represented by P3, P4 and P5 in Fig. 2 feature maps into the feature pyramid. The feature cycle fusion and refinement structure of the present invention can increase the consistency of multispectral features. Assuming that in the i-th fusion module, in order to obtain a new fusion feature f, the fusion process of the visible light image feature X _V and the infrared light image feature X _T can be expressed as:

where σ is the feature fusion function, and F is the feature fusion module.

To avoid overfitting, the operation F in all cycles shares weights, and then combines the fusion features with the original features to build a residual network:

To prevent the vanishing gradient problem when learning network parameters and perform better multispectral feature fusion, an auxiliary semantic segmentation task is used to bring individual information to each refined spectral feature.

The similarity between the modalities increases with the number of cycles, while their coherence increases and complementarity decreases as the similarity between spectral features increases. The consistency between multispectral features is very important, but on the contrary, too much consistency will lead to a sharp rise or fall of feature values, and redundant cyclic fusion is meaningless. According to experiments, the performance of feature fusion begins to decline after the fourth cycle, so in practice, we choose three cycles to balance consistency and complementarity.

At the same time, the three feature fusion modules will input the processed feature maps of large, medium and small to the feature pyramid three times.

Further, in step 2, the present invention is illustrated by using a dynamic convolution that uses a multi-head self-attention mechanism operation on the dimension of the convolution kernel. For the dynamic convolution layer, it uses a linear combination of n convolution kernels, dynamically weighted by the attention mechanism, so that the convolution operation depends on the input feature map. The overall operation of ODConv can be expressed as:

X'＝ODConv(concat(X _v ,X _T ))

Among them, X _V and X _T are the feature map inputs of visible light and infrared light modes respectively, concat represents the superposition of two inputs along the channel number dimension, and ODConv represents the dynamic convolution operation.

Specifically, mathematically, the dynamic convolution operation on a single dimension can be defined as:

y＝(α _w1 W ₁ +...+α _wn W _n )*x

in, and /> Represent the input and output of the feature map matrix whose height is h, width is w, and the number of channels is c. W _i denote by the output convolution filter /> The i-th convolution kernel composed of m=1,...,c _out . is the attention coefficient matrix of the convolution kernel dimension, which is calculated by the attention function π _wi (x) conditioned on the input features; * indicates the convolution operation, and the bias term is omitted here.

According to the dynamic convolution calculation equation, dynamic convolution has two basic components: Given n convolution kernels, the convolution kernel W _i and the attention function used to calculate its attention scalar The corresponding kernel space has four dimensions with respect to the spatial kernel size k×k, and each convolution kernel has the number of input channels c _in and the number of output channels c _out .

The ODConv module in the present invention takes into account the convolution kernel dimension, the spatial dimension, the input channel dimension and the output channel dimension at the same time, which makes the multi-modal feature fusion in the convolution operation more comprehensive, and the formula of each dimension is related to the convolution kernel dimension The dynamic convolution of is similar. As shown in Figure 3, its dynamic convolution formula that combines four dimensions can be expressed as:

y＝(α _w1 ⊙α _f1 ⊙α _c1 ⊙α _s1 ⊙W ₁ +...+α _wn ⊙α _fn ⊙α _cn ⊙α _sn ⊙W _n )*x

is the attention coefficient matrix of the convolution kernel W _i , /> and /> Respectively represent the dynamic convolution attention coefficient matrix along the spatial dimension, input channel dimension, and output channel dimension of the convolution kernel W _i , and ⊙ represents the multiplication along different dimensions of the kernel space.

Among them, α _si assigns different attention scalars to each convolution filter at k×k spatial positions; α _ci assigns different attention scalars to the c _in channel of each convolution filter W _i ^m ; α _fi A different attention scalar is assigned to the c _out channel of each convolution filter W _i ^m ; α _wi assigns the attention scalar to the entire convolution kernel. It multiplies the attention coefficient matrix of these four dimensions with the corresponding dimensions for n convolution kernels to obtain the output of the module.

Specifically, firstly, the input X is compressed into a feature vector with a length of c _in through the global average pooling operation, and the fully connected layer maps the compressed feature vector to a low-dimensional space with a reduction rate r after a fully connected layer and a ReLU unit. After four branches, each branch corresponds to the above four dimensions, each of which has an FC layer with an output size of k×k, c _in ×1, c _out ×1 and n×1, and a Softmax or Sigmoid function, Generate normalized attention coefficient matrices α _si , α _ci , α _fi and α _wi respectively.

Since these four dimensions are complementary and able to capture rich contextual cues. Therefore, ODConv can significantly enhance the feature extraction capability of basic convolution operations of CNNs.

Further, in step 3, the MS-CBAM module based on channel attention and spatial attention is established, the feature maps are weighted in the channel dimension and spatial dimension respectively, and the residual network is used for feature refinement.

For the input feature map X _V ∈ ^{V B×C×H×W} , X _T ∈ T ^B×C×H×W , where V represents the visible light image, T represents the infrared light image, B represents the Batch Size, and C represents the number of channels , H and W respectively represent the length and width of the feature map, and the unit is pixel. The calculation of the MS-CBAM module can be expressed as:

X＝M _s [concat[M _c (X _V ),M _c (X _T )]]

Among them, M _c represents the channel attention mechanism, and M _s represents the spatial attention mechanism. Concat means to stack the feature map in the channel dimension, and X represents the module output. Through channel attention and spatial attention, feature weighting can be carried out in the channel dimension and spatial dimension, which can reduce the adverse effects caused by using a certain type of pooling operation alone, and increase the accuracy performance of the neural network.

The Channel Attention Module (CAM) improves the representation ability of feature maps by learning the interactions between each channel. Specifically, the channel attention module first performs the maximum pooling and average pooling operations on each channel in the input feature map in turn to obtain the maximum pooling and average pooling feature maps. Then these two feature maps are used as input, the weight of each channel is obtained through two fully connected layers and the Sigmoid function, and the channel weight is multiplied by the original feature map to obtain a weighted feature map. The channel attention mechanism can be expressed as:

In the formula, and /> represent average pooling and maximum pooling, respectively.

The Spatial Attention Module (SAM) improves the representation ability of feature maps by learning the interaction between each pixel in the feature map. The input feature map of this module is the feature map output by the channel attention module. First, for an input feature map, the spatial attention module first performs maximum pooling and average pooling operations on it to obtain maximum pooling and average pooling feature maps. Then the two feature maps are stitched together, the weight of each pixel is obtained through a convolutional layer and the Sigmoid function, and the weighted feature map is obtained by multiplying the pixel weight with the original feature map. Then, average pooling and maximum pooling are performed on the channel dimensions of the visible and infrared feature maps, respectively, to obtain two feature maps of size . Then, the two feature maps are concatenated in the channel dimension to obtain a feature map of size . Finally, the feature map is reduced to 1 channel through a 7×7 convolution operation, and then the spatial attention feature is generated through the Sigmoid activation function.

Finally, the output features of spatial attention are multiplied element-wise by the input features to obtain the final generated features. The spatial attention mechanism can be expressed as:

In the formula, and /> represent average pooling and maximum pooling, respectively.

The present invention uses channel attention and spatial attention, and then refines the residual network built by X. The process can be expressed as:

_X'V ＝ _XV +X

X' _T = X _T + X

The finally obtained feature maps are X' _V ∈ ^{V B×C×H×W} , X' _T ∈ T ^B×C×H×W , representing the final output of the MS-CBAM module.

Further, in step 4, the feature map sizes H, W, and C are respectively 80×80×256, 40×40×512, 20×20×1024, that is, the features of the three positions of P3, P4, and P5 in Figure 2 Figures use MS-CBAM, ODConv, and ODConv for multimodal feature fusion, and then input the three large, medium, and small feature maps into the YOLO v5 Neck feature pyramid for further feature fusion and extraction.

In step 5, the loss function is divided into localization loss, classification loss, and confidence loss, which can be expressed as:

The positioning loss uses NWD, and other losses use the default loss function of YOLO v5:

NWD uses a metric based on Wasserstein distance, which greatly improves the performance of small target detection.

For small objects, there will always be some background pixels in the bounding box, because the real object cannot be exactly a rectangle. In the bounding box, the foreground pixels are generally concentrated in the middle, and the background pixels are generally concentrated on the sides. To better weight each pixel in the bounding box, the bounding box can be modeled as a 2D Gaussian distribution. Specifically, for a horizontal bounding box R=(cx, cy, w, h), an inscribed ellipse can be expressed as:

where (μ _x ,μ _y ) is the center point of the ellipse, and (σ _x ,σ _y ) are the radii of the x and y axes. Corresponding to the bounding box:

μ _x = cx, μ _y = cy,

The probability density function of a 2D Gaussian distribution is:

Among them, X, μ, ∑ represent the coordinates (x, y), mean and variance respectively. when:

This ellipse is a distribution profile of the 2D Gaussian distribution. Therefore, the horizontal bounding box R=(cx,cy,w,h) can be modeled as a 2D Gaussian distribution:

In this way, the similarity between two bounding boxes can be represented by the distance between these two Gaussian distributions.

Next, the present invention uses the Wasserstein distance in optimal transport theory to calculate the distance of the two distributions. For two 2D Gaussian distributions, the 2nd-order Wasserstein distance can be defined as:

Right now:

For two bounding boxes:

However, this is a distance metric and cannot be used directly for similarity. We use the normalized exponent to get a new metric called the normalized Wasserstein distance:

Here C is a constant, related to the data set.

Afterwards, the constructed model input data set is trained, and the model parameters of the current epoch are saved for each iteration of an epoch, and the classification accuracy of the current epoch is compared with the classification accuracy of the previous optimal model. When the set maximum epoch is reached, the pedestrian target recognition model with the best recognition accuracy is output. After training, the model can realize the detection and recognition of targets in poor light environments, including the detection and recognition of objects such as people, animals, cars, other vehicles, and obstacles.

Finally, it is noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be carried out Modifications or equivalent replacements, without departing from the spirit and scope of the technical solution, should be included in the scope of the claims of the present invention.

Claims (6)

1. The multi-mode characteristic target detection method based on the dynamic convolution and the attention mechanism is characterized by comprising the following steps of: the method comprises the following steps:

s1: a neural network model of a double-flow convolution detection network based on YOLOv5 is established, wherein a back bone adopts convolution operation and a feature fusion module to perform modal fusion and feature learning;

s2: the method comprises the steps that a multispectral module MS-CBAM is formed by adopting channel attention and spatial attention, the channel attention is used for respectively carrying out feature weighting on visible light and infrared light image feature graphs, infrared light and visible light images are stacked together, the spatial attention is used for carrying out feature weighting on the feature graphs, and then a residual network is used for refining features;

s3: introducing a multi-head attention mechanism to a convolution structure, and establishing a dynamic convolution ODConv module by endowing different attention coefficient matrixes of convolution to the four dimensions of an input channel dimension, an output channel dimension, a space dimension and a convolution kernel;

s4: setting an MS-CBAM module as a position with a larger characteristic diagram of 80 multiplied by 256 to output, and setting an ODConv module as a position with a medium and small characteristic diagrams of 40 multiplied by 512 and 20 multiplied by 1024 to output; three feature graphs with different sizes are output to enter a Neck layer, namely a feature pyramid, feature extraction is carried out, the output features are predicted, and a prediction result is output;

s5: in the training stage, visible light and infrared light data enter a double-flow neural network training after being subjected to specific Mosaic data enhancement, self-adaptive anchor frame calculation and self-adaptive picture scaling process; initializing by using a YOLOv5l pre-training weight, and learning parameters of a network by using a random gradient descent algorithm;

in the prediction stage, a softmax classifier is used for obtaining the final classification probability of the category to which the model belongs;

in the optimization stage, the error between the true value and the predicted value is reduced by adopting a mode of combining and optimizing the positioning loss, the classification loss and the confidence coefficient loss, and NWD is introduced into the positioning loss, so that the precision of small target detection is improved; and S5, continuously repeating the step until the iteration times reach the set iteration times, completing model training, and carrying out target detection tasks.

2. The method for detecting the multi-modal feature object based on the dynamic convolution and attention mechanism according to claim 1, wherein the method comprises the following steps: in the step S1, an input of a double-flow convolution target detection network frame based on YOLOv5 is an image pair of different modes, a Backbone is a double-flow convolution network, and a double-flow neural network model comprises Backbone, neck and a prediction layer;

let the input visible light characteristic diagram be X _V The input infrared light characteristic diagram is X _T The length, width and channel number of the feature map are H, W, C respectively;

the feature extraction network structure uses three feature fusion modules and a residual network to form a three-time feature extraction circulation and refinement structure, and the ith feature fusion calculation process is expressed as follows:

wherein sigma is a feature fusion function, and the visible light image input feature graph is X _V The infrared light image input characteristic diagram is X _T F is a feature fusion module for carrying out batch normalization operation; fusion feature mapThe length, the width and the channel number of the device are H, W and 2C respectively; and then constructing a residual error network by combining the fusion characteristic and the original characteristic:

obtaining new characteristic diagram f for visible light and infrared light _t ⁱ And

3. the method for detecting the multi-modal feature object based on the dynamic convolution and the attention mechanism according to claim 2, wherein the method comprises the following steps: in the step S2, the visible light and the infrared light are input into the image, the channel attention mechanism calculation is respectively carried out on the visible light and the infrared light, then the feature images are overlapped in a mode of overlapping the channel dimensions, and then the feature images are input into the space attention for operation;

the calculation of the MS-CBAM module is expressed as:

X＝M _S [concat[M _C (X _V ),M _C (X _T )]]

wherein M is _C Representing the channel attention mechanism, M _S Representing a spatial attention mechanism; concat means stacking the feature graphs in the channel dimension;

and then refining the X constructed residual error network, wherein the process is expressed as follows:

X' _V ＝X _V +X

X' _T ＝X _T +X

the finally obtained characteristic diagram is X' _V ∈V ^B×C×H×W 、X' _T ∈T ^B×C×H×W Representing the final output of the MS-CBAM module.

4. The method for detecting a multi-modal feature object based on dynamic convolution and attention mechanism as claimed in claim 3, wherein: in the step S3, a multi-head self-attention mechanism is introduced in the convolution process, and different attention coefficient matrixes ODConv of convolution are endowed to four dimensions of an input channel dimension, an output channel dimension, a space dimension and a convolution kernel, so that the capability of feature extraction is improved; the operation of the ODConv module as a whole is expressed as:

X'＝ODConv(concat(X _V ,X _T ))

wherein X is _V And X _T Respectively inputting feature graphs of visible light and infrared light modes, wherein concat represents that the two inputs are overlapped along the channel number dimension, and ODConv represents dynamic convolution operation;

it integrates the four-dimensional dynamic convolution formulas into:

y＝(α _w1 ⊙α _f1 ⊙α _c1 ⊙α _s1 ⊙W ₁ +...+α _wn ⊙α _fn ⊙α _cn ⊙α _sn ⊙W _n )*x

dimension W is the convolution kernel _i Attention coefficient matrix of->And->Respectively along the convolution kernel W _i Dynamic convolution attention coefficient matrix in the spatial dimension, input channel dimension, output channel dimension, +..

5. The method for detecting the multi-modal feature object based on the dynamic convolution and attention mechanism according to claim 4, wherein: the input and output of the MS-CBAM module and the ODConv module are both visible light and infrared light characteristic diagrams, and the output and the input form a residual error network;

the loss functions of the positioning loss, the classification loss and the confidence loss are expressed as follows:

L _total ＝L _box +L _cls +L _conf

wherein the positioning loss adopts an NWD loss function; the NWD loss function calculates the similarity by introducing Normalized Wasserstein Distance calculation method through the corresponding gaussian distribution.

6. The method for detecting the multi-modal feature object based on the dynamic convolution and attention mechanism according to claim 5, wherein: the NWD loss function is expressed as:

wherein,,for Wasserstein distance, +.>And C is a fixed constant related to the data set, so that the detection performance of the small target is improved.