CN114694001A - A target detection method and device based on multimodal image fusion - Google Patents

Abstract

The invention provides a target detection method and device based on multi-modal image fusion. The method comprises the following steps: acquiring a video image and an infrared image in real time, and respectively inputting the video image and the infrared image into a target detection model formed by a transform; respectively extracting global features of the video image and the infrared image; fusing the extracted video image features and the infrared image features; and inputting the fusion characteristics of the video image and the infrared image into a prediction module consisting of a transform full-link layer, and outputting the target type and the target position. According to the method, a pure Transformer is utilized to construct a target detection model, so that model advantages brought by the overall structure of the Transformer can be fully exerted; the invention carries out target detection based on the feature fusion of the video image and the infrared image, can realize target detection under any illumination condition, and solves the problem of poor detection effect of the existing detection system in dark environments such as night and the like.

Description

A target detection method and device based on multimodal image fusion

technical field

The invention belongs to the technical field of target detection, and in particular relates to a target detection method and device based on multimodal image fusion.

Background technique

For a long time, how to help disadvantaged people with visual impairment to obtain better mobility has been a social issue that has attracted much attention. Timely and correct perception of the surrounding environment is an essential condition to help improve the safety and quality of life of the target individual. With the rapid development of computer vision technology in recent years, various deep learning models based on convolutional neural networks (CNN) have been able to show outstanding capabilities in real-time recognition tasks for natural scene images, and even have an accuracy that surpasses that of humans. It is stable and has been successfully deployed in products, such as autonomous driving technology that has achieved excellent results recently.

Some wearable electronic devices with visual aids and perception developed for the visually impaired have also benefited from this. They collect images or video data in real-time scenes with the help of miniature cameras or sensors on the devices, and the models on them perform corresponding calculations. , so as to provide the wearer with the result information of scene target detection. However, most object detection models are modeled based on visible light color image data with sufficient brightness, which makes the model perform well when receiving visible light image input with poor ambient lighting conditions (such as night, dark spaces, and other life scenes). If it is greatly reduced, sufficient recognition ability cannot be achieved, and the corresponding visually impaired auxiliary equipment cannot provide the wearer with timely danger warnings.

SUMMARY OF THE INVENTION

In order to solve the above problems existing in the prior art, the present invention provides a target detection method and device based on multimodal image fusion.

In order to achieve the above objects, the present invention adopts the following technical solutions.

In a first aspect, the present invention provides a target detection method based on multimodal image fusion, comprising the following steps:

Real-time acquisition of video images and infrared images captured by video cameras and infrared cameras, respectively, and input to the target detection model composed of Transformer;

Utilize the feature encoding module composed of Transformer encoder to perform global feature extraction on the video image and the infrared image respectively;

Use the feature fusion module composed of Transformer decoder to fuse the extracted video image features and infrared image features;

The fusion features of video images and infrared images are input into the prediction module composed of the Transformer fully connected layer, and the target category and target position are output.

Further, the method further includes the following operations respectively performed on the input video image and the infrared image before performing the global feature extraction:

Cut the image into N slices;

Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector;

The sine and cosine position codes of the slice row and column directions are calculated and added to the output of the linear fully connected layer to obtain an N×d coding matrix.

Further, the feature encoding module is formed by stacking Transformer encoders, and each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer and a normalization layer and residual unit connected with each layer; The N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module, through three different linear transformations, the query vector, key vector and value vector of size N×d′ are obtained. The similarity is calculated by the vector dot product with the scaling factor, and the attention weight matrix is obtained after normalization by the softmax function, and the weight matrix is multiplied by the value vector to obtain one attention result; Then map back to the original dimension d' to obtain the feature code of the video image or infrared image.

Further, the feature fusion module is formed by stacking Transformer decoders, and each Transformer decoder includes a multi-head self-attention module layer, a multi-head mutual attention module layer, a feedforward network layer and a layer connected to each layer. A normalization layer and residual unit; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i and the value vector V _i come from the feature encoding module respectively. The output video image feature A and infrared image feature B; the query vector Q _i+1 of the multi-head mutual attention module layer of the i+1 Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 The sum value vector V _i+1 comes from B and A respectively; the key vector K _i and the value vector V _i are both N×d′ matrices, and the query vector Q _i is an N′×d′ matrix, N′<N; i=1 ,2,….

Further, the method further includes: judging the dangerous target and its orientation according to the target category and the target position, and sending out danger warning information.

In a second aspect, the present invention provides a target detection device based on multimodal image fusion, including:

The image acquisition module is used to acquire the video images and infrared images captured by the video camera and the infrared camera in real time, and input them to the target detection model composed of the Transformer respectively;

A feature extraction module, used for using the feature encoding module formed by the Transformer encoder to perform global feature extraction on the video image and the infrared image respectively;

The feature fusion module is used to fuse the extracted video image features and infrared image features by using the feature fusion module composed of the Transformer decoder;

The target prediction module is used to input the fusion features of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and target position.

Further, the device also includes a vector embedding module for:

Cut the image into N slices;

Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector;

The sine and cosine position codes of the slice row and column directions are calculated and added to the output of the linear fully connected layer to obtain an N×d coding matrix.

Further, the feature encoding module is formed by stacking Transformer encoders, and each Transformer encoder includes a multi-head self-attention module layer and a feedforward network layer and a normalization layer and residual unit connected with each layer; input The N×d encoding matrix of the video image or infrared image to the multi-head self-attention module, after three different linear transformations, the query vector, key vector and value vector of size N×d′ are obtained. The similarity is calculated by the vector dot product with the scaling factor, and normalized by the softmax function to obtain the attention weight matrix. The weight matrix is multiplied by the value vector to obtain one attention result; after splicing the multi-way attention results Then map back to the original dimension d' to obtain the feature code of the video image or infrared image.

Further, the feature fusion module is formed by stacking Transformer decoders, and each Transformer decoder includes a multi-head self-attention module layer, a multi-head mutual attention module layer, a feedforward network layer and a layer connected to each layer. A normalization layer and residual unit; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i and the value vector V _i come from the feature encoding module respectively. The output video image feature A and infrared image feature B; the query vector Q _i+1 of the multi-head mutual attention module layer of the i+1 Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 The sum value vector V _i+1 comes from B and A respectively; the key vector K _i and the value vector V _i are both N×d′ matrices, and the query vector Q _i is an N′×d′ matrix, N′<N; i=1 ,2,….

Further, the device further includes a danger warning module, which is used for judging the dangerous target and its orientation according to the target category and target position, and sending out danger warning information.

Compared with the prior art, the present invention has the following beneficial effects.

The present invention obtains video images and infrared images in real time, uses a target detection model composed of pure Transformers to extract global features of the video images and infrared images respectively, and fuses the extracted video image features and infrared image features. The feature is used to predict the target category, and the target detection based on multimodal image fusion is realized. The invention uses pure Transformer to construct a target detection model, which can give full play to the model advantages brought by the overall structure of the Transformer; the invention performs target detection based on the feature fusion of video images and infrared images, can realize target detection under any lighting conditions, and solves the problem of current There is a problem that the detection system has a poor detection effect in dark environments such as night.

Description of drawings

FIG. 1 is a flowchart of a target detection method based on multimodal image fusion according to an embodiment of the present invention.

FIG. 2 is a schematic diagram of an overall structure of a target detection model according to an embodiment of the present invention.

Figure 3 is a schematic diagram of the principle of the self-attention mechanism.

Figure 4 is a schematic diagram of the connection of two Transformer decoders.

FIG. 5 is a block diagram of a target detection apparatus based on multimodal image fusion according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer and more comprehensible, the present invention will be further described below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

1 is a flowchart of a target detection method for multimodal image fusion according to an embodiment of the present invention, including the following steps:

Step 101, acquire video images and infrared images captured by a video camera and an infrared camera in real time, and input them into a target detection model formed by a Transformer respectively;

Step 102, utilize the feature encoding module that Transformer encoder is formed to carry out global feature extraction to described video image and infrared image respectively;

Step 103, utilize the feature fusion module formed by Transformer decoder to fuse the extracted video image features and infrared image features;

Step 104: Input the fusion feature of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and target position.

In this embodiment, step 101 is mainly used to acquire video images and infrared images in real time. Most of the existing target detection models for assisting visually impaired people are based on visible light color image data with sufficient brightness, which makes the model in poor ambient lighting conditions (such as night, dark spaces, etc. The performance of the visible light image input of the scene) is greatly reduced, and the required recognition ability cannot be achieved. To this end, this embodiment also acquires infrared images while acquiring video images. Since the imaging principle of the infrared camera is not affected by the lighting conditions, the collected infrared images can provide a powerful supplement to the scene target information in the dark environment. Therefore, the target detection model based on the fusion of video and infrared images can be used in light or darkness. It has a high level of generalization ability in all scenarios. The target detection model in this embodiment adopts a pure Transformer structure, which fully utilizes the model advantages brought by the overall Transformer structure, and can achieve better effects and generalization capabilities than the convolutional neural network CNN model in the image recognition task. The overall structure of the target detection model is shown in Figure 2.

In this embodiment, step 102 is mainly used for image feature extraction. In this embodiment, a feature encoding module composed of a Transformer encoder is used to perform feature extraction on the video image and the infrared image respectively. The Transformer encoder adopts the attention mechanism, which is mainly composed of multi-head self-attention modules, which can extract the global features of the input image. Compared with CNN, which can only extract the local features of the image, the target detection accuracy is greatly improved.

In this embodiment, step 103 is mainly used to perform multi-modal feature fusion. In this embodiment, a feature fusion module composed of a Transformer decoder is used to fuse the extracted video image features and infrared image features. There are mainly three fusion schemes used by the existing CNN-based network structure models to complete the multimodal image fusion task, which are called early, middle and late fusion: The input end of the model concatenates images from multiple modalities directly in the channel dimension as the input of the entire network; in the mid-term fusion, different modalities have independent feature extractors, and use various defined fusion calculation methods to fuse The feature map of each modal at a certain level; late fusion is to fuse the final results of each modal through independent feature extractors for prediction. No matter which fusion method is used, in fact, it simply seeks the input of fusion, without sufficient theoretical explanatory support, or task orientation, and assumes that the characteristics between modalities are the same in space. Correspondingly, convolution only performs local fusion calculations. However, images of different modalities and even feature maps will have a certain positional deviation. Only performing local calculations may cause the corresponding features to not be aligned, which will lead to low fusion efficiency and poor detection effect. In this embodiment, the Transformer is used to provide a multi-modal fusion method based on attention (the Transformer decoder includes a multi-head self-attention module and a multi-head mutual attention module) to replace the CNN, so that the information of different modalities can be used for mutual attention on a global scale , so that it is not limited by the position deviation, making the fusion more effective and more theoretically supportive.

In this embodiment, step 104 is mainly used to predict the target category. In this embodiment, the prediction of the target category is realized by inputting the fusion feature of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer. The target in this embodiment refers to a dangerous target that may pose a threat to action, and is classified into target categories according to the risk level. The prediction module generally outputs the target position while outputting the target category. The prediction module consists of two fully connected layer branches, one of which consists of an N1 layer fully connected layer to complete the prediction of the target category, and the other branch consists of an N2 layer fully connected layer to complete the target position (the upper left corner of the detection frame and The lower right corner coordinates) of the regression prediction, so as to achieve the target detection task. The inputs of the two branches are the same, which are the fusion features finally output by the feature fusion module.

As an optional embodiment, before the global feature extraction is performed, the method further includes the following operations respectively performed on the input video image and the infrared image:

Cut the image into N slices;

Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector;

The sine and cosine position codes of the slice row and column directions are calculated and added to the output of the linear fully connected layer to obtain an N×d coding matrix.

This embodiment presents a technical solution for vector embedding of input video images and infrared images. For the input video image and infrared image, it is necessary to perform embedding encoding to convert it into a sequence input that the Transformer can accept. Specifically, for an image input with a size of C×H×W, slice it (patch), it may be assumed that the spatial size of each patch is h×w, then N=(H/h)×( W/w) slices of size C×h×w. Flatten each slice along the dimension of channel C to obtain a C×h×w-dimensional vector, and the N×(C×h×w) matrix is input into a linear fully connected layer to calculate and change the dimension to d-dimension. In addition, in order to make the patch code contain two-dimensional position information instead of showing arrangement invariance, a fixed d-dimensional sine or cosine position code is calculated for the row and column directions respectively, and added to the output of the linear layer, and finally obtained The N×d matrix is the linear embedded coding representation of the input image, in which the d-dimensional vector of each row is the representative vector of a patch, and the number of rows N of the matrix can be called the number of representative vectors. It should be pointed out that N varies with the size of the set patch, and N can be flexibly set according to the actual needs of specific tasks.

As an optional embodiment, the feature encoding module is formed by stacking Transformer encoders, and each Transformer encoder includes a multi-head self-attention module layer, a feedforward network layer, and a normalization layer and residual layer connected to each layer. Difference unit; the N×d coding matrix of the video image or infrared image input to the multi-head self-attention module, after three different linear transformations, the query vector, key vector and value vector of size N×d′ are obtained. The query vector and The similarity between the key vectors is calculated by the vector dot product with the scaling factor, and the attention weight matrix is obtained after normalization by the softmax function. The weight matrix is multiplied with the value vector to obtain one attention result; The force results are spliced and then mapped back to the original dimension d' to obtain the feature code of the video image or infrared image.

This embodiment presents a specific technical solution for feature extraction. Feature extraction is implemented by the feature encoding module, which is obtained by stacking Transformer encoders. The specific stacking layers can be determined according to specific task debugging, and the encoders corresponding to the two branches of the two images are independent of each other, and the number of stacked layers can be the same or not. can be different. The specific structure of each Transformer encoder consists (in order) of a multi-head self-attention module layer, a forward propagation module layer, and residual connections and normalization applied at each layer. The calculation process of the self-attention mechanism can refer to Figure 3. The input N×d coding matrix is transformed by the linear mapping functions W _q , W _k , and W _v to obtain the query vector (Query) and the key vector ( Key), value vector (Value), the similarity between the query vector and the key vector is calculated according to the vector dot product with scaling factor, and normalized by the softmax function to obtain the attention weight matrix, which is expressed as follows:

where α is the weight matrix, Q is the query vector, and K ^T is the transpose of the key vector. The weight matrix is used to multiply the value vector (that is, equivalent to performing a column-wise weighted summation of the value vector according to the weight to obtain the value of a point on the result matrix). Multi-head self-attention is to repeat the process independently multiple times, concatenate the results of multiple times and map them back to the original feature dimension d'. The forward relay module layer is a multilayer perceptron (MLP) structure with one hidden layer. After the Transformer encoder, the input image can encode its own features on the global scale, that is, each representative vector will calculate the similarity with all other representative vectors, including itself, which has the globality that is not available in CNN extraction of image features.

As an optional embodiment, the feature fusion module is formed by stacking Transformer decoders, each Transformer decoder includes a multi-head self-attention module layer, a multi-head mutual attention module layer and a feedforward network layer A normalization layer and a residual unit connected by layers; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i and value vector _Vi come from respectively The video image feature A and infrared image feature B output by the feature encoding module; the query vector Q _i+1 of the multi-head mutual attention module layer of the i+1 Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 and value vector V _i+1 come from B and A respectively; both key vector K _i and value vector V _i are N×d′ matrices, query vector Q _i is N′×d′ matrix, N′<N; i = 1, 2, . . .

This embodiment presents a specific technical solution for feature fusion. The fusion of two modal image features is realized by the feature fusion module, which is obtained by stacking Transformer decoders. The schematic diagram of the structure of stacking two Transformer decoders in succession is shown in Figure 4. Likewise, the specific stacking layers of the decoder can be determined by task-specific debugging. The detailed structure of each Transformer decoder (in order) consists of a multi-head self-attention module layer, a multi-head mutual attention module layer, a forward relay module layer, and residual connections and normalization applied at each layer. The multi-head self-attention module layer and the forward propagation module layer are the same as those in the Transformer encoder. The computing mechanism of the multi-head mutual attention module layer is the same as that of self-attention, the only difference is that the query vector it receives comes from the output of the previous multi-head self-attention module layer, and the key vector and value vector come from the output of the feature encoding module respectively. The video image feature A and the infrared image feature B. It is worth noting that the order of the image features A and B connected by the key vector and the value vector of adjacent decoders is just opposite. For example, if the key vector and the value vector of the current decoder are connected to A and B respectively, then The key vector and the value vector of the previous decoder and the next decoder are respectively connected to B and A, so that the query vector can alternately perform attention calculation and fusion on the features of the two modalities. This design can effectively balance some information deviations that may exist between the two modalities, including positional deviations, not only extract effective content with similar distributions, but also model the key global relationships that may exist. However, it should be noted that a specially defined query vector needs to be initialized separately for the first layer Transformer decoder as input. The query vector is a set of learnable parameters that can implicitly learn how to extract the presence of targets in multimodal images. The location of the region encodes and plays a mediating role in the fusion. It has good task orientation and a priori, and is a key component to complete the target detection task and the multimodal fusion task. The dimension of this query vector is the same as that of the modality image encoding, but the size N' (or the number, i.e. the number of rows of the encoding matrix) should be much smaller than the number N of the modality image encoding, i.e. N'<<N, slightly larger than the data The maximum number of objects to be detected in the image can reduce missed detections, and only interact with necessary features in the process of attention calculation, reduce information redundancy, and greatly reduce computational costs.

As an optional embodiment, the method further includes: judging the dangerous target and its orientation according to the output target category and target position, and sending out danger warning information.

This embodiment presents a technical solution for danger early warning. The danger warning belongs to the post-processing step. In this embodiment, the dangerous target is judged based on the target category and target position output by the prediction module, and the orientation of the target relative to the user (which may also include the distance) is calculated. Finally, an alarm message is sent to the user through the voice module to remind the user. To draw attention or to evade.

FIG. 5 is a schematic diagram of the composition of a target detection device for multi-modal image fusion according to an embodiment of the present invention, and the device includes:

The image acquisition module 11 is used for real-time acquisition of video images and infrared images captured by the video camera and the infrared camera, respectively, and input to the target detection model formed by the Transformer;

The feature extraction module 12 is used to perform global feature extraction on the video image and the infrared image respectively by utilizing the feature encoding module formed by the Transformer encoder;

The feature fusion module 13 is used to fuse the extracted video image features and infrared image features by using the feature fusion module formed by the Transformer decoder;

The target prediction module 14 is used to input the fusion feature of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and the target position.

The apparatus of this embodiment can be used to execute the technical solution of the method embodiment shown in FIG. 1 , and the implementation principle and technical effect thereof are similar, and are not repeated here. The same is true for the following embodiments, which will not be further described.

As an optional embodiment, the apparatus further includes a vector embedding module for:

Cut the image into N slices;

Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector;

The sine and cosine position codes of the slice row and column directions are calculated and added to the output of the linear fully connected layer to obtain an N×d coding matrix.

As an optional embodiment, the feature encoding module is formed by stacking Transformer encoders, and each Transformer encoder includes a multi-head self-attention module layer, a feedforward network layer, and a normalization layer and residual layer connected to each layer. Difference unit; the N×d coding matrix of the video image or infrared image input to the multi-head self-attention module, after three different linear transformations, the query vector, key vector and value vector of size N×d′ are obtained. The query vector and The similarity between the key vectors is calculated by the vector dot product with the scaling factor, and the attention weight matrix is obtained after normalization by the softmax function. The weight matrix is multiplied with the value vector to obtain one attention result; The force results are spliced and then mapped back to the original dimension d' to obtain the feature code of the video image or infrared image.

As an optional embodiment, the feature fusion module is formed by stacking Transformer decoders, each Transformer decoder includes a multi-head self-attention module layer, a multi-head mutual attention module layer and a feedforward network layer A normalization layer and a residual unit connected by layers; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i and value vector _Vi come from respectively The video image feature A and infrared image feature B output by the feature encoding module; the query vector Q _i+1 of the multi-head mutual attention module layer of the i+1 Transformer decoder comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 and value vector V _i+1 come from B and A respectively; both key vector K _i and value vector V _i are N×d′ matrices, query vector Q _i is N′×d′ matrix, N′<N; i = 1, 2, . . .

As an optional embodiment, the device further includes a danger early warning module, configured to judge the dangerous target and its orientation according to the target category and target location, and send out danger warning information.

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope disclosed by the present invention can easily think of changes or substitutions. All should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. a target detection method based on multimodal image fusion, is characterized in that, comprises the following steps: Real-time acquisition of video images and infrared images captured by video cameras and infrared cameras, respectively, and input to the target detection model composed of Transformer; Utilize the feature encoding module formed by the Transformer encoder to perform global feature extraction on the video image and the infrared image respectively; Use the feature fusion module composed of Transformer decoder to fuse the extracted video image features and infrared image features; The fusion features of video images and infrared images are input into the prediction module composed of the Transformer fully connected layer, and the target category and target position are output. 2. the target detection method based on multimodal image fusion according to claim 1, is characterized in that, described method also comprises the following operations that the video image of input and infrared image are carried out respectively before carrying out global feature extraction: Cut the image into N slices; Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector; The sine and cosine position codes of the slice row and column directions are calculated and added to the output of the linear fully connected layer to obtain an N×d coding matrix. 3. the target detection method based on multimodal image fusion according to claim 2, is characterized in that, described feature encoding module is formed by stacking Transformer encoder, and each Transformer encoder comprises a multi-head self-attention module layer and a feedforward network layer and a normalization layer and residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module, after three different linear transformations, the size is N×d' query vector, key vector and value vector, the similarity between the query vector and the key vector is calculated by the vector dot product with scaling factor, and normalized by the softmax function to obtain the attention weight matrix, the weight The matrix and the value vector are multiplied to obtain one-way attention results; the multi-way attention results are spliced and then mapped back to the original dimension d' to obtain the feature encoding of the video image or infrared image. 4. The target detection method based on multimodal image fusion according to claim 3, wherein the feature fusion module is formed by stacking Transformer decoders, and each Transformer decoder includes a multi-head self-attention module layer , a multi-head mutual attention module layer and a feedforward network layer, and a normalization layer and residual unit connected to each layer; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from multi-head self-attention The output of the force module layer, the key vector K _i and the value vector V _i come from the video image feature A and the infrared image feature B output by the feature encoding module respectively; the query vector of the multi-head mutual attention module layer of the i+1 Transformer decoder Q _i+1 comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 and the value vector V _i+1 come from B and A, respectively; the key vector K _i and the value vector V _i are both N×d' matrices, The query vector Q _i is an N'×d' matrix, N'<N; i=1, 2, . . . 5 . The target detection method based on multimodal image fusion according to claim 1 , wherein the method further comprises: judging the dangerous target and its orientation according to the target category and target location, and issuing danger warning information. 6 . 6. A target detection device based on multimodal image fusion, characterized in that, comprising: The image acquisition module is used to acquire the video images and infrared images captured by the video camera and the infrared camera in real time, and input them to the target detection model composed of the Transformer respectively; A feature extraction module, used for using the feature encoding module formed by the Transformer encoder to perform global feature extraction on the video image and the infrared image respectively; The feature fusion module is used to fuse the extracted video image features and infrared image features by using the feature fusion module composed of the Transformer decoder; The target prediction module is used to input the fusion features of the video image and the infrared image into the prediction module composed of the Transformer fully connected layer, and output the target category and target position. 7. The target detection device based on multimodal image fusion according to claim 6, wherein the device further comprises a vector embedding module for: Cut the image into N slices; Expand each slice in the channel dimension and input it to a linear fully connected layer to get a d-dimensional vector; The sine and cosine position codes of the slice row and column directions are computed and added to the output of the linear fully connected layer to obtain an N×d coding matrix. 8. The target detection device based on multimodal image fusion according to claim 7, wherein the feature encoding module is formed by stacking Transformer encoders, and each Transformer encoder comprises a multi-head self-attention module layer and a feedforward network layer and a normalization layer and residual unit connected to each layer; the N×d encoding matrix of the video image or infrared image input to the multi-head self-attention module, after three different linear transformations, the size is N×d' query vector, key vector and value vector, the similarity between the query vector and the key vector is calculated by the vector dot product with scaling factor, and normalized by the softmax function to obtain the attention weight matrix, the weight The matrix and the value vector are multiplied to obtain one-way attention result; the multi-way attention results are spliced and then mapped back to the original dimension d' to obtain the feature encoding of the video image or infrared image. 9. The target detection device based on multimodal image fusion according to claim 8, wherein the feature fusion module is formed by stacking Transformer decoders, and each Transformer decoder includes a multi-head self-attention module layer , a multi-head mutual attention module layer and a feed-forward network layer, and a normalization layer and residual unit connected to each layer; the query vector Q _i of the multi-head mutual attention module layer of the i-th Transformer decoder comes from multi-head self-attention The output of the force module layer, the key vector K _i and the value vector V _i come from the video image feature A and infrared image feature B output by the feature encoding module, respectively; the query vector of the multi-head mutual attention module layer of the i+1 Transformer decoder Q _i+1 comes from the output of the multi-head self-attention module layer, and the key vector K _i+1 and the value vector V _i+1 come from B and A, respectively; the key vector K _i and the value vector V _i are both N×d' matrices, The query vector Q _i is an N'×d' matrix, N'<N; i=1, 2, . . . 10 . The target detection device based on multimodal image fusion according to claim 6 , wherein the device further comprises a danger warning module for judging the dangerous target and its orientation according to the target category and target position, and sending out the 10 . Hazard warning information.

Images