CN112418163A - A Multispectral Target Detection Guide System - Google Patents

Abstract

The invention provides a multi-spectral target detection guidance system, comprising: a data input module for acquiring visible light image and infrared thermal image; a deformable feature extractor module for extracting visible light image feature map and infrared thermal image feature respectively Figure; candidate frame extraction network, used to extract visible light image candidate frame and infrared thermal image candidate frame; candidate frame complementary module, used to add the part of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image In the candidate frame, the part of the infrared thermal image candidate frame that is not covered by the visible light image candidate frame is added to the visible light image candidate frame to obtain the visible light image region feature map and the infrared thermal image region feature map; cross-modal attention fusion module , which is used to fuse the visible light image region feature map into the infrared thermal image region feature map according to the similarity between the features of each region to obtain enhanced thermal image features; the classification and regression modules are used to obtain target detection results.

Description

A Multispectral Target Detection Guide System

technical field

The invention relates to the field of computers, in particular to a multispectral target detection guidance system.

Background technique

The tremendous development of computer vision in recent years has brought new opportunities and possibilities to the guidance system. Deep learning models based on Convolutional Neural Networks (CNN) have reached or surpassed human-level performance in image classification (ImageNet dataset) and object detection (COCO dataset) tasks. Visual perception systems (especially object detection systems) based on deep learning technology have achieved good results in applications such as unmanned driving. Therefore, the use of this technology to assist the blind to perceive the environment has formed a new trend. However, the previous target detection models are generally constructed based on visible light color images, and the applicable scenes are limited by the lighting conditions, and cannot be applied at night or in places with strong light. Similarly, the blind guidance system using this technology also has this problem and cannot assist the blind to perceive the environment around the clock.

SUMMARY OF THE INVENTION

The present invention aims to provide a multispectral target detection guidance system that overcomes the above problems or at least partially solves the above problems.

In order to achieve the above object, the technical scheme of the present invention is specifically realized in this way:

One aspect of the present invention provides a multispectral target detection guidance system, including: a data input module for acquiring visible light images and infrared thermal images; a deformable feature extractor module for using deformable convolution to extract the The image features of the visible light image and the infrared thermal image, and output the visible light image feature map and the infrared thermal image feature map; the candidate frame extraction network is used to extract the candidate frame of the target object according to the visible light image feature map and the infrared thermal image feature map to obtain the visible light image. The candidate frame and the infrared thermal image candidate frame; the candidate frame complementary module is used to add the part of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image candidate frame, and the infrared thermal image candidate frame is not covered. The part covered by the visible light image candidate frame is added to the visible light image candidate frame to obtain the visible light image region feature map and the infrared thermal image region feature map; the cross-modal attention fusion module is used to query the infrared thermal image region feature map vector, using the visible light image region feature map as the key vector and the value vector, referring to the self-attention module, the visible light image region feature map is fused into the infrared thermal image region feature map according to the similarity between the features of each region, and the color image feature enhancement is obtained. The classification and regression module is used to perform convolution calculation on the thermal image features enhanced by the color image features and the visible light image region feature maps to obtain the target detection results, wherein the target detection results include: Category and candidate box offsets.

Among them, the data input module is also used to determine the category and position of the training target; the system also includes: a loss calculation module, which is used to calculate the two tasks of the model in box regression and box classification according to the target detection result and the training target by using the loss function The comprehensive prediction error in , returns the gradient of the error, updates the model parameters, performs model training, and iterates continuously. The prediction error of the model will continue to decrease until convergence, and a model that can be applied and deployed is obtained.

The deformable feature extractor module includes: a first deformable feature extractor for extracting image features of visible light images to obtain visible light image feature maps; a second deformable feature extractor for extracting image features of infrared thermal images , the infrared thermal image feature map is obtained; the visible light image feature map and the infrared thermal image feature map have the same size.

其中，

为常规卷积操作公式，x表示输入特征图，y表示输出特征图，p是当前待计算的像素点位置(w₀,h₀)，k表示卷积范围内的位置序号，p_k是相对p的位置偏移，w_k表示k点位置所对应的权重，Δp_k表示卷积中k点额外增加的位置偏移量，Δm_k表示卷积中k点的额外权重。Among them, the deformable convolution formula is:

in,

It is a conventional convolution operation formula, x represents the input feature map, y represents the output feature map, p is the current pixel position to be calculated (w ₀ , h ₀ ), k represents the position number within the convolution range, and p _k is the relative The position offset of p, w _k represents the weight corresponding to the position of the k point, Δp _k represents the additional position offset of the k point in the convolution, and Δm _k represents the additional weight of the k point in the convolution.

Wherein, the first deformable feature extractor and the second deformable feature extractor learn w _k , Δp _k and Δm _k independently, respectively; or the first deformable feature extractor and the second deformable feature extractor learn w _k independently, respectively , shared learning Δp _k and Δm _k .

Wherein, the candidate frame extraction network includes: a first candidate frame extraction network, which is used for connecting the first deformable feature extractor to extract visible light image candidate frames in which objects exist in the visible light image feature map; the second candidate frame extraction network is used for connecting The second deformable feature extractor extracts the infrared thermal image candidate frame of the object in the infrared thermal image feature map.

Among them, the candidate frame complementary module is specifically used to add the part of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image candidate frame, and to add the infrared thermal image candidate frame that is not covered by the visible light image candidate frame. The obtained part is added to the visible light image candidate frame. According to the selected candidate frame, the regional features of different corresponding positions and sizes on the initial feature map are extracted, and the regional features are unified to the same size through the regional pooling layer to obtain the size. The same visible light image area feature map and infrared thermal image area feature map.

Among them, the cross-modal attention fusion module and the cross-modal attention fusion module are specifically used to reduce the dimension of the infrared thermal image region feature map and the visible light image region feature map through independent convolution, and calculate the infrared thermal image region feature map and the two similar relationships between the features of each region in the feature map of the visible light image region to obtain a relationship matrix, and normalize the weights of the similarity. The dual-modal complementary enhanced region features are output, and the thermal image features enhanced by the color image features are obtained.

Among them, the cross-modal attention fusion module also adds or merges the thermal image features enhanced by the color image features with the infrared thermal image region feature maps; the classification and regression module is also used to enhance the color image features. The thermal image features of the infrared thermal image are added or combined with the infrared thermal image region feature map and the visible light image region feature map to perform convolution calculation to obtain the target detection result, wherein the target detection result includes: the category of each region and the candidate frame offset quantity.

It can be seen that, through the multi-spectral target detection guidance system provided by the present invention, an all-weather end-to-end multi-modal/multi-spectral target detection guidance system is constructed by combining visible light color images and infrared thermal images, so as to solve the problem of the existing guidance system. Issues that are not supported or perform poorly in no-light, low-light, or high-light scenes.

Description of drawings

In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings used in the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without any creative effort.

1 is a schematic structural diagram of a multispectral target detection blind guidance system provided by an embodiment of the present invention;

2 is a schematic structural diagram of a specific structure of a multispectral target detection blind guidance system provided by an embodiment of the present invention;

FIG. 3 is a schematic diagram of a cross-attention fusion module provided by an embodiment of the present invention.

Detailed ways

Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be more thoroughly understood, and will fully convey the scope of the present disclosure to those skilled in the art.

The core of the present invention is:

Existing multispectral/multimodal object detection systems generally assume that the color image and thermal image of the same scene are perfectly aligned, but this is not the case, and the images of the two modalities often have positional offsets. This wrong assumption will lead to errors or even failure of the detection system. Since the fusion of multimodal data is currently performed in a pixel-by-pixel manner, it not only reduces the alignment robustness but also affects the effectiveness of complementary feature fusion. The present invention aims to propose a new solution to the above-mentioned problems.

The present invention takes into account the positional offset of different modal images in the network design. On the one hand, the network implicitly learns the alignment relationship of the two modal images, so as to avoid possible errors in previous systems; on the other hand, it introduces a sense of A region-of-interest (ROI)-level feature fusion module further improves the robustness to misalignment problems. In addition, this scheme does not require additional marking, which saves costs.

It should be pointed out that the fusion module is the core of the one-step multispectral object detection system, because it determines how a system can use the information of multiple modal images to improve the prediction performance. In previous systems, no matter where the fusion module is located, the fusion methods are very simple, such as addition, concat or weighting of corresponding positions, which do not effectively utilize the complementary information of different modalities , which limits the generalization ability of the model in complex real-world scenarios. For the feature fusion part, the present invention proposes two modules, the candidate frame complementary module and the cross-modal attention module, which make full use of the relevant information of the two modalities and realize a more comprehensive modeling of the relationship between the two modalities. The destination promotes the information exchange of the features of the two modalities in the network, and improves the accuracy and generalization ability of the system through the exchange of presence or absence.

FIG. 1 shows a schematic structural diagram of a multispectral target detection blind guidance system provided by an embodiment of the present invention. Referring to FIG. 1 , the multispectral target detection blind guidance system provided by an embodiment of the present invention includes:

Data input module for acquiring visible light images and infrared thermal images;

The deformable feature extractor module is used to use deformable convolution to extract the image features of the visible light image and the infrared thermal image respectively, and output the visible light image feature map and the infrared thermal image feature map;

The candidate frame extraction network is used to extract the candidate frame of the target object according to the visible light image feature map and the infrared thermal image feature map, and obtain the visible light image candidate frame and the infrared thermal image candidate frame;

The candidate frame complementary module is used to add the part of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image candidate frame, and add the part of the infrared thermal image candidate frame that is not covered by the visible light image candidate frame Add it to the visible light image candidate box to obtain the visible light image region feature map and the infrared thermal image region feature map;

The cross-modal attention fusion module is used to use the infrared thermal image region feature map as the query vector, the visible light image region feature map as the key vector and the value vector, and refer to the self-attention module. The similarity relationship of the infrared thermal image is fused into the feature map of the infrared thermal image area, and the thermal image feature enhanced by the color image feature is obtained;

The classification and regression module is used to convolve the thermal image features enhanced by the color image features and the visible light image region feature maps to obtain the target detection results, wherein the target detection results include: the category of each region and the candidate frame offset quantity.

It can be seen that the present invention provides an all-weather, end-to-end, multimodal/multispectral target detection guidance system that combines visible light color images and infrared thermal images. The present invention does not need position offset supervision information, and is an end-to-end target detection system that aggregates two modal information through a self-attention module at the regional feature level. The model in the present invention is based on the two-stage detection algorithm Faster-RCNN. The feature extraction and region candidate network (RPN) stages have two independent branches, which are respectively used to extract the regional features of the visible light and infrared light images, and then complement each other through the candidate frame. The module and the cross-modal self-attention module fuse the regional features of the two branches, and finally perform the category and coordinate prediction of the region. In a specific embodiment, a general two-stage detection model such as FPN and RFCN can be used as the basic model, which is not limited to Faster-RCNN.

Hereinafter, with reference to FIG. 2 and FIG. 3 , the multispectral target detection guidance system provided by the embodiment of the present invention will be described in detail:

As an optional implementation manner of the embodiment of the present invention, the data input module is further configured to determine the category and position of the training target. In this way, the learning target can be input to the detection network during training, and the model can be trained.

specifically:

Data input module: The data input of the neural network includes two parts, the first is the image input, and the second is the detection target input. The image input is the dual-modal paired image to be detected, one is a visible light color image (with three RGB channels), and the other is an infrared thermal image (a grayscale image, with only one channel of raw data). Assuming that the length and width of the input image are H, W, the image input to the network is [N×3×H×W, N×1×H×W] (in some common implementations, the channel of the infrared thermal image can also be Copy three times to get a three-channel input, that is, N×3×H×W), where N represents the batch size. The detection target input is the category and position of the target object marked in the original image, and the position is represented by the coordinate points [x1, y1, x2, y2] of the bounding rectangle of the target object, where [x1, y1] and [x2] ,y2] are the coordinates of the upper left corner and the lower right corner of the bounding box, respectively. The detection target is first marked manually, and is input to the detection network as a learning target during training for model training.

As an optional implementation of the embodiment of the present invention, the deformable feature extractor module includes: a first deformable feature extractor, configured to extract image features of a visible light image to obtain a visible light image feature map; a second deformable feature extractor , which is used to extract the image features of the infrared thermal image to obtain the infrared thermal image feature map; the visible light image feature map and the infrared thermal image feature map have the same size.

其中，yp＝1K wk·xp+pk为常规卷积操作公式，x表示输入特征图，y表示输出特征图，p是当前待计算的像素点位置(w₀,h₀)，k表示卷积范围内的位置序号，p_k是相对p的位置偏移，w_k表示k点位置所对应的权重，Δp_k表示卷积中k点额外增加的位置偏移量，Δm_k表示卷积中k点的额外权重。As an optional implementation of the embodiment of the present invention, the deformable convolution formula is:

Among them, yp=1K wk xp+pk is the conventional convolution operation formula, x represents the input feature map, y represents the output feature map, p is the current pixel position to be calculated (w ₀ , h ₀ ), k represents the convolution The position number in the range, p _k is the position offset relative to p, w _k represents the weight corresponding to the k point position, Δp _k represents the additional position offset of the k point in the convolution, Δm _k represents the k in the convolution Additional weight for points.

As an optional implementation of the embodiment of the present invention, the first deformable feature extractor and the second deformable feature extractor learn w _k , Δp _k and Δm _k independently; or the first deformable feature extractor and the second deformable feature extractor The deformable feature extractors learn w _k independently and share Δp _k and Δm _{k respectively} .

specifically:

Deformable feature extractor module: Since the image input of the two modalities has a position offset, the present invention uses a deformable convolution in the feature extraction module, allowing the two branch networks to independently extract image features. , feature-level alignment is implicitly implemented. In the current mainstream feature extractors such as Resnet50, convolutions use convolution kernels with relatively fixed geometric structures, such as square 3×3 or 7×7 convolution kernels. Their geometric transformation modeling capabilities are essentially is limited. The deformable convolution adds a displacement variable to the conventional convolution, and the displacement is automatically learned during model training. After the offset, the receptive field of the convolution is no longer square, but becomes arbitrary according to the training data. polygon. In view of the positional offset existing in the multimodal image, the two feature extractors of the present invention can automatically adjust their respective deformable convolutions through training, so that the extracted features can be aligned at the level of the convolution receptive field. At the same time, the method does not need to provide additional supervision information to achieve the calibration of the two modal images, so it can save costs and be easy to implement. Taking the two modal images in the data input module as input, the final output feature map size of the two feature extractors is N×C×H'×W', where H'=H/16, W'=W/ 16, C represents the feature dimension or the number of channels, such as 512 or 2048.

The conventional convolution operation is as formula (1), x represents the input feature map, y represents the output feature map, p is the current pixel position to be calculated (w ₀ , h ₀ ), k represents the position number within the convolution range, for example In the 3×3 convolution, k=9; p _k is the position offset relative to p, and w _k represents the weight corresponding to the position of k point, which is a learnable parameter. Deformable convolution is shown in formula (2), which adds two learnable parameters, Δp _k and Δm _k , on the basis of formula (1). Δp _k represents the additional position offset of k points in the convolution, and Δm _k represents the additional weight of k points in the convolution. As a supplementary explanation, the present invention provides two implementation forms of deformable convolutional networks as implementation examples. The first is that the feature extraction networks of the visible light image and the thermal infrared image are completely independent of each other, and the deformation parameters of the deformable convolution are learned from the respective feature extraction networks, that is, the learnable parameters of the deformable convolution in the two branch networks, w _k , Δp _k , Δm _k are all independent. Second, the feature extraction networks of visible light images and thermal infrared images are partially independent, that is, the main computation of feature extraction is independent of each other, that is, the learning of w _k is independent of each other. However, the deformation parameters Δp _k and Δm _k in the deformable convolution are shared, and the specific latter two deformation parameters are learned from the fusion features of the two modal features as input.

As an optional implementation of the embodiment of the present invention, the candidate frame extraction network includes: a first candidate frame extraction network, configured to connect to the first deformable feature extractor, and extract the visible light image candidate frame of the object in the visible light image feature map; The second candidate frame extraction network is used to connect the second deformable feature extractor to extract the infrared thermal image candidate frame of the object existing in the infrared thermal image feature map.

specifically:

The candidate frame extraction network: Taking the feature map output by the feature extractor as input, the candidate frame extraction network module aims to extract the candidate frame of the object - the prediction of the true bounding rectangle of the target object, regardless of which object it belongs to. kind. Specifically, k anchor boxes of different sizes are generated for each pixel in the feature map, and then the feature maps in the k anchor boxes are input into the extraction network, and the network predicts that each anchor box exists after calculation. The probability of the object - k × 2 classification results, and the offset of the anchor box relative to the true position of the object - k × 4 regression results. For a feature map of size N×C×H’×W’, the extraction network will output N×H’×W’×k×2+N×H’×W’×k×4 results. Finally, after non-maximum suppression and culling, M (usually M=1024) candidate frames with the most likely objects are selected and stored in a matrix of size M×4. The two branch networks will output M relatively independent candidate boxes based on their respective feature maps.

As an optional implementation of the embodiment of the present invention, the candidate frame complementary module is specifically configured to add the part of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image candidate frame, and the infrared thermal image The part of the candidate frame that is not covered by the visible light image candidate frame is added to the visible light image candidate frame. According to the selected candidate frame, the regional features of different positions and sizes on the initial feature map are extracted. The features of each region are unified to the same size to obtain the region feature map of visible light image and the region feature map of infrared thermal image with the same size.

specifically:

Candidate frame complementation module: This module is intended to fuse the candidate frames extracted by the two branch networks, in order to achieve the complementarity of the two modalities at the target object position level. In the case of poor lighting conditions, the candidate frame extracted by the color image branch may be missing, while the candidate frame extracted by the thermal image branch is relatively stable; in addition, there are also cases where the thermal image branch is missed but the color image can be detected. Such as electricity poles with low temperature on cloudy days. This module uses two modalities to obtain more complete candidate boxes. Specifically, this module takes the candidate boxes of the two modalities obtained in the previous stage as input, and adds the parts of the candidate boxes of the two modalities whose IoU is smaller than p∈[0.5, 0.8], such as m, to another module. The number of candidate boxes for each of the two modalities is increased to M'=M+m, and the value of the threshold p may vary slightly according to the specific embodiment. According to the selected candidate frame, the regional features of different corresponding positions and sizes on the initial feature map are extracted, and then the regional features are unified to the same size L×L (such as L=7) through the region pooling layer (ROI pooling). . Finally, the module will output M' regional feature maps of the two modalities, respectively, whose size is 2×N×M’×C×L×L.

As an optional implementation of the embodiment of the present invention, the cross-modal attention fusion module is specifically used to reduce the dimension of the infrared thermal image region feature map and the visible light image region feature map through independent convolution, and calculate the infrared thermal image region The feature map and the two similar relationships between the features of each region in the feature map of the visible light image area are obtained, the relationship matrix is obtained, the weights are normalized for the similarity, the features in the feature map of the visible light image region are convolved, and the matrix of the relationship matrix is obtained. Multiply, output the dual-modal complementary enhanced region features, and obtain the thermal image features enhanced by the color image features.

specifically:

Cross-modal attention fusion module: For the effect of feature model fusion, the present invention introduces a bidirectional feature enhancement module. Due to the different imaging mechanisms of different modalities on the environment, point-to-point feature fusion is difficult to provide substantial help. For example, pedestrians in dark light conditions are mostly dark and invisible on the color image (the entire upper half of life), and only a small part is illuminated (below the calf), then the color image branch network extracts features from these points. It has discriminative power, and it does not play much complementary or enhanced role in the features of thermal images that are fused one by one according to the position coordinates. The regional level fusion is relatively more effective method. Specifically, with the help of the candidate frame complementary module, the color map branch can also obtain the candidate frames of most of the invisible pedestrians in the above example, and the corresponding regional features contain the information of a small part of the illuminated lower legs. Features containing a certain part of an object can be fused into thermal image features as better supplementary information, enhancing the latter's representational power. Therefore, this module adopts a fusion strategy at the regional feature level.

In addition, different objects in a scene often have certain dependencies, and this inter-object relationship can help improve the representation ability of the model, thereby improving the prediction accuracy. For example, when one person appears on a zebra crossing, there are often other people (because of the green light); park benches or trash cans are generally placed outside the lawn, not on the lawn. Therefore, in the process of fusing the regional features of the two modalities, this module also models the relationship between various regions/potential objects, and uses this relationship between objects to enhance the expression of regional features.

Specifically, this module takes the infrared photothermal image area feature and the visible light color image area feature as input, the former as the query vector (query), the latter as the key vector (key) and the value vector (value), referring to self-attention The module fuses the latter into the former according to the similar relationship between the features of each region, so as to realize the enhancement effect of the visible light feature on the infrared light feature, or realize the complementary effect of the dual light feature, as shown in Figure 2. In order to facilitate the description, take N=1, first, the thermal image area feature and the color image area feature are respectively subjected to the same dimension reduction through independent L×L convolution, and become M'×C×1×1 size query and key; Then calculate the similarity relationship between the two regional features in the two modalities, and obtain a relationship matrix of size M'×M'. The method of calculating the similarity can take the negative number of the Euclidean distance, matrix multiplication or other, and then compare the similarity The degree of weight is normalized. In the embodiment, the line-by-line softmax operation is used by default; at the same time, the color image area feature will also undergo a 1×1 convolution to become a value of size M'×C×L×L, and finally pass through and The matrix multiplication of the relation matrix outputs the bimodal complementary enhanced regional features of size M'×C×L×L, that is, the thermal image features enhanced by the color image features are obtained at the regional feature level.

This module models the relationship between candidate regions, not the relationship between all pixels in the original method, so the computational complexity is relatively small (the former O(M'×M')～10 ⁶ , the latter O( H×W×H×W)～10 ⁸ ), the calculation efficiency is high.

As an optional implementation of the embodiment of the present invention, the cross-modal attention fusion module also adds or merges the thermal image features enhanced by the color image features and the infrared thermal image region feature map; the classification and regression module, It is also used to perform convolution calculation on the thermal image feature enhanced by the color image feature, the feature map obtained by adding or combining the infrared thermal image region feature map, and the visible light image region feature map to obtain target detection results, wherein the target detection results include : The category and candidate frame offset of each region. And the output of the cross-modal attention fusion module can be added or concatenated with the thermal image region features.

As an optional implementation of the embodiment of the present invention, the system further includes: a loss calculation module, configured to use a loss function to calculate the comprehensive prediction error of the model in the two tasks of frame regression and frame classification according to the target detection result and the training target , return the gradient of the error, update the model parameters, perform model training, and iterate continuously. The prediction error of the model will continue to decrease until convergence, and a model that can be applied and deployed will be obtained.

specifically:

Loss calculation module: This module takes the prediction result of the entire model and the corresponding training target as input, uses the conventional loss function to calculate the comprehensive prediction error of the model in the two tasks of box regression and box classification, and then returns the gradient of the error, according to A certain learning rate updates the parameters of the entire model to achieve model training. In this way, the prediction error of the model will continue to decrease until convergence, and finally a model that can be applied and deployed will be obtained. During training, you can use Mixed Precision Training for network parameters to reduce video memory and speed up training.

It can be seen that, through the multi-spectral target detection guidance system provided by the embodiment of the present invention, an all-weather guidance system based on the target detection network is constructed by using multi-spectral images, and the deformable convolution is applied in the feature extractor to implicitly learn the multi-spectral images. In order to cope with the possible position offset, the candidate frame complementary module and the cross-modal attention module are proposed to fully utilize the complementary information of multispectral images to achieve more accurate all-weather target detection. At the same time, the robustness to feature misalignment problem is further enhanced. This can support all-weather guidance for blindness, implicitly learn the alignment relationship of multispectral images without additional annotation, save costs, and make full use of the complementary information of multispectral images by the candidate frame complementary module and the cross-modal attention module, and the fusion is more effective. Better results. In addition, the cross-modal attention module has low computational complexity and high efficiency.

The above are merely examples of the present application, and are not intended to limit the present application. Various modifications and variations of this application are possible for those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application shall be included within the scope of the claims of this application.

Claims (9)

1. A multi-spectral target detection blind guide system, comprising:

the data input module is used for acquiring a visible light image and an infrared thermal image;

the deformable feature extractor module is used for respectively extracting the image features of the visible light image and the infrared thermal image by adopting deformable convolution and outputting a visible light image feature map and an infrared thermal image feature map;

the candidate frame extraction network is used for extracting candidate frames of the target object according to the visible light image characteristic diagram and the infrared thermal image characteristic diagram to obtain a visible light image candidate frame and an infrared thermal image candidate frame;

a candidate frame complementing module, configured to add, to the infrared thermal image candidate frame, a portion of the visible light image candidate frame that is not covered by the infrared thermal image candidate frame, and add, to the visible light image candidate frame, a portion of the infrared thermal image candidate frame that is not covered by the visible light image candidate frame, so as to obtain a visible light image area feature map and an infrared thermal image area feature map;

the cross-mode attention fusion module is used for taking the infrared thermal image region feature map as a query vector, taking the visible light image region feature map as a key vector and a value vector, and fusing the visible light image region feature map into the infrared thermal image region feature map according to the similarity relation among the region features by referring to the self-attention module to obtain thermal image features enhanced by color image features;

a classification and regression module, configured to perform convolution calculation on the thermal image features enhanced by the color image features and the visible light image region feature map to obtain a target detection result, where the target detection result includes: the category of each region and the candidate box offset.

2. The system of claim 1,

the data input module is also used for determining the category and the position of a training target;

the system further comprises:

and the loss calculation module is used for calculating the comprehensive prediction error of the model in two tasks of frame regression and frame classification by adopting a loss function according to the target detection result and the training target, returning the gradient of the error, updating the model parameters, training the model, and continuously iterating until the prediction error of the model is continuously reduced until the model is converged to obtain the model capable of being applied and deployed.

3. The system of claim 1, wherein the deformable feature extractor module comprises:

the first deformable feature extractor is used for extracting image features of the visible light image to obtain a visible light image feature map;

the second deformable feature extractor is used for extracting image features of the infrared thermal image to obtain an infrared thermal image feature map;

the visible light image feature map and the infrared thermal image feature map are the same in size.

4. The system of claim 3, wherein the deformable convolution formula is:

wherein,

for the conventional convolution operation formula, x represents the input feature map, y represents the output feature map, and p is the current pixel point position to be calculated (w)₀,h₀) K denotes the position number in the convolution range, p_kIs a positional shift with respect to p, w_kRepresenting the weight, Δ p, corresponding to the position of the k point_kRepresenting an additional added position offset, Δ m, of k points in the convolution_kRepresenting the additional weight of the k points in the convolution.

5. The system of claim 4, wherein the first deformable feature extractor and the second deformable feature extractor each independently learn w_k、Δp_kAnd Δ m_k(ii) a Or the first and second deformable feature extractors learn w independently_kShared learning Δ p_kAnd Δ m_k。

6. The system of claim 3, wherein the candidate box extraction network comprises:

a first candidate frame extraction network, connected to the first deformable feature extractor, for extracting the visible light image candidate frame in which an object exists in the visible light image feature map;

and the second candidate frame extraction network is connected with the second deformable feature extractor and is used for extracting the infrared thermal image candidate frames of the objects in the infrared thermal image feature map.

7. The system according to claim 1, wherein the candidate frame complementing module is specifically configured to add a portion of the visible-light image candidate frame that is not covered by the infrared thermal image candidate frame to the infrared thermal image candidate frame, add a portion of the infrared thermal image candidate frame that is not covered by the visible-light image candidate frame to the visible-light image candidate frame, extract, according to the selected candidate frame, the region features at different sizes of the corresponding positions on the initial feature map, and unify the region features to the same size through a region pooling layer, thereby obtaining the visible-light image region feature map and the infrared thermal image region feature map that have the same size.

8. The system of claim 1,

the cross-modal attention fusion module is specifically configured to perform dimension reduction on the infrared thermal image region feature map and the visible light image region feature map through independent convolution, calculate two similarity relations between the infrared thermal image region feature map and each region feature in the visible light image region feature map, obtain a relation matrix, normalize the similarity by a weight, perform convolution on the features in the visible light image region feature map, multiply the convolved features with the matrix of the relation matrix, output bimodal complementary enhanced region features, and obtain the thermal image features enhanced by the color image features.

9. The system of claim 1,

the cross-modal attention fusion module is used for adding or merging the thermal image characteristics enhanced by the color image characteristics and the infrared thermal image area characteristic map;

the classification and regression module is further configured to perform convolution calculation on the thermal image features enhanced by the color image features, the feature map obtained by adding or merging the infrared thermal image region feature map and the visible light image region feature map to obtain a target detection result, where the target detection result includes: the category of each region and the candidate box offset.

Images