CN115797684A - Infrared small target detection method and system based on context information - Google Patents

Abstract

The invention relates to an infrared small target detection method and system based on context information, belonging to the technical field of computer vision. For the infrared small target dataset, the classification loss function, confidence loss function and position loss function are used to train the small target detection network. Then use the trained small target detection network to extract features from the infrared image and get the feature results. Finally, the extracted features are further fused, and infrared small target detection is performed on the fused features to obtain the final target detection result. At the same time, the invention proposes an infrared small target detection system based on context information. The invention does not rely on additional infrared image denoising, enhancement and other processing modules, and the training process is carried out end-to-end, with simple implementation, high performance and strong robustness. The extra calculation cost of the present invention is extremely low, which is beneficial to realize low-delay and high-speed infrared small target detection, effectively improves the detection rate, and reduces the missed detection rate.

Description

A method and system for infrared small target detection based on context information

technical field

The invention relates to a method and device for detecting small targets in infrared images, in particular to a method and device for detecting small infrared targets based on context information, and belongs to the technical field of computer vision processing.

Background technique

Compared with visible light images, infrared images are not affected by extreme climates and environments, and can be imaged without external light. They have strong detection capabilities and a long range. From infrared monitoring systems to infrared guidance systems, infrared images are used in both civilian and military fields. It has important research and application significance. However, compared with visible light images, infrared images have disadvantages such as poor resolution, blurred imaging, and low signal-to-noise ratio, and small objects in them are easily overwhelmed by noise. Therefore, efficiently detecting small objects in infrared images is a challenging task that has received extensive attention from the signal processing and computer vision communities.

Small object detection is a technique capable of detecting small objects from images. This technology can detect the category and location of small objects under natural lighting conditions. At present, the main small target detection methods are based on deep learning and deep convolutional neural network. This technology is widely used in monitoring security, automatic driving, remote sensing satellite and other fields. According to the definition of the COCO dataset, in principle, objects smaller than 32×32 are called small objects. Small targets account for a low proportion of pixels, and the detection performance is significantly different from that of large targets. If the detection scene is more complex, such as occlusion between targets, the target is occluded by the background, or in dense situations, the impact of small targets will be more severe than that of large targets, and the difficulty of small target detection will be further increased.

Contextual information. Objects usually appear with the corresponding environment. In addition to the characteristics of the object itself, there is also a close connection between the object and the surrounding environment. This information is the so-called feature context information. Because infrared images are seriously affected by noise and clutter, small targets are easily disturbed. Therefore, using other information related to the target in the image and combining the characteristics of small targets to detect objects can effectively improve the detection results and reduce the probability of missing small targets.

Contents of the invention

The purpose of the present invention is to address the defects and deficiencies in the existing infrared small target detection technology, in order to solve the problem that the existing methods do not make full use of the local context information around the target and the global context information in the overall image, and adapt to the feature changes of different types of small targets Insufficient ability, and technical problems such as improper fusion of shallow features and deep features, creatively propose a method and system for infrared small target detection based on context information. The invention of the present invention effectively improves the detection performance for small infrared targets, and has good practical application effect.

In order to achieve the above object, the present invention adopts the following technical solutions to achieve.

A method for detecting infrared small targets based on context information, comprising the following steps:

Step 1: Obtain and process the infrared small target dataset.

Step 2: Use the classification loss function, confidence loss function and position loss function to train the small object detection network.

Step 3: Use the trained small target detection network to extract features from the infrared image and obtain feature results. In this method, there is no need to preprocess the infrared image.

Step 4: The extracted features are further fused, and infrared small target detection is performed on the fused features to obtain the final target detection result.

In order to achieve the purpose of the present invention, the present invention further proposes a small infrared target detection system based on context information, including an image processing module, a target information learning module, a feature extraction module, a feature fusion and a target detection module.

Beneficial effect

Compared with the prior art, the method and system of the present invention have the following advantages:

1. The present invention does not rely on additional infrared image denoising, enhancement and other processing modules, and the training process is carried out end-to-end, with simple implementation, high performance and strong robustness.

2. The extra calculation overhead of the present invention is extremely low, which is beneficial to realize low-latency, high-speed infrared small target detection, effectively improves the detection rate of small targets, reduces the missed detection rate, and improves the accuracy of targets of other scales .

Description of drawings

Figure 1 is a flow chart of the method of the present invention.

Fig. 2 is a schematic diagram of the feature extraction method described in the method of the present invention.

Fig. 3 is a schematic diagram of the frame diagram and internal details of feature fusion described in the method of the present invention.

Figure 4 is a flow chart of the system of the present invention.

Detailed ways

In order to better illustrate the purpose and advantages of the present invention, the content of the invention will be described in detail below in conjunction with the accompanying drawings.

As shown in Figure 1, a small infrared target detection method based on context information includes the following steps:

Step 1: Obtain the infrared small target data set and perform data enhancement processing.

A high-quality dataset is an essential choice for deep learning-based infrared small object detection methods to achieve good performance. However, the existing infrared small target data sets are small in number and small in scale, and the proportion of small targets is not good. Therefore, the present invention first expands the infrared small target data set through data enhancement, thereby enhancing the robustness of the entire detection method.

Specifically, in an image containing small infrared targets, find out the small targets that do not overlap with other targets, and randomly copy and paste them to other positions in the image. Among them, the copied small target does not block other targets, and keeps a distance from other targets.

Furthermore, on the basis of copying and pasting small objects, other data enhancement operations (such as: rotation and translation, scaling and cropping, mosaic enhancement, etc.) can be superimposed, among which mosaic enhancement is preferred.

Step 2: Use the classification loss function, confidence loss function and position loss function to train the small object detection network.

Specifically, let the total loss function L(x,x′) be expressed as:

Among them, x and x′ represent the predicted value and the real value respectively, α _box , α _obj , and α _cls represent the weights of the three loss functions respectively, L _CIoU , L _obj , and L _cls represent the target detection task position loss function, confidence Loss function and classification loss function; k, s ² , and B represent the output feature map, grid, and the number of anchors (ie positions) on each grid, respectively, and I _kij represents the kth output feature map and the i-th grid , Whether the jth anchor box is a positive sample, if it is a positive sample, it is 1, if it is a negative sample, it is 0; α _k is used to balance the output feature weights of different scales.

Step 3: Use the trained small target detection network to perform feature extraction on the infrared image, and obtain the feature result. Also, no preprocessing of infrared images is required.

The ordinary infrared small target detection method mainly extracts the features contained in the object itself, lacks the ability to acquire and analyze context information, and the texture information of small targets is insufficient. When there is no supplementary enhancement of context information, low-contrast infrared images with complex backgrounds are easy to detect. If it is submerged, the different shapes of small targets will also cause certain difficulties in detection.

To this end, this method proposes to first perform feature extraction on the image. During the extraction process, fully considering that features of different shapes have different requirements for context information, through dynamic context information extraction, as shown in Figure 2, a long-distance dependence between each information is established for features. Among them, the position code added in the input part makes up the position information of the feature, and improves the problem of insufficient feature information of the infrared target in the distance. After flattening the input deep feature block into a sequence and introducing position information, it is sent to the multi-head attention mechanism for weighted summation. Then, the residual connection is used to optimize the results and speed up the convergence, through two fully connected layers and residual connection again. The follow-up is a layer of deformable convolution, and a bias term is added at the same time of convolution to ensure that when there are objects of different shapes and sizes at the same position, the characteristics of the objects can also be well expressed.

Specifically, the process of dynamic context information extraction is as follows:

For the input feature F, its feature size is C×H×W, where C represents the number of channels, H represents the height, and W represents the width; given the size P of the block, divide C×H×W into N P× P×C blocks, P means block.

After obtaining N blocks, linearly transform them into feature vectors of N lengths, and add a flag vector x _p at the starting position of the vector;

F ₁ =E+F ₀

表示第N个区块，W_N为权重参数，Concat[]为拼接操作。Among them, F ₀ represents the output vector result,

Indicates the Nth block, W _N is the weight parameter, and Concat[] is the splicing operation.

The final F ₀ obtained is the output result of block embedding. After obtaining the F ₀ blocks, the embedded features still lack the relative position information between the blocks. Therefore, add the position coding information E and F ₀ to get F ₁ , and F ₁ represents the result after adding the position information.

The _F1 after embedding position information is multiplied by three different parameter matrices respectively, and mapped into query matrix, queried key-value matrix and value matrix. After being processed by the attention mechanism, multiple attention results are obtained to represent different contextual information in the image. These attention results are concatenated and normalized to obtain the final summary result of contextual information:

head _i = Attention(F ₁ W _q ; F ₁ W _k ; F ₁ W _v )

F _M =Concat[head _i ; head _i ; head _i ; . . . ; head _i ] W _M

表示缩放因子；F₁表示添加位置信息后的结果，W_q、W_k、W_v、W_M是可学习参数矩阵，Softmax表示进行Softmax操作，head_i表示多个注意力结果的输出，F_M表示多头注意力输出特征。Concat表示相加操作。Among them, Attention() represents the operation of the attention mechanism, Q, K, and V represent the query matrix, the queried key-value matrix and the value matrix, respectively, T represents the transpose operation,

Represents the scaling factor; F ₁ represents the result after adding position information, W _q , W _k , W _v , W _M are learnable parameter matrices, Softmax represents the Softmax operation, head _i represents the output of multiple attention results, F _M Denotes multi-head attention output features. Concat represents an addition operation.

The feed-forward neural network includes two layers of fully connected layers. The multi-head attention output feature F _M after residual normalization is mapped to the high-dimensional space by the first fully connected layer, and the low-dimensional space is mapped by the second fully connected layer. , to further retain useful information, the process is:

F ₂ =F _M [0]+F ₁

X＝F ₂ W _fc1 W _fc2 +F ₁

Among them, F ₂ represents the result after the residual, F _M [0] is the flag bit vector, X is the output result, W _fc1 and W _fc2 are the weights of the two fully connected layers.

After processing the context information, the output result X dynamically adjusts the effective information through deformable convolution, and links the relationship between different small targets and context information:

Among them, Y(p ₀ ) represents the output result of deformable convolution, X and Y are the input feature map and output feature map respectively, p ₀ represents the position in the output feature, p _n represents the adjacent position, and R represents the range of real numbers. The function W() represents the weight at p _n . p _n is the offset value, learned by parallel convolution from the input features.

Step 4: The extracted features are further fused, and infrared small target detection is performed on the fused features to obtain the final target detection result.

Affected by noise, the feature performance of different small targets in the infrared image is very different, which is a test of the feature fusion ability of the model. However, the ordinary object detection model simply performs operations such as upsampling, convolution, and connection features, and does not analyze the positional features of objects from the spatial dimension and the semantic features of objects from the channel dimension, or can only fuse obvious features in the image, but ignores them. The small target information is lost, which leads to low final detection accuracy of small targets.

Therefore, in this method, after feature extraction is performed on the image, the extracted features are subjected to feature fusion, and target detection is performed on the fused features. In the process of feature fusion, the multi-information fusion layer is used to aggregate channel and spatial information in multiple features.

The aggregated features greatly improve the expression of the object's location information and semantic information. In the feature fusion, a new feature scale is added and fused to complement the deep small target features, which is conducive to enriching the detailed features of small targets.

In order to increase the spatio-temporal information of the object as much as possible during feature fusion, more target features are retained. As shown in Figure 3, the multi-information fusion layer performs information fusion operations in each feature scale.

The multi-information fusion module (MFM) fuses the information of different layers through multiple residual structures. Its structure is shown in Figure 3(c), which includes three parts. The first one is the IC layer, as shown in Figure 3(b). Responsible for refining the feature information, and then perform global pooling and maximum pooling from the channel level, and organize the information through the fully connected layer with shared weights, multiply and add, and then normalize through the softmax function to obtain the extracted channel information. Multiplied with the input information to achieve the effect of enhancing the channel information.

After the channel information extraction is enhanced, continue to perform global pooling and maximum pooling on each position of the image respectively. After adding, 7×7 convolution superimposition features can be taken and normalized by the softmax function to achieve the effect of enhancing position information. . Finally, 1×1 convolution can be used to further integrate channel and spatial information.

The deep features contain rich semantic information, but most of them are the semantic information of the target. After multiple downsampling operations, the relevant features of the small target are easily covered by noise and difficult to locate, while the shallow features have rich texture information and position of the small target. information. At the same time, in order to effectively use shallow features to enhance the details of small objects and supplement the location information of small objects, an additional feature scale is added to focus on small objects, and a detection head is added to output detection results. The names of related structures are shown in Figure 3(a). The outputs of the dynamic context information extraction module and the subsequent three multi-information fusion modules (MFM) are T5, T4, T3, and T2, and the sizes of these outputs are 1/1 of the original image respectively. 32, 1/16, 1/8, 1/4. Features of the same size connected with T5, T4, T3 are denoted as R4, R3, R2.

In this method, when the feature map is processed to the T3 layer, the feature is continuously up-sampled, and the T2 layer is added after the up-sampling, and the T2 layer is connected to the features of the same size as the second layer of the backbone network. Improve the characterization ability of small target details, transfer shallow detail information, and connect small target detection heads after the T2 layer to reduce the feature coupling between small targets and other targets on the same layer, reduce the missed detection rate of small targets, and improve the detection accuracy. The probability of small targets alleviates the poor accuracy caused by excessive scale. In order to correspond to the channel of the subsequent network, the R2 layer is added after the T2 layer, and is connected with the T3 layer features of the same dimension.

In order to achieve the stated purpose of the present invention, the present invention further proposes an end-to-end infrared small target detection system based on context information, as shown in Figure 4, including an infrared image processing module 10, a target information learning module 20, and a feature extraction module 30. Feature fusion and object detection module 40.

Wherein, the infrared image processing module 10 is used for processing the infrared image data set used for training the small target detection model. This module can increase the number of small targets, enrich the changing scenarios of the dataset, and enhance the robustness of the model.

The small target information learning module 20 is used to guide the small target detection model to learn robust image features. This module uses the infrared small target data set to use the information to learn and train the model, and outputs the trained small target detection model.

The image feature extraction module 30 uses the dynamic context information extraction module to extract the target surrounding information and global related information in the image features, and adapt to the contour changes of different small targets. Extract stable and clean small target features from infrared images to achieve precise infrared small target detection.

The feature fusion and target detection module 40 can fuse the extracted features. The category, position, size, and shape of the target of interest are identified and extracted from the fused image features, and the final infrared small target detection result is obtained.

The connections between the above modules are as follows:

The output end of the infrared image processing module 10 is connected with the input end of the small target information learning module 20 .

The output end of the small target information learning module 20 is connected to the input end of the image feature extraction module 30 .

The output terminal of the image feature extraction module 30 is connected to the input terminal of the feature fusion and object detection module 40 .

Claims (8)

1. a kind of infrared small target detection method based on context information, it is characterized in that, comprising the following steps: Step 1: Obtain the infrared small target data set and perform data enhancement processing; Step 2: Use the classification loss function, confidence loss function and position loss function to train the small target detection network; Step 3: Use the trained small target detection network to extract features from the infrared image and obtain feature results; First, feature extraction is performed on the image; in the process of extraction, the long-distance dependence between each information is established for the feature through the extraction of dynamic context information; the input deep feature block is flattened into a sequence and the location information is introduced, and then sent to the multi-head The attention mechanism performs weighted summation; Then, use the residual connection to optimize the result and speed up the convergence, pass through two fully connected layers and connect the residual again; the follow-up is a layer of deformable convolution, adding a bias term while convolution; Step 4: The extracted features are further fused, and infrared small target detection is performed on the fused features to obtain the final target detection result. 2. a kind of infrared small target detection method based on contextual information as claimed in claim 1, is characterized in that, when step 1 carries out data enhancement processing, in the image that contains infrared small target, find out and do not overlap with other targets small target, and randomly copy and paste it to other positions in the image; among them, the copied small target does not block other targets, and keeps a distance from other targets. 3. A method for detecting infrared small targets based on context information as claimed in claim 2, characterized in that, on the basis of copying and pasting small targets, other data enhancement operations are further superimposed, including rotation and translation, zooming and cutting, mosaic enhanced. 4. A kind of infrared small target detection method based on context information as claimed in claim 1, is characterized in that, in step 2, let total loss function L (x, x ') be expressed as:

Among them, x and x′ represent the predicted value and the real value respectively, α _box , α _obj , and α _cls represent the weights of the three loss functions respectively, L _CIoU , L _obj , and L _cls represent the target detection task position loss function, confidence Loss function and classification loss function; k, s ² , and B represent the number of output feature maps, grids, and positions on each grid, respectively, and I _kij represents the kth output feature map, the i-th grid, and the j-th Whether an anchor box is a positive sample, if it is a positive sample, it is 1, if it is a negative sample, it is 0; α _k is used to balance the output feature weights of different scales. 5. a kind of infrared small target detection method based on context information as claimed in claim 1, is characterized in that, in step 3, the process of dynamic context information extraction is as follows: For the input feature F, its feature size is C×H×W, where C represents the number of channels, H represents the height, and W represents the width; given the size P of the block, divide C×H×W into N P× P×C block, P means block; After obtaining N blocks, linearly transform them into feature vectors of N lengths, and add a flag vector x _p at the starting position of the vector;

F ₁ =E+F ₀ Among them, F ₀ represents the output vector result,

Indicates the Nth block, W _N is the weight parameter, Concat[] is the splicing operation; the final F ₀ is the output result of block embedding; Add the location coding information E and add it to F ₀ to get F ₁ , and F ₁ represents the result after adding the location information; F ₁ after embedding position information is multiplied by three different parameter matrices, and mapped to query matrix, queried key-value matrix and value matrix; after being processed by the attention mechanism, multiple attention results are obtained, which are used to represent the image Different context information in different context information; these attention results are stitched together and standardized to get the final summary result of context information:

head _i = Attention(F ₁ W _q ; F ₁ W _k ; F ₁ W _v ) F _M =Concat[head _i ; head _i ; head _i ; . . . ; head _i ] W _M Among them, Attention() represents the operation of the attention mechanism, Q, K, and V represent the query matrix, the queried key-value matrix and the value matrix, respectively, T represents the transpose operation,

Represents the scaling factor; F ₁ represents the result after adding position information, W _q , W _k , W _v , W _M are learnable parameter matrices, Softmax represents the Softmax operation, head _i represents the output of multiple attention results, F _M Represents multi-head attention output features; Concat represents addition operation; The feed-forward neural network includes two layers of fully connected layers. The multi-head attention output feature F _M after residual normalization is mapped to the high-dimensional space by the first fully connected layer, and the low-dimensional space is mapped by the second fully connected layer. , to further retain useful information, the process is: F ₂ =F _M [0]+F ₁ X＝F ₂ W _fc1 W _fc2 +F ₁ Among them, F ₂ represents the result after the residual, F _M [0] is the flag bit vector, X is the output result, W _fc1 and W _fc2 are the weights of the two fully connected layers; After processing the context information, the output result X dynamically adjusts the effective information through deformable convolution, and links the relationship between different small targets and context information:

Among them, Y(p ₀ ) represents the output result of deformable convolution, X and Y are the input feature map and output feature map respectively, p ₀ represents the position in the output feature, p _n represents the adjacent position, and R represents the range of real numbers; the function W() denotes the weight at p _n ; p _n is the offset value, learned by parallel convolution from the input features. 6. A kind of infrared small target detection method based on contextual information as claimed in claim 1, it is characterized in that, step 4 is in feature fusion process, utilizes channel, spatial information in multi-information fusion layer aggregation multiple features, multi-information The fusion layer performs information fusion operation in each feature scale; Among them, the multi-information fusion module fuses the information of different layers through multiple residual structures, which consists of three parts. The first is the IC layer, which is responsible for refining the feature information, and then performs global pooling and maximum pooling from the channel level. , and organize the information through the fully connected layer of shared weights, multiply and add, and then normalize through the softmax function to obtain the extracted channel information and multiply it with the input information; After the channel information extraction is enhanced, continue to perform global pooling and maximum pooling on each position of the image respectively, and after the addition, take the convolution superposition feature and normalize it through the softmax function to achieve the effect of enhancing the position information; finally, after Convolution, integrating channel and spatial information. 7. A kind of infrared small target detection method based on contextual information as claimed in claim 6, is characterized in that, increases a feature scale, is used for paying special attention to small object; Increases a detection head to output detection result; Among them, the output of the dynamic context information extraction module and the subsequent three multi-information fusion modules are T5, T4, T3, and T2, and the sizes of these outputs are 1/32, 1/16, 1/8, and 1/4 of the original image, respectively. , the features of the same size connected with T5, T4, T3 are denoted as R4, R3, R2; When the feature map is processed to the T3 layer, continue to upsample the feature, and add the T2 layer after the upsampling, and connect the T2 layer with the features of the same size as the second layer of the backbone network; connect the small target detection head after the T2 layer, Add the R2 layer after the T2 layer, and connect with the T3 layer features of the same dimension. 8. a kind of infrared small target detection system based on the context information of method described in claim 1 is characterized in that, comprises infrared image processing module (10), target information learning module (20), feature extraction module (30), Feature fusion and object detection module (40); Wherein, the infrared image processing module (10) is used for processing the infrared image dataset used for training the small target detection model; The small target information learning module (20) is used to guide the small target detection model to learn robust image features; this module utilizes the infrared small target data set to use the information learning training model, and outputs the trained small target detection model; The image feature extraction module (30) uses the dynamic context information extraction module to extract target surrounding information and global related information in the image feature, and adapt to the contour changes of different small targets; extract stable and clean small target features on the infrared image; The feature fusion and target detection module (40) fuses the extracted features, identifies and extracts the category, position, size and shape of the target of interest from the fused image features, and obtains the final infrared small target detection result; The connections between the above modules are as follows: The output end of the infrared image processing module (10) is connected with the input end of the small target information learning module (20); The output end of the small target information learning module (20) is connected with the input end of the image feature extraction module (30); The output end of the image feature extraction module (30) is connected with the input end of the feature fusion and target detection module (40).

Images