CN117522923A - Target tracking system and method integrating multi-mode characteristics - Google Patents
Target tracking system and method integrating multi-mode characteristics Download PDFInfo
- Publication number
- CN117522923A CN117522923A CN202311554028.5A CN202311554028A CN117522923A CN 117522923 A CN117522923 A CN 117522923A CN 202311554028 A CN202311554028 A CN 202311554028A CN 117522923 A CN117522923 A CN 117522923A
- Authority
- CN
- China
- Prior art keywords
- feature
- feature map
- attention
- classification
- map
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000007246 mechanism Effects 0.000 claims abstract description 26
- 230000004927 fusion Effects 0.000 claims description 69
- 239000011159 matrix material Substances 0.000 claims description 45
- 238000010586 diagram Methods 0.000 claims description 40
- 238000013527 convolutional neural network Methods 0.000 claims description 25
- 230000005284 excitation Effects 0.000 claims description 25
- 238000012545 processing Methods 0.000 claims description 21
- 239000013598 vector Substances 0.000 claims description 19
- 238000009826 distribution Methods 0.000 claims description 16
- 230000006870 function Effects 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 15
- 238000005728 strengthening Methods 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 14
- 238000003062 neural network model Methods 0.000 claims description 14
- 238000011176 pooling Methods 0.000 claims description 13
- 238000013139 quantization Methods 0.000 claims description 7
- 230000002787 reinforcement Effects 0.000 claims description 7
- 238000012546 transfer Methods 0.000 claims description 7
- 230000003213 activating effect Effects 0.000 claims description 6
- 238000013507 mapping Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 3
- 230000003416 augmentation Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 5
- 238000005516 engineering process Methods 0.000 abstract description 5
- 238000013135 deep learning Methods 0.000 abstract description 4
- 230000010354 integration Effects 0.000 abstract description 2
- 230000000295 complement effect Effects 0.000 description 2
- 230000000875 corresponding effect Effects 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000008034 disappearance Effects 0.000 description 1
- 238000004821 distillation Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000007499 fusion processing Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003252 repetitive effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
- G06V10/451—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters with interaction between the filter responses, e.g. cortical complex cells
- G06V10/454—Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
- G06V10/765—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects using rules for classification or partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10048—Infrared image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Biomedical Technology (AREA)
- Molecular Biology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Biodiversity & Conservation Biology (AREA)
- Image Analysis (AREA)
Abstract
The application relates to the technical field of target tracking, and particularly discloses a target tracking system and method integrating multi-mode features, which are characterized in that the RGB features and the thermal infrared features are respectively extracted from RGB images and thermal infrared images of target tracking by utilizing a deep learning technology, the RGB features and the thermal infrared features are subjected to feature integration in a linear combination mode, then a residual attention mechanism is used for adaptively adjusting feature weights on space dimensions and channel dimensions, and finally whether a target object is separated from a tracking range is judged by a classifier. Therefore, the accuracy of target tracking is improved by effectively fusing the multi-mode characteristics in target tracking, so that a better target tracking effect is achieved.
Description
Technical Field
The present disclosure relates to the field of target tracking technologies, and in particular, to a target tracking system and method that incorporates multi-modal features.
Background
Target tracking is an important problem in the field of computer vision, and is widely applied to the fields of sports event rebroadcasting, security monitoring, unmanned aerial vehicles, unmanned vehicles, robots and the like at present. In the process of target tracking, the target object needs to be identified and tracked, and the position, the gesture and other information of the target object are monitored in real time. Conventional target tracking methods typically utilize features of only one modality for target tracking, such as visible light images. However, in a practical application scenario, it is often difficult to obtain a good tracking effect by using only the features of one mode due to the complexity of the environmental factors and the features of the target itself.
In order to improve the accuracy and the robustness of target tracking, a target tracking method with multi-mode feature fusion is provided. The method effectively fuses the characteristics of different modes, so that richer target information is obtained, and the accuracy and the robustness of target tracking are improved. At present, a common multi-modal feature fusion method is based on weighted fusion. However, the weighted fusion method is highly subjective, and it is difficult to adaptively determine the weight of each modal feature.
Therefore, a target tracking method and method that incorporates multi-modal features is desired.
Disclosure of Invention
The present application has been made in order to solve the above technical problems. The embodiment of the application provides a target tracking system and method integrating multi-mode features, which are characterized in that a deep learning technology is utilized to extract RGB features and thermal infrared features from RGB images and thermal infrared images of target tracking respectively, the RGB features and the thermal infrared features are integrated in a linear combination mode, then a residual attention mechanism is used for adaptively adjusting feature weights in space dimension and channel dimension, and finally whether a target object is separated from a tracking range is judged through a classifier. Therefore, the accuracy of target tracking is improved by effectively fusing the multi-mode characteristics in target tracking, so that a better target tracking effect is achieved.
Accordingly, according to one aspect of the present application, there is provided a target tracking system incorporating multi-modal features, comprising:
the data acquisition module is used for acquiring an RGB image and a thermal infrared image of target tracking in real time;
the RGB image feature extraction module is used for enabling the RGB image to pass through a first convolution neural network model serving as a filter to obtain an RGB feature map;
the thermal infrared image feature extraction module is used for enabling the thermal infrared image to pass through a second convolution neural network model serving as a filter so as to obtain a thermal infrared feature map;
the linear combination module is used for carrying out feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode so as to obtain a multi-mode fusion feature map;
the residual double-attention module is used for enabling the multi-mode fusion feature map to pass through a residual double-attention mechanism model to obtain a classification feature map;
the characteristic strengthening module is used for carrying out autocorrelation strengthening of characteristic value quantization distribution characteristics on the classification characteristic diagram so as to obtain a strengthened classification characteristic diagram;
and the tracking result generation module is used for enabling the enhanced classification characteristic diagram to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the target object is out of a tracking range.
In the above target tracking system with multi-modal feature fusion, the RGB image feature extraction module is configured to: each layer of the first convolutional neural network model used as the filter performs the following steps on input data in forward transfer of the layer: performing convolution processing on the input data based on a two-dimensional convolution check to generate a convolution feature map; carrying out mean pooling processing based on a local feature matrix on the convolution feature map to generate a pooled feature map; non-linear activation is carried out on the characteristic values of all the positions in the pooled characteristic map so as to generate an activated characteristic map; the output of the last layer of the first convolutional neural network model is the RGB feature map, the input from the second layer to the last layer of the first convolutional neural network model is the output of the last layer, and the input of the first layer of the first convolutional neural network model is the RGB image.
In the target tracking system with the multi-mode feature fusion, the thermal infrared image feature extraction module is configured to: and respectively carrying out two-dimensional convolution processing, local feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a filter so as to output the thermal infrared feature map by the last layer of the second convolution neural network model.
In the above target tracking system fusing multi-modal features, the residual dual-attention module includes: the spatial attention unit is used for inputting the multi-mode fusion feature map into a spatial attention module of the residual double-attention mechanism model to obtain a spatial attention map; the channel attention unit is used for inputting the multi-mode fusion feature map into a channel attention module of the residual double-attention mechanism model to obtain a channel attention map; an attention fusion unit for fusing the spatial attention map and the channel attention map to obtain a fused attention map; the activating unit is used for inputting the fusion attention map into a Sigmoid activating function to activate so as to obtain a fusion attention feature map; the attention applying unit is used for calculating the weighted feature map obtained by multiplying the position-by-position points of the fusion attention feature map and the multi-mode fusion feature map; and the residual fusion unit is used for fusing the weighted feature map and the multi-mode fusion feature map to obtain the classification feature map.
In the above target tracking system that merges multi-modal features, the spatial attention unit includes: the space perception subunit is used for carrying out convolution encoding on the multi-mode fusion feature map by using a convolution layer of a space attention module of the residual double-attention mechanism model so as to obtain an initial convolution feature map; a probability subunit, configured to pass the initial convolution feature map through a Softmax function to obtain a spatial attention score map; and the spatial attention applying subunit is used for multiplying the spatial attention score graph and the multi-mode fusion feature graph by position points to obtain the spatial attention graph.
In the above target tracking system that merges multi-modal features, the channel attention unit includes: the channel dimension pooling subunit is used for carrying out global average pooling along the channel dimension on the multi-mode fusion feature map so as to obtain a channel feature vector; a nonlinear activation subunit, configured to pass the channel feature vector through a Softmax activation function to obtain a channel weight feature vector; and the channel attention applying subunit is used for weighting each feature matrix of the multi-mode fusion feature map along the channel dimension by taking the feature value of each position in the channel weight feature vector as a weight so as to obtain the channel attention map.
In the above target tracking system fusing multi-modal features, the feature enhancement module includes: the characteristic squeezing unit is used for passing the classification characteristic map through a characteristic squeezing module based on a convolution layer to obtain a squeezing classification characteristic map; the characteristic excitation unit is used for enabling the squeezing classification characteristic diagram to pass through a characteristic excitation module based on a deconvolution layer to obtain an excitation classification characteristic diagram; the cosine similarity calculation unit is used for calculating cosine similarity between channel feature vectors of every two pixel positions of the excitation classification feature map to obtain a classification feature autocorrelation matrix; the normalization unit is used for normalizing the classification characteristic autocorrelation matrix through a Softmax function to obtain an autocorrelation class attention matrix; and the strengthening unit is used for modeling the relation between any two pixel points in the excitation classification characteristic diagram by utilizing the autocorrelation class attention matrix so as to obtain the strengthening classification characteristic diagram.
In the target tracking system integrating the multi-mode features, the reinforcement unit is configured to: modeling the relation between any two pixel points in the excitation classification feature map by using the autocorrelation class attention matrix according to the following reinforcement formula to obtain the reinforcement classification feature map after the association feature mapping; wherein, the strengthening formula is:
wherein S represents the autocorrelation class attention focusing matrix, F 1 Representing the excitation classification feature map, W representing a matrix of learnable parameters,representing matrix multiplication, F 2 And representing the enhanced classification characteristic diagram after the association characteristic mapping.
In the target tracking system integrating the multi-mode features, the tracking result generating module is configured to: processing the enhanced classification feature map with the classifier in the following classification formula to obtain the classification result; wherein, the classification formula is: o=softmax { (M) c ，B c )|Projeot(F 2 ) }, where Projeot (F) 2 ) Representing the projection of the enhanced classification feature map as a vector, M c Weight matrix of full connection layer, B c Representing the bias matrix of the fully connected layer, softmax representing the normalized exponential function, and O representing the classification result.
According to another aspect of the present application, there is provided a target tracking method fusing multi-modal characteristics, including:
acquiring an RGB image and a thermal infrared image of target tracking in real time;
passing the RGB image through a first convolutional neural network model serving as a filter to obtain an RGB feature map;
the thermal infrared image is passed through a second convolution neural network model serving as a filter to obtain a thermal infrared characteristic diagram;
performing feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode to obtain a multi-mode fusion feature map;
the multi-mode fusion feature map is subjected to a residual double-attention mechanism model to obtain a classification feature map;
performing autocorrelation strengthening of characteristic value quantization distribution characteristics on the classification characteristic map to obtain a strengthened classification characteristic map;
and the enhanced classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the target object is out of the tracking range.
Compared with the prior art, the target tracking system and method integrating the multi-mode features, provided by the application, utilize a deep learning technology to extract RGB features and thermal infrared features from an RGB image and a thermal infrared image of target tracking respectively, perform feature integration on the RGB features and the thermal infrared features in a linear combination mode, then adaptively adjust feature weights in space dimension and channel dimension by using a residual attention mechanism, and finally judge whether a target object is separated from a tracking range by using a classifier. Therefore, the accuracy of target tracking is improved by effectively fusing the multi-mode characteristics in target tracking, so that a better target tracking effect is achieved.
Drawings
The foregoing and other objects, features and advantages of the present application will become more apparent from the following more particular description of embodiments of the present application, as illustrated in the accompanying drawings. The accompanying drawings are included to provide a further understanding of embodiments of the application and are incorporated in and constitute a part of this specification, illustrate the application and not constitute a limitation to the application. In the drawings, like reference numerals generally refer to like parts or steps.
FIG. 1 is a block diagram of a target tracking system incorporating multi-modal features according to an embodiment of the present application.
Fig. 2 is a schematic architecture diagram of a target tracking system incorporating multi-modal features according to an embodiment of the present application.
Fig. 3 is a block diagram of a residual dual attention module in a target tracking system incorporating multi-modal features in accordance with an embodiment of the present application.
Fig. 4 is a block diagram of a spatial attention unit in a target tracking system incorporating multi-modal features in accordance with an embodiment of the present application.
Fig. 5 is a block diagram of a channel attention unit in a target tracking system incorporating multi-modal features according to an embodiment of the present application.
Fig. 6 is a flowchart of a target tracking method incorporating multi-modal features according to an embodiment of the present application.
Detailed Description
Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be apparent that the described embodiments are only some of the embodiments of the present application and not all of the embodiments of the present application, and it should be understood that the present application is not limited by the example embodiments described herein.
FIG. 1 is a block diagram of a target tracking system incorporating multi-modal features according to an embodiment of the present application. Fig. 2 is a schematic architecture diagram of a target tracking system incorporating multi-modal features according to an embodiment of the present application. As shown in fig. 1 and 2, a target tracking system 100 that incorporates multi-modal features according to an embodiment of the present application includes: the data acquisition module 110 is used for acquiring an RGB image and a thermal infrared image of target tracking in real time; an RGB image feature extraction module 120, configured to pass the RGB image through a first convolutional neural network model serving as a filter to obtain an RGB feature map; a thermal infrared image feature extraction module 130, configured to pass the thermal infrared image through a second convolutional neural network model serving as a filter to obtain a thermal infrared feature map; the linear combination module 140 is configured to perform feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination manner to obtain a multi-mode fusion feature map; the residual dual-attention module 150 is configured to pass the multi-mode fusion feature map through a residual dual-attention mechanism model to obtain a classification feature map; the feature enhancement module 160 is configured to perform auto-correlation enhancement of feature value quantization distribution features on the classification feature map to obtain an enhanced classification feature map; the tracking result generating module 170 is configured to pass the enhanced classification feature map through a classifier to obtain a classification result, where the classification result is used to indicate whether the target object deviates from the tracking range.
In the above-mentioned target tracking system 100 that fuses multi-modal features, the data acquisition module 110 is configured to acquire an RGB image and a thermal infrared image of target tracking in real time. As described above in the background art, in order to improve the accuracy and robustness of target tracking, a target tracking method with multi-modal feature fusion is proposed. Accordingly, considering that the RGB image can provide clear color and texture information under good illumination conditions, the method can be used for extracting the appearance characteristics of the target tracking object. The thermal infrared image can provide thermal energy information of the target tracking object, and has advantages for target tracking in low light or complex background. Therefore, the RGB image and the thermal infrared image can complement each other, and the target tracking accuracy is improved. Based on this, in the technical scheme of the application, firstly, an RGB image of target tracking is acquired through a camera, and a thermal infrared image of target tracking is acquired through a thermal infrared instrument.
In the above-mentioned target tracking system 100 with multi-modal feature fusion, the RGB image feature extraction module 120 is configured to pass the RGB image through a first convolutional neural network model as a filter to obtain an RGB feature map. In target tracking, the RGB image contains rich color and texture information, and the appearance features have important significance for target tracking identification. In order to extract the appearance feature information of the RGB image, the RGB image is further feature-mined using a convolutional neural network having good performance in image feature extraction. It should be appreciated that convolutional neural networks (Convolutional Neural Network, CNN) are a deep learning model that is adept at processing image data. By using the CNN model, useful features can be effectively extracted from RGB images and applied to target tracking tasks. Specifically, by inputting the RGB image into a convolutional neural network model, the network extracts local implicit features in the image through a series of convolution, pooling and activation operations to capture information such as edges, textures, shapes and the like of the target, further distinguish tracking targets from backgrounds and provide valuable feature representations.
Accordingly, in one specific example, the RGB image feature extraction module 120 is configured to: each layer of the first convolutional neural network model used as the filter performs the following steps on input data in forward transfer of the layer: performing convolution processing on the input data based on a two-dimensional convolution check to generate a convolution feature map; carrying out mean pooling processing based on a local feature matrix on the convolution feature map to generate a pooled feature map; non-linear activation is carried out on the characteristic values of all the positions in the pooled characteristic map so as to generate an activated characteristic map; the output of the last layer of the first convolutional neural network model is the RGB feature map, the input from the second layer to the last layer of the first convolutional neural network model is the output of the last layer, and the input of the first layer of the first convolutional neural network model is the RGB image.
In the target tracking system 100 with multi-modal feature fusion, the thermal infrared image feature extraction module 130 is configured to pass the thermal infrared image through a second convolutional neural network model serving as a filter to obtain a thermal infrared feature map. It should be appreciated that thermal infrared images are obtained by measuring the thermal energy emitted by the tracked object, which can provide thermal energy distribution information of the object. In the object tracking task, thermal infrared images have advantages for object tracking in low light or complex backgrounds, because it is not affected by lighting conditions and can highlight the thermal energy characteristics of the object. Similarly, the thermal infrared image is input into a second convolutional neural network model, which captures local implicit features in the thermal infrared image through convolution, pooling and activation operations to reflect information such as thermal energy distribution, shape and texture of the target, and for subsequent feature fusion and target tracking tasks, similar to the processing of the RGB image.
Accordingly, in one specific example, the thermal infrared image feature extraction module 130 is configured to: and respectively carrying out two-dimensional convolution processing, local feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a filter so as to output the thermal infrared feature map by the last layer of the second convolution neural network model.
In the target tracking system 100 with multi-modal feature fusion, the linear combination module 140 is configured to perform feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination manner to obtain a multi-modal fusion feature map. Considering that the RGB image and the thermal infrared image represent different sources of information, they provide complementary features in the object tracking task. The advantages of the two modes are comprehensively utilized by fusing the information of the two modes, so that the accuracy of target tracking is improved. Here, it is described. And carrying out feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode, and adjusting according to the characteristics of specific tasks and data sets so as to adapt to different tracking scenes. Meanwhile, through reasonable weight distribution, the fused feature images can better capture key features of the target and provide more accurate target characterization, so that the follow-up target tracking performance is improved.
In the target tracking system 100 that merges the multi-modal features, the residual dual-attention module 150 is configured to pass the multi-modal merged feature map through a residual dual-attention mechanism model to obtain a classification feature map. It should be appreciated that the residual dual-attention mechanism model is a variation of the attention mechanism that is capable of adaptively learning important information in the feature map and enhancing meaningful feature representations. Specifically, the residual dual-attention mechanism model adaptively selects and weights different position features in the feature map through an attention mechanism, and can adjust feature weights of the feature map from two directions of a channel dimension and a space dimension at the same time, so that the model focuses on a target area and important features more, and background interference and irrelevant information are restrained. And by introducing residual connection, the model can effectively transfer and integrate characteristic representations of different levels, reduce the problems of information loss and gradient disappearance, and improve the expression capability and discriminant of the characteristics.
Fig. 3 is a block diagram of a residual dual attention module in a target tracking system incorporating multi-modal features in accordance with an embodiment of the present application. As shown in fig. 3, the residual dual-attention module 150 includes: a spatial attention unit 151, configured to input the multi-modal fusion feature map into a spatial attention module of the residual dual-attention mechanism model to obtain a spatial attention map; a channel attention unit 152, configured to input the multimodal fusion feature map into a channel attention module of the residual dual-attention mechanism model to obtain a channel attention map; an attention fusion unit 153 for fusing the spatial attention profile and the channel attention profile to obtain a fused attention profile; an activating unit 154, configured to activate the fused attention map by inputting a Sigmoid activating function to obtain a fused attention profile; an attention applying unit 155 for calculating a weighted feature map obtained by multiplying the fused attention feature map and the multi-modal fused feature map by the position points; and a residual fusion unit 156, configured to fuse the weighted feature map and the multi-mode fusion feature map to obtain the classification feature map.
Fig. 4 is a block diagram of a spatial attention unit in a target tracking system incorporating multi-modal features in accordance with an embodiment of the present application. As shown in fig. 4, the spatial attention unit 151 includes: a spatial perception subunit 1511, configured to convolutionally encode the multi-modal fusion feature map using a convolution layer of a spatial attention module of the residual dual-attention mechanism model to obtain an initial convolution feature map; a probabilizing subunit 1512, configured to pass the initial convolution feature map through a Softmax function to obtain a spatial attention score map; a spatial attention applying subunit 1513 is configured to multiply the spatial attention score map and the multi-modal fusion feature map by location points to obtain the spatial attention map.
Fig. 5 is a block diagram of a channel attention unit in a target tracking system incorporating multi-modal features according to an embodiment of the present application. As shown in fig. 5, the channel attention unit 152 includes: a channel dimension pooling subunit 1521, configured to pool the multi-modal fusion feature map along a global average of channel dimensions to obtain a channel feature vector; a nonlinear activation subunit 1522, configured to obtain a channel weight feature vector by using a Softmax activation function on the channel feature vector; a channel attention applying subunit 1523, configured to weight each feature matrix of the multi-modal fusion feature map along a channel dimension with a feature value of each position in the channel weight feature vector as a weight to obtain the channel attention map.
In the target tracking system 100 with multi-modal feature fusion, the feature enhancement module 160 is configured to perform auto-correlation enhancement of feature value quantization distribution features on the classification feature map to obtain an enhanced classification feature map. In particular, in the technical scheme of the application, the RGB feature map and the thermal infrared feature map are considered to be fused in a linear combination mode in the feature fusion process. If the distributions of the two feature maps do not match or there is a large difference, outliers and outliers may occur in the fused feature maps. The residual dual-attention mechanism model may contain a large number of parameters and complex computational processes, especially when processing the multi-modal fusion feature map. Such complexity may result in the model being overly sensitive to outliers and outliers, thereby enlarging or introducing it into the classification map. In order to reduce abnormal values and outliers existing in the classification feature map, correlation among feature values of all positions in the classification feature map is considered, namely, the overall quantized distribution of all positions of the classification feature map contains important mode features, so if the feature distribution feature can be quantized by utilizing the feature values of the classification feature map, the feature distribution autocorrelation enhancement can be carried out on the feature distribution feature, the information distillation degree and the certainty of the classification feature map can be improved, and the accuracy of classification results obtained by the classifier of the classification feature map is improved.
Specifically, the feature enhancement module 160 includes: the characteristic squeezing unit is used for passing the classification characteristic map through a characteristic squeezing module based on a convolution layer to obtain a squeezing classification characteristic map; the characteristic excitation unit is used for enabling the squeezing classification characteristic diagram to pass through a characteristic excitation module based on a deconvolution layer to obtain an excitation classification characteristic diagram; the cosine similarity calculation unit is used for calculating cosine similarity between channel feature vectors of every two pixel positions of the excitation classification feature map to obtain a classification feature autocorrelation matrix; the normalization unit is used for normalizing the classification characteristic autocorrelation matrix through a Softmax function to obtain an autocorrelation class attention matrix; and the strengthening unit is used for modeling the relation between any two pixel points in the excitation classification characteristic diagram by utilizing the autocorrelation class attention matrix so as to obtain the strengthening classification characteristic diagram.
In one embodiment of the present application, the reinforcement unit is configured to: and modeling the relation between any two pixel points in the excitation classification characteristic map by utilizing the autocorrelation class attention matrix by utilizing element-by-element multiplication operation so as to obtain the enhanced classification characteristic map after the association characteristic mapping. The correlation feature effectively aggregates the complete information of each target according to the similarity of pixels, and the calculation formula is as follows:
wherein S represents the autocorrelation class attention focusing matrix, F 1 Representing the excitation classification feature map, W representing a matrix of learnable parameters,representing matrix multiplication, F 2 And representing the enhanced classification characteristic diagram after the association characteristic mapping.
That is, in the technical solution of the present application, the classification characteristic map is first information-purified by using a press-excitation mechanism. And in the step of obtaining the excitation classification characteristic diagram, a cosine similarity operation is utilized to obtain a relation matrix among pixels in the excitation classification characteristic diagram, namely, the cosine similarity between channel characteristic vectors of every two pixel positions of the excitation classification characteristic diagram is calculated to obtain a classification characteristic autocorrelation matrix. After dynamically learning the pixel relationship, the classification feature autocorrelation matrix is subjected to a Softmax function to normalize a similarity matrix by using the Softmax so as to obtain an autocorrelation type attention focusing matrix. And modeling the relation between any two pixel points in the excitation classification characteristic map by using the autocorrelation class attention matrix to obtain the enhanced classification characteristic map.
In the target tracking system 100 that merges the multi-modal features, the tracking result generating module 170 is configured to pass the enhanced classification feature map through a classifier to obtain a classification result, where the classification result is used to indicate whether the target object deviates from the tracking range. It should be appreciated that a classifier is a machine learning model that is capable of learning input feature representations and class discriminant rules and outputting corresponding class labels or probability distributions based on the input features. Here, the enhanced classification feature map is processed by a classifier to obtain a classification result, for example, the classification result is a classification label such as "target object is out of tracking range" or "target object is in tracking range". In this way, the system can be assisted in determining if the target deviates from the original tracking area based on the classification results and taking corresponding actions, such as repositioning the target or triggering an alarm, etc.
Accordingly, in one specific example, the tracking result generating module 170 is configured to: processing the enhanced classification feature map with the classifier in the following classification formula to obtain the classification result; wherein, the classification formula is: o=softmax { (M) c ，B c )|Project(F 2 ) Of which Project (F) 2 ) Representing the projection of the enhanced classification feature map as a vector, M c Weight matrix of full connection layer, B c Representing the bias matrix of the fully connected layer, softmax representing the normalized exponential function, and O representing the classification result.
In summary, a target tracking system with multi-mode feature fusion according to an embodiment of the present application is illustrated, which extracts RGB features and thermal infrared features from an RGB image and a thermal infrared image of target tracking respectively by using a deep learning technology, performs feature fusion on the RGB features and the thermal infrared features by using a linear combination manner, then adaptively adjusts feature weights in a space dimension and a channel dimension by using a residual attention mechanism, and finally determines whether a target object is separated from a tracking range by using a classifier. Therefore, the accuracy of target tracking is improved by effectively fusing the multi-mode characteristics in target tracking, so that a better target tracking effect is achieved.
Fig. 6 is a flowchart of a target tracking method incorporating multi-modal features according to an embodiment of the present application. As shown in fig. 6, a target tracking method for fusing multi-modal features according to an embodiment of the present application includes the steps of: s110, acquiring an RGB image and a thermal infrared image of target tracking in real time; s120, passing the RGB image through a first convolution neural network model serving as a filter to obtain an RGB feature map; s130, passing the thermal infrared image through a second convolution neural network model serving as a filter to obtain a thermal infrared characteristic diagram; s140, carrying out feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode to obtain a multi-mode fusion feature map; s150, the multi-mode fusion feature map is subjected to a residual double-attention mechanism model to obtain a classification feature map; s160, carrying out autocorrelation strengthening of characteristic value quantization distribution characteristics on the classification characteristic map to obtain a strengthened classification characteristic map; and S170, the enhanced classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the target object is out of the tracking range.
Here, it will be understood by those skilled in the art that the specific operations of the respective steps in the above-described multi-modal feature-fused object tracking method have been described in detail in the above description of the multi-modal feature-fused object tracking system with reference to fig. 1 to 5, and thus, repetitive descriptions thereof will be omitted.
Claims (10)
1. A target tracking system incorporating multi-modal features, comprising:
the data acquisition module is used for acquiring an RGB image and a thermal infrared image of target tracking in real time;
the RGB image feature extraction module is used for enabling the RGB image to pass through a first convolution neural network model serving as a filter to obtain an RGB feature map;
the thermal infrared image feature extraction module is used for enabling the thermal infrared image to pass through a second convolution neural network model serving as a filter so as to obtain a thermal infrared feature map;
the linear combination module is used for carrying out feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode so as to obtain a multi-mode fusion feature map;
the residual double-attention module is used for enabling the multi-mode fusion feature map to pass through a residual double-attention mechanism model to obtain a classification feature map;
the characteristic strengthening module is used for carrying out autocorrelation strengthening of characteristic value quantization distribution characteristics on the classification characteristic diagram so as to obtain a strengthened classification characteristic diagram;
and the tracking result generation module is used for enabling the enhanced classification characteristic diagram to pass through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the target object is out of a tracking range.
2. The multi-modal feature fused target tracking system of claim 1, wherein the RGB image feature extraction module is configured to: each layer of the first convolutional neural network model used as the filter performs the following steps on input data in forward transfer of the layer:
performing convolution processing on the input data based on a two-dimensional convolution check to generate a convolution feature map;
carrying out mean pooling processing based on a local feature matrix on the convolution feature map to generate a pooled feature map;
non-linear activation is carried out on the characteristic values of all the positions in the pooled characteristic map so as to generate an activated characteristic map;
the output of the last layer of the first convolutional neural network model is the RGB feature map, the input from the second layer to the last layer of the first convolutional neural network model is the output of the last layer, and the input of the first layer of the first convolutional neural network model is the RGB image.
3. The multi-modal feature fused target tracking system of claim 2, wherein the thermal infrared image feature extraction module is configured to: and respectively carrying out two-dimensional convolution processing, local feature matrix-based mean pooling processing and nonlinear activation processing on input data in forward transfer of layers by using each layer of the second convolution neural network model serving as a filter so as to output the thermal infrared feature map by the last layer of the second convolution neural network model.
4. A multi-modal feature fused target tracking system as claimed in claim 3, wherein the residual dual-attention module comprises:
the spatial attention unit is used for inputting the multi-mode fusion feature map into a spatial attention module of the residual double-attention mechanism model to obtain a spatial attention map;
the channel attention unit is used for inputting the multi-mode fusion feature map into a channel attention module of the residual double-attention mechanism model to obtain a channel attention map;
an attention fusion unit for fusing the spatial attention map and the channel attention map to obtain a fused attention map;
the activating unit is used for inputting the fusion attention map into a Sigmoid activating function to activate so as to obtain a fusion attention feature map;
the attention applying unit is used for calculating the weighted feature map obtained by multiplying the position-by-position points of the fusion attention feature map and the multi-mode fusion feature map;
and the residual fusion unit is used for fusing the weighted feature map and the multi-mode fusion feature map to obtain the classification feature map.
5. The multi-modal feature fused target tracking system of claim 4, wherein the spatial attention unit comprises:
the space perception subunit is used for carrying out convolution encoding on the multi-mode fusion feature map by using a convolution layer of a space attention module of the residual double-attention mechanism model so as to obtain an initial convolution feature map;
a probability subunit, configured to pass the initial convolution feature map through a Softmax function to obtain a spatial attention score map;
and the spatial attention applying subunit is used for multiplying the spatial attention score graph and the multi-mode fusion feature graph by position points to obtain the spatial attention graph.
6. The multi-modal feature fused target tracking system of claim 5, wherein the channel attention unit comprises:
the channel dimension pooling subunit is used for carrying out global average pooling along the channel dimension on the multi-mode fusion feature map so as to obtain a channel feature vector;
a nonlinear activation subunit, configured to pass the channel feature vector through a Softmax activation function to obtain a channel weight feature vector;
and the channel attention applying subunit is used for weighting each feature matrix of the multi-mode fusion feature map along the channel dimension by taking the feature value of each position in the channel weight feature vector as a weight so as to obtain the channel attention map.
7. The multi-modal feature fused target tracking system of claim 6, wherein the feature augmentation module comprises:
the characteristic squeezing unit is used for passing the classification characteristic map through a characteristic squeezing module based on a convolution layer to obtain a squeezing classification characteristic map;
the characteristic excitation unit is used for enabling the squeezing classification characteristic diagram to pass through a characteristic excitation module based on a deconvolution layer to obtain an excitation classification characteristic diagram;
the cosine similarity calculation unit is used for calculating cosine similarity between channel feature vectors of every two pixel positions of the excitation classification feature map to obtain a classification feature autocorrelation matrix;
the normalization unit is used for normalizing the classification characteristic autocorrelation matrix through a Softmax function to obtain an autocorrelation class attention matrix;
and the strengthening unit is used for modeling the relation between any two pixel points in the excitation classification characteristic diagram by utilizing the autocorrelation class attention matrix so as to obtain the strengthening classification characteristic diagram.
8. The multi-modal feature fused target tracking system of claim 7, wherein the reinforcement unit is configured to: modeling the relation between any two pixel points in the excitation classification feature map by using the autocorrelation class attention matrix according to the following reinforcement formula to obtain the reinforcement classification feature map after the association feature mapping; wherein, the strengthening formula is:
wherein S represents the autocorrelation class attention focusing matrix, F 1 Representing the excitation classification feature map, W representing a matrix of learnable parameters,representing matrix multiplication, F 2 And representing the enhanced classification characteristic diagram after the association characteristic mapping.
9. The target tracking system for merging multi-modal features according to claim 8, wherein the tracking result generating module is configured to: processing the enhanced classification feature map with the classifier in the following classification formula to obtain the classification result;
wherein, the classification formula is: o=softmax { (M) c ，B c )|Project(F 2 ) Of which Project (F) 2 ) Representing the projection of the enhanced classification feature map as a vector, M c Weight matrix of full connection layer, B c Representing the bias matrix of the fully connected layer, softmax representing the normalized exponential function, and O representing the classification result.
10. The target tracking method integrating the multi-mode features is characterized by comprising the following steps of:
acquiring an RGB image and a thermal infrared image of target tracking in real time;
passing the RGB image through a first convolutional neural network model serving as a filter to obtain an RGB feature map;
the thermal infrared image is passed through a second convolution neural network model serving as a filter to obtain a thermal infrared characteristic diagram;
performing feature fusion on the RGB feature map and the thermal infrared feature map in a linear combination mode to obtain a multi-mode fusion feature map;
the multi-mode fusion feature map is subjected to a residual double-attention mechanism model to obtain a classification feature map;
performing autocorrelation strengthening of characteristic value quantization distribution characteristics on the classification characteristic map to obtain a strengthened classification characteristic map;
and the enhanced classification characteristic diagram is passed through a classifier to obtain a classification result, wherein the classification result is used for indicating whether the target object is out of the tracking range.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311554028.5A CN117522923A (en) | 2023-11-21 | 2023-11-21 | Target tracking system and method integrating multi-mode characteristics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311554028.5A CN117522923A (en) | 2023-11-21 | 2023-11-21 | Target tracking system and method integrating multi-mode characteristics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117522923A true CN117522923A (en) | 2024-02-06 |
Family
ID=89760411
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311554028.5A Withdrawn CN117522923A (en) | 2023-11-21 | 2023-11-21 | Target tracking system and method integrating multi-mode characteristics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117522923A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117611245A (en) * | 2023-12-14 | 2024-02-27 | 浙江博观瑞思科技有限公司 | Data analysis management system and method for planning E-business operation activities |
| CN117611245B (en) * | 2023-12-14 | 2024-05-31 | 浙江博观瑞思科技有限公司 | Data analysis management system and method for planning E-business operation activities |
-
2023
- 2023-11-21 CN CN202311554028.5A patent/CN117522923A/en not_active Withdrawn
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117611245A (en) * | 2023-12-14 | 2024-02-27 | 浙江博观瑞思科技有限公司 | Data analysis management system and method for planning E-business operation activities |
| CN117611245B (en) * | 2023-12-14 | 2024-05-31 | 浙江博观瑞思科技有限公司 | Data analysis management system and method for planning E-business operation activities |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Goh et al. | Micro-expression recognition: an updated review of current trends, challenges and solutions | |
| US10346464B2 (en) | Cross-modiality image matching method | |
| CN108229267B (en) | Object attribute detection, neural network training and region detection method and device | |
| CN109902548B (en) | Object attribute identification method and device, computing equipment and system | |
| CN107239730B (en) | Quaternion deep neural network model method for intelligent automobile traffic sign recognition | |
| CN110717411A (en) | Pedestrian re-identification method based on deep layer feature fusion | |
| Fendri et al. | Fusion of thermal infrared and visible spectra for robust moving object detection | |
| CN111709311A (en) | Pedestrian re-identification method based on multi-scale convolution feature fusion | |
| CN111723822B (en) | RGBD image significance detection method and system based on multi-level fusion | |
| CN114663670A (en) | Image detection method and device, electronic equipment and storage medium | |
| CN112991350B (en) | RGB-T image semantic segmentation method based on modal difference reduction | |
| CN113052185A (en) | Small sample target detection method based on fast R-CNN | |
| Das et al. | Automated Indian sign language recognition system by fusing deep and handcrafted feature | |
| WO2019167784A1 (en) | Position specifying device, position specifying method, and computer program | |
| CN117252904B (en) | Target tracking method and system based on long-range space perception and channel enhancement | |
| JP7327077B2 (en) | Road obstacle detection device, road obstacle detection method, and road obstacle detection program | |
| CN116824625A (en) | Target re-identification method based on generation type multi-mode image fusion | |
| Gao et al. | CI-Net: Contextual information for joint semantic segmentation and depth estimation | |
| US20230196841A1 (en) | Behavior recognition artificial intelligence network system and method for efficient recognition of hand signals and gestures | |
| CN113033587A (en) | Image recognition result evaluation method and device, electronic equipment and storage medium | |
| CN117522923A (en) | Target tracking system and method integrating multi-mode characteristics | |
| CN116110118A (en) | Pedestrian re-recognition and gait recognition method based on space-time feature complementary fusion | |
| CN114494934A (en) | Unsupervised moving object detection method based on information reduction rate | |
| Vaidya et al. | Lightweight Hardware Architecture for Object Detection in Driver Assistance Systems | |
| CN113728355A (en) | Image processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20240206 |