CN116543178A - Infrared small target tracking method and system based on local tracker - Google Patents

Abstract

The invention discloses an infrared small target tracking method and system based on a local tracker, wherein the method comprises the following steps: step (1): generating a filter corresponding to each convolution layer in the network based on the pre-trained multi-convolution layer network; step (2): generating a plurality of local trackers, each local tracker including a multi-layer convolutional layer network, a plurality of filters, a fusion module, and a maximum solution module; step (3): training the local tracker to obtain a trained local tracker; step (4): dividing the area to be identified into a plurality of local areas, deploying the trained local tracker in different local areas to track the target, and if the local tracker is positioned to the target in the tracking process, moving the local tracker along with the target, and continuing to track the target in the subsequent frame until the target cannot be tracked. The invention can realize long-term tracking of frequently discontinuous moving targets.

Description

Infrared small target tracking method and system based on local tracker

Technical Field

The invention relates to the technical field of infrared identification, in particular to an infrared small target tracking method and system based on a local tracker.

Background

The infrared weak and small point-shaped target tracking is to firstly give the target position in an initial frame from an infrared image sequence, then estimate the state of the target in the motion process, and predict the position of the target in an infrared scene in a subsequent frame so as to realize tracking. Because the infrared weak and small target tracking algorithm has high requirements on stability, the method mainly comprises a model generation method and a model discrimination method.

The key to long-term tracking of small infrared targets is the difficulty in tracking the discontinuous movement of the target due to out-of-view or occlusion. Existing long-term tracking methods follow two typical strategies. The first strategy uses one local tracker for smooth tracking and another re-detector to detect the target when it is lost. While it may take advantage of temporal backgrounds, such as historical appearance and location of objects, one potential limitation of this strategy is that local trackers tend to erroneously identify nearby interferers as objects, rather than activating a re-detector when the real object is not in view. Another long-term tracking strategy is to perform global tracking on the target in the whole image instead of performing local tracking based on the previous tracking result, but the existing global tracking strategy cannot effectively utilize the time background, and most algorithms search the target in a local area, so that the problem that the target frequently and discontinuously moves in a long-term tracking task cannot be solved.

Disclosure of Invention

The invention aims to: aiming at the problems existing in the prior art, the invention provides a local tracker-based infrared small target tracking method and system capable of achieving long-term tracking of a target frequently and discontinuously.

The technical scheme is as follows: the infrared small target tracking method based on the local tracker comprises the following steps:

step (1): generating a filter corresponding to each convolution layer in the network based on the pre-trained multi-convolution layer network;

step (2): generating a plurality of local trackers, wherein each local tracker comprises a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for outputting a corresponding feature map on each convolution layer after an input image frame is processed, each filter is connected with the corresponding convolution layer of the multi-convolution layer network and is used for filtering the feature map output by the corresponding convolution layer, the fusion module is connected with all the filters and is used for fusing the outputs of all the filters to obtain a fusion feature map, and the maximum value solving module is connected with the fusion module and is used for solving the maximum value in the fusion feature map and outputting the position corresponding to the maximum value as a target position;

step (3): training the local tracker to obtain a trained local tracker;

step (4): dividing the area to be identified into a plurality of local areas, deploying the trained local tracker in different local areas to track the target, and if the local tracker is positioned to the target in the tracking process, moving the local tracker along with the target, and continuing to track the target in the subsequent frame until the target cannot be tracked.

Further, the step (1) specifically includes:

step (1-1): acquiring a plurality of infrared images and generating corresponding Gaussian labels;

step (1-2): inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network;

step (1-3): the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () representing the discrete fourier transform, X ^k ＝Ｆ(x ^k ) Feature map x representing output of a kth convolutional layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

Further, the filter adopts the following formula to realize the filtration:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

Further, the fusion module adopts the following formula to realize fusion:

s.t.∑q _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements of the ith row and jth column of Q are represented.

Further, the filter is updated by the following formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.

The infrared small target tracking system based on the local tracker comprises a plurality of local trackers which are deployed in different local areas, wherein the local trackers are obtained through training and comprise a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for processing an input image frame and then outputting a corresponding characteristic map on each layer of convolution layer, each filter is connected with the corresponding convolution layer of the multi-layer convolution layer network and is used for filtering the characteristic map output by the corresponding convolution layer, the fusion module is connected with all filters and is used for fusing the outputs of all filters to obtain a fusion characteristic map, and the maximum value solving module is connected with the fusion module and is used for solving the maximum value in the fusion characteristic map and outputting the position corresponding to the maximum value as a target position; the filter is generated based on a pre-trained multi-convolution layer network; and in the tracking process, if the target is positioned, the local tracker moves along with the target, and continues to track the target in the subsequent frame until the target cannot be tracked.

Further, the filter is generated by:

acquiring a plurality of infrared images and generating corresponding Gaussian labels;

inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network;

the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () representing the discrete fourier transform, X ^k ＝Ｆ(x ^k ) Feature map x representing output of a kth convolutional layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

Further, the filter adopts the following formula to realize the filtration:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

Further, the fusion module adopts the following formula to realize fusion:

s.t.∑q _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements of the ith row and jth column of Q are represented.

Further, the filter is updated by the following formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.

The beneficial effects are that: compared with the prior art, the invention has the remarkable advantages that: the invention provides a global re-detection function for the local tracker to detect the target, switches between local tracking and global detection according to prior information, and can solve the problem of frequent discontinuous movement of the target in a long-term tracking task.

Drawings

FIG. 1 is a schematic flow chart of a local tracker-based infrared small target tracking method provided by the invention;

FIG. 2 is a schematic diagram of a local tracker in accordance with the present invention;

fig. 3 is a schematic diagram of object recognition according to the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.

Example 1

The embodiment provides an infrared small target tracking method based on a local tracker, as shown in fig. 1, comprising the following steps:

step (1): and generating a filter corresponding to each convolution layer in the network based on the pre-trained multi-convolution layer network.

Wherein the multi-volume lamination network is specifically VGG-Net, and the steps of adopting VGG-Net to generate the filter are as follows: step (1-1): acquiring a plurality of infrared images and generating corresponding Gaussian labels; step (1-2): inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network; step (1-3): the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () tableShowing the discrete Fourier transform, X ^k ＝Ｆ(x ^k ) Feature map x representing output of a kth convolutional layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

Step (2): generating a plurality of local trackers, wherein each local tracker comprises a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for outputting a corresponding feature map on each convolution layer after an input image frame is processed, each filter is connected with the corresponding convolution layer of the multi-layer convolution layer network and used for filtering the feature map output by the corresponding convolution layer, the fusion module is connected with all the filters and used for fusing the outputs of all the filters to obtain a fusion feature map, and the maximum value solving module is connected with the fusion module and used for solving the maximum value in the fusion feature map and outputting the position corresponding to the maximum value as a target position.

The structure of the local tracker is shown in fig. 2, the multi-layer convolution layer network is the pretrained VGG-Net, the filter is generated by the step (1), and the following formula is adopted to realize the filtering:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

The fusion module adopts the following steps:

s.t.∑q _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements representing row i and column j of Q, in effect, each feature map P ^k Can be regarded as a probability map which is defined by a probability distribution +.>This probability distribution represents the probability that the position (i, j) becomes the target center, assuming that the filtered feature map set a= { p ^1，2 ，p ^1，3 ，···，p ^2，3 ，···，P ^n-1，n And n (n-1)/2 feature maps with less noise. Now, our aim is to fuse these filtered response maps. The above function can be changed to

The solution of the equation can be obtained by Lagrangian multiplier method, and the final result is as follows:

the equation is the average of all filtered probabilities, i.e. the final result is enhanced by all filtered feature maps using a weighted sum.

The maximum value solving module solves for the maximum value in the fusion characteristic diagramAnd taking the position corresponding to the maximum value as a target position.

In order to achieve adaptation of the filter to the target appearance, the filter is updated over time, with the update formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.

Step (3): and training the local tracker to obtain a trained local tracker.

During training, a plurality of samples with infrared images and corresponding target positions as labels are adopted, and the training method adopts the existing method and is not repeated.

Step (4): dividing the area to be identified into a plurality of local areas, deploying the trained local tracker in different local areas to track the target, and if the local tracker is positioned to the target in the tracking process, moving the local tracker along with the target, and continuing to track the target in the subsequent frame until the target cannot be tracked. The tracked target is shown in fig. 3.

When the target motion is stationary, the active local tracker (the local tracker positioned to the target) may track the target in successive frames, forming an active local tracking stream. With such a flow, it is possible to easily transfer and utilize time contexts from a plurality of history frames, and perform local tracking in the frames. On the other hand, when the target does not continuously move due to occlusion or disappearance, while the activated local tracker may lose the target, another local tracker close to the target may take over tracking to locate the target, thereby achieving a long-term tracking task in which the target frequently does not continuously move.

Example two

The embodiment provides an infrared small target tracking system based on local trackers, which comprises a plurality of local trackers which are deployed in different local areas, wherein the local trackers are obtained through training and comprise a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for processing an input image frame and then outputting a corresponding feature map on each layer of convolution layer, each filter is connected with the corresponding convolution layer of the multi-convolution layer network and is used for filtering the feature map output by the corresponding convolution layer, the fusion module is connected with all filters and is used for fusing the outputs of all filters to obtain a fusion feature map, and the maximum value solving module is connected with the fusion module and is used for solving the maximum value in the fusion feature map and outputting the position corresponding to the maximum value as a target position; the filter is generated based on a pre-trained multi-convolution layer network; and in the tracking process, if the target is positioned, the local tracker moves along with the target, and continues to track the target in the subsequent frame until the target cannot be tracked.

Further, the filter is generated by:

acquiring a plurality of infrared images and generating corresponding Gaussian labels;

inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network;

the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () representing the discrete fourier transform, X ^k ＝Ｆ(x ^k ) Feature map x representing output of a kth convolutional layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

Further, the filter adopts the following formula to realize the filtration:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

Further, the fusion module adopts the following formula to realize fusion:

s.t.∑q _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements of the ith row and jth column of Q are represented.

Further, the filter is updated by the following formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.

It should be noted that, in the embodiment of the determining apparatus, each unit and module included are only divided according to the functional logic, but not limited to the above-mentioned division, so long as the corresponding function can be implemented; in addition, the specific names of the functional units are also only for distinguishing from each other, and are not used to limit the protection scope of the present invention.

Claims (10)

1. The infrared small target tracking method based on the local tracker is characterized by comprising the following steps of:

step (1): generating a filter corresponding to each convolution layer in the network based on the pre-trained multi-convolution layer network;

step (2): generating a plurality of local trackers, wherein each local tracker comprises a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for outputting a corresponding feature map on each convolution layer after an input image frame is processed, each filter is connected with the corresponding convolution layer of the multi-convolution layer network and is used for filtering the feature map output by the corresponding convolution layer, the fusion module is connected with all the filters and is used for fusing the outputs of all the filters to obtain a fusion feature map, and the maximum value solving module is connected with the fusion module and is used for solving the maximum value in the fusion feature map and outputting the position corresponding to the maximum value as a target position;

step (3): training the local tracker to obtain a trained local tracker;

step (4): dividing the area to be identified into a plurality of local areas, deploying the trained local tracker in different local areas to track the target, and if the local tracker is positioned to the target in the tracking process, moving the local tracker along with the target, and continuing to track the target in the subsequent frame until the target cannot be tracked.

2. The local tracker-based infrared small target tracking method of claim 1, wherein: the step (1) specifically comprises the following steps:

step (1-1): acquiring a plurality of infrared images and generating corresponding Gaussian labels;

step (1-2): inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network;

step (1-3): the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () representing the discrete fourier transform, X ^k ＝F(x ^k ) Representation ofFeature map x of the output of the kth convolution layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

3. The local tracker-based infrared small target tracking method of claim 1, wherein: the filter adopts the following formula to realize the filtration:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

4. The local tracker-based infrared small target tracking method of claim 1, wherein: the fusion module adopts the following formula to realize fusion:

s.t.Σq _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements of the ith row and jth column of Q are represented.

5. The local tracker-based infrared small target tracking method of claim 1, wherein: the filter is updated by adopting the following formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.

6. An infrared small target tracking system based on a local tracker is characterized in that: the system comprises a plurality of local trackers which are deployed in different local areas, wherein the local trackers are obtained through training and comprise a multi-layer convolution layer network, a plurality of filters, a fusion module and a maximum value solving module, the multi-layer convolution layer network is used for outputting a corresponding characteristic image on each layer of convolution layer after an input image frame is processed, each filter is connected with the corresponding convolution layer of the multi-convolution layer network and is used for filtering the characteristic image output by the corresponding convolution layer, the fusion module is connected with all the filters and is used for fusing the outputs of all the filters to obtain a fusion characteristic image, and the maximum value solving module is connected with the fusion module and is used for solving the maximum value in the fusion characteristic image and outputting the position corresponding to the maximum value as a target position; the filter is generated based on a pre-trained multi-convolution layer network; and in the tracking process, if the target is positioned, the local tracker moves along with the target, and continues to track the target in the subsequent frame until the target cannot be tracked.

7. The local tracker-based infrared small target tracking system of claim 6, wherein: the filter is generated by the steps of:

acquiring a plurality of infrared images and generating corresponding Gaussian labels;

inputting the infrared image into a pre-trained multi-convolution layer network, and generating a characteristic diagram by each convolution layer of the multi-convolution layer network;

the gaussian labels based on the infrared image and the feature map of each convolution layer generate the corresponding filter using the following formula:

in which W is ^k Representing the filter corresponding to the kth convolution layer, y=f (Y) representing the discrete fourier transform of the gaussian tag Y, F () representing the discrete fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k Is a discrete fourier transform of (c), λ represents a regularization parameter,representing the quadratic norm, W represents the matrix formed by all filters.

8. The local tracker-based infrared small target tracking system of claim 6, wherein: the filter adopts the following formula to realize the filtration:

P ^k ＝F ^-1 (X ^k ·W ^k )

wherein P is ^k Representing a filtered characteristic diagram obtained by filtering the characteristic diagram output by the kth convolution layer by a filter, F ^-1 () Representing the inverse discrete Fourier transform, X ^k ＝F(x ^k ) Feature map x representing output of a kth convolutional layer ^k F () represents the discrete fourier transform, W ^k Representing the filter corresponding to the kth convolution layer.

9. The local tracker-based infrared small target tracking system of claim 6, wherein: the fusion module adopts the following formula to realize fusion:

s.t.Σq _ij ＝1

wherein Q represents a fusion feature map, P ^k Represents a filtering characteristic diagram obtained by filtering a characteristic diagram output by a kth convolution layer by a filter, n represents the number of filters, KL represents the Kullback-Leibler divergence,representing P ^k Elements of row i and column j, q _ij Elements of the ith row and jth column of Q are represented.

10. The local tracker-based infrared small target tracking system of claim 6, wherein: the filter is updated by adopting the following formula:

in which W is _t ^k 、The filter corresponding to the kth layer convolution layer at the time t and the time t-1 is shown, gamma is the learning rate,indicating the filter update amount, +.indicates the element multiplication, +.Y indicates the discrete Fourier transform of the Gaussian tag Y, +.>And represents an infrared image acquired at the time t, and lambda represents a regularization parameter.