CN114708306A - A single target tracking method, device and storage medium - Google Patents

Abstract

The invention discloses a single-target tracking method, device and storage medium. The method includes: extracting features of a target area and a search area by using a transformer backbone network; using a transformer-based encoding and decoding architecture to fuse the features of the target area and the search area, To serve the subsequent prediction task, M (100) target pre-selection boxes based on the target position of the previous frame are added when the target area feature code and the search area feature code are sent to the transformer decoder together. IoU prediction module: Iteratively optimizes the target prediction frame output by the Transformer decoder structure decoding for N times to obtain the optimized prediction frame, and then calculates the IoU between it and the labeled frame, and selects the top three optimized prediction frames of the IoU to take the average as the final prediction result. . While improving accuracy, speed is also guaranteed.

Description

A single target tracking method, device and storage medium

technical field

The invention belongs to the technical field of target tracking, relates to a single target tracking method, device and storage medium, and in particular relates to a single target tracking method, device and storage medium for IoU prediction based on a transformer.

Background technique

Single object tracking is a very challenging task in computer vision tasks. The purpose of the target tracking task is that given the target information of the first frame, the tracker needs to automatically determine the target state in subsequent frames. Unlike object detection tasks, object features are only available at the inference stage, which means that there is no prior information about the object, such as class and surrounding environment. In recent years, thanks to the rapid development of deep learning, many research results have been achieved in the field of object tracking.

Existing techniques suffer from one shortcoming: However, in the face of a complex environment (e.g., blur, scale changes, color shifts, and fast motion, etc.), most existing trackers still cannot handle these scenes well.

SUMMARY OF THE INVENTION

Purpose: In order to overcome the deficiencies in the prior art, the present invention provides a single-target tracking method, device and storage medium; IoU prediction based on the transformer makes full use of the transformer's powerful local and global information integration capabilities, and also avoids the need for The complex parameter design and post-processing of traditional trackers are eliminated, and the extraction of target features is more efficient.

Technical scheme: in order to solve the above-mentioned technical problems, the technical scheme adopted in the present invention is:

In a first aspect, a single-target tracking method is provided, including:

get video data;

Determine the target area of the first frame of the video data, perform feature extraction on the target area through the transformer backbone network, and encode the transformer encoder to obtain the feature encoding of the target area;

According to the target area of the first frame of the video data, use a Gaussian function to generate M target a priori frames, where M is an integer greater than 3;

Determine the search area according to the target area of the first frame of the video data;

Perform transformer backbone network feature extraction and transformer encoder encoding on the search area to obtain the search area feature encoding;

Input the target area feature encoding, search area feature encoding and M target a priori boxes into the transformer decoder for decoding, and obtain the IoU value of the intersection ratio between the output M target prediction boxes and the annotation boxes;

IoU loss optimizes the network parameters, uses the gradient descent method to iteratively optimize the target prediction frame for N times, and obtains the optimized optimized prediction frame;

Select the optimized prediction frame of the first three IoU values after optimization and take the mean value as the final target tracking frame.

In some embodiments, the determining the target area of the first frame of the video data includes;

In the first frame of the video data, the target is marked by a rectangular frame, and the target area includes the target position and size, which are respectively determined by the position and size of the rectangular frame.

In some embodiments, determining the search area according to the target area of the first frame of the video data includes:

Among them, crop_sz represents the size of the search area cropped in the current frame, w represents the width of the target frame in the previous frame, h represents the height of the target frame in the previous frame, and s represents the search factor.

In some embodiments, as shown in FIG. 2 and FIG. 3 , the target region feature encoding, the search region feature encoding and M target a priori boxes are input into the transformer decoder for decoding, and the output M target prediction boxes and annotation boxes are obtained. The IoU value of the intersection and union ratio between them includes: splicing the target area feature code and the search area feature code in the channel dimension to obtain a feature map, and obtaining M target prediction frames according to the feature map and M target a priori frames, and then after Feedforward neural network (FFN) obtains the IoU value between M target prediction boxes and annotation boxes.

In some embodiments, the IoU loss optimizes network parameters, and uses the gradient descent method to perform N iterations of optimization on the target prediction frame, including:

Among them, A is the prediction frame, and B is the annotation frame.

In some embodiments, M is 100.

In a second aspect, the present invention provides a single target tracking device, including a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is operable according to the instructions to perform the steps of the method according to the first aspect.

In a third aspect, the present invention provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the method in the first aspect.

Beneficial effect: The single target tracking method and system provided by the present invention, compared with other target tracking algorithms based on transformers, add 100 targets based on the target position of the previous frame when the template and the search image are sent to the transformer decoder together The a priori frame is proposed to consider that the state of the target between each frame in the target tracking task is not very large. Therefore, 100 target a priori frames are equivalent to providing the network with a priori of the target position , so as to improve the accuracy and also ensure the speed. Effective use of template frame information gives the model appropriate priors to ensure tracking performance. In addition, it can not only effectively avoid the need to finely design the anchor frame in the traditional anchor frame-based method, but also eliminate the tedious post-processing steps in most target tracking algorithms, improve the tracking accuracy of the model and meet the real-time performance.

Description of drawings

1 is a flowchart of a single-target tracking method according to an embodiment of the present invention;

2 and 3 are schematic diagrams of a network of a single-target tracking system according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

In the description of the present invention, the meaning of several means one or more, the meaning of multiple means two or more, greater than, less than, exceeding, etc. are understood as not including this number, above, below, within, etc. are understood as including this number. If it is described that the first and the second are only for the purpose of distinguishing technical features, it cannot be understood as indicating or implying relative importance, or indicating the number of the indicated technical features or the order of the indicated technical features. relation.

In the description of the present invention, reference to the terms "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" or the like is meant to be used in conjunction with the embodiment. A particular feature, structure, material or characteristic described or exemplified is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

Example 1

A single-target tracking method, comprising:

get video data;

Determine the target area of the first frame of the video data, perform feature extraction on the target area through the transformer backbone network, and encode the transformer encoder to obtain the feature encoding of the target area;

According to the target area of the first frame of the video data, use a Gaussian function to generate M target a priori frames, where M is an integer greater than 3;

Determine the search area according to the target area of the first frame of the video data;

Perform feature extraction on the transformer backbone network for the search area, and encode the transformer encoder to obtain the feature code of the search area;

Input the target area feature encoding, search area feature encoding and M target a priori boxes into the transformer decoder for decoding, and obtain the IoU value of the intersection ratio between the output M target prediction boxes and the annotation boxes;

IoU loss optimizes the network parameters, uses the gradient descent method to iteratively optimize the target prediction frame for N times, and obtains the optimized optimized prediction frame;

Select the optimized prediction frame of the top three IoU values through the network forward calculation and take the mean value as the final target tracking frame.

In some embodiments, the determining the target area of the first frame of the video data includes;

In the first frame of the video data, the target is marked by a rectangular frame, and the target area includes the target position and size, which are respectively determined by the position and size of the rectangular frame.

When the initial frame image passes through the backbone network, it needs to go through a position encoding (Position Embedding) module to save the relative or absolute position of a single pixel of the image in the sequence. The position calculation formula is as follows:

PE _{(pos, 2i)} = sin(pos/10000 ^2i/d )

PE _{(pos, 2i+1} )=cos(pos/10000 ^2i/d )

The transformer encoder structure specifically includes six identical encoders. Each encoder is composed of Multi-Head Self-Attention, Norm layer and Feed-Forward Network. The structural formulas are expressed as follows: Self-Attention formula:

Multi-Head Self-Attention:

MultiHead(Q, K, V) = Concat(head ₁ , ..., head _h )W

Among them, head refers to the result of each Self-Attention, h refers to the number of heads, and W is the weighting matrix.

Feed-Forward Network:

FFW(x)=max(0, xW ₁ +b ₁ )W ₂ +b ₂ .

In some embodiments, according to the target area of the first frame of the video data, the Gaussian function is used to generate M target a priori frames, including: the Gaussian function adopted is as follows:

Among them, μ=0, σ takes 0.05 and 0.5 respectively. In some specific embodiments, M is 100. Generate 100 target a priori boxes relative to the target box from the previous frame.

In some embodiments, determining the search area according to the target area of the first frame of the video data includes:

Among them, crop_sz represents the size of the search area cropped in the current frame, w represents the width of the target frame in the previous frame, h represents the height of the target frame in the previous frame, and s represents the search factor.

Further, in some embodiments, after the search area is determined, the image is set to a fixed size of 320*320 using a function in pytorch; then it is sent to the transformer backbone network for feature extraction.

In some embodiments, the target area feature code, the search area feature code, and the M prediction target a priori boxes are input into the transformer decoder for decoding, to obtain the output IoU value of the intersection ratio between the M prediction target frames and the annotation frame. , including: splicing the feature code of the target area and the feature code of the search area in the channel dimension to obtain a feature map, obtaining M target prediction frames according to the feature map and M target a priori frames, and then obtaining M target prediction frames through a feedforward neural network (FFN) The IoU value between the M target prediction boxes and the annotation boxes.

In some embodiments, the IoU loss optimizes network parameters, and uses the gradient descent method to perform N iterations of optimization on the target prediction frame, including:

Among them, A is the prediction frame, and B is the annotation frame.

The target prediction frame is optimized by N iterations using the gradient descent method, and N is 10 in some embodiments. The specific process is as follows:

Step1: IoU loss optimizes network parameters θ ₀ and sets step size α=1;

经由前馈神经网络得到相对于标准框之间的IoU loss记为L_IoU；Step2: M target prediction boxes are marked as

Obtaining the IoU loss relative to the standard frame via the feedforward neural network is denoted as L _IoU ;

Step3: Calculate

Step4: Update the M target prediction boxes,

Step5: Iterate N times to get

This optimization process only iteratively optimizes the target prediction frame without updating the network parameters.

Finally, after N times of optimization, the top three optimized prediction frames of the IoU value are calculated forward by the network, and the average is taken as the final prediction frame A _final for target tracking.

Example 2

In a second aspect, this embodiment provides a single-target tracking device, including a processor and a storage medium;

the storage medium is used for storing instructions;

The processor is configured to operate in accordance with the instructions to perform the steps of the method according to Embodiment 1.

Example 3

In a third aspect, this embodiment provides a storage medium on which a computer program is stored, and when the computer program is executed by a processor, implements the steps of the method in Embodiment 1.

As will be appreciated by those skilled in the art, the embodiments of the present application may be provided as a method, a system, or a computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and/or block in the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general purpose computer, special purpose computer, embedded processor or other programmable data processing device to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing device produce Means for implementing the functions specified in a flow or flow of a flowchart and/or a block or blocks of a block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory result in an article of manufacture comprising instruction means, the instructions The apparatus implements the functions specified in the flow or flow of the flowcharts and/or the block or blocks of the block diagrams.

These computer program instructions can also be loaded on a computer or other programmable data processing device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process such that The instructions provide steps for implementing the functions specified in the flow or blocks of the flowcharts and/or the block or blocks of the block diagrams.

The above is only the preferred embodiment of the present invention, it should be pointed out that: for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can also be made, and these improvements and modifications are also It should be regarded as the protection scope of the present invention.

Claims (8)

1. A method for single target tracking, comprising:

acquiring video data;

determining a target area of a first frame of video data, and performing feature extraction and transform encoder encoding on the target area through a transform backbone network to obtain target area feature encoding;

generating M target prior frames by using a Gaussian function according to a target area of a first frame of video data, wherein M is an integer greater than 3;

determining a search area according to a target area of a first frame of video data;

performing transformer backbone network feature extraction and transformer encoder encoding on the search area to obtain search area feature encoding;

inputting the target region feature coding, the search region feature coding and the M target prior frames into a transform decoder for decoding to obtain IoU values of intersection ratios between the output M target prediction frames and the output labeling frames;

the IoU loss optimizes network parameters, and a gradient descent method is adopted to carry out N times of iterative optimization on the target prediction frame to obtain an optimized prediction frame;

and selecting the average value of the optimized prediction boxes of the optimized first three IoU values as a final target tracking box.

2. The method of single-target tracking according to claim 1, wherein said determining a target region of a first frame of video data comprises;

and labeling the target in a first frame of the video data through a rectangular frame, wherein the target area comprises a target position and a target size which are respectively determined by the position and the size of the rectangular frame.

3. The single-target tracking method of claim 1 or 2, wherein determining the search area based on the target area of the first frame of video data comprises:

wherein crop _ sz represents the size of a search area cropped in the current frame, w represents the width of an object frame in the previous frame, h represents the height of the object frame in the previous frame, and s represents a search factor.

4. The single-target tracking method of claim 1, wherein the inputting the target region feature code, the search region feature code and the M target prior boxes into a transform decoder for decoding results in IoU values of intersection ratios between the output M target prediction boxes and the output annotation boxes comprises: and splicing the target region feature code and the search region feature code on a channel dimension to obtain a feature map, obtaining M target prediction frames according to the feature map and the M target prior frames, and then obtaining IoU values between the M target prediction frames and the labeling frames through a feedforward neural network.

5. The single-target tracking method according to claim 1, wherein the IoU loss optimizes network parameters and performs N times of iterative optimization on the target prediction frame by adopting a gradient descent method, and the method comprises the following steps:

wherein, A is a prediction box, and B is a marking box.

6. The method of claim 5, wherein M is 100.

7. A single target tracking device, comprising: comprising a processor and a storage medium;

the storage medium is to store instructions;

the processor is configured to operate in accordance with the instructions to perform the steps of the method according to any one of claims 1 to 6.

8. A storage medium having a computer program stored thereon, the computer program, when being executed by a processor, implementing the steps of the method of any one of claims 1 to 6.

Images