CN112785626A - Twin network small target tracking method based on multi-scale feature fusion - Google Patents

Abstract

The invention discloses a twin network small target tracking method based on multi-scale feature fusion. The multi-scale fusion feature module and the optimized twin neural network comprehensively take into account the precise position of the target in the lower layer of the deep neural network structure, and the high layer can capture the target's precise position. The advantages of semantic information, through the effective fusion of different levels, make full use of the underlying information to avoid the problem that the convolution operation of the deep network will discard the information of small targets, solve the challenges of small targets in the tracking process, and achieve good tracking. Effect.

Description

A Siamese Network Small Object Tracking Method Based on Multi-scale Feature Fusion

technical field

The invention relates to visual recognition technology, in particular to a twin network small target tracking method based on multi-scale feature fusion.

Background technique

Moving object tracking means that after the position information of the target of interest in the first frame of a video sequence is given, the tracker can continue to track the target accurately and in real time in the subsequent sequence and return the position information. In recent years, the theoretical method of object tracking has developed very rapidly. It is an important research direction in the field of computer vision, and has been successfully applied to many fields such as video surveillance, unmanned driving, and semantic segmentation. The emergence of deep learning methods has greatly promoted the development of tracking problems, but the problem of small target tracking is still a very big challenge, especially in complex environments, how to accurately track small targets in real time is a key issue of research. .

At present, the challenges of small target tracking mainly come from two aspects: the features of small target objects are very difficult to obtain with the increase of the depth of the neural network, so it is difficult to obtain the feature representation. On the other hand, during tracking, small targets tend to drift abruptly and drastically compared to normal-sized targets due to lens shake. Current research only focuses on the tracking results of normal-sized target objects on general datasets, but ignores the problem of small target tracking.

The existing small target tracking algorithms are all based on traditional machine learning algorithms, which have great limitations in terms of accuracy improvement or real-time tracking, while deep neural networks can extract high-level layers due to their deep network layers. However, for objects with small targets, as the number of network layers deepens, continuous convolution operations will cause the network to gradually lose the location information of small targets.

Therefore, using the deep neural network structure of the Siamese network, from the perspective of multi-scale feature fusion, by fusing the complementary feature information of different network layers, real-time and robust small target object tracking in complex scenes and environments can be achieved. The application of Siamese network has the following problems: how to effectively integrate multi-scale features of different network layers, fuzzy target position of existing deep neural network and less semantic information, etc., which ultimately lead to difficulty in obtaining small target features.

SUMMARY OF THE INVENTION

Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art, and to provide a twin network small target tracking method based on multi-scale feature fusion. The scale feature fusion module increases the robustness of the network to scale changes, enabling end-to-end training to track small-scale targets accurately and in real time.

Technical solution: A twin network small target tracking method based on multi-scale feature fusion of the present invention includes the following steps:

Step (1): Perform size modification and data augmentation preprocessing on the template image x and the image y to be searched in turn, to obtain a pair of cropped training samples with a fixed size, and input them into the template branch and the siamese network structure respectively. search branch;

Step (2), the template branch and the search branch share the feature extractor, that is, the multi-scale feature fusion module is used to obtain the multi-scale fusion feature vector, including two stages of extracting features from bottom to top and horizontal fusion features from top to bottom;

When extracting features from bottom to top, an optimized twin network structure is constructed. The optimized twin network structure includes 5 convolutional layers, and the output of each layer is recorded as {C1, C2, C3, C4, C5} in turn;

When fusing features horizontally from top to bottom, first upsampling and expanding the size of high-level features and then merging them with the features of the lower layer, and then iteratively generate the multi-scale fusion feature maps of the template branch and the branch to be searched respectively;

In step (3), the template feature map and the search feature map obtained in step (2) are input into the similarity function, and the relevant cross operation is performed to obtain the response map. Similar positions, so as to determine the position of the target, that is, the target position in the image to be searched (that is, the frame to be tracked);

Step (4), expand the response map to the original y size of the image to be searched (for example, 225*225), then analyze the response map to obtain the final tracking result, multiply the position with the largest score by the total steps of the five-layer convolution of the optimized twin network structure long, the position information of the current target on the image to be searched can be obtained.

Further, in the described step (1), the specific method for modifying the size of the template image x is as follows:

During the target tracking process, the size of the target frame of the first frame is known, and the size of the target frame of the first frame is set as (x_min, y_min, w, h); then the size of the template image x is calculated according to the target frame of the first frame, That is, a square area is cut out with the target to be tracked as the center. The calculation formula is as follows:

s(w+2p)×s(h+2p)=A

Among them, (x_min, y_min) refers to the coordinate value of the lower left corner of the target frame, w and h represent the width and height of the frame, s refers to the modified size, and A is set to 127*127 size; Expand, then resize to 127*127 to get the template image x.

In the present invention, the first frame in a video frame is called a template frame (ie, template image x), and subsequent frames are all target positions to be searched (ie, image y to be searched), and the positions are represented by four coordinates of the lower left corner and width and height .

The specific method for modifying the size of the image y to be searched:

First, the center of the target frame predicted according to the previous frame is the cropping center, and then the side length of the square area cropped from the template image x is determined in proportion to the side length of the search box; finally, the size is modified to 255*255.

Further, in the step (2), an optimized twin network structure is constructed to extract features from bottom to top, and the optimized twin network structure is set as follows:

1. The first layer is a convolution layer, using a 11*11*96 convolution kernel with a stride of 2 to perform a convolution operation on the image. Then use a 3*3 maximum pooling operation and batch normalization operation to output C1;

2. The second layer is a convolution layer, using a convolution kernel of 5*5*256 and a stride of 1, using two groups of GPUs to perform convolution operations respectively, and then using 3*3 maximum pooling operations and batch normalization operations to Extract feature information and output C2;

3. The third layer is the convolution layer, which uses a 3*3*192 convolution kernel grouping to perform the convolution operation and continue the batch normalization operation to output C3;

④. The fourth layer is the convolution layer, which uses 3*3*192 convolution kernel grouping for operation and continues the batch normalization operation, and outputs C4;

⑤. The fifth layer is the convolution layer, which only uses 3*3*128 convolution operations, and finally outputs a 256-dimensional high-level semantic feature C5.

Further, in described step (2), the specific method of horizontally fusing features from top to bottom is:

(A), using the interpolation method, firstly, on the basis of the feature map pixels of the fifth layer, use 2 times upsampling (nearest neighbor upsampling method) to insert new elements between pixels, and change their size to the features of the fourth layer size, so as to expand the high-level feature size to facilitate the next step of fusion; then expand the feature size of the fourth layer, the third layer and the second layer in turn;

(B), use a 1*1 convolution operation in the C5 layer to obtain the low-resolution feature P5, and then use a 1×1 convolution kernel to change the fourth layer feature map C4 generated in the bottom-up process. The number of channels is fixed to 256-d, which is convenient for subsequent feature fusion. Next, the result of the fourth layer processing and the sampled result of the fifth layer are added and fused, and a 3*3 volume is used. The fusion result is processed by the kernel to solve the aliasing effect that may occur in the upsampling process, and the final result is recorded as P4;

Iterating the above (B) process finally generates a more accurate feature map, and obtains the feature map after multi-scale fusion of the template branch and the branch to be searched, respectively.

Further, in the step (3), the multi-scale fusion feature map corresponding to the template branch and the branch to be searched is used to obtain a response map by using a cross-correlation operation. Cross-correlation operation. Specifically, the template branch and the branch to be searched are used to correspond to multi-scale fusion features. The sizes of these two features are 22*22*256 and 6*6*256 respectively, and then 6*6* 256 is used as a convolution kernel to perform a convolution operation on the features of 22*22*256 to obtain a 17*17 response map. The target position tracked on this 17*17 response map has a higher score;

In the training process, after obtaining the 17*17 response map, then determine the positive and negative samples: as long as the value of the distance target is less than R on the search image, it is regarded as a positive sample, otherwise, it is regarded as a negative sample;

Finally, the binary cross-entropy logistic loss function is used, and the stochastic gradient descent method is used, the number of training iterations is set to 50, the minimum batch is set to 8, and the learning rate is decayed from ^10-2 to ^10-8 to train the entire deep network;

The above similarity function formula is as follows:

为卷积核，在

上进行卷积，b₁表示得分图上每个位置的取值。in,

is the convolution kernel, in

Convolution is performed on the score map, and b ₁ represents the value of each position on the score map.

Beneficial effects: The present invention is provided with a multi-scale fusion feature module, which fully considers the precise position of the target in the lower layer of the deep neural network structure, and the upper layer can capture the advantage of the semantic information of the target. Through effective fusion at different levels, the bottom layer can be fully utilized. The information avoids the problem that the convolution operation of the deep network will discard the information of the small target. In addition, the present invention optimizes the existing twin network structure, and can accurately track the visual target tracking method of small target objects in real time.

To sum up, the present invention can comprehensively and effectively integrate the structure of features of different network layers, solves the challenge of small targets in the tracking process, and thus achieves a good tracking effect.

Description of drawings

Fig. 1 is the overall flow schematic diagram of the present invention;

2 is a schematic structural diagram of a multi-scale feature fusion module of a branch to be searched in an embodiment of the present invention;

Fig. 3 is a comparative schematic diagram of an embodiment of the present invention;

Among them, Fig. 3(a) is a visualized feature map obtained by using the present invention, and Fig. 3(b) is a visualized feature map obtained by using an existing twin network.

Detailed ways

The technical solutions of the present invention are described in detail below, but the protection scope of the present invention is not limited to the embodiments.

In the practical application of target tracking, there is a situation in which the target captured by the camera is tracked at medium and high altitudes. How to continuously and accurately track the target in the long-distance scene has always been a difficult problem in the field of tracking research.

Based on the optimized twin network, the present invention performs feature fusion through a top-down multi-scale fusion method, which solves the problem of difficulty in tracking small objects in the prior art. As shown in FIG. 1, the present invention is based on multi-scale feature fusion. The Siamese network small target tracking method includes the following steps:

Step (1): Perform the preprocessing of modified size data augmentation on the template image x and the image to be searched y in turn to obtain a pair of cropped training samples with a corresponding fixed size, and input them into the template branch and search branch;

During the target tracking process, set the size of the target frame of the first frame as (x_min, y_min, w, h); then calculate the size of the template image x according to the target frame of the first frame, that is, cut out a For a square area, the calculation formula is as follows:

s(w+2p)×s(h+2p)=A

Among them, s refers to the modified size, and A is set to 127*127 size; through the above operations, expand the size of the target frame, and then modify the size to 127*127 size to obtain the template image x;

During training, the specific method to modify the size of the search image y:

First, the center of the target frame predicted in the previous frame is the cropping center, and then the side length of the square area cropped from the template image x is determined proportionally to the side length of the search box; finally, the size is modified to 255*255;

Step (2), the template branch and the search branch share the feature extractor, that is, the multi-scale feature fusion module is used to obtain the multi-scale fusion feature vector, including two stages of extracting features from bottom to top and horizontal fusion features from top to bottom;

As shown in Figure 2, an optimized Siamese network structure is constructed to extract features from bottom to top. The optimized Siamese network structure is set as follows:

1. The first layer is a convolution layer, using a 11*11*96 convolution kernel with a stride of 2 to perform a convolution operation on the image. Then use a 3*3 maximum pooling operation and batch normalization operation to output C1;

2. The second layer is a convolution layer, using a convolution kernel of 5*5*256 and a stride of 1, using two groups of GPUs to perform convolution operations respectively, and then using 3*3 maximum pooling operations and batch normalization operations to Extract feature information and output C2;

3. The third layer is the convolution layer, which uses a 3*3*192 convolution kernel grouping to perform the convolution operation and continue the batch normalization operation to output C3;

④. The fourth layer is the convolution layer, which uses 3*3*192 convolution kernel grouping for operation and continues the batch normalization operation, and outputs C4;

⑤. The fifth layer is the convolution layer, which only uses 3*3*128 convolution operations, and finally outputs a 256-dimensional high-level semantic feature C5.

The specific method of horizontally fusing features from top to bottom is as follows:

(A), using the interpolation method, firstly, on the basis of the feature map pixels of the fifth layer, use 2 times upsampling (nearest neighbor upsampling method) to insert new elements between pixels, and change their size to the features of the fourth layer size, so as to expand the high-level feature size to facilitate the next step of fusion; then expand the feature size of the fourth layer, the third layer and the second layer in turn;

(B), use a 1*1 convolution operation in the C5 layer to obtain the low-resolution feature P5, and then use a 1×1 convolution kernel to change the fourth layer feature map C4 generated in the bottom-up process. The number of channels is fixed to 256-d, which is convenient for subsequent feature fusion. Next, the result of the fourth layer processing and the sampled result of the fifth layer are added and fused, and a 3*3 volume is used. The fusion result is processed by the kernel to solve the aliasing effect that may occur in the upsampling process, and the final result is recorded as P4;

Iterating the above (B) process finally generates a more accurate feature map, and obtains the feature map after the multi-scale fusion of the template branch and the branch to be searched respectively;

Step (3), in the step (3), the multi-scale fusion feature map corresponding to the template branch and the branch to be searched is used to obtain a response map by using a cross-correlation operation. Cross-correlation operation. Specifically, the template branch and the branch to be searched are used to correspond to multi-scale fusion features. The sizes of these two features are 22*22*256 and 6*6*256 respectively, and then 6*6* 256 is used as a convolution kernel to perform a convolution operation on the features of 22*22*256 to obtain a 17*17 response map. The target position tracked on this 17*17 response map has a higher score;

In the training process, after obtaining the response map, it is necessary to determine the positive and negative samples: on the search image, as long as the value of the distance to the target is less than R, it is regarded as a positive sample, otherwise, it is regarded as a negative sample;

Finally, the binary cross-entropy logistic loss function is used, and the stochastic gradient descent method is used, the number of training iterations is set to 50, the minimum batch is set to 8, and the learning rate is decayed from ^10-2 to ^10-8 to train the entire deep network;

The above similarity function formula is as follows:

为卷积核，在

上进行卷积，b₁表示得分图上每个位置的取值；in,

is the convolution kernel, in

Convolution is performed on, b ₁ represents the value of each position on the score map;

Step (4), expand the response map to the original image size, and then analyze the response map to get the final tracking result, multiply the position with the largest score by the total step size of the five-layer convolution of the optimized twin network structure, and then the current target can be obtained. Search for location information on an image.

As shown in FIG. 3 , the target positioning obtained by the method of the present invention is accurate, and the effect is clearer.

It can be seen from the above embodiments that the present invention regards target tracking as the learning of the similarity measurement problem. For the template image x and the image to be searched y, we input them into the Siamese network structure for the same transformation, and design a multi-scale feature fusion module to obtain the corresponding feature vectors respectively, and finally use the template feature map as a convolution kernel on the search feature. Perform a cross-correlation operation to generate a response map to compare the similarity between the two. The position with higher similarity returns a high score, that is, the target position, otherwise it returns a low score.

Claims (6)

1. a Siamese network small target tracking method based on multi-scale feature fusion, is characterized in that: comprise the following steps: Step (1): Perform size modification and data augmentation preprocessing on the template image x and the image y to be searched in turn, to obtain a pair of cropped training samples with a fixed size, and input them into the template branch and the siamese network structure respectively. search branch; Step (2), the template branch and the search branch share the feature extractor, that is, the multi-scale feature fusion module is used to obtain the multi-scale fusion feature vector, including two stages of extracting features from bottom to top and horizontal fusion features from top to bottom; When extracting features from bottom to top, an optimized twin network structure is constructed. The optimized twin network structure includes 5 convolutional layers, and the output of each layer is recorded as {C1, C2, C3, C4, C5} in turn; When fusing features horizontally from top to bottom, first upsampling and expanding the size of high-level features and then merging them with the features of the lower layer, and then iteratively generate the multi-scale fusion feature maps of the template branch and the branch to be searched respectively; In step (3), the template feature map and the search feature map obtained in step (2) are input into the similarity function, and the relevant cross operation is performed to obtain the response map. Similar positions, so as to determine the position of the target in the image y to be searched; Step (4), expand the response map to the size of the original image y to be searched, and then analyze the response map to obtain the final tracking result, multiply the position with the largest score by the total step size of the five-layer convolution of the optimized twin network structure, and then obtain The position information of the current target on the image to be searched. 2. the twin network small target tracking method based on multi-scale feature fusion according to claim 1, is characterized in that: in described step (1), the concrete method that modifies size to template image x is as follows: Set the size of the target frame of the first frame as (x_min, y_min, w, h); then calculate the size of the template image x according to the target frame of the first frame, that is, cut out a square area with the target to be tracked as the center, the calculation formula as follows: s(w+2p)×s(h+2p)=A

Among them, s refers to the modified size transformation, and A is set to 127*127 size; through the above operations, the size of the target frame is expanded, and then the modified size is transformed to 127*127 size to obtain the template image x; The specific method to modify the size of the search image y: First, the center of the target frame predicted according to the previous frame is the cropping center, and then the side length of the square area cropped from the template image x is determined in proportion to the side length of the search box; finally, the size is modified to 255*255. 3. the twin network small target tracking method based on multi-scale feature fusion according to claim 1, is characterized in that: the method for data augmentation in the described step (1) comprises in order to increase deep learning training data, here we use four. Data augmentation methods: randomstretch random stretch, randomcrop random crop, normalize normalize and convert totensor into tensor; Finally, modify the size to the size that needs to be input into the network structure. 4. the twin network small target tracking method based on multi-scale feature fusion according to claim 1, it is characterized in that: in described step (2), construct optimized twin network structure to extract features from bottom to top, optimize twin network structure The settings are as follows: 1. The first layer is a convolution layer, using a 11*11*96 convolution kernel with a stride of 2 to perform a convolution operation on the image. Then use a 3*3 maximum pooling operation and batch normalization operation to output C1; 2. The second layer is a convolution layer, using a convolution kernel of 5*5*256 and a stride of 1, using two groups of GPUs to perform convolution operations respectively, and then using 3*3 maximum pooling operations and batch normalization operations to Extract feature information and output C2; 3. The third layer is the convolution layer, which uses a 3*3*192 convolution kernel grouping to perform the convolution operation and continue the batch normalization operation to output C3; ④. The fourth layer is the convolution layer, which uses 3*3*192 convolution kernel grouping for operation and continues the batch normalization operation, and outputs C4; ⑤. The fifth layer is the convolution layer, which only uses 3*3*128 convolution operations, and finally outputs a 256-dimensional high-level semantic feature C5. 5. The Siamese network small target tracking method based on multi-scale feature fusion according to claim 1, is characterized in that: the concrete method of horizontal fusion feature from top to bottom in described step (2) is: (A) Using the interpolation method, firstly, on the basis of the feature map pixels of the fifth layer, 2 times upsampling is used to insert new elements between the pixels, and the size is changed to the feature size of the fourth layer, thereby expanding the high-level feature size. It is convenient for the next step of fusion; then expand the feature size of the fourth layer, the third layer and the second layer in turn; (B), use a 1*1 convolution operation in the C5 layer to obtain the low-resolution feature P5, and then use a 1×1 convolution kernel to change the fourth layer feature map C4 generated in the bottom-up process. The number of channels is fixed as 256-d. Next, the result of the fourth layer processing and the sampling result of the fifth layer are added and fused, and a 3*3 convolution kernel is used to process the fusion. , denote the final result as P4; Iterate the above (B) process to finally generate a feature map, and obtain the feature map after multi-scale fusion of the template branch and the branch to be searched, respectively. 6. Siamese network small target tracking method based on multi-scale feature fusion according to claim 1, is characterized in that: described step (3) will template branch and the feature map after multi-scale fusion corresponding to the branch to be searched, using The cross-correlation operation obtains the response graph; The specific process of the cross-correlation operation is as follows: the template branch and the branch to be searched are used to correspond to the multi-scale fusion features. As a convolution kernel, perform the convolution operation on the features of 22*22*256 to obtain a 17*17 response map; In the training process, after obtaining the 17*17 response map, then determine the positive and negative samples: as long as the value of the distance target is less than R on the search image, it will be regarded as a positive sample, otherwise, it will be regarded as a negative sample; Finally, the binary cross-entropy logistic loss function is used, and the stochastic gradient descent method is used for iterative training to train the entire deep network; The above similarity function formula is as follows:

in,

is the convolution kernel, in

Convolution is performed on the score map, and b ₁ represents the value of each position on the score map.

Images