CN110517285A - Tracking of very small objects in large scenes based on motion estimation ME-CNN network - Google Patents

Abstract

The present invention proposes a method for tracking extremely small targets in a large scene based on the motion estimation ME-CNN network, which solves the problem of using motion parameters to track extremely small targets without registration. The implementation steps are: obtaining the target motion estimation network ME-CNN The initial training set D; construct the network ME-CNN to estimate the target motion; use the target motion parameters to calculate the network ME-CNN loss function; judge whether it is the initial training set; update the loss function training label; get the initial model of the predicted target motion position; correct Predict the position of the model; update the training data set with the corrected target position, and complete the target tracking of one frame; obtain the remote sensing video target tracking result. The invention uses the deep learning network ME-CNN to predict the moving position of the target, avoiding the registration of large scene images in tracking and the difficulty of extracting super-fuzzy target features, reducing the dependence of target features, and improving the accuracy of target tracking in super-fuzzy videos Spend.

Description

Tracking of very small objects in large scenes based on motion estimation ME-CNN network

technical field

The invention belongs to the technical field of remote sensing video processing, and relates to remote sensing video target tracking of very small targets in large scenes, in particular to a method for remote sensing video tracking of very small targets in large scenes based on a motion estimation ME-CNN network. It is used in security monitoring, smart city construction and traffic facility monitoring.

Background technique

Remote sensing target tracking is an important research direction in the field of computer vision, in which the target tracking of large scenes, extremely small targets and low-resolution remote sensing videos captured by moving satellites is a very challenging research problem. The remote sensing video of a large scene and a small target records the daily activities of a certain area for a period of time. Because the satellite shooting is at a high altitude and covers more than half of the city, the resolution of the video is not high. Vehicles, ships and aircraft are in the video The size of the vehicle is extremely small, and the size of the vehicle in the video even reaches about 3*3 pixels, and the contrast with the surrounding environment is also extremely low. Human eyes can only observe a small bright spot, so this ultra-low pixel and extremely small target The tracking problem is a very small target tracking problem in a large scene, which is more difficult; and because the satellites that shoot the video are constantly moving, the overall video has a relatively obvious shift in one direction, and some areas will zoom due to the height of the area, making it difficult to The method of image registration first and then frame difference method to obtain the moving position of the target brings great challenges to remote sensing video target tracking of very small targets in large scenes.

Video target tracking is to predict the target position and size in subsequent video frames after given the target position and size in an initial video frame. At present, most of the algorithms in the field of video tracking are based on neural network (Neural Network) and correlation filter (Correlation Filter). The main idea based on neural network algorithms, such as the CNN-SVM method, is to first input the target into a multi-layer neural network to learn target features. Then use the traditional SVM method for tracking. The extracted target features are learned through a large amount of training data, which is more discriminative than traditional features; the algorithm based on correlation filtering, such as the basic idea of the KCF method is to find a filter template, so that The image of the next frame is convolved with the filter template, and the area with the largest response is the predicted target. Use this template and other search areas for convolution operation, and the search area with the largest response is the target position. The calculation speed of this method is Fast and high accuracy.

Natural optical video tracking algorithms are difficult to apply to remote sensing videos of very small targets in large scenes, because the target size is extremely small and blurred, and neural networks cannot learn effective target features. However, the tracking method of traditional remote sensing video is not suitable for the video where the background is continuously shifted and some areas are zoomed in. The technical methods of image registration and frame difference method cannot be realized, and the contrast between the target and the surrounding environment is extremely low, and it is easy to lose track. .

Contents of the invention

The purpose of the present invention is to overcome the defects in the above-mentioned prior art, and propose a remote sensing video tracking method based on motion estimation with low computational complexity and higher precision for large scenes and small targets.

The present invention is a kind of remote sensing video tracking method based on motion estimation ME-CNN network of very small target in large scene, it is characterized in that, comprises the following steps:

(1) Obtain the initial training set D of the minimal target motion estimation network ME-CNN:

Take the first F frames of the original remote sensing data video A, continuously mark the bounding box for the same target in each image, and arrange the vertex coordinates of the upper left corner of each bounding box in order of video frame number as the training set D;

(2) Construct a network ME-CNN for estimating the motion of a very small target: including three parallel convolution modules that extract different features of the training data, and then sequentially stack the connection layer, the fully connected layer and the output layer;

(3) Calculate the loss function of the network ME-CNN with the minimum target motion parameters: calculate the motion trend of the target according to the motion law of the target, and use it as the training label corresponding to the target, and then calculate the relationship between the training label and the ME-CNN network The Euclidean spatial distance between the prediction results is used as the loss function of the ME-CNN network optimization training;

(4) Determine whether it is the initial training set: determine whether the current training set is the initial training set, if it is not the initial training set, perform step (5), and update the training label in the loss function; otherwise, if it is the initial training set, perform step ( 6), enter the cycle training of the network;

(5) Update the training label in the loss function: when the current training set is not the initial training set, use the data of the current training set to recalculate the training label of the loss function, the calculation method uses the minimum target motion parameters to calculate the training label method, and steps The method of (3) is the same, recalculate the training label that obtains, participate in motion estimation network ME-CNN training, enter step (6);

(6) Obtain the initial model M1 of the predicted target motion position: input the training set D into the target motion estimation network ME-CNN, train the network according to the current loss function, and obtain the initial model M1 of the predicted target motion position;

(7) Correct the position result of the prediction model: calculate the auxiliary position offset of the target, and use the offset to correct the position result predicted by the motion estimation network ME-CNN;

(7a) Get the target grayscale image block: get the target position (P _x , P _y ) of the next frame according to the initial model M1 of the predicted target motion position, and get the target position (P _x , P _y ) in the next frame according to the obtained target position (P x , P y ) The grayscale image block of the target is taken out from the image, and normalized to obtain the normalized target grayscale image block;

(7b) Obtain the target position offset: perform brightness classification on the normalized target grayscale image block, use the vertical projection method to determine the position of the target in the image block, and calculate the distance between the target center position and the center position of the image block Distance, that is, to get the target position offset;

(7c) Get the corrected target position: Use the obtained target position offset to correct the position of the target predicted by the motion estimation network ME-CNN, and obtain all corrected positions of the target;

(8) Update the training data set with the corrected target position to complete the target tracking of one frame: add the obtained upper left corner position of the target to the last line of the training set D, and remove the first line of the training set D to perform a one-time operation , get a corrected and updated training set D, complete the training of one frame, and get the target position result of one frame;

(9) Determine whether the current video frame number is less than the total video frame number: if it is less than the total video frame number, repeat steps (4) to (9) cyclically to perform target tracking optimization training until all video frames are traversed, and continue Training, otherwise if it is equal to the total number of video frames, end the training and perform step (10);

(10) Obtain the remote sensing video target tracking result: the accumulated output is the remote sensing video target tracking result.

The invention solves the problems of high computational complexity and low tracking precision existing in existing video tracking algorithms.

Compared with the prior art, the present invention has the following advantages:

(1) The motion estimation network ME-CNN adopted in the present invention does not need to perform image registration first in the traditional method, and then perform frame difference method or complex image background modeling to obtain the motion track of the target, and can use the neural network for the previous F frame images The analysis of the training set composed of the target position and the network prediction can obtain the target's motion trend, and the self-circulation training of the network can be realized without manually labeling the target position label in the subsequent video frame, which not only greatly reduces the complexity of the tracking algorithm, but also improves the The usefulness of the algorithm.

(2) The algorithm used in the present invention adopts the method of combining ME-CNN network and auxiliary position offset method to correct the remote sensing video target position by itself, and according to the motion law of the target, the loss function of the motion estimation network is modified, which reduces the network The amount of calculation improves the robustness of target tracking.

Description of drawings

Fig. 1 is the realization flowchart of the present invention;

Fig. 2 is the structural representation of the ME-CNN network that the present invention proposes;

Fig. 3 is a graph comparing curves between the predicted trajectory results of extremely small targets in large scenes and standard target trajectories using the present invention. The predicted results of the present invention are green curves, and red is the accurate target trajectory.

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

Example 1

Remote sensing video tracking of very small targets in large scenes plays an important role in security monitoring, smart city construction, and traffic facility monitoring. The remote sensing video studied by the present invention is a remote sensing video of a large scene, a very small target, and a low resolution taken by a moving satellite. The target of the video tracking researched by the present invention is extremely blurred, and the target is extremely small, and the contrast with the surrounding environment is also very low. It is difficult for human eyes to see that the target is a vehicle when the target does not move, and the video is due to the movement and shooting of satellites. Altitude changes in the region will cause image translation and partial zooming, and the difficulty of target tracking is greatly improved compared with clear video target tracking, which is also a challenge for remote sensing video tracking. There are mainly two existing methods. One is to use neural network to learn and extract target features, extract multiple search boxes in the next frame, and select the box with the highest score of target features as the target location. Extremely small, effective features cannot be extracted, and cannot be applied to the video of the present invention. The other is to first register the image and then perform the frame difference method to obtain the target motion trajectory, then find a filter template, and let the image of the next frame be convolved with the filter template, and the area with the largest response is the predicted target. Method Because the video of the present invention not only has image translation, but also partial zooming, the complexity of image registration is greatly increased, the difficulty of calculation is increased, and it is difficult to extract effective motion trajectories. Therefore, the present invention aims at these present situations, proposes a kind of remote sensing video tracking method based on motion estimation ME-CNN network of motion estimation ME-CNN network, referring to Fig. 1, comprising the following steps:

(1) Obtain the initial training set D of the minimal target motion estimation network ME-CNN:

Take the first F frame images of the original remote sensing data video A, select only one target in each image, and mark the bounding box continuously for the same target in each image. In the present invention, the minimal target is used as the target, and each boundary The coordinates of the vertices in the upper left corner of the frame are arranged together in order of the number of video frames as the training set D, in which the image coordinate system is used, and the training set D is a matrix of F rows and 2 columns, and each row corresponds to the target coordinate position of a frame, where the target The position of can be expressed by the coordinates of the upper left corner vertex, or by the center coordinates, which does not affect the analysis of the target movement.

(2) Build a network ME-CNN to estimate the motion of a very small target: the ME-CNN network of the present invention includes three parallel convolution modules that extract different features of the training data to obtain different motion features of the target, and then sequentially stack the connection layer fusion The extracted different motion features, the fully connected layer and the output layer get the output results, which constitute the ME-CNN network as a whole. The present invention uses three convolution modules to obtain different motion characteristics of the target. It is difficult for a single convolution module to process the characteristics of the entire training set. If the network layer is deep, the problem of gradient disappearance will occur. Therefore, the present invention increases the network Wide, multiple scales extract the features of different situations in the training set, which reduces the network complexity and speeds up the network speed. Because the video of the present invention is constantly moving and offset and some areas are zoomed due to the difference in height of the region, methods such as image registration plus frame difference method and background modeling cannot be used for this kind of video. At this time, the ME-CNN network can be used Obtaining the target trajectory has lower model complexity and less calculation than the existing methods.

(3) Calculate the loss function of the network ME-CNN with the minimum target motion parameters: calculate the motion trend of the target according to the motion law of the target, and use it as the training label corresponding to the target, and then calculate the relationship between the training label and the ME-CNN network The Euclidean space distance between the prediction results is used as the loss function optimized by the ME-CNN network. The loss function used for training in the present invention can strengthen the analysis of the training data and help the network to quickly extract effective features for optimizing motion estimation. Network ME-CNN.

(4) Determine whether it is the initial training set: determine whether the current training set is the initial training set, if the current training set is not the initial training set, perform step (5), update the training label in the loss function, and then participate in network training. On the contrary, if the current training set is the initial training set, perform step (6) to enter the network cycle training.

(5) Update the training label in the loss function: Since the training set D is continuously updated in the subsequent step (8), it is necessary to continuously adjust the training label in the loss function according to the updated training set D during the training process. The current training set is not the initial When training the set, the training label of the loss function should be recalculated using the data of the current training set. The calculation method uses the minimum target motion parameters to calculate the training label, which is the same as the method in step (3); the recalculated training label, participate in Motion estimation network ME-CNN training, enter step (6).

(6) Obtain the initial model M1 for predicting the target motion position: input the training set D into the target motion estimation network ME-CNN, train the network according to the current loss function, and obtain the initial model M1 for predicting the target motion position.

(7) Correct the position result of the prediction model: calculate the auxiliary position offset of the target, and use the offset to correct the position result predicted by the motion estimation network ME-CNN.

(7a) Get the target grayscale image block: get the target position (P _x , P _y ) of the next frame according to the initial model M1 of the predicted target motion position, and get the target position (P _x , P _y ) in the next frame according to the obtained target position (P x , P y ) The grayscale image block of the target is taken out from the image and normalized to obtain the normalized grayscale image block of the target. Since the size of the target is extremely small and the contrast with the surrounding environment is extremely low, the offset is judged by the neural network The effect of the method is poor, so the method of obtaining a smaller target frame first, and then judging the offset in the frame is better.

(7b) Obtain the target position offset: perform brightness grading on the normalized target grayscale image block, and display the target and the road with different brightness. At the same time, due to the extremely low contrast between the surrounding environment of the road and the target, the vertical projection method is used Determine the position of the target in the image block, and calculate the distance between the center position of the target and the center of the image block to obtain the target position offset.

(7c) Get the corrected target position: Use the obtained target position offset to correct the position of the target predicted by the motion estimation network ME-CNN, and obtain all corrected positions of the target, including the position of the upper left corner of the target.

(8) Update the training data set with the corrected target position to complete the target tracking of one frame: add the obtained upper left corner position of the target to the last line of the training set D, and remove the first line of the training set D to perform a one-time operation , a corrected and updated training set D is obtained, one frame of training is completed, and the target position result of one frame is obtained. This method of cyclically modifying the training set updates the network parameters, reduces the difference between target frames, and adapts to the target position. sports.

(9) Determine whether the current number of video frames is less than the total number of video frames. If it is less than the total number of video frames, repeat steps (4) to (9) cyclically, re-update model parameters, improve model adaptability, and perform target tracking optimization. Train until all video frames are traversed, and continue training, otherwise if the current video frame number is equal to the total video frame number, end the training and perform step (10).

(10) Get the remote sensing video target tracking result: After the training, the accumulated target position output is the remote sensing video target tracking result.

The motion estimation network ME-CNN adopted in the present invention does not need to perform image registration first in the traditional method and then perform frame difference method or complex image background modeling to obtain the target motion trajectory. The analysis of the training set composed of locations can effectively extract the target motion features. Due to problems such as gradient disappearance when the network is too deep, the ME-CNN network with multi-scale analysis is used to predict the movement trend of the target. It does not need to manually mark the target position label in the subsequent video frames, and the network self-circulation training can be realized, not only The complexity of the tracking algorithm is greatly reduced, and the practicability of the algorithm is improved. Through the target motion estimation network, the target position can be found quickly and accurately without image registration. Using the combination of ME-CNN network and auxiliary position offset method to judge the position of the remote sensing video target by itself, according to the motion of the target, get the motion speed of the target, analyze its possible motion trend, and modify the loss of the motion estimation network function, which improves the robustness of object tracking.

The present invention uses a method based on deep learning to analyze the motion of the ultra-fuzzy target, predict its next prediction direction, and then use the position offset to correct the motion estimation network, so that the target can be tracked without subsequent labels , so as to avoid the problems of image registration in large scenes and difficult feature extraction of super-fuzzy targets in tracking, and significantly improve the accuracy of target tracking in super-fuzzy videos, and is also suitable for tracking in various other remote sensing videos.

Example 2

The remote sensing video tracking method for very small objects in large scenes based on motion estimation ME-CNN network is the same as embodiment 1, and the network ME-CNN for estimating the motion of extremely small objects described in step (2), as shown in Figure 2, includes the following steps :

(2a) The overall structure of the motion estimation network: the motion estimation network ME-CNN, including three convolution modules connected in parallel, the present invention uses the three convolution modules connected in parallel to extract different motion features, and then the three convolution modules connected in parallel Then connect to the connection layer, fully connected layer and output layer in turn. The invention constructs a network ME-CNN for estimating the motion of a very small target, uses a connection layer to fuse and extract different motion features, uses a fully connected layer to refine and analyze, and outputs the result through an output layer.

(2b) Structure of three convolution modules connected in parallel: three convolution modules connected in parallel are respectively convolution module I, convolution module II and convolution module III, wherein

The convolution module I includes a locally connected LocallyConnected1D convolution layer with a step size of 2 to extract the coordinate position information of the target;

Convolution module ΙΙ contains dilated convolution with a step size of 1;

The convolution module ΙΙΙ contains a one-dimensional convolution with a step size of 2;

The convolution modules Ι, ΙΙ, and ΙΙΙ obtain the location features of the target at different scales, and obtain three output data, and then connect the outputs of the three convolution modules in series to obtain the fusion convolution result; then input the fully connected layer and the output layer to obtain The final prediction result. The present invention uses three convolution modules to obtain different motion characteristics of the target. It is difficult for a single convolution module to process the characteristics of the entire training set. If the network layer is deep, the problem of gradient disappearance will occur. Therefore, the present invention increases the network Wide, multiple scales extract the features of different situations in the training set, which reduces the network complexity and speeds up the network speed. Because the video of the present invention is constantly moving and offset and some areas are zoomed due to the difference in height of the region, methods such as image registration plus frame difference method and background modeling cannot be used for this kind of video. At this time, the ME-CNN network can be used Obtaining the target trajectory has lower model complexity and less calculation than the existing methods.

Example 3

Based on the motion estimation ME-CNN network, the large scene minimal target remote sensing video tracking method is the same as embodiment 1-2, and the loss function of the minimal target motion parameter calculation network ME-CNN described in step 3 is processed by processing the training set D The data, a rough analysis of the movement of the target, has a certain guiding effect on the optimization direction of the motion estimation network ME-CNN, including the following steps:

(3a) Obtain the target displacement of the training set D: take out the data of the Fth row, the F-2th row, and the F-4th row of the training set D, and subtract them from the data of the first row of the training set D respectively to obtain the Fth frame, The target displacements between frame F-2, frame F-4 and the first frame are S ₁ , S ₂ , and S ₃ in sequence. S ₁ is the target displacement between frame F and the first frame, S ₂ is the target displacement between frame F-2 and the first frame, S ₃ is the target between frame F-4 and the first frame displacement. If the training set is not the initial training set, but the training set D that has been updated i times, the number of frames corresponding to each row of the training set will also change accordingly, becoming the 1st+i frame, 2+i frame, ..., the th In frame F+i, the data in row F, row F-2, and row F-4 of training set D are taken out, and subtracted from the data in the first row of training set D, respectively, and the results are frame F+i, The distances between the target displacements of the F+i-2th frame and the F+i-4th frame and the first frame are respectively S ₁ , S ₂ , and S ₃ in sequence.

(3b) Obtain the movement trend of the target:

According to the movement law of the target, use the obtained target displacement to calculate the target's movement trend (G _x , G _y ) in the x and y directions of the image coordinate system according to the following formula;

V ₁ =(S ₁ -S ₂ )/2

V ₂ =(S ₂ −S ₃ )/2

a=(V ₁ -V ₂ )/2

G=V ₁ +a/2

The image coordinate system is used in the present invention, and the image coordinate system is based on the upper left corner of the image as the origin, the horizontal direction to the right is the x direction, and the vertical downward direction is the y direction. In the above formula, V ₁ is the target motion speed between displacement S ₁ and S ₂ , V ₂ is the target motion speed between displacement S ₂ and S ₃ , a is the motion acceleration, and G is the motion trend of the target.

(3c) Construct the loss function of the motion estimation network ME-CNN:

Calculate the movement trend of the target according to the movement law of the target, and use it as the training label corresponding to the target. The calculated target movement trend (G _x , G _y ) and the predicted position (P _x , P The Euclidean spatial distance between _y ) is constructed as the loss function of the motion estimation network ME-CNN;

In the formula, G _x is the target motion trend in the x direction in the image coordinate system, G _y is the target motion trend in the y direction in the image coordinate system, P _x is the prediction result of the motion estimation network in the x direction in the image coordinate system, and P _y is the prediction result of the motion estimation network in the y direction in the image coordinate system.

A comprehensive example is given below to further illustrate the present invention

Example 4

The remote sensing video tracking method for a very small target in a large scene based on a motion estimation ME-CNN network is the same as in Embodiment 1-3,

Referring to Fig. 1, a method for remote sensing video tracking of very small targets in a large scene based on a motion estimation ME-CNN network comprises the following steps:

(1) Obtain the initial training set D of the minimal target motion estimation network ME-CNN:

Take the first F frame images of the original remote sensing data video A, continuously mark a bounding box for a target in each image, and superimpose the vertex coordinates of the upper left corner of each bounding box together as a training set D, where the training set D is an F row 2 The matrix of columns, each row corresponds to the target coordinates of a frame in the video, wherein the position of the target can be represented by the coordinates of the upper left corner vertex, or can be represented by the center coordinates, which does not affect the analysis of the target motion situation. In the present invention, Minimal targets are simply referred to as targets.

(2) Build a network ME-CNN to estimate the motion of extremely small targets: including three parallel convolution modules that extract different features of the training data to obtain different motion features of the target. It is difficult for a single convolutional layer to process the features of the entire training set , if the network layer is deep, there will be a problem of gradient disappearance. Therefore, widening the network and extracting the characteristics of different situations in the training set at multiple scales can reduce the complexity of the network and speed up the network. Fusion extracted motion features, fully connected layer for analysis and output layer to get the result.

(2a) The overall structure of the motion estimation network: the motion estimation network ME-CNN, including three convolution modules connected in parallel, and then stacking the connection layer, the fully connected layer, and the output layer in sequence;

(2b) Structure of three convolution modules connected in parallel: three convolution modules connected in parallel are respectively convolution module I, convolution module II and convolution module III, wherein

The convolution module I includes a locally connected LocallyConnected1D convolution layer with a step size of 2 to extract the coordinate position information of the target;

Convolution module ΙΙ contains dilated convolution with a step size of 1;

The convolution module ΙΙΙ contains a one-dimensional convolution with a step size of 2;

The convolution modules Ι, ΙΙ, and ΙΙΙ obtain the location features of the target at different scales, and obtain three output data, and then connect the outputs of the three convolution modules in series to obtain the fusion convolution result; then input the fully connected layer and the output layer to obtain The final prediction result.

(3) Construct the loss function of the minimal target motion estimation network ME-CNN: calculate the target's motion trend according to the target's motion law, and use it as the training label corresponding to the target, and then calculate its prediction result with the ME-CNN network The Euclidean spatial distance between is used as the loss function of the ME-CNN network;

(3a) Obtain the target displacement of training set D: if the training set is the initial training set, take out the data of row F, row F-2, and row F-4 of training set D, and compare them with the data of the first row of training set D By subtracting, the target displacements between frame F, frame F-2, and frame F-4 and the first frame are respectively S ₁ , S ₂ , and S ₃ . S ₁ is the target displacement between frame F and the first frame, S ₂ is the target displacement between frame F-2 and the first frame, S ₃ is the target between frame F-4 and the first frame displacement. If the training set is not the initial training set, but the training set D that has been updated i times, the number of frames corresponding to each row of the training set will also change accordingly, becoming the 1st+i frame, 2+i frame, ..., the th In frame F+i, the data in row F, row F-2, and row F-4 of training set D are taken out, and subtracted from the data in the first row of training set D, respectively, and the results are frame F+i, The distances between the target displacements of the F+i-2th frame and the F+i-4th frame and the first frame are respectively S ₁ , S ₂ , and S ₃ in sequence.

(3b) Obtain the movement trend of the target:

According to the movement law of the target, using the target displacement obtained from the training data, the movement trend (G _x , G _y ) of the target is calculated according to the following formula in the x and y directions of the image coordinate system respectively.

V ₁ =(S ₁ -S ₂ )/2

V ₂ =(S ₂ −S ₃ )/2

a=(V ₁ -V ₂ )/2

G=V ₁ +a/2

(3c) Construct the loss function of the motion estimation network ME-CNN:

The calculated Euclidean spatial distance between the target motion trend (G _x , G _y ) and the predicted position (P _x , P _y ) output by the estimation network is constructed as the loss function of the motion estimation network ME-CNN.

(4) Update the training label in the loss function: Since the training set D is continuously updated in the subsequent step (7), it is necessary to continuously adjust the training label in the loss function according to the updated training set D during the training process, and participate in the motion estimation network ME - CNN training.

(5) Obtain the initial model M1 for predicting the target motion position: input the training set D into the target motion estimation network ME-CNN, train the network according to the loss function, and obtain the initial model M1 for predicting the target motion position.

(6) Correct the position result of the prediction model: calculate the auxiliary position offset of the target, and use the offset to correct the position result predicted by the motion estimation network ME-CNN.

(6a) Get the target grayscale image block: get the target position (P _x , P _y ) of the next frame according to the initial model M1 of the predicted target motion position, and get the target position (P _x , P _y ) in the next frame according to the obtained target position (P x , P y ) The grayscale image block of the target is taken out from the image and normalized to obtain the normalized grayscale image block of the target. Since the size of the target is extremely small and the contrast with the surrounding environment is extremely low, the offset is judged by the neural network The effect of the method is poor, so the method of obtaining a smaller target frame first, and then judging the offset in the frame is better.

(6b) Obtain the target position offset: perform brightness classification on the normalized target grayscale image block, use the vertical projection method to determine the position of the target in the image block, and calculate the center position of the target and the position of the center of the image block Distance, that is, to get the target position offset.

(6c) Obtain the corrected target position: use the obtained target position offset to correct the position of the target predicted by the motion estimation network ME-CNN, and obtain all corrected positions of the target, including the position of the upper left corner of the target.

(7) Update the training data set with the corrected target position to complete the target tracking of one frame: add the obtained upper left corner position of the target to the last row of the training set D, and remove the first row of the training set D to perform a one-time operation , a corrected and updated training set is obtained, one frame of training is completed, and the target position result of one frame is obtained.

(8) Obtain the remote sensing video target tracking result: repeat steps (4) to (7) in a loop, continuously use the updated training set to re-obtain the training label according to the method in step (3), update the network model, and iterate repeatedly. Carry out target tracking and approximation training until all video frames are traversed, and the accumulated output is the remote sensing video target tracking result.

In this example, the motion estimation model of the target can also extract the road information of the target through the target motion in the previous few frames, find the city where the target is located at the same longitude and latitude on the map, match the corresponding road conditions through the road, and predict the target motion. Make full use of the three-dimensional information of the road, and track the target more accurately when the height of the road changes sharply and there is some zoom in the video; the auxiliary position offset of the target can also be obtained by training the neural network, but it needs to be adjusted in advance. The surrounding environment is processed to obtain image blocks with high contrast, and then the neural network can be trained.

Below in conjunction with simulation experiment, technical effect of the present invention is further described:

Example 5

The remote sensing video tracking method of a very small target in a large scene based on a motion estimation ME-CNN network is the same as in Embodiment 1-4,

Simulation conditions and content:

The emulation platform of the present invention is: main frequency is the Intel Xeon CPU E5-2630v3CPU of 2.40GHz, the operating memory of 64GB, Ubuntu16.04 operating system, software platform is Keras and Python. Graphics card: GeForce GTX TITANX/PCIe/SSE2×2.

What the present invention uses is the remote sensing video of Libya's Derna area that Jilin No. 1 video satellite shoots, and the vehicle of the previous 10 frames of images is used as the target, and the frame is marked on the target in the image, and the position of the upper left vertex of the frame is used as the training set DateSet. The target video is tracked and simulated by the present invention and the existing KCF-based target tracking method respectively.

Simulation content and results:

The comparison method is the existing KCF-based target tracking method, and the method of the present invention and the comparison method are used to carry out experiments under the above-mentioned simulation conditions, that is, the comparison method and the present invention are used to track the vehicle target in the remote sensing video of the Libyan Derna area, and the ME- The comparison of the predicted target trajectory results (green curve) and the accurate target trajectory (red curve) of the CNN network is shown in Figure 3, and the results in Table 1 are as follows.

Table 1. List of remote sensing video target tracking results in Derna, Libya

Simulation result analysis:

In Table 1, Precision represents the area overlap rate between the target position predicted by the ME-CNN network and the label position, and IOU represents the percentage of the average Euclidean distance between the center position of the bounding box and the center position of the label less than a given threshold. In this example, The given threshold is selected as 5, KCF represents the comparison method, and ME-CNN represents the method of the present invention.

Referring to Table 1, it can be seen from the data comparison in Table 1 that the present invention greatly improves the tracking accuracy, and the present invention improves the Precision from 63.21% to 85.63%,

It can be seen from Table 1 that the average Euclidean distance between the center position of the bounding box and the center position of the label is less than the percentage IOU of the given threshold, and the present invention improves the comparison method based on the KCF target tracking method from 58.72% to 76.51%.

Referring to Fig. 3, the red curve in Fig. 3 is the standard target trajectory curve, the green curve is the tracking prediction estimation curve of the same target by the present invention, and the extremely small target in the large scene is displayed in the green box, comparing the two curves, It can be seen that the two curves are highly consistent and basically overlap, which proves that the tracking accuracy of the present invention is high.

In short, the present invention proposes a remote sensing video tracking method based on a motion estimation ME-CNN network for very small targets in a large scene, which can capture satellites in constant motion, overall panning and partial zooming in the video, and the resolution of the video is extremely low and The tracking accuracy is improved when the target size is extremely small, and the problem of using motion parameters to track extremely small objects without registration is solved. The implementation steps are: obtain the initial training set D of the extremely small object motion estimation network ME-CNN; Network ME-CNN for small target motion; calculate the loss function of network ME-CNN with extremely small target motion parameters; judge whether it is the initial training set; update the training label in the loss function; obtain the initial model M1 for predicting the target motion position; correct Predict the position result of the model; update the training data set with the corrected target position to complete the target tracking of one frame; judge whether the current video frame number is less than the total video frame number; obtain the remote sensing video target tracking result. The invention uses the deep learning network ME-CNN to predict the moving position of the target, which avoids the problems of difficult feature extraction of super-fuzzy target features in the registration of medium and large scene images in the existing method of tracking, reduces the dependence on target features, and significantly improves the super-ambiguity. The accuracy of object tracking in blurry videos is also applicable to tracking in various other remote sensing videos.

Claims (3)

1. a kind of minimum method for tracking target of large scene based on estimation ME-CNN network, which is characterized in that including as follows Step:

(1) the initial training collection D of minimum target estimation network ME-CNN is obtained:

The preceding F frame image for taking original remotely-sensed data video A will be every to the same target continued labelling bounding box of each image A bounding box top left corner apex coordinate is arranged sequentially by video frame number together as training set D；

(2) the network ME-CNN of minimum target movement is estimated in building: including three extraction training data different characteristics in parallel Convolution module, then stack gradually articulamentum, full articulamentum and output layer；

(3) loss function of network ME-CNN is calculated with the minimum parameters of target motion: being calculated according to the characteristics of motion of target The movement tendency of target, and using it as the corresponding trained label of target, then calculate the prediction of trained label Yu ME-CNN network As a result the European space distance between, the loss function as ME-CNN network optimization training；

(4) judge whether it is initial training to integrate: judging current training set whether as initial training collection, if not initial training Collection executes step (5), updates the training label in loss function；If instead being initial training collection, execute step (6), enters The circuit training of network；

(5) it updates the training label in loss function: when current training set is not initial training collection, using the number of current training set According to the training label for recalculating loss function, calculation method calculates the method for training label with the minimum parameters of target motion, with The method of step (3) is identical, the training label recalculated, participates in estimation network ME-CNN training, enters step (6)；

(6) it obtains the initial model M1 of prediction target movement position: training set D is inputted into target estimation network ME-CNN, According to current loss function training network, the initial model M1 of prediction target movement position is obtained；

(7) it corrects the position result of prediction model: the aided location offset of target is calculated, with offset correction estimation net The position result of network ME-CNN prediction；

(7a) obtains target gray image block: obtaining the target position of next frame according to the initial model M1 of prediction target movement position Set (P_x,P_y), according to obtained target position (P_x,P_y) the gray level image block of target is taken out in the image of next frame, and carry out Normalization, the target gray image block after being normalized；

(7b) obtains target position offset: carrying out brightness classification to the target gray image block after normalization, is thrown using vertical Shadow method determines the position of target in image block, and the target's center position being calculated is at a distance from the position at image block center, i.e., Obtain target position offset；

(7c) obtains revised target position: pre- using obtained target position offset correction estimation network ME-CNN The position for surveying target, obtains the revised all positions of target；

(8) training dataset is updated with revised target position, completes the target following of a frame: the target upper left corner that will be obtained Training set D last line is added in position, and removes the first row of training set D, is disposably operated, and has obtained an amendment simultaneously The training set D of update completes the training of a frame, has obtained the target position result of a frame；

(9) judge whether current video frame number is less than total video frame number: if it is less than total video frame number with regard to circulating repetition step (4)~step (9), carries out the tracking optimization training of target, until all video frames are traversed, if being equal to total video frame number, Terminate training, executes step (10)；

(10) obtain remote sensing video frequency object tracking result: the output accumulated is remote sensing video frequency object tracking result.

2. the large scene minimum target remote sensing video tracking side according to claim 1 based on estimation ME-CNN network Method, it is characterised in that: the network ME-CNN of minimum target movement is estimated in building described in step (2), comprises the following steps that

The overall structure of (2a) estimation network: estimation network ME-CNN, including three convolution modules in parallel, then according to Secondary stacking articulamentum, full articulamentum, output layer；

The structure of three convolution modules of (2b) parallel connection: three convolution modules in parallel, respectively convolution module Ι, convolution module Ι Ι and convolution module Ι Ι Ι, wherein

Convolution module I includes locally-attached LocallyConnected1D convolutional layer, and step-length is 2 coordinate positions for extracting target Information；

Convolution module Ι Ι includes empty convolution, step-length 1；

Convolution module Ι Ι Ι includes the one-dimensional convolution that step-length is 2；

Convolution module Ι, Ι Ι and Ι Ι Ι obtains the position feature of target different scale, obtains three output datas, then rolls up three The output of volume module is cascaded to obtain fusion convolution results；Full articulamentum and output layer are inputted again, obtain prediction knot to the end Fruit.

3. the large scene minimum target remote sensing video tracking side according to claim 1 based on estimation ME-CNN network Method, it is characterised in that: with the loss function of minimum parameters of target motion calculating network ME-CNN described in step 3, include Following steps:

(3a) obtains training set D displacement of targets: the data of training set D line f, F-2 row, F-4 row are taken out, respectively with instruction Practice collection D the first row data to subtract each other, obtains F frame, F-2 frame, displacement of targets of the F-4 frame respectively between first frame and be followed successively by S₁、S₂、S₃。

(3b) obtains the movement tendency of target:

According to the characteristics of motion of target, using obtained displacement of targets, respectively in the x of image coordinate system, the direction y is according to following public affairs Movement tendency (the G of target is calculated in formula_x,G_y)；

V₁=(S₁-S₂)/2

V₂=(S₂-S₃)/2

A=(V₁-V₂)/2

G=V₁+a/2

In formula, V₁To be displaced S₁With S₂Between target speed, V₂To be displaced S₂With S₃Between target speed, a be movement Acceleration, G are the movement tendency of target.

The loss function of (3c) building estimation network ME-CNN:

The movement tendency of target is calculated according to the characteristics of motion of target, and using it as the corresponding trained label of target, meter Obtained target movement tendency (G_x,G_y) with estimation network ME-CNN output predicted position (P_x,P_y) between theorem in Euclid space Distance is configured to the loss function of estimation network ME-CNN；

In formula, G_xFor the target movement tendency in the direction x under image coordinate system, G_yTarget movement for the direction y under image coordinate system becomes Gesture, P_xFor the prediction result in estimation network direction x under image coordinate system, P_yIt is estimation network in image coordinate system The prediction result in the direction lower y.