CN110956653B - Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation - Google Patents

Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation Download PDF

Info

Publication number
CN110956653B
CN110956653B CN201911206496.7A CN201911206496A CN110956653B CN 110956653 B CN110956653 B CN 110956653B CN 201911206496 A CN201911206496 A CN 201911206496A CN 110956653 B CN110956653 B CN 110956653B
Authority
CN
China
Prior art keywords
target
filter
image
motion estimation
tracking
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201911206496.7A
Other languages
Chinese (zh)
Other versions
CN110956653A (en
Inventor
李盛阳
轩诗宇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Technology and Engineering Center for Space Utilization of CAS
Original Assignee
Technology and Engineering Center for Space Utilization of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Technology and Engineering Center for Space Utilization of CAS filed Critical Technology and Engineering Center for Space Utilization of CAS
Priority to CN201911206496.7A priority Critical patent/CN110956653B/en
Publication of CN110956653A publication Critical patent/CN110956653A/en
Application granted granted Critical
Publication of CN110956653B publication Critical patent/CN110956653B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20076—Probabilistic image processing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30241—Trajectory

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a satellite video dynamic target tracking method with a correlation filter and motion estimation fused, which relates to the field of information processing and analysis of space and earth observation, and comprises the following steps: acquiring a satellite video, and cutting the satellite video into continuous single-frame images; selecting a target to be tracked in a single-frame image, constructing image characteristics of the target, and inputting the image characteristics into a relevant filter to obtain the position of the target; respectively carrying out motion estimation processing on the position of the target through a track averaging algorithm and a Kalman filter, and taking the processing result of the track averaging algorithm as a motion estimation result when the Kalman filter is not stable; and when the Kalman filter is stable, taking the processing result of the Kalman filter as a motion estimation result. The method is suitable for tracking the satellite video ground target, can effectively track the target when the target is shielded, and can finish various applications such as high-precision target tracking, real-time sensitive target positioning, traffic flow monitoring and the like.

Description

Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation
Technical Field
The invention relates to the field of information processing and analysis of space and earth observation, in particular to a satellite video dynamic target tracking method and device based on fusion of a correlation filter and motion estimation.
Background
The satellite video dynamic target tracking refers to the steps of giving an initial state such as position and size of a specified typical dynamic target (vehicle, ship, airplane and the like) in a first frame of a satellite video, and automatically estimating the state such as position, speed and the like of the target in a subsequent video.
The satellite video ground typical targets are vehicles, airplanes, trains, ships and the like, the targets are generally small in images due to the limitation of video spatial resolution, and many existing tracking algorithms based on template matching or optical flow and the like cannot complete many applications such as target high-precision tracking, sensitive target real-time positioning and traffic flow monitoring due to the lack of characteristics of ground moving targets.
Disclosure of Invention
The invention aims to solve the technical problem of the prior art, and provides a satellite video dynamic target tracking method and device based on the fusion of a correlation filter and motion estimation, which can realize rapid and accurate satellite video dynamic target tracking.
The technical scheme for solving the technical problems is as follows:
a satellite video dynamic target tracking method with fusion of correlation filters and motion estimation comprises the following steps:
acquiring a satellite video, and cutting the satellite video into continuous single-frame images;
selecting a target to be tracked in the single-frame image, constructing an image characteristic of the target, and inputting the image characteristic into a related filter to obtain the position of the target;
respectively carrying out motion estimation processing on the position of the target through a track averaging algorithm and a Kalman filter, and taking the processing result of the track averaging algorithm as a motion estimation result when the Kalman filter is not stable; and when the Kalman filter is stable, taking the processing result of the Kalman filter as a motion estimation result.
The invention has the beneficial effects that: according to the dynamic target tracking method provided by the invention, the related filter is fused with the motion estimation, so that the rapid and accurate satellite video dynamic target tracking is realized, the target is kept at the center of a search area as much as possible through the motion estimation to relieve the problem of boundary effect caused by insufficient samples in the related filtering, the accuracy of the algorithm is greatly improved, particularly the capability of tracking small targets such as vehicles and the like, the problem of slow convergence of a Kalman filter is solved by combining the track averaging algorithm and the Kalman filter, the motion estimation can be carried out in time before the Kalman filter converges, and a plurality of applications such as target high-precision tracking, sensitive target real-time positioning, traffic flow monitoring and the like can be completed.
Another technical solution of the present invention for solving the above technical problems is as follows:
a storage medium, wherein instructions are stored in the storage medium, and when the instructions are read by a computer, the instructions cause the computer to execute the satellite video dynamic target tracking method with fusion of correlation filter and motion estimation according to the above technical solution.
Another technical solution of the present invention for solving the above technical problems is as follows:
a satellite video dynamic target tracking device with a correlation filter fused with motion estimation comprises:
a memory for storing a computer program;
and the processor is used for executing the computer program to realize the satellite video dynamic target tracking method with the fusion of the correlation filter and the motion estimation according to the technical scheme.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
FIG. 1 is a schematic flow chart diagram illustrating a dynamic target tracking method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a dynamic target tracking device according to an embodiment of the present invention.
Detailed Description
The principles and features of this invention are described below in conjunction with the following drawings, which are set forth to illustrate, but are not to be construed to limit the scope of the invention.
The dynamic target tracking method provided by the invention is suitable for a video satellite, the video satellite is a novel earth observation satellite, and the greatest difference of the method compared with the traditional earth observation satellite is that the method can perform 'staring' on a certain area and acquire a high-time resolution image of the area through 'staring' observation on the area.
The launching lift-off sign of the video satellite indicates that people can obtain remote sensing images with high spatial resolution and high spectral resolution as well as high temporal resolution, is beneficial to improving the accuracy of some traditional remote sensing applications such as disaster monitoring, ocean monitoring and ecosystem disturbance monitoring, and changes the applications such as traffic condition monitoring and the like which cannot be well finished by the traditional remote sensing into reality. In order to complete the applications, breakthrough and development of key technologies such as high-resolution video reconstruction, moving object detection, identification and tracking, three-dimensional object reconstruction and the like are needed.
The satellite video dynamic target tracking refers to the steps of giving an initial state such as position and size of a specified typical dynamic target (vehicle, ship, airplane and the like) in a first frame of a satellite video, and automatically estimating the state such as position, speed and the like of the target in a subsequent video. The research of the direction technology can realize various applications in the aspects of specific target/group monitoring, flow estimation of airports, shopping malls and the like, traffic condition monitoring and military affairs.
Optical satellite video generally has the following problems in dynamic target tracking: for a ground moving object, the size of a vehicle is usually 4 × 4 pixels to 20 × 20 pixels in a video image (e.g. spatial resolution ≦ 1m), which results in a typical dynamic object being generally small and having insignificant features such as texture; the dynamic target can be partially or completely shielded by thin clouds, architectural shadows and the like; when the targets such as vehicles pass through the bottom of the overpass, the targets are completely shielded and disappear from the image; intensive motion may occur in a small area such as a highway, a port, an airport, and the like; background shake caused by a shooting mode; the problem of large-range rotation of the target caused by changing the shooting visual angle from the common head-up visual angle into a overlook visual angle.
Based on the above problems, the present invention provides a satellite video dynamic target tracking method based on the fusion of correlation filter and motion estimation and an improved scheme thereof, which solves the above problems.
Since the correlation filter is fused with the motion estimation, the following is a brief description of the conventional correlation filtering target tracking algorithm.
The correlation filter sees the tracking as a regression problem. For each sample, the correlation filter constructs a label between 0 and 1. The objective function for training the correlation filter is:
Figure BDA0002297050020000041
where x is the training sample, y is the label, and λ is the regular coefficient used to prevent overfitting,
Figure BDA0002297050020000042
is a kernel function.
The formula can be directly calculated to obtain the result
Figure BDA0002297050020000043
In the formula:
Figure BDA0002297050020000044
in the formula, kxxFor nuclear correlation, F is the Fourier transform.
During the tracking process, the response map can be calculated using the following equation:
Figure BDA0002297050020000045
wherein, the position where the response graph has the maximum response is the position of the target.
The related filtering algorithm can train the filter more quickly and complete the tracking of the target, but due to the loss of the training sample, the robustness is reduced because the filter is over-fitted; when the target changes rapidly and the background complexity increases, the target is lost by the algorithm; further, when the target appears at the edge of the search area, the accuracy of the algorithm may further decrease due to the missing target information, which may cause a boundary effect.
The existing method cannot complete many applications such as high-precision target tracking, real-time sensitive target positioning, traffic flow monitoring and the like, and the problems need to be solved in a targeted manner and the real-time performance of the algorithm is kept. Therefore, the method improves the relevant filter, fuses the relevant filter with motion estimation, and solves the defects that the ground target tracking accuracy of the satellite video is low, the speed is slow, and the complicated conditions that the target is blocked cannot be met in the prior art method, and the like, and the detailed description is provided below.
As shown in fig. 1, a schematic flow chart is provided for an embodiment of a dynamic target tracking method of the present invention, which is implemented based on fusion of a correlation filter and motion estimation, and is suitable for satellite video ground dynamic target tracking, such as an aircraft, an automobile, a ship, and the like, and the method includes:
and S1, acquiring the satellite video, and cutting the satellite video into continuous single-frame images.
And S2, selecting a target to be tracked from the single-frame image, constructing the image characteristics of the target, and inputting the image characteristics into a relevant filter to obtain the position of the target.
For example, the position of the target may be obtained according to the following formula:
Figure BDA0002297050020000051
where F is a fourier transform, α can be solved by the following equation:
Figure BDA0002297050020000052
if a Gaussian kernel is used, kxx'This can be written in the form of the following equation, where c represents the number of channels of the feature.
Figure BDA0002297050020000053
Where x is the training sample, y is the label, λ is the regular coefficient used to prevent overfitting, and σ is the gaussian kernel bandwidth.
Figure BDA0002297050020000054
DFT transform for ω, X, y.
Figure BDA0002297050020000055
Is composed of
Figure BDA0002297050020000056
Complex conjugation of (a). Represents an element-by-element multiplication, λ being the regularization coefficient of ridge regression.
S3, motion estimation processing is carried out on the position of the target through a track averaging algorithm and a Kalman filter respectively, and when the Kalman filter is not stable, the processing result of the track averaging algorithm is used as a motion estimation result; and when the Kalman filter is stable, taking the processing result of the Kalman filter as a motion estimation result.
It is to be understood that n frames of images are required in order to achieve the trajectory averaging, and therefore, the number of frames of processing for a single frame of image needs to be made to meet the requirement of the trajectory averaging.
It should be understood that the accuracy of the kalman filter for motion state estimation is high, but the kalman filter is complex, and requires a certain number of frames for iteration, and the state of the system can be stabilized, and through experiments, the satellite video dynamic target motion estimation generally requires 30-50 frames, and the kalman filter can be stabilized. In order to solve the problem of motion estimation of a Kalman filter before stabilization, the invention provides a track averaging method with simple calculation. And before the Kalman filter is stabilized, using the result of the track averaging as the output of the motion estimation, and after the Kalman filter is stabilized, using the result of the Kalman filter as the output of the motion estimation.
By using motion estimation, the approximate position of the target of the current frame can be estimated according to the position of the previous frame. At this time, we use the position of motion estimation as the center of the search area of the current frame correlation filtering, and then obtain the accurate position of the target through the correlation filtering calculation. By using the method, the target to be tracked can be moved to the center of the search area to a great extent, the problem of boundary effect is improved, and experiments prove that the accuracy of related filtering is greatly improved.
The kalman filter will be explained below.
The state equations and observation equations of the system can be written as follows:
Xk＝φk,k-1Xk-1+Wk-1
Yk＝HkXk+Vk
wherein, Xk，Xk-1Which are the state variables of the system at time k and time k-1, respectively. Phi is ak,k-1For the state transition matrix of the system, HkIs the observation matrix of the system. W and V are the noise matrices corresponding to the state and observation, respectively, and are Gaussian distributions with covariance matrices Q and R, respectively. In this system, we select the state variable of the system to be Xk＝(xsk ysk Δxk Δyk)TWherein xsk，yskHorizontal and vertical position of the target at time k, Δ xk，ΔykThe horizontal and vertical velocities of the target at time k.
Since the time between every two frames is short, the motion of a ground moving object such as a vehicle approaches a constant speed, so the state transition matrix can be written as:
Figure BDA0002297050020000071
observed state vector of Yk＝(xwk ywk)TThe target position observed at k time is shown. HKCan be expressed as:
Figure BDA0002297050020000072
the kalman filter motion state estimate can be written as follows:
Figure BDA0002297050020000073
Figure BDA0002297050020000074
Figure BDA0002297050020000075
Figure BDA0002297050020000076
Pk+1＝(I-Kk+1Hk+1)Pk+1,k
wherein, Xk，Xk-1Which are the state variables of the system at time k and time k-1, respectively. Phi is ak,k-1For the state transition matrix of the system, HkIs an observation matrix of the system, Xk+1For the optimal state estimation, K is a Kalman filtering gain matrix, and Q and R are covariance matrices of noise, and can be adjusted according to actual conditions. P0Random data other than 0 is typically used for initialization.
After the position of the target is obtained through the correlation filter, the position of the target is used as a state variable of an initial system and is input into the Kalman filter, so that the motion state estimation of the target can be realized, and for the motion estimation before the Kalman filter is stable, the track averaging can be realized through the following modes:
the typical dynamic target of the satellite video is generally a vehicle, an airplane, a ship, etc., and the speed is slow relative to the observation of the satellite, so that the motion of the target can be assumed to be a uniform linear motion in a relatively short time. Based on this assumption, the motion velocity of the object of the current frame can be estimated using the average of the displacements of the previous frames. The position of the current frame target can be estimated by using the speed of the target and the position of the previous frame target. The trajectory average can be described as:
Figure BDA0002297050020000081
Figure BDA0002297050020000082
Pt＝ASt-1
wherein S ist-1＝(xt-1 yt-1 Δxt-1 Δyt-1)TIs the state vector of the target at time t-1, Pt＝(xt yt)TFor the position vector of the target at time t, a is the transition matrix, which can be written as:
Figure BDA0002297050020000083
n is the frame number of the track average, the frame number used by the track average is determined by considering the FPS of the satellite video, and if the frame number is too small, the track average is not smooth enough and is easily influenced by the sudden change of the motion state of the vehicle. The assumption that the vehicle moves linearly at a constant speed is not true if the number of frames is too large. According to experiments, n is preferably set to be 5, so that the requirement of practical application can be met.
It should be understood that the motion estimation in this embodiment is described by using a conventional kalman filter, and those skilled in the art may also select an improved method for implementing the kalman filter according to actual requirements, such as unscented kalman filtering, volumetric kalman filtering, and the like.
The unscented kalman filter is explained as an example.
The unscented Kalman filtering is developed on the basis of Kalman filtering and transformation, and the unscented Kalman filtering under a linear assumption is applied to a nonlinear system by utilizing lossless transformation. The UT transform is to approximate a gaussian distribution with a fixed number of parameters, and the implementation principle is: taking some points in the original distribution according to a certain rule, and enabling the mean value and the covariance of the points to be equal to the mean value and the covariance of the original state distribution; and substituting the points into a nonlinear function to correspondingly obtain a nonlinear function value point set, and calculating the mean value and the covariance of the transformation through the point sets. For any nonlinear system, when the Gaussian state is transferred through the nonlinear system, the posterior mean and covariance accurate to the third moment can be obtained by using the group of sampling points.
The UT transformation is characterized in that the probability density distribution of the nonlinear function is approximated instead of the nonlinear function, and even if a system model is complex, the difficulty of algorithm realization is not increased; the accuracy of the statistic of the obtained nonlinear function can reach three orders; in addition to this, it does not require the calculation of jacobian matrices and can handle non-derivable non-linear functions.
The UT transform rationale is as follows: let a nonlinear system y ═ f (a), where a is the n-dimensional state vector and its average value is known as
Figure BDA0002297050020000091
Variance is PxThen 2n +1 Sigma points chi can be constructed by UT transformationiSimultaneous structure χiCorresponding weight value WiAnd then the statistical property of y is obtained.
Figure BDA0002297050020000092
Figure BDA0002297050020000093
Figure BDA0002297050020000094
Figure BDA0002297050020000095
Figure BDA0002297050020000096
Wi m＝Wi c＝1/{2(n+λ)}i＝1,…,2n
Wherein λ ═ α2(n + κ) -n is a scaling factor.
Figure BDA0002297050020000097
The distribution of surrounding Sigma points is determined by alpha, which is adjusted to reduce the effect of higher order terms, usually set to a small positive number. The value of κ is not particularly limited, but it is necessary to ensure that the matrix (n + λ) P is a semi-positive definite matrix, and κ is usually set to 3-n. The state distribution parameter beta is more than or equal to 0, the precision of the variance can be improved by setting beta,
Figure BDA0002297050020000098
is the matrix (n + lambda) PxItem I of (1). The mean and variance of y can be obtained by the following equations.
yi＝f(χi)
Figure BDA0002297050020000099
Figure BDA00022970500200000910
In the kalman wave algorithm, for a one-step prediction equation, the UT transform is used to handle the nonlinear transfer of mean and covariance. Unscented kalman filtering is to approximate the probability density distribution of a nonlinear function, approximating the posterior probability density of the state with a series of determined samples, rather than approximating the nonlinear function, without the need to derive a computational jacobian matrix. For a nonlinear system, the unscented Kalman filtering has higher precision and stability.
By using this alternative, the accuracy of the algorithm can be improved theoretically but the amount of calculation of the algorithm is increased greatly.
According to the dynamic target tracking method provided by the embodiment, the related filter is fused with the motion estimation, so that the rapid and accurate satellite video dynamic target tracking is realized, the target is kept at the center of the search area as much as possible through the motion estimation to relieve the problem of boundary effect caused by insufficient samples in the related filtering, the accuracy of the algorithm is greatly improved, particularly the capability of tracking small targets such as vehicles is greatly improved, the track averaging algorithm is combined with the Kalman filter, the problem of slow convergence of the Kalman filter is solved, the motion estimation can be timely carried out before the Kalman filter converges, and a plurality of applications such as target high-precision tracking, sensitive target real-time positioning and traffic flow monitoring can be completed.
Optionally, in some embodiments, the dynamic target tracking method provided by the present invention may further include:
obtaining a target response image of the current frame image according to the position of the motion estimation processing result and the relevant filter;
judging whether the peak value of the target response graph is larger than a preset threshold value or not, if so, determining that the target is not shielded, and updating filter parameters of a relevant filter and a Kalman filter; if not, the target is considered to be shielded, updating of filter parameters of the correlation filter and the Kalman filter is stopped, and the position of the motion estimation processing result is taken as the current position of the target.
It should be noted that, compared with the ordinary video tracking task, it is very common that the target is completely occluded in the satellite video. Taking a vehicle as an example, when the target passes through the bottom of the overpass, if the overpass is large, the vehicle is completely shielded and disappears from the image, and the method can solve the problem by using motion estimation.
To deal with the complete occlusion problem, the following three sub-problems typically need to be solved.
(1) Occlusion detection: the algorithm needs to detect the occurrence of complete occlusion or large area occlusion of the target.
(2) Shielding treatment: when complete shielding or large-area shielding occurs, processing is required to ensure that the target is not lost.
(3) And (3) shielding end detection: the algorithm needs to detect the end of occlusion.
The method for solving the three problems provided by the invention comprises the following steps:
first, the correlation filtering tracker can obtain an objective function, and the target position is calculated by the offset of the peak of the objective function from the center of the search area. The larger the peak, the higher the confidence. Therefore, whether the target is occluded can be determined by determining the magnitude of the peak value. When the peak value is larger than the threshold value, the target is not shielded, and when the peak value is smaller than the threshold value, the target is shielded.
When the target is occluded, the position obtained by the correlation filter will be inaccurate, and at this time, the position obtained by the correlation filter is discarded, and the position of the motion estimation is used as the position of the target of the current frame. The position accuracy obtained using motion estimation is limited compared to the correlation filter, so the kalman filter stops updating, and the correlation filter also stops updating, preventing the occluded feature from being learned.
When the peak value of the objective function obtained by the correlation filter is larger than the threshold value again, the occlusion is considered to be finished, and the algorithm flow when the occlusion is not performed is returned.
By using this method, the algorithm can quickly get the exact location of the target when it reappears. The condition that the target is lost by the tracker when the target is completely shielded is avoided.
Preferably, the preset threshold may be 0.3, and the target is considered to be completely occluded when the maximum value of the response map is lower than 0.3, and the target is considered to reappear when the maximum value of the response map is higher than 0.3.
Optionally, in some embodiments, selecting a target to be tracked from a single frame image, constructing an image feature of the target, and inputting the image feature into the correlation filter to obtain a position of the target, specifically including:
determining a target to be tracked in a first frame of image, constructing a first image characteristic of the target, inputting the first image characteristic into a related filter for training, and obtaining a filter parameter;
constructing a second image characteristic of the target in the second frame image, and inputting the second image characteristic into the trained correlation filter to obtain a target response image of the second frame image;
and calculating to obtain the position of the target in the second frame image according to the target response image of the second frame image, and updating filter parameters of the correlation filter and the Kalman filter according to the position of the target.
Preferably, the image feature may be a HOG feature.
Optionally, in some embodiments, determining a target to be tracked in the first frame image, and constructing a first image feature of the target specifically includes:
determining a target to be tracked in the first frame image, selecting a region with a preset size by taking the central coordinate of the target as the region center, cutting out the region from the first frame image, and constructing a first image feature of the target according to the region.
Optionally, in some embodiments, constructing the second image feature of the target in the second frame image specifically includes:
in the second frame image, the center coordinates of the target in the first frame image are used as the area center, a search area with a preset size is selected, the search area is cut out from the second frame image, and the second image characteristics of the target are constructed according to the search area.
It should be appreciated that overfitting can result from missing samples during the training of the correlation filter, and the tracking accuracy can decrease when the target is rapidly deformed or the background complexity increases. On the other hand, when the target appears at the edge of the search area, the robustness of the tracker is further influenced due to the lack of information, and the boundary effect is relieved on the premise of not influencing the closed-form of the relevant filter by placing the target to be tracked at the center of the search box as much as possible.
Preferably, the target to be tracked can be determined through horizontal rectangular box labeling, for example, the central coordinate of the target to be tracked is taken as the center of the area, the size of the target is 2.5 times as large as the size of the area, the area is cut out from the first frame image, and the HOG feature is constructed, so that the target can be ensured to be located in the center of the area, and the feature of the target can be completely extracted.
Similarly, for the processing of the subsequent frame image, the central coordinate of the target to be tracked in the previous frame may be used as the center of the area, the size of the target which is 2.5 times the size of the target may be used as the search area, and the area may be cut out from the original image to construct the HOG feature. And then calculating a response graph according to the HOG characteristics to obtain a target position, and updating filter parameters of the correlation filter and the Kalman filter according to the target position. The process is repeated, and other frames are continuously processed until the number of processed frames meets the requirement of track averaging.
Optionally, in some embodiments, the calculating the position of the target in the second frame image according to the target response map of the second frame image specifically includes:
and calculating the offset position of the peak value of the target response image and the center of the target response image to obtain the position of the target in the second frame image.
Optionally, in some embodiments, before inputting the first image feature into the correlation filter for training, the method further includes:
representing the data input into the classifier into a cyclic displacement form, and linking the data in the cyclic displacement form into a cyclic matrix;
and transforming the cyclic matrix to a Fourier domain, constructing an objective function by using a ridge regression form, and processing the closed solution of the ridge regression to obtain a training model of the correlation filter.
It should be understood that, in the initialization stage of target tracking, when the classifier is trained by using the first frame image, the classifier is easy to over-fit due to the lack of the number of samples, thereby causing the accuracy to decrease. The present invention increases the number of samples by using a cyclic shift method. Unlike a general algorithm, the present invention uses a regression method to construct a classifier, and points closer to the target center are closer to 1, and points farther from the target center are closer to 0, that is, closer to 1 indicates that the point is more likely to be a target.
Suppose the data is x ═ x1，x2，x3，...，xn]A cyclic shift of the data can be represented as Px＝[xn，x1，x2，...，xn-1]. All cyclic shifts of the data may be linked into a data matrix X ═ c (X), as shown below, such a matrix is referred to as a cyclic matrix
Figure BDA0002297050020000131
All circulant matrices have the following properties:
X＝FHdiag(Fx)F
wherein F is the DFT momentArray, meaning the transformation of data to the Fourier domain, FHFor the hermitian matrix of F, the solution of linear regression can be simplified by exploiting the properties of the circulant matrix.
The invention constructs an objective function using a ridge regression form:
Figure BDA0002297050020000141
wherein f (x) ═ ωTx and y are Gaussian functions with the bandwidth of the Gaussian distribution
Figure BDA0002297050020000142
Where w and h are the width and height of the target, the center of the target is 1, and decays to 0 as the distance from the center of the target increases.
The closed-form of ridge regression can be written as:
ω＝(XTX+λI)-1XTy
substituting the property expression of the circulant matrix into the formula can obtain:
Figure BDA0002297050020000143
where x is the training sample, y is the label, and λ is the regular coefficient used to prevent overfitting, can be set to 0.0001,
Figure BDA0002297050020000144
in order to be a kernel function, the kernel function,
Figure BDA0002297050020000145
DFT transform for ω, X, y.
Figure BDA0002297050020000146
Is composed of
Figure BDA0002297050020000147
Denotes element-by-element multiplication. λ is the regularization coefficient of ridge regression.
By using the property represented by the cyclic matrix and calculating in the Fourier domain, the invention does not need to iteratively construct cyclic displacement samples, but the samples participating in the calculation have the effect of cyclic displacement, and the calculation consumption of matrix inversion is avoided, so that the related filtering operation achieves very high calculation efficiency, and the ridge regression mode is used, so that overfitting can be avoided.
To improve the performance of the classifier, the present invention uses a kernel technique to change the linear regression of the classifier into a non-linear regression. Then f (x) can be written as follows:
Figure BDA0002297050020000148
for most kernel functions, the properties of the circulant matrix are still true, and α in the above equation can be solved by:
Figure BDA0002297050020000149
if a Gaussian kernel is used, kxx'The following can be written:
Figure BDA00022970500200001410
where c represents the number of channels of the feature, then f (x) can be further written as follows:
Figure BDA0002297050020000151
it will be appreciated that the filter needs to be updated in order to ensure that the filter can adapt to changes in the object. The new filter parameters are obtained by training samples sampled from the new position using the formula of α and linear interpolation with the old filter parameters, and the update rate can be set to 0.012.
Optionally, in some embodiments, determining whether the kalman filter is stable is performed according to the following steps:
and when the Euclidean distance between the predicted position of the continuous n frames of Kalman filter and the accurate position obtained by the relevant filter is within m pixels, considering that the state of the Kalman filter is stable, otherwise, considering that the state of the Kalman filter is not stable, wherein n is more than 2, and m is more than 0.
Preferably, when the Euclidean distance between the predicted position of the Kalman filter of 4 continuous frames and the accurate position of the target obtained by the relevant filtering is within 4 pixels, the state of the Kalman filter is considered to be stable.
The effects of the present invention will be further described with reference to examples.
The satellite video data comes from Jilin satellite constellation developed by Changchun satellite technology, Inc. of China. We used 11 segments of video, 13 targets for experiments, with a data spatial resolution of about 1 meter and a frame rate of 10 frames/second. Two of the 2-segment videos are targeted for airplanes, from Frankfurt and China Guizhou airports, which are approximately 50 x 40 pixels in size. A total of 11 targets of the remaining videos are vehicles, and traffic conditions of debye, hong kong, boston, and the like are observed, and the maximum of the vehicles is 23 × 8 pixels, and the minimum is only 8 × 8 pixels.
As technical scheme algorithm verification comparison, we choose to use the KCF tracking method, ECO tracking method, TLD tracking method, MIL tracking method, BOOSTING tracking method, MEDIANFLOW tracking method and other tracking methods in the image video field as reference.
The main evaluation indexes of the tracking result comprise: AUC, success rate, average positioning accuracy and frame number per second.
The overlapping rate is the overlapping rate of the target area estimated by the tracking algorithm and the artificially labeled target area, and if the overlapping rate is greater than a given threshold value, the frame tracking is considered to be successful. According to different thresholds, different tracking Success rates can be obtained, so that a curve Success Plot can be obtained. The area under the line of the curve can be used to measure the overall performance of the tracking algorithm.
The center error is the distance between the center point of the target position estimated by the tracking method and the center point of the artificially labeled target. The index can measure the positioning accuracy of the algorithm. If the central error is less than a given threshold, the frame tracking is considered to be successful. According to different thresholds, different tracking success rates can be obtained, so that a curve Precision Plot can be obtained.
Processing frame data per second is the number of frames that the algorithm can process per second, and can be used to measure the operating efficiency of the tracking method.
Vehicle tracking experimental results:
the technical scheme of the invention tests the performance of the vehicle when tracking.
In table 1, compared with KCF, the method greatly improves the tracking accuracy AUC of KCF by about 14%, improves the average tracking success rate by about 20%, improves the average positioning accuracy by about 17%, and makes TLD and medionnew invalid on the vehicle target, possibly because two algorithms need to extract feature points to calculate optical flow, but the feature points of the low-resolution vehicle target are not obvious or even have no feature points. The invention greatly exceeds other tracking methods in various accuracy rates.
In order to test the anti-occlusion performance of the technical scheme provided by the invention, the video data set comprises two parts, namely an occlusion part and an non-occlusion part. 7 targets are not shielded, and 4 targets have partial or full shielding phenomena. We performed experiments on these two parts separately.
Without occlusion, the accuracy of ECO is not as good as KCF, which may be caused by scale adaptation of ECO. Compared with KCF, the AUC of the technical scheme of the invention is improved by 8.5%, thus proving the effectiveness of motion estimation.
With occlusion, the ECO exceeds KCF. When the target is partially shielded, the ECO can not lose the target compared with the KCF, and strong robustness is embodied. However, when the target is completely shielded, except the method, other tracking methods can lose the target, the method can ensure the normal tracking during the shielding period, and the tracking problem when the target is completely shielded is solved.
In the aspect of efficiency, the technical scheme provided by the invention slightly improves the calculation complexity of motion estimation due to the introduction.
TABLE 1 vehicle target tracking results
Figure BDA0002297050020000171
And (3) tracking the aircraft target:
in order to test the adaptability of the tracking method to various targets, an airplane is used as a tracking target for testing.
The test results are shown in table 2. It can be seen that for the large target of the airplane with clear texture and shape features, the ECO has obvious advantage of reaching 76% in the aspect of AUC, and the MEDIANFLOW achieves the highest performance at 155FPS in the aspect of speed. The AUC of MIL is over 50% in the whole video sequence, but the positioning accuracy is particularly low, only 17.2, which proves that it can track the target correctly, but the deviation is large. Compared with other methods, the method provided by the invention has the advantages that the speed reaches 102FPS, the AUC reaches 69.28% and is arranged at the second position, the positioning precision reaches 66.2% and is arranged at the third position of all algorithms, and compared with the KCF (common correlation filtering) method, the accuracy is greatly improved while a large amount of calculation is not increased. The effectiveness of the technical scheme provided by the invention under the condition of a larger target is proved.
TABLE 2 aircraft target tracking results
Figure BDA0002297050020000172
Figure BDA0002297050020000181
Therefore, the method can complete continuous and accurate tracking of the satellite video ground target with very low computational complexity. The technical scheme of the invention can realize 120 frames/second on an Intel xeon E5. Meanwhile, compared with other common methods such as nuclear cross-correlation filtering, ECO and the like, the technical scheme of the invention can ensure continuous and accurate tracking of the target in various complex environments such as dense traffic, overpass shielding and the like, and especially ensure that the target is not lost during tracking when the target is completely shielded, so that the precision is achieved. By using the technical scheme of the invention, the high-precision continuous tracking of the specified target can be completed within the staring time of the satellite to the area.
The technical scheme of the invention can play an important application value in the fields of sensitive target monitoring, specific target positioning, traffic flow monitoring, related security and protection, military and the like.
Alternatively, in some embodiments, a combination of some or all of the above embodiments may be included.
In another embodiment of the present invention, there is provided a storage medium having instructions stored therein, which when read by a computer, cause the computer to execute the method for tracking a satellite video dynamic target by fusing a correlation filter and motion estimation as described in any of the above embodiments or a combination thereof.
As shown in fig. 2, a structural framework diagram is provided for a dynamic target tracking device according to an embodiment of the present invention, the dynamic target tracking device is implemented by fusing a correlation filter and motion estimation, and is suitable for tracking a satellite video ground dynamic target, the device includes:
a memory 1 for storing a computer program;
and a processor 2, configured to execute a computer program to implement the method for tracking a satellite video dynamic target by fusing correlation filter and motion estimation as described in any of the above embodiments and combinations thereof.
The reader should understand that in the description of this specification, reference to the description of the terms "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms used above are not necessarily intended to refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
In the several embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. For example, the above-described method embodiments are merely illustrative, and for example, the division of steps into only one logical functional division may be implemented in practice in another way, for example, multiple steps may be combined or integrated into another step, or some features may be omitted, or not implemented.
The above method, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention essentially or partially contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product stored in a storage medium and including instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a Read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
While the invention has been described with reference to specific embodiments, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims (9)

1. A satellite video dynamic target tracking method with a correlation filter and motion estimation fused is characterized by comprising the following steps:
acquiring a satellite video, and cutting the satellite video into continuous single-frame images;
selecting a target to be tracked in the single-frame image, constructing an image characteristic of the target, and inputting the image characteristic into a related filter to obtain the position of the target;
respectively carrying out motion estimation processing on the position of the target through a track averaging algorithm and a Kalman filter, and taking the processing result of the track averaging algorithm as a motion estimation result when the Kalman filter is not stable; when the Kalman filter is stable, taking a processing result of the Kalman filter as a motion estimation result;
judging whether the Kalman filter is stable according to the following steps:
when the Euclidean distance between the predicted position of the Kalman filter of n continuous frames and the accurate position obtained by the relevant filter is within m pixels, the state of the Kalman filter is considered to be stable, otherwise, the state of the Kalman filter is considered to be unstable, wherein n is more than 2, and m is more than 0;
wherein the trajectory averaging algorithm comprises: and estimating the motion speed of the target in the current frame according to the displacement average value of the targets in the preset number of frames before the current frame of the image, and estimating the position of the target in the current frame according to the motion speed of the target in the current frame and the position of the target in the previous frame of the current frame.
2. The method for tracking a satellite video dynamic target with a correlation filter fused with motion estimation according to claim 1, further comprising:
obtaining a target response image of the current frame image according to the position of the motion estimation processing result and the relevant filter;
judging whether the peak value of the target response graph is larger than a preset threshold value or not, if so, determining that the target is not shielded, and updating filter parameters of the correlation filter and the Kalman filter; if not, the target is considered to be shielded, the filter parameters of the correlation filter and the Kalman filter are stopped to be updated, and the position of the motion estimation processing result is taken as the current position of the target.
3. The method for tracking a satellite video dynamic target fused with correlation filter and motion estimation according to claim 1, wherein the method comprises the steps of selecting a target to be tracked from the single-frame image, constructing an image feature of the target, and inputting the image feature into the correlation filter to obtain the position of the target, and specifically comprises the following steps:
determining a target to be tracked in a first frame of image, constructing a first image characteristic of the target, inputting the first image characteristic into a correlation filter for training to obtain a filter parameter;
constructing a second image characteristic of the target in a second frame image, and inputting the second image characteristic into the trained correlation filter to obtain a target response image of the second frame image;
and calculating the position of the target in the second frame image according to the target response image of the second frame image, and updating filter parameters of the correlation filter and the Kalman filter according to the position of the target.
4. The method for tracking a satellite video dynamic target fused with a correlation filter and motion estimation according to claim 3, wherein the determining a target to be tracked in a first frame image and constructing a first image feature of the target specifically comprise:
determining a target to be tracked in a first frame image, selecting a region with a preset size by taking the central coordinate of the target as a region center, cutting out the region from the first frame image, and constructing a first image feature of the target according to the region.
5. The method for tracking a satellite video dynamic target fused with correlation filter and motion estimation according to claim 3, wherein constructing the second image feature of the target in the second frame image specifically comprises:
in a second frame image, taking the central coordinate of the target in the first frame image as an area center, selecting a search area with a preset size, cutting out the search area from the second frame image, and constructing a second image feature of the target according to the search area.
6. The method for tracking a satellite video dynamic target with a correlation filter fused with motion estimation according to claim 3, wherein the calculating of the position of the target in the second frame image according to the target response map of the second frame image specifically comprises:
and calculating the offset position of the peak value of the target response image and the center of the target response image to obtain the position of the target in the second frame image.
7. The method for tracking a satellite video dynamic target fused with correlation filter and motion estimation according to claim 3, wherein before inputting the first image feature into the correlation filter for training, the method further comprises:
representing the data input into the classifier into a cyclic displacement form, and linking the data in the cyclic displacement form into a cyclic matrix;
and transforming the cyclic matrix to a Fourier domain, constructing an objective function by using a ridge regression form, and processing a closed solution of the ridge regression to obtain a training model of the correlation filter.
8. A storage medium having stored therein instructions which, when read by a computer, cause the computer to perform a satellite video dynamic target tracking method with fusion of correlation filters and motion estimation according to any one of claims 1 to 7.
9. A satellite video dynamic target tracking device with a correlation filter fused with motion estimation is characterized by comprising:
a memory for storing a computer program;
a processor for executing the computer program to implement the correlation filter and motion estimation fused satellite video dynamic target tracking method according to any one of claims 1 to 7.
CN201911206496.7A 2019-11-29 2019-11-29 Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation Active CN110956653B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201911206496.7A CN110956653B (en) 2019-11-29 2019-11-29 Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911206496.7A CN110956653B (en) 2019-11-29 2019-11-29 Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation

Publications (2)

Publication Number Publication Date
CN110956653A CN110956653A (en) 2020-04-03
CN110956653B true CN110956653B (en) 2021-05-04

Family

ID=69979188

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911206496.7A Active CN110956653B (en) 2019-11-29 2019-11-29 Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation

Country Status (1)

Country Link
CN (1) CN110956653B (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111652906B (en) * 2020-05-11 2021-04-20 中国科学院空间应用工程与技术中心 Adaptive tracking method, device and equipment for satellite video ground dynamic target rotation
CN111696132B (en) * 2020-05-15 2023-12-29 深圳市优必选科技股份有限公司 Target tracking method, device, computer readable storage medium and robot
CN111949123B (en) * 2020-07-01 2023-08-08 青岛小鸟看看科技有限公司 Multi-sensor handle controller hybrid tracking method and device
CN112200126A (en) * 2020-10-26 2021-01-08 上海盛奕数字科技有限公司 Method for identifying limb shielding gesture based on artificial intelligence running
CN112464886A (en) * 2020-12-14 2021-03-09 上海交通大学 Aircraft identification tracking method
CN112634333B (en) * 2020-12-30 2022-07-05 武汉卓目科技有限公司 Tracking device method and device based on ECO algorithm and Kalman filtering
CN112927261A (en) * 2021-03-08 2021-06-08 电子科技大学 Target tracking method integrating position prediction and related filtering
CN113075086B (en) * 2021-03-25 2024-02-02 苏州未艾视智能科技有限公司 Wood chip detection and screening device based on X-rays
CN113963025B (en) * 2021-10-22 2022-08-16 西北工业大学深圳研究院 Underwater self-adaptive maneuvering target rapid tracking and tracing method
CN115082519A (en) * 2022-06-02 2022-09-20 长光卫星技术股份有限公司 Airplane tracking method based on background perception correlation filtering, storage medium and electronic equipment
CN116681731B (en) * 2023-08-02 2023-10-20 北京观微科技有限公司 Target object tracking method, target object tracking device, electronic equipment and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103873743A (en) * 2014-03-24 2014-06-18 中国人民解放军国防科学技术大学 Video de-noising method based on structure tensor and Kalman filtering
US9305362B1 (en) * 2014-02-28 2016-04-05 Xilinx, Inc. Image stabilization
CN107292911A (en) * 2017-05-23 2017-10-24 南京邮电大学 A kind of multi-object tracking method merged based on multi-model with data correlation

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101951464A (en) * 2010-05-19 2011-01-19 上海稳像信息技术有限公司 Real-time video image stabilizing method based on integral image characteristic block matching
CN102005054B (en) * 2010-11-24 2012-01-18 中国电子科技集团公司第二十八研究所 Real-time infrared image target tracking method
US20130004028A1 (en) * 2011-06-28 2013-01-03 Jones Michael J Method for Filtering Using Block-Gabor Filters for Determining Descriptors for Images

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9305362B1 (en) * 2014-02-28 2016-04-05 Xilinx, Inc. Image stabilization
CN103873743A (en) * 2014-03-24 2014-06-18 中国人民解放军国防科学技术大学 Video de-noising method based on structure tensor and Kalman filtering
CN107292911A (en) * 2017-05-23 2017-10-24 南京邮电大学 A kind of multi-object tracking method merged based on multi-model with data correlation

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Object tracking using block motion estimation with adaptive Kalman estimates;B.V Sarala等;《IEEE》;20180115;第751-754页 *
基于高斯混合模型的移动目标跟踪;时磊等;《微型电脑应用》;20171220;第33卷(第12期);第75-77页 *

Also Published As

Publication number Publication date
CN110956653A (en) 2020-04-03

Similar Documents

Publication Publication Date Title
CN110956653B (en) Satellite video dynamic target tracking method with fusion of correlation filter and motion estimation
Xuan et al. Object tracking in satellite videos by improved correlation filters with motion estimations
WO2022217840A1 (en) Method for high-precision multi-target tracking against complex background
CN107424171B (en) Block-based anti-occlusion target tracking method
CN110276785B (en) Anti-shielding infrared target tracking method
CN111311647B (en) Global-local and Kalman filtering-based target tracking method and device
Han et al. Visual tracking by continuous density propagation in sequential Bayesian filtering framework
CN106097383A (en) A kind of method for tracking target for occlusion issue and equipment
CN110827262B (en) Weak and small target detection method based on continuous limited frame infrared image
CN112036381B (en) Visual tracking method, video monitoring method and terminal equipment
CN104050685A (en) Moving target detection method based on particle filtering visual attention model
Li et al. Simultaneous tracking and verification via sequential posterior estimation
CN107194950B (en) Multi-person tracking method based on slow feature analysis
CN106780567B (en) Immune particle filter extension target tracking method fusing color histogram and gradient histogram
CN111402303A (en) Target tracking architecture based on KFSTRCF
Wang et al. Low-altitude infrared small target detection based on fully convolutional regression network and graph matching
Zhang et al. Unsupervised depth estimation from monocular videos with hybrid geometric-refined loss and contextual attention
CN114648547B (en) Weak and small target detection method and device for anti-unmanned aerial vehicle infrared detection system
CN110276788B (en) Method and apparatus for infrared imaging seeker target tracking
CN113033356B (en) Scale-adaptive long-term correlation target tracking method
Bouaynaya et al. A complete system for head tracking using motion-based particle filter and randomly perturbed active contour
Wu et al. Dynamic LiDAR Re-simulation using Compositional Neural Fields
CN111652906B (en) Adaptive tracking method, device and equipment for satellite video ground dynamic target rotation
Zakharov et al. Object Tracking and Anomaly Detection in Full Motion Video
CN109492530A (en) Robustness vision object tracking algorithm based on the multiple dimensioned space-time characteristic of depth

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant