CN108734103B - Detection and Tracking Method of Moving Objects in Satellite Video - Google Patents

Abstract

The invention belongs to the technical field of image and video processing, in particular to a method for detecting and tracking moving objects in satellite videos. The method of the invention includes: using the detection technology based on motion and local clutter modeling to detect moving objects-vehicles in the city in the satellite video; The target is identified; the target tracking technology based on Kalman filter is adopted, and the detected multiple vehicle targets are associated with the multiple trajectories being tracked to further track the ground moving targets. The invention has strong practicability and can be used in the fields of traffic monitoring, urban monitoring, military monitoring and the like, and has broad application prospects.

Description

Detection and Tracking Method of Moving Objects in Satellite Video

Field of Invention

The invention belongs to the technical field of image processing and video processing, and in particular relates to a method for detecting and tracking moving objects in satellite videos.

Background technique

Earth observation (EO) has a broad field of view and can reach the observation range of the size of a city. It is the current research hotspot and future research trend. It plays a vital role in civil and military fields such as urban transportation systems, maritime surveillance, aerial espionage and battlefield surveillance. At present, many observation technologies including optical satellite imagery, spaceborne synthetic aperture radar and aerial imagery have gradually developed and are of great significance. Both optical satellite imagery and spaceborne SAR can observe a wide field of view with high resolution. However, optical satellite images and SAR images are static pictures and cannot observe the dynamic changes of objects, which limits their application scope.

With the development of unmanned aerial vehicle (UAV), more and more aerial imagery can be used. However, aerial imagery must rely on complex stabilization processing due to platform instability fluctuations. For object tracking, image registration needs to be implemented to separate camera motion from object motion. The height of aerial photography is limited, and according to the new civil aviation regulations, the scope of application of drones is compressed, which greatly reduces the scope of video obtained by aerial photography.

Satellite video is an advanced monitoring technology that uses optical sensors to acquire continuous images on a moving satellite platform. Compared with the aerial images of drones, in addition to a wider field of view, satellite video is more stable than aerial video due to the stabilization of satellite motion. Another advantage of satellite video is that it can always keep the camera on the region of interest (ROI) through its own flight.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for detecting and tracking moving objects in satellite imaging video (hereinafter referred to as "satellite video").

The present invention is the first algorithm specially for satellite video processing, which can automatically realize the detection and tracking of vehicle targets in satellite video. Among them, the present invention proposes a new moving target detection algorithm and a new false alarm target elimination algorithm.

The method for detecting and tracking moving objects in satellite video proposed by the present invention, the moving objects here are mainly aimed at vehicles, and the specific steps are as follows:

(1) The detection technology based on motion and local clutter modeling is used to detect the moving target-vehicle in the city in the satellite video;

(2) Using the identification technology based on regional growth and multi-cues to identify the detected moving targets;

(3) The multi-target tracking technology based on Kalman filter is adopted, and the detected vehicle target states are associated with the multiple trajectories being tracked to further track the ground moving target; here, the target refers to any frame The target state is a 6-dimensional state vector composed of its position, velocity and acceleration, and the trajectory refers to a set of state vectors with a certain vehicle target in a continuous period of time. Any trajectory has a unique number for the entire video. .

In step (1), the detection technology based on motion and local clutter modeling assumes that the original image is the sum of the image signal and the noise signal, so the difference image between frames (the absolute value obtained after subtracting the two frame images is obtained) image) represents the noise signal, the abnormal pixels in the differential image represent the possible location of the vehicle target, select the exponential distribution to fit the noise clutter distribution, calculate the binarization threshold, and realize the binarization of the difference image between frames. The foreground in the image represents the vehicle target, the background in the binary image represents the background area of the image, and the current position of the target is determined by the intersection of two consecutive binary images.

In step (2), the identification technology based on region growth and multiple clues is to reconstruct the geometric shape of the moving target, and use a variety of morphological clues to identify the moving target and the false alarm target.

In step (3), the Kalman filter-based target tracking technology uses a linear model to describe the vehicle motion, combines the observed value and the predicted value, estimates the current state of the moving target, and updates the system state.

In step (3), the association between the detected multiple vehicle targets and the multiple trajectories being tracked is to use the Hungarian algorithm to achieve all the optimal matching of vehicle targets and trajectories. After the matching is completed, some targets match the upper part. Track, initialize a new tracking track for the target that does not match the track, and use the nearest neighbor search technology to find the matching target again for the track that does not match the target.

The algorithm of each step is further described below:

1. Detection technology based on motion and local clutter modeling, the specific steps are:

(1) Rasterization, that is: in each frame, the image is divided into two-dimensional grids along the vertical and horizontal directions, each grid cell contains several pixels, and the detection algorithm is executed in each cell;

(1a) Consider the image as the sum of the original image signal and the noise signal, namely:

G _i (x, y)=g _i (x, y)+n _i (x, y) (1)

Among them, G _i (x, y) is the magnitude of the pixel point (x, y) in the ith frame of video, and _gi (x, y) is the point (x, y) in the original image in the ith frame of video. Amplitude, the amplitude of the noise signal corresponding to n _i (x, y);

(1b) Choose an exponential distribution to fit the distribution of the noise signal, and the cumulative density function (CDF) of the exponential distribution is:

where c _E (*) is the CDF of the exponential distribution and λ is the parameter of the exponential distribution.

(1c) Find the area with large grayscale change as the target, set the false alarm rate in advance, and calculate the binarization threshold:

是逆函数，p_fa是预先设置的概率值；in,

is the inverse function, _pfa is the preset probability value;

(1d) Use a predefined probability value to calculate the binarization threshold to classify the pixels, the pixel amplitude value greater than th is the foreground, and the pixel amplitude value less than th is the background. In a binary image, '1' is the foreground, and '0' is the foreground background;

(1e) Take the intersection map of the two binary images obtained, that is, perform the "AND" Boolean operation pixel by pixel to eliminate the scene targets at the previous moment and the next moment, so as to uniquely retain the current position of the moving vehicle target.

2. Identification technology based on regional growth and multiple clues, the specific steps are:

(2a) Using the region growing algorithm based on Gaussian distribution, using all foreground pixels of a connected region detected by the detector to estimate the parameters of the Gaussian distribution; the PDF of the Gaussian distribution is;

where μ and σ represent the mean and standard deviation, respectively, and the CDF of the Gaussian distribution is:

Among them, erf(*) represents the correlation error function;

和上边界概率

分别计算幅度值的下边界和上边界，即：(2b) Predefined lower bound probability

and the upper bound probability

Calculate the lower and upper bounds of the magnitude value separately, namely:

和

分别代表阈值的下界和上界；in,

and

represent the lower and upper bounds of the threshold, respectively;

范围内的像素为前景；(2c) Re-judging the foreground target in this area, the amplitude value is in

The pixels within the range are the foreground;

(2d) Retain all pixels connected to the original foreground, and construct a new connected region, that is, the result of region growth;

(2e) Finally, using area, duty cycle, length, and eccentricity as morphological cues, the results of region growth are morphologically discriminated with the aim of eliminating non-vehicle targets from the detected targets, where the definition of duty cycle Yes, the ratio of the connected region pixels to the area of the connected region bounding box.

3. Target tracking technology based on Kalman filter, the specific steps are:

(3a) KF is generally suitable for linear motion target tracking, while nonlinear motion can generally be decomposed into a series of linear motions. First, only the KF tracking problem of one moving target is considered. The dynamic equation of the prediction system is:

x _i =F _i ·x _i-1 +v _i (8)

Among them, F _i is the evolution matrix of the ith frame moment of the satellite video, vi is the noise vector, x _{i -1} is the state vector of the i-1th frame moment, and x _i is the state vector of the current moment;

The state vector x _i is defined as:

x _i = [x, y, v _x , v _y , a _x , a _y ] ^T (9)

Among them, x, y, v _x , v _y , a _x , a _y represent the position in the X-axis direction, the position in the Y-axis direction, the velocity in the X-axis direction, the velocity in the Y-axis direction, the acceleration in the X-axis direction, Y acceleration in the axis direction;

Assuming that the vehicle target moves in a straight line with uniform acceleration at the moment between two frames, the evolution matrix F _i is:

Among them, τ represents the time interval between two frames in the video, in "seconds", which is the inverse of the frame rate.

(3b) Using the area and position of the detected vehicle as the observation value, the observation equation is obtained as:

Among them, y _i , H _i and _ni represent the measurement vector, measurement matrix and measurement noise respectively;

The observation matrix H _i is:

(3c) Derive the posterior probability density function (PDF) of the state vector x _i by Bayes' theorem, and then use the chain rule to simplify it, namely:

Among them, the probability p(x _i |y _1:i-1 ) is derived by the Chapman-Kolmogorov equation, namely:

p(x _i |y _{1 : i-1} )=∫p(x _i |x _i-1 )p(x _i-1 |y _1:i-1 )dx _i-1 (14)

(3d) Using equations (13) and (14), the posterior probability density function PDF of the state vector x _i is obtained:

(3e) Combine the observed value and the predicted value, using formula (15), estimate the current state, and continuously update the state of the system.

4. The association of multiple vehicle detection and tracking, the specific steps are:

(4a) In the target tracking technology based on Kalman filter, the motion trajectory is associated with the moving target through the Hungarian algorithm, and the associated target is tracked;

(4b) Using the nearest neighbor search based on SSIM (structural similarity index), the trajectory of the unrelated target is re-searched for the moving target associated with it. If the association is successful, the tracking will continue after correction. If the trajectory is still unrelated target, the trajectory is terminated.

The algorithm of the invention has high precision and practical value, can be used in traffic monitoring, urban monitoring, military monitoring and other fields, and has broad application prospects.

Description of drawings

FIG. 1 is a schematic diagram of an example of satellite video of the present invention. where (a) the video location is in Valencia, Spain, and (b) is the corresponding optical map downloaded from Google Earth.

FIG. 2 is a flow chart of a detection algorithm for fitting noise clutter distribution using exponential distribution according to the present invention.

FIG. 3 is a schematic diagram of foreground segmentation results and region growth results of moving targets and false alarms according to the present invention. Among them, (a), (b), (c), and (d) represent the four hypotheses, respectively, and the first, second, and third rows represent the color map, foreground segmentation results, and region growth results, respectively. For ease of observation, the original grayscale image is correspondingly converted to the image shown in the first row.

FIG. 4 is a schematic diagram of a moving vehicle sample and a non-moving vehicle sample according to the present invention. where (a), (b) and (c) are real moving targets, but the black vehicle in (b), which is not significantly different from the background, (d) is a false alarm target, which may be a similar A stationary vehicle with a real moving target.

FIG. 5 is a schematic diagram of the annotation area of the satellite video used in the experiments of the present invention.

Fig. 6 is the effect diagram after the annotation of the satellite video used in the experiment of the present invention. Among them, (a) shows the location of the vehicle and (b) shows its corresponding name. Annotated in the figure are representative scenarios.

7 is a schematic diagram of vehicle detection and tracking results in four frames of the satellite video of the present invention: the 50th frame, the 100th frame, the 150th frame, and the 200th frame, and each vehicle has its own number.

FIG. 8 is a flow chart of the present invention.

Detailed ways

First, the moving target in the satellite video is detected, and the method based on motion and local clutter modeling is used. The specific implementation is as follows:

Step 1: In each frame, the image is divided into two-dimensional grids along the vertical and horizontal directions, and each grid cell contains several pixels;

Step 2: The image is regarded as the sum of the original image signal and the noise signal, and the exponential distribution is selected to fit the distribution of the noise signal;

Step 3: Find the area with large grayscale change as the target, set the false alarm rate to 1×10 ^-2 in advance, calculate the binarization threshold, and use the threshold to classify the pixels, that is: the pixel amplitude value is greater than the threshold is the foreground, and is the background anyway (in a binary image, '1' is the foreground, and '0' is the background);

Step 4: Take the intersection map of the two binary images obtained, that is, perform the "AND" Boolean operation pixel by pixel to eliminate the scene target at the previous moment and the next moment, so as to uniquely retain the current position of the moving vehicle target and obtain the detection result.

Then, the detected moving objects in the satellite video are identified, and the method based on regional growth and multiple clues is used, and the specific implementation is as follows:

Step 1: Using the region growth algorithm based on Gaussian distribution, use all foreground pixels of a connected region detected by the detector to estimate the parameters of the Gaussian distribution, and then obtain the PDF of the Gaussian distribution;

Step 2: Predefine the values of the upper justification probability and the lower bound probability, design the lower bound probability as 1×10 ^-2 , and design the upper bound probability as 1-1×10 ^-2 , respectively calculate the upper bound and lower bound of the amplitude value, and use The new amplitude boundary value, re-judging the foreground target in this area;

Step 3: Retain all pixels connected to the original foreground, build a new connected area, and obtain the result of area growth;

Step 4: Use area, duty cycle, length, and eccentricity as morphological clues. The parameters of morphological clues can be designed according to the actual situation. The results of regional growth are morphologically identified, and non-vehicle targets are excluded from the detected targets. , get the identification result.

Finally, the moving target in the detected and identified satellite video is tracked and associated, and the method for target tracking using the Kalman filter is implemented as follows:

Step 1: Consider only the KF tracking problem of a moving target, predict the dynamic equation of the system, and use the state vector to obtain the evolution matrix;

Step 2: Use the area and position of the detected vehicle as the observation value to obtain the observation equation, and use the observation vector to obtain the observation matrix;

Step 3: Derive the posterior probability density function of the state vector by Bayesian rule, and use the chain rule to simplify it;

Step 4: Combine the observed value and the predicted value, estimate the current state, and update the system state;

Step 5: Use the Hungarian algorithm to associate the detected vehicle target with the track being tracked. If it can be associated, update the state of the corresponding track with the target state. If the target is not associated with the track, initialize a new track and track it. , if the trajectory is not associated with the target, then use the SSIM algorithm to search for the corresponding target again. If it can be searched, update the state of the trajectory with the state of the searched target. If it cannot be searched, terminate the tracking of this trajectory.

The algorithm was tested on the video shown in Figure 1, which has a frame rate of 20 frames per second, a resolution of 1 meter, a duration of 29 seconds, an image height of 3072 pixels, and an image width of 4096 pixels. Some areas are manually marked as shown in Figure 6. The manually marked data in these areas is compared with the output results of the algorithm to quantitatively evaluate the performance of the algorithm. The evaluation test shows that the algorithm recall rate is 63.06%, the accuracy rate is 81.04%, and the algorithm precision is high. Has practical value.

Claims (5)

1. the detection and tracking method of moving target in a satellite video, is characterized in that, concrete steps are as follows: (1) The detection technology based on motion and local clutter modeling is used to detect the moving target-vehicle in the city in the satellite video; (2) Using the identification technology based on regional growth and multi-cues to identify the detected moving targets; (3) The multi-target tracking technology based on Kalman filter is adopted, and the detected multiple vehicle targets are associated with the multiple trajectories being tracked to further track the ground moving targets; here, the target refers to any frame in any frame. For the detected vehicle, the target state refers to the 6-dimensional state vector composed of its position, speed and acceleration, and the trajectory refers to the set of state vectors with a certain vehicle target in a continuous period of time, and any trajectory has a unique number for the entire video; In step (1), the described detection technology based on motion and local clutter modeling detects the moving target in the city in the satellite video---the vehicle, assuming that the original image is the sum of the image signal and the noise signal, The difference image between frames, that is, the image obtained by subtracting the absolute value of the two frame images represents the noise signal, and the abnormal pixels in the difference image represent the possible location of the vehicle target. Select the exponential distribution to fit the noise clutter distribution, and calculate the binarization threshold. , realize the binarization of the difference image between frames, the foreground in the binary image represents the vehicle target, the background in the binary image represents the background area of the image, and the current position of the target is determined by the intersection of two consecutive binary images; In step (2), the described identification technology based on regional growth and multiple clues, to identify the detected moving target, is to reconstruct the geometric shape of the moving target, and use a variety of morphological clues to identify the moving target and the false alarm. Target; In step (3), the described target tracking technology based on Kalman filter is to use a linear model to describe the vehicle motion, combine the observed value and the predicted value, estimate the current state of the moving target, and update the system state; In step (3), the association between the detected multiple vehicle targets and the multiple trajectories being tracked is to use the Hungarian algorithm to achieve all the optimal matching of vehicle targets and trajectories. After the matching is completed, some targets match the upper part. Track, initialize a new tracking track for the target that does not match the track, and use the nearest neighbor search technology to find the matching target again for the track that does not match the target. 2. the detection and tracking method of moving target in satellite video according to claim 1, is characterized in that, in step (1), the concrete steps based on the detection of motion and local clutter modeling are: (2a) rasterization, that is: in each frame, the image is divided into two-dimensional grids along the vertical and horizontal directions, each grid cell contains several pixels, and the detection algorithm is executed in each cell; (2b) regard the image as the sum of the original image signal and the noise signal, namely: G _i (x, y)=g _i (x, y)+n _i (x, y) (1) Among them, G _i (x, y) is the magnitude of the pixel point (x, y) in the ith frame of video, and _gi (x, y) is the point (x, y) in the original image in the ith frame of video. Amplitude, the amplitude of the noise signal corresponding to n _i (x, y); (2c) Select the exponential distribution to fit the distribution of the noise signal, and the cumulative density function CDF of the exponential distribution is:

where c _E (*) is the CDF of the exponential distribution and λ is the parameter of the exponential distribution; (2d) Find the area with large grayscale change as the target, set the false alarm rate in advance, and calculate the binarization threshold:

in,

is the inverse function, _pfa is the preset probability value; (2e) Use a predefined probability value to calculate the binarization threshold to classify the pixels. The pixel amplitude value greater than th is the foreground, and the pixel amplitude value less than th is the background. In a binary image, '1' is the foreground, and '0' is the foreground. background; (2f) Take the intersection map of the two binary images obtained, that is, perform the "AND" Boolean operation pixel by pixel to eliminate the scene targets at the previous moment and the next moment, so as to uniquely retain the current position of the moving vehicle target. 3. the detection and tracking method of moving target in the satellite video according to claim 1, is characterized in that, in step (2), the concrete steps based on the discrimination of regional growth and multiple clues are: (3a) Using the region growing algorithm based on Gaussian distribution, using all foreground pixels of a connected region detected by the detector to estimate the parameters of the Gaussian distribution; the PDF of the Gaussian distribution is;

where μ and σ represent the mean and standard deviation, respectively, and the CDF of the Gaussian distribution is:

Among them, erf(*) represents the correlation error function; (3b) Predefined lower bound probability

and the upper bound probability

Calculate the lower and upper bounds of the magnitude value separately, namely:

in,

and

represent the lower and upper bounds of the threshold, respectively; (3c) Re-judging the foreground target in this area, the amplitude value is in

The pixels within the range are the foreground; (3d) Retain all the pixels connected to the original foreground, and construct a new connected region, that is, the result of region growth; (3e) Finally, using area, duty cycle, length, and eccentricity as morphological cues, the results of region growth are morphologically discriminated with the aim of culling non-vehicle targets from the detected targets, where the definition of duty cycle Yes, the ratio of the connected region pixels to the area of the connected region bounding box. 4. the detection and tracking method of moving target in the satellite video according to claim 1, is characterized in that, in step (3), the concrete steps of the target tracking based on Kalman filter KF are: (4a) KF is suitable for target tracking of linear motion. The nonlinear motion is decomposed into a series of linear motions. First, only the KF tracking problem of one moving target is considered. The dynamic equation of the prediction system is: x _i =F _i ·x _i-1 +v _i (8) Among them, F _i is the evolution matrix of the ith frame moment of the satellite video, vi is the noise vector, x _{i -1} is the state vector of the i-1th frame moment, and x _i is the state vector of the current moment; The state vector x _i is defined as: x _i = [x, y, v _x , v _y , a _x , a _y ] ^T (9) Among them, x, y, v _x , v _y , a _x , a _y represent the position in the X-axis direction, the position in the Y-axis direction, the velocity in the X-axis direction, the velocity in the Y-axis direction, the acceleration in the X-axis direction, Y acceleration in the axis direction; Assuming that the vehicle target moves in a straight line with uniform acceleration at the moment between two frames, the evolution matrix F _i is:

Among them, τ represents the time interval between two frames in the video, in "seconds", which is the inverse of the frame rate; (4b) Using the area and position of the detected vehicle as the observation value, the observation equation is obtained as:

Among them, y _i , H _i and _ni represent the measurement vector, measurement matrix and measurement noise respectively; The observation matrix H _i is:

(4c) The posterior probability density function PDF of the state vector x _i is derived by Bayes' theorem, and then simplified by the chain rule, namely:

Among them, the probability p(x _i |y _1:i-1 ) is derived by the Chapman-Kolmogorov equation, namely: p(x _i |y _{1 : i-1} )=∫p(x _i |x _i-1 )p(x _i-1 |y _1:i-1 )dx _i-1 (14) (4d) Using formulas (13) and (14), the posterior probability density function PDF of the state vector x _i is obtained:

(4e) Combine the observed value and the predicted value, using formula (15), estimate the current state, and continuously update the state of the system. 5. the detection and tracking method of moving target in satellite video according to claim 1 is characterized in that, the concrete steps that the multiple vehicle targets detected in step (3) are associated with the multiple tracks being tracked are: (5a) In the target tracking technology based on Kalman filter, the motion trajectory is associated with the moving target through the Hungarian algorithm, and the associated target is tracked; (5b) Using the nearest neighbor search based on SSIM, the trajectory of the unassociated target is re-searched for the moving target associated with it. If the association is successful, the tracking is continued after correction. If the trajectory still does not have an associated target, then Trajectory terminated.

Images