CN108492324A - Aircraft method for tracing based on fully-connected network and Kalman filter - Google Patents

Abstract

The present invention relates to a kind of aircraft method for tracing based on fully-connected network and Kalman filter, includes the following steps：Video is detected frame by frame using fully-connected network R FCN, obtains the bounding box of previous frame image, so that trajectory corrector uses；State vector, description airplane motion track are built, state vector should represent the position of target aircraft central point, show the size and aspect ratio of bounding box again；Kalman filter and extended Kalman filter are combined, subvector is isolated from the state vector of structure to describe moving target；When testing result deviation is larger, effective range is limited according to the size of target object and improves detection speed, corrects the position of bounding box in consecutive frame, to realize the correction to movement locus.

Description

airplane tracking method based on full-connection network and Kalman filter

Technical Field

The invention belongs to the field of computer vision, and relates to a deep learning tracking method for an airplane in a video.

Background

Aircraft tracking is an important technology in the fields of aviation safety and the like, and the introduction of scientific and technological means to strengthen safety measures in military reconnaissance is also gradually paid national attention. The target tracking is an important field in computer vision, and can realize the functions of checking and even tracking the airplane in the acquired video image.

In recent years, with the development of deep learning, machine learning algorithms are gradually applied to various visual fields, and target detection and tracking technologies based on deep learning are rapidly developed. Compared with the traditional method, the tracking performance is greatly improved. Popular object detection strategies include two categories, area proposal and area step. However, the accuracy of aircraft tracking is mainly affected by complex environmental conditions, and the visual tracking algorithm still has some challenging problems, such as sudden motion, attitude change, deformation, occlusion, background clutter, illumination or viewpoint change, etc., which all cause the accuracy of tracking to be reduced, and even cause the tracking to fail. At present, no effective algorithm is available for solving the problem of airplane tracking.

Disclosure of Invention

The invention aims to establish a more accurate airplane tracking method. The aircraft tracking method provided by the invention comprises three main parts, namely an R-FCN-based detection model, a Kalman filter-based state estimation model and an aircraft motion track correction module. The technical scheme is as follows:

an aircraft tracking method based on a fully-connected network and a Kalman filter comprises the following steps:

the first step is as follows: detecting the video frame by using a full connection network R-FCN to obtain a bounding box of a previous frame image for track correction;

the second step is that: constructing a state vector, describing an airplane motion track, wherein the state vector not only expresses the position of the center point of the target airplane, but also shows the size and the aspect ratio of the bounding box;

the third step: in order to avoid target drift caused by detection failure of target detection on a certain frame, combining a Kalman filter and an extended Kalman filter, and separating a sub-vector from a constructed state vector to describe a moving target; the specific method comprises the following steps:

(1) kalman filter process linear part: a sub-vector representing the position of the central point of the airplane in the state vector is approximated by a linear model, then Kalman gain is calculated according to the uncertainty of the prediction result and the current observation result, and the prediction result and the observation result are weighted and averaged to obtain the state estimation of the current moment and the uncertainty of the state estimation;

(2) the extended kalman filter is used to fit the non-linear part that is not suitable for the linear model: creating a state subvector which can represent the size and the aspect ratio of a bounding box in the same way as in the step (1), but the related state matrix and the mapping matrix are no longer constant matrixes, and obtaining the state estimation of the nonlinear part at the current moment and the uncertainty of the state estimation;

(3) adding a nonlinear part into a linear system to describe the motion state of the airplane;

the fourth step: when the deviation of the detection result is large, limiting an effective range according to the size of a target object to improve the detection speed, and correcting the position of a bounding box in an adjacent frame so as to realize the correction of the motion track; otherwise, the target is taken as the center, a bounding box is drawn and input into the detection network for training, and the correction formula is as follows:

wherein, delta_a,δ_bRespectively representing the confidence degrees of detection results between the previous frame and the next frame, wherein the higher the value of the confidence degree is, the more accurate the model is; w is a_a,w_b,w_cRespectively representing the widths, h, of the preceding and following frames and the corrected bounding box_a,h_b,h_cRespectively representing the heights of the front frame and the rear frame and the bounding box after correction; (x)_a,y_a),(x_b,y_b),(x_c,y_c) Respectively representing horizontal and vertical coordinates of the central points of the front and rear frames and the bounding box after correction;

the fifth step: collecting videos containing airplane motion, uniformly processing the videos into videos with fixed lengths, jointly forming a training database, wherein the videos are randomly divided into two parts, 80% of the videos are input into an R-FCN as a training set, and predicting the remaining 20% of the videos according to a model obtained through training to obtain an airplane tracking result.

The specific airplane tracking algorithm provided by the invention is realized on the basis of an R-FCN and a Kalman filter. The R-FCN is a detection model used for obtaining position information of the airplane. In order to reduce the detection time, we also cut out a specific area in each frame according to the position of the bounding box in the previous frame. In addition, the size of the detection area may also be changed depending on the size of the target. The kalman filter is used as an evaluation model to adjust the predicted motion trace. When the detection difference between adjacent frames is large, the bounding box of the next frame can be adjusted according to the detection result in order to improve the determination rate of detection. Therefore, the method can accurately detect and track the motion trail of the airplane.

Drawings

FIG. 1: flow chart

FIG. 2: schematic diagram of correction method

Detailed Description

The invention provides a method for using R-FCN^[1]And Kalman filter^[2,3]The airplane tracking method based on the KF comprises three main parts, namely an R-FCN-based detection model, a KF-based state estimation model and an airplane motion track correction module. Specifically, the method can be represented as the following steps:

the first step is as follows: the bounding box (bounding box) of the previous frame is obtained through the R-FCN.

The specific method comprises the following steps:

(1) relative spatial position information of a Region of interest (ROI) is encoded by creating a spatially sensitive score map according to the R-FCN principle. The present invention generates a 4-dimensional vector v ═ v (v) for each ROI by a bounding box regression method_x,v_y,v_w,v_h) For subsequent calculation of bounding boxes. Wherein v is_x,v_yX, y coordinates, v, representing the center point_w,v_hRepresenting the width and height of the bounding box.

(2) Dividing the ROI into upper left,Four word areas of left lower, right upper and right lower, these sub-areas are used as the score map. The bounding box vertices can be respectively denoted as B_tl,B_tr,B_bl,B_btAnd v is_x,v_y,v_w,v_hThe relationship of (a) is expressed as follows:

the second step is that: constructing state vectorsDescribing the motion trail of the airplane.

Wherein,the position of the center point of the object is represented,representing the bounding box ratios and aspect ratios, and black dots representing derivations. The step mainly processes the linear part, and the specific method is as follows:

(1)separating the subvectors from the constructed state vector by approximation of a linear modelTo describe the moving object.

x_k＝Ax_k-1+Bu_k+w_k(2)

z_k＝Hx_k+v_k(3)

Wherein x represents the state vector of the system, z is the observed value, A is the state transition matrix, and B and u constitute the control part which can be ignored in the non-controllable system to which the invention is applied. H is an observation model, i.e., an observation matrix, which can map the true state to the sensing space. w and v represent noise during state update and observation, respectively. Previous studies have shown that the above noise follows a gaussian distribution. Equation (2) is referred to as the equation of state, equation (3) is referred to as the observation equation, and the subscript k denotes the value at time k.

(2) Known from the KF principle:

wherein, P is an error covariance matrix between the predicted value and the true value, which is used for representing the uncertainty of the prediction result, and Q is a new uncertainty added in the prediction process. Equation (4) shows that the current state is predicted from the state at the previous time plus external input. Equation (5) represents the previously existing uncertainty, which in turn adds a new uncertainty Q in the prediction process.

(3) Calculating Kalman gain K and K-th predicted valueCan be calculated by the following formula:

equation (6) represents the uncertainty of the prediction resultComputing Kalman gain from the uncertainty R of the observation(weight) K. Formula (7) shows that the prediction result and the observation result are weighted and averaged to obtain the state estimation of the current time.

(4) UpdatingAnd representing the uncertainty of the state estimation.

The third step: the non-linear part of the s and r, which are not fit to the linear model, was fitted using EKF. Creating sub-vectors in the same manner as in the third stepBut the state matrix a and the mapping matrix H are no longer constant matrices, they are represented by the following equations:

wherein, the matrix F_k，H_kDerived from Jacobian (Jacobian) matrix, which correspond to A and H in KF, respectively.

The fourth step: the motion state of the airplane can be described by adding the nonlinear part into the linear system.

The fifth step: and correcting the motion trail. It is determined whether the bounding box is capable of properly detecting an aircraft in motion. And if the value of the overlapping degree (IOU) of the bounding box and the window where the airplane is located is larger than a predefined threshold value T, modifying the position and the size of the current detection frame according to the bounding box of the previous frame. Otherwise, the target is taken as the center, and a bounding box is drawn and input into the detection network for training. The specific correction formula is as follows:

wherein, delta_a,δ_bRespectively representing the front and rear framesThe higher the confidence of the detection result, the more accurate the model. w is a_a,w_b,w_cRespectively representing the widths, h, of the preceding and following frames and the corrected bounding box_a,h_b,h_cThe heights of the preceding and following frames and the corrected bounding box are indicated, respectively. (x)_a,y_a),(x_b,y_b),(x_c,y_c) The horizontal and vertical coordinates of the center points of the previous and following frames and the corrected bounding box are indicated, respectively.

And a sixth step: videos containing airplane motion are collected and processed into videos with fixed length for convenient training, and a training database is formed together. The video is randomly divided into two parts, 80% of the video is input into R-FCN as a training set, and the remaining 20% of the video is predicted according to a model obtained by training to obtain an airplane tracking result.

Reference documents:

[1]Dai J,Li Y,He K,Sun J(2016)R-fcn:Object detection via region-basedfully convolutional networks.In:Advances in Neural Information ProcessingSystems,pp 379C387.

[2]Kalman RE,et al(1960)A new approach to linear filtering andprediction problems. Journal of basic Engineering 82(1):35C45.

[3]Kalman RE,Bucy RS(1961)New results in linear filtering andprediction theory.Journal of basic engineering 83(1):95C108。

Claims (1)

1. An aircraft tracking method based on a fully-connected network and a Kalman filter comprises the following steps:

the first step is as follows: detecting the video frame by using a full connection network R-FCN to obtain a bounding box of a previous frame image for track correction;

the second step is that: constructing a state vector, describing an airplane motion track, wherein the state vector not only expresses the position of the center point of the target airplane, but also shows the size and the aspect ratio of the bounding box;

the third step: in order to avoid target drift caused by detection failure of target detection on a certain frame, combining a Kalman filter and an extended Kalman filter, and separating a sub-vector from a constructed state vector to describe a moving target; the specific method comprises the following steps:

(1) kalman filter process linear part: a sub-vector representing the position of the central point of the airplane in the state vector is approximated by a linear model, then Kalman gain is calculated according to the uncertainty of the prediction result and the current observation result, and the prediction result and the observation result are weighted and averaged to obtain the state estimation of the current moment and the uncertainty of the state estimation;

(2) the extended kalman filter is used to fit the non-linear part that is not suitable for the linear model: creating a state subvector which can represent the size and the aspect ratio of a bounding box in the same way as in the step (1), but the related state matrix and the mapping matrix are no longer constant matrixes, and obtaining the state estimation of the nonlinear part at the current moment and the uncertainty of the state estimation;

(3) adding a nonlinear part into a linear system to describe the motion state of the airplane;

the fourth step: when the deviation of the detection result is large, limiting an effective range according to the size of a target object to improve the detection speed, and correcting the position of a bounding box in an adjacent frame so as to realize the correction of the motion track; otherwise, the target is taken as the center, a bounding box is drawn and input into the detection network for training, and the correction formula is as follows:

wherein, delta_a,δ_bRespectively representing the confidence degrees of detection results between the previous frame and the next frame, wherein the higher the value of the confidence degree is, the more accurate the model is; w is a_a,w_b,w_cRespectively representing the widths, h, of the preceding and following frames and the corrected bounding box_a,h_b,h_cRespectively representing preceding and following frames and corrected framingThe height of the box; (x)_a,y_a),(x_b,y_b),(x_c,y_c) Respectively representing horizontal and vertical coordinates of the central points of the front and rear frames and the bounding box after correction;

the fifth step: collecting videos containing airplane motion, uniformly processing the videos into videos with fixed lengths, jointly forming a training database, wherein the videos are randomly divided into two parts, 80% of the videos are input into an R-FCN as a training set, and predicting the remaining 20% of the videos according to a model obtained through training to obtain an airplane tracking result.