CN111161325B - Three-dimensional multi-target tracking method based on Kalman filtering and LSTM - Google Patents
Three-dimensional multi-target tracking method based on Kalman filtering and LSTM Download PDFInfo
- Publication number
- CN111161325B CN111161325B CN201911416915.XA CN201911416915A CN111161325B CN 111161325 B CN111161325 B CN 111161325B CN 201911416915 A CN201911416915 A CN 201911416915A CN 111161325 B CN111161325 B CN 111161325B
- Authority
- CN
- China
- Prior art keywords
- frame
- dimensional
- lstm
- dimensional target
- track
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30241—Trajectory
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a three-dimensional target tracking method based on Kalman filtering and LSTM, which comprises the following steps: initializing the track of an input three-dimensional target frame; updating and denoising the three-dimensional target frame track by using a constant-rate Kalman filtering algorithm to obtain a predicted track set; carrying out data association on the predicted track and the three-dimensional target frame of the current frame by using a Hungary algorithm and updating a Kalman filter; using the denoised three-dimensional target frame sequence for training a long-short-term memory network; and tracking and predicting the three-dimensional target by using a constant-rate Kalman filtering algorithm and a Hungary algorithm and the trained LSTM. The traditional target tracking method based on Kalman filtering has the problem of insufficient nonlinear fitting capability, and the method is the biggest difference from the traditional method in that the strong characteristic extraction capability of a deep learning model LSTM is used, so that a more complex motion model can be fitted, the tracking result is smoother, and meanwhile, the speed of a tracking system is improved.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a three-dimensional multi-target tracking method based on Kalman filtering and LSTM.
Background
Three-dimensional multi-target tracking is an important component of video processing and computer vision, is widely applied to automatic driving, benefits from improvement of accuracy of a detection algorithm, and is mainly based on detection. In a tracking algorithm based on detection, a target detector detects an image of each frame to obtain a target detection frame, and then the motion information and frame information of the target are utilized to correlate and track the target frame to obtain a track of the target.
When the traditional tracking algorithm based on detection is applied to three-dimensional multi-target tracking, the real-time performance of the tracking algorithm is seriously dependent on the detection speed of a three-dimensional target detector. The current mainstream three-dimensional target detector has a low speed, so that the existing tracking algorithm based on detection cannot be directly applied to three-dimensional multi-target tracking. Meanwhile, the three-dimensional frame of the tracking target is obvious in jitter due to noise of the three-dimensional target frame output by the detector, so that the tracking result is not smooth and stable enough.
Disclosure of Invention
In order to solve the above technical problems, an embodiment of the present invention provides a three-dimensional multi-target tracking method based on kalman filtering and LSTM, including:
s1, initializing tracks of input three-dimensional target frames, wherein whether a track is newly established is determined according to whether the three-dimensional frames of a t+1st frame are matched with the three-dimensional frames of the t frame, and because false positive samples possibly exist in a three-dimensional target detection result, the new track is initialized only when two continuous frames have the same target;
s2, updating and denoising the frame track of the t-frame three-dimensional target by using a constant-rate Kalman filtering algorithm to obtain a real track set
Then predicting to obtain a predicted track set +.>
Wherein the prediction track set->
Representing a predicted track set of the t+1st frame;
s3, carrying out data association on the predicted track and the three-dimensional target frame of the current frame by using a Hungary algorithm and updating a Kalman filter;
s4, using the denoised three-dimensional target frame sequence for training a long-short-term memory network LSTM;
s5, tracking and predicting a three-dimensional target by using a constant-rate Kalman filtering algorithm and a Hungary algorithm and a trained LSTM, if each frame is used for three-dimensional target frame detection, a three-dimensional target detection result can be obtained every F frames because the main stream three-dimensional target detector generally has the problem of low detection speed, and the middle F frames are predicted by using the LSTM model, so that the tracking result is smoother and the speed is increased.
Further, the track initialization process in the step S1 is as follows:
by using
An ith three-dimensional object border representing a t-th frame,>
wherein x, y, z, l, w, h and θ respectively represent the x-axis coordinate, the y-axis coordinate and the z-axis coordinate of the three-dimensional target frame in the camera coordinate system;
length, width and height of three-dimensional target frame and observation angle of target, set D t A set representing all three-dimensional object frames of the t-th frame;
if the ratio of cross-over
That is, when the intersection ratio (Intersection over Union, ioU) of the ith three-dimensional target frame of the t frame and the jth three-dimensional target frame of the (t+1) frame is greater than or equal to the threshold value threshold, a track is newly created>
Wherein k represents the kth track, and the track set at time t+1 is denoted as T t+1 The remaining three-dimensional object frames are discarded.
Further, the data association in the step S3 is specifically as follows:
three-dimensional target frame set D of current t-th frame t Prediction track set T obtained by Kalman filtering algorithm t p The input Hungary algorithm of the system obtains a data association result;
in the result, three-dimensional object frame set D t Divided into two sets
Respectively representing a matched three-dimensional detection frame set and an unmatched three-dimensional target frame set by +.>
Update, for collection->
Step S1 is executed to initialize a three-dimensional target track;
in the result of the data association, the unmatched tracks will be discarded and the matched tracks remain.
Further, the training long-short-time memory network LSTM of step S4 is specifically as follows:
setting a time step L of LSTM, cutting tracks in a track set according to a frame number L+1, discarding tracks with a length less than L+1, inputting a three-dimensional target frame sequence of the previous L frames as LSTM, and training the LSTM to obtain a three-dimensional target track prediction model by using a label of the last frame as LSTM.
Further, the tracking and predicting of the three-dimensional object in the step S5 is specifically as follows:
setting an interval frame number F, if the current track frame number is not equal to N (F+1) +1 and is larger than L, starting an LSTM network to predict a three-dimensional target frame of a next frame, wherein N is a natural number, acquiring the three-dimensional target frame every other F frames, and performing target tracking on the three-dimensional target frame sequence of the interval F frames by adopting a constant rate Kalman filtering algorithm and a Hungary algorithm, namely executing steps S1 to S3 to obtain a denoised three-dimensional target frame, wherein the union of two model prediction results is a final tracking result.
Compared with the prior art, the invention has the following advantages and effects:
high efficiency: according to the invention, the LSTM is utilized to predict the three-dimensional target frame of the middle interval frame number, so that the frequency of acquiring the three-dimensional target frame by a tracking algorithm is reduced, and the speed of the three-dimensional multi-target tracking method is greatly improved;
stability: because of the strong nonlinear fitting capability of the depth network LSTM, the track jitter output by the tracking algorithm is smaller and more stable.
Drawings
Fig. 1 is a schematic diagram of a kalman filter and LSTM fusion for processing different frames according to a first embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
First embodiment
The embodiment discloses a layer-by-layer updating algorithm based on generation of an countermeasure network, which specifically comprises the following steps:
step S1, initializing tracks of input three-dimensional target frames, wherein the initialization determines whether to newly establish a track according to whether the three-dimensional frames of the t+1st frame are matched with the three-dimensional frames of the t frame, and because false positive samples possibly exist in the three-dimensional target detection result, the new track is initialized only when the same target appears in two continuous frames. The specific process is as follows:
by using
An ith three-dimensional object border representing a t-th frame,>
wherein x, y, z, l, w, h and θ respectively represent the x-axis coordinate, y-axis coordinate and z-axis coordinate of the three-dimensional target frame in the camera coordinate system, the length, width and height of the three-dimensional target frame, the observation angle of the target, and the set D t Representing the set of all three-dimensional object frames of the t-th frame, if the cross-ratios are
Taking threshold=0.7, that is, the intersection ratio IoU of the ith three-dimensional target frame of the t frame and the jth three-dimensional target frame of the t+1st frame is greater than or equal to the threshold, then creating a track
Wherein k represents the kth track, and the track set at time t+1 is denoted as T t+1 The remaining three-dimensional object frames are discarded.
S2, updating and denoising the frame track of the t-frame three-dimensional target by using a constant-rate Kalman filtering algorithm to obtain a real track set
Then predicting to obtain a predicted track set +.>
Wherein->
Representing a predicted track set of the t+1st frame;
and S3, carrying out data association on the predicted track and the three-dimensional target frame of the current frame by using a Hungary algorithm and updating a Kalman filter. The specific process is as follows:
three-dimensional target frame set D of current t-th frame t Prediction track set T obtained by Kalman filtering algorithm t p The input Hungary algorithm of (2) to obtain a data association result, and a three-dimensional target frame set D is obtained in the result t Divided into two sets
Respectively representing a matched three-dimensional detection frame set and an unmatched three-dimensional target frame set by +.>
Update, for collection->
And step S1 is executed to initialize the three-dimensional target track, in the result of data association, the unmatched track is discarded, and the matched track is reserved.
And S4, using the denoised three-dimensional target frame sequence for training the long-short-term memory network LSTM. The specific process is as follows:
setting a time step L of LSTM, taking L=30, wherein the video frame rate is 30 frames per second, cutting tracks in a track set according to the frame number L+1, discarding tracks with the length less than L+1, inputting a three-dimensional target frame sequence of the previous L frames as LSTM, and training the LSTM to obtain a three-dimensional target track prediction model by using a label of the last frame as LSTM.
Step S5, tracking and predicting three-dimensional targets by using a constant-rate Kalman filtering algorithm and a Hungary algorithm and a trained LSTM, if each frame is used for three-dimensional target frame detection, a three-dimensional target detection result can be obtained every F frames because the main stream three-dimensional target detector generally has the problem of low detection speed, F=5 is taken here, the middle F frames are predicted by using the LSTM model, and the tracking result can be smoother and the speed is increased. The specific process is as follows:
setting an interval frame number F, if the current track frame number is not equal to N (F+1) +1 and is larger than L, starting an LSTM network to predict a three-dimensional target frame of a next frame, wherein N is a natural number, acquiring the three-dimensional target frame every other F frames, and performing target tracking on the three-dimensional target frame sequence of the interval F frames by adopting a constant rate Kalman filtering algorithm and a Hungary algorithm, namely executing steps S1 to S3 to obtain a denoised three-dimensional target frame, wherein the union of two model prediction results is a final tracking result.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (4)
1. The three-dimensional multi-target tracking method based on Kalman filtering and LSTM is characterized by comprising the following steps of:
s1, initializing tracks of input three-dimensional target frames, and determining whether to establish a track according to whether the three-dimensional frames of the (t+1) th frame are matched with the three-dimensional frames of the (t) th frame;
s2, updating and denoising the frame track of the t frame three-dimensional target by using a constant-rate Kalman filtering algorithm to obtain a real track set, and predicting to obtain a predicted track set, wherein the predicted track set represents the predicted track set of the t+1st frame;
s3, carrying out data association on the predicted track and the three-dimensional target frame of the current frame by using a Hungary algorithm and updating a Kalman filter;
s4, using the denoised three-dimensional target frame sequence for training a long-short-term memory network LSTM;
s5, tracking and predicting a three-dimensional target with the trained LSTM by using a constant-rate Kalman filtering algorithm and a Hungary algorithm;
the tracking and predicting of the three-dimensional target in step S5 is specifically as follows:
setting an interval frame number F, if the current track frame number is not equal to N (F+1) +1 and is larger than L, starting an LSTM network to predict a three-dimensional target frame of a next frame, wherein N is a natural number, acquiring the three-dimensional target frame every other F frames, and performing target tracking on the three-dimensional target frame sequence of the interval F frames by adopting a constant rate Kalman filtering algorithm and a Hungary algorithm, namely executing steps S1 to S3 to obtain a denoised three-dimensional target frame, wherein the union of two model prediction results is a final tracking result.
2. The three-dimensional multi-target tracking method based on kalman filtering and LSTM according to claim 1, wherein the trajectory initialization process of step S1 is as follows:
by using
An ith three-dimensional object border representing a t-th frame,>
wherein x, y, z, l, w, h and θ respectively represent the x-axis coordinate, the y-axis coordinate and the z-axis coordinate of the three-dimensional target frame in the camera coordinate system;
length, width and height of three-dimensional target frame and observation angle of target, set D t A set representing all three-dimensional object frames of the t-th frame;
if the ratio of cross-over
Namely, when the intersection ratio of the ith three-dimensional target frame of the t frame and the jth three-dimensional target frame of the t+1st frame is greater than or equal to a threshold value threshold; newly created track +.>
Wherein k represents the kth track, and the track set at time t+1 is denoted as T t+1 The remaining three-dimensional object frames are discarded.
3. The three-dimensional multi-target tracking method based on kalman filtering and LSTM according to claim 1, wherein the data association in step S3 is specifically as follows:
three-dimensional target frame set D of current t-th frame t Obtained by Kalman filtering algorithmIs set of predicted trajectories T of (1) t p The input Hungary algorithm of the system obtains a data association result;
in the result, three-dimensional object frame set D t Divided into two sets
Respectively representing a matched three-dimensional detection frame set and an unmatched three-dimensional target frame set by +.>
Update, for collection->
Step S1 is executed to initialize a three-dimensional target track;
in the result of the data association, the unmatched tracks will be discarded and the matched tracks remain.
4. The three-dimensional multi-target tracking method based on kalman filtering and LSTM according to claim 1, wherein the training long-short-term memory network LSTM of step S4 is specifically as follows:
setting a time step L of LSTM, cutting tracks in a track set according to a frame number L+1, discarding tracks with a length less than L+1, inputting a three-dimensional target frame sequence of the previous L frames as LSTM, and training the LSTM to obtain a three-dimensional target track prediction model by using a label of the last frame as LSTM.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911416915.XA CN111161325B (en) | 2019-12-31 | 2019-12-31 | Three-dimensional multi-target tracking method based on Kalman filtering and LSTM |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911416915.XA CN111161325B (en) | 2019-12-31 | 2019-12-31 | Three-dimensional multi-target tracking method based on Kalman filtering and LSTM |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111161325A CN111161325A (en) | 2020-05-15 |
| CN111161325B true CN111161325B (en) | 2023-05-23 |
Family
ID=70560260
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911416915.XA Active CN111161325B (en) | 2019-12-31 | 2019-12-31 | Three-dimensional multi-target tracking method based on Kalman filtering and LSTM |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111161325B (en) |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111932580A (en) * | 2020-07-03 | 2020-11-13 | 江苏大学 | Road 3D vehicle tracking method and system based on Kalman filtering and Hungary algorithm |
| CN112965494A (en) * | 2021-02-09 | 2021-06-15 | 武汉理工大学 | Control system and method for pure electric automatic driving special vehicle in fixed area |
| CN113763434B (en) * | 2021-09-26 | 2024-02-02 | 东风汽车集团股份有限公司 | Target track prediction method based on Kalman filtering multi-motion model switching |
| CN116228989A (en) * | 2023-03-30 | 2023-06-06 | 北京数原数字化城市研究中心 | Three-dimensional track prediction method, device, equipment and medium |
| CN117192063B (en) * | 2023-11-06 | 2024-03-15 | 山东大学 | Water quality prediction method and system based on coupled Kalman filtering data assimilation |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CA3041651A1 (en) * | 2016-10-25 | 2018-05-03 | Deep North, Inc. | Vision based target tracking using tracklets |
| CN109816690A (en) * | 2018-12-25 | 2019-05-28 | 北京飞搜科技有限公司 | Multi-target tracking method and system based on depth characteristic |
| CN110400347B (en) * | 2019-06-25 | 2022-10-28 | 哈尔滨工程大学 | Target tracking method for judging occlusion and target relocation |
| CN110399808A (en) * | 2019-07-05 | 2019-11-01 | 桂林安维科技有限公司 | A kind of Human bodys' response method and system based on multiple target tracking |
-
2019
- 2019-12-31 CN CN201911416915.XA patent/CN111161325B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN111161325A (en) | 2020-05-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111161325B (en) | Three-dimensional multi-target tracking method based on Kalman filtering and LSTM | |
| CN109636829B (en) | Multi-target tracking method based on semantic information and scene information | |
| CN106846359B (en) | Moving target rapid detection method based on video sequence | |
| EP2858008A2 (en) | Target detecting method and system | |
| CN110853078B (en) | On-line multi-target tracking method based on shielding pair | |
| WO2021036367A1 (en) | Target tracking method and apparatus based on measurement allocation | |
| CN111080673B (en) | Anti-occlusion target tracking method | |
| CN101527044A (en) | Automatic segmenting and tracking method of multiple-video moving target | |
| CN107516321A (en) | A kind of video multi-target tracking and device | |
| CN110659600B (en) | Object detection method, device and equipment | |
| CN110555868A (en) | method for detecting small moving target under complex ground background | |
| CN113593219B (en) | Traffic flow statistical method and device, electronic equipment and storage medium | |
| CN105374049B (en) | Multi-corner point tracking method and device based on sparse optical flow method | |
| CN112528786A (en) | Vehicle tracking method and device and electronic equipment | |
| CN111402293B (en) | Intelligent traffic-oriented vehicle tracking method and device | |
| CN110443829A (en) | It is a kind of that track algorithm is blocked based on motion feature and the anti-of similarity feature | |
| WO2019172172A1 (en) | Object tracker, object tracking method, and computer program | |
| CN113763427A (en) | Multi-target tracking method based on coarse-fine shielding processing | |
| CN112364865A (en) | Method for detecting small moving target in complex scene | |
| CN112330589A (en) | Method and device for estimating pose and computer readable storage medium | |
| CN109118516A (en) | A kind of target is from moving to static tracking and device | |
| CN111178261A (en) | Face detection acceleration method based on video coding technology | |
| CN116630376A (en) | Unmanned aerial vehicle multi-target tracking method based on ByteTrack | |
| CN116012421A (en) | Target tracking method and device | |
| CN115861386A (en) | Unmanned aerial vehicle multi-target tracking method and device through divide-and-conquer association |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |