CN115953437A - Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation - Google Patents
Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation Download PDFInfo
- Publication number
- CN115953437A CN115953437A CN202310119716.2A CN202310119716A CN115953437A CN 115953437 A CN115953437 A CN 115953437A CN 202310119716 A CN202310119716 A CN 202310119716A CN 115953437 A CN115953437 A CN 115953437A
- Authority
- CN
- China
- Prior art keywords
- target
- frame
- tracking
- rectangular frame
- rectangular
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Image Analysis (AREA)
Abstract
The invention provides a multi-target real-time tracking method for intelligent driving scene oriented fusion of visual light stream feature point tracking and motion trend estimation, which extracts light stream features from an area where a target tracked in a previous frame is located, predicts the position of the target in a current frame, predicts the motion trend of a rectangular frame through a rectangular frame motion trend estimation algorithm, and compares two prediction results to obtain the predicted position of the target in the previous frame in the current frame; determining the accuracy of target tracking according to the characteristics of similarity, overlapping degree, difference degree and the like of the target detection result of the current frame; and finally, performing position filtering on the successfully tracked target to improve the stability of the target rectangular frame, and outputting the target rectangular frame to the intelligent driving decision module. The invention is mainly applied to an intelligent driving visual target tracking module and provides stable and reliable target output for the system.
Description
Technical Field
The invention relates to the field of intelligent driving visual multi-target tracking, in particular to a multi-target real-time tracking method integrating visual optical flow characteristic point tracking and motion trend estimation.
Background
The research of the visual target tracking technology has very important significance in the field of intelligent driving perception, and the motion state and the trend of the target can be estimated through the technology, so that the motion direction and the motion speed of the target are calculated, and a basis is provided for an intelligent driving decision module.
The current visual target tracking technology has more research bases, and can be divided into the following steps according to different algorithm types: correlation filtering, optical flow, motion state estimation, deep learning and the like. Early target tracking algorithms mainly included: media Flow, kalman filtering, etc.; the algorithm based on the relevant filtering mainly comprises MOSSE, CSK, KCF and the like; in recent years, target tracking algorithms based on deep learning are becoming more popular, such as: ECO, MDNet, SANet, deepsORT.
However, in the application of the intelligent driving field, because the vehicle moves fast and the scene is complex, how to solve the problems of target shielding, fast movement, light change and the like is a big difficulty, and meanwhile, the number of targets is large, the real-time requirement is high, the chip computing power is limited, and higher requirements are also put forward on the performance of the tracking algorithm.
At present, many algorithms cannot meet actual requirements in the deployment process, such as: algorithms such as media Flow and KCF are single-target tracking, and when the algorithms are expanded to multi-target tracking, the algorithms cannot achieve real-time performance; the algorithm based on deep learning needs to label the video tracking data set independently, so that the cost is high, and the performance is not as expected.
Therefore, a multi-target real-time tracking algorithm applicable to an intelligent driving scene is needed.
Disclosure of Invention
In order to solve the above-mentioned defects in the prior art, the present invention aims to provide a multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation, which aims to solve the problem of complex scene tracking, improve tracking accuracy and algorithm operation efficiency, and make the tracking result smoother.
According to a first aspect of the present invention, a multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation is provided, which includes:
step 10: by passingImage acquisition equipment acquires image M of current ith frame in real time i Detection of M Using the YOLO target detection Algorithm i The target in (1) obtains the rectangular frame position of the target, and each target in the ith frame is added into the target detection result list Objects in the ith frame i 。
Step 20: if the ith frame is the first frame, for Objects i Configuring a new ID, setting the first parameter of the object, tracked _ frames to 1, the second parameter of the object, patched _ frames to 0, and Objects i Target tracking result list assigned to current ith frame
Skipping to step 70; if the ith frame is not the first frame, execution continues at step 30.
And step 30: target tracking result list for frame i-1
Is processed using a rectangular-box motion trend estimation algorithm to obtain a list of target motion state predicted locations->
Processing by using a rectangular frame area image optical flow characteristic point tracking algorithm to obtain a target optical flow prediction position list->
Step 40: comparison of
And &>
Calculating the position of the rectangular frame of the target j with the same ID, and calculating the overlapping degree of the first rectangular frame IoU 1 Based on the first degree of overlap a predetermined threshold value is->
Is judged if >>
If the target tracking is successful, adding the target into the position tracking prediction result list of the target of the i-1 th frame in the i-th frame, and keeping the position of the target in the prediction result list>
And adding 1 to the patched _ frames of the target; if->
When the target tracking fails, the tracked _ frames of the target is set to 0.
Step 50: for Objects i Each of the objects respectively with
Is calculated, including a second degree of overlap match IoU 2 Normalizing the distance of the central point Dist and the difference Diff, and determining the similarity SimM of the image area, if the matching degree meets the hyper-parameter threshold, then IoU is used 2 And as the weights of the two target matching degrees, if the matching degree does not meet the hyper-parameter threshold, setting the weight corresponding to the two target matching degrees to 0, and constructing a matching degree weight matrix based on the weight.
Step 60: according to the matching degree weight matrix, obtaining a matching relation between targets of an i-1 th frame and an i-th frame by using a KM optimal matching algorithm, assigning the ID of the target in the i-1 th frame to the matching target of the i-th frame for the target which is successfully matched, adding 1 to a tracked _ frames, and setting a patched \\/u frame as 0; for targets that do not match in the ith frame, a new ID is assigned, the tracked _ frames is set to 1, and the patched_frames is set to 0; for the unmatched targets in the (i-1) th frame, keeping the ID unchanged, and adding 1 to the patched _ frames; each object is added
And (4) listing.
Step 70: to pair
If the patched _ frames is greater than the maximum target forecast frame number Pn, the target is judged to be in the sub-frame number Pn>
Removing the intermediate layer and recovering the ID; if the patched _ frames is less than the maximum target prediction frame number Pn and the tracked _ frames is greater than the minimum target tracking frame number Tn, then the target is slave ≧>
Is removed and added->
And (4) listing.
Step 80: to pair
Filtering each target by using a rectangular frame smooth filtering algorithm, outputting a filtered result, and finishing the tracking of the current ith frame of target; and will->
Adding the target in the buffer list for tracking the next frame; and returning to the step 10 to process the next frame image.
Further, the multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation is characterized in that the rectangular frame of the target comprises coordinates (x, y) of the upper left corner of a rectangular frame area of an object in an image, a width w and a height h.
Further, the multi-target real-time tracking method integrating the visual optical flow feature point tracking and the motion trend estimation is characterized in that a rectangular frame motion trend estimation algorithm comprises the following steps: for is to
The rectangular frame position of the medium target j->
Tracking and predicting by using a Kalman filter, predicting the rectangular frame position of the current frame target according to the historical rectangular frame position of the target, and calculating to obtain the motion state prediction position->
And adds to the list pick>
/>
The tracking algorithm of the optical flow characteristic points of the rectangular frame area image comprises the following steps: are respectively paired with M i And M i-1 Build a pyramid of gray level images at
Rectangular frame position +for medium target j>
Uniformly selecting M i-1 Calculating K coordinate points at M by using LK optical flow point matching algorithm based on gray level image pyramid i The positions of all the corresponding points are counted to be M i And M i-1 Averaged as the optical-flow tracking offset of target j>
Computing optical flow tracking result for target j>
Is->
And join the list
Furthermore, the invention provides a multi-target real-time tracking method integrating visual optical flow characteristic point tracking and motion trend estimation,characterized in that for a rectangular frame R a (x a ,y a ,w a ,h a ) And a rectangular frame R b (x b ,y b ,w b ,h b ) The rectangular frame overlap is: ioU = the intersection area of two rectangular boxes/union area of two rectangular boxes.
The normalized center point distance is:
wherein: />
Are respectively a rectangular frame R a And R b The coordinates of the center point of (a); w m Representing the image width, H m Representing the image height.
The difference degree is as follows: diff = abs (log (w) a /w b ))+abs(log(h a /h b ))，w a ,h a ,w b ,h b Are respectively a rectangular frame R a And R b Width and height of (a).
The calculation of the similarity SimM of the image areas comprises the following steps: for two rectangular frame area images M a And M b The RGB channels are separated, a color histogram is counted, and normalization processing is carried out to obtain
Wherein S is {a,b} Image M representing rectangular frame region a Or M b The area of the pixel (c) is,
representing an image M a Or M b The pixel value at the (i, j) position. For a color histogram vector>
And
calculating the similarity SimM = V of the image areas of the two rectangular frames by adopting a cosine similarity formula a ·V b /(|V a |+|V b L) wherein V a ·V b Representing two vector dot-product, | V a I and I V b And | represents the modulus of the two vectors, respectively.
Further, the multi-target real-time tracking method integrating the tracking of the visual optical flow feature points and the estimation of the motion trend, provided by the invention, is characterized in that the matching degree meeting the hyper-parameter threshold comprises the following steps:
Dist<thresh dist 、Diff<thresh diff 、SimM>thresh sim wherein->
Presetting a threshold value for the second degree of overlap, thresh dist For the distance from the center point a preset threshold, thresh diff Presetting a threshold for the degree of difference, thresh sim A threshold is preset for the similarity.
Further, the multi-target real-time tracking method integrating the tracking of the feature points of the visual optical flow and the estimation of the motion trend provided by the invention is characterized in that a rectangular frame smoothing filtering algorithm is as follows:
the rectangular frame positions of the target j from the i-2 th frame to the i-th frame are respectively marked as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j )、R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j )、R i,j (x i,j ,y i,j ,w i,j ,h i,j ) (ii) a The positions of the rectangular frames filtered from the ith-3 frame to the ith-1 frame are respectively marked as R' i-3,j (x’ i-3,j ,y’ i-3,j ,w’ i-3,j ,h’ i-3,j )、R’ i-2,j (x’ i-2,j ,y’ i-2,j ,w’ i-2,j ,h’ i-2,j )、R’ i-1,j (x’ i-1,j ,y’ i-1,j ,w’ i-1,j ,h’ i-1,j )。
The four parameters of the rectangular frame are filtered by a 2-order Butterworth low-pass filter respectively as follows:
wherein (a) 1 ,a 2 ,a 3 ) And (b) 1 ,b 2 ,b 3 ) The parameters of the Butterworth low-pass filter are calculated by setting a sampling frequency and a cut-off frequency. Obtaining a target rectangular frame filtering result: r' i,j (x’ i,j ,y’ i,j ,w’ i,j ,w’ i,j )。
Furthermore, the multi-target real-time tracking method integrating the tracking of the visual optical flow feature points and the estimation of the motion trend is characterized in that the image acquisition equipment is arranged on a vehicle, and a sensing area covers the front of the vehicle; the target includes: vehicles, pedestrians, non-motorized vehicles.
According to a second aspect of the present invention, there is provided a computer apparatus, comprising:
a memory to store instructions; and
and the processor is used for calling the instructions stored in the memory to execute the multi-target real-time tracking method for fusing the visual optical flow characteristic point tracking and the motion trend estimation in the first aspect.
According to a third aspect of the present invention, there is provided a computer-readable storage medium storing instructions which, when executed by a processor, perform the multi-target real-time tracking method of the first aspect, which combines visual optical flow feature point tracking and motion trend estimation.
Compared with the prior art, the technical scheme of the invention at least has the following beneficial effects:
the position of the target of the previous frame in the current frame is predicted by combining a Kalman state filter and an optical flow tracking algorithm; the problem that a single predictor cannot cover a complex scene is solved.
Aiming at the problem of calculating the similarity of the targets of the front frame and the rear frame, the method evaluates from two dimensions, namely respectively designing an evaluation algorithm according to the overlapping degree of rectangular frames, the distance of central points and the size of the rectangular frames; secondly, a cosine similarity evaluation algorithm based on a color histogram is provided on the image similarity of the rectangular frame region; therefore, the similarity of the target is calculated more comprehensively, and the target tracking precision is improved.
On the aspect of the target bipartite graph matching, compared with the Hungarian matching algorithm commonly used in the industry, the algorithm can only realize maximum matching, and the KM matching algorithm adopted by the invention further considers the matching weight on the basis of the Hungarian algorithm to realize optimal matching.
The target tracking algorithm realized by the invention provides a rectangular frame filtering algorithm based on a Butterworth low-pass filter on the basis of the rectangular frame jitter problem of the target, so that the target tracking result is smoother.
The invention finally achieves the operation efficiency of 30 frames per second in the embedded system, and the maximum number of tracking targets reaches 64; tracking performance is advantageous.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention, as claimed.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and together with the description, serve to explain the principles of the invention.
FIG. 1 is a schematic flow diagram shown in accordance with an exemplary embodiment.
FIG. 2 is a diagram illustrating rectangular box optimal binary matching, according to an exemplary embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. In addition, the technical features involved in the embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.
Description of algorithm label:
initializing a target label array, and recording the target label array as ListID; the list adopts a queue structure.
Recording the current frame as i; the current frame image is marked as M i (ii) a The current target detection result is recorded as Objects i (ii) a The current target tracking result is recorded as
The last frame image is marked as M i-1 The last frame of target tracking result is recorded as
The prediction result of the target motion state in the previous frame is recorded as ^ er>
The last frame of the target optical flow prediction is recorded as->
The result of the predicted position of the target in the previous frame in the current frame is recorded as->
Current target tracking result usage
And (4) showing.
Each tracked target contains the parameters: target index id, target tracking frame number tracked _ frames, target prediction frame number patched _ frames, target rectangular frame coordinates R (x, y, w, h).
The parameter Tn represents the minimum target tracking frame number, and the parameter Pn represents the maximum target prediction frame number.
As shown in fig. 1, in an embodiment, the multi-target real-time tracking method for integrating visual optical flow feature point tracking and motion trend estimation provided by the present invention comprises the following steps:
1) System construction: the camera is arranged on the front windshield of the vehicle, covers the forward sensing area, is connected with the controller through a video transmission line, and supplies power to the whole system;
2) Initializing a system: starting a system, loading a driver, carrying out hardware function self-checking, and alarming and quitting the system if the hardware fails; if the system self-check is normal, the next step is carried out;
3) The algorithm collects the camera image of the current frame i in real time and records as M i And detecting the target in the image by using a YOLO target detection algorithm, comprising: vehicles, pedestrians, non-motor vehicles, noted as Objects i ；
4) If the current frame is the first frame, the target detection result is assigned to the current tracking result
The step 10); if not, entering the next step;
5) Tracking results for the i-1 st frame
Obtaining the motion state prediction position of each target in the current frame by using a rectangular frame motion trend estimation algorithm (see (4) the rectangular frame motion trend estimation algorithm in an algorithm key module), and recording the motion state prediction position as the motion state prediction position
6) Tracking results for the i-1 st frame
Obtaining the optical flow predicted position of each target in the current frame by using the rectangular frame area image optical flow feature point tracking algorithm (see the 5 th partial rectangular frame area image optical flow feature point tracking algorithm in the key module of the algorithm in section 5), and recording the optical flow predicted position as being ^ er>
7) Comparison of
And &>
The position of the rectangular frame corresponding to the target in the list is calculated as overlap IoU, if
Then the target is added->
Listing, and adding 1 to the parameter number of patched _ frames of each target, otherwise, failing to track the target;
8) Detect the result Objects for the ith frame i Respectively tracking the predicted result with the (i-1) th frame
Each target in (1) calculates the matching degree of the rectangular frame (see (1) calculation of the matching degree of the rectangular frame in the algorithm key module) and the similarity SimM of the image area where the rectangular frame is located (see (2) calculation of the similarity of the image area of the rectangular frame in the algorithm key module), if the matching degree between the two rectangular frames satisfies: />
Dist<thresh dist 、Diff<thresh diff 、SimM>thresh sim Then a matrix is constructed based on IoU as the weights;
9) Using the weight matrix to execute a rectangular frame optimal binary matching algorithm (see (6) in an algorithm key module) to obtain a corresponding relation; for the successfully matched target, assigning a corresponding target id in the i-1 th frame to the target of the current i-th frame, wherein the target parameter tracked _ frames is added with 1, and the patched_frames is set with 0; for objects that do not match in the ith frame, label at objectNew id is distributed in array ListID, target parameter tracked _ frames is set as 1, and patched _framesis set as 0; for unmatched targets in the (i-1) th frame, the target labels are unchanged, and the patched _ frames is added with 1; adding the three targets
A list;
10 A pair of
Analyzing and judging the targets in the list, if the patched _ frames is greater than the maximum target prediction frame number Pn, removing the targets from the list, and recycling the id of the targets to a target label list ListID; if the patched _ frames is less than the maximum target prediction frame number Pn and the tracked _ frames is greater than the minimum target tracking frame number Tn, then the target is removed @>
And add it>
A list; will->
Caching the target in the database for subsequent target tracking calculation;
11 A pair of
Filtering each target in the list by using a rectangular frame smooth filtering algorithm (see (3) the rectangular frame smooth filtering algorithm in the algorithm key module), reducing the jitter of the rectangular frame, and adding a filtered result into a cache list for tracking the next frame; and simultaneously outputting a target tracking result to an intelligent driving decision module for decision making, and finishing the target tracking of the current frame.
12 ) returning to the step 3).
The key modules of the algorithm involved in the above steps include the following parts:
(1) Calculating the matching degree of the rectangular frame: rectangular frame R a (x a ,y a ,w a ,h a ) And a rectangular frame R b (x b ,y b ,w b ,h b ) The matching degree adopts three calculation modes:
overlap IoU = the intersection area of two rectangular boxes/union area of two rectangular boxes.
Normalized center point distance
Wherein:
are respectively a rectangular frame R a And R b The coordinates of the center point of (a); w is a group of m Representing the width of the image, H m Representing the image height.
Diff = abs (log (w) a /w b ))+abs(log(h a /h b ))。
(2) Calculating the similarity SimM of the rectangular frame area image:
for rectangular frame area image M a And M b The RGB channels are separated, the color histogram is counted, normalization processing is carried out, and the calculation formula is as follows:
wherein S {a,b} Image M representing rectangular frame region a Or M b The area of the pixel of (a),
representing an image M a Or M b The pixel value at the (i, j) position.
Correspondingly forming two color histogram vectors with the same dimension
And &>
Calculating the similarity of the two rectangular images by adopting a cosine similarity formula:
SimM＝V a ·V b /(|V a |+|V b |)
wherein V a ·V b Representing two vector dot-product, | V a I and I V b L represents the modulus of the two vectors, respectively.
(3) And (3) a rectangular frame smoothing filtering algorithm: in order to enable the target tracking rectangular frame to be more stable, the position of the tracked target rectangular frame is adjusted through a smoothing filtering algorithm. Marking the current frame as i, and performing smooth filtering on the target with the mark j, wherein the steps are as follows:
the rectangular frame positions of the target from the i-2 th frame to the i-th frame are respectively marked as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j )、R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j )、R i,j (x i,j ,y i,j ,w i,j ,h i,j ) (ii) a The positions of the rectangular frames filtered from the ith-3 frame to the ith-1 frame are respectively marked as R' i-3,j (x’ i-3,j ,y’ i-3,j ,w’ i-3,j ,h’ i-3,j )、R’ i-2,j (x’ i-2,j ,y’ i-2,j ,w’ i-2,j ,h’ i-2,j )、R’ i-1,j (x’ i-1,j ,y’ i-1,j ,w’ i-1,j ,h’ i-1,j )。
Filtering the four parameters of the rectangular frame by using a 2-order Butterworth low-pass filter respectively, wherein the calculation formula is as follows:
wherein (a) 1 ,a 2 ,a 3 ) And (b) 1 ,b 2 ,b 3 ) The parameters of the Butterworth low-pass filter are calculated by setting a sampling frequency and a cut-off frequency.
And returning and storing a target rectangular frame filtering result: r' i,j (x’ i,j ,y’ i,j ,w’ i,j ,w’ i,j )。
(4) Rectangular box motion trend estimation algorithm: in order to improve the accuracy of target tracking, the invention predicts the position of a rectangular frame of each target, adopts a 4-dimensional Kalman state filter comprising (x, y, w, h) of the rectangular frame, and predicts the position of the rectangular frame of the target of the current frame according to the historical positions of the rectangular frames of the target.
(5) The method comprises the following steps of (1) carrying out a rectangular frame area image optical flow characteristic point tracking algorithm: the invention realizes the optical flow tracking by constructing a gray pyramid for the front frame image and the rear frame image and calculating the position of the rectangular frame area image of the first frame in the current frame by an LK optical flow tracking algorithm, wherein the algorithm comprises the following steps:
and respectively constructing a gray pyramid for the front frame image and the rear frame image.
For the last frame object list
And uniformly selecting K points in the rectangular frame area of each target.
The positions of the K points in the current frame image are calculated by using an LK optical flow point matching algorithm.
And deleting the points which fail to be matched, and respectively calculating the horizontal and vertical offsets of the front frame and the rear frame for the rest points.
Counting the offset, and taking a median value as a target offset of the target optical flow tracking; and calculates the rectangular frame position of the target in the current frame.
Adding optical flow trace results
Tabulated, and returned.
(6) The optimal binary matching algorithm for the rectangular frame is as follows: for the target matching problem between the rectangular frame list A and the rectangular frame list B, the method is converted into a bipartite graph matching problem, and the optimal matching result is calculated by adopting a KM optimal matching algorithm.
The IoU overlap between the rectangular boxes of the two lists is first calculated.
And constructing a weight matrix by taking the overlapping degree as a weight.
And calculating the weight matrix through a KM algorithm to obtain the matching relation between the rectangular boxes of the list A and the list B under the condition of optimal matching.
And the successfully matched rectangular boxes are regarded as the same target, and the matching result is returned.
As shown in fig. 2, 3 targets exist in the list a, 4 targets exist in the list B, the overlapping degree IoU is calculated pairwise for each target rectangular frame in the list, the value range is (0-1), a weight matrix is obtained, and then a matching result is obtained by using a KM optimal binary matching algorithm. Through operation, A1, A2, A3 in fig. 2 are successfully matched with B1, B2, B4 in the list B, respectively.
Specifically, in some embodiments, the invention is implemented as the following steps:
1) System construction: the camera is arranged on the front windshield of the vehicle, covers the forward sensing area, is connected with the controller through the video transmission line and supplies power to the whole system.
2) Initializing a system: starting a system, loading a driver, carrying out hardware function self-checking, and alarming and quitting the system if the hardware fails; and if the system self-checking is normal, the next step is carried out.
3) Acquiring a camera image of a current frame i in real time, and recording the image as M i And detecting targets in the image by using a YOLO target detection algorithm, wherein each target position comprises the pixel coordinates x and y of the upper left corner of a rectangular frame area of the object in the image and the width and height w and h, and the target types comprise: the recognition results of vehicles, pedestrians and non-motor vehicles are recorded as Objects i 。
4) If the current frame i is the first frame, initializing a target label queue ListID, wherein the length of the queue is 128, the range of the queue data is 0-127, and the input and output of the data in the queue adopt a first-in first-out principle as the detection result Objects i Assigns an ID, sets the target parameter tracked _ frames to 1, the patched _framesto 0, puts the target into a list
To step 12); if the current frame i is not the first frame, the next step is proceeded to。
5) Fetching the target tracking result of the (i-1) th frame in the buffer memory
Rectangular frame position for each tracking target j
Performing tracking prediction by using a Kalman filter to obtain a predicted position ^ of each target on the motion state of the current frame>
And adds to the list pick>
This module is named rectangular box motion trend estimation algorithm.
6) Fetching the i-1 frame image M in the buffer i-1 And respectively to the images M i And M i-1 Constructing a gray level image pyramid, and then tracking the result of the (i-1) th frame
The rectangular frame position of each tracking target j in>
K coordinate points are respectively taken in the row and column directions, and K is calculated by using an LK optical flow point matching algorithm based on a gray image pyramid 2 The positions of the points in the current frame image are recorded as the median of the position offset and the distance offset between the corresponding points
Calculating to obtain the optical flow tracking result of each tracking target in the current frame by adopting the following formula
And adds to the list pick>
The module is named as a rectangular frame area image optical flow feature point tracking algorithm:
7) Comparison of
And &>
The rectangular box position of the corresponding object in the list is calculated to have an overlap of IoU, if IoU>thresh I 1 oU Then add the target to &>
Tabulated and the patched _ frames parameter value for each target is added to 1, otherwise the target tracking fails and the tracked _ frames parameter is set to 0.
8) Detecting the result Objects of the current frame i i Each target in the system is respectively tracked with the (i-1) th frame to predict the result
The matching degree of the rectangular frame of each target is calculated, wherein the matching degree comprises an overlapping degree IoU, a normalized central point distance Dist and a difference degree Diff, and the calculation formula is as follows:
iou = intersection area of two rectangular boxes/union area of two rectangular boxes.
b.
Wherein: />
Are respectively a rectangular frame R a And R b The coordinates of the center point of (a); w m Representing the width of the image, H m Representing the image height.
c.Diff＝abs(log(w a /w b ))+abs(log(h a /h b ))。
9) Recalculating Objects i Each of which is respectively connected with
The similarity SimM of the image area of the medium target is calculated in two steps:
a. for two rectangular frame area images M a And M b The RGB channels of (1) are separated, a color histogram is counted, normalization processing is carried out, and a calculation formula is as follows:
wherein S {a,b} Image M representing rectangular frame region a Or M b The area of the pixel (c) is,
representing an image M a Or M b The pixel value at the (i, j) position.
b. Correspondingly forming two color histogram vectors with the same dimension
And &>
Calculating the similarity of the two rectangular images by adopting a cosine similarity formula:
SimM＝V a ·V b /(|V a |+|V b |)
wherein V a ·V b Representing two vector dot-product, | V a I and I V b And | represents the modulus of the two vectors, respectively.
10 If Objects) i And
the degree of match between the targets in (1) satisfies a hyper-parameter threshold:
Dist<thresh dist 、Diff<thresh diff 、SimM>thresh sim taking IoU as a weight construction matrix; if the hyperparametric threshold is not met, the weights in the matrix are set to 0.
11 For the weight matrix, calculate Objects using KM optimal binary matching algorithm i And
the corresponding relation between each target; for a successfully matched target, the i-1 th frame is @>
Assigning corresponding object id to Objects in current ith frame i Target parameter tracked _ frames plus 1, patched _framesset 0; for Objects i In the unmatched targets, new id is distributed in the target index array ListID, the target parameter tracked _ frames is set to be 1, and the patched_frames is set to be 0; for->
The index of the target is not changed, and the patched _ frames is added with 1; these three targets are added to->
And (4) listing.
12 ) emptying
List, pair->
Analyzing and judging the targets in the list, if the patched _ frames is greater than the maximum target prediction frame number Pn, removing the targets from the list, and recycling id of the targets to a target label queue ListID; if the patched _ frames is less than the maximum target predicted frame number Pn and the tracked _ frames is greater than the minimum target tracking frame number Tn, then it will beThe target removal +>
And add it>
And &>
A list; if the patched _ frames is less than the maximum target prediction frame number Pn and the tracked _ frames is less than the minimum target tracking frame number Tn, then the target is added ≧>
A list for subsequent target tracking calculations.
13 A pair of
Each target in the list is filtered by using a rectangular frame smoothing filter algorithm, so that the jitter of a rectangular frame is reduced, and the calculation steps are as follows:
a. the original positions of rectangular frames from the i-2 th frame to the i-th frame of the target with the mark j are respectively marked as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j )、R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j )、R i,j (x i,j ,y i,j ,w i,j ,h i,j ) (ii) a The positions of the rectangular frames filtered from the ith-3 frame to the ith-1 frame are respectively marked as R' i-3,j (x’ i-3,j ,y’ i-3,j ,w’ i-3,j ,h’ i-3,j )、R’ i-2,j (x’ i-2,j ,y’ i-2,j ,w’ i-2,j ,h’ i-2,j )、R’ i-1,j (x’ i-1,j ,y’ i-1,j ,w’ i-1,j ,h’ i-1,j )。
b. Filtering the four parameters of the rectangular frame by using a 2-order Butterworth low-pass filter respectively, wherein the calculation formula is as follows:
wherein (a) 1 ,a 2 ,a 3 ) And (b) 1 ,b 2 ,b 3 ) The parameters of the Butterworth low-pass filter are calculated by setting a sampling frequency and a cut-off frequency.
c. And returning and storing a target rectangular frame filtering result: r' i,j (x’ i,j ,y’ i,j ,w’ i,j ,w’ i,j ) And update
The target rectangular frame position of (1).
14 Will be
Adding a target tracking list into a cache for tracking the next frame; and outputting the target to an intelligent driving decision module for decision making, and finishing the target tracking of the current frame.
15 Returning to the step 3).
It will be understood that the invention is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the invention is limited only by the appended claims.
Claims (9)
1. A multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation is characterized by comprising the following steps:
step 10: obtaining the image M of the current ith frame in real time through image acquisition equipment i Detection of M Using the YOLO target detection Algorithm i Obtaining the rectangular frame position of the target, adding each target of the ith frame into the target detection result list Objects of the ith frame i ；
Step 20: if the ith frame is the first frame, for Objects i Configuring a new ID, setting a first parameter of the target, namely, tracked _ frames to 1, and setting a second parameter of the target, namely, patched _ frames to 10, will Objects i Target tracking result list assigned to current ith frame
Skipping to step 70; if the ith frame is not the first frame, proceed to step 30;
and step 30: target tracking result list for frame i-1
Is processed by using a rectangular frame motion trend estimation algorithm to obtain a list of predicted positions of the target in the motion state of the current frame->
Processing by using a rectangular frame area image optical flow characteristic point tracking algorithm to obtain a target optical flow prediction position list->
Step 40: comparison of
And &>
Calculating the position of the rectangular frame of the target j with the same ID, and calculating the overlapping degree of the first rectangular frame IoU 1 Based on a first degree of overlap preset threshold values>
Is judged if->
If the target tracking is successful, adding the target into a position tracking prediction result list of the target of the i-1 th frame in the i frame->
And adding 1 to the patched _ frames of the target; if->
When the target tracking fails, setting the tracked _ frames of the target to be 0;
step 50: for Objects i Each of the objects in (1) is respectively connected with
Is calculated, including a second degree of overlap match IoU 2 Normalizing the distance of the central point Dist and the difference Diff, and determining the similarity SimM of the image area, if the matching degree meets the hyper-parameter threshold, then IoU is used 2 As the weights of the two target matching degrees, if the matching degree does not meet the hyper-parameter threshold, setting the weights corresponding to the two target matching degrees as 0, and constructing a matching degree weight matrix based on the weights;
step 60: according to the matching degree weight matrix, obtaining a matching relation between targets of an i-1 th frame and an i-th frame by using a KM optimal matching algorithm, assigning the ID of the target in the i-1 th frame to the matching target of the i-th frame for the target which is successfully matched, adding 1 to a tracked _ frames, and setting a patched \\/u frame as 0; for targets that do not match in the ith frame, a new ID is assigned, the tracked _ frames is set to 1, and the patched_frames is set to 0; for the unmatched target in the (i-1) th frame, the ID is unchanged, and the patched _ frames is added with 1; each object is added
A list;
step 70: to pair
If the patched _ frames is greater than the maximum target forecast frame number Pn, the target is judged to be in the sub-frame number Pn>
Removing and recovering the ID; if the patched _ frames is less than the maximum target prediction frame number Pn and the tracked _ frames is greater than the minimum target tracking frame number Tn, then the target is slave @>
Is removed and added
A list;
step 80: to pair
Filtering each target by using a rectangular frame smooth filtering algorithm, outputting a filtered result, and finishing the tracking of the current ith frame of target; and will->
Adding the target in the cache list for tracking the next frame; and returning to the step 10 to process the next frame image. />
2. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in claim 1, wherein the rectangular frame of the target comprises coordinates (x, y) of the upper left corner of the rectangular frame region of the object in the image, and width w and height h.
3. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in claim 2, wherein the rectangular frame motion trend estimation algorithm comprises: to pair
Rectangular frame position of middle target j
Tracking prediction using Kalman filter based on target historical momentsThe shape frame position is used for predicting the rectangular frame position of the current frame target, and the motion state prediction position of the target j is calculated and obtained>
And adds to the list pick>
The rectangular frame area image optical flow characteristic point tracking algorithm comprises the following steps: are respectively paired with M i And M i-1 Construct a pyramid of gray scale images at
The rectangular frame position of the medium target j->
Uniformly selecting M i-1 Calculating K coordinate points at M by using LK optical flow point matching algorithm based on gray level image pyramid i The positions of all the corresponding points are counted to be M i And M i-1 The average is taken as the optical flow tracking offset ≥ of target j>
Computing optical flow tracking result->
Is->
And join the list
4. The method of claim 3, wherein the method comprises multi-target real-time tracking with fusion of tracking of feature points of visual optical flow and estimation of motion trendCharacterized by for a rectangular frame R a (x a ,y a ,w a ,h a ) And a rectangular frame R b (x b ,y b ,w b ,h b ) The rectangular frame overlap is: ioU = the intersection area of the two rectangular boxes/the union area of the two rectangular boxes;
the normalized center point distance is:
wherein: />
Are respectively a rectangular frame R a And R b The coordinates of the center point of (a); w m Representing the width of the image, H m Representing the image height;
the degree of difference is: diff = abs (log (w) a /w b ))+abs(log(h a /h b ))，w a ,h a ,w b ,h b Are respectively a rectangular frame R a And R b Width and height of (d);
the calculation of the similarity SimM of the image areas comprises the following steps:
for two rectangular frame area images M a And M b The RGB channels are separated, the color histogram is counted, and normalization processing is carried out to obtain
g is from {0 to 255}, wherein S is {a,b} Image M representing rectangular frame region a Or M b Pixel area of (d), based on the pixel area of (d)>
Representing an image M a Or M b The pixel value at the (i, j) position;
for color histogram vector
And &>
Calculating the similarity SimM = V of the image areas of the two rectangular frames by adopting a cosine similarity formula a ·V b /(|V a |+|V b L) wherein V a ·V b Representing two vector dot-product, | V a I and I V b And | represents the modulus of the two vectors, respectively.
5. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in claim 4, wherein the matching degree satisfying the hyper-parameter threshold comprises:
Dist<thresh dist 、Diff<thresh diff 、SimM>thresh sim in which>
Presetting a threshold value for the second degree of overlap, thresh dist For the distance from the center point a preset threshold, thresh diff Presetting a threshold for the degree of difference, thresh sim A threshold is preset for the similarity.
6. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in claim 5, wherein the rectangular frame smoothing filtering algorithm is:
the rectangular frame positions of the target j from the i-2 th frame to the i-th frame are respectively marked as R i-2,j (x i-2,j ,y i-2,j ,w i-2,j ,h i-2,j )、R i-1,j (x i-1,j ,y i-1,j ,w i-1,j ,h i-1,j )、R i,j (x i,j ,y i,j ,w i,j ,h i,j ) (ii) a The positions of the rectangular frames filtered from the ith-3 frame to the ith-1 frame are respectively marked as R' i-3,j (x’ i-3,j ,y’ i-3,j ,w’ i-3,j ,h’ i-3,j )、R’ i-2,j (x’ i-2,j ,y’ i-2,j ,w’ i-2,j ,h’ i-2,j )、R’ i-1,j (x’ i-1,j ,y’ i-1,j ,w’ i-1,j ,h’ i-1,j )；
The four parameters of the rectangular frame are respectively filtered by a 2-order Butterworth low-pass filter as follows:
wherein (a) 1 ,a 2 ,a 3 ) And (b) 1 ,b 2 ,b 3 ) The parameters of the Butterworth low-pass filter are obtained by setting sampling frequency and cut-off frequency;
obtaining a target rectangular frame filtering result: r' i,j (x’ i,j ,y’ i,j ,w’ i,j ,w’ i,j )。
7. The multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in claims 1-6, wherein the image acquisition device is installed on a vehicle, and a sensing area covers the front of the vehicle; the target includes: vehicles, pedestrians, non-motorized vehicles.
8. A computer device, comprising:
a memory to store instructions; and a processor for invoking the instructions stored in the memory to execute the multi-target real-time tracking method integrating visual optical flow feature point tracking and motion trend estimation as claimed in any one of claims 1-7.
9. A computer-readable storage medium storing instructions which, when executed by a processor, perform the multi-target real-time tracking method with fusion of visual optical flow feature point tracking and motion trend estimation according to any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310119716.2A CN115953437A (en) | 2023-02-16 | 2023-02-16 | Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310119716.2A CN115953437A (en) | 2023-02-16 | 2023-02-16 | Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115953437A true CN115953437A (en) | 2023-04-11 |
Family
ID=87296883
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310119716.2A Pending CN115953437A (en) | 2023-02-16 | 2023-02-16 | Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115953437A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116385497A (en) * | 2023-05-29 | 2023-07-04 | 成都与睿创新科技有限公司 | Custom target tracking method and system for body cavity |
| CN117761678A (en) * | 2024-02-22 | 2024-03-26 | 成都鹰谷米特科技有限公司 | Complex environment target detection method and chip based on V frequency band |
-
2023
- 2023-02-16 CN CN202310119716.2A patent/CN115953437A/en active Pending
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116385497A (en) * | 2023-05-29 | 2023-07-04 | 成都与睿创新科技有限公司 | Custom target tracking method and system for body cavity |
| CN116385497B (en) * | 2023-05-29 | 2023-08-22 | 成都与睿创新科技有限公司 | Custom target tracking method and system for body cavity |
| CN117761678A (en) * | 2024-02-22 | 2024-03-26 | 成都鹰谷米特科技有限公司 | Complex environment target detection method and chip based on V frequency band |
| CN117761678B (en) * | 2024-02-22 | 2024-04-26 | 成都鹰谷米特科技有限公司 | Complex environment target detection method and chip based on V frequency band |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Tang et al. | Single-camera and inter-camera vehicle tracking and 3D speed estimation based on fusion of visual and semantic features | |
| CN115953437A (en) | Multi-target real-time tracking method integrating visual light stream feature point tracking and motion trend estimation | |
| CN102609953B (en) | Many objects outward appearance of video camera and range-sensor data strengthens and merges | |
| EP1329850B1 (en) | Apparatus, program and method for detecting both stationary objects and moving objects in an image | |
| CN111260683A (en) | Target detection and tracking method and device for three-dimensional point cloud data | |
| JP2020052694A (en) | Object detection apparatus, object detection method, and computer program for object detection | |
| CN111950394B (en) | Method and device for predicting lane change of vehicle and computer storage medium | |
| Huang et al. | Efficient image stitching of continuous image sequence with image and seam selections | |
| JP7209115B2 (en) | Detection, 3D reconstruction and tracking of multiple rigid objects moving in relatively close proximity | |
| CN115063454B (en) | Multi-target tracking matching method, device, terminal and storage medium | |
| Chen et al. | Moving vehicle detection based on optical flow estimation of edge | |
| CN113723190A (en) | Multi-target tracking method for synchronous moving target | |
| CN113409361B (en) | Multi-target tracking method and device, computer and storage medium | |
| WO2020154990A1 (en) | Target object motion state detection method and device, and storage medium | |
| US20210174113A1 (en) | Method for limiting object detection area in a mobile system equipped with a rotation sensor or a position sensor with an image sensor, and apparatus for performing the same | |
| CN112069969A (en) | Method and system for tracking highway monitoring video mirror-crossing vehicle | |
| CN115546705B (en) | Target identification method, terminal device and storage medium | |
| Funde et al. | Object detection and tracking approaches for video surveillance over camera network | |
| Urdiales et al. | An improved deep learning architecture for multi-object tracking systems | |
| CN114763136A (en) | Guide vehicle driving auxiliary system based on deep learning | |
| CN110992424A (en) | Positioning method and system based on binocular vision | |
| CN112580565A (en) | Lane line detection method, lane line detection device, computer device, and storage medium | |
| JP7003972B2 (en) | Distance estimation device, distance estimation method and computer program for distance estimation | |
| CN116931583A (en) | Method, device, equipment and storage medium for determining and avoiding moving object | |
| CN116681730A (en) | Target tracking method, device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |