CN116580066B - Pedestrian target tracking method under low frame rate scene and readable storage medium - Google Patents
Pedestrian target tracking method under low frame rate scene and readable storage medium Download PDFInfo
- Publication number
- CN116580066B CN116580066B CN202310809902.9A CN202310809902A CN116580066B CN 116580066 B CN116580066 B CN 116580066B CN 202310809902 A CN202310809902 A CN 202310809902A CN 116580066 B CN116580066 B CN 116580066B
- Authority
- CN
- China
- Prior art keywords
- pedestrian
- target
- tracking
- matrix
- frame
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000003860 storage Methods 0.000 title claims abstract description 7
- 239000011159 matrix material Substances 0.000 claims abstract description 100
- 238000001514 detection method Methods 0.000 claims abstract description 46
- 230000008859 change Effects 0.000 claims abstract description 12
- 238000001914 filtration Methods 0.000 claims abstract description 12
- 238000009826 distribution Methods 0.000 claims description 48
- 230000008569 process Effects 0.000 claims description 17
- 238000012549 training Methods 0.000 claims description 9
- 230000007704 transition Effects 0.000 claims description 7
- 238000013527 convolutional neural network Methods 0.000 claims description 6
- 238000005259 measurement Methods 0.000 claims description 6
- 238000005457 optimization Methods 0.000 claims description 5
- 238000002372 labelling Methods 0.000 claims description 4
- 238000012545 processing Methods 0.000 abstract description 2
- 238000005286 illumination Methods 0.000 description 4
- 238000012806 monitoring device Methods 0.000 description 4
- 239000002245 particle Substances 0.000 description 4
- 230000000007 visual effect Effects 0.000 description 4
- 230000007547 defect Effects 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000012706 support-vector machine Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000000903 blocking effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/277—Analysis of motion involving stochastic approaches, e.g. using Kalman filters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/761—Proximity, similarity or dissimilarity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a pedestrian target tracking method under a low frame rate scene and a readable storage medium, wherein the method comprises the following steps: initializing Kalman filtering parameters, and determining a covariance matrix of observation noise of a Kalman filter for updating a pedestrian tracking target in each iteration according to the confidence coefficient of a pedestrian target frame obtained by a pedestrian target detection model and the change rate of the aspect ratio of the pedestrian target frame; calculating a globally optimal matching pair of a pedestrian tracking target and a pedestrian target frame by using a Hungary algorithm; and performing corresponding processing according to the matching result. The invention optimizes the low frame rate scene, solves the problem of easy target follow-up under the low frame rate scene, is applicable to security and protection scenes and low frame rate scenes, reduces the operation amount and parameter amount of a tracking algorithm by only utilizing the position and motion information of a target frame, and decouples the detection and tracking algorithm, so that a proper detection model can be used for solving the problem of complex scenes.
Description
Technical Field
The present invention relates to the field of target tracking, and in particular, to a pedestrian target tracking method and readable storage medium in a low frame rate scene.
Background
Target tracking algorithms are currently one of the focus of research in computer vision as a typical problem in machine vision applications. With the development of artificial intelligence, intelligent monitoring devices are widely used. At present, a target tracking technology used on monitoring equipment mainly performs image capture on a target through a monitoring camera, and then detects and analyzes image features and motion models between frames in edge equipment or a central server by using a computer vision technology, so that a tracking result is obtained.
The intelligent monitoring device usually needs to process multiple paths of video streams, and under the condition of limited computational power (such as an edge device or a scene with low computing resources of a central server), the tracking algorithm used by the intelligent monitoring device is required to complete tracking under the condition of low frame rate, so that the intelligent monitoring device is an optional method. While the low frame rate scheme is adopted, there are a plurality of problems to be solved when target tracking is performed.
For example, the invention application number is CN201810452935.1, the invention name is a low frame rate video target detection and tracking method and application thereof in unmanned aerial vehicle, firstly, sampling the periphery of a target by an edge block algorithm (edgeboxes), and the sampled target is a potential target, and the potential target is discarded beyond a sampling frame; then, based on the local data, extracting local visual characteristics of the image, and simultaneously learning the local data to improve the stability of target change; and then judging the potential target by using a support vector machine, if the potential target is higher than a threshold value, considering the potential target as a new target position, and if the value of a classification result obtained by the support vector machine is lower, sampling the whole picture by using an edge block algorithm, and searching the target in the whole picture. And finally, updating the support vector machine model according to the predicted position. The technical scheme has the following defects: (1) Determining possible positions of the target frame by adopting a discriminant model, extracting features from a large number of possible positions around the target frame, and using the discriminant model, so that the speed is reduced; (2) The characteristic mode used by the judging model and the selection of the kernel function may have poor generalization capability for different visual target effects; (3) Each iteration needs to update parameters of the discriminant model, and the additional learning process is difficult to complete in real time at the edge end with weak calculation power.
For another example, the technical scheme of the invention with the application number of CN201210089248.0 and the invention name of "target tracking method for low frame rate video" includes the following main steps: (1) The target area is represented by a method of fusing main colors and spatial distribution characteristics of the main colors; (2) Matching the similarity between the candidate region and the target region by adopting a matching criterion based on the cross color proportion; (3) Characterizing the matching degree of the sample particles and the target template by adopting an adaptive value function based on a parameter integral graph; (4) An annealed particle swarm optimization framework that mimics the intelligence of a biological swarm is utilized to reduce mutations caused by searching for low frame rate video. The technical scheme has the following defects: (1) The main color mode obtained by clustering is used as a characteristic, and is not suitable for large scene with changed illumination conditions; (2) The use of the particle swarm algorithm avoids that a large number of particle iterative optimization processes cannot find a globally optimal solution, resulting in slower running speeds.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art and provides a pedestrian target tracking method in a low-frame-rate scene.
The aim of the invention is achieved by the following technical scheme:
a pedestrian target tracking method in a low frame rate scene comprises the following steps:
s1, initializing Kalman filtering parameters of a Kalman filter;
s2, acquiring a video image, inputting a picture of the acquired video image into a pedestrian target detection model,
determining a covariance matrix of observation noise of a Kalman filter in each iteration according to the confidence coefficient of a pedestrian target frame obtained by the pedestrian target detection model and the change rate of the aspect ratio of the pedestrian target frame;
s3, calculating a cost matrix according to the measurement between the pedestrian tracking target and the pedestrian target frame, wherein the cost matrix is used as the input of the Hungary algorithm; calculating a globally optimal matching pair of a pedestrian tracking target and a pedestrian target frame by using a Hungary algorithm;
s4, updating posterior distribution of the pedestrian tracking target for the matched pedestrian tracking target and the pedestrian target frame, and outputting a result to a Kalman filter;
if the maximum value of probability density of prior distribution of the unmatched pedestrian tracking targets is smaller than a preset threshold value, deleting the pedestrian tracking targets, and ending the life cycle; otherwise, the pedestrian tracking target is reserved, and the reserved pedestrian tracking target enters the next iteration period;
for the unmatched pedestrian target frame, a new pedestrian tracking target is initialized, and the initial state quantity of the new pedestrian tracking target is determined by the coordinates of the pedestrian target frame.
In step S1, the kalman filter parameter includes an observation matrix of a kalman filterCovariance matrix of process noise->；
The abscissa, the ordinate, the width and the height of the pedestrian target frame are obtained through a detection algorithm, and are assigned to the observation matrix of the Kalman filter；
Covariance matrix of process noise of Kalman filterInitializing to an empirical constant;
wherein Is->Time of day.
In step S2, the covariance matrix of the observed noise of the Kalman filterOptimization is performed by:
definition of karlUncertainty of observed noise of Mannheim filter：
；
wherein ,、/>is the weight of two coefficients, +.>Confidence level of pedestrian target frame obtained for pedestrian target detection model, < >>The rate of change of aspect ratio for the pedestrian target frame; />，/>The width and the height of the pedestrian target frame are respectively;
covariance matrix of observed noise of Kalman filterThe definition is as follows:
；
wherein Is->Time of day.
The pedestrian target detection model is based on a convolutional neural network model, is a yolov5s pedestrian target detection model, and defines input and output of the pedestrian target detection model: defining the number and the format of input pictures, and defining output as a target envelope frame and a class; and manually labeling the acquired data by using a defined output scheme to obtain training data, and finally training the target detection model and updating the model weight.
In step S3, the measurement between the pedestrian tracking target and the pedestrian target frame is defined as：
；
；
wherein ,the probability density of the location of the pedestrian target box in the prior distribution of pedestrian tracking target locations,is Gaussian distribution->Covariance matrix of prior distribution of target (i.e. covariance matrix of prior distribution of predicted state) for pedestrian tracking,>for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>Is the transpose operation of the matrix.
A priori distribution of the pedestrian tracking targetsCovariance matrixThe method comprises the following steps of obtaining by a state transition matrix and process noise:
；
wherein ,covariance matrix for the distribution of state variables, +.>Covariance matrix of process noise for Kalman filter, +.>Is->Time of day.
In step S3, the cost matrixIs defined as follows:
；
wherein ,tracking target for nth pedestrian>，/>Probability density of the position of the target frame for the pedestrian in a priori distribution of tracking target positions for the pedestrian,/->、/>The positions of the mth pedestrian target frame are respectively the abscissa and the ordinate, and the +.>Is the transpose operation of the matrix.
In step S3, the method uses a hungarian algorithm to calculate a globally optimal matching pair of the pedestrian tracking target and the pedestrian target frame, specifically:
performing row permutation operation on the cost matrix by using a Hungary algorithm to minimize the trace of the matrix:
；
wherein ,for the permutation matrix +.>A cost matrix;
after being matched by the Hungary algorithm, the global optimal matching pair is obtained, and the definition of the matching pair is as follows:
；
wherein ,for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>Confidence level of pedestrian target frame obtained for pedestrian target detection model, < >>Id information of the target is tracked for pedestrians.
In step S4, for the matched pedestrian tracking target and pedestrian target frame, the following operations are performed:
calculating residual error of pedestrian observance：
；
wherein ,observation matrix for Kalman filter, < >>For observing the model matrix>Tracking a priori state quantity of a target for a pedestrian;
calculating residual error of pedestrian observanceCovariance matrix>：
；
wherein ,for transpose operation of matrix,/->Covariance matrix of a priori distribution of tracking targets for pedestrians,>covariance matrix for distribution of state quantity; />The covariance matrix of the observed noise of the Kalman filter is used for updating the prior distribution of the pedestrian tracking target;
computing optimal gain for Kalman filtering：
；
Updating posterior distribution of pedestrian state quantity:
；
wherein ,covariance matrix of posterior distribution of state quantity;
updating posterior state quantity of pedestrian tracking targets:
；
wherein ,、/>and respectively using the current state quantity and the distribution of the pedestrian tracking target in the next iteration period.
The prior state quantity of the pedestrian tracking targetThe method comprises the following steps of:
；
；
wherein ,for state transition matrix>For the state quantity->For the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>、/>、/>、/>Respectively->、/>、/>、Derivative with respect to time.
Meanwhile, the invention provides:
a server comprising a processor and a memory, the memory having stored therein at least one program loaded and executed by the processor to implement the pedestrian target tracking method in a low frame rate scenario described above.
A computer-readable storage medium having stored therein at least one program loaded and executed by a processor to implement the pedestrian target tracking method in a low frame rate scenario described above.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the invention optimizes the covariance matrix, so that the pedestrian observation noise is more in line with the definition of Kalman filtering. The covariance matrix parameters are determined by the confidence of the pedestrian target box and the rate of change of the aspect ratio. When the pedestrian target is blocked or blurred, the confidence of the pedestrian target frame is reduced, and the variance is increased. When the aspect ratio change rate of the pedestrian frame is larger, the blocking condition of the tracked pedestrian target is likely to occur, and the variance is larger. Compared with the prior tracking algorithm scheme which mostly uses a covariance matrix with a fixed constant, the characteristic of uncertainty of observed quantity cannot be well expressed, and the uncertainty is better described by using two proxy variables for representation aiming at two scenes, namely uncertainty of a pedestrian detection model and occlusion of a tracking target.
2. The invention optimizes the definition of metrics between pedestrian tracking targets and pedestrian target frames. Compared with the use of iou and mahalanobis distances, the measurement is defined as the probability density of the position of the pedestrian target frame in the prior distribution of the pedestrian tracking target, and finally the optimal matching is carried out through a Hungary algorithm. Compared with the iou distance, the phenomenon that the iou distance can occur with the iou=0 under the condition of low frame rate is avoided. The problem that the optimal matching cannot be performed by the iou=0 is avoided, namely the pedestrian is lost. In comparison with the mahalanobis distance, in the case of tracking, the smaller and smaller characteristic of the mahalanobis distance enables the tracking to be matched with the tracking pedestrian target preferentially, so that the tracking is unstable. The probability density is directly used as a measure to solve the two problems.
3. The invention can realize the life cycle self-adaption of the pedestrian tracking target. By designating a probability threshold, the pedestrian tracking target below the threshold is deleted, and if the tracking loss occurs, the prior covariance of the pedestrian target with the tracking loss can be slowly increased for a relatively stable or stationary pedestrian target, such as a person sitting or standing still, so that the pedestrian target can be kept for a longer time. And for the pedestrian tracking target with large variation, the probability density is rapidly increased, so that the maximum value of the probability density is rapidly reduced, and if the probability density is lower than the threshold value, the probability density is deleted by the tracker. In contrast to previous tracking algorithms, a fixed lifecycle of k frames is typically given, without considering that different targets should be given different lengths of life. For example, a very stable or stationary target should remain for a long time after it is lost. On the contrary, a tracking target which moves rapidly cannot estimate its position in a short time after losing, so that it can be kept for a short time.
Drawings
Fig. 1 is a flowchart of a pedestrian target tracking method in a low frame rate scene.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but embodiments of the present invention are not limited thereto.
The following preparation work is done before the implementation process in this embodiment:
1. and acquiring a video sequence of the pedestrian by using the installed camera, and storing image data to the edge equipment or the central server. The collected data are mainly used for training and testing a pedestrian target detection algorithm model. In order to ensure the performance of the model, data under different conditions need to be acquired, and the data cover different time periods, different illumination changes and different weather conditions.
2. The target detection algorithm is mainly based on a convolutional neural network model, and the specific model is a yolov5s pedestrian detection model. The algorithm based on the convolutional neural network model can be more suitable for the conditions of illumination change, shielding and the like on the detection effect. The convolutional neural network model needs to define input and output in advance, wherein the input is the number and format of pictures in the input of the defined model, and the output is the target envelope frame and class. And manually labeling the acquired data by using a defined output scheme to obtain training data, and finally training the target detection model and updating the model weight. In the invention, the model is a yolov5 target detection model, a picture with the size of 640 x 384 is input, the speed and the precision are both considered, and the detection model is quantized into an int8 quantization model of a corresponding hardware platform. The model outputs 1 category corresponding to the detection target of the pedestrian category.
After the preparation work is completed, the following specific implementation is as follows:
referring to fig. 1, a pedestrian target tracking method in a low frame rate scene includes the following steps:
s1, initializing Kalman filtering parameters of a Kalman filter;
the Kalman filtering parameter comprises an observation matrix of a Kalman filterCovariance matrix of process noise；
The abscissa, the ordinate, the width and the height of the pedestrian target frame are obtained through a detection algorithm, and are assigned to the observation matrix of the Kalman filter；
Covariance matrix of process noise of Kalman filterInitializing to an empirical constant;
wherein Is->Time of day.
S2, acquiring a video image, inputting a picture of the acquired video image into a pedestrian target detection model,
determining a covariance matrix of observation noise of a Kalman filter in each iteration according to the confidence coefficient of a pedestrian target frame obtained by the pedestrian target detection model and the change rate of the aspect ratio of the pedestrian target frame;
covariance matrix of observed noise of the Kalman filterOptimization is performed by:
definition of uncertainty of observed noise of Kalman filter：
；
wherein ,、/>is the weight of two coefficients, +.>Confidence level of pedestrian target frame obtained for pedestrian target detection model, < >>The rate of change of aspect ratio for the pedestrian target frame; />，/>The width and the height of the pedestrian target frame are respectively;
covariance matrix of observed noise of Kalman filterThe definition is as follows:
；
wherein Is->Time of day.
In the present embodiment, covariance matrix of observed noise of Kalman filter is optimizedSuch that the observed noise complies with the definition of kalman filtering (i.e. the observed noise represents the uncertainty of the observed quantity), this uncertainty is represented by two proxy variables, namely: confidence degree of pedestrian target frame obtained by pedestrian target detection model +.>And the aspect ratio of the pedestrian target frame +.>。
When the pedestrian target is blocked or the pedestrian detection model is unstable, the uncertainty of the position of the pedestrian target frame is increased, and meanwhile, the confidence of the pedestrian target frame is also reduced, so that the uncertainty of the observation noise can be represented by the confidence of the pedestrian target frame.
When a pedestrian target is shielded, there is often a gradual shielding process, i.e. the shielding area is larger and larger, and the uncertainty of the observed noise can be represented by the change rate of the aspect ratio of the pedestrian target frame.
The present embodiment combines these two variables and can be used to describe the uncertainty of this observation noise. Compared with the traditional tracking algorithm, such as a sort algorithm, the observed noise of the Kalman filtering is usually given a constant, the uncertainty of the observed quantity cannot be measured, and the Kalman filtering plays no role.
The pedestrian target detection model is based on a convolutional neural network model, is a yolov5s pedestrian target detection model, and defines input and output of the pedestrian target detection model: defining the number and the format of input pictures, and defining output as a target envelope frame and a class; and manually labeling the acquired data by using a defined output scheme to obtain training data, and finally training the target detection model and updating the model weight.
S3, calculating a cost matrix according to the measurement between the pedestrian tracking target and the pedestrian target frame, wherein the cost matrix is used as the input of the Hungary algorithm; calculating a globally optimal matching pair of a pedestrian tracking target and a pedestrian target frame by using a Hungary algorithm;
conventional tracking algorithms typically use an iou distance (sort tracking algorithm) defined as the ratio of the intersection and union of the areas of two target frames and a mahalanobis distance (deepsorts tracking algorithm) defined as the distance in terms of the standard deviation of the distribution. Unlike with the iou distance and the mahalanobis distance, the metric of the present embodiment is defined as the probability density of the position of the detection frame in the prior distribution of the pedestrian tracking target position. Compared with iou, the probability density is taken as a measure to avoid the problem of heel-and-toe caused by that a pedestrian target frame and a pedestrian tracking target are not overlapped at all (iou=0) in a low frame rate scene. Compared with the mahalanobis distance, the method can avoid the problem that after the mahalanobis distance is lost, the covariance is larger and larger, so that the tracker is preferentially matched with the lost pedestrian target, and the pedestrian target which is continuously tracked is influenced, so that the normal pedestrian tracking target is unstable.
The measurement between the pedestrian tracking target and the pedestrian target frame is defined as：
；
；
wherein ,the probability density of the location of the pedestrian target box in the prior distribution of pedestrian tracking target locations,is Gaussian distribution->Covariance matrix of prior distribution of target (i.e. covariance matrix of prior distribution of predicted state) for pedestrian tracking,>for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>Is the transpose operation of the matrix.
Covariance matrix of prior distribution of the pedestrian tracking targetThe method comprises the following steps of obtaining by a state transition matrix and process noise:
；
wherein ,covariance matrix for the distribution of state variables, +.>Covariance matrix of process noise for Kalman filter, +.>Is->Time of day.
The cost matrixIs defined as follows:
；
wherein ,tracking target for nth pedestrian>，/>Probability density of the position of the target frame for the pedestrian in a priori distribution of tracking target positions for the pedestrian,/->、/>The positions of the mth pedestrian target frame are respectively the abscissa and the ordinate, and the +.>Is the transpose operation of the matrix.
The method comprises the steps of calculating a globally optimal matching pair of a pedestrian tracking target and a pedestrian target frame by using a Hungary algorithm, wherein the globally optimal matching pair is specifically as follows:
performing row permutation operation on the cost matrix by using a Hungary algorithm to minimize the trace of the matrix:
；
wherein ,for the permutation matrix +.>A cost matrix;
after being matched by the Hungary algorithm, the global optimal matching pair is obtained, and the definition of the matching pair is as follows:
；
wherein ,for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>Confidence level of pedestrian target frame obtained for pedestrian target detection model, < >>Id information of the target is tracked for pedestrians.
S4, updating posterior distribution of the pedestrian tracking target for the matched pedestrian tracking target and the pedestrian target frame, and outputting a result to a Kalman filter;
and for the matched pedestrian tracking target and pedestrian target frame, performing the following operations:
calculating residual error of pedestrian observance：
；
wherein ,observation matrix for Kalman filter, < >>For observing the model matrix>Tracking a priori state quantity of a target for a pedestrian;
calculating residual error of pedestrian observanceCovariance matrix>：
；
wherein ,for transpose operation of matrix,/->Tracking a target for a pedestrianCovariance matrix of the experimental distribution +.>Covariance matrix for distribution of state quantity; />The covariance matrix of the observed noise of the Kalman filter is used for updating the prior distribution of the pedestrian tracking target;
computing optimal gain for Kalman filtering：
；
Updating posterior distribution of pedestrian state quantity:
；
wherein ,covariance matrix of posterior distribution of state quantity;
updating posterior state quantity of pedestrian tracking targets:
；
wherein ,、/>and respectively using the current state quantity and the distribution of the pedestrian tracking target in the next iteration period.
The prior state quantity of the pedestrian tracking targetState of passageAnd (3) obtaining a conversion matrix:
；
；
wherein ,for state transition matrix>For the state quantity->For the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>、/>、/>、/>Respectively->、/>、/>、Derivative with respect to time.
If the maximum value of probability density of prior distribution of the unmatched pedestrian tracking targets is smaller than a preset threshold value, deleting the pedestrian tracking targets, and ending the life cycle; otherwise, the pedestrian tracking target is reserved, and the reserved pedestrian tracking target enters the next iteration period;
；
in comparison to previous tracking algorithms, a fixed constant is typically used to determine the life cycle of the tracked object, i.e., the tracked object is deleted at k frames after the tracking is lost, which is simple but not good. The pedestrian tracking target in this embodiment does not have a specific k-frame life cycle, but dynamically determines the stability of the pedestrian tracking target, and if the pedestrian tracking target is always stable, for example, standing or sitting at a certain position, the prior distribution of the pedestrian tracking target will slowly increase with time after losing, so that a longer time is required for the pedestrian tracking target to be deleted. On the contrary, the pedestrian tracking targets with rapid motion changes, such as people running or being blocked, the prior distribution of the pedestrian tracking targets can rapidly increase along with time after losing, and the pedestrian tracking targets can be deleted in a short time.
At the same time, the unmatched pedestrian tracks the target、/>And respectively using the current state quantity and the distribution of the pedestrian tracking target in the next iteration period.
For the unmatched pedestrian target frame, a new pedestrian tracking target is initialized, and the initial state quantity of the new pedestrian tracking target is determined by the coordinates of the pedestrian target frame.
The low frame rate in this embodiment refers to an image processing speed of 5-10 fps.
Meanwhile, the present embodiment provides:
a server comprising a processor and a memory, the memory having stored therein at least one program loaded and executed by the processor to implement the pedestrian target tracking method in a low frame rate scenario described above.
A computer-readable storage medium having stored therein at least one program loaded and executed by a processor to implement the pedestrian target tracking method in a low frame rate scenario described above.
The pedestrian target tracking method in the low-frame-rate scene can be suitable for security scenes and low-frame-rate scenes. The method is different from other methods in that local visual features are required to be extracted, and the tracking algorithm only uses the position and motion information of the target frame, but does not use visual features in the target frame, so that the operation amount and parameter amount of the tracking algorithm are reduced. The invention decouples detection and tracking algorithms so that suitable detection models can be used to solve complex scene problems such as dense targets, partial occlusion, low resolution, illumination changes, etc. Meanwhile, the method provided by the embodiment optimizes the low-frame-rate scene, and solves the problem that other algorithms are easy to lose targets in the low-frame-rate scene.
The above examples are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above examples, and any other changes, modifications, substitutions, combinations, and simplifications that do not depart from the spirit and principle of the present invention should be made in the equivalent manner, and the embodiments are included in the protection scope of the present invention.
Claims (11)
1. The pedestrian target tracking method in the low frame rate scene is characterized by comprising the following steps:
s1, initializing Kalman filtering parameters of a Kalman filter;
s2, acquiring a video image, inputting a picture of the acquired video image into a pedestrian target detection model, and determining a covariance matrix of observation noise of a Kalman filter in each iteration according to the confidence coefficient of a pedestrian target frame obtained by the pedestrian target detection model and the change rate of the aspect ratio of the pedestrian target frame;
s3, calculating a cost matrix according to the measurement between the pedestrian tracking target and the pedestrian target frame, wherein the cost matrix is used as the input of the Hungary algorithm; calculating a globally optimal matching pair of a pedestrian tracking target and a pedestrian target frame by using a Hungary algorithm;
s4, updating posterior distribution of the pedestrian tracking target for the matched pedestrian tracking target and the pedestrian target frame, and outputting a result to a Kalman filter;
and for the matched pedestrian tracking target and pedestrian target frame, performing the following operations:
calculating residual error of pedestrian observance：
；
wherein ,observation matrix for Kalman filter, < >>For observing the model matrix>Tracking a priori state quantity of a target for a pedestrian;
calculating residual error of pedestrian observanceCovariance matrix>：
；
wherein ,for transpose operation of matrix,/->Covariance matrix of a priori distribution of tracking targets for pedestrians,>covariance matrix for distribution of state quantity; />The covariance matrix of the observed noise of the Kalman filter is used for updating the prior distribution of the pedestrian tracking target;
computing optimal gain for Kalman filtering：
；
Updating posterior distribution of pedestrian state quantity:
；
wherein ,covariance matrix of posterior distribution of state quantity;
updating posterior state quantity of pedestrian tracking targets:
；
wherein ,、 />respectively serving as the current state quantity and distribution of the pedestrian tracking target in the next iteration period;
if the maximum value of probability density of prior distribution of the unmatched pedestrian tracking targets is smaller than a preset threshold value, deleting the pedestrian tracking targets, and ending the life cycle; otherwise, the pedestrian tracking target is reserved, and the reserved pedestrian tracking target enters the next iteration period;
for the unmatched pedestrian target frame, a new pedestrian tracking target is initialized, and the initial state quantity of the new pedestrian tracking target is determined by the coordinates of the pedestrian target frame.
2. The pedestrian target tracking method in a low frame rate scene as claimed in claim 1, wherein in step S1, the kalman filter parameters include an observation matrix of a kalman filterCovariance matrix of process noise->；
The abscissa, the ordinate, the width and the height of the pedestrian target frame are obtained through a detection algorithm, and are assigned to the observation matrix of the Kalman filter；
Covariance matrix of process noise of Kalman filterInitializing to an empirical constant;
wherein Is->Time of day.
3. The pedestrian target tracking method in a low frame rate scene as claimed in claim 1, wherein in step S2, a covariance matrix of observed noise of the kalman filterOptimization is performed by:
definition of uncertainty of observed noise of Kalman filter：
；
wherein ,、/>is the weight of two coefficients, +.>Confidence level of pedestrian target frame obtained for pedestrian target detection model, < >>The rate of change of aspect ratio for the pedestrian target frame; />，/>The width and the height of the pedestrian target frame are respectively;
covariance matrix of observed noise of Kalman filterThe definition is as follows:
；
wherein Is->Time of day.
4. The pedestrian target tracking method in a low frame rate scene as claimed in claim 1, wherein the pedestrian target detection model is based on a convolutional neural network model, the pedestrian target detection model is a yolov5s pedestrian target detection model, and input and output of the pedestrian target detection model are defined: defining the number and the format of input pictures, and defining output as a target envelope frame and a class; and manually labeling the acquired data by using a defined output scheme to obtain training data, and finally training the target detection model and updating the model weight.
5. The method according to claim 1, wherein in step S3, the measure between the pedestrian tracking target and the pedestrian target frame is defined as：
；
；
wherein ,probability density of the position of the target frame for the pedestrian in a priori distribution of tracking target positions for the pedestrian,/->Is Gaussian distribution->Covariance matrix of a priori distribution of tracking targets for pedestrians,>for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>Is the transpose operation of the matrix.
6. The method for pedestrian target tracking in a low frame rate scene as set forth in claim 5, wherein the covariance matrix of the prior distribution of pedestrian targets is a matrixThe method comprises the following steps of obtaining by a state transition matrix and process noise:
；
wherein ,covariance matrix for distribution of state quantity; />Covariance matrix of process noise for Kalman filter, +.>Is->Time of day (I)>Is a state transition matrix.
7. The pedestrian target tracking method in a low frame rate scene as claimed in claim 1, wherein in step S3, the cost matrixIs defined as follows:
；
wherein ,tracking target for nth pedestrian>，/>Probability density of the position of the target frame for the pedestrian in a priori distribution of tracking target positions for the pedestrian,/->、/>The positions of the mth pedestrian target frame are respectively the abscissa and the ordinate, and the +.>Is the transpose operation of the matrix.
8. The method for tracking pedestrian target in low frame rate scene according to claim 1, wherein in step S3, the matching pair of the pedestrian tracking target and the pedestrian target frame that is globally optimal is calculated by using a hungarian algorithm, specifically:
performing row permutation operation on the cost matrix by using a Hungary algorithm to minimize the trace of the matrix:
；
wherein ,for the permutation matrix +.>A cost matrix;
after being matched by the Hungary algorithm, the global optimal matching pair is obtained, and the definition of the matching pair is as follows:
；
wherein ,for the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>Confidence of the pedestrian target frame obtained for the pedestrian target detection model,id information of the target is tracked for pedestrians.
9. The pedestrian target tracking method in a low frame rate scene as set forth in claim 1, wherein the a priori state quantity of the pedestrian tracking targetThe method comprises the following steps of:
；
；
wherein ,for state transition matrix>For the state quantity->For the abscissa of the pedestrian target frame, +.>Is the ordinate of the pedestrian target frame, +.>For the width of the pedestrian target frame +.>For the height of the pedestrian target frame, +.>、/>、、/>Respectively->、/>、/>、/>Derivative with respect to time.
10. A server comprising a processor and a memory, wherein the memory has stored therein at least one program that is loaded and executed by the processor to implement the pedestrian target tracking method in the low frame rate scenario of any one of claims 1 to 9.
11. A computer readable storage medium having stored therein at least one program loaded and executed by a processor to implement the pedestrian target tracking method in a low frame rate scenario of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310809902.9A CN116580066B (en) | 2023-07-04 | 2023-07-04 | Pedestrian target tracking method under low frame rate scene and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310809902.9A CN116580066B (en) | 2023-07-04 | 2023-07-04 | Pedestrian target tracking method under low frame rate scene and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116580066A CN116580066A (en) | 2023-08-11 |
| CN116580066B true CN116580066B (en) | 2023-10-03 |
Family
ID=87536085
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310809902.9A Active CN116580066B (en) | 2023-07-04 | 2023-07-04 | Pedestrian target tracking method under low frame rate scene and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116580066B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188690A (en) * | 2019-05-30 | 2019-08-30 | 青岛伴星智能科技有限公司 | A kind of intelligent vision analysis system based on unmanned plane, intelligent vision analysis system and method |
| CN110378259A (en) * | 2019-07-05 | 2019-10-25 | 桂林电子科技大学 | A kind of multiple target Activity recognition method and system towards monitor video |
| CN111192296A (en) * | 2019-12-30 | 2020-05-22 | 长沙品先信息技术有限公司 | Pedestrian multi-target detection and tracking method based on video monitoring |
| CN112750150A (en) * | 2021-01-18 | 2021-05-04 | 西安电子科技大学 | Vehicle flow statistical method based on vehicle detection and multi-target tracking |
| CN113160274A (en) * | 2021-04-19 | 2021-07-23 | 桂林电子科技大学 | Improved deep sort target detection tracking method based on YOLOv4 |
| CN113985445A (en) * | 2021-08-24 | 2022-01-28 | 中国北方车辆研究所 | 3D target detection algorithm based on data fusion of camera and laser radar |
| CN114220176A (en) * | 2021-12-22 | 2022-03-22 | 南京华苏科技有限公司 | Human behavior recognition method based on deep learning |
| CN114299456A (en) * | 2021-12-24 | 2022-04-08 | 北京航空航天大学 | Intersection pedestrian crossing risk assessment method based on real-time track detection |
| CN114612933A (en) * | 2022-03-11 | 2022-06-10 | 电子科技大学 | Monocular social distance detection tracking method |
| CN114663795A (en) * | 2021-12-13 | 2022-06-24 | 江苏集萃未来城市应用技术研究所有限公司 | Target detection method for obtaining rear image of glass curtain wall by range gating imaging equipment |
| CN114782985A (en) * | 2022-03-28 | 2022-07-22 | 广东工业大学 | Pedestrian tracking and re-identification method based on monocular camera |
| CN115330833A (en) * | 2022-06-06 | 2022-11-11 | 华南农业大学 | Fruit yield estimation method with improved multi-target tracking |
| CN116129302A (en) * | 2022-12-09 | 2023-05-16 | 西南交通大学烟台新一代信息技术研究院 | Method for counting people flow in deep building site based on Yolov5 |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7492400B2 (en) * | 2004-06-01 | 2009-02-17 | The Board Of Trustees Of The Leland Stanford Junior University | Adaptive pixel for high dynamic range and disturbance detection and correction |
| JP6331785B2 (en) * | 2014-07-08 | 2018-05-30 | 日本電気株式会社 | Object tracking device, object tracking method, and object tracking program |
| JP6997106B2 (en) * | 2016-06-27 | 2022-01-17 | モービルアイ ビジョン テクノロジーズ リミテッド | Systems, methods and computer programs for navigating vehicles |
| US11030476B2 (en) * | 2018-11-29 | 2021-06-08 | Element Ai Inc. | System and method for detecting and tracking objects |
| US20220261593A1 (en) * | 2021-02-16 | 2022-08-18 | Nvidia Corporation | Using neural networks to perform object detection, instance segmentation, and semantic correspondence from bounding box supervision |
-
2023
- 2023-07-04 CN CN202310809902.9A patent/CN116580066B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110188690A (en) * | 2019-05-30 | 2019-08-30 | 青岛伴星智能科技有限公司 | A kind of intelligent vision analysis system based on unmanned plane, intelligent vision analysis system and method |
| CN110378259A (en) * | 2019-07-05 | 2019-10-25 | 桂林电子科技大学 | A kind of multiple target Activity recognition method and system towards monitor video |
| CN111192296A (en) * | 2019-12-30 | 2020-05-22 | 长沙品先信息技术有限公司 | Pedestrian multi-target detection and tracking method based on video monitoring |
| CN112750150A (en) * | 2021-01-18 | 2021-05-04 | 西安电子科技大学 | Vehicle flow statistical method based on vehicle detection and multi-target tracking |
| CN113160274A (en) * | 2021-04-19 | 2021-07-23 | 桂林电子科技大学 | Improved deep sort target detection tracking method based on YOLOv4 |
| CN113985445A (en) * | 2021-08-24 | 2022-01-28 | 中国北方车辆研究所 | 3D target detection algorithm based on data fusion of camera and laser radar |
| CN114663795A (en) * | 2021-12-13 | 2022-06-24 | 江苏集萃未来城市应用技术研究所有限公司 | Target detection method for obtaining rear image of glass curtain wall by range gating imaging equipment |
| CN114220176A (en) * | 2021-12-22 | 2022-03-22 | 南京华苏科技有限公司 | Human behavior recognition method based on deep learning |
| CN114299456A (en) * | 2021-12-24 | 2022-04-08 | 北京航空航天大学 | Intersection pedestrian crossing risk assessment method based on real-time track detection |
| CN114612933A (en) * | 2022-03-11 | 2022-06-10 | 电子科技大学 | Monocular social distance detection tracking method |
| CN114782985A (en) * | 2022-03-28 | 2022-07-22 | 广东工业大学 | Pedestrian tracking and re-identification method based on monocular camera |
| CN115330833A (en) * | 2022-06-06 | 2022-11-11 | 华南农业大学 | Fruit yield estimation method with improved multi-target tracking |
| CN116129302A (en) * | 2022-12-09 | 2023-05-16 | 西南交通大学烟台新一代信息技术研究院 | Method for counting people flow in deep building site based on Yolov5 |
Non-Patent Citations (1)
| Title |
|---|
| 一种邻帧匹配与卡尔曼滤波相结合的多目标跟踪算法;梅立雪 等;《科学技术与工程》;第19卷(第15期);第204页-第210页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116580066A (en) | 2023-08-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yazdi et al. | New trends on moving object detection in video images captured by a moving camera: A survey | |
| Sultana et al. | Unsupervised deep context prediction for background estimation and foreground segmentation | |
| WO2022217840A1 (en) | Method for high-precision multi-target tracking against complex background | |
| JP5036580B2 (en) | How to adapt the boost classifier to new samples | |
| Liu et al. | Nonparametric background generation | |
| Lin et al. | Integrating graph partitioning and matching for trajectory analysis in video surveillance | |
| CN111027493A (en) | Pedestrian detection method based on deep learning multi-network soft fusion | |
| JP2006209755A (en) | Method for tracing moving object inside frame sequence acquired from scene | |
| CN110569754A (en) | Image target detection method, device, storage medium and equipment | |
| CN110009060B (en) | Robustness long-term tracking method based on correlation filtering and target detection | |
| CN109063549B (en) | High-resolution aerial video moving target detection method based on deep neural network | |
| CN111274917B (en) | Long-time target tracking method based on depth detection | |
| CN113763427B (en) | Multi-target tracking method based on coarse-to-fine shielding processing | |
| WO2022218396A1 (en) | Image processing method and apparatus, and computer readable storage medium | |
| CN116402850A (en) | Multi-target tracking method for intelligent driving | |
| Hao et al. | Multiple person tracking based on slow feature analysis | |
| Zhan et al. | Salient superpixel visual tracking with graph model and iterative segmentation | |
| CN116309705A (en) | Satellite video single-target tracking method and system based on feature interaction | |
| Hao et al. | Recognition of basketball players’ action detection based on visual image and Harris corner extraction algorithm | |
| CN111931571A (en) | Video character target tracking method based on online enhanced detection and electronic equipment | |
| CN113379789B (en) | Moving target tracking method in complex environment | |
| CN116580066B (en) | Pedestrian target tracking method under low frame rate scene and readable storage medium | |
| Bardas et al. | 3D tracking and classification system using a monocular camera | |
| CN114612545A (en) | Image analysis method and training method, device, equipment and medium of related model | |
| CN113129332A (en) | Method and apparatus for performing target object tracking |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |