CN103699908A - Joint reasoning-based video multi-target tracking method - Google Patents

Abstract

A video multi-target tracking method based on joint reasoning in the field of video processing technology, by first reading in a frame image of a video file and performing image rasterization on it, and then using an online detector and a KLT tracking algorithm as a tracker Calibrate the candidate positions of the target, screen the results separately and synthesize the results, then quantify and score the candidate position results obtained, and finally use the joint function to describe the target tracking situation and use the optimal solution based on the joint function as the position of the target in this frame , that is, to achieve target tracking. The present invention can solve the processing method for the combination of detection and tracking algorithms and the processing of multi-target interrelationships in the tracking technology under multi-target tracking, and uses the joint function to describe the relationship between multiple targets, which not only solves the problem of fusion of detection and tracking results, but also Also considering the overall situation, the relationship between each objective is synthesized, and the global optimal solution is obtained.

Description

Multi-object Tracking Method for Video Based on Joint Reasoning

technical field

The invention relates to a method in the technical field of video processing, in particular to a video multi-target tracking method based on joint reasoning.

Background technique

With the development and popularization of camera equipment, video tracking plays an increasingly important role in production and life. Especially in video surveillance systems, tracking algorithms can effectively reduce labor costs and save time. However, due to the imperfection of the tracking technology itself and the complex and changeable tracking environment, the accuracy of the tracking results is affected, and the application of the tracking algorithm is limited.

Target tracking in video is a very challenging subject, because there are many uncertain factors in the tracking process, such as: complex background, if the background is similar to the tracked target, it will interfere with the tracking algorithm's judgment of the target position; obvious Object occlusion and rapid changes in the shape of the target will cause obvious changes in the appearance of the target in the screen, causing the algorithm to lose the tracked target. For multi-target tracking, in addition to the above-mentioned problems, the similarity, interaction and mutual occlusion between various targets will bring difficulties to correct tracking.

The general solution to these problems is often to use a combination of detection algorithms and tracking algorithms, that is, to use detection algorithms to improve the final tracking effect. However, under the premise that the correctness of the detection algorithm is difficult to guarantee, whether it can improve the tracking rate is still a problem.

Through the literature search of the existing technology, it is found that many scholars use the method of adaptive learning to ensure the correctness of the detector, so as to deal with the problem of coordination between the detection results and the tracking results. For example, Zdenek et al. in "IEEE Transaction on Pattern Analysis and Machine Intelligence" (2012 Issue 34, Pages 1409-1422) published "Tracking-Learning-Detection", which uses the framework structure of "Tracking-Learning-Detection (TLD)" and uses the method of combining tracking and detection to deal with Track issues. Specifically, first, a frame of image is input, and a large number of image blocks are obtained after the image is rasterized; then, the tracker screens out the image blocks that meet the requirements, and the detector also marks all images that may become targets in appearance block; finally, adopt the method of combining the tracking link and the detection link. When the tracker fails, the result of the detector can be used to reinitialize the tracker, and at the same time, the tracking result can be used to expand the training sample set to train the detector online to improve the accuracy of the detector. . From the experimental results published by this technology, it can be seen that TLD has a good effect on long-term tracking. However, this method still has many limitations: 1) It is only suitable for single target tracking 2) If the appearance of the target changes greatly, or if a complete occlusion occurs, this method does not work well, because in this case the online The detector cannot correctly give the possible position of the object.

Chinese Patent Document No. CN103176185A Application Publication Date: 2013.06.26, discloses a method and system for detecting road obstacles, the technology is based on the first obstacle detection model of the video camera device, based on the video camera device and millimeter wave radar The second obstacle detection model based on the three-dimensional laser radar and the third obstacle detection model based on the infrared camera device, and through the rough set-based fuzzy neural network algorithm, the multiple models form complementary detection, so as to obtain road obstacles in real time feature information. However, the equipment cost of this technology is high, and it cannot effectively track obstacles so as to combine historical information to make more accurate detection results.

Contents of the invention

Aiming at the above-mentioned deficiencies existing in the prior art, the present invention proposes a video multi-target tracking method based on joint reasoning, which can solve the processing method for the combination of detection and tracking algorithms and the processing of multi-target interrelationships in the tracking technology under multi-target tracking. The joint function is used to describe the relationship between multiple targets, which not only solves the problem of fusion of detection and tracking results, but also considers the overall situation, synthesizes the relationship between each target, and obtains the global optimal solution.

The present invention is achieved through the following technical solutions. The present invention firstly reads in a frame image of a video file and performs image rasterization processing on it, and then uses an online detector and the KLT tracking algorithm as a tracker to demarcate the candidate position of the target , respectively screen and synthesize the results, and then quantify and score the candidate position results obtained, and finally use the joint function to describe the target tracking situation and use the optimal solution based on the joint function as the position of the target in this frame, that is, to achieve target tracking .

When the image is the first frame of the video file, an initialization operation is performed before rasterizing the image, specifically: manually input the number of targets to be tracked, and then manually frame the targets.

The initialization operation refers to: initialize the detector and the tracker, update the detector through the method of random fern online learning, and at the same time, the tracker based on the KLT algorithm will also select feature points within the target range for the next frame. Target Tracking.

The image rasterization process refers to: scanning the whole frame of image with sliding windows of different sizes to obtain image blocks of different sizes in different positions, which are used as candidate targets, specifically: firstly, according to the size of the initialization target, equal A series of sliding windows of different scales are obtained, and the scaling range of scale transformation is 1.2 ^-10 ~ 1.2 ¹⁰ ; each sliding window traverses the entire image in sequence from left to right and top to bottom, and the displacement of the sliding window is 0.1 of the window size. In this way, many image blocks with different sizes at different positions can be obtained.

The candidate positions are obtained in the following ways:

其中：p_i指的是第i幅图像块的灰度图像，E()指的是求平均函数，当

则第i幅图像块就被保留，其中：l为固定参数，表示模板图像的方差。First, calculate the image block density variance. If the image block density variance is too small, it will be excluded. The target template image to be tracked is obtained when the first frame is initialized. The image block square variance calculation formula is:

Among them: p _i refers to the grayscale image of the i-th image block, E() refers to the averaging function, when

Then the i-th image block is reserved, wherein: l is a fixed parameter, Indicates the variance of the template image.

Then, binary features are extracted from the input image block, and the classifier obtained by online training using the random fern algorithm is used to estimate the similarity between each image block judged by the density variance and the detected target. The similarity judgment formula is: P(c ₁ |x)= Among them: c ₁ represents the training category. There are only two categories during training here. It is represented by c ₁ if it is similar to the detected target, and it is represented by c ₀ if it is not similar to the detected target; P _i (c ₁ |x) represents the ith object The posterior probability obtained by fern.

Finally, the posterior probability obtained by all ferns is averaged to obtain the final posterior probability value. When the similarity P(c ₁ |x)>50%, the input image block is similar to the detected target, and the image block is retained.

The calibration refers to: do the KLT algorithm initialization process in the first frame of the image, do not do tracking, and then use the KLT algorithm to select the tracked target feature from the target position of the previous frame in each frame, and find it in the current frame. The corresponding feature area; then the tracker decides whether to keep it according to the number of feature points in each image block of the image block. If the number of feature points in an image block area exceeds a certain empirical threshold, then the The image block is judged as a candidate state and retained.

其中：

的意思是被检测目标停在了级联Adaboost分类器的第

层，

即代表了被检测矩形框

的量化评分；Described quantitative score refers to: extract the Haar feature of candidate image block, evaluate the authenticity of candidate position by cascading Adaboost classifier, carry out the evaluation of quantification:

in:

means the detected target stopped at the first cascaded Adaboost classifier

layer,

That is, it represents the detected rectangular frame

quantitative score;

其中：代表了级联分类器中的一个弱分类器，s(L)代表了第L层的一系列弱分类器；当函数F_i大于某个经验阈值，那么就可以通过这一层，反之则不能通过；所述的弱分类器权重w_i,l是通过AdaBoost的学习方法学习得出的。

in: Represents a weak classifier in the cascade classifier, s(L) represents a series of weak classifiers in the L-th layer; when the function F _i is greater than a certain empirical threshold, then can pass this layer, and vice versa; the weak classifier weights w _i,l are learned through the learning method of AdaBoost.

The joint function is composed of the spatial position relationship between each target and the scoring of each target candidate position. By constructing the joint function model and finding the optimal solution of the function, the best candidate position can be obtained as the tracking result .

描述为：The spatial positional relationship mentioned above means that in the case of multiple targets, it can also be considered that the positional relationship between the targets does not change significantly between adjacent frames. Then the mutual position between the targets in the previous frame can also be used as a reference to help improve the tracking result of this frame. Based on this principle, at time t, the positional relationship probability between the i-th target and the N-th target is

described as:

函数ν和函数θ分别代表了在欧式空间上的距离和角度。β₁和β₂起到了平衡距离和角度权重的作用。函数

描述了t时刻目标i与目标N之间的空间位置信息，包括了距离的变化与角度的变化。 $d(x_i^t,x_N^t)=e^{-{\mathrm{\β}}_1{\left(\mathrm{\Δv}\right)}^2-{\mathrm{\β}}_2{\left(\mathrm{\Δ\θ}\right)}^2},$ in: $\mathrm{\Δ\ν}=\mathrm{\ν}(x_i^t,x_N^t)-\mathrm{\ν}(x_i^{t-1},x_N^{t-1}),\mathrm{\Δ\θ}=\mathrm{\θ}(x_i^t,x_N^t)-$

Function ν and function θ represent distance and angle in Euclidean space, respectively. β ₁ and β ₂ play the role of balancing distance and angle weights. function

It describes the spatial position information between target i and target N at time t, including the change of distance and angle.

代表了t时刻目标i的状态空间置信度，函数

描述了t时刻目标i与目标N之间的空间位置信息，包括了距离的变化与角度的变化。The joint function model refers to the joint probability of all targets, $p(x_1^t,......,x_N^t)\∝\exp ({\mathrm{\Σ}}_{i=1}^{N}g\left(x_i^t\right)+\mathrm{\γ}{\mathrm{\Σ}}_{i=1}^{N-1}d(x_i^t,x_N^t)),$ Where: γ is the weight parameter, the function

Represents the state space confidence of target i at time t, the function

It describes the spatial position information between target i and target N at time t, including the change of distance and angle.

每一个叶节点都传递信息至根节点处，那么根节点即被跟踪目标N节点处的置信度可以表示为：

当

取得最大值时，便是该帧多目标跟踪的跟踪结果。The optimal solution uses the standard belief propagation algorithm to effectively solve the multi-target tracking problem, and describes the relationship between the targets with a tree structure; when the tracked target is a node in the Markov random field, it is randomly selected One target is the root node, and the rest are child nodes, and transmit information to the root node. The last target is defined as the root node, and the others are the child nodes in the tree structure; in this way, the information transfer function for child nodes to transmit information to the root node is constructed :

Each leaf node transmits information to the root node, then the confidence of the root node, which is the tracked target N node, can be expressed as:

when

When the maximum value is obtained, it is the tracking result of multi-target tracking in this frame.

technical effect

Although the online detector can quickly detect the candidate position of the target, it produces more erroneous results and has a higher false alarm rate. Second, online detectors are only sensitive to objects whose appearance changes slowly. In the case of few training samples, the method is unstable for rotation problems, non-rigid targets, etc. Compared with online detectors, offline detection is more accurate but slower. In order to balance the functions of these two detectors, the present invention uses online detection to mark the target candidate positions, and uses offline detection to score the candidate positions, taking into account the advantages of the two technologies.

The invention not only balances off-line detection and on-line detection, makes the detector have better robustness, can achieve better detection results, and is used to improve the final tracking effect. Moreover, a joint function is proposed to describe the target tracking situation, which integrates the structural information between multiple targets, and also effectively fuses the results of the detector and tracker. Compared with the general tracking method, the present invention solves the tracking problem from a global perspective, and can better solve problems such as occlusion and appearance changes in the process of multi-target tracking, making it widely used in daily life and production application prospects.

Description of drawings

Fig. 1 is a schematic diagram of the processing flow of the embodiment of the present invention.

Fig. 2 is a schematic diagram 1 of an application example of the present invention.

Fig. 3 is a schematic diagram 2 of an application example of the present invention.

Detailed ways

The embodiments of the present invention are described in detail below. This embodiment is implemented on the premise of the technical solution of the present invention, and detailed implementation methods and specific operating procedures are provided, but the protection scope of the present invention is not limited to the following implementation example.

Example 1

The implementation environment of this embodiment is that the present invention can be used for tracking processing for videos collected indoors or outdoors by movable or fixed cameras. The specific method of use is as follows, at first, set up a fixed or movable camera in the place where monitoring and tracking is required to collect data; The first frame of the stream will require input of the number of targets to be tracked, and the tracked targets will be framed manually with a rectangle; finally, the present invention will automatically and accurately track and mark the selected targets in the video stream.

As shown in Figure 1, this embodiment includes the following steps: first read a frame image of the input video, if it is the first frame, do initialization settings; afterward, after the frame image is rasterized, it is input to the online detector Mark the candidate target positions with the tracker; then, input the candidate target positions into the scorer, and use the offline classifier to make a quantitative score for each candidate position; finally, input the quantified candidate position information and the spatial structure information of each target into the joint function , output the tracking results of each target through the optimal solution calculation.

As shown in Figure 1, in this embodiment, the user first inputs the video to be tracked, reads a frame of the video, and performs initialization processing at the first frame of the video, and the user selects the target to be tracked, which can be one or more. After the initial target selection is completed, the present invention automatically tracks the target.

Next, the frame image is rasterized to obtain many image blocks of different sizes as preliminary candidate positions. The rasterized images are then fed into the online detector and tracker respectively. The online detector first excludes image blocks with too small density variance, that is, image blocks containing less information, and then the program extracts the features of the remaining image blocks, and the online trained detector retains the image blocks that meet the requirements as candidate target positions; tracking In the device, the program first selects key points from the target result of the previous frame, and then finds the corresponding key points in this frame. The present invention will analyze the tracked points in each image block according to the empirical threshold value determined in the experiment. The number of key points to determine which image blocks to keep as target candidate positions.

After obtaining the target candidate positions, this program will use an offline classifier to quantify and score each candidate image block, and then combine the spatial relationship model between each target to input a joint function to describe the tracking situation of all targets. The program outputs the optimal solution of the joint function as the final tracking result. After getting the tracking result of the current frame, the program will also train the online detector to ensure the stability of the detection result of the next frame.

After each frame of the video image is traversed, this embodiment ends.

As shown in FIG. 2 , it is a schematic diagram of an application example of this embodiment. In this example, the parameters mentioned in the above formula are set as: λ=10, γ=60, β ₁ =0.1, β ₂ =2, 4 targets in the scene move from the left side of the lens to the right side at the same time, and the targets not only Being very close to each other will cause mutual occlusion, and the appearance of the four tracked targets is quite similar, which brings certain difficulties to tracking. Due to the use of structural reasoning functions, this method considers all target tracking results from a global perspective to obtain the optimal solution, which reduces the influence of mutual occlusion between targets, and the green target in frame#187 is not affected by the surrounding during the tracking process. Due to the influence of similar objects, it can accurately track 4 targets at the same time. It can be clearly seen from frame#220 that this method has a good tracking effect in this scene.

As shown in FIG. 3 , it is also a schematic diagram of an application example of the present invention. In this example, the parameters mentioned in the above formula are set as: λ=10, γ=60, β ₁ =0.1, β ₂ =2. In order to have a direct understanding of the tracking model of this method, the structural objects Did a track test. The test video shown in Figure 3 is self-recorded, and the tracked targets are the eyes, nose and mouth corners of the human face. The human face can be regarded as a structural object, consisting of five fixed parts, with the nose as the center, two eyes and the corners of the mouth around it. The algorithm proposed by this method can handle the occlusion problem very well. Even if there is an instantaneous occlusion, the tracking model in this paper does not lose the target. This is because the joint probability function of structural inference proposed in this paper utilizes the information of the interrelationship between various targets instead of only considering the probability of a single target. The mutual spatial relationship is used to infer the position of the tracked target.

Claims (10)

1. A video multi-target tracking method based on joint reasoning, characterized in that by first reading in a frame image of a video file and carrying out image rasterization processing to it, then using online detector and KLT tracking as tracker The algorithm calibrates the candidate position of the target, screens the results separately and synthesizes the results, then quantifies and scores the candidate position results obtained, and finally uses the joint function to describe the target tracking situation and takes the optimal solution based on the joint function as the target in this frame. position, that is, to achieve target tracking. 2. method according to claim 1, it is characterized in that, described initialization operation refers to: initialize detector and tracker, update detector by the method for random fern online learning, simultaneously the tracker based on KLT algorithm will also Feature points are selected within the target range for target tracking in the next frame. 3. The method according to claim 1, wherein said image rasterization processing refers to: scanning the whole frame of image with sliding windows of different sizes to obtain image blocks of different sizes in different positions, which are used as Candidate targets, specifically: firstly, according to the size of the initialization target, a series of sliding windows of different scales are obtained in proportion, and the scale range of scale transformation is 1.2 ^-10 ~ 1.2 ¹⁰ ; The entire image is traversed in order from top to bottom, and the sliding window displacement size is 0.1 of the window size, that is, many image blocks with different sizes at different positions are obtained. 4. method according to claim 1, is characterized in that, described candidate position obtains by following way: First, calculate the image block density variance. If the image block density variance is too small, it will be excluded. The target template image to be tracked is obtained when the first frame is initialized. The image block square variance calculation formula is: Among them: p _i refers to the grayscale image of the i-th image block, E() refers to the averaging function, when

Then the i-th image block is reserved, wherein: l is a fixed parameter,

Indicates the variance of the template image; Then, binary features are extracted from the input image block, and the classifier obtained by online training using the random fern algorithm is used to estimate the similarity between each image block judged by the density variance and the detected target. The similarity judgment formula is: Pc ₁ |x )=

Among them: c ₁ represents the training category. There are only two categories during training here. It is represented by c ₁ if it is similar to the detected target, and it is represented by c ₀ if it is not similar to the detected target; P _i (c ₁ |x) represents the ith object The posterior probability obtained by fern; Finally, the posterior probability obtained by all ferns is averaged to obtain the final posterior probability value. When the similarity P(c ₁ |x)>50%, the input image block is similar to the detected target, and the image block is retained. 5. The method according to claim 1, characterized in that, said calibration refers to: do KLT algorithm initialization processing in the first frame of the image, do not track, and then each frame from the previous frame target position In the method, the KLT algorithm is used to select the features of the tracked target, and the corresponding feature area is found in the current frame; then the tracker decides whether to keep it according to the number of feature points inside each image block of the image block, if an image block area If the number of feature points in the image exceeds a certain empirical threshold, then the image block is judged as a candidate state and is retained. 6. method according to claim 1, it is characterized in that, described quantitative scoring refers to: extract the Haar feature of candidate image block, evaluate the authenticity of candidate position by cascade Adaboost classifier, carry out the evaluation of quantification:

in:

means the detected target

stopped at the first cascaded Adaboost classifier

layer,

That is, it represents the detected rectangular frame

quantitative score;

in:

Represents a weak classifier in the cascade classifier, s(L) represents a series of weak classifiers in the L-th layer; when the function F _i is greater than a certain empirical threshold, then

can pass this layer, and vice versa; the weak classifier weights w _i,l are learned through the learning method of AdaBoost. 7. method according to claim 1, it is characterized in that, described joint function is made up of the spatial position relationship between each target and the scoring of each target candidate position, by building this joint function model, seek the optimum of function solution, the best candidate position can be obtained as the tracking result. 8. The method according to claim 7, wherein the spatial positional relationship means that, in the case of multiple targets, the mutual positions between targets in the previous frame can also be used as a reference to help improve the tracking of this frame As a result, at time t, the positional relationship probability between the i-th target and the N-th target

described as: $d(x_i^t,x_N^T)=e^{-{\mathrm{\β}}_1{\left(\mathrm{\Δv}\right)}^2-{\mathrm{\β}}_2{\left(\mathrm{\Δ\θ}\right)}^2},$ in: $\mathrm{\Δ\ν}=\mathrm{\ν}(x_i^t,x_N^t)-\mathrm{\ν}(x_i^{t-1},x_N^{t-1}),\mathrm{\Δ\θ}=\mathrm{\θ}(x_i^t,x_N^t)-$

Function ν and function θ respectively represent the distance and angle in Euclidean space; β ₁ and β ₂ play a role in balancing the weight of distance and angle; the function

It describes the spatial position information between target i and target N at time t, including the change of distance and angle. 9. method according to claim 7, is characterized in that, described joint function model refers to, the joint probability of all targets,

Among them: ε and γ are weight parameters, and the function

Represents the state space confidence of target i at time t, the function

It describes the spatial position information between target i and target N at time t, including the change of distance and angle. 10. The method according to claim 7, characterized in that, said optimal solution utilizes a standard belief propagation algorithm to effectively solve the multi-target tracking problem, and describes the relationship between each target with a tree structure; when being tracked The target is a node in the Markov random field, randomly select a target as the root node, and the rest are child nodes, and transmit information to the root node, define the last target as the root node, and the others as child nodes in the tree structure node; thus construct the information transfer function for the child node to transfer information to the root node:

Each leaf node transmits information to the root node, then the confidence of the root node, which is the tracked target N node, can be expressed as: when

When the maximum value is obtained, it is the tracking result of multi-target tracking in this frame.

Images