CN107240120B - Method and device for tracking moving target in video - Google Patents

Abstract

The invention relates to a method and device for tracking a moving object in a video. The method for tracking a moving object in a video includes the following steps: calculating the occlusion rate of the moving object in the current frame of the video under a first viewing angle; Calculate the learning rate of the spatio-temporal context model, and update the spatio-temporal context model of the moving target according to the learning rate; obtain the image feature value of the moving target in the current frame, and update the context prior model of the moving target according to the image feature value; update the updated The spatio-temporal context model and the context prior model perform a convolution operation to obtain the tracking position of the moving target in the next frame of the video from the first perspective. The tracking method of the moving target in the above video has computational complexity, high tracking efficiency and high tracking accuracy. Correspondingly, the present invention also provides a tracking device for a moving object in a video.

Description

Method and device for tracking moving target in video

technical field

The present invention relates to the technical field of video tracking, in particular to a method and device for tracking a moving object in a video.

Background technique

With the vigorous development of information technology, computer vision technology is more and more widely used in the field of video tracking, especially in the video analysis of sports events. Using computer vision to track moving targets for sports event analysis can greatly reduce labor costs and improve analysis accuracy. . In recent years, tracking algorithms based on online machine learning have developed rapidly, such as online Boosting algorithm, "tracking-detection-learning" algorithm, and tracking algorithms based on compressed sensing. However, the above-mentioned moving target tracking based on online machine learning Because the method needs to learn new models continuously, the calculation complexity is high, which affects the tracking efficiency, and is prone to tracking drift, and the tracking accuracy is low.

Contents of the invention

Based on this, it is necessary to provide a fast, accurate and effective tracking method and device for moving targets in video for the problems of low tracking efficiency and low tracking accuracy in traditional moving target tracking methods.

A method for tracking a moving target in a video, comprising the following steps:

Calculate the occlusion rate of the moving target in the current frame of the video from the first perspective;

Calculate the learning rate of the spatio-temporal context model according to the occlusion rate of the moving target, and update the spatio-temporal context model of the moving target according to the learning rate;

Obtain the image feature value of the moving target in the current frame, and update the context prior model of the moving target according to the image feature value;

Convolute the updated spatio-temporal context model and the context prior model to obtain the tracking position of the moving target in the next frame of the video from the first perspective.

The tracking method of the moving object in the above video calculates the learning rate of the space-time context model by calculating the occlusion rate of the moving object in the current frame of the video under the first perspective, and updates the space-time context model of the moving object according to the learning rate; then according to the image The feature value updates the context prior model of the moving target; performs convolution operation according to the updated spatio-temporal context model and the context prior model, and obtains the tracking position of the moving target in the next frame of the video from the first perspective. The moving target tracking method in the above video can realize the tracking and positioning of the moving target in the next frame by updating the context prior model of the space-time context model of the moving target, as long as the model is updated, it is not necessary to learn a new model all the time, which effectively reduces the calculation Complexity, effectively improve tracking efficiency, and, the tracking method of the moving object in the above video dynamically determines the learning rate of the spatio-temporal context model according to the occlusion of the moving object to update the spatio-temporal context model, which can avoid learning errors when the moving object is occluded by other objects The model can effectively avoid tracking drift and greatly improve tracking accuracy.

In one of the embodiments, the step of calculating the occlusion rate of the moving target of the current frame in the video under the first viewing angle includes:

Detect whether there is an intersection between the tracking frames of different moving objects in the current frame;

When intersecting points are included between the tracking frames of different moving objects, calculate the length and width of overlapping parts between the tracking frames of different moving objects, and calculate the occlusion area where the moving objects are blocked according to the length and width;

Obtain the tracking frame area of the pre-stored moving target, and calculate the occlusion rate of the moving target as the ratio of the occlusion area to the tracking frame area.

In one of the embodiments, the learning rate is calculated using the following formula:

in:

e is the natural logarithm;

ΔS is the occlusion rate of the moving target;

k, are constant parameters.

In one of the embodiments, the step of obtaining the image feature value of the moving target in the current frame includes:

Obtain the color intensity of the moving target in the current frame on the red channel, the color intensity on the green channel and the color intensity on the blue channel;

Assign corresponding color intensity weight values to the color intensity of the moving target on the red channel, the color intensity on the green channel, and the color intensity on the blue channel;

The weighted summation is carried out on the color intensity on each channel to obtain the image feature value of the moving target in the current frame.

In one of the embodiments, the method for tracking the moving target in the above video also includes:

Extract the edge area of the tracking site, establish a two-dimensional model of the tracking site, and project the tracking position to the first projection coordinates in the two-dimensional model of the tracking site.

In one of the embodiments, the method for tracking the moving target in the above video also includes:

Obtain the video under the second viewing angle, and calculate the second projection coordinates of the tracking position of the moving target in the video frame corresponding to the next frame in the video under the second viewing angle in the tracking site overlooking the two-dimensional model;

The occlusion rate of the current frame moving target in the video under the first viewing angle and the occlusion rate of the current frame moving target in the video under the second viewing angle are compared with the preset occlusion rate threshold;

When the occlusion rate of the moving object in the current frame of the video under the first viewing angle and the occlusion rate of the moving object in the current frame of the video under the second viewing angle are both less than or equal to the preset occlusion rate threshold, according to the first projected coordinates and the second Two projection coordinates calculate the target projection coordinates of the moving target in the two-dimensional model of the tracking site;

When the occlusion rate of the current frame moving target in the video under the first viewing angle is greater than the preset occlusion rate threshold, select the second projected coordinates as the target projected coordinates of the moving target in the two-dimensional model of the tracking site; when the second viewing angle When the occlusion rate of the moving target in the current frame of the video is greater than the preset occlusion rate threshold, the first projected coordinates are selected as the target projected coordinates of the moving target in the two-dimensional model of the tracking site.

In one of the embodiments, after selecting the second projected coordinates as the moving target after the step of tracking the projected coordinates of the target in the two-dimensional model of the ground, it also includes: correcting the first projected coordinates according to the second projected coordinates;

After the step of selecting the first projected coordinates as the moving target and tracking the projected coordinates of the target in the top-view two-dimensional model of the site, it also includes: correcting the second projected coordinates according to the first projected coordinates.

A tracking device for a moving target in a video, comprising:

The occlusion rate calculation module is used to calculate the occlusion rate of the moving target of the current frame in the video under the first viewing angle;

The spatio-temporal context model updating module is used to calculate the learning rate of the spatio-temporal context model according to the occlusion rate of the moving target, and updates the spatio-temporal context model of the moving target according to the learning rate;

The context prior model updating module is used to obtain the image feature value of the moving target in the current frame, and update the context prior model of the moving target according to the image feature value;

The tracking module is used to perform convolution operation on the updated spatio-temporal context model and the context prior model to obtain the tracking position of the moving target in the next frame in the video under the first perspective.

In one of the embodiments, the spatio-temporal context model update module includes:

The intersection detection submodule is used to detect whether an intersection is included between the tracking frames of different moving objects of the current frame;

The occlusion area calculation sub-module is used to calculate the length and width of the overlapping part between the tracking frames of different moving targets when the tracking frames of different moving targets include intersection points, and calculate the occlusion rate of the moving target according to the length and width Blocking area;

The occlusion rate calculation sub-module is used to obtain the tracking frame area of the pre-stored moving target, and calculate the occlusion rate of the moving target as the ratio of the occlusion area to the tracking frame area.

In one of the embodiments, the learning rate calculation module adopts the following formula to calculate the learning rate:

in:

e is the natural logarithm;

ΔS is the occlusion rate of the moving target;

k, are constant parameters.

Description of drawings

Fig. 1 is the flowchart of the tracking method of moving target in video in one embodiment;

Fig. 2 is a flowchart of calculating the occlusion rate of a moving target in one embodiment;

Fig. 3 is a schematic diagram of the calculation principle of the occlusion area of a moving target in an embodiment;

Fig. 4 is the flowchart of the tracking method of moving object in video in another embodiment;

Fig. 5 is a schematic diagram showing the space-time context information interface of a moving object in an embodiment;

Fig. 6 is a schematic view showing a two-dimensional model of a tracking site in an embodiment;

FIG. 7 is a schematic structural diagram of a tracking device for a moving target in a video in an embodiment;

FIG. 8 is a schematic structural diagram of a spatio-temporal context model update module in an embodiment;

FIG. 9 is a schematic structural diagram of a context prior model update module in an embodiment;

Fig. 10 is a schematic structural diagram of a spatio-temporal context model updating module in another embodiment.

Detailed ways

In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in further detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Please refer to Figure 1, a method for tracking a moving target in a video, including the following steps:

Step 102: Calculate the occlusion rate of the moving object in the current frame of the video under the first viewing angle.

Specifically, the occlusion rate of the moving object indicates the extent to which the moving object is occluded, and is calculated according to the area where the moving object is occluded. The terminal detects whether the moving target is blocked, and when the moving target is blocked, it calculates the blocking rate of the moving target; otherwise, the blocking rate of the moving target is 0.

Step 104: Calculate the learning rate of the spatio-temporal context model according to the occlusion rate of the moving target, and update the spatio-temporal context model of the moving target according to the learning rate.

Specifically, the spatial context model focuses on the spatial position relationship between moving objects and their local context, including distance and direction relationships. The video sequence is continuous, and the temporal context correlation is also very important for the tracking results. The spatio-temporal context model of the moving target in each frame is obtained by learning the spatio-temporal context model and the spatial context model of the tracking target in the previous frame through the learning rate. When the moving object is blocked by other objects, the appearance model of the moving object will be changed, and the reliability of the spatial context model of the moving object will decrease at this time. In this embodiment, the learning rate of the space-time context model of the moving object is adjusted to prevent learning to the wrong model. Specifically, dynamically determine the learning rate of the spatio-temporal context model according to the occlusion of the moving target, and update the spatio-temporal context model of the moving target in the next frame of the video under the first view according to the learning rate.

Step 106: Obtain the image feature value of the moving object in the current frame, and update the context prior model of the moving object according to the image feature value.

Specifically, the context prior model reflects the spatial composition of the current local context of the moving object itself, and is related to the image features of the context space and its spatial position structure. In this embodiment, the terminal obtains the image feature value of the moving object in the current frame, and according to the image The eigenvalues update the contextual prior model of moving objects.

Step 108: Convolute the updated spatio-temporal context model and the context prior model to obtain the tracking position of the moving target in the next frame of the video from the first perspective.

The tracking method of the moving object in the above video calculates the learning rate of the space-time context model by calculating the occlusion rate of the moving object in the current frame of the video under the first perspective, and updates the space-time context model of the moving object according to the learning rate; then according to the image The feature value updates the context prior model of the moving target; performs convolution operation according to the updated spatio-temporal context model and the context prior model, and obtains the tracking position of the moving target in the next frame of the video from the first perspective. The moving target tracking method in the above video can realize the tracking and positioning of the moving target in the next frame by updating the context prior model of the space-time context model of the moving target, as long as the model is updated, it is not necessary to learn a new model all the time, which effectively reduces the calculation Complexity, effectively improve tracking efficiency, and, the tracking method of the moving object in the above video dynamically determines the learning rate of the spatio-temporal context model according to the occlusion of the moving object to update the spatio-temporal context model, which can avoid learning errors when the moving object is occluded by other objects The model can effectively avoid tracking drift and greatly improve tracking accuracy.

As shown in Figure 2, in one embodiment, step 102 includes:

Step 1022: Detect whether there is an intersection between the tracking frames of different moving objects in the current frame.

In order to ensure the accuracy of the initial position of the moving target and lay a good foundation for subsequent tracking, in this embodiment, the initial position of the moving target in the first frame of the video under the first viewing angle is manually calibrated through human-computer interaction. A tracking frame is selected to determine the initial position of each moving target. Specifically, in this embodiment, the tracking frame is a rectangular frame. During the tracking process, the terminal detects in real time whether there is an intersection between the tracking frames of different moving objects. If there is an intersection between the tracking frames of different moving objects, it means that occlusion occurs between the moving objects, and step 1024 is executed; otherwise, the moving object No occlusion occurs, and the occlusion rate of the moving target is directly obtained as 0.

Step 1024: When the tracking frames of different moving objects include intersections, calculate the length and width of the overlapping parts between the tracking frames of different moving objects, and calculate the occlusion area where the moving objects are occluded according to the length and width.

As shown in Figure 3, in this embodiment, a coordinate system is established with the upper left corner of the tracking frame as the origin, the rightwards as the X axis, and the downwards as the Y axis. After completing the current frame tracking to obtain the tracking position of the moving target, the position coordinates of the apex of the tracking frame can be obtained. For the convenience of calculation, this embodiment selects the vertex coordinates of the upper left corner and the lower right corner of the tracking frame for calculation, wherein the vertex coordinates of the upper left corner of the tracking frame K1 are (minX1, minY1), and the vertex coordinates of the lower right corner are (maxX1, maxY1) , the vertex coordinates of the upper left corner of the tracking frame K2 are (minX2, minY2), and the vertex coordinates of the lower right corner are (maxX2, maxY2). The tracking frame K1 and the tracking frame K2 include two intersection points, which are the intersection point E and the intersection point F respectively. According to the abscissa of the lower right corner vertex of the tracking frame K1 and the ordinate of the upper left corner vertex of the tracking frame K2, the coordinates of the intersection point E can be obtained as (maxX1, minY2); Similarly, according to the abscissa of the upper left corner of the tracking frame K2 and the ordinate of the lower right corner of the tracking frame K1, the coordinates of the intersection point F are (minX2, maxY1). After obtaining the coordinates of the intersection point E and the intersection point F, the length and width of the overlapping portion of the tracking frame K1 and the tracking frame K2 can be calculated. Among them, the difference between the abscissa of the intersection point E and the abscissa of the upper left corner of the second tracking frame K2 is calculated to obtain the width of the overlapping part of the tracking frame K1 and the tracking frame K2, and the ordinate of the intersection point F and the second tracking frame K2 are calculated. The difference between the vertical coordinates of the upper left corner vertex is the length of the overlapping part of the tracking frame K1 and the tracking frame K2. Further, calculate the product of the length and width to obtain the occlusion area where the moving target is occluded, and the occlusion area Soverlap=(maxX1-minX2)*(maxY1-minY2).

In this embodiment, the length and width of the overlapping parts between the tracking frames are further calculated by calculating the intersection coordinates to calculate the occlusion area. However, it should be noted that the above embodiments are not used to specifically limit the calculation of the shading area. For example, in other embodiments, the occlusion area can also be calculated directly through the coordinates of the upper left vertex and the lower right vertex of the tracking frame K1 and the coordinates of the upper left vertex and the lower right vertex of the tracking frame K2. For ease of description, Fig. 3 is still taken as an example. In one embodiment, minX=max(minX1, minX2) is defined, that is, minX is the larger value among minX1 and minX2; at the same time, maxX=min(maxX1, maxX2), maxX is the smaller value of maxX1 and maxX2; minY=max(minY1,minY2), minY is the larger value of minY1 and minY2; maxY=min(maxY1,maxY2), maxY is the value of maxY1 and maxY2 smaller value. During the tracking process, the terminal compares the size of minX and maxdX, and the size of minY and maxY in real time, judges whether occlusion occurs between the tracking frames according to the comparison result, and calculates the occlusion area when occlusion occurs. Specifically, if minX<maxX and minY<maxY, then the tracking frame K1 and the tracking frame K2 overlap, and the moving object is occluded, and the occlusion area is calculated as: Soverlap=(maxX-minX)*(maxY-minY). As shown in Figure 3, in this embodiment, minX=minX2, minY=minY2, maxX=maxX1, maxY=maxY1, and the occlusion area Soverlap=(maxX1-minX2)*(maxY1-minY2).

Step 1026: Acquire the pre-stored tracking frame area of the moving object, and calculate the ratio of the occlusion rate of the moving object to the area of the tracking frame.

Specifically, in step 1022, after the tracking frame of the moving object is calibrated, the area of the tracking frame is calculated and stored. After the occlusion area of the moving object is calculated in step 1024, the terminal reads the area of the tracking frame, calculates the ratio of the occlusion area to the area of the tracking frame, and obtains the motion The occlusion rate of the target is as follows:

Among them, S _overlap is the occlusion area of the moving target, and S ₀ is the area of the tracking frame.

In one embodiment, in step 104, the following formula is used to calculate the learning rate:

Among them: e is the natural logarithm; ΔS is the occlusion rate of the moving target; k, are constant parameters. Specifically, the value range of k is 2-4; The range of values is 1.5 to 2.5. In one embodiment, k=3,

In one embodiment, in step 106, the step of obtaining the image feature value of the moving object in the current frame includes: obtaining the color intensity of the moving object in the current frame on the red channel, the color intensity on the green channel, and the color intensity on the blue channel. The color intensity on the color channel; assign corresponding color intensity weight values to the color intensity of the moving target on the red channel, the color intensity on the green channel, and the color intensity on the blue channel; The color intensity is weighted and summed to obtain the image feature value of the moving target in the current frame.

Specifically, the color intensity of the moving object on the red channel, the color intensity on the green channel, and the color intensity on the blue channel are given corresponding color intensity weight values according to the red channel, green channel and blue channel of different moving objects. The difference in color intensity on a channel is determined, and the greater the difference in color intensity, the greater the weight value of the color intensity on that channel. In this embodiment, the color feature value of each moving object is determined by the color difference between different moving objects for updating the context prior model, so as to ensure the accurate tracking of the context prior model and further improve the tracking accuracy.

In one embodiment, the method for tracking a moving target in the above-mentioned video further includes: extracting the sideline area of the tracking site, establishing a two-dimensional model of the tracking site, and projecting the tracking position to the first projected coordinates in the two-dimensional model of the tracking site.

The coordinates and positional relationship of the moving target obtained by tracking through steps 102 to 108 are the positions in the original view captured by the camera of the first viewing angle. In order to visualize the tracking results for easy tracking and analysis, in this embodiment , establish a top-view two-dimensional model of the tracking site to simultaneously display the tracking position of each moving target. In the two-dimensional model of the tracking site, each moving target has a target identification. After each frame of tracking is completed, the target identification of the moving target is moved from the corresponding first projected coordinates in the previous frame to the currently determined tracking position. corresponding to the first projected coordinates.

Generally, the 2D model map of the tracking site is a top view, while the shooting angle of the original video is generally a side view with a certain angle. In this embodiment, the angle of view and data scale conversion are performed according to the position and angle of the camera, and the tracking position of the moving target is synchronously displayed on the top-view two-dimensional model of the tracking site. Specifically, this embodiment establishes the conversion relationship between the original video image and the two-dimensional model through homogeneous transformation. First, the projection transformation of the two-dimensional plane is expressed as the product of the vector under the homogeneous coordinates and a 3x3 matrix, which is x’=Hx, and the specific homography transformation matrix is expressed as follows:

From the above homography transformation matrix, it can be seen that the planar homography transformation has eight degrees of freedom, and the homography transformation matrix can be obtained by solving the eight unknowns in the transformation matrix, and the target projection transformation is completed. Since a set of corresponding point coordinates can be obtained by the above matrix multiplication, two equations are required, and four sets of equations are required for all unknowns in the original transformation matrix. Therefore, if the homography transformation matrix is required, it is only necessary to know the corresponding four sets of point coordinates. . Specifically, in this embodiment, four sets of vertex coordinates of the tracking site are determined by extracting the edge area of the tracking site to obtain a transformation matrix and realize two-dimensional projection transformation. In this embodiment, the two-dimensional projection transformation of the three-dimensional video image is calculated through the homography transformation matrix, without obtaining the parameter information of the camera equipment, the video analysis system is simple and easy to use, and the conversion flexibility is high.

In one embodiment, the above-mentioned method for tracking a moving object in a video further includes: acquiring the video under the second viewing angle, and calculating the tracking position of the moving object in the video frame corresponding to the next frame in the video under the second viewing angle. Track the second projected coordinates in the two-dimensional model of the field overlooking; respectively compare the occlusion rate of the current frame moving object in the video under the first viewing angle and the occlusion rate of the current frame moving object in the video under the second viewing angle with the preset occlusion When the occlusion rate of the moving object in the current frame of the video under the first viewing angle and the occlusion rate of the moving object in the current frame of the video under the second viewing angle are less than or equal to the preset occlusion rate threshold, according to the first The first projection coordinates and the second projection coordinates calculate the target projection coordinates of the moving target in the two-dimensional model of the tracking site; when the occlusion rate of the current frame of the moving target in the video under the first viewing angle is greater than the preset occlusion rate threshold, select the second The second projection coordinates are used as the target projection coordinates of the moving target in the two-dimensional model of the tracking site; when the occlusion rate of the current frame of the moving target in the video under the second viewing angle is greater than the preset occlusion rate threshold, the first projection coordinate is selected as the motion The projected coordinates of the target in the 2D model of the tracking site.

Specifically, the determination of the tracking position of the moving target in the next frame in the video under the second viewing angle and the conversion process and principle of the second projected coordinates of the tracking position in the two-dimensional model of the tracking site overlooked are the same as those in the video under the first viewing angle. The determination of the tracking position of a moving target in one frame and the conversion of the second projected coordinates of the tracking position in the two-dimensional model of the tracking site overlooked are the same, and will not be repeated here.

In the multi-moving target tracking scene, due to the complex movement of the target, it is extremely easy to have large-area occlusion or even complete occlusion. If two tracking frames are superimposed together, drift and jump will occur. During occlusion, even if there is no drift jump, the coordinate information obtained by the occluded object is inaccurate because the distance between objects with occlusion relationship cannot be judged under this viewing angle. Therefore, this embodiment judges whether the first projected coordinates of the moving object obtained under the first viewing angle and the second projected coordinates of the moving object obtained under the second viewing angle are wrong according to the occlusion of the moving object. The occlusion of the target is more serious. If the occlusion rate of the moving target is greater than the preset occlusion rate threshold, the projected coordinates of the moving target obtained from this viewing angle are wrong. Select the projected coordinates of the moving target obtained from another viewing angle as the final target projection Coordinates; if the occlusion rate of the moving target in the two viewing angles is less than or equal to the preset occlusion rate threshold, the projected coordinates of the moving target obtained in the two viewing angles are all correct, which is the first projection coordinate and the second projection The coordinates are assigned weight values, and the weighted sum calculation is performed according to the first projected coordinates and the second projected coordinates to obtain the target projected coordinates of the moving target in the two-dimensional model of the tracking site, and at the same time, the pair is tracked according to the first projected coordinates and the second projected coordinates Results are optimized to ensure accurate tracking.

Specifically, in the tracking process, the size of the tracking frame is fixed, but the tracked moving target has a relationship between near and far being small in the perspective of the camera. Therefore, in one embodiment, the preset occlusion rate threshold defined to define the occlusion situation is related to the distance of the target from the camera on the 2D model, and the preset occlusion is calculated according to the distance of the moving target from the camera in the 2D model of the tracking field overlooking rate threshold, and calculate the tracking position of the moving target in the next frame of the first viewing angle video and the tracking position of the moving target in the next frame of the second viewing angle video according to the distance between the moving target in the first viewing angle video and the second viewing angle video. Weights.

In this embodiment, through simultaneous tracking of videos under different viewing angles, the tracking position of the moving target obtained under each viewing angle is projected into the two-dimensional model of the tracking site, and according to the occlusion of the moving target, the two viewing angles The tracking results of the same moving target are unified, and the tracking results are optimized based on dual-view video tracking to ensure accurate tracking and greatly improve tracking accuracy.

In one embodiment, after selecting the second projected coordinates as the moving target, after the step of tracking the projected coordinates of the target in the top view two-dimensional model of the field, it further includes: correcting the first projected coordinates according to the second projected coordinates; selecting the first projected coordinates Coordinates as the moving target After the step of tracking the projected coordinates of the target in the top view two-dimensional model of the site, the method further includes: correcting the second projected coordinates according to the first projected coordinates. In this embodiment, when the tracking result under a certain perspective is wrong, the wrong tracking result is corrected through the tracking result under another perspective, and the space-time context model is updated according to the corrected tracking result to ensure that the follow-up tracking results Accurate, further improving tracking accuracy.

Further, in order to facilitate the understanding of the technical solution of the present invention, below in conjunction with Fig. 4 to Fig. 6, take football video tracking as an example to describe the tracking method of the moving target in the above-mentioned video in detail. For the convenience of explanation, two football teams are defined as team A and team B, where the players of team A are identified as rectangles in the two-dimensional model of the field overlooking the stadium, and the players of team B in the two-dimensional model of the field overlooking the player are identified as circles shape.

A method for tracking a moving target in a video, comprising the following steps:

1) Determine the initial position of the moving target, and calibrate the tracking frame of the moving target.

First, read the t-th frame image, and determine the initial position of each player (that is, the moving target) in the t-th frame by manually marking the tracking frame. Specifically, when manually marking the player tracking frame, the mouse can be used to select the tracking frame, and the initial position of the player in the first frame of the video under the first viewing angle and the video under the second viewing angle are respectively calibrated to determine the first viewing angle. The player's initial position in the first frame of the video below and in the second-view video. Further, after the player's initial position calibration is completed, the terminal further calculates and stores the tracking frame area of each player's tracking frame.

2) Calculate the occlusion rate of the moving target in the current frame.

Specifically, calculate the occlusion rate of each player in the corresponding current frame in the video under the first viewing angle and the video under the second viewing angle, according to the tracking frame area of each player’s tracking frame in the current frame and the occlusion of the current player Calculate the occlusion rate of each player in the corresponding current frame in the video under the first viewing angle and the video under the second viewing angle. The calculation principle and process of the occlusion rate of a specific moving target have been described in detail in the foregoing embodiments. , which will not be repeated here.

3) Calculate the learning rate of the spatio-temporal context model and update the spatio-temporal context model.

The temporal context information is the time correlation between consecutive frames, and the spatial context information is the combination of the tracking target and the background image within a definite range around it. Using spatio-temporal context information to track a target first requires the establishment of a tracking model. Specifically, the target tracking problem is the probability of a target's location. Let o be the target to be tracked, x be a two-dimensional coordinate point on an image, P(x|o) means that the coordinate x appears in the target o, and the target tracking is transformed into a calculation problem of maximum confidence.

make:

M(x)＝P(x|o); formula (4)

Then when the value of the confidence map M(x) is the largest, the corresponding coordinate x can be considered as the most likely position of the target o. As shown in Figure 5, the area within the solid line frame is the area where the target is located, and the area within the outer dashed line frame is the local context area. Use the target center position coordinate x ^* to represent the position of the target, and z to be a point in the local context area. Define the local context area of the target x ^* as Ω _c (x ^* ), and define the context feature set of this local area as X ^C ＝{c(z)＝(I(z),z)|z∈Ω _c (x ^* )}, where I(z) is the image feature value at the z coordinate. Using the full probability formula and taking the local context feature as the intermediate quantity, the formula (4) can be expanded to get:

Among them, P(x|c(z),o) represents the probability of the target appearing at point x when the target o and its local context feature c(z) are given, which is the spatial relationship between the location of the tracking target and its context information A spatial context model is established. And P(c(z)|o) represents the probability that a contextual feature c(z) appears in the target o, which is the context prior probability of the target o, and is the appearance prior model of the current local context. Among them, the context prior model indicates that when the target position is predicted by calculating the confidence map M(x), the selected context is similar to the target position in the previous frame, while the spatial context model guarantees that the selected new The target position is not only similar to the original target in appearance, but also reasonable in spatial position, so as to avoid interference caused by other similar-looking objects to a certain extent, and avoid drifting in tracking.

Based on the above, in this embodiment, a specific mathematical model is established in advance for each part in the formula (5), specifically including confidence map modeling, spatial context model modeling and context prior model modeling.

First, the modeling of the confidence map is as follows: Since the target position in the first frame of the video is known (according to the tracking frame calibration of the initial frame), the confidence map M(x) should satisfy the closer the distance to the target x* position, its The larger the confidence is of this property. Therefore, let:

Among them, b is a normalized constant parameter; α is a scale constant parameter, and β is a function curve image control constant parameter. α is related to the size of the tracking target, and its value ranges from 1.75 to 2.75; β ranges from 0.5 to 1.5. In one embodiment, α=2.25 and β=1.

Secondly, the modeling of the spatial context model P(x|c(z),o) is as follows: Since the spatial context model focuses on the spatial position relationship between the tracking target and its local context, that is, including the distance and direction relationship, the definition of P(x |c(z),o) is a non-radial symmetric function:

P(x|c(z),o)=hsc(x-z); formula (7)

Among them: x is the position of the target, z is a certain position in the local context, even when there are two points z1, z2 with the same distance from the center position x* of the target, since their positions are different, hsc(x*-z1 )≠hsc(x*-z2), which means that they represent different contexts for x*, so as to effectively distinguish different spatial relationships and prevent ambiguity.

Finally, the modeling of the context prior model P(c(z)|o) is as follows: the context prior model reflects the spatial composition of the current local/context itself, intuitively considered, it should be related to the image features of the context space and its spatial position structure related. Therefore, let:

P(c(z)|o)＝I(z)ω _σ (zx ^* ); formula (8)

Among them, I(z) is the image feature value at point z in the local context area, and ω _σ (Δ) is the weight function.

Specifically, in the tracking process, it can be compared to the process of human eyes tracking something. The closer the context area to the tracking target, the more relevant it is to the tracking target, and therefore the higher the importance, while the farther away from the tracking target The context area, which can be considered as a part that is relatively irrelevant to the tracking target, is therefore less important. Accordingly, define:

Among them, Δ is the distance between two points, λ is a normalized constant parameter, which is used to make the value of P(c(z)|o) between 0 and 1, so as to conform to the definition of probability function; σ is a scale parameter , which is related to the size of the tracking target.

Substitute formula (9) into formula (8) to get the context prior model as follows:

That is, the spatial composition of the local context is modeled as the Gaussian weighted sum of the image feature values of each point in this region.

Further, in this embodiment, after completing the above confidence map modeling, spatial context model modeling and context prior model modeling, the spatiotemporal context model is further updated according to the above confidence map, spatial context model and context prior model:

First, substitute formula (5) into formula (6), formula (7) and formula (10), and get:

Among them, hsc(x-z) is the spatial context model, that is, the object to be calculated and learned for each frame of image.

According to convolution Definition:

Equation (11) can be changed to:

According to the convolution theorem, there are:

but:

in, and denote the Fourier and inverse Fourier transforms, respectively.

Assume that at the tth frame, the center position of the target is known and the local context area Ω _c (x ^* ) of the target, the spatial context model of the tracking target and its local context area in the tth frame can be calculated, denoted as Since we are dealing with continuous video sequences, temporal contextual correlation is also crucial for tracking results. In order to take this dimension into consideration, set the spatio-temporal context model learning rate ρ, and express the spatio-temporal context model of each frame tracking target as two parts: the historical spatio-temporal context model and the newly acquired spatial context model, as follows:

in, is the spatial context model of frame t; is the spatio-temporal context model of frame t, is the spatio-temporal context model of frame t+1.

Generally, in a situation where there are multiple tracking targets with similar appearance, when an occlusion occurs, that is, the appearance model of the target has changed greatly, and the spatio-temporal context model is still learning and updating at the same rate at this time. It will continue to learn the wrong model and eventually lose the tracking target. In this embodiment, the learning rate is dynamically determined according to the occlusion of the moving object. The learning rate ρ is an automatically updated dynamic value, which can effectively prevent the historical model information from being completely lost due to too fast updating. Specifically, when the tracking target is occluded by other objects, the target appearance model will be changed, and the reliability of the spatial context model will be reduced, so the learning rate needs to be reduced to prevent learning the wrong model and ensure accurate tracking. In this embodiment, the learning rate of the spatiotemporal context model is calculated according to the occlusion rate of the moving object to update the spatiotemporal context model of the moving object obtained above. Specifically, the learning rate of the spatio-temporal context model is calculated according to formula (3), which will not be repeated here.

4) Obtain the image feature value of the moving target, and update the context prior model of the moving target.

Specifically, the image feature value of the moving target is calculated by the following formula:

I(x)=w ₁ ·I _R (x)+w ₂ ·I _G (x)+w ₃ ·I _B (x); formula (17)

Among them, I _R (x) is the color intensity of x on the red channel; I _G (x) is the color intensity of x on the green channel; I _B (x) is the color intensity of x on the blue channel ; w ₁ , w ₂ , and w ₃ are weights and w ₁ +w ₂ +w ₃ =1. In one embodiment, the color of the uniforms of the two teams is significantly different in the R channel, and I _R (x) is assigned a greater weight, w ₁ =0.4, w ₂ =0.3, and w ₃ =0.3.

5) Perform convolution operation on the updated spatio-temporal context model and context prior model to obtain the tracking position of the moving target in the next frame.

Specifically, at frame t+1, it is known that its spatio-temporal context model is updated Then calculate the context prior model of t+1 frame That is, the confidence map of the t+1th frame can be obtained by performing convolution calculation from formula (7) to formula (11), as follows:

Then, when the confidence map M _t+鯐(x) of the t+1th frame takes the maximum value, the corresponding x value is considered to be the center position of the moving target in the t+1th frame Therefore, the tracking position of the moving target is determined, that is, the tracking position of the next frame of the tracked player in the video under the first viewing angle and the tracking position of the next frame of the tracking player in the video of the second viewing angle are respectively determined.

6) Establish a two-dimensional model of the stadium overlooking, project the tracking position of the next frame of the moving target in the video under the first viewing angle to the first projection coordinates in the two-dimensional model of the stadium overlooking, and project the next frame of the moving target in the video under the second viewing angle The tracking position of the moving target in the frame is projected to the second projected coordinates in the two-dimensional model overlooking the stadium.

Specifically, in this embodiment, the four corner points of the football match field are selected as the four reference points for calculating the plane homography transformation matrix. First, through a series of threshold processing techniques in digital image processing and Hough transform line detection, extract the sideline area of the court; then merge the scattered line segments to obtain the straight line equation of the sideline of the court and four sets of coordinates of calibration points. Finally, according to the four Group the coordinates of the calibration points to obtain the transformation matrix of the two viewing angles. The specific two-dimensional model of the stadium is shown in Figure 6.

7) Detecting whether the first projected coordinates and the second projected coordinates are wrong.

Specifically, the occlusion rate of the current frame player in the video under the first viewing angle and the occlusion rate of the current frame player in the video under the second viewing angle are compared with the preset occlusion rate threshold, and the first projection coordinate and the second projection coordinate are determined. 2 Whether the projected coordinates are wrong. If the occlusion rate of the current frame player in the video from the first viewing angle is greater than the preset occlusion rate threshold, the first projection coordinates are wrong; if the occlusion rate of the current frame player in the video from the second viewing angle is greater than the preset occlusion rate threshold , the coordinates of the second projection are wrong. If the occlusion rate of the current frame player in the video from the first viewing angle and the current frame player’s occlusion rate in the video from the second viewing angle are both less than or equal to the preset occlusion rate threshold, then the first projection coordinate and the second projection coordinate All are correct. In this embodiment, the preset occlusion rate threshold is calculated according to the distance from the camera to the player in the two-dimensional model overlooking the stadium, wherein the distance from the camera to the player in the two-dimensional model overlooking the stadium is:

Among them, [x, y] are the coordinates of the players in the current frame on the two-dimensional model of the stadium, and height and width are the height and width of the two-dimensional model of the stadium.

Then, the preset occlusion rate threshold is:

threshold＝γe- ^μ·Δd ; formula (20)

Among them, threshold is the preset occlusion rate threshold; γ and μ are constant parameters, γ is used to adjust the range of the preset occlusion rate threshold, and μ is used to adjust the speed of the preset occlusion rate threshold change.

8), when the first projected coordinates or the second projected coordinates are wrong, choose the projected coordinates of another angle of view as the player's target projected coordinates.

Specifically, the shooting angles of the video under the first angle of view and the video under the second angle of view are different, and the players in the video captured under the first angle of view are blocked, but at this time, the players in the video captured under the second angle of view will not Occlusion occurs so that both the first projected coordinate and the second projected coordinate cannot be wrong at the same time. Therefore, when the first projected coordinates are wrong, select the second projected coordinates as the player's target projected coordinates, and the tracking of the tth frame ends. When the second projected coordinates are wrong, the first projected coordinates are selected as the player's target projected coordinates, and the tracking of frame t ends.

Further, in one embodiment, when the tracking result under a certain viewing angle is wrong, the wrong tracking result is also corrected by using the tracking result under another viewing angle, so as to ensure the accuracy of the subsequent tracking result. Assuming that the coordinates of the first projection are wrong, in the first viewing angle, the maximum likelihood position of the tracked player with tracking drift given by the confidence map is P ₁ , and the projection matrix for converting the first viewing angle video to the two-dimensional model of the pitch is H ₁ , at this time, the maximum likelihood position of the tracked player in the second perspective is P ₂ , and the projection matrix for converting the second perspective video to the two-dimensional model of the pitch is H ₂ , then P ₂ looks down on the two-dimensional model of the pitch The second projected coordinates on is H ₂ ·P ₂ , since P ₂ is the result of correct tracking, the wrong tracking position P ₁ is updated to the correct tracking position. The first projected coordinates of the first viewing angle are:

P ₁ =H ₁ ⁻¹ ·H ₂ ·P ₂ ; formula (21)

Similarly, if the second projected coordinates are wrong, the second projected coordinates are corrected according to the first projected coordinates. The specific correction principle is the same as that of the first projected coordinates described above, and will not be repeated here. After completing the correction of the first projected coordinates or the second projected coordinates, further update the spatio-temporal context model under the corresponding perspective according to the corrected tracking results to ensure the accuracy of subsequent tracking results.

9), when the first projected coordinates and the second projected coordinates are correct, calculate the target projected coordinates of the player according to the first projected coordinates and the second projected coordinates.

When the first projected coordinates and the second projected coordinates are all correct, through the first projected coordinates and the second projected coordinates, the mutual auxiliary adjustment determines the player's target projected coordinates, after determining the target projected coordinates, the tth frame tracking ends. Specifically, in the image after projection transformation, the position of the player is relatively clear at a position closer to the camera, while at a position farther from the camera, the specific position of the player is relatively clear due to deformation and stretching. Vague. Therefore, when the target is closer to the camera at a certain angle of view, the tracking result in the video shot by the camera is considered to be more reliable, that is, the tracking result obtained from this angle of view, when the target position is finally determined, the more weight it occupies. is large, so the weight values of the first projected coordinates and the second projected coordinates are determined according to the distance between the target and the camera.

Assume that in the first viewing angle, the camera is at the position shown in Figure 6 . Define the position of the photography in the first perspective as the origin, and the coordinates of player M of team B on the two-dimensional model of the pitch are pos _model 1=[x ₁ y ₁ ], then:

As shown in Figure 6, the camera under the second viewing angle is located opposite to the camera under the first viewing angle, and the coordinates of the tracking result of the player M obtained by the camera under the second viewing angle are converted to the two-dimensional model of the pitch as pos _model 2= [x ₂ y ₂ ], then the distance between player M and the camera in the second perspective is:

Among them, width and height are the width and height of the two-dimensional model of the stadium overlooking.

Then, after fusing the first projected coordinates and the second projected coordinates, the final position of the player M on the two-dimensional model of the stadium is: pos _final = [xy],

Further, in one embodiment, according to above-mentioned steps (1) to step (10), the football video under two viewing angles is tracked, and the tracking operation is realized on the PC computer, hardware environment: central processing unit: Intel Core i5 , The main frequency is 2.5GHz, and the memory is 8GB. The programming environment is Matlab 2014a. The original video under the two viewing angles is in avi format, the size of each frame is 1696x1080, and the video size is about 20MB. It reaches 1s/frame, and the tracking accuracy reaches 100%.

Please refer to FIG. 7 , a tracking device 700 for a moving target in a video, including:

The occlusion rate calculation module 702 is used to calculate the occlusion rate of the moving target in the current frame in the video under the first viewing angle.

The spatio-temporal context model update module 704 is used to calculate the learning rate of the spatio-temporal context model according to the occlusion rate of the moving target, and update the spatio-temporal context model of the moving target according to the learning rate.

The context prior model updating module 706 is configured to acquire the image feature value of the moving object in the current frame, and update the context prior model of the moving object according to the image feature value.

The tracking module 708 is used to perform convolution operations on the updated spatio-temporal context model and the context prior model to obtain the tracking position of the moving target in the next frame of the video under the first viewing angle.

As shown in Figure 8, in one embodiment, the spatio-temporal context model updating module 704 includes:

The intersection detection sub-module 7042 is used to detect whether an intersection is included between the tracking frames of different moving objects in the current frame.

The occlusion area calculation sub-module 7044 is used to calculate the length and width of the overlapping part between the tracking frames of different moving objects when the tracking frames of different moving objects include intersection points, and calculate the occlusion of the moving object according to the length and width shading area.

The occlusion rate calculation sub-module 7046 is used to obtain the pre-stored tracking frame area of the moving object, and calculate the ratio of the occlusion rate of the moving object to the area of the tracking frame.

In one of the embodiments, the learning rate calculation module 702 uses the following formula to calculate the learning rate:

Among them: e is the natural logarithm; ΔS is the occlusion rate of the moving target; k, are constant parameters.

As shown in FIG. 9, in one embodiment, the context prior model update module 706 includes:

Color intensity acquisition sub-module 7062: used to acquire the color intensity of the moving target in the current frame on the red channel, the color intensity on the green channel and the color intensity on the blue channel.

The color intensity weight value selection sub-module 7064 is used to assign corresponding color intensity weight values to the color intensity of the moving target on the red channel, the color intensity on the green channel, and the color intensity on the blue channel.

The image feature value calculation module 7066 is configured to weight and sum the color intensity on each channel to obtain the image feature value of the moving object in the current frame.

As shown in FIG. 10, in one embodiment, the tracking device 700 of a moving object in a video further includes:

The two-dimensional model projection module 710 is used to extract the tracking site sideline area, establish a tracking site overlooking the two-dimensional model, and project the tracking position to the first projection coordinates in the tracking site overlooking the two-dimensional model.

In one embodiment, the device 700 for tracking a moving object in a video is configured to acquire a video under a second viewing angle, and calculate the tracking position of the moving object in the video frame corresponding to the next frame in the video under the second viewing angle. The site looks down on the second projected coordinates in the 2D model. As shown in Figure 10, the tracking device 700 of the moving target in the video also includes:

The occlusion rate comparison module 712 is used to compare the occlusion rate of the moving object in the current frame in the video under the first viewing angle and the moving object in the current frame in the video under the second viewing angle with the preset occlusion rate threshold.

The first target projection coordinate selection module 714 is used for when the occlusion rate of the moving object in the current frame in the video under the first viewing angle and the occlusion rate of the moving object in the current frame in the video under the second viewing angle are both less than or equal to the preset occlusion When the rate threshold is calculated, the target projected coordinates of the moving target in the two-dimensional model of the tracking field overlooked are calculated according to the first projected coordinates and the second projected coordinates.

The second target projection coordinate selection module 716 is used to select the second projected coordinates as the moving target to look down on the two-dimensional model in the tracking field when the occlusion rate of the current frame moving target in the video under the first viewing angle is greater than the preset occlusion rate threshold The target projection coordinates in ; when the occlusion rate of the moving target in the current frame of the video under the second viewing angle is greater than the preset occlusion rate threshold, select the first projected coordinates as the target projection coordinates of the moving target in the two-dimensional model of the tracking site. .

As shown in FIG. 10, in one embodiment, the tracking device 700 of a moving object in a video further includes:

The projected coordinate correction module 718 is configured to correct the first projected coordinates according to the second projected coordinates when the occlusion rate of the moving object in the current frame in the video under the first viewing angle is greater than a preset occlusion rate threshold; and, when the second projected coordinates When the occlusion rate of the moving object in the current frame in the video under the viewing angle is greater than the preset occlusion rate threshold, the second projection coordinates are corrected according to the first projection coordinates.

The technical features of the above-mentioned embodiments can be combined arbitrarily. To make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, should be considered as within the scope of this specification.

The above-mentioned embodiments only express several implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but should not be construed as limiting the scope of the patent for the invention. It should be noted that for those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent for the present invention should be based on the appended claims.

Claims (8)

1. A method for tracking a moving object in a video is characterized by comprising the following steps:

calculating the shielding rate of a moving target of a current frame in a video under a first visual angle;

calculating the learning rate of the space-time context model according to the shielding rate of the moving target, and updating the space-time context model of the moving target according to the learning rate;

Acquiring an image characteristic value of a moving target in a current frame, and updating a context prior model of the moving target according to the image characteristic value;

performing convolution operation on the updated space-time context model and the context prior model to obtain the tracking position of the moving target of the next frame in the video under the first visual angle;

calculating the learning rate using the following formula:

wherein:

e is a natural logarithm;

Delta S is the shielding rate of the moving target;

k、are all constant parameters.

2. The method according to claim 1, wherein the step of calculating the occlusion rate of the moving object of the current frame in the video at the first view comprises:

detecting whether the tracking frames of different moving targets of the current frame comprise an intersection point or not;

When the tracking frames of different moving targets comprise intersection points, calculating the length and the width of an overlapped part between the tracking frames of different moving targets, and calculating the shielding area of the moving target shielded according to the length and the width;

The method comprises the steps of obtaining a pre-stored area of a tracking frame of a moving target, and calculating the shielding rate of the moving target as the ratio of the shielding area to the area of the tracking frame.

3. the method of claim 1, wherein the step of obtaining the image feature value of the moving object in the current frame comprises:

Acquiring the color intensity of a moving object in the current frame on a red channel, the color intensity on a green channel and the color intensity on a blue channel;

Assigning corresponding color intensity weighted values to the color intensity of the moving object on a red channel, the color intensity of the moving object on a green channel and the color intensity of the moving object on a blue channel;

And carrying out weighted summation on the color intensity on each channel to obtain the image characteristic value of the moving object in the current frame.

4. the method of claim 1, further comprising:

and extracting a sideline area of the tracking field, establishing a tracking field overlook two-dimensional model, and projecting the tracking position to a first projection coordinate in the tracking field overlook two-dimensional model.

5. The method of claim 4, further comprising:

Acquiring a video under a second visual angle, and calculating a second projection coordinate of a tracking position of a moving target of a video frame corresponding to a next frame in the video under the second visual angle in the tracking field overlooking two-dimensional model;

Respectively comparing the occlusion rate of the current frame moving target in the video under the first visual angle and the occlusion rate of the current frame moving target in the video under the second visual angle with a preset occlusion rate threshold value;

when the shielding rate of the current frame moving target in the video under the first visual angle and the shielding rate of the current frame moving target in the video under the second visual angle are both smaller than or equal to the preset shielding rate threshold value, calculating target projection coordinates of the moving target in the two-dimensional model of the tracking field overlook according to the first projection coordinates and the second projection coordinates;

When the shielding rate of a current frame moving target in the video under the first visual angle is greater than the preset shielding rate threshold value, selecting the second projection coordinate as a target projection coordinate of the moving target in the tracking field overlooking two-dimensional model; and when the shielding rate of the current frame moving target in the video under the second visual angle is greater than the preset shielding rate threshold value, selecting the first projection coordinate as a target projection coordinate of the moving target in the tracking field overlooking two-dimensional model.

6. The method of claim 5, wherein after the step of selecting the second projection coordinate as the target projection coordinate of the moving target in the two-dimensional model of the tracking field overhead view, the method further comprises:

Correcting the first projection coordinate according to the second projection coordinate;

After the step of selecting the first projection coordinate as the target projection coordinate of the moving target in the tracking field overlooking two-dimensional model, the method further comprises the following steps:

And correcting the second projection coordinate according to the first projection coordinate.

7. An apparatus for tracking a moving object in a video, comprising:

the occlusion rate calculation module is used for calculating the occlusion rate of a moving target of a current frame in the video under the first visual angle;

The space-time context model updating module is used for calculating the learning rate of the space-time context model according to the shielding rate of the moving target and updating the space-time context model of the moving target according to the learning rate;

the context prior model updating module is used for acquiring the image characteristic value of the moving target in the current frame and updating the context prior model of the moving target according to the image characteristic value;

The tracking module is used for carrying out convolution operation on the updated space-time context model and the context prior model to obtain the tracking position of the moving target of the next frame in the video under the first visual angle;

The learning rate calculation module calculates the learning rate using the following formula:

Wherein:

e is a natural logarithm;

delta S is the shielding rate of the moving target;

k、are all constant parameters.

8. the apparatus of claim 7, wherein the spatiotemporal context model update module comprises:

The intersection point detection submodule is used for detecting whether intersection points are included among tracking frames of different moving targets of the current frame;

The occlusion area calculation submodule is used for calculating the length and the width of an overlapped part between the tracking frames of different moving targets when the tracking frames of different moving targets comprise intersection points, and calculating the occlusion area of the moving target for occlusion according to the length and the width;

And the shielding rate calculation submodule is used for acquiring the pre-stored tracking frame area of the moving target and calculating the shielding rate of the moving target as the ratio of the shielding area to the tracking frame area.