KR102335755B1 - Method for object tracking using extended control of search window and object tracking system thereof - Google Patents

Abstract

The present invention relates to an object tracking technology through extended control of a search window, and a method for an object tracking system to track an object receives an image, sets a search window for a target object to be tracked, and sets a search window set inside the search window. Divide the target object into upper and lower body, extract corner feature points, respectively, compare the pixel and corner feature points of the color histogram extracted from the upper and lower body to determine the final feature point of the target object, The center of color density is traced while adjusting the size of the search window according to the size change, and the ellipse is centered on the inlier that excludes the outlier corresponding to false information using the final feature point of the target object being tracked. Control the expansion of the search window by fitting the model.

Description

An object tracking method through extended control of a search window and an object tracking system according to the method

The present invention relates to a technology for tracking a moving object in an image, and in particular, a method and method for continuously tracking a target object even when overlapping objects occur in a situation in which a plurality of objects and a background are mixed in a moving image to a computer program for executing on a computer, and an object tracking system according to the method.

As digital images become more common, interest in video analysis is increasing, and the need for tracking research on specific moving objects in video is also increasing. Such video analysis can be used in various fields such as continuous tracking for analyzing the behavior of a specific player in an athletic game through a broadcast camera, or tracking a specific object such as a criminal through CCTV (Closed Circuit Television) or security. Conventionally, tracking and analysis of a specific object has been done using a DVR (Digital Video Recorder) through an image recorded after an athletic event or after a crime has occurred. There is a report that if a person continuously tracks specific objects such as athletes and criminals through recorded images, the monitoring efficiency decreases as time goes by. In order to solve this problem, it is necessary to develop an intelligent camera and CCTV that can track specific objects such as athletes and criminals by the camera or CCTV, and determine that the target object is different from other objects and perform continuous tracking.

In general, as mentioned above, intelligent cameras and intelligent CCTVs require a process of detecting and recognizing a moving object to track a specific object by themselves. It can be continuously tracked. In order to detect a moving object, a difference image technique is often used. The difference image technique is a method of detecting a moving object by separating the background and foreground from an input image. One of the widely used differential imaging techniques is Gaussian Mixture Models (GMM). GMM is a method that learns and adapts to changes in the background, and is robust against changes in ambient noise such as light. However, it is not suitable for use in sports games and dynamic camera environments because the background continuously changes and learning cannot be performed properly.

J. H. Lee, S. W. Cho, J. M. Kim, and S. T. Chung, "Layered object detection using adaptive Gaussian mixture model in the complex and dynamic environment.," Journal of Korean Institute of Intelligent Systems, Vol. 18, No. 3, pp. 387-391, 2008. J. H. Park, G. S. Lee, N. D. Toan, W. H. Cho, and S. Y. Park, "Moving object detection using clausius entropy and adaptive Gaussian mixture model.," The Institute of Electronics Engineers of Korea- Computer and Information, Vol. 47, No. 1, pp. 22-29, 2010.

The technical problem to be solved by the present invention is to overcome the limitation that the conventional technique of tracking a moving object in an image is difficult to learn accurately in an environment in which a background and a large number of objects are mixed, and due to an excessive amount of computation in a dynamic camera image Resolves the weakness of real-time tracking delay or load on the system. When a background or another object with a similar color to the target object approaches the target object, the object tracking does not work correctly and the search window expands to areas other than the object. We want to solve the problem that

In order to solve the above technical problem, in an object tracking method according to an embodiment of the present invention, (a) an object tracking system receives an image including at least one object and a background and searches for a target object to be tracked through a search window setting a (search window); (b) dividing, by the object tracking system, the target object set in the search window into an upper body and a lower body and extracting corner feature points; (c) determining, by the object tracking system, a final feature point of the target object by comparing the corner feature points with pixels of a color histogram extracted from the upper body and the lower body, respectively; (d) tracking, by the object tracking system, the center of color density while adaptively adjusting the size of a search window according to a change in the size of the object with respect to the upper body and the lower body; and (e) the search window by fitting an ellipse model around an inlier excluding outliers corresponding to false information using the final feature point of the target object being tracked by the object tracking system. Including; controlling the expansion of.

In the object tracking method according to an embodiment, the step (b) of extracting corner feature points includes: (b1) dividing the target object set in the search window into an upper body and a lower body using the human body proportion model of the target object to do; and (b2) a corner response function to find a point where the sum of squared difference (SSD) according to the movement of the corner detection window for each of the divided regions of the upper body and the lower body is locally maximized. , CRF) to extract the corner feature points; may include.

In addition, in the step (b2) of extracting the corner feature points, a determinant is calculated using a product of a partial differential value in each axial direction and a corner detection window function in a pixel of a two-dimensional image, and the difference between the diagonal sum of the matrix from the calculated determinant A corner response function using ? is calculated for all pixels, and a value of the calculated corner response function equal to or greater than a preset threshold may be selected as a corner feature point.

In the object tracking method according to an embodiment, the step (c) of determining the final feature point includes: (c1) continuously extracting a color histogram from the upper body and the lower body; and (c2) determining the final feature point by comparing the extracted color histogram pixel with the corner feature point, leaving only common coordinates, and removing the selected corner feature point from the background located around the target object.

In the object tracking method according to an embodiment, the step (e) of controlling the expansion of the search window includes: (e1) a color similar to the target object while tracking the density center of the color for each of the upper body and the lower body Continuously detecting whether the search window is extended due to overlap with the background or other objects of the Separating with inliers; and (e3) controlling the expansion of the search window by fitting an ellipse model based on the divided inlier when the search window is to be expanded.

In the object tracking method according to an embodiment, the step (e) of controlling the expansion of the search window includes (e4) setting the final feature point of the target object as a candidate inlier and selecting a preset number of samples from among the candidate inliers. generating an elliptic model by using; (e5) examining whether a preset termination condition or number of repetitions is reached in consideration of the final feature point and the generated elliptic model; and (e6) repeating steps (e4) and (e5) when the end condition or the number of iterations is not reached, and when the end condition or the number of iterations is reached, the search is performed based on the generated ellipse model It may include; controlling the extension of the window. Also, in the step (e6) of controlling the expansion of the search window, an elliptic model having the same or similar angle to the existing search window among the generated elliptic models may be set as a new search window.

On the other hand, below, a computer program stored in a medium for executing the above-described object tracking method in a computer is provided.

In order to solve the above technical problem, an object tracking system according to an embodiment of the present invention includes: an input unit for receiving an image including at least one object and a background; a memory for storing an object tracking program for tracking a target object in an input image; and a processor for driving the object tracking program, wherein the object tracking program stored in the memory sets a search window for a target object to be tracked from an input image, and sets the search window inside the search window. The target object is divided into an upper body and a lower body, corner feature points are extracted, respectively, and pixels of a color histogram extracted from the upper body and the lower body are compared with the corner feature points to compare the target object determines the final feature point of , and tracks the density center of color while adaptively adjusting the size of the search window according to the change in the size of the object for the upper body and the lower body, and using the final feature point of the target object being tracked and a command for controlling the expansion of the search window by fitting an elliptic model around an inlier excluding outliers corresponding to information.

In the object tracking system according to an embodiment, the object tracking program stored in the memory divides the target object set in the search window into an upper body and a lower body using a human body proportion model of the target object, and divides the divided upper body and a corner feature point using a corner response function (CRF) to find a point where the sum of squared difference (SSD) according to the movement of the corner detection window for each region of the lower body is locally maximized. can be extracted.

In addition, in the object tracking system according to an embodiment, the object tracking program stored in the memory calculates a determinant by using a product of a partial differential value in each axial direction and a corner detection window function in a pixel of a two-dimensional image, and the calculated A corner response function using the difference of the diagonal sum of the matrix from the determinant is calculated for all pixels, and a value of the calculated corner response function equal to or greater than a preset threshold may be selected as a corner feature point.

In the object tracking system according to an embodiment, the object tracking program stored in the memory continuously extracts color histograms from the upper body and the lower body, respectively, and compares the pixels of the extracted color histogram with the corner feature points to obtain only common coordinates The final feature point may be determined by removing the selected corner feature point from the background located around the target object.

In the object tracking system according to an embodiment, the object tracking program stored in the memory overlaps with a background or other object of a color similar to the target object while tracking the center of color density for each of the upper body and the lower body It is continuously detected whether the search window is expanded due to overlap, and the final feature point of the target object is divided into an outlier corresponding to false information and an inlier having a consistent data distribution, and the search window is to be expanded. In this case, the extension of the search window may be controlled by fitting an elliptic model based on the divided inlier.

In the object tracking system according to an embodiment, the object tracking program stored in the memory sets the final feature point of the target object as a candidate inlier and generates an ellipse model using a preset number of samples among the candidate inliers, In consideration of the final feature point and the generated elliptic model, it is checked whether a preset end condition or number of iterations is reached, and if the end condition or number of iterations is not reached, the elliptic model creation process and the inspection process are repeated and performed and, when the end condition or the number of iterations is reached, the expansion of the search window may be controlled based on the generated ellipse model.

In addition, in the object tracking system according to an embodiment, the object tracking program stored in the memory sets, as a new search window, an ellipse model having the same or similar angle to an existing search window among the generated elliptic models, You can control the expansion of the navigation window.

In the embodiments of the present invention, when extracting feature points from an input image, unnecessary corner feature points of the background are effectively excluded by separating the upper body and lower body within the search window and determining the main color in the search window as the target object using a color histogram. It is possible to quickly and continuously track the target object in the dynamic image by controlling the expansion of the search window by performing elliptic fitting based on the inlier when tracking the final feature point.

1 is a flowchart illustrating an object tracking method through extended control of a search window according to an embodiment of the present invention.
2 is a diagram illustrating an image in which a search window is set on a target object.
3 is a view for explaining a human body proportion model for dividing an upper body and a lower body.
4 is an exemplary diagram for explaining a process of determining a final feature point from a corner feature point.
5 is a diagram for explaining the expansion of a search window that appears according to overlapping objects.
6 is a flowchart illustrating in more detail a process of controlling the expansion of a search window in the object tracking method of FIG. 1 according to an embodiment of the present invention.
7 is a diagram for explaining a process of setting a new search window from a candidate inlier using an ellipse model.
8 is a flowchart illustrating a system implementation example according to the object tracking method proposed by the embodiments of the present invention.
9 is a block diagram illustrating an object tracking system through extended control of a search window according to an embodiment of the present invention.
10 is a diagram experimentally illustrating a process of controlling the expansion of a search window in the object tracking system proposed by the embodiments of the present invention.
11 is a result of a comparative experiment with respect to the effect of the extended control of the search window.

Prior to describing the embodiments of the present invention, problems occurring in techniques used to track a moving object are overviewed, and technical means adopted by embodiments of the present invention to solve these problems are sequentially described. to be introduced as

Previously, it was pointed out that GMM and SIFT algorithm, which requires a lot of computation, are not suitable for use in dynamic camera images. Therefore, we will look at clues that can compensate for these weaknesses by referring to the CAM-shift algorithm.

The CAM-shift (Continuously Adaptive Mean-shift) algorithm, like the Mean-shift algorithm, tracks the center of color density. If the size of the object is reduced or expanded due to the invariance of the scale of the navigation window, which is a disadvantage of Mean-shift, the ratio of the color of the tracking target changes, so the problem of tracking a background or other moving object other than the target to be tracked is eliminated. To improve, the CAM-shift algorithm was developed. The CAM-shift algorithm is an improved algorithm that enables continuous tracking by adjusting the size of the search window according to the change in the size of the object being tracked.

If an object to be tracked in the image is designated as a search window and the pixel position probability value of x,y within the search window is I(x,y), the initial moment (zeroth moment) can be obtained through Equation 1 below. .

_{M 00} in Equation 1 is the initial moment. Then, the first moment of x,y can be found by Equation (2).

The position of the center x _c , y _c of the search window can be obtained through Equation (3).

The output result of Mean-shift is used as the input result of CAM-shift. The second moment can be obtained using Equation 4 below.

In order to obtain the width (l) and length (h) of the search window, the components of the width and length expressions are calculated using the first moment, the _{center of the search window x c ,y c} values, and the second moment in Equations 5, 6, 7 can be obtained through

Equations for calculating the width and length of the search window using Equations 5, 6, and 7 are the same as Equations 8 and 9.

In Equation 8, l represents a width, and h in Equation 9 represents a length. Therefore, the width and length of the search window can be obtained using Equations 8 and 9.

However, the problem of this conventional CAM-shift is that the search window is expanded when multiple colors are inside the search window when overlapping with a background or other moving object of a similar color occurs due to tracking the color density center or when specifying the search window. Therefore, a problem was found that the object to be tracked could not be accurately tracked.

Accordingly, the embodiments of the present invention described below are devised in recognition of such a problem, and it is intended to propose a technology for setting a search window that can achieve robust object tracking through extended control of a search window for tracking a moving target object. . In particular, it is intended to provide a technical means for appropriately controlling the expansion of the search window when overlapping with an object or background occurs.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, detailed descriptions of well-known functions or configurations that may obscure the gist of the present invention in the following description and accompanying drawings will be omitted. In addition, throughout the specification, 'including' a certain component does not exclude other components unless otherwise stated, but means that other components may be further included.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or a combination thereof exists, but one or more other features or numbers , it is to be understood that it does not preclude the possibility of the existence or addition of steps, operations, components, parts, or combinations thereof.

Unless specifically defined otherwise, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as meanings consistent with the context of the related art, and unless explicitly defined in the present application, they are not to be interpreted in an ideal or excessively formal meaning. .

1 is a flowchart illustrating an object tracking method through extended control of a search window according to an embodiment of the present invention. In terms of implementation, a series of processes described below may be implemented as an object tracking program including instructions for performing these operations, loaded into a memory and driven in an object tracking system including at least one processor can be

In step S110, the object tracking system receives an image including at least one or more objects and a background and sets a search window for a target object to be tracked. To this end, the user can select the object to be tracked by directly designating a specific object to be tracked from the input RGB (Red-Green-Blue) image as a search window, and geometrical or color characteristics appearing in the specific object can be checked. It may be used to designate the target object. If the target object is a human, it is preferable to set the search window except for the head.

If the color of the object to be tracked is multi-colored, which is a problem with CAM-shift, it is difficult to continuously track, and CAM-shift is input to control the expansion of the search window due to overlapping with backgrounds of similar colors and other moving objects. It goes through a pre-processing step instead of directly applying it to the image. First, in order to designate an object to be tracked in an input image, an object to be tracked is designated.

Referring to FIG. 2 illustrating an image in which a search window is set in a target object, (A) is an input image original, and (B) is a result in which the search window is specified. In (B) of FIG. 2, the human body can be further subdivided and recognized as different objects. This will be described through the following step S120.

In step S120, the object tracking system divides the target object set in the search window into an upper body and a lower body, and extracts corner feature points, respectively. After the search window is designated, the inside of the search window may be divided into upper and lower body by using the human body ratio model in case the color of the upper body and lower body of the object to be tracked is different. That is, in the search window selected to track multiple colors, upper and lower body parts are divided through the human body proportion model, and corner feature points of the upper and lower body can be extracted.

3 is a diagram for explaining a human body proportion model for dividing the upper body and lower body, and when a person's height is H, it can be divided into 7 to 8 equal parts based on the length of the head. In the embodiments of the present invention, it is assumed that the human body is divided into seven equal parts, and the upper body and lower body are divided. The upper body can be assigned 2/7 of the total height and the lower body can be assigned 4/7. The equation for dividing the upper body and the lower body can be obtained using l of Equation 8 described above, and the equation for dividing the upper body and lower body is the same as Equation 10.

In Equation 10, Upper means the upper body, and Lower means the lower body. Since the search window does not include the head, the entire length l of the search window is divided into 6 equal parts, and the upper body can be obtained using the human body proportion model. For the lower body, only the lower body is obtained by subtracting the Upper value, which is the upper body, from the total length l of the search window.

In summary, the target object set inside the search window is divided into the upper body and lower body using the human body proportion model of the target object, but the upper body according to the division ratio of the preset human body proportion model for the entire height of the set search window excluding the head The height of the search window corresponding to The upper body and lower body can be divided accordingly.

Now, in order to find a point where the sum of squared difference (SSD) according to the movement of the corner detection window for each of the divided regions of the upper body and the lower body is locally maximized, a corner response function (CRF) ) to extract corner feature points.

In order to separate the background and the tracking object in the navigation window that divides the upper and lower body, the characteristic point of the tracking object is needed. This feature point is used to control the extension of the search window. Feature point extraction may utilize, for example, a Harris Corner algorithm. First, Harris Corner feature points are extracted from the upper body and lower body respectively. Equations for extracting the Harris Corner feature point are the same as in Equations 11 and 12.

In Equation 11, I _x ,I _y is a partial differential value in the x-direction and y-direction in the image pixel (x,y), ω(x,y) is a window function, and a rectangular kernel and a Gaussian kernel are used. If the Eigen Value of the determinant M is λ ₁ , λ ₂ , the corner detection equation is the same as Equation 12.

값이 매우 작은 값이면 평면을 의미한다. 본 발명의 실시예에서는 R > 0 값인 코너를 추출하였다.Here, det(M) = λ ₁ λ ₂ , and trace(M) = λ ₁ + λ ₂ . The resulting value of the formula in R is R > 0: corner, R < 0: edge,

A very small value means flat. In an embodiment of the present invention, a corner with a value of R > 0 was extracted.

In summary, a determinant is calculated using the product of a corner detection window function and a partial differential value in each axial direction in a pixel of a 2D image, and a corner response function using the difference of the diagonal sum of the matrix from the calculated determinant is calculated for all pixels and a value of the calculated corner response function greater than or equal to a preset threshold may be selected as a corner feature point.

However, when a corner feature point is extracted, not only the feature point of the object to be tracked but also the corner point of the background around the tracking object are extracted, so that the reliability as a feature point is lowered. In the embodiments of the present invention, technical means to supplement this are provided through the following step S130.

In step S130, the object tracking system determines the final feature point of the target object by comparing the corner feature points with pixels of color histograms extracted from the upper body and the lower body, respectively. That is, the color histograms of the upper and lower bodies were continuously extracted, and the corner points and pixels of the color histogram were compared to determine the final feature points of the upper and lower body of the tracking object to be tracked.

4 is an exemplary view for explaining the process of determining the final feature point from the corner feature point, and shows the final feature point determination of the tracking object through pixel comparison of the corner feature point and the color histogram.

Figure 4 (A) shows the result of extracting the corner point inside the upper body search window, and Figure 4 (B) shows the result of extracting the corner point inside the lower body search window. Now, (C) and (D) of Figs. 4 show the results of color histograms extracted from the upper body and the lower body and the result of the feature points finally obtained by comparing the pixels of the corner feature points in blue and green. As described above, it can be confirmed that inappropriate corner points located in the background beyond the object regions of the upper and lower body have been removed.

In summary, by continuously extracting each color histogram from the upper and lower body, comparing the pixels and corner feature points of the extracted color histogram, leaving only common coordinates and removing the selected corner feature points from the background located around the target object, it can be determined as the final feature point. have.

Returning to FIG. 1 , in step S140 , the object tracking system tracks the center of color density while adaptively adjusting the size of the search window according to the change in the size of the object with respect to the upper body and the lower body, respectively. From an implementation point of view, for each of the upper and lower body, for example, a CAM-shift algorithm can be used to find and track the color density centroid.

However, as pointed out earlier, in the CAM-shift algorithm, due to the nature of tracking the center of color density, if the background and other moving objects of a similar color to the object to be tracked are close to each other, the search window is unintentionally extended and the object to be tracked. A problem was found that made it difficult to track accurately. To prevent this problem, extended control of the search window is required.

First, in the object tracking method proposed by the embodiments of the present invention, when the search windows of the upper body and the lower body of the object overlap each other, it is determined that the search window is extended.

5 is a diagram for explaining the expansion of a search window that appears according to overlapping objects. 5(A) shows that the search window expansion did not occur because the search windows of the upper and lower body did not overlap each other because the search window of the upper and lower body did not expand because the object of a similar color was overlapped with the tracking object. judged. In contrast, in (B) of FIG. 5 , the search window of the upper body was expanded due to overlap between objects of similar color, and overlapped with the search window of the lower body, so it was determined that the search window was extended.

Accordingly, in the embodiments of the present invention, another technical means for controlling the search window extension through the following step S150 is proposed.

In step S150, the object tracking system uses the final feature point of the target object being tracked to fit an elliptic model around an inlier excluding outliers corresponding to false information, thereby making the search window control the expansion of To control the expansion of the search window when it overlaps with other similar colored backgrounds and moving objects, for example, a model prediction algorithm is applied to create a new search window using the final feature points for the tracking object as input data. By doing so, it can be implemented to continuously control the expansion of the search window and continuously track only the tracking object.

However, while accuracy is important in the model prediction algorithm, noise, that is, outliers, is included in the original data (final feature points) measured through the embodiments of the present invention. This false information prevents the model from predicting accurately. In order to remove such false information, an algorithm strong in the classification of inliers and outliers, for example, RANdom SAmple Consensus (RANSAC) may be used. A typical model prediction algorithm uses a lot of data to increase the model prediction rate. However, the RANSAC algorithm is an algorithm that predicts the optimal model by repeating the process of randomly selecting a part from the given original data while using the minimum initial data and expanding the set of consistent data. .

The final feature points obtained from the upper and lower body search windows of the tracking object are used as RANSAC data. A new search window is created by classifying inliers and outliers and continuously performing ellipse fitting around the inliers. Through this, it is possible to continuously track an object to be tracked by controlling the expansion of the search window in a situation where an overlap occurs between objects or between an object and a background.

In order to control the search window through the RANSAC algorithm, an ellipse must first be modeled. Ellipse modeling may be performed through Equation (13).

Equation 13 is an elliptic equation for modeling an ellipse, and is an expression created by substituting _{n data (x 1} ,y ₁ ), (x ₂ ,y ₂ ), ..., (x _n ,y n ). When expressed as a matrix, it is shown in Equation 14. For n data, the feature points of the finally extracted object were used.

Equation 14 is a matrix expressing an expression created by substituting n pieces of data (x ₁ ,y ₁ ), (x ₂ ,y ₂ ), ..., (x _n ,y _{n ).} In the embodiments of the present invention, although the minimum number of data is designated as 6, it is of course not limited thereto. Equation 14 is expressed as Ax = 0, and solutions other than x = 0 can be obtained by decomposing the SVD (singular value decomposition). SVD decomposes a matrix into a diagonal matrix D and two orthogonal matrices U and V as shown in Equation 15 below.

In Equation 15, the solution becomes the last column vector of V. And the formula for finding the inlier and the optimal model by randomly selecting n pieces of data is the same as Equation 16.

In Equation 16, p is the probability of extracting only the inlier at least once among n sampling times, and in the embodiments of the present invention, p is set to 90%. m is the number of samples to be drawn at one time, and α is the ratio of inliers in the input data. The higher the number of n times, the higher the probability, but the number of times n cannot be increased indefinitely. The expression for obtaining the optimal number of n is as shown in Equation 17.

By continuously applying the new search window of CAM-shift by setting the search window based on the elliptic modeling and inliers obtained in the same way as the above equations, the search window does not expand when an overlap occurs, but by controlling the expansion to continuously only the target object can be tracked

In summary, while tracing the color density center for each of the upper and lower body, it is continuously detected whether the search window is expanded due to a background of a color similar to the target object or overlap with another object, and the target The final feature point of the object is divided into an outlier corresponding to false information and an inlier with a consistent data distribution. can be controlled

6 is a flowchart illustrating in more detail a process (step S150) of controlling the expansion of a search window in the object tracking method of FIG. 1 according to an embodiment of the present invention, and a time-series operation process is presented in terms of implementation. .

In step S151, the final feature point of the target object is set as a candidate inlier, and an ellipse model is generated using a preset number of samples among the candidate inliers. To this end, by comparing the main color histogram of the tracking object with the feature point of the Harris corner, the common coordinates can be judged as the final feature point and used as data for RANSAC Ellipse Fitting. In order to designate the final feature point as a candidate inlier and extract the final inlier, the candidate inlier is modeled as an ellipse using Equation 13 previously presented.

Then, in step S152, it is checked whether a preset termination condition or the number of repetitions is reached in consideration of the final feature point and the generated elliptic model. If the end condition or the number of iterations is not reached, the process returns to step S151 and repeats the generation of the ellipse model and the inspection of step S152. In this case, the number of repetitions may be determined in consideration of the probability of extracting only the inlier during sampling, the number of samples to be selected at one time, and the ratio of the inlier.

On the other hand, when the end condition or the number of iterations is reached, the process proceeds to step S153 to control the expansion of the search window based on the generated ellipse model. In this process, inappropriate expansion of the search window can be suppressed by setting an elliptic model having the same or similar angle to the existing search window among the previously generated elliptic models as a new search window. That is, a candidate inlier that is the same (or similar within a critical range) of an ellipse modeled based on the candidate inlier and the angle of the existing search window is extracted as the final inlier, and ellipse fitting is performed based on the extracted final inlier. Set up a new navigation window.

7 is a diagram for explaining a process of setting a new search window from a candidate inlier using an ellipse model. 7A shows the angle of the search window of the conventional CAM-shift with a purple arc. 7(B) shows the designation of the final feature point as an inlier candidate to set up a new search window as a green dot, and FIG. 7(C) shows an ellipse modeled in advance among the candidate inliers and the existing CAM-shift By comparing the angles of the search window of , the ellipse with the same angle is extracted as the final inlier. In (C) of FIG. 7 , the final inlier is indicated by a pink dot, and the result of creating a new search window is indicated in yellow.

8 is a flowchart illustrating a system implementation example according to the object tracking method proposed by the embodiments of the present invention.

First, an image is received, a search window is designated for the tracking object through block 810, and upper/lower body separation is performed through the human body model ratio for the search window. The separated upper body and lower body are tracked in a similar manner through the 820A and 820B blocks, respectively.

Now, by extracting corner feature points from the separated upper and lower body, and determining whether the extracted feature points match using a color histogram, the final feature point is determined. An algorithm (eg, CAM-shift) 825A and 825B for tracking the color density center is performed for the final feature point determined in this way. If the search window is to be extended, a control algorithm (eg, RANSAC) 830 that suppresses the extension of the search window with respect to the final feature points may be performed. Through the above series of processes, it is possible to track the target object quickly and accurately even in a situation where a large number of objects and backgrounds are mixed through a relatively small amount of computation, and it is possible to prevent the problem of unnecessary expansion of the search window due to the similarity of colors. can

9 is a block diagram illustrating an object tracking system 10 through extended control of a search window according to an embodiment of the present invention, wherein the configuration performed in each step of FIG. 1 is reconstructed from a hardware point of view. Therefore, in order to avoid duplication of description, only an outline thereof is briefly summarized herein.

The input unit 11 receives an image including at least one object and a background acquired through an image acquisition means such as the camera 20 . The memory 13 stores an object tracking program for tracking a target object in an input image, and the processor 15 drives the object tracking program.

Here, the object tracking program stored in the memory 13 sets a search window for a target object to be tracked from an input image, and divides the target object set in the search window into an upper body and a lower body and extracting corner feature points, comparing the corner feature points with pixels of color histograms extracted from the upper body and the lower body, respectively, to determine the final feature point of the target object, and the upper body and tracking the center of color density while adaptively adjusting the size of the search window according to the size change of the object for the lower body, and using the final feature point of the target object being tracked as an outlier corresponding to false information. and a command for controlling the expansion of the search window by fitting an elliptic model around an inlier excluding .

The object tracking program stored in the memory 13 divides the target object set in the search window into an upper body and a lower body using the human body proportion model of the target object, and for each divided area of the upper body and the lower body Corner feature points may be extracted using a corner response function (CRF) to find a point where the sum of squared difference (SSD) according to the movement of the corner detection window is locally maximized.

The object tracking program stored in the memory 13 determines the height of the search window corresponding to the upper body according to the division ratio of the human body proportion model preset with respect to the total height of the set search window except for the head, and determines the height of the search window. The height of the search window corresponding to the lower body may be determined by subtracting the height of the search window corresponding to the upper body from the height, and the upper body and the lower body may be divided according to the determined heights of the upper body and the lower body.

The object tracking program stored in the memory 13 calculates a determinant by using the product of a corner detection window function and a partial differential value in each axial direction in a pixel of a two-dimensional image, and uses the difference between the diagonal sum of the matrix from the calculated determinant. A response function may be calculated for all pixels, and a corner feature point having a calculated value of the corner response function equal to or greater than a preset threshold may be selected.

In addition, the object tracking program stored in the memory 13 continuously extracts color histograms from the upper body and the lower body, respectively, and compares the pixels of the extracted color histogram with the corner feature points, leaving only common coordinates and surrounding the target object. By removing the selected corner feature point from the background, it can be determined as the final feature point.

In addition, the object tracking program stored in the memory 13, while tracking the center of color density for each of the upper body and the lower body, due to the background or overlap with another object of a color similar to the target object, the It is continuously detected whether the search window is extended, and the final feature point of the target object is divided into an outlier corresponding to false information and an inlier in which data distribution is consistent. The expansion of the search window may be controlled by fitting an elliptic model based on the inlier.

More specifically, the object tracking program stored in the memory 13 sets the final feature point of the target object as a candidate inlier and generates an ellipse model using a preset number of samples among the candidate inliers, and the final feature point and In consideration of the generated ellipse model, it is checked whether a preset end condition or number of iterations is reached, and if the end condition or number of iterations is not reached, the ellipse model creation process and the inspection process are repeatedly performed, and the end condition Alternatively, when the number of iterations is reached, the expansion of the search window may be controlled based on the generated elliptic model. In particular, the object tracking program stored in the memory 13 may control the expansion of the search window by setting an elliptic model having the same or similar angle to the existing search window among the generated ellipse models as a new search window. have. Furthermore, it is preferable that the object tracking program stored in the memory 13 determines the number of repetitions in consideration of the probability of extracting only inliers during sampling, the number of samples to be selected at one time, and the ratio of inliers.

Hereinafter, comparison experiments are performed using a prototype implementing the object tracking algorithm proposed by the embodiments of the present invention, and the results are briefly introduced.

10 is a diagram experimentally illustrating a process of controlling the expansion of a search window in the object tracking system proposed by the embodiments of the present invention. 10(A) is a result of performing CAM-shift on the search windows of the upper body and lower body, respectively, in which the object is separated into the human body proportion model. Green represents the upper body, and red represents the lower body. As a result of using only the conventional algorithm (eg, CAM-shift) for tracking the density center of the color, (B) of FIG. 10 shows that the search window of the lower body overlaps with the background of a similar color, so an object other than the tracking object It is an example diagram showing the problem of finding . 10C is an exemplary diagram illustrating the result of appropriately controlling the expansion of the search window despite overlapping with a background of a similar color according to the object tracking method proposed by the embodiments of the present invention.

11 is a comparison experimental result regarding the effect of the extended control of the search window, and compares the size change of the search window according to the conventional CAM-shift algorithm and the search window extension control algorithm proposed by the embodiments of the present invention. was indicated. In both graphs, the x-axis represents the frame, and the y-axis represents the size of the search window.

Referring to (A) of FIG. 11 , it can be seen that the length H and the width W of the search window are continuously extended when similar colors and overlaps occur when only the CAM-shift algorithm is used. In contrast, referring to FIG. 11B , which is an experimental result according to embodiments of the present invention, it can be seen that H and W are appropriately controlled without being expanded.

Table 1 shows the extended control rate by determining that the size of H and W of the initial search window is up to an error rate of 20% as the search window extension control. (A) shows a typical CAM-shift, (B) shows the extended control method proposed by the embodiments of the present invention. When the background of a similar color and another moving object overlap, the control rate of the search window expansion is relatively higher in (B) than in (A), and as a result, it can be seen that the performance is better.

According to the above-described embodiments of the present invention, when extracting feature points from an input image, an unnecessary corner feature point of the background is separated by separating the upper body and lower body in the search window and determining the main color in the search window as the target object using a color histogram. can be effectively excluded, and the target object in the dynamic image can be quickly and continuously tracked by controlling the expansion of the search window by performing ellipse fitting centered on the inlier when tracking the final feature point.

Meanwhile, embodiments of the present invention can be implemented as computer-readable codes on a computer-readable recording medium. The computer-readable recording medium includes all types of recording devices in which data readable by a computer system is stored.

Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like. In addition, the computer-readable recording medium is distributed in a computer system connected to a network, so that the computer-readable code can be stored and executed in a distributed manner. And functional programs, codes, and code segments for implementing the present invention can be easily inferred by programmers in the technical field to which the present invention pertains.

In the above, the present invention has been looked at focusing on various embodiments thereof. Those of ordinary skill in the art to which the present invention pertains will understand that the present invention can be implemented in modified forms without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments are to be considered in an illustrative rather than a restrictive sense. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the scope equivalent thereto should be construed as being included in the present invention.

10: Object tracking system
11: input
13: memory
15: processor
20: camera

Claims (20)

(a) an object tracking system receiving an image including at least one object and a background and setting a search window for a target object to be tracked;
(b) dividing, by the object tracking system, the target object set in the search window into an upper body and a lower body and extracting corner feature points;
(c) determining, by the object tracking system, a final feature point of the target object by comparing the corner feature points with pixels of a color histogram extracted from the upper body and the lower body, respectively;
(d) tracking, by the object tracking system, the center of color density while adaptively adjusting the size of a search window according to a change in the size of the object with respect to the upper body and the lower body; and
(e) of the search window by fitting an ellipse model around an inlier excluding outliers corresponding to false information using the final feature point of the target object being tracked by the object tracking system Including; controlling the expansion;
Step (e) is,
(e1) While tracking the color density center for each of the upper body and the lower body, it is continuously detected whether the search window is expanded due to a background color similar to the target object or overlap with another object Steps to do:
(e2) dividing the final feature point of the target object into an outlier corresponding to false information and an inlier having a consistent data distribution; and
(e3) when the search window is to be expanded, controlling the expansion of the search window by fitting an ellipse model based on the divided inlier. The method of claim 1,
The step (b) is,
(b1) dividing the target object set in the search window into an upper body and a lower body using the body proportion model of the target object; and
(b2) a corner response function to find a point where the sum of squared difference (SSD) according to the movement of the corner detection window for each of the divided regions of the upper body and the lower body is locally maximized; CRF) using the step of extracting corner feature points; including; object tracking method. 3. The method of claim 2,
The step (b1) is,
The height of the search window corresponding to the upper body is determined according to the division ratio of the human body proportion model preset with respect to the total height of the set search window except for the head, and the height of the search window corresponding to the upper body is determined from the total height of the search window. The height of the search window corresponding to the lower body is determined by subtracting , and the upper body and the lower body are divided according to the determined heights of the upper body and the lower body. 3. The method of claim 2,
The step (b2) is,
A determinant is calculated using the product of a corner detection window function and a partial differential value in each axial direction in a pixel of a two-dimensional image, and a corner response function using the difference of the diagonal sum of the matrix from the calculated determinant is calculated for all pixels, and calculated An object tracking method for selecting a corner feature point in which the value of the corner response function is greater than or equal to a preset threshold. The method of claim 1,
Step (c) is,
(c1) continuously extracting color histograms from the upper body and the lower body; and
(c2) determining the final feature point by comparing the extracted color histogram pixel with the corner feature point, leaving only common coordinates, and removing the selected corner feature point from the background located around the target object. delete The method of claim 1,
Step (e) is,
(e4) setting the final feature point of the target object as a candidate inlier and generating an elliptic model using a preset number of samples among the candidate inliers;
(e5) examining whether a preset termination condition or number of repetitions is reached in consideration of the final feature point and the generated elliptic model; and
(e6) If the end condition or the number of iterations is not reached, the steps (e4) and (e5) are repeatedly performed, and when the end condition or the number of iterations is reached, the search window is based on the generated ellipse model Controlling the extension of; Containing, object tracking method. 8. The method of claim 7,
The step (e6) is,
An object tracking method for setting an elliptic model having the same or similar angle as an existing search window among the generated elliptic models as a new search window. 8. The method of claim 7,
Step (e) is,
An object tracking method that determines the number of iterations in consideration of the probability of extracting only inliers during sampling, the number of samples selected at one time, and the ratio of inliers. The method of claim 1,
The step (d) tracks the density center of the color for the upper body and the lower body using a CAM-shift (Continuously Adaptive Mean-shift) algorithm,
In the step (e), the extension of the search window is controlled by classifying an inlier from the final feature point using a RANdom SAmple Consensus (RANSAC) algorithm. A computer program stored in a medium for executing the method of any one of claims 1 to 5 and 7 to 10 in a computer. an input unit for receiving an image including at least one object and a background;
a memory for storing an object tracking program for tracking a target object in an input image; and
Including; a processor for driving the object tracking program;
The object tracking program stored in the memory,
A search window is set for the target object to be tracked from the input image, the target object set in the search window is divided into upper and lower body parts, and corner feature points are extracted, respectively, the upper body and the lower body The final feature point of the target object is determined by comparing pixels of a color histogram extracted from the lower body with the corner feature points, and adaptively according to the size change of the object with respect to the upper body and the lower body, respectively Adjusts the size of the search window with , and tracks the center of color density, and uses the final feature point of the target object being tracked to fit an elliptical model centered on the inlier excluding outliers corresponding to false information. (fitting) comprising a command to control the expansion of the search window,
The object tracking program stored in the memory,
While tracking the color density center for each of the upper body and the lower body, it is continuously detected whether the search window is expanded due to an overlap with a background or another object of a color similar to the target object,
The final feature point of the target object is divided into an outlier corresponding to false information and an inlier having a consistent data distribution,
When the search window is to be expanded, the object tracking system controls the expansion of the search window by fitting an elliptic model based on the divided inlier. 13. The method of claim 12,
The object tracking program stored in the memory,
dividing the target object set in the search window into an upper body and a lower body using the human body proportion model of the target object;
A corner response function (CRF) is used to find a point at which the sum of squared difference (SSD) according to the movement of the corner detection window for each of the divided regions of the upper body and the lower body is locally maximized. An object tracking system that extracts corner feature points using 14. The method of claim 13,
The object tracking program stored in the memory,
The height of the search window corresponding to the upper body is determined according to the division ratio of the human body proportion model preset with respect to the total height of the set search window except for the head, and the height of the search window corresponding to the upper body is determined from the total height of the search window. The height of the search window corresponding to the lower body is determined by subtracting , and the upper body and the lower body are divided according to the determined heights of the upper body and the lower body. 14. The method of claim 13,
The object tracking program stored in the memory,
A determinant is calculated using the product of a corner detection window function and a partial differential value in each axial direction in a pixel of a two-dimensional image, and a corner response function using the difference of the diagonal sum of the matrix from the calculated determinant is calculated for all pixels, and calculated An object tracking system that selects a value of the corner response function that is greater than or equal to a preset threshold as a corner feature point. 13. The method of claim 12,
The object tracking program stored in the memory,
Continuously extracting color histograms from the upper body and the lower body,
An object tracking system that compares the extracted color histogram pixels with the corner feature points, leaves only common coordinates, and determines the final feature point by removing the selected corner feature point from the background located around the target object. delete 13. The method of claim 12,
The object tracking program stored in the memory,
setting the final feature point of the target object as a candidate inlier and generating an ellipse model using a preset number of samples among the candidate inliers;
In consideration of the final feature point and the generated ellipse model, it is checked whether a preset termination condition or number of repetitions has been reached,
When the end condition or the number of iterations is not reached, the elliptic model creation process and the inspection process are repeatedly performed, and when the end condition or the number of iterations is reached, the expansion of the search window is controlled based on the generated elliptic model , an object tracking system. 19. The method of claim 18,
The object tracking program stored in the memory,
An object tracking system for controlling the expansion of the search window by setting an elliptic model having the same or similar angle to an existing search window among the generated elliptic models as a new search window. 19. The method of claim 18,
The object tracking program stored in the memory,
An object tracking system that determines the number of iterations in consideration of the probability of extracting only inliers during sampling, the number of samples selected at one time, and the ratio of inliers.

Images