KR102399974B1 - Method of tracking multiple objects and apparatus for the same - Google Patents

Abstract

An embodiment of the present invention, the object detection step for detecting a plurality of objects; generating, by a tracker generating unit, a tracker for tracking a target object that is any one of the objects; The tracker learning unit sets the target object as a positive sample for the tracker, sets any one or more objects different from the target object among the objects as a negative sample, and uses the samples to set the target object as a negative sample. training the tracker; and tracking the target object by an object tracking unit using the tracker. Discloses a multi-object tracking method comprising: a.

Description

Multi-object tracking method and device for same

Embodiments of the present invention relate to a multi-object tracking method and an apparatus therefor, and more particularly, a multi-object tracking method for tracking multiple objects by learning a tracker using a positive sample and a negative sample, and It relates to a device for this.

The technology to track people and objects is a representative application in the field of computer vision, and is a core technology required for a surveillance system using a camera that is emerging these days.

In recent years, when a user directly checks images one by one in a situation where the demand for a monitoring system increases, much time and effort are required. By using a tracker that automatically tracks the target in such a surveillance system, the above effort can be saved.

Korean Patent Publication No. 10-2010-0026123

Embodiments of the present invention set the object tracked by a specific tracker as a positive sample, and train all trackers in a manner that sets the objects tracked by the other trackers as negative samples. Provided are a multi-object tracking method and apparatus for stably discriminating and tracking each object even in a complex situation where they overlap each other.

An embodiment of the present invention, the object detection step for detecting a plurality of objects; generating, by a tracker generating unit, a tracker for tracking a target object that is any one of the objects; The tracker learning unit sets the target object as a positive sample for the tracker, sets any one or more objects different from the target object among the objects as a negative sample, and uses the samples to set the target object as a negative sample. training the tracker; and tracking the target object by an object tracking unit using the tracker. Discloses a multi-object tracking method comprising: a.

In the step of generating the tracker, the tracker generating unit further generates an additional tracker for tracking an additional target object that is different from the target object among the objects, and the step of learning the tracker includes the tracker learning unit in the tracker For, set the additional target object as a negative sample, for the additional tracker, set the additional target object as a positive sample, and set the target object as a negative sample, so that the tracker and the additional tracker can mutually learn there is.

In the training of the tracker, the tracker learning unit determines a weight of each of the classifiers by calculating a log likelihood value of each of the classifiers included in the tracker or each of any two or more combinations of the classifiers. By doing so, the tracker can be trained.

Another embodiment of the present invention, an object detection unit for detecting a plurality of objects; a tracker generating unit that generates a tracker for tracking a target object that is any one of the objects; For the tracker, set the target object as a positive sample, set any one or more objects different from the target object among the objects as a negative sample, and learn the tracker using the samples Letting Tracker Learning Department; and an object tracking unit that tracks the target object using the tracker.

Other aspects, features and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

By the multi-object tracking method and apparatus therefor according to embodiments of the present invention, an object tracked by a specific tracker is set as a positive sample, and objects tracked by the other trackers are set as negative samples By learning all the trackers in this way, even in a complex situation where all objects in the image overlap each other, it is possible to discriminate and track each object more stably.

1 is a diagram showing the configuration of a multi-object tracking apparatus according to an embodiment of the present invention.
2 is a flowchart illustrating a multi-object tracking method according to an embodiment of the present invention.
3 is a diagram illustrating a method for setting and learning a tracker according to an embodiment of the present invention.
4 is a diagram illustrating a method of calculating an overlap ratio between objects according to an embodiment of the present invention.
5 is a diagram illustrating an operation process of a tracker according to an embodiment of the present invention.

Since the present invention can apply various transformations and can have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

In the following embodiments, terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from another.

Terms used in the following examples are used only to describe specific examples, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the following embodiments, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification is present, but one or more It is to be understood that this does not preclude the possibility of addition or presence of other features or numbers, steps, operations, components, parts, or combinations thereof.

Embodiments of the present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented in any number of hardware and/or software configurations that perform specific functions. For example, embodiments of the present invention may be implemented directly, such as memory, processing, logic, look-up table, etc., capable of executing various functions by control of one or more microprocessors or other control devices. Circuit configurations may be employed. Similar to how components of an embodiment of the invention may be implemented as software programming or software elements, embodiments of the invention may include various algorithms implemented in combinations of data structures, processes, routines, or other programming constructs. , C, C++, Java, assembler, etc. may be implemented in a programming or scripting language. Functional aspects may be implemented in an algorithm running on one or more processors. In addition, embodiments of the present invention may employ conventional techniques for electronic environment setting, signal processing, and/or data processing, and the like. Terms such as mechanism, element, means, and configuration may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in connection with a processor or the like.

In addition, in this specification and drawing, about the component which has substantially the same structure, duplicate description is abbreviate|omitted by using the same code|symbol.

1 is a diagram showing the configuration of a multi-object tracking apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , a multi-object tracking apparatus according to an embodiment of the present invention includes an object detector 110 , a tracker generator 120 , a tracker learner 130 , and an object tracker 140 .

The object detection unit 110 detects a plurality of objects in the image.

For example, the object detection unit 110 may detect the local motion of the object as a difference image between the current image and the reference image obtained while the camera gazes in one direction. The reference image is captured and determined at the moment the motion detection function for object detection is activated. The current image is an image taken after the reference image is determined. The reference image and the current image are determined by the angle of view of the camera.

The object detector 110 may detect an object by simple comparison of a reference image and a current image generated in the same direction. The method of detecting the local motion of the object by the object detection unit 110 of the present invention is not limited to the method using the above-described image difference, and of course, various known methods may be applied.

In an optional embodiment, the object detection unit 110 may generate metadata that is a set of object detection results. The metadata may include the number of objects detected in the image and coordinates of each object. The coordinates of the object may include upper-left coordinates and lower-right coordinates of the object block.

In an optional embodiment, the object detector 110 may detect objects using an integral channel feature (ICF). The integral channel feature may consist of a plurality of images on which color-related and direction-related integral operations have been performed, and by using the integral channel feature, the sum of feature values for a desired region can be obtained with only four addition or subtraction operations. .

The tracker generator 120 generates a tracker for tracking objects detected by the object detector 110 . The tracker generator 120 may generate a tracker for tracking any one of the objects as a target object.

A tracker may consist of a combination of a plurality of classifiers. The classifier performs a test on the input image and provides a probability that the input image is positive or negative as a result value as a result value. When the specific image is positive, it may mean that the corresponding image corresponds to the target object, and when the specific image is negative, it may mean that the corresponding image does not correspond to the target object.

The classifier may include a plurality of binary tests. For example, the classifier may be a randomized fern including arbitrarily selected binary tests.

A binary test is a test in which the result value is determined as one of two values. The binary test may be a test that compares the sums of values in two arbitrary rectangular regions and determines whether the sum of the specific regions is greater. For example, the binary test may be a binary test such as Haar-like.

A plurality of binary tests may be performed by being connected in a hierarchical structure. Binary tests may be connected in a hierarchical structure in the form of a binary tree. For example, if the first binary test is a root node and the second and third binary tests are child nodes of the first binary test, the first binary test is performed and the second binary test is performed if the result value is the first result value. It may be connected to perform a binary test, and to perform a third binary test in case of a second result value. In each of the binary tests, a result value of a child node or a classifier may be mapped to a result value.

In an optional embodiment, the tracker generating unit 120 may generate an additional tracker for tracking by using any one different from the target object among the objects as the additional target object. Furthermore, the tracker generating unit 120 may generate a plurality of trackers that track each of the objects. For example, when three objects are detected by the object detector 110 , three trackers for tracking each object may be generated.

The tracker learner 130 learns the tracker generated by the tracker generator 120 using a positive sample and a negative sample. The positive sample may be information obtained in a target area tracked by the tracker, and the negative sample may be information obtained in an area other than the target area. For example, the tracker learning unit 130 may learn the tracker using a multiple-Instance Learning boosting (MIL boosting) method.

With respect to the tracker, the tracker learning unit 130 may set a target object as a positive sample, set any one or more objects different from the target object among objects as a negative sample, and train the tracker using the samples.

In an optional embodiment, when the additional tracker is generated by the tracker generating unit 120 , the target object is set as a positive sample for the tracker, the additional target object is set as a negative sample, and for the additional tracker, the additional target object is set. The tracker and the additional tracker can be mutually learned by setting , as a positive sample, and learning by setting the target object as a negative sample. Furthermore, even when three or more trackers are generated, the tracker learning unit 130 sets its own target object as a positive sample for each tracker and sets the target object of another tracker as a negative sample, so that the corresponding tracker can learn from each other.

The tracker learning unit 130 may train the tracker by determining a weight for each of the classifiers included in the tracker. The tracker provides a final probability that the input image is positive, and the final probability may be a sum or average value of values obtained by multiplying each of the positive probabilities by classifiers included in the tracker by a weight. For example, if the weight is a value in the range of 0 to '1/(number of trackers)', the final probability is the sum of values obtained by multiplying each of the positive probabilities by the weight, and if the weight is in the range of 0 to 1, the final probability is It may be an average value of values obtained by multiplying each of the positive probabilities by a weight.

The tracker learner 130 may calculate a log likelihood value for each classifier and determine a weight using the log likelihood value.

For example, the weight of each of the classifiers may be determined to be 0 or 1 according to a log likelihood value. That is, when the weight is 0, the result value of the corresponding classifier is not reflected in the tracker. When the weight is 1, it means that the result value of the corresponding classifier is reflected in the tracker.

More specifically, a method of determining the weight of each of the classifiers will be described below.

First, a log likelihood value is calculated for each of the classifiers, and the weight of the classifier having the largest log likelihood value is determined as 1.

Next, after generating combinations using one classifier whose weight is determined to be 1 and each of the remaining classifiers, a log likelihood value is calculated for each of the combinations, and the remainder used in the combination with the largest log likelihood value The weight of the classifier is determined to be 1.

Next, after generating combinations using the two classifiers whose weight is determined to be 1 and each of the remaining classifiers, a log likelihood value is calculated for each of the combinations, and the remainder used in the combination with the largest log likelihood value The weight of the classifier is determined to be 1. Whenever this process is repeated once, the classifier whose weight is determined to be 1 can be increased by one. That is, by repeating this process, the weight of a preset number of classifiers may be determined to be 1, and the weights of the remaining classifiers may be determined to be 0.

In an optional embodiment, the tracker learning unit 130 may train the tracker using a positive bag and a negative bag. A positive bag is a sample group including at least one positive sample, and a negative bag is a sample group including only a negative sample.

The object tracking unit 140 tracks the object using the tracker learned by the tracker learning unit 130 . According to the learning result of the tracker learning unit 130 , the specific tracker may more stably discriminate and track the specific object even if the specific object overlaps the other objects.

In an optional embodiment, when overlapping between objects occurs, the object tracking unit 140 calculates the overlap ratio by calculating the ratio of the area of the union area and the area of the intersection area for the areas corresponding to the objects. can do.

That is, the multi-object tracking apparatus according to an embodiment of the present invention sets the object tracked by the corresponding tracker as positive samples for a specific tracker, and sets the remaining objects as negative samples. By mutually learning the entire tracker, even in a complex situation where all objects in the image overlap each other, each of the objects can be discriminated and tracked more reliably.

2 is a flowchart illustrating a multi-object tracking method according to an embodiment of the present invention.

Referring to FIG. 2 , the multi-object tracking method according to an embodiment of the present invention includes a step of detecting a plurality of objects by an object detection unit (S210), and a step of generating a tracker for tracking a target object by a tracker generation unit (S220) , the tracker learning unit learning the tracker using the samples (S230), and the object tracking unit tracking the target object (S240).

First, the object detector detects a plurality of objects in the image (S210). In operation S210 , the object detector may detect objects using an integral channel feature.

Next, the tracker generating unit generates a tracker for tracking the objects detected in step S210 (S220). In step S220, the tracker generating unit may generate a plurality of trackers for tracking each of the plurality of objects.

Next, the tracker learning unit learns the tracker generated in step S220 using the positive sample and the negative sample (S230). In step S230, the tracker learning unit sets a target object, which is a target for tracking by the corresponding tracker, as a positive sample for a specific tracker, and sets at least one object different from the target object as a negative sample to learn the tracker. . When a plurality of trackers are generated in step S220, the tracker learning unit sets its own target object as a positive sample for each tracker and sets the target object of another tracker as a negative sample, thereby mutually learning the trackers. can

Finally, the object detection unit tracks the object using the tracker learned in step S230 (S240). When overlapping between objects occurs in step S240, the object detection unit calculates the overlap ratio by calculating the ratio of the area of the union area and the area of the intersection area with respect to the areas corresponding to the objects.

As such, in the multi-object tracking method according to an embodiment of the present invention, for a specific tracker, an object tracked by the corresponding tracker is set as positive samples, and the remaining objects are set as negative samples. By mutually learning the entire tracker with , it is possible to discriminate and track each object more stably even in a complex situation where all objects in the image overlap each other.

3 is a diagram illustrating a method for setting and learning a tracker according to an embodiment of the present invention.

Referring to FIG. 3 , a method for setting a tracker according to an embodiment of the present invention may be described through a process in which trackers 331 , 332 , and 333 are set for each of the objects 311 , 312 , and 313 . .

For example, as shown in FIG. 3A , the object detector recognizes people 301 , 302 , and 303 in the image 310 , and as shown in FIG. 3B , an area including the people 301 , 302 , 303 may be obtained as objects 311 , 312 , and 313 . 3 illustrates an example of recognizing people to obtain an object, the recognition target of the present invention is not limited, and it is of course possible to obtain an object for various recognition targets according to a known recognition algorithm.

Next, as shown in FIG. 3C , the tracker generating unit may generate a tracker for an object to be tracked among all objects 320 . 3 illustrates a case in which trackers 331 , 332 , 333 are generated for all 311 , 312 , 313 of all objects 320 , but some of all objects 320 (eg, 311 , 312 ) ) may create trackers (eg, 331 , 332 ) only for . In this case, the tracker 331 may track the object 311 , the tracker 332 may track the object 312 , and the tracker 333 may track the object 313 as a target object.

In this way, the tracker according to an embodiment of the present invention may be set to track each of the recognized objects in the image.

Referring to FIG. 3 , in the tracker learning method according to an embodiment of the present invention, positive samples 341a , 342a , 343a and negative samples 341b , 342b , 343b for each of the trackers 331 , 332 , 333 ) This can be explained through the setting process.

For example, the tracker learning unit sets the target object 311 to the positive sample 341a for the tracker 331 and sets the target objects 312 and 313 of the other trackers 332 and 333 to the negative sample 341b. can In addition, the tracker learning unit sets the target object 312 as a positive sample 342a for the tracker 332 and sets the target objects 311 and 313 of the other trackers 331 and 333 as a negative sample 342b. can In addition, the tracker learning unit sets the target object 313 to the positive sample 343a for the tracker 333, and sets the target objects 311 and 312 of the other trackers 331 and 332 to the negative sample 343b. can

In this way, the tracker according to an embodiment of the present invention can be mutually learned by setting its own target object as a positive sample for each of the trackers and setting the target object of another tracker as a negative sample.

4 is a diagram illustrating a method of calculating an overlap ratio between objects according to an embodiment of the present invention.

Referring to FIG. 4A , it can be seen that the overlapping between the two objects 311 and 312 of the objects 311 , 312 and 313 has occurred.

As shown in FIG. 4(b) , the object detection unit has the area of the union area A∪B and the area of the intersection area A∩B with respect to the areas A and B corresponding to the objects 311 and 312 . By calculating the ratio of , the overlap ratio can be calculated.

5 is a diagram illustrating an operation process of a tracker according to an embodiment of the present invention.

Referring to FIG. 5 , the tracker according to an embodiment of the present invention provides an image (x) as an input and a probability value (H(x)) that the image (x) is positive as a result.

The tracker includes a plurality of classifiers (h1, hi, hr), and each of the classifiers (h1, hi, hr) performs a plurality of binary tests on the image (x) to obtain a probability value (eg, the image ( It gives a probability value c0 that x) is positive or a probability value c1) that is negative. The tracker multiplies the probability values calculated from the classifiers (h1, hi, hr) by the weights (a1, ai, ar) and finally provides a probability value (H(x)) that the image (x) is positive. .

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and a ROM. , RAM, flash memory, and the like, and hardware devices specially configured to store and execute program instructions. Furthermore, the medium may include an intangible medium implemented in a form that can be transmitted on a network, for example, may be a type of medium that is implemented in the form of software or an application and can be transmitted and distributed through a network.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

The specific implementations described in the present invention are only examples, and do not limit the scope of the present invention in any way. For brevity of the specification, descriptions of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection or connection members of the lines between the components shown in the drawings illustratively represent functional connections and/or physical or circuit connections, and in an actual device, various functional connections, physical connections that are replaceable or additional may be referred to as connections, or circuit connections. In addition, unless there is a specific reference such as “essential” or “importantly”, it may not be a necessary component for the application of the present invention.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the spirit of the present invention is not limited to the scope of the scope of the present invention. will be said to belong to

110: object detection unit 120: tracker generation unit
130: tracker learning unit 140: object tracking unit

Claims (6)

detecting, by an object detection unit, a plurality of objects;
generating, by a tracker generating unit, a tracker for tracking a target object that is any one of the objects;
The tracker learning unit sets the target object as a positive sample for the tracker, sets any one or more objects different from the target object among the objects as a negative sample, and uses the samples to set the target object as a negative sample. training the tracker; and
Including; tracking the target object by an object tracking unit using the tracker;
In the tracker generation step, a plurality of trackers for tracking different target objects are generated by the tracker generator,
In the tracker learning step, each of the plurality of trackers is learned by the tracker learning unit using positive samples that are target objects to be tracked among the objects and negative samples that are objects tracked by other trackers. Multi-object tracking method, characterized in that the trackers of the mutual learning. delete According to claim 1,
The step of learning the tracker is
Learning the tracker by the tracker learning unit calculating a log likelihood value of each of the classifiers included in the tracker or each of any two or more combinations of the classifiers and determining the weight of each of the classifiers Characterized, a multi-object tracking method. an object detection unit detecting a plurality of objects;
a tracker generating unit that generates a tracker for tracking a target object that is any one of the objects;
For the tracker, the target object is set as a positive sample, any one or more objects different from the target object among the objects are set as a negative sample, and the tracker is trained using the samples. Letting Tracker Learning Department; and
Including; an object tracking unit for tracking the target object using the tracker,
The tracker generating unit generates a plurality of trackers that each track different target objects,
The tracker learning unit mutually learns the plurality of trackers so that each of the plurality of trackers is learned by using a positive sample that is a target object to be tracked among the objects and negative samples that are objects tracked by other trackers. , which is a multi-object tracking device. delete 5. The method of claim 4,
The tracker learning unit
Characterized in that the tracker is trained by calculating a log likelihood value of each of the classifiers included in the tracker or each of combinations of any two or more of the classifiers and determining the weight of each of the classifiers, Multi-object tracking device.

Images