KR20190023389A - Multi-Class Multi-Object Tracking Method using Changing Point Detection - Google Patents

Abstract

The present invention relates to a multi-class multi-object tracking method capable of tracking a class of an object without limit. According to the present invention, the multi-class multi-object tracking method comprises: a step of calculating an observation likelihood about presence and absence of an object included in an input image sequence; a step of setting a plurality of tracking paths; a step of detecting a change point about a time point in which data of segments on the tracking path is changed by a threshold value or more; and a step of determining final tracking segments.

Description

[0001] The present invention relates to a multi-class multi-object tracking method,

The present invention relates to a multi-object tracking method, and more particularly, to a multi-class multi-object tracking method capable of tracking an object class with an unlimited number of objects using a combination of change point detection technology and object detection based multi object tracking technology.

Multi - object tracking is emerging as a technology applicable to various real world such as image security, gesture recognition, robot vision, and human robot interaction. A challenging problem in tracking multiple objects is the occurrence of trajectory drift due to changes in appearance caused by noise, light changes, pauses, complex backgrounds, interactions, obstructions due to objects, and camera movement. Most multi-object tracking methods suffer from performance degradation due to the number of objects changing per frame, and this is more noticeable in complex backgrounds. In addition, most multi-object tracking methods focus primarily on a limited number of categories, such as pedestrians and car tracking. Multi-object tracking techniques that utilize an object's class indefinitely have been studied very rarely due to very high complexity and high computational demands.

The Bayesian filter consists of a motion model and an observation model for estimating the posterior probability. One of the object tracking methods based on the Bayesian filter is the Markov chain Monte Carlo based method. It can deal with the interaction between multiple objects and the movement of various objects. Most Markov chain Monte Carlo-based methods assume that the number of objects does not change with time, but these assumptions are not suitable for real world applications. The existing Markov chain Monte Carlo-based tracking method has a problem due to one limited assumption. To solve this problem, Monte Carlo, a reversible jump marker, was proposed. Reversible Jump Markov Chain Monte Carlo can deal with motion changes such as updating, replacing, creating, and destroying to accommodate changes in the number of objects over time. The Reversible Jump Markov Chain Monte Carlo ends the current track by initiating a new track by initializing a new object, or by removing an object.

Thus, the multi-object tracking approach based on the Markov chain Monte Carlo has been somewhat successful. Nevertheless, a very high computational complexity is required due to the high dimensional state space. Changes in object appearance, interaction, occlusion due to objects, and the number of changing objects also result in high computational overloads.

For example, in the invention, " Sakaino, H.: Video-based tracking, learning, and recognition methods for multiple moving objects, IEEE Trans. On circuits and systems for video technology 23 (2013) 1661-1674 " The motion model that determines the extinction state is isolated, and the iterative loop of the Markov chain reduces the computational overload of the Markov chain Monte Carlo sampling by only determining the update and replacement states. If the behavior that determines the creation and extinction of an object is determined within the Monte Carlo chain of Markov chains, dimensional changes in the Markov chain Monte Carlo sampling are required. By separating the actions that determine generation and extinction from the iterative loop of the Markov chain, the Markov chain no longer changes in dimension, which makes it possible to reach a stable state with a smaller amount of computation. However, a simple approach to isolate such generation and decay behavior can not efficiently handle complex situations arising from multiple object tracking. In most cases, tracking drift problems arise due to changes in object shape.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made keeping in mind the above problems occurring in the prior art, and it is an object of the present invention to provide a multi- Class multi-object tracking method.

In the present invention, the change point detection model detects a singular point caused by a drift or an obstruction due to an object, or a sudden change based on the spatial state of the track state and the temporal characteristics. A convolution neural network based object detector and a Lucas-Kanade tracker based motion detector were applied to calculate the likelihood for different object classes in the proposed domain. The present invention is based on recently introduced image benchmark data sets, ImageNet VID and MOT benchmark, and confirmed very encouraging results comparable to the latest multi-object tracking techniques.

In accordance with an aspect of the present invention, there is provided a method of tracking a multi-class multiple object for an input video sequence, the method comprising: detecting a presence or absence of an object included in an input image sequence, Calculating an observation likelihood; Determining, based on the computation of likelihood, segments for object presence probability positions and establishing a plurality of trajectories connecting the segments; Detecting a change point with respect to a time point at which the data of the segments on the trajectory change exceeds a threshold value; And performing a forward-backward check on the segments on the trajectory containing the change point, and combining the segments on the trajectory based on the result of the determination to determine the final trace segments.

Wherein the determining step comprises: classifying the segment into a confidence segment and a drifted unstable segment using scoring for detection of the change point; maintaining the confidence segment and excluding the unstable segment; Can be combined.

Wherein the confidence level is used for estimating a confidence level as a forward-backward error for segments on a corresponding trajectory to exclude the retention of the confidence segment and the unstable segment, Can be computed using the Euclidean distance between the trajectory-based segment trajectory and its backward trajectory based on backward tracing.

According to another aspect of the present invention, there is provided a recording medium embodied as computer-readable code for multi-class multi-object tracking of an input video sequence, the method comprising the steps of: ; Determining, based on the computation of likelihood, segments for object presence probability positions and setting a plurality of trajectory trains connecting the segments; Detecting a change point with respect to a time point at which the data of the segments on the trajectory change more than a threshold value; And performing a forward-backward check on the segments on the trajectory containing the change point, and combining the segments on the trajectory based on the result of the determination to determine the final trace segments.

According to the multi-class multi-object tracking method according to the present invention, it is possible to track by changing the likelihood of a proposed region that ideally moves a multi-object unlimited-class which changes in various ways. Unlike existing techniques that estimate only limited types of objects, such as pedestrians and automobiles, an efficient convolution neural network based multi-class object detector can be applied to calculate the likelihood of multiple object classes.

In addition, a change point detection algorithm based on static observations and dynamic observations evaluates the failure of tracking. The drift of multiple object tracking can be investigated by detecting a sudden change point in the time series without change represented by the tracking segment.

BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.
FIG. 1 shows a main concept of a framework for multi-class multi-object tracking according to an embodiment of the present invention.
2 is a flow chart for explaining change point detection according to an embodiment of the present invention.
FIGS. 3A and 3B illustrate examples of detection of change points from segments of the MOT16-02 and MOT16-09 sequences by applying the MCMOT of the present invention.
Figure 4 is an illustration of the result of MCMOT tracing in a verification sequence of an ImageNet VID dataset using the MCMOT of the present invention.
FIG. 5 shows an example of visualizing the tracking result of MCMOT in a test image sequence by applying the MCMOT of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are denoted by the same reference symbols as possible. In addition, detailed descriptions of known functions and / or configurations are omitted. The following description will focus on the parts necessary for understanding the operation according to various embodiments, and a description of elements that may obscure the gist of the description will be omitted. Also, some of the elements of the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore the contents described herein are not limited by the relative sizes or spacings of the components drawn in the respective drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. The following terms are defined in consideration of the functions of the present invention, and may be changed according to the intention or custom of the user, the operator, and the like. Therefore, the definition should be based on the contents throughout this specification. The terms used in the detailed description are intended only to describe embodiments of the invention and should in no way be limiting. Unless specifically stated otherwise, the singular form of a term includes plural forms of meaning. In this description, the expressions " comprising " or " comprising " are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, Should not be construed to preclude the presence or possibility of other features, numbers, steps, operations, elements, portions or combinations thereof.

It is also to be understood that the terms first, second, etc. may be used to describe various components, but the components are not limited by the terms, and the terms may be used to distinguish one component from another .

First, the present invention proposes a robust multi-class multi-object tracking combining an object detection reaction and a change point detection algorithm. Faster R-CNN, and ResNet-based Deep Learning-based object detection technology has made it possible to detect objects that can handle an object's class indefinitely. The ensemble of object detection is realized by Faster R-CNN based object detector and Lucas-Kanade tracking algorithm based motion detector. The method of separating the motion dynamics model of the Bayesian filter is implemented by entity switching and motion movement. Entity transitions are modeled as events of creation and destruction. Deep learning-based algorithms calculate posterior probabilities. One of the most annoying problems of multi-object tracking, the drift problem, is solved through a change point detection algorithm. Assuming the smoothness of the motion dynamics, it can be handled with a rapidly changing point detection algorithm of observation associated with lighting, complex background, pose, scale, and so on. The main contributions of the present invention are as follows.

That is, according to the multi-class multi-object tracking method according to the present invention, it is possible to track by changing the likelihood of the proposed region, which ideally moves the multi-object unlimited-class changing in various ways. Unlike existing techniques that estimate only limited types of objects, such as pedestrians and automobiles, an efficient convolution neural network based multi-class object detector can be applied to calculate the likelihood of multiple object classes. In addition, a change point detection algorithm based on static observations and dynamic observations evaluates the failure of tracking. The drift of multiple object tracking can be investigated by detecting a sudden change point in the time series without change represented by the tracking segment.

<Multi Object Tracking>

Recent multi-object tracking studies are focused on detection-based-tracking theory for performing data clustering that links object detection, present in continuous images. The offline based multi-object tracking method uses the information of the pre-frame to perform better multi-object tracking using hierarchical track clustering, network flow, and global tracking trajectory optimization. However, offline methods require high computational complexity. On the other hand, the on-line method requires only a small amount of computation to use only the past and present frame information to solve a multi-object tracking problem. The on-line method is more suitable for real-world problems, but drifts easily occur due to noise, light intensity, pose, camera angle, shadow, changes in the number of objects, and sudden changes. The number of dynamically changing objects is difficult to handle, especially in crowded or crowded objects. Most multi-object tracking methods rely on observations based on different kinds of features and are susceptible to drift. For these nonstationary and nonlinearities, probability-based tracking methods such as Kalman filters or particle filters are known to dominate deterministic based tracking methods.

<Convolution Neural Network Based Object Detection>

In recent years, the use of convolutional neural networks in the field of computer vision has been remarkable. Particularly noteworthy is the region based Convolutional Neural Networks (R-CNN). In R-CNN, we proposed a method of transforming an object classifier based on convolutional neural network into an object detector based on convolutional neural network through an area-based approach of interest. SPPNet (Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition) and Fast R-CNN extended the R-CNN method to pool the convolution feature vectors from the convolution feature map. More recently, RPN (Region Proposal Networks) has been proposed that can create regions of interest within the R-CNN framework. These domain-based convolutional neural network frameworks have demonstrated superior performance over most previous studies. Despite the great success of convolutional neural networks, only a small number of multi-object tracking algorithms have been proposed by applying convolutional neural network based representation. In the conventional method, a multi-object tracking using a simple object tracking algorithm together with a convolutional neural network-based framework has been verified in the ImageNet VID data set, and a convolution neural network-based object detector has been used in the MOT Challenge.

In the present invention, an observational model is constructed by applying the region-based convolutional neural network paradigm.

<Overview of multi-class multi-object tracking>

The present invention proposes an efficient multi-class multi-object tracking unit MCMOT (Multi-Object Multi-Object Tracking Unit) that can efficiently handle generation, extinction, blindness, interaction, and drift of an object. MCMOT may fail due to posterior probability calculations, interaction models, entity models, and incorrect calculations that may occur in motion models. The purpose of MCMOT is therefore to stop the drift as soon as possible, recover from the false detection, and resume tracking again.

In the multi-class multi-object tracker (MCMOT) according to an exemplary embodiment of the present invention, the components for realizing each function may be realized through hardware such as a semiconductor processor, software such as an application program, Software. &Lt; / RTI &gt;

In particular, the functions used for data processing in the multi-class multi-object tracker (MCMOT) according to an embodiment of the present invention can be implemented as computer-readable codes on a recording medium readable by an apparatus such as a computer , And may be implemented by a combination of such a recording medium and a computer or the like to input, output, and display data or information necessary for performing a function. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, hard disk, and storage medium such as a removable storage device. In addition, the computer-readable recording medium may be distributed in a computer system connected to a network (e.g., the Internet, a mobile communication network, or the like) and may store computer readable codes in a distributed manner and may be executed through a network.

FIG. 1 shows a main concept of a framework for multi-class multi-object tracking according to an embodiment of the present invention. As shown in FIG. 1, a framework for multi-class multi-object tracking in a multi-class multi-object tracker (MCMOT) according to an exemplary embodiment of the present invention includes: (a) a likelihood calculation based on an observation model; Tracking segment creation, (c) change point detection, and (d) trajectory combination. The class of objects can be tracked indefinitely. The drift of the segment is effectively controlled through forward-backward word verification with change point detection.

In the MCMOT of the present invention, the object is represented by the proposed regions traced by the tracking algorithm. The tracking algorithm divides the tracking trajectory into several segments based on likelihood calculation, considering possible interactions or occlusion. Taking into account the decision of the easy-to-follow tracker, we observe within the segment, compare the localized bounding boxes, and monitor sudden changes through change point detection. The motion-based tracking component adaptively applies the Lucas-Kanade tracker to predict the area of the next tracking point from the current tracking point. In the present invention, a multi-class object detector based on a deep feature point is used as a global object detector and a local object detector. At this time, the number of object categories may be expanded according to the capability of the object detector.

Hereinafter, the operation of the multi-class multi-object tracker (MCMOT) according to an embodiment of the present invention will be described in detail.

<Multi-class multi-object tracking>

The MCMOT of the present invention adopts a data-centric approach to deal with object-level events such as object creation and object extinction, inter-object phase interactions and inter-object occlusion, tracking-step event generation, updating, and extinction do. Possible drifts from observation failures are addressed using change point detection based on anomaly detection methods.

Trace segments are defined using generation and extinction detection. It only tracks visible objects and divides the overall tracking trajectory into multiple tracking segments if occlusion occurs. If the object becomes ambiguous due to occlusion or noise, the corresponding trace segment is terminated (associated with object extinction). If the same object reappears near the ending area, the tracker begins tracking again (associated with object creation) and builds the tracking segments successively.

<Observation model>

In the " likelihood calculation based on (a) observation model " of FIG. 1 of the present invention, the observation model (observation likelihood) is defined as follows. The observed likelihood of a tracked object must evaluate both the object class and its exact position. In MCMOT, the image detection of the detectors with different characteristics is ensembled to accurately calculate the observation likelihood for the existence and nonexistence of the objects included in the input image sequence. Since the dimension of the scene state of the image can be changed, the measurement value defines the existence and non-presence of the object id at time t as a ratio of likelihood as in [Equation 1]. Since the likelihood of non-presence can not be measured, the present invention adopts the softmax of the probability model as shown in [Equation 1].

[Equation 1]

는 시간 t에서의 물체가 검출되는 경우를 나타낸다. Where _{id, t} is the presence of an object id at time t, ø _{id, t} is the absence of an object id at time t, λ _e is a weight for the object detector e, Max function.

Indicates a case where an object at time t is detected.

Since each detector has its own advantages and disadvantages, this ensemble approach is expected to be robust to sporadic noise. In the present invention, an object detector ensemble consisting of a global object detector (GT), a color detector (CT), and a motion detector (MT) is used. As a global object detector (GT), a deep feature point based object detector is used to detect objects based on deep feature points. The color detector (CT) calculates the similarity score between the observed appearance model and the reference object over the Bhattacharyya distance using RGB (Red, Green, Blue) or HSV (Hue, Saturation, Value) color histogram of the bounding box. The motion detector MT determines the presence or absence of an object using a Lucas-Kanade tracker-based motion detector that detects whether or not the motion of the scene exists.

<Create Track Segment>

Here, the "(b) generation of trace segments" in FIG. 1 of the present invention will be described. Based on the above likelihood calculation, the tracking algorithm determines the segments of the object existence possibility position by using the judgment about possible interactions or occlusion, and sets a plurality of tracking trajectories for connecting the segments Is applied.

MOCOT is modeled as a tracking problem that determines ideal scene particles from a given image sequence. The MCMOT can be considered to perform successive iterations of reallocating objects from the current scene state to the next scene state. First, generation and destruction allocation are performed in the entity state transition step. Second, build immediate tracking segments under the assumption that the shape and position of the tracking segments change smoothly in the data-driven Markov chain Monte Carlo sampling stage. In the final step, detection of tracking drift is performed through a change point detection algorithm to prevent drift. The change point means the time point at which the characteristic of the data changes abruptly, and it can be expected that the probability of the start point of the drift is high. A more detailed discussion of data-centric Markov chain Monte Carlo sampling and entity state transition is described below.

<Data-based Markov chain Monte Carlo sampling>

의 분포를 다음 시간(t+1)의 파티클을 제안하기 위해 사용한다. MCMOT에서는 위와 같은 논문에서 기술한

수식처럼, 마르코프 체인 몬테 카를로 샘플링을 위한 제안 밀도가 순수한 모션 모델을 따른다고 가정한다. 파티클 또는 물체 상태 x_t의 집합인 장면 파티클이 주어졌을 때, 후보 장면 파티클 x_t'가 랜덤하게 물체 존재 o_id,t를 선택함으로써 결정되고, 물체 존재 o_id,t에 의존하는 결정 상태 x_t'가 균일 확률 분포 가정 아래에서 결정된다. 본 발명에서는, 데이터 중심의 제안 밀도가 논문 "Zhao, T., Nevatia, R., Wu, B.: Segmentation and tracking of multiple humans in crowded environments. TPAMI 30 (2008) 1198-1211"을 참조하여 보다 나은 마르코프 체인 수용률을 위해 채택되었다. MCMOT에서 새로운 물체 상태 o'_id,t가 [수학식2]와 같이 표현된다. In the Markov chain Monte Carlo sampling, the proposed density function is one of the important factors. It has been confirmed that this has a great influence on tracking performance because it has a large influence on building a stable distribution of Markov chains. The proposed density function must be measurable and should be able to be sampled efficiently from the proposed distribution proportional to the desired target distribution. In the present invention, a strategy of 'one object at a time' strategy, that is, a strategy in which one object state is modified is described in the article "Khan, Z., Balch, T., Dellaert, F .: MCMC- K., Matas, J .: Forward-backward error: Automatic detection of tracking failures. In: ICPR. (2010) Quot;). &Lt; / RTI &gt; When a particle x _t in the image is given at time t, the probability density function of x 'for x _t

Is used to suggest particles at the next time (t + 1). In the MCMOT,

As the formula suggests, the proposed density for Markov chain Monte Carlo sampling follows a pure motion model. Given a particle or object conditions, a set of scene particles x _t, the candidate scene particles x _t to "random object is present o _id, is determined by selecting a _t, the object exists o _id, it determined depending on the _t state x _t Is determined under the uniform probability distribution assumption. In the present invention, the proposed density of the data center is referred to as "Zhao, T., Nevatia, R., Wu, B .: Segmentation and tracking of multiple humans in crowded environments. TPAMI 30 (2008) 1198-1211" Better Markov chain was adopted for the acceptance rate. In MCMOT, the new object state o ' _{id, t} is expressed as [Equation 2].

&Quot; (2) &quot;

Here,? ₁ +? ₂ = 1. In Equation (2), the first term is the shape of the motion model, and the second term is the detector ensemble using all detections close to the object id. It is assumed that the posterior probability for time step t-1 is represented by N (natural number) samples (scene particles). When observations are initially given at the current time t, the current likelihood is computed by Monte Carlo sampling of Markov chains using N samples. Where the initial B samples are used as burn-in samples. The B burn-in samples are used initially and then removed to converge to a stable state distribution. More practical considerations of the Markov chain Monte Carlo can be found in the article "Gilks, WR, Richardson, S., Spiegelhalter, DJ: Introducing markov chain monte carlo. Markov chain Monte Carlo in practice 1 (1996)".

&Lt; Determination of entity state transition >

에 따라 물체 id가 위치 (x,y)에 존재할 때, 생성을 긍정하는 상태는 v=1으로, v=0은 생성을 부정하는 상태를 가리킨다. 마찬가지로,

는 소멸 상태를 나타내고, 즉, 소멸을 긍정하는 상태는 v=1으로, v=0은 소멸을 부정하는 상태를 가리킨다. 시간 t에서의 엔터티 상태 사후 확률(posterior probability) P_ES(o_id,t/o_id,t-1)은 [수학식3]과 같이 각 경우에 대해 P_b, P_d, P_a, P_ø로 나타낼 수 있다. Entity-state transitions can be predicted according to the generation state and the extinction state probability, which are two binomial probabilities, according to the time model at time t and at time t-1. Equation

, When the object id exists at the position (x, y), the state that affirms the generation is v = 1, and the state v = 0 denies the generation. Likewise,

Indicates a disappearance state, that is, a state in which vanishes are extinguished is v = 1, and v = 0 indicates a state in which extinction is denied. The posterior probability P _ES (o _{id, t} / o _{id, t-1} ) at time _t is given by P _b , P _d , P _a , P _ø .

&Quot; (3) &quot;

는 1로 설정되며, 그렇지 않을 경우에는 0으로 설정된다. 만약 물체 id가 시간 t의 위치 (x,y)에서 관찰되지 않고 시간 t-1에서 관찰되었을 경우, 물체 id의 소멸 상태

는 1로 설정되며, 그렇지 않을 경우 이는 0으로 설정된다.If a new object id is observed by observation likelihood at position (x, y) at time t and not observed at time t-1,

Is set to 1, and is set to 0 otherwise. If the object id is observed at time t-1 without being observed at the position (x, y) of time t, the disappearance state of the object id

Is set to 1, otherwise it is set to zero.

&Lt; Change point detection >

Reference is now made to Fig. 2 for the purpose of describing the " (c) change point detection " in Fig. 1 of the present invention.

2 is a flow chart for explaining change point detection according to an embodiment of the present invention.

If the tracking trajectory is divided into a plurality of segments based on the likelihood calculation as described above, the MCMOT first detects a change point (S100) for the corresponding tracking segment as shown in FIG. The change point means a time point at which the data of the segments on the trajectory change suddenly beyond the threshold value, which is a high probability of starting the drift of the trajectory.

The score for change point detection (CPD) may be calculated by a change point detection algorithm (S200). If a high change point score is detected for the change point, a forward-backward error (FB error) is checked for the segments on the trajectory including the change point (S300). A forward-backward error (FB error) is the basis for determining whether the segment has drifted. If the score for the change point is low or the forward-backward error (FB error) is low for that change point, the segment is kept as a confident segment (S500) and a forward-backward error Is excluded as a drifted unstable segment (S600). In this way, it is possible to efficiently determine the tracking trajectory according to the final tracking segments by combining the segments on the tracking trajectory through the change point detection together with the forward-backward confirmation of drift probability of the tracking segment. Forward-backward error (FB error) confirmation is described in the paper, "Kalal, Z., Mikolajczyk, K., Matas, J .: Forward-backward error: Can be referenced. Thus, by scoring the segment for detection of the change point, classifying the segment into a confidence segment and a drifted unstable segment, maintaining the confidence segment, excluding the unstable segment, and combining the segments on the corresponding tracking trajectory, Segments can be determined.

MCMOT may fail to track an object if the object is confused by a cluttered background. MCMOT decides whether to terminate tracing or continue tracing. The MCMOT drift problem is investigated by detecting sudden change points in a fixed time series representing track segments. A high response indicates a high uncertainty that is likely to occur in the drift. The tracking drift probability is determined through a change point detection method. Here, the two-stage time series learning algorithm is described in "Takeuchi, JI, Yamanishi, K .: A unifying framework for detecting outliers and change points from time series, IEEE Transactions on Knowledge and Data Engineering 18 (2006) 482-492" Can be similarly applied. To reduce time series outliers, a two-phase time series is constructed using the average response of the first-level time series. Drift can be prevented by the procedure shown in FIG.

)의 연결 궤도를 나타내는 포워드 궤도 τ_t 를 예측할 수 있다. τ_t ^r는

의 역순 연결 상태

로 표현된 백워드 궤도에 해당한다. 랜덤한 추적 경로인 백워드 추적은 올바른 포워드 추적과 유사할 것으로 기대한다. 추적 세그먼트의 확신도는 포워드와 백워드 두 가지 추적 세그먼트 사이의 유클리디안 거리

로 정의된다. 여기서 거리 함수는

와 같이 포워드와 백워드 두 가지 추적 세그먼트에 기초한 유효한 추적궤도들(시작점과 끝점 사이)의 유클리디안 거리로써 표현될 수 있다. 예를 들어, 포워드-백워드 에러(FB error)가 임계치를 넘지만 이와 같은 유클리디안 거리(Conf)가 작고 FB error가 소정의 값 이하이면, 확신 세그먼트가 될 수 있다.For example, if a high change point detection response is found in the tracking segment, the backward tracking of the segment to be tracked in the forward-backward error (FB error) ) Can be estimated. The confidence level (Conf) can then be used to determine the confidence segment to be retained and the drifted unstable segment to be excluded. For example, given an image, the segments in the trajectory from 1 to t in the image frame (

Can be predicted from the forward trajectory &lt; RTI ID = 0.0 &gt; _ti &lt; / RTI &gt; τ _t ^r

Reverse Link State

Which corresponds to the backward orbit represented by &lt; RTI ID = 0.0 &gt; Backward tracking, a random tracking path, is expected to be similar to correct forward tracking. The confidence level of the tracking segment is the Euclidean distance between the two tracking segments, forward and backward.

. Here, the distance function

Can be expressed as the Euclidean distance of valid tracking trajectories (between the start and end points) based on the two tracking segments, forward and backward, as shown in FIG. For example, if the forward-backward error (FB error) exceeds the threshold but the Euclidean distance Conf is small and the FB error is below a predetermined value, it may become a confidence segment.

The proposed MCMOT algorithm of the present invention can be summarized as follows.

Hereinafter, a more detailed description of the MCMOT experiment of the present invention will be described, and the performance will be described by comparison with the state of the art methods in a challenging image sequence.

<Experiment setting>

In order to construct a global object detector and a local object detector, the present invention uses a publicly available 16-layered VGG-Net (see the article, Simonyan, K., Zisserman, A .: Very deep convolutional networks for large- recognition. In: arXiv preprint arXiv: 1409.1556. (2014)] and ResNet [He, K., Zhang, X., Ren, S., Sun, J .: Deep residual learning for image recognition. arXiv pre-print arXiv: 1512.03385. (2015) is pre-learned in the ImageNet object classification data set. The initial model is fine-tuned to 280,000 iterations and a learning rate of 0.001 using the ImageNet Challenge Object Detection Data Set (ImageNet DET). After 280,000 iterations, 70,000 iterations are performed with a learning rate of 1/10 lower. In order to generate the proposed region, RPN which can generate the fast and accurate proposed region by the method of sharing the convolution feature [References, Ren, S., He, K., Girshick, R., Sun, J .: Faster R -CNN: Towards real-time object detection with region proposal networks. In: NIPS. (2015)]. After building the initial model, fine-tuning each dataset in a manner similar to that described above, and constructing an adaptive model for each domain. For the change point detection algorithm, a two-stage time series learning algorithm with effective computational complexity and high detection accuracy [see papers, Takeuchi, J. I., Yamanishi, K .: A unifying framework for detecting outliers and change points from time series. IEEE Transactions on Knowledge and Data Engineering 18 (2006) 482-492. The change point is considered when the change point score is higher than the change point score threshold. The change point score threshold is set to 0.3 empirically.

<Experimental Data Set>

Existing multi-object tracking benchmark datasets are only concerned with two or three classes. In the present invention, MCMOT is classified into ImageNet VID having 30 object classes (see papers, Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Berg, AC: Imagenet large scale visual recognition challenge. IJCV 115 (2015) 211-252] and the latest multi-object tracking benchmark, MOT 2016 [see papers, Magee, D. R .: Tracking multiple vehicles using foreground, background and motion models. Image and vision Computing 22 (2004) 143-155] Evaluate performance using benchmark datasets. Performance comparisons of MCMOT with state-of-the-art methods are performed in ImageNet VID and MOT benchmark 2016.

<ImageNet VID>

The algorithm proposed in the present invention is verified through an ImageNet video object detection (VID, object detection from video, video image detection or visual identification detection) data set. ImageNet VID is a dataset used to evaluate object detection performance for image data. Nevertheless, the data set can also be used to evaluate multi-class multi-object tracking. This is because the data set consists of video sequences recorded through a moving camera in the real world, and the number of 30 object categories varies with scene and time flow. Object categories that exist in a scene are photographed at different points in time, and there are obstacles in various stages. For an easy comparison with other state-of-the-art methods, the performance of the MCMOT measured in the dataset is primarily determined by the ImageNet Challenge [Russkovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., ... Berg, AC: Imagenet large scale visual recognition challenge. Measured based on the mean average precision (mAP) used in IJCV 115 (2015) 211-252. The ImageNet VID dataset used in the present invention uses the first published version, which consists of three parts: learning, verification, and testing.

<MOT Benchmark 2016>

The multi-object tracking framework proposed by the present invention can be applied to a multi-object tracking framework such as MOT Benchmark [see papers, Milan, A., Leal-Taixe, L., Reid, I., Roth, S., Schindler, K .: MOT16: A Benchmark for Multi -Object Tracking. arXiv preprint arXiv: 1603.00831. (2016). The MOT benchmark is the latest multi-object tracking benchmark. The MOT benchmark consists of several new challenging sequences and video sequences commonly used in the existing multi-object tracking community. The MOT benchmark 2016 generated data from fixed cameras and moving cameras in a total of 14 constrained environments. All sequences contain only pedestrians, and these challenging sequences are made up of various settings. For example, other settings or other weather conditions are included in these settings. Therefore, certain scenarios or scene-specific tracking algorithms can not perform well. In the present invention, the CLEAR MOT tracking measurement included in the MOT Benchmark Development Kit [see the paper, Bernardin, K., Stiefelhagen, R .: Evaluating multiple object tracking performance: the CLEAR MOT metrics. (2008)] for the performance evaluation.

<MCMOT change point detection analysis>

FIGS. 3A and 3B illustrate examples of detection of change points from segments of the MOT16-02 and MOT16-09 sequences by applying the MCMOT of the present invention. 3A and 3B show graphs of observed likelihood and detected change points for the corresponding image frame. High change point reaction shows high drift probability with high uncertainty. It can be seen from the figure that the method proposed by the present invention can effectively detect drift in a challenging environment.

To verify the proposed MCMOT change point detection component, two sequences MOT16-02 (Fig. 3a) and MOT16-09 (Fig. 3b) were selected from the MOT 2016 training data. For the change point detection, if the change point score is higher than the change point threshold value of 0.3, it is recognized as the change point. As in FIGS. 3A and 3B, it is an important factor for detecting the possibility of changing points of low likelihood or sudden likelihood. As can be seen from the tracking results of the MOT16-02 (FIG. 3A) sequence, the unstable likelihood continues to the image frame # 438 as a motion blurred person moving only the upper body moves. In this case, since the proposed region of the region where the drift occurs is unstable, a strong fluctuation of likelihood can be observed. The change point detection algorithm of the present invention detects this change and indicates a high change point score in the image frame # 440. The MOT16-09 (FIG. 3B) sequence can likewise observe similar phenomena as described above. From these observation results, it can be seen that the change point detection method implicitly processes the possibility of tracking drift.

<ImageNet VID performance evaluation>

Table 1 below shows the impact measurement results of other components in the ImageNet VID verification set.

[Table 1]

Because the official ImageNet Challenge test server is being used primarily for annual competitions and there are a limited number of evaluations, the ImageNet VID performance evaluation in the present invention is practically evaluated in the verification set instead of the test set [ K., Ouyang, W., Li, H., Wang, X .: Object detection from video tubes with convolutional neural networks. In: arXiv preprint arXiv: 1604.04053. (2016). In the ImageNet VID learning / verification experiment, the images used for all learning and testing are scaled so that the short side of the image is 600 pixels in length. The corresponding scaling values were chosen due to the limitations of GPU (Graphical Processing Unit) memory in the fine tuning of VGG16 (dataset classification method) or ResNet.

[Table 1] shows the influence of each component of MCMOT. Each method is denoted by MCMOT, the change point algorithm (MCMOT CPD), and the MCMOT and the change point algorithm with forward-backward word verification (MCMOT CPD FB). Segments with an average observation score of less than 0.3 were filtered and evaluated. As can be seen in Table 1, the MCMOT CPD showed 71.1% of the baseline detection performance, which is a significant improvement of 9.8%. After applying forward-backward detection, the overall detection performance is 74.5% mAP It can be seen from the ImageNet VID verification set.

Table 2 below shows the results of tracking performance comparisons with state-of-the-art algorithms in the ImageNet VID verification set.

[Table 2]

Table 2 summarizes the accuracy of MCMOT, evaluates through 281 validated video sequences, and compares it with the most advanced algorithms. The MCMOT proposed by the present invention achieves a total of 74.5% mAP, and this is achieved by using a state-of-the-art algorithm, T-CNN (Kang, K., Li, H., Yan, J., Zeng, X., Yang, B., Xiao, T., Ouyang, W .: T-cnn: Tubelets with convolutional neural networks for object detection from videos. In: arXiv preprint arXiv: 1604.02532. (2016)] than the other methods. These results are mainly due to the MCMOT approach, which can be used to construct high-accuracy segments through change point detection.

Figure 4 is an illustration of the result of MCMOT tracing in a verification sequence of an ImageNet VID dataset using the MCMOT of the present invention. For each object bounding box, an identifier number is assigned, the class predicted for each object, and the confidence score of the segment are managed and displayed on the image together with the identifier. It is recommended that these figures be enlarged to digital devices.

As can be seen in FIG. 4, when using MCMOT, it can be seen that an unlimited variety of classes are tracked with high accuracy.

<Performance evaluation of MOT Benchmark 2016>

To evaluate the MCMOT of the present invention, a comparison is made between the approach proposed in the MOT Challenge 2016 benchmark and other state-of-the-art algorithms. For the MOT 2016 experiment, all training images and test images are scaled so that the shorter side of the image is 800 pixels in length. The reason for selecting a value larger than the scale pixel value used above is that the average bounding box of the pedestrian is smaller than the ImageNet VID. The tracking execution time was measured excluding the object detection time.

Table 3 below shows the tracking performance evaluation and comparison results on the MOT benchmark 2016 (as of July 14, 2016). The ↑ mark indicates that the higher score shows better performance, and the ↓ mark indicates that the lower score shows better performance. Table 3 summarizes the MCMOT metrics and compares them with other state-of-the-art methods in test video sequences.

[Table 3]

FIG. 5 shows an example of visualizing the tracking result of MCMOT in a test image sequence by applying the MCMOT of the present invention. Figure 5 is an example of MCMOT trace results in the MOT Benchmark 2016 test sequence. Each frame was sampled every 100 frames. The color of each bounding box represents the identifier of the target. It is recommended that these figures be viewed in a color image environment.

As can be seen in [Table 3], MCMOT shows better results than Multi-Object Tracking Accuracy (MOTA), which is a more comprehensive performance measure than the most advanced methods previously released. It also shows a much smaller number of mostly lost targets (ML). Although the method proposed by the present invention shows excellent results in most of the evaluation criteria, the tracker speed in frames per second (HZ), which measures the speed of the tracker, is faster than other trackers Can be confirmed. This result is due to the improved tracking speed of multiple object tracking through selective state forward-backward error detection through entity state transition and change point detection along with a concise Markov chain Monte Carlo-based tracking structure. However, it can be seen that a higher identity switch (IDS) and higher fragmentation (FRAG) are observed compared to state-of-the-art algorithms because of the lack of identifier mapping between trace segments. More importantly, because the proposed MCMOT achieved the most advanced performance in the other two data sets, the proposed method proved to be highly applicable to universal multi-class multi-object tracking in an unlimited number of classes with any kind of situation .

As described above, the multi-class multi-object tracking device (MCMOT) according to an embodiment of the present invention showed superior performance results compared with the state-of-the-art algorithms in ImageNet VID and MOT Benchmark 2016. [ MCMOT forms an unlimited number of object class connections based on the object detection results. The change point detection model observes sudden changes or anomalies caused by tracking drift. We calculated the likelihood based on the ensemble of Lucas-Kanade based motion detector and convolution neural network based object detector.

Thus, according to the multi-class multi-object tracking device (MCMOT) of the present invention, a variety of changing multi-object unlimited-classes can be traced by formulating the likelihood of a proposed region that ideally moves. Unlike existing techniques that estimate only limited types of objects, such as pedestrians and automobiles, an efficient convolution neural network based multi-class object detector can be applied to calculate the likelihood of multiple object classes. In addition, a change point detection algorithm based on static observations and dynamic observations evaluates the failure of tracking. The drift of multiple object tracking can be investigated by detecting a sudden change point in the time series without change represented by the tracking segment.

As described above, the present invention has been described with reference to particular embodiments, such as specific elements, and specific embodiments and drawings. However, it should be understood that the present invention is not limited to the above- Those skilled in the art will appreciate that various modifications, additions and substitutions are possible, without departing from the essential characteristics of the invention. Therefore, the spirit of the present invention should not be construed as being limited to the embodiments described, and all technical ideas which are equivalent to or equivalent to the claims of the present invention are included in the scope of the present invention .

Claims (4)

A multi-class multi-object tracking method for an input video sequence,
Calculating an observation likelihood of existence and nonexistence of an object included in the input image sequence;
Determining, based on the computation of likelihood, segments for object presence probability positions and establishing a plurality of trajectories connecting the segments;
Detecting a change point with respect to a time point at which the data of the segments on the trajectory change exceeds a threshold value; And
Performing a forward-backward check on the segments on the trajectory containing the change point, and combining the segments on the trajectory based on the result of the determination to determine final trace segments
Wherein the multi-class multi-object tracking method comprises: The method according to claim 1,
Wherein the determining step comprises: classifying the segment into a confidence segment and a drifted unstable segment using scoring for detection of the change point; maintaining the confidence segment and excluding the unstable segment; Wherein the multi-class multi-object tracking method comprises the steps of: 3. The method of claim 2,
Wherein the confidence level is used for estimating a confidence level as a forward-backward error for segments on a corresponding trajectory to exclude the retention of the confidence segment and the unstable segment, Wherein the Euclidean distance is calculated using the Euclidean distance between the trajectory-based segment trajectory and the backward trajectory based on backward tracing. A function of calculating an observation likelihood of existence and nonexistence of an object included in an input image sequence;
Determining, based on the computation of likelihood, segments for object presence probability positions and setting a plurality of trajectory trains connecting the segments;
Detecting a change point with respect to a time point at which the data of the segments on the trajectory change more than a threshold value; And
Performing backward-word verification on the segments on the trajectory containing the change point, and combining the segments on the trajectory based on the result of the determination to determine final trace segments
And a computer-readable code for multi-class multi-object tracking for an input video sequence.

Images