KR20180108123A - Method for tracking multi object - Google Patents

Abstract

The present invention relates to a method for tracking multiple objects, the method comprising the steps of: (a) generating a 2D image data set composed of a target object of a previous 2D image frame and a target candidate object of a current 2D image frame, and a depth data set composed of a target object of a previous depth frame and a target candidate object of a current 2D image frame; (b) applying the 2D image data set and the depth data set to a first matching network and a matching network independent from each other, outputting a 2D image tracking result for the 2D image data set from the first matching network, and outputting a depth tracking result for the depth data set from the second matching network; and (c) combining the 2D image tracking result and the depth tracking result by using a basic belief assignment to determine a final tracking result. Accordingly, when multiple target objects are tracked by using the 2D image frame and the depth frame acquired from multiple sensors, a tracking failure due to an error of data is overcome, and more accurate and stable tracking is possible.

Description

{METHOD FOR TRACKING MULTI OBJECT}

The present invention relates to a multi-object tracking method, and more particularly, to tracking multiple target objects using 2D image frames and depth frames acquired from multiple sensors, overcoming tracking errors due to errors in data, To a possible multi-object tracking method.

Object tracking is an important task used in many areas such as security, sports analysis, human-computer interaction, and autonomous navigation systems. As a result, various types of tracker have been developed, including multi-object tracking, tracking using multiple sensors, and modelless tracker.

An important objective of multi-object tracking is to track the status of the target object from the frames of a given video sequence. However, although various types of multi-object trackers have been proposed, there are still limitations in developing tracking performance due to various obstacles such as illumination changes, object size changes, and masking.

One way to address these obstacles is to model the disturbances a priori in the tracker. For example, in the article "An iterative image registration technique with an application to stereo vision (IJCAI, 1981, Vol. 81, pp. 674-679)" by Lucas, BD, Kanade, T. et al. transformation, and Nguyen, HT And Smeulders, AW, "Robust tracking using foreground-background texture discrimination" (International Journal of Computer Vision 2006, 69, 277-293) &Quot; Robust occlusion handling in object tracking. "IEEE Proceedings of Computer Vision and Pattern Recognition (IEEE, 2007, pp. 1-8.). However, a tracker modeled with a particular disturbance exhibits robust performance for that disturbance, but has limitations when overcoming other disturbances.

Another way is to adaptively update the morphological model while the tracker is running. However, even if the morphological model is adaptively updated, it is possible to miss the dramatically changing morphology if the temporarily changing situation is introduced into the updated morphological model.

Moreover, if the tracker only operates on RGB frames, the so-called bounding box shifting problem, in which the bounding box of the target object being tracked is considered a similar object with a similar shape or color in the next frame, .

To compensate for tracking failures resulting from various disturbances in each modality, multiple sensor fusion has been proposed. Tracking failures on RGB frames can be compensated using depth information from the 3D point cloud and stereo vision sensors. However, existing trackers using multiple sensors are modeled under the assumption that all sensors operate normally, so there is a weakness that they can not deal with the noise generated by one or more sensors.

SUMMARY OF THE INVENTION Accordingly, the present invention has been made to solve the above problems, and it is an object of the present invention to overcome the tracking failure due to data errors in tracking multiple target objects using 2D image frames and depth frames acquired from multiple sensors, The object of the present invention is to provide a multi-object tracking method capable of more accurate and stable tracking.

(A) a 2D image data set composed of a target object of a previous 2D image frame and a target candidate object of a current 2D image frame, and a 2D image data set of a current depth frame, Generating a depth data set comprising a target candidate object of a 2D image frame; (b) the 2D image data set and the depth data set are respectively applied to a first matching network and a matching network that are mutually independent, a 2D image tracking result of the 2D image data set is output from the first matching network, Outputting a depth tracking result for the depth data set from a matching network; and (c) the 2D tracing result and the depth tracing result are fused using a basic belief assignment to determine a final tracing result.

In the step (a), an instance including the target object and the target candidate object of the 2D image frame is expressed using a pre-trained convolution neural network; (A) extracting an output of a convolution layer of the 2D image frame from the pre-trained convolution neural network to generate an output characteristic map of the 2D image frame; ; (a2) a representation of each of the instances is pooled from the output property map according to a scale of each of the instances using ORI pooling; (a) the representation of the instance is normalized.

In addition, in step (a1), Max pooling is applied to an instance of a larger scale than the input of the first matching network, and an instance of a smaller scale than an input of the first matching network A deconvolution operation may be applied for sampling.

In the step (a), a supervision transfer may be applied to represent an instance including the target object and the target candidate object in the depth frame.

In the step (b), the first matching network and the second matching network are connected to two sub-networks sharing a weight, and two sub-networks are connected to each other, and a soft- neural network) is applied; The target object and the target candidate object may be separately input to the subnetwork of the Convolution Neural Network.

After the steps (a) to (c) are performed for a predetermined number of the 2D image frames and the depth frame, the first matching network and the depth frame are determined based on the matching score of the final tracking result. 2 matching network may be fine tuned to update the target appearance model of the first matching network and the second matching network.

According to the present invention, in tracking multiple target objects using 2D image frames and depth frames acquired from multiple sensors, it is possible to overcome tracking failure due to data errors, A possible multi-object tracking method is provided.

FIG. 1 is a diagram illustrating a configuration of a multi-object tracking system according to the present invention,
FIG. 2 is a view for explaining a multi-object tracking method according to the present invention,
FIG. 3 is a view for explaining a method of representing an instance in the multi-object tracking method according to the present invention,
4 is a view for explaining a structure of a convolutional neural network applied to a multi-object tracking method according to the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a configuration of a multi-object tracking system according to the present invention. 1, a multi-object tracking system according to the present invention includes a 2D frame acquisition unit, a depth frame acquisition unit 12, a data set generation unit 20, a first matching network 30, a second matching network 40 and a tracking result fusion unit 50.

The 2D image frame acquisition unit 11 acquires a 2D image frame. For example, a 2D image frame is acquired through a 2D image sensor such as a 2D image camera or a laser scanner. In the following description, it is assumed that the 2D image frame is an RGB frame, and the technical idea of the present invention is not limited thereto.

The depth frame acquisition section 12 acquires a depth frame. For example, a depth frame can be acquired from a three-dimensional image captured by an Angan camera using two image sensors, and depth information is included in the depth frame.

The data set generation unit 20 receives the RGB frame and the depth frame from the 2D image frame acquisition unit 11 and the depth frame acquisition unit 12 and receives the 2D image data set corresponding to the RGB frame, And generates a depth data set corresponding to the depth frame.

Here, the RGB data set is composed of the target object of the previous RGB frame (k) and the target candidate object of the current RGB frame (k + 1), and similarly, the depth data set includes the target object of the previous depth frame And a target candidate object of the depth frame (k + 1).

The first matching network 30 and the second matching network 40 operate independently of each other. The first matching network 30 receives the RGB data set, determines whether the target object matches the target candidate object, and outputs the 2D image tracking result, i.e., the RGB tracking result. The second matching network 40 receives the depth data set, determines whether the target object matches the target candidate object, and outputs the depth tracking result.

 Tracking Result The convergence unit 50 fuses the RGB tracking result with the depth tracking result to determine the final tracking result. In the present invention, it is assumed that the two tracking results are fused using the basic belief assignment.

As described above, the tracking result using the RGB frame and the tracking result of the depth frame are derived independently, and the independently derived tracking result is fused in the final determination process. Thus, The influence is minimized, and more accurate and stable tracking of the target object becomes possible.

Hereinafter, a multi-object tracking method applied to the multi-object tracking system according to the present invention will be described in detail with reference to FIG.

2, in the multi-object tracking method according to the present invention, a representation composed of a target object in a frame k and a target candidate object in a frame k + 1 is referred to as a first matching network 30 and a second matching network And the RGB data set and the depth data set input to the matching network 40 are generated.

Here, the output of the convolutional layer is adaptively used as a representation of the instance according to the scale of the instance to represent the instances in the RGB frame and the depth frame, i.e., the target object or the target candidate object.

More specifically, the target object undergoes a large change in its scale depending on factors such as the pose change and the movement state. If a lower resolution instance, for example a small scale instance, passes through the next layer on the matching network, then the fine characteristics gradually disappear by operations such as Convolution and Pooling . In the present invention, the instances are classified according to the scale of the instances and are represented.

3 is a diagram for explaining a method of representing an instance in the multi-object tracking method according to the present invention. Referring to FIG. 3, in the process of generating RGB data sets in the multi-object tracking method according to the present invention, an instance is created using a pre-trained convolution neural network (CNN) Is expressed. Here, when the entire instance is input to the CNN, a high calculation cost is incurred. In the present invention, ROI pooling is applied to the representation of the instance.

First, all the outputs of the convolutionary of the RGB frame are extracted from the trained CNN. The representation of each instance is then pooled from the Output Feature Map of the corresponding RGB frame according to the scale level using ROI pooling. Because the output sizes from each convolutional are different, in the present invention, they are sampled separately by applying different sampling layers as matching functions. That is, Max pooling is applied for subsampling to instances of a larger scale than the input of the first matching network 30, and upsampling is applied to instances of a smaller scale than the input of the first matching network 30 A deconvolution operation is applied. In addition, normalization of the expression of the instance is performed. In the present invention, local response normalization (LRN) is applied.

In the multi-object tracking method according to the present invention, supervision transfer is applied to a depth frame to represent an instance.

,

를 각각 RGB 프레임과 깊이 프레임의 레이어드 표현(layered representation)한다. 여기서, i는 레이어의 개수이다. 슈퍼비전 트랜스퍼는 고정된 CNN 구조로부터 언어노우티드(Unannotated)된 깊이 이미지를 표현하기 위해 가중치 파라미터

를 충분히 훈련시킨다. 슈퍼비전 트랜스퍼는 손실 함수

(본 발명에서는 L2 거리가 손실함수로 사용하는 것을 예로 한다)를 이용하여 RGB 프레임과 깊이 프레임의 표현 간의 유사성의 측정한다. 유사성은 [수학식 1]과 같이 측정된다.first,

,

Layered representation of the RGB frame and the depth frame, respectively. Here, i is the number of layers. The supervision transfer uses a weight parameter to represent the unannotated depth image from the fixed CNN structure

. Super Vision Transfer Loss Function

(In the present invention, L2 distance is used as a loss function as an example), the similarity between the representation of the RGB frame and the depth frame is measured. The similarity is measured as in Equation (1).

[Equation 1]

를

의 동일 차원(same dimension)으로 임베딩(embedding)하기 위한 변환 함수(transformation function)이고,

는 학습된 가중치 파라이터이다. 본 발명에서는 만약 깊이 이미지가 3D 포인트 클라우드로부터 얻어지면, 업-스케일링 방법이 변환 함수로 사용될 수 있다.In Equation (1), t

To

Is a transformation function for embedding in the same dimension of the input image,

Is a learned weighted wave distributor. In the present invention, if a depth image is obtained from a 3D point cloud, the up-scaling method can be used as a transform function.

When an instance constituting the RGB data set and the depth data set, that is, the target object and the target candidate object are represented through the above-described method, the RGB data set and the depth data set are respectively transmitted to the first matching network 30 and the second matching network (40). In the present invention, a convolutional neural network composed of two sub-networks sharing a weight to the first matching network 30 and the second matching network 40 is applied.

4 is a view for explaining a structure of a convolutional neural network applied to a multi-object tracking method according to the present invention. The convolutional neural network shown in FIG. 4 is applied to the first matching network 30 and the second matching network 40. The convolutional neural network is applied to the first matching network 30, A description of the convolutional neural network applied to the convolutional neural network 40 is omitted.

를 구성하는 타겟 객체

와 타겟 후보 객체

는 컨벌루션 신경망의 두 개의 서브 네트워크로 각각 분리되어 입력된다. 본 발명에 따른 컨벌루션 신경망의 서브 네트워크는 각각 3개의 컨벌루션 레이어(Convolution layer)와 2개의 풀리-커넥티드 레이어(Fully-connected layer)로 구성되는 것을 예로 한다.RGB data set

The target object

And target candidate objects

Are separately input into two subnetworks of the convolutional neural network. The subnetwork of the convolutional neural network according to the present invention includes three convolution layers and two fully-connected layers.

In the conventional convolution neural network structure, when Max pooling is applied as input, only strong values in the local neighbors are activated to transition to the next layer. That is, the spatial resolution of the activated value is significantly reduced. The benefits of maximum pooling are strong for local variants, but it is important to keep small shape changes of objects over time. Therefore, in the present invention, the maximum value pooling layer is excluded from each subnetwork.

와

이 매칭이 되는지 판단한다. 본 발명에서는 1 및 0이 매칭(positive)과 비매칭(dis-matching, negative)을 각각 나타내는 분류로 제1 매칭 네트워크(30)의 출력, 즉 RGB 프레임에 대한 RGB 추적 결과(깊이 프레임의 경우 깊이 추적 결과)로 출력된다.The output of the last pulley-connected layer of each subnetwork is coupled and transitioned to a two-way soft max layer with one vector. The soft max layer

Wow

It is determined whether or not this is a match. In the present invention, the output of the first matching network 30, i.e., the RGB tracking result for the RGB frame (the depth in the case of the depth frame, Tracking result).

As described above, when the RGB tracking result for the RGB frame and the depth tracking result for the depth frame are independently generated, the RGB tracking result and the depth tracking result are fused and more accurate tracking results are obtained. In the present invention, the RGB tracing result and the depth tracing result are merged using the basic belief assignment (BBA) to determine the final tracing result.

Using the BBA, RGB tracking results and depth tracking results evaluate the discounting factor. The discount factor α represents the reliability of each tracking result.

을 경우의 수라고 할 때, 0, 1, Ω는 각각 비매칭, 매칭 및 불확실을 나타내는 것으로 가정하면, 추적 결과에 대한 BBA는 [수학식 2]와 같이 정의될 수 있다.

Assuming that 0, 1, and? Denote mismatching, matching, and uncertainty, respectively, the BBA for the tracking result can be defined as: " (2) "

&Quot; (2) "

가 [수학식 3]과 같이 정의된다.In Equation (2), m (A) is a basic belief mass (BBM) representing a portion of the belief determined for the subset. A conjunctive rule is used for the combination of the BBA of the RGB frame and the BBA of the depth frame. Let mD and mR denote the BBM for the tracking results of the depth frame and the RGB frame, respectively. Conjunctive combination,

Is defined as " (3) "

&Quot; (3) "

The convergence of the multi-object tracking method according to the present invention assigns a weight according to the tracking result. To this end, a discount factor was added to each BBM in the present invention. The RGB tracking result and the discount factor for the depth tracking result can be defined as [Equation 4] and [Equation 5].

&Quot; (4) "

&Quot; (5) "

The present invention uses the normalized confidence function disclosed in Smets, P., "The combination of evidence in the transferable belief model. &Quot; IEEE Transactions on Pattern Analysis and Machine Intelligence (1990, 12, 447-458) Is set as an example.

와 깊이 프레임의 할인된 BBM

로부터 결합된 BBM

는 [수학식 6]을 통해 계산된다.Before merging RGB trace results with depth trace results, each trace result is discounted by their discount factor. Discounted BBM for RGB frames

And deep frame BBM

0.0 > BBM <

Is calculated by the following equation (6).

&Quot; (6) "

In Equation (6), α is calculated through a linear function and a quadratic function of the minimizing program of Smets, P. In the present invention, α _R and α _D are set to 0.22 and 0.31, respectively.

1 and 2, the multi-object tracking method according to the present invention may further include a fine tuning process of the first matching network 30 and the second matching network 40 .

For the configuration of the robust first matching network 30 and the second matching network 40 the first matching network 30 and the second matching network 40 which were initialized in the external video sequence are structured model is traced more than a certain number of frames and then fine tuned. Here, the tracking result is accumulated in the tracking result accumulation unit 60, and the fine tuning unit 70 performs a fine tuning process using the accumulation result.

The fine tuned first matching network 30 and the second matching network 40 may be robust to a temporally changing target shape while preserving model consistency. Here, the previously fine tuned matching network is maintained as a path of the target appearance model.

The structured target morph model according to the present invention is configured in a hierarchical structure by fine tuning the first matching network 30 and the second matching network 40 adaptively.

를 파인 튜닝된 매칭 네트워크(제1 매칭 네트워크(30) 또는 제2 매칭 네트워크(40), 이하 동일)에 대한 노드라 하고,

및

는 각각 파인 튜닝된 매칭 네트워크와 관련된 정점(vertex)과, 정점 간의 경로 관계(path relationship)를 나타내는 방향성 엣지(Directed edge)라 하면, 두 정점(하나의 에지)은 [수학식 7]과 같이 정의될 수 있다.

Is referred to as a node for a fine tuned matching network (the first matching network 30 or the second matching network 40, hereinafter the same)

And

Is defined as a directed edge representing a path relationship between a vertex and a vertex associated with a fine tuned matching network, and two vertices (one edge) are defined as in Equation (7) .

&Quot; (7) "

는 정점

와

사이의 관계 점수(Relationship score)를 의미하고,

는 정점

까지 매칭 네트워크를 이용하여 추적이 수행된 연속된 프레임의 세트이고,

는

가 정점

를 이용하여 이전의 타겟 객체

와 매칭된 후보로 판단되었을 때의 매칭 점수이다.In Equation (7)

Apex

Wow

The relationship score between the two groups,

Apex

Lt; RTI ID = 0.0 > a < / RTI > matching network,

The

Apex

≪ RTI ID = 0.0 >

Is a matching score when it is judged as a matching candidate.

가 프레임

에서의 타겟 객체의 세트라고 하면, 프레임

에서의 타겟 후보 객체는

이다. 본 발명에서는 세트

와

중 가장 유사한 쌍 C의 세트를 찾는 것이다. 이는 [수학식 8]을 만족시킨다.

Frame

A set of target objects in the frame < RTI ID = 0.0 >

The target candidate object in

to be. In the present invention,

Wow

The most likely pair C is found. This satisfies [Expression 8].

&Quot; (8) "

는 n번째 타겟 객체의 활성화된 파인 튜닝 경로로부터 얻은 가중치 평균 매칭 점수이다.

가 n번째 타겟 형태 모델의 활성화된 파인 튜닝 경로라 하면, 가중치 평균 매칭 점수는 [수학식 9]와 같이 측정될 수 있다.In Equation 8,

Is the weighted average matching score obtained from the activated fine tuning path of the nth target object.

Is an activated fine tuning path of the n-th target form model, the weighted average matching score can be measured as shown in Equation (9).

&Quot; (9) "

는 n번째 타겟 형태 모델의 정점

에 대응하는

와

사이의 매칭 점수이고, 분류 1(matching)에 대한 확률이다. 그리고,

는 n번째 타겟 객체의 경로에서 프레임

안의 정점

의 가중치이다. 후보가 타겟 객체와 매칭되지 않으면, 추적에서 새로운 삭제된 객체로 여겨진다. 또한, 타겟 객체가 모든 후보에 대해

보다 작은 매칭 점수를 가지면, 사라진 객체로 여겨진다. 본 발명에서는 실험적으로

을 0.6으로 설정하는 것을 예로 한다.In Equation (9)

Is the vertex of the nth target type model

Corresponding to

Wow

, And is the probability for matching. And,

Is the frame in the path of the nth target object

The inner peak

. If the candidate does not match the target object, it is considered a new deleted object in the trace. Also, if the target object has

Having a smaller matching score is considered a lost object. In the present invention,

Is set to 0.6.

를 결정하기 위해, 본 발명에서는 매칭 네트워크의 신뢰성(reliability)을 고려한다. 이와 같은 가중치는 노이즈 인스턴스에도 불구하고 높은 매칭 점수가 측정되는 것과 같이, 파인 튜닝이 신뢰할 수 없는 케이스로 생성되는 것을 방지하기 위해 할당된다. 매칭 네트워크의 신뢰성 측정을 위해, 파인 튜닝 경로(모든 경로가 아님)는 [수학식 10]과 같이 회귀적으로 탐색(Recursively explored)된다.weight

The reliability of the matching network is considered in the present invention. Such a weight is assigned to prevent fine tuning from being created as an unreliable case, such as a high matching score being measured despite the noise instance. In order to measure the reliability of the matching network, the fine tuning path (not all paths) is recursively explored as shown in Equation (10).

&Quot; (10) "

는 정점

의 부모 노드(Parent node)이다.In Equation (10)

Apex

(Parent node).

In a structured target morph model, the node includes a fine tuning matching network. In the present invention, an adaptive model update method for a fine tuning matching network for a new training frame is applied.

)의 추적이 완료된 후에 수행되는 것을 예로 한다. 새로운 파인 튜닝된 매칭 네트워크는 부모 노드

를 가지며, [수학식 11]을 만족한다.z is a newly generated node for fine tuning the matching network. Fine tuning of the matching network can be performed in 15 consecutive frames (

) Is completed. The new fine-tuned matching network includes a parent node

And satisfies the following expression (11).

&Quot; (11) "

는 임시 엣지(Interim edge)이다. 새로 생성된 노드 크의 부모 노드를 찾은 후에, 두 세트의 프레임

와

에 대한 파인 튜닝이 수행된다. 마지막으로 노드 z의 새로이 파인 튜닝된 매칭 네트워크가

에 추가됨으로써, 파인 튜닝이 완료된다.In Equation (11)

Is an interim edge. After finding the parent node of the newly created node, two sets of frames

Wow

The fine tuning is performed. Finally, a new fine tuned matching network of node z

The fine tuning is completed.

Although several embodiments of the present invention have been shown and described, those skilled in the art will appreciate that various modifications may be made without departing from the principles and spirit of the invention . The scope of the invention will be determined by the appended claims and their equivalents.

11: 2D image frame acquisition unit 12: depth frame acquisition unit
20: data set generation unit 30: first matching network
40: second matching network 50: tracking result fusion unit
60: tracking result accumulation unit 70: fine tuning unit

Claims (6)

In a multi-object tracking method,
(a) a 2D image data set composed of a target object of a previous 2D image frame and a target candidate object of a current 2D image frame, and a depth data set composed of a target object of a previous depth frame and a target candidate object of a current 2D image frame ;
(b) the 2D image data set and the depth data set are respectively applied to a first matching network and a matching network that are mutually independent, a 2D image tracking result of the 2D image data set is output from the first matching network, Outputting a depth tracking result for the depth data set from a matching network;
and (c) determining a final tracking result by fusing the 2D image tracking result and the depth tracking result using a basic belief assignment. The method according to claim 1,
In the step (a), an instance including the target object and the target candidate object of the 2D image frame is expressed using a pre-trained convolution neural network;
The step (a)
(a1) extracting an output of a convolution layer of the 2D image frame from the pre-trained convolution neural network to generate an output characteristic map of the 2D image frame;
(a2) a representation of each of the instances is pooled from the output property map according to a scale of each of the instances using ORI pooling;
(a) the representation of the instance is normalized. The method according to claim 1,
In the step (a1)
Max pooling is applied for subsampling to instances of a larger scale than the input of the first matching network,
Wherein a deconvolution operation is applied to an instance of a smaller scale than the input of the first matching network for upsampling. The method according to claim 1,
Wherein in the step (a), an instance including the target object of the depth frame and the target candidate object is represented by a supervision transfer. The method according to claim 1,
In the step (b), the first matching network and the second matching network are connected with two sub-networks sharing a weight, and two sub-networks are connected to each other, and a soft max layer );
Wherein the target object and the target candidate object are separately input to the subnetwork of the convergence neural network. The method according to claim 1,
Wherein, after performing the steps (a) to (c) for a predetermined number of the 2D image frames and the depth frame, the first matching network and the second matching Wherein the network is fine tuned to update a target appearance model of the first matching network and the second matching network.

Images