KR20200061118A - Tracking method and system multi-object in video - Google Patents

Abstract

The present invention relates to a method and a system for tracking multiple objects in a video, and more specifically, to a technique and a system for tracking objects by using the characteristics of the objects in a video, tracking movements of the objects, and performing correction when missing or incorrectly tracking the objects to accurately track the objects. The method comprises the steps of: obtaining an image of an area of interest; identifying a plurality of objects included in the image respectively, and tracking a tracklet of the objects; evaluating the reliability of the tracking results; and re-identifying the objects whose reliability evaluation value is below a threshold value and re-tracking the tracklet of the objects.

Description

Tracking method and system for multiple objects in an image {TRACKING METHOD AND SYSTEM MULTI-OBJECT IN VIDEO}

The present invention relates to a method and system for tracking multiple objects in an image, and more specifically, to track an object using characteristics of the object in the image, track the movement of the object, and correct the correct object when the object is missed or incorrectly tracked. It is related to technology and systems capable of performing tracking.

One of the fundamental and challenging problems of computer vision, MULTIPLE-OBJECT TRACKING (MOT) has a wide range of applications, such as autonomous driving, video surveillance and advanced driver assistance systems (ADAS). The goal of multi-object tracking is to maintain the object's identity consistently in order to estimate the object's position and calculate the object's trajectory. MOT has evolved tremendously over the past decade. However, it is still a difficult task due to frequent or long-term occlusion by different objects or backgrounds of similar shapes and complex scenes of different objects. Due to recent advances in object detectors, the tracking detection approach has shown a remarkable performance improvement in MOT.

The tracking method by detection can be classified into offline and online tracking methods. The offline method builds a robust multi-orbit for occlusion and false detection using a batch method. On the other hand, the online method can be applied to actual applications because it uses the previous and current frame information to create trajectories for each frame. The offline method shows better performance than the online method because it uses future information as well as previous and current information, but is not suitable for real-time applications such as ADAS.

The tracking approach by detection has made great progress in the online MOT, but fragmented trajectory generation and object position drift problems can still occur. The cause of this problem is mainly that (i) the object characteristics are not strong enough to distinguish the internal categories due to similar appearances, and (ii) occlusion and unreliable detection cause the object to be temporarily lost or incorrect association. In general, object feature expression and data association are essential to improve MOT performance. Moreover, the balance between accuracy and processing time is also important in practical applications.

Fortunately, with the development of large-scale visual data sets and increased computing power, the Convolutional Neural Network (CNN), with its powerful differentiated and feature-learning skills, has shown breakthrough performance in computer vision work. MOT also benefits from CNNs. However, there is still a lack of discrimination dealing with internal classification objects and occlusion. A common way to solve the above problem is to maintain an instance-specific shape model for each tracked object and fine-tune the CNN model during testing through back presentation. It's a great way to improve the performance of the MOT, but given the unacceptable computational cost and processing time of the CNN for online education, low-level manual features are still widely used in the MOT. To overcome this limitation, we first train the CNN model offline using a set of object identification data that is useful for distinguishing inner-category objects. Next, a correlation filter-based reliability, which is one of the single object tracking methods robust to appearance change, is introduced into the MOT to create a robust appearance model for each tracklet.

In addition, the reliability based on the correlation filter can improve the problem of temporary object loss and incorrect association. Data relevance can be divided into two levels (hierarchically) according to the reliability score. The first step solves the connection problem between all tracklets and detections, and the second step involves two, including a drifting tracklet, with incorrect association results determined by a confidence score and detection of objects reoccurring in occlusal or other uncoupled objects. Applies to tracklets of this type.

Therefore, there is a need for a technique for balancing accuracy and processing time.

Accordingly, the present invention is to solve the above problems, and an object of the object tracking method and system in an image is to provide a technique for improving accuracy and improving processing speed.

A multi-object tracking method in an image according to an embodiment of the present invention for solving the above problems may include obtaining an image of a region of interest; Identifying a plurality of objects included in the image, and tracking tracklets of the objects; Evaluating the reliability of the tracking result; And re-identifying an object whose reliability evaluation is less than or equal to a threshold and re-tracking the tracklet of the object.

According to an embodiment of the present invention, the image acquiring step includes: acquiring an image including the identified object and tracklet information; And, extract the external features of the identified object. Learning; may include.

According to an embodiment of the present invention, the first tracking step may be tracked through data correlation using the appearance, shape, and motion information of the object.

According to an embodiment of the present invention, the second tracking step may be tracked through data correlation using appearance and shape information of the object.

According to an embodiment of the present invention, updating the tracklet using the appearance and shape information of the object according to the re-tracking result; And updating the appearance model of the object using the appearance, shape, and tracklet information of the object.

In order to solve the above problems, a multi-object tracking system in an image according to an embodiment of the present invention includes an image acquisition unit for acquiring an image of a region of interest; An object detection unit that identifies a plurality of objects included in the image and tracks a tracklet of the object, respectively; Includes; a verification unit for evaluating the reliability of the tracking result, the object detection unit, the reliability evaluation can re-identify the object below the threshold value and retrace the tracklet.

According to an embodiment of the present invention, the image acquisition unit may acquire an image including the identified object and tracklet information, and learn by extracting an external characteristic of the identified object.

According to an embodiment of the present invention, the object detection unit may track through data correlation using the appearance, shape, and motion information of the object.

According to an embodiment of the present invention, the object detection unit updates the tracklet using the appearance and shape information of the object according to the retrace result, and the object's appearance, shape, and tracklet information is used to track the object. The appearance model can be updated.

According to the present invention, it is possible to accurately track an object by dealing with an object loss and an incorrect association problem than a conventional tracking technique.

In addition, by updating the appearance model based on the trust score, it is possible to determine whether the tracking result is correct and correct it by itself if it is incorrect, and to improve the tracking accuracy of the repeatedly tracked object.

Meanwhile, the effects of the present invention are not limited to the above-mentioned effects, and various effects may be included within a range obvious to those skilled in the art from the following description.

1 is a schematic diagram of a multi-object tracking method according to an embodiment of the present invention.
2 is a block diagram of a multi-object tracking system according to an embodiment of the present invention.
3 is an example of incorrect data correlation of a multi-object tracking method according to an embodiment of the present invention.
4 illustrates a reliability map according to a situation of a multi-object tracking method according to an embodiment of the present invention.
5 is a result of comparing performance in different Loss Functions of a multi-object tracking method and an existing method according to an embodiment of the present invention.
6 is a result of comparing the multiple object tracking method according to an embodiment of the present invention and the hierarchical and single stage data correlation of the existing method.
7 is a result of comparing the performance of a multi-object tracking method and an existing method in various datasets according to an embodiment of the present invention.
8 is a flowchart of a multi-object tracking method according to an embodiment of the present invention.

Hereinafter, a'multi-object tracking method and system in an image' according to the present invention will be described in detail with reference to the accompanying drawings. The described embodiments are provided so that those skilled in the art can easily understand the technical idea of the present invention, and the present invention is not limited thereby. In addition, matters expressed in the accompanying drawings may be different from those actually implemented in schematic drawings to easily describe embodiments of the present invention.

On the other hand, each component represented below is only an example for implementing the present invention. Accordingly, other components may be used in other implementations of the present invention without departing from the spirit and scope of the present invention.

In addition, each component may be implemented solely in the configuration of hardware or software, but may also be implemented in a combination of various hardware and software components that perform the same function. Also, two or more components may be implemented together by one hardware or software.

In addition, the expression'includes' certain components, as an expression of'open', simply refers to the existence of the components, and should not be understood as excluding additional components.

1 is a schematic diagram of a multi-object tracking method according to an embodiment of the present invention.

Referring to FIG. 1, in the multi-object tracking method according to an embodiment of the present invention, in each frame, detection may be performed at an input to obtain a position of an object. Various types of feature information including the shape, location and appearance of the object can be fused to each detected object. Existing tracklets can be associated with detection in the current frame, after which confidence is calculated for each associated tracklet-detection pair. If the reliability is lower than a predefined threshold value, the multi-object tracking method according to an embodiment of the present invention may perform second stage data association with non-associative detection, assuming that this is a non-standard tracklet, that is, incorrect tracking. After data association, the tracklet status is updated and, in particular, the appearance model can be selectively updated according to the reliability. Finally, the updated tracklet of the current frame can be used as input to the data association of the next frame.

2 is a block diagram of a multi-object tracking system according to an embodiment of the present invention.

Referring to FIG. 2, a multi-object tracking system according to an embodiment of the present invention may include an image acquisition unit 210, an object detection unit 220, and a verification unit 230.

The image acquisition unit 210 may acquire an image of a region of interest. The image acquisition unit 210 may acquire an image including the identified object and tracklet information. The identified object information and tracklet information included in the image acquired by the image acquisition unit 210 may be received from the outside. The identified object information and tracklet information included in the image acquired by the image acquisition unit 210 may be derived by recognizing an object showing motion in the image captured by the image acquisition unit 210. The image acquisition unit 210 may include a camera capable of taking an image. The image acquisition unit 210 may include a computer or artificial neural network capable of identifying an object showing movement in the acquired image and tracking the movement.

The object detection unit 220 may respectively identify a plurality of objects included in the image, and track a tracklet of the object.

The object detection unit 220 may re-identify an object whose reliability evaluation of the verification unit 230 is less than or equal to a threshold value, and retrace a tracklet of the object. The object detection unit 220 may perform object tracking through data correlation using the appearance, shape, and motion information of the object. The object detection unit 220 may perform object tracking through data correlation using appearance and shape information of the object. The object detection unit 220 may update the tracklet using the appearance and shape information of the object according to the retrace result. The object detection unit 220 may update the appearance model of the object using the appearance, shape, and tracklet information of the object.

The object detection unit 220 may extract and learn the external characteristics of the identified object. The object detection unit 220 may generate an external model of the object using the appearance and shape information of the identified object. The object detection unit 220 may update the appearance model using the tracking and re-tracking results. The object detection unit 220 may generate and update the appearance model by machine learning the tracking and retrace results.

The object detection in the frame t by the object detection unit 220 is Dt = {d1t; d2t; :::; dnt} and n may be the number of detections. Rt = {r1t; r2t; :::; rmt} may be a set of tracklets. Here, r ⁱ _t may be the i-th tracklet in frame t and m may be the number of tracklets. In the object detection unit 220, the state of the i-th tracklet updated in frame t may be an appearance feature vector extracted by a CNN model, pi = (xi, yi) is location information, and si = (wi; hi ) May be form information composed of width and height. A similarity model for determining how well a tracklet associated with detection including appearance, motion and shape information is defined may be defined as in Equation 1 below.

[Equation 1]

The appearance, motion, and shape similarity scores used in the calculation of the similarity model may be calculated as in Equations 2 to 4, respectively.

[Equation 2]

[Equation 3]

[Equation 4]

에 대하여, 본 발명의 일 실시 예에 따른 CNN 모델에 의해 검출을 위한 특징이 추출되고, 이전의 대응하는 검출에 의해 갱신된 트랙렛 외관 특징으로 L2 거리가 계산된다. 움직임 유사성

는 트랙렛의 위치, 탐지 및 탐지 스케일을 사용하여 계산될 수 있다. 형태 유사성

는 폭 w와 높이 h를 사용하여 계산될 수 있다.Appearance similarity

With respect to, the feature for detection is extracted by the CNN model according to an embodiment of the present invention, and the L2 distance is calculated as the tracklet appearance feature updated by the previous corresponding detection. Motion similarity

Can be calculated using the tracklet's position, detection and detection scale. Shape similarity

Can be calculated using width w and height h.

From the similarity model in the object detection unit 220, bidirectional association between all detections and tracklets can be performed. The score matrix Smxn can be calculated as in Equation 5 between m tracklets and n detections in frame t.

[Equation 5]

는 수학식 1에서 계산된 값이다. 그런 다음 헝가리 알고리즘이 수행되어 트랙 크렛 감지 연관을 결정할 수 있다.here,

Is a value calculated in Equation 1. The Hungarian algorithm can then be performed to determine the track cret detection association.

All tracklets are associated with the first stage data association, but false internal trackwork detection may occur due to occlusion or unreliable detection in complex scenes. This is especially true if the characteristics of the internal category objects are similar.

보다 낮으면, 상기 검증부(230)는 상기 트랙렛을 신뢰할 수 없는 것으로 분류할 수 있다. 또한, 대응하는 검출은 제 1 스테이지 데이터 연관에서 비연관 검출으로 새로운 비연관 검출과 결합될 수 있다. 비신뢰적인 트랙렛과 비연관 검출이있는 경우 두 가지 모두에 대해 전역 연관이 수행될 수 있다. 형태 유사성 모델은 다음과 같이 수학식 6을 연산해 두 번째 연관성에 대해 재정의될 수 있다.The verification unit 230 may verify the detection result of the object detection unit 220. The verification unit 230 may trust a single tracklet as a single object tracking after data association. Reliability scores can be computed using a correlation filter to determine the confidence between detection and tracklet to solve incorrect associations and problems. The confidence score of the tracklet is the threshold

If lower, the verification unit 230 may classify the tracklet as unreliable. In addition, the corresponding detection can be combined with a new non-associative detection from non-associative detection in the first stage data association. If there is unreliable tracklet and uncorrelated detection, global association can be performed for both. The shape similarity model can be redefined for the second association by calculating Equation 6 as follows.

[Equation 6]

The tracklet may not be associated with the same detection associated with the first step data association.

After the hierarchical data association, (i) the tracklet position and shape information is updated with the associated detection, and (ii) the appearance information can be updated as shown in Equation 7.

[Equation 7]

는 하이퍼 매개 변수이고 t는 프레임 인덱스다.

는 상관 필터로 계산 된 신뢰도다.here

Is a hyper parameter and t is a frame index.

Is the reliability calculated with the correlation filter.

The verification unit 230 may perform a robust tracking by combining the correlation filter with a CNN-based function, particularly in a single object tracking problem. It is also necessary to compute the dot product and vector standards in the frequency domain, both of which are simple operations. Therefore, a correlation filter can be applied to the MOT to balance accuracy and processing time.

를 다음 수학식 8과 같이 계산할 수 있다.The verification unit 230 can be viewed as ridge regression because the correlation filter allows a simple closed solution. The key is that an increase in negative samples is used to improve the discriminative ability of the detection-specific tracking system while exploring the structure of the circular matrix to achieve high efficiency. The verification unit 230 finds the location by finding the maximum confidence score. By synchronizing this process and the high-speed process, the verification unit 230 has a reliability between the tracklet and the detection.

Can be calculated as in Equation 8 below.

[Equation 8]

은 역 이산 푸리에 변환을 나타내며,

는 요소 단위의 곱 연산자이며,

는 변수가 주파수 영역에 있음을 의미한다.here

Denotes the inverse discrete Fourier transform,

Is the element-wise multiplication operator,

Means that the variable is in the frequency domain.

The verification unit 230, the best CNN model for MOT needs appropriately differentiated internal categories and offline training to improve processing time. To this end, the object detection unit 220 or the verification unit 230 according to an embodiment of the present invention may use the Pretrained ResNet-50 architecture by discarding the last layer and adding two completely connected layers. The first fully linked layer has 1024 units, then the ReLU, and the second item can be divided into 128 units. The network can be trained by the MARS data set. The MARS data set is a large person reconfirmation data set containing 1,261 identities and 1,191,003 images. The cosine soft max loss function can be used because all samples are pushed at the decision boundary in the direction of the parameterized class mean. That is, it can not only classify objects, but also converge functions according to different classification criteria, which can help in classifying internal categories. Therefore, the robustness of the CNN model can be improved.

The verification unit 230 is given a data set B = {(xi, yi)} Ni = 1 of given N training samples and related class labels. yi ∈ {1; 2; In ::: C}, the basic softmax classifier can select a class having the maximum probability for an object based on a parametric function as shown in Equation 9.

[Equation 9]

Here, f(x) is a feature expression extracted by the CNN model, which can be trained jointly with a classifier. The basic softmax classifier can be modified to create a compact cluster in the expression space with a small adaptation that can improve the robustness of the internal adaptive classification. L2 normalization must first be calculated for the final layer of the network to ensure that the representation is unit length. Weight wk, ∀k = 1, 2,… C should be standardized to unit length. The cosine softmax classifier can be expressed as Equation 10 below.

[Equation 10]

Where κ is a free scaling parameter. As a method of accelerating the convergence of the stochastic gradient drop, there is a method of separating the length of the weight vector κ from that direction.

를 볼 때, 클래스 조건부 확률이 수학식 11, 수학식 12와 같이 von Mises-Fisher (vMF) 분포를 따를 수 있다.The verification unit 230, from a generating point of view,

Looking at, the class conditional probability can follow the von Mises-Fisher (vMF) distribution as shown in Equation 11 and Equation 12.

[Equation 11]

[Equation 12]

Here, I _v represents the modified Bessel function of the first kind in order v. Consider p = 2 in one embodiment of the present invention. This distribution decreases from the circle to the von Mises distribution, so Equation 11 becomes Equation 13 as follows.

[Equation 13]

는 공유 농도 매개 변수다. 작은 C의 경우 분포가 거의 균일하므로 CNN 모델의 내부 범주 분류 기능을 향상시키기 위해 적절한

을 선택할 수 있다. 결과적으로, 사후 클래스 확률은 베이스 (Bayes) 규칙에 의해 수학식 14로 계산될 수 있다.Where I ₀ is the modified Bessel function of order 0

Is the shared concentration parameter. For small C, the distribution is almost uniform, so it is appropriate to improve the internal categorization of the CNN model.

You can choose As a result, the posterior class probability can be calculated by Equation 14 according to the Bayes rule.

[Equation 14]

주위로 정점을 이루고 붕괴하는 1 차원 구에 대한 등방성 확률 분포이다. 해당 코사인 소프트 맥스 손실 함수는 다음 수학식 15과 같이 쓸 수 있다.The verification unit 230, the von Mises distribution is the direction as the cosine similarity decreases

An isotropic probability distribution for a one-dimensional sphere that peaks and collapses around. The cosine soft max loss function can be written as in Equation 15 below.

[Equation 15]

값을 가진 기본 softmax로 적용하여 견고한 클러스터로 샘플을 밀어 넣을 수 있다. This is different

The sample can be pushed into a solid cluster by applying it as a default softmax with a value.

3 is an example of incorrect data correlation of a multi-object tracking method according to an embodiment of the present invention.

As can be seen in FIG. 3, the objects 4 in (a) and (b) have similar appearances and positions, and thus are incorrect association results. In addition, there is a possibility of generating a drift tracklet due to a long detection error, and in this case, it is difficult to detect a recurrence by reassigning the drift tracklet.

4 illustrates a reliability map according to a situation of a multi-object tracking method according to an embodiment of the present invention.

Referring to FIG. 4, the correlation filter is strong in tracking a single object even when the appearance model is similar or severely occluded. 4 highlights the relationship between the occlusion and the response map obtained by the correlation filter. The response maps for objects 43 and 49 are sharp, the confidence score (maximum value) for object 1 is 43, and #49 is sharp and confidence score (peak value) is similar and high with no occlusion. However, the response map for Subject 2 in #49 fluctuates considerably, and the confidence score (peak value) is lower than that in #43 because it was occluded in #49. It can be seen that the response map including the reliability can reflect the conditions of occlusion with the MOT.

5 is a result of comparing performance in different Loss Functions of a multi-object tracking method and an existing method according to an embodiment of the present invention.

는 0.95로 설정될 수 있다. 비신뢰 관계의 임계 값은

= 0.5이고 모양 모델 업데이트의 임계 값은

= 0.7 이다. 모든 매개 변수 설정은 모든 이미지 시퀀스에 대해 동일하게 유지된다.Referring to FIG. 5, the multi-object tracking method according to an embodiment of the present invention is evaluated with three different data sets of PETS S2.L2 and MOT Challenge to analyze different contributing factors and the entire MOT system. During appearance feature extraction, all input images are resized to 128x64x3 and all parameter settings are fixed for all test data sets. Hyper parameters

Can be set to 0.95. The threshold for untrusted relationships is

= 0.5 and the threshold for updating the shape model is

= 0.7. All parameter settings remain the same for all image sequences.

In the multi-object tracking method according to an embodiment of the present invention, multi-object accuracy (MOTA) and multi-object tracking accuracy (MOTP) for calculating accuracy composed of false positive (FP), false negative (FN), and mismatch are ground truths. Evaluate true positive trajectory alignment in relation to. In addition, the existing methods include most track targets (MT, ground truth trajectory covered by the tracking hypothesis for at least 80%), mainly lost track targets (ML, ground truth track covered by the tracking hypothesis for up to 20%). And the total number of identification switches (IDS) is utilized. In addition, treatment time (Hz) should be included in the evaluation scale.

Another commonly used object re-identification loss function is for the tuples of three examples x _a , x _p and x _n that contain the positive pair y _a = y _p and the negative pair y _a ≠ y _n . It is a triplet as defined in Equation 16. The triplet loss requires that the difference in distance between the negative and positive pairs is more important than the predefined margin.

[Equation 16]

는 마진이다. 이 두 손실 함수는 PETS S2.L2 데이터 세트에서 평가되며, 그 비교 결과는 도 5에 나와 있다. 이러한 결과로부터 코사인 소프트 맥스 손실 함수는 객체 재 식별에서 훨씬 강력하기 때문에 코사인 소프트 맥스 손실 함수는 삼중 항보다 우수한 성능을 보여준다. MOT에 대한 내부 범주 분류를 개선할 수 있다.here

Is the margin. These two loss functions are evaluated in the PETS S2.L2 data set, and the comparison results are shown in FIG. 5. From these results, since the cosine soft max loss function is much more powerful in object re-identification, the cosine soft max loss function shows better performance than the triplet term. The internal categorization of MOT can be improved.

6 is a result of comparing the hierarchical and single stage data correlation of a multi-object tracking method and an existing method according to an embodiment of the present invention.

Referring to FIG. 6, in order to demonstrate the performance of hierarchical data association, the multi-object tracking method and the existing method according to an embodiment of the present invention are evaluated in two versions. One is the whole system and the other skips the second stage data association in the PETS S2.L2 data set. The results of the comparison are shown in FIG. 6 and show that the trust score based hierarchical data association is robust to deal with the wrong association and tracklet drift problems.

7 is a result of comparing the performance of a multi-object tracking method and an existing method according to an embodiment of the present invention in various data sets.

Referring to FIG. 7, the multi-object tracking method according to an embodiment of the present invention is compared with other advanced methods for PETS S2.L2 and MOT Challenge. The results are summarized in FIG. 7.

The multi-object tracking method according to an embodiment of the present invention obtains reasonable performance in a PETS S2.L2 data set showing the effect of a trust score-based hierarchical data association and shape model update strategy. Compared to the offline MOT method, the performance of the multi-object tracking method according to an embodiment of the present invention is also acceptable and is superior in some factors.

For the MOT Challenge data set, compare different methods and methods of the MOT16 test data set containing 7 image sequences. Since it is a weighted combination of FN, FP and IDS, it focuses on MOTA comparison. Offline methods generally use tracking information to perform better than online methods. The online method CNN-based tracking methods STAM16 and CAD_DDAL both update the appearance model for each object online through a backflow, which affects processing time, so you can get the best performance from most metrics. STAM16 improves computational efficiency by sharing features and using ROI pooling to obtain individual features for each object. It reaches 0.2 and 0.5 frames per second (fps), so it's too slow for real applications like ADAS. In the multi-object tracking method according to an embodiment of the present invention, since the appearance model is updated using a reliability inspired by a correlation filter well known as high-speed processing, it may be much more effective to balance accuracy and processing time. The processing time of the multi-object tracking method according to an embodiment of the present invention may reach 8 fps with acceptable performance.

In the multi-object tracking method according to an embodiment of the present invention, reliability-based hierarchical data association for MOT is disclosed. To alleviate the problem of false associations due to occlusion and unreliable detection, the first step is to divide the data connection into relevant, trusted tracklet detections based on reliability. The second level of data relevance applies between untrusted tracklets and the rest of the searches, including mismatched and mismatched searches. In addition, due to the high-speed processing, since reliability is used for updating the appearance model, the robustness of the appearance model is improved and processing time is saved. The representative experimental results compared with other state-of-the-art trackers show that the multi-object tracking method according to an embodiment of the present invention is generally excellent in balancing various kinds of problems, especially accuracy and processing time.

8 is a flowchart of a multi-object tracking method according to an embodiment of the present invention.

Referring to FIG. 8, a multi-object tracking method according to an embodiment of the present invention may include obtaining an image of a region of interest (S810 ).

In step S810, the image acquisition unit 210 may acquire an image of the region of interest. The image acquisition unit 210 may acquire an image including the identified object and tracklet information. The identified object information and tracklet information included in the image acquired by the image acquisition unit 210 may be received from the outside. The identified object information and tracklet information included in the image acquired by the image acquisition unit 210 may be derived by recognizing an object showing motion in the image captured by the image acquisition unit 210. The image acquisition unit 210 may include a camera capable of taking an image. The image acquisition unit 210 may include a computer or artificial neural network capable of identifying an object showing movement in the acquired image and tracking the movement.

The multi-object tracking method according to an embodiment of the present invention may include a step (S820) of identifying a plurality of objects included in the image and tracking a tracklet of the object, respectively.

In step S820, the object detection unit 220 may re-identify an object whose reliability evaluation of the verification unit 230 is equal to or less than a threshold value and retrace the tracklet of the object. The object detection unit 220 may perform object tracking through data correlation using the appearance, shape, and motion information of the object. The object detection unit 220 may perform object tracking through data correlation using appearance and shape information of the object. The object detection unit 220 may update the tracklet using the appearance and shape information of the object according to the retrace result. The object detection unit 220 may update the appearance model of the object using the appearance, shape, and tracklet information of the object.

The object detection unit 220 may extract and learn the external characteristics of the identified object. The object detection unit 220 may generate an external model of the object using the appearance and shape information of the identified object. The object detection unit 220 may update the appearance model using the tracking and re-tracking results. The object detection unit 220 may generate and update the appearance model by machine learning the tracking and retrace results.

The object detection in the frame t by the object detection unit 220 is Dt = {d1t; d2t; :::; dnt} and n may be the number of detections. Rt = {r1t; r2t; :::; rmt} may be a set of tracklets. Here, r ⁱ _t may be the i-th tracklet in frame t and m may be the number of tracklets. In the object detection unit 220, the state of the i-th tracklet updated in frame t may be an appearance feature vector extracted by a CNN model, pi = (xi, yi) is location information, and si = (wi; hi ) May be form information composed of width and height. A similarity model for determining how well a tracklet associated with detection including appearance, motion, and shape information is defined may be defined as in Equation 1 below.

[Equation 1]

The appearance, motion, and shape similarity scores used in the calculation of the similarity model may be calculated as in Equations 2 to 4, respectively.

[Equation 2]

[Equation 3]

[Equation 4]

에 대하여, 본 발명의 일 실시 예에 따른 CNN 모델에 의해 검출을 위한 특징이 추출되고, 이전의 대응하는 검출에 의해 갱신된 트랙렛 외관 특징으로 L2 거리가 계산된다. 움직임 유사성

는 트랙렛의 위치, 탐지 및 탐지 스케일을 사용하여 계산될 수 있다. 형태 유사성

는 폭 w와 높이 h를 사용하여 계산될 수 있다.Appearance similarity

With respect to, the feature for detection is extracted by the CNN model according to an embodiment of the present invention, and the L2 distance is calculated as the tracklet appearance feature updated by the previous corresponding detection. Motion similarity

Can be calculated using the tracklet's position, detection and detection scale. Shape similarity

Can be calculated using width w and height h.

From the similarity model in the object detection unit 220, bidirectional association between all detections and tracklets can be performed. The score matrix Smxn can be calculated as in Equation 5 between m tracklets and n detections in frame t.

[Equation 5]

는 수학식 1에서 계산된 값이다. 그런 다음 헝가리 알고리즘이 수행되어 트랙 크렛 감지 연관을 결정할 수 있다.here,

Is a value calculated in Equation 1. The Hungarian algorithm can then be performed to determine the track cret detection association.

All tracklets are associated with the first stage data association, but false internal trackwork detection may occur due to occlusion or unreliable detection in complex scenes. This is especially true if the characteristics of the internal category objects are similar.

The multi-object tracking method according to an embodiment of the present invention may include evaluating the reliability of the tracking result (S830 ).

In step S830, the verification unit 230 may perform robust tracking by combining the correlation filter with a CNN-based function, particularly in a single object tracking problem. It is also necessary to compute the dot product and vector standards in the frequency domain, both of which are simple operations. Therefore, a correlation filter can be applied to the MOT to balance accuracy and processing time.

를 다음 수학식 8과 같이 계산할 수 있다.The verification unit 230 can be viewed as ridge regression because the correlation filter allows a simple closed solution. The key is that an increase in negative samples is used to improve the discriminative ability of the detection-specific tracking system while exploring the structure of the circular matrix to achieve high efficiency. The verification unit 230 finds the location by finding the maximum confidence score. By synchronizing this process with the high-speed process, the verification unit 230 has a reliability between the tracklet and the detection.

Can be calculated as in Equation 8 below.

[Equation 8]

은 역 이산 푸리에 변환을 나타내며,

는 요소 단위의 곱 연산자이며,

는 변수가 주파수 영역에 있음을 의미한다.here

Denotes the inverse discrete Fourier transform,

Is the element-wise multiplication operator,

Means that the variable is in the frequency domain.

The verification unit 230, the best CNN model for the MOT needs appropriately differentiated internal categories and offline training to improve processing time. To this end, the object detection unit 220 or the verification unit 230 according to an embodiment of the present invention can use the Pretrained ResNet-50 architecture by discarding the last layer and adding two completely connected layers. The first fully connected layer has 1024 units, then the ReLU, and the second item can be divided into 128 units. The network can be trained by the MARS data set. The MARS data set is a large person reconfirmation data set containing 1,261 identities and 1,191,003 images. The cosine soft max loss function can be used because all samples are pushed at the decision boundary in the direction of the parameterized class mean. That is, it can not only classify objects, but also converge functions according to different classification criteria, which can help in classifying internal categories. Therefore, the robustness of the CNN model can be improved.

The verification unit 230 is given a data set B = {(xi, yi)} Ni = 1 of given N training samples and related class labels. yi ∈ {1; 2; In ::: C}, the basic softmax classifier can select a class having the maximum probability for an object based on a parametric function as shown in Equation 9.

[Equation 9]

Here, f(x) is a feature expression extracted by a CNN model, which can be trained jointly with a classifier. The basic softmax classifier can be modified to create a compact cluster in the expression space with a small adaptation that can improve the robustness of the internal adaptive classification. L2 normalization must first be calculated for the final layer of the network to ensure that the representation is unit length. Weight wk, ∀k = 1, 2,… C should be standardized to unit length. The cosine softmax classifier can be expressed by Equation 10 below.

[Equation 10]

Where κ is a free scaling parameter. As a method of accelerating the convergence of the stochastic gradient drop, there is a method of separating the length of the weight vector κ from the direction.

를 볼 때, 클래스 조건부 확률이 수학식 11, 수학식 12와 같이 von Mises-Fisher (vMF) 분포를 따를 수 있다.The verification unit 230, from a generative point of view,

Looking at, the class conditional probability can follow the von Mises-Fisher (vMF) distribution as shown in Equation 11 and Equation 12.

[Equation 11]

[Equation 12]

Here, I _v represents the modified Bessel function of the first kind in order v. Consider p = 2 in one embodiment of the present invention. This distribution decreases from the circle to the von Mises distribution, so Equation 11 becomes Equation 13 as follows.

[Equation 13]

는 공유 농도 매개 변수다. 작은 C의 경우 분포가 거의 균일하므로 CNN 모델의 내부 범주 분류 기능을 향상시키기 위해 적절한

을 선택할 수 있다. 결과적으로, 사후 클래스 확률은 베이스 (Bayes) 규칙에 의해 수학식 14로 계산될 수 있다.Where I ₀ is the modified Bessel function of order 0

Is the shared concentration parameter. For small C, the distribution is almost uniform, so it is appropriate to improve the internal categorization of the CNN model.

You can choose As a result, the posterior class probability can be calculated by Equation 14 according to the Bayes rule.

[Equation 14]

주위로 정점을 이루고 붕괴하는 1 차원 구에 대한 등방성 확률 분포이다. 해당 코사인 소프트 맥스 손실 함수는 다음 수학식 15과 같이 쓸 수 있다.The verification unit 230, the von Mises distribution is the direction as the cosine similarity decreases

An isotropic probability distribution for a one-dimensional sphere that peaks and collapses around. The cosine soft max loss function can be written as in Equation 15 below.

[Equation 15]

값을 가진 기본 softmax로 적용하여 견고한 클러스터로 샘플을 밀어 넣을 수 있다. This is different

The sample can be pushed into a solid cluster by applying it as a default softmax with a value.

The multi-object tracking method according to an embodiment of the present invention may include re-identifying an object whose reliability evaluation is less than or equal to a threshold value and re-tracking the tracklet of the object (S840 ).

In step S840, the object detection unit 220 may re-identify an object whose reliability evaluation of the verification unit 230 is less than or equal to a threshold value and retrace the tracklet of the object. The object detection unit 220 may perform object tracking through data correlation using the appearance, shape, and motion information of the object. The object detection unit 220 may perform object tracking through data correlation using appearance and shape information of the object. The object detection unit 220 may update the tracklet using the appearance and shape information of the object according to the retrace result. The object detection unit 220 may update the appearance model of the object using the appearance, shape, and tracklet information of the object.

The object detection unit 220 may extract and learn the external characteristics of the identified object. The object detection unit 220 may generate an external model of the object using the appearance and shape information of the identified object. The object detection unit 220 may update the appearance model using the tracking and re-tracking results. The object detection unit 220 may generate and update the appearance model by machine learning the tracking and retrace results.

So far, the present invention has been focused on the preferred embodiments. Those skilled in the art to which the present invention pertains will appreciate that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered in terms of explanation, not limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent range should be interpreted as being included in the present invention.

Claims (9)

Obtaining an image of a region of interest;
Identifying a plurality of objects included in the image, and tracking tracklets of the objects;
Evaluating the reliability of the tracking result; And
And re-identifying an object whose reliability evaluation is below a threshold and retracking the tracklet of the object.
According to claim 1,
The image acquisition step,
Obtaining an image including the identified object and tracklet information; And,
Extract the external features of the identified object. Learning step; Multi-object tracking method in an image comprising a.
According to claim 1,
The first tracking step,
A method for tracking multiple objects in an image, characterized by tracking through data correlation using the appearance, shape, and motion information of the object.
According to claim 1,
The second tracking step,
A method for tracking multiple objects in an image, characterized by tracking through data correlation using the appearance and shape information of the object.
The method of claim 4,
Updating a tracklet using the appearance and shape information of the object according to the retrace result; And
And updating the appearance model of the object using the appearance, shape, and tracklet information of the object.
An image acquisition unit that acquires an image of a region of interest;
An object detection unit that identifies a plurality of objects included in the image and tracks a tracklet of the object, respectively;
Includes; a verification unit for evaluating the reliability of the tracking results,
The object detection unit,
Multi-object tracking system in an image that re-identifies an object whose reliability evaluation is below a threshold and retraces a tracklet.
The method of claim 6,
The image acquisition unit,
An image including the identified object and tracklet information is acquired, and external features of the identified object are extracted. Multi-object tracking system in an image characterized by learning.
The method of claim 6,
The object detection unit,
Multi-object tracking system in an image, characterized by tracking through data correlation using the appearance, shape, and motion information of the object.
The method of claim 8,
The object detection unit,
Update the tracklet using the appearance and shape information of the object according to the retrace result,
Multi-object tracking system in the image, characterized in that for updating the appearance model of the object using the appearance, shape and tracklet information of the object.

Images