KR20210092672A - Method and apparatus for tracking target - Google Patents

Abstract

Disclosed are a method and an apparatus for tracking a target. According to one embodiment, the apparatus for tracking the target includes at least one processor. The processor obtains a first depth feature from a target region image, and obtains a second depth feature from a search region image. The processor obtains a global response diagram between the first depth feature and the second depth feature. The processor obtains temporary bounding box information based on the global response diagram. The processor updates the second depth feature based on the temporary bounding box information to obtain the updated second depth feature. The processor obtains a plurality of local feature blocks based on the first depth feature. The processor obtains a local response diagram based on the local feature blocks and the updated second depth feature. The processor obtains output bounding box information based on the local response diagram.

Description

Target tracking method and apparatus {METHOD AND APPARATUS FOR TRACKING TARGET}

It relates to a technique for tracking a target in an image, and to a technique for tracking a target in stages.

Visual object tracking is one of the important fields of computer vision. The visual target tracking technique is to continuously predict the bounding box of a target in a subsequent frame image according to a first frame image in one video sequence and a given bounding box. The target may be an object or part of an object.

According to an embodiment, a target tracking method includes: obtaining a first depth feature from a target area image, and obtaining a second depth feature from a search area image; obtaining a global response diagram between the first depth feature and the second depth feature; obtaining temporary bounding box information based on the global response diagram; acquiring an updated second depth feature by updating the second depth feature based on the temporary bounding box information; obtaining a plurality of local feature blocks based on the first depth feature; obtaining a local response diagram based on the plurality of local feature blocks and the updated second depth feature; and obtaining output bounding box information based on the local response diagram.

The obtaining of the plurality of local feature blocks includes obtaining the plurality of local feature blocks by dividing the first depth feature or a third depth feature additionally extracted from the first depth feature, The obtaining of the response diagram may include: obtaining the local response diagram based on a fourth depth feature additionally extracted from the updated second depth feature or a correlation between the second depth feature and the plurality of local feature blocks; may include.

The obtaining of the local response diagrams based on the correlation includes: acquiring a plurality of local sub-response diagrams based on the correlation between each of the plurality of local feature blocks and the second depth feature or the fourth depth feature to do; and synthesizing the plurality of local sub-response diagrams to obtain the local response diagram.

The step of synthesizing the plurality of local sub-response diagrams to obtain the local response diagram may include: classifying the plurality of local feature blocks into target feature blocks or background feature blocks; and synthesizing the local sub-response diagram based on the classification result to obtain the local response diagram.

The classifying may include classifying the plurality of local feature blocks as a target feature block or a background feature block based on an overlap ratio of the temporary bounding box and each of the plurality of local feature blocks.

The output bounding box information includes a coordinate offset between the center position coordinates of the temporary bounding box and the center position coordinates of the output bounding box included in the temporary bounding box information, and a size offset between the size of the output bounding box and a preset size. , the obtaining of the output bounding box information includes outputting the temporary bounding box information as the output bounding box information when the sum of the absolute values of the coordinate offsets is greater than a threshold value, and the sum of the absolute values of the coordinate offsets. When it is less than or equal to this threshold, a result of summing the center position of the temporary bounding box and the coordinate offset and a result of summing the size of the temporary bounding box and the size offset may be output as the output bounding box information.

The obtaining of the temporary bounding box information includes outputting the coordinates having the greatest correlation in the global response diagram of the current frame as the coordinates of the center position of the temporary bounding box of the current frame, and outputting the estimated output bounding box in the previous frame. may be output as the size of the temporary bounding box of the current frame.

The step of obtaining the plurality of local feature blocks by dividing the third depth feature or the first depth feature additionally extracted from the first depth feature may include adding the first depth feature or the third depth feature to the plurality of local feature blocks. The local feature blocks may be divided so that they do not overlap, the plurality of local feature blocks may be divided such that they overlap, or the partition may be divided based on a preset block distribution.

A computer program according to an embodiment may be combined with hardware and stored in a computer-readable recording medium to execute the method.

According to an embodiment, the target tracking apparatus may include at least one processor. The processor may obtain a first depth feature from the target area image, and obtain a second depth feature from the search area image. The processor may obtain a global response diagram between the first depth feature and the second depth feature. The processor may obtain temporary bounding box information based on the global response diagram. The processor may acquire the updated second depth feature by updating the second depth feature based on the temporary bounding box information. The processor may obtain a plurality of local feature blocks based on the first depth feature. The processor may obtain a local response diagram based on the plurality of local feature blocks and the updated second depth feature. The processor may obtain the output bounding box information based on the local response diagram.

1 is a flowchart illustrating an overall operation of a method for tracking a target according to an embodiment.
2 is a flowchart illustrating an operation of a target tracking method according to an embodiment.
3 is a flowchart illustrating a method for tracking a target according to an embodiment.
4 is an exemplary diagram illustrating a global correlation operation according to the present invention.
5 is an exemplary diagram of a first stage of a method for tracking a target according to an embodiment.
6 is an exemplary diagram of a block partitioning method according to an embodiment.
7 is an exemplary diagram for block correlation operation according to an embodiment.
8 is an exemplary diagram in which interference suppression and response diagrams are fused according to an embodiment.
9 is an exemplary diagram of self-adaptive prediction according to an embodiment.
10 is an exemplary diagram of a second stage operation of a method for tracking a target according to an embodiment.
11 is an exemplary diagram for network training according to an embodiment.
12 shows a comparison of the effect difference between the block correlation method and the global correlation method and the block correlation method combining interference suppression according to an embodiment.
13 is a diagram illustrating a configuration of an apparatus for estimating a target according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

1 is a flowchart illustrating an overall operation of a method for tracking a target according to an embodiment.

According to an embodiment, the target tracking apparatus may track the target on successive images using two stages. The target tracking device may determine the target area image 101 and the search area image 103 . In the first stage 105 , the target tracking apparatus may perform coarse prediction 107 on an area matching the target area image 101 from the search area image 103 . Temporary bounding box information may be obtained as a result of the coarse prediction 107 . The target tracking apparatus may perform the accurate prediction 111 using the temporary bounding box information in the second stage 109 . Output bounding box information may be output as a result of the correct prediction 111 .

The target tracking apparatus performs target tracking by dividing two stages, and may track the target using block correlation and global correlation. The target tracking device may extract information for adjusting the temporary bounding box information by applying block correlation. Through this, the target tracking apparatus can achieve high-accuracy target tracking using fewer resources. Because it uses few resources, the target tracking device can perform high-accuracy and stable real-time tracking even in a mobile environment.

Hereinafter, the input image means an image input to the target tracking device. The input image may include a continuous frame, but is not limited thereto. The target refers to an object tracked in the input image. The target area image refers to a reference image indicating a target, and may be referred to as a template image. The search area image refers to an image as an area in which a target is searched.

The first depth feature means a feature extracted from the target area image. The second depth feature means a feature extracted from the search area image. The global correlation refers to a correlation operation between the entire first depth feature and the second depth feature. The global response diagram represents the result of global correlation. The third depth feature means a depth feature extracted through an additional convolution operation with respect to the first depth feature, and the fourth depth feature means a depth feature extracted through an additional convolution operation with respect to the second depth feature.

The local feature block means a result of segmenting the feature of the target region image. Here, the characteristic of the target region may include a first depth characteristic or a third depth characteristic derived from the first depth characteristic. The local feature block need not be a rectangular block and may include blocks of various shapes. The local correlation refers to a correlation operation between each of the local feature blocks and the second depth feature or the updated second depth feature. The local response diagram represents the result of local correlation.

The temporary bounding box information means information about the temporary bounding box as a result of the first stage, and the output bounding box information means information about the output bounding box as a result of the second stage. Each bounding box information may include location information and size information of the target within the search area.

2 is a flowchart illustrating an operation of a target tracking method according to an embodiment.

In Fig. 2, steps 201 to 205 correspond to the first stage, and steps 207 to 213 correspond to the second stage. Here, the distinction between the first stage and the second stage is for convenience of description, and it is not necessary to separate the stages.

According to an embodiment, in step 201 , the target tracking device may obtain a first depth feature from the target area image, and obtain a second depth feature from the search area image.

For example, the target tracking device may acquire successive frames. The target tracking apparatus may set a partial area of the first frame image among successive frames as the target area image by using the first neural network. For example, the first neural network may be a Siamese twin convolutional network, but is not limited thereto. The target tracking apparatus may extract the first depth feature from the target area image. The target tracking device sets the current frame as a search area image, and extracts a second depth feature from the search area image. Here, the target area image may be obtained by clipping the first frame image according to the manually set initial bounding box or the output bounding box for the previous frame, but is not limited thereto. Further, the first depth feature is a global feature of the target region image, and the second depth feature is a global feature of the search region image.

In step 203 , the target tracking device may obtain a global response diagram between the first depth feature and the second depth feature. For example, the target tracking apparatus may obtain a global response diagram by calculating a global correlation between the first depth feature and the second depth feature.

By applying the correlation operation to the two images, a response diagram Y representing the similarity between the two images can be obtained. The larger the similarity value, the higher the similarity between the corresponding area of the search area image Z and the target area image X is. For example, the correlation operation may be performed by Equation (1).

In Equation 1, h and w represent the size of the image X, and i, j, u, and v represent the coordinates of the image, respectively.

In step 205, the target tracking device may obtain temporary bounding box information based on the global response diagram. The target tracking device outputs the coordinates having the greatest correlation in the global response diagram of the current frame as the center position coordinates of the temporary bounding box of the current frame, and sets the size of the output bounding box estimated in the previous frame to the temporary bounding box of the current frame. It can be output as the size of .

In step 207 , the target tracking device may acquire an updated second depth feature by updating the second depth feature based on the temporary bounding box information. For example, the target tracking apparatus may obtain the search area image of the reduced area by clipping the search area image according to the temporary bounding box. The target tracking apparatus may update the second depth feature by extracting the depth feature from the search region image of the reduced region.

In step 209 , the target tracking device may obtain a plurality of local feature blocks based on the first depth feature. The target tracking apparatus may obtain a plurality of local feature blocks by dividing the third depth feature or the first depth feature additionally extracted from the first depth feature. The target tracking apparatus may additionally extract the third depth feature by inputting the first depth feature into the second neural network.

The target tracking apparatus may divide the first depth feature or the third depth feature into a plurality of local feature blocks. The target tracking apparatus may divide the first depth feature or the third depth feature so that a plurality of local feature blocks do not overlap, divide a plurality of local feature blocks so that they overlap, or divide based on a preset block distribution.

Here, the preset block distribution may be an artificially determined distribution or a distribution derived by a learned neural network. The artificially determined distribution may include a Gaussian distribution. A neural network that outputs a distribution can be trained in the direction of finding optimized parameters by targeting parameters of a specific distribution such as a Gaussian distribution (eg, the mean and variance of a Gaussian distribution).

In step 211 , the target tracking device may obtain a local response diagram based on the plurality of local feature blocks and the updated second depth feature. The target tracking apparatus may obtain a local response diagram based on a fourth depth feature additionally extracted from the updated second depth feature or a correlation between the second depth feature and a plurality of local feature blocks. The target tracking apparatus may additionally extract the fourth depth feature by inputting the second depth feature to the second neural network.

The target tracking apparatus may obtain a plurality of local sub-response diagrams based on a correlation between each of the plurality of local feature blocks and the second depth feature or the fourth depth feature.

The target tracking device may obtain a local response diagram by synthesizing a plurality of local sub-response diagrams. The target tracking apparatus may classify the plurality of local feature blocks as target feature blocks or background feature blocks. The plurality of local feature blocks may be classified as a target feature block or a background feature block based on an overlap ratio of the temporary bounding box and each of the plurality of local feature blocks. The target tracking device may obtain a local response diagram by synthesizing a local sub-response diagram based on the classification result.

In step 213, the target tracking device may obtain the output bounding box information based on the local response diagram. The output bounding box information may include a coordinate offset between the center position coordinates of the temporary bounding box and the center position coordinates of the output bounding box included in the temporary bounding box information, and a size offset between the size of the output bounding box and a preset size. When the sum of the absolute values of the coordinate offsets is greater than the threshold value, the target tracking apparatus may output temporary bounding box information as output bounding box information. When the sum of the absolute values of the coordinate offsets is less than or equal to the threshold, the target tracking device outputs the result of summing the center position and the coordinate offset of the temporary bounding box and the result of summing the size and size offset of the temporary bounding box as output bounding box information can do.

3 is a flowchart illustrating a method for tracking a target according to an embodiment.

As shown in Fig. 3, the target tracking method includes two stages. In the first stage, a coarse prediction 107 is performed. In the first stage 105 , a target target image 101 and a search area image 103 may be determined. The target tracking apparatus extracts global features 311 and 312 of the target area image 101 and the search area image 103, respectively, and calculates a global correlation 313 for the extracted global features 311 and 312 to obtain a global response You can get a diagram. The target tracking device may acquire temporary bounding box information based on the global response diagram.

In the second stage, an accurate prediction 111 is made. The target tracking device obtains a plurality of local feature blocks 321 and 322 by segmenting the features of the target region image, and includes a plurality of local feature blocks 321 and 322 that are the segmented features of the target region image and the updated search region image. A local response diagram can be obtained by computing the block correlation 323 between the features. The target tracking device may output the output bounding box information according to the local response diagram.

4 is an exemplary diagram of a global correlation operation according to an embodiment.

Referring to FIG. 4 , the target tracking apparatus may perform a correlation operation between _{the first depth feature F T} 401 and the second depth feature F _{St 402 .} The target tracking device may calculate the correlation at each location while sliding _{the first depth feature F T} 401 relative to the second depth feature F _{St 402 .} The target tracking device may obtain the global response diagram 403 through global correlation.

5 is an exemplary diagram of a first stage of a method for tracking a target according to an embodiment.

In the first stage 105 , the target tracking device may output a temporary bounding box. To this end, the target tracking apparatus may perform feature extraction, global correlation 313 , and feature clipping 505 .

를 포함할 수 있다. 타겟 추적 장치는 타겟 영역 이미지 Z(101)를 컨벌루션 뉴럴 네트워크

에 입력하여 제1 깊이 특징

(Z) 을 출력할 수 있다. 타겟 추적 장치는 검색 영역 이미지 X(103)를 컨벌루션 뉴럴 네트워크

에 입력하여 제2 깊이 특징

(X)을 출력할 수 있다. 예를 들어, 컨벌루션 뉴럴 네트워크

는 샴쌍둥이 컨볼루션 네트워크일 수 있으며, 도 5의 두개의 분기(branch)의 매개 변수는 입력되는 이미지가 동일한 특징 공간에 매핑되도록 공유될 수 있다.The target tracking apparatus may extract features from each of the target area image 101 and the search area image 103 using the first neural network. The first neural network is a convolutional neural network

may include. The target tracking device converts the target area image Z 101 into a convolutional neural network

Enter in the first depth feature

(Z) can be printed. The target tracking device converts the search area image X 103 into a convolutional neural network

Enter in the second depth feature

(X) can be printed. For example, convolutional neural networks

may be a Siamese twin convolution network, and the parameters of the two branches of FIG. 5 may be shared so that an input image is mapped to the same feature space.

(Z) 과 제2 깊이 특징

(X)에 대해 글로벌 상관(313)을 수행하여 글로벌 응답 다이어그램 f(501)을 출력할 수 있다. 타겟 추적 장치는 글로벌 응답 다이어그램 f(501)을 기초로 제1 스테이지 예측 결과인 임시 바운딩 박스 정보 P1(503)을 출력할 수 있다. 타겟 추적 장치는 수학식 2를 이용하여 제1 깊이 특징

(Z) 과 제2 깊이 특징

(X) 간의 글로벌 응답 다이어그램을 획득할 수 있다.The target tracking device extracts the first depth feature

(Z) and the second depth feature

A global correlation 313 may be performed on (X) to output a global response diagram f(501). The target tracking apparatus may output temporary bounding box information P1 503 that is a first stage prediction result based on the global response diagram f 501 . The target tracking device uses Equation 2 to determine the first depth feature

(Z) and the second depth feature

A global response diagram between (X) can be obtained.

The target tracking device may output the position having the largest value in the global response diagram as position information of the temporary bounding box. The target tracking apparatus may output the size of the output bounding box of the previous frame as size information of the temporary bounding box.

과 제1 깊이 특징

(Z) 을 출력할 수 있다. 임시 바운딩 박스 정보 P1(503)은 P1= (x1, y1, w1, h1)로 표현될 수 있다. 여기서, x1와 y1는 각각 제1 스테이지의 임시 바운딩 박스의 중심 위치의 가로 좌표와 세로 좌표이고, w1과 h1은 각각 임시 바운딩 박스의 폭과 높이를 의미한다.The target tracking apparatus may perform feature clipping 505 on the first stage prediction result P1 503 . The target tracking apparatus may update the second depth feature by extracting the depth feature from the search region image clipped by the feature clipping 505 . As a result, the target tracking device has the updated second depth feature in the first stage 105 .

and first depth feature

(Z) can be printed. The temporary bounding box information P1 503 may be expressed as P1 = (x1, y1, w1, h1). Here, x1 and y1 are the horizontal and vertical coordinates of the center position of the temporary bounding box of the first stage, respectively, and w1 and h1 are the width and height of the temporary bounding box, respectively.

을 획득할 수 있다.The target tracking apparatus may obtain a smaller search area image X' by clipping the search area image X according to the center position and size of the temporary bounding box. The target tracking device extracts the depth feature of the search area image X' and updates the second depth feature

can be obtained.

6 is an exemplary diagram of a block partitioning method according to an embodiment.

Referring to FIG. 6 , various types of segmentation methods 600 performed by a target tracking apparatus are disclosed. The target tracking apparatus may divide the first depth feature 401 or the third depth feature (not shown) into a plurality of local feature blocks. According to the non-overlapping image segmentation 601 method, the target tracking apparatus may segment the first depth feature 401 or the third depth feature so that a plurality of local feature blocks do not overlap. According to the overlapped image segmentation 603 method, the target tracking apparatus may segment the first depth feature 401 or the third depth feature so that a plurality of local feature blocks overlap. According to the division 605 method based on a predetermined block distribution, the target tracking apparatus may divide the first depth feature 401 or the third depth characteristic based on a preset block distribution.

7 is an exemplary diagram for block correlation operation according to an embodiment.

Although FIG. 7 is illustrated assuming a first depth feature 401 for convenience of description, a block correlation operation may be performed with respect to the third depth feature. Referring to FIG. 7 , a region feature segmentation may be performed on the first depth feature 401 . The first depth feature 401 may be divided into a plurality of local feature blocks.

The target tracking device may calculate a correlation between each of the local features and the second depth feature 402 . The target tracking device may output a plurality of local sub-diagrams 701 , 702 , 703 , 704 , 705 , 706 , 707 , 708 and 709 as a result of the calculation. The target tracking device may synthesize a plurality of local sub-diagrams 701 , 702 , 703 , 704 , 705 , 706 , 707 , 708 , and 709 to obtain a local response diagram 711 .

For example, the target tracking apparatus may classify each local feature block among a plurality of local feature blocks as a target feature block or a background feature block. The target tracking device may obtain a local sub-response diagram corresponding to the target feature block, and obtain a local sub-response diagram corresponding to the background feature block. The target tracking device may output a local response diagram 711 by synthesizing all local sub-response diagrams.

In the target area image, a background area exists in addition to the target area, and the characteristics of the background area may affect stability and accuracy of target tracking. The target tracking apparatus may increase the stability and accuracy of the target tracking through the method shown in FIG. 7 . The target tracking apparatus can effectively reduce background interference by classifying and synthesizing a plurality of local feature blocks into a target feature block and a background feature block.

8 is an exemplary diagram in which interference suppression and response diagrams are fused according to an embodiment.

The target tracking device may obtain a local response diagram by synthesizing a plurality of local sub-response diagrams. The target tracking apparatus may classify the plurality of local feature blocks as target feature blocks or background feature blocks. Based on the overlap ratio of each of the temporary bounding box 801 and the plurality of local feature blocks 801, 802, 803, 804, 805, 806, 807, 808, 809, the plurality of local feature blocks 801, 802, 803, 804, 805, 806, 807, 808, 809) may be classified as a target feature block or a background feature block.

The target tracking device is configured for each local feature block 801, 802, 803, 804, 805, 806, 807, 808, 809 and each local feature block 801, 802, 803, in the overlapping area between the temporary bounding box 801, Each local feature block 801, 802, 803, 804, 805, 806, 807, 808, 809 is used as a target feature block or a background feature block, depending on the proportion of 804, 805, 806, 807, 808, 809. can be classified. For example, based on the corrected temporary bounding box 801 on the target region image, if the local feature block has an area of p% or more in the temporary bounding box, it is classified as a target target feature block, and the local feature blocks 801 and 802 , 803, 804, 805, 806, 807, 808, 809) and the temporary bounding box 801 may be classified as a background feature block when the overlapping portion is less than p%. Here, p may be a predetermined threshold value.

The target tracking apparatus may obtain a local response diagram by synthesizing the local sub-response diagram corresponding to the target feature block and the local sub-response diagram corresponding to the background feature block using Equation (3).

In Equation 3, S is a local response diagram, So is a local sub-response diagram corresponding to a target feature block, Sb is a sub-response diagram corresponding to a background feature block, no is the number of target feature blocks, and nb is The number of background feature blocks.

The target tracking device may output an output bounding box based on the local response diagram. The target tracking device may predict the position offset and the size offset of the temporary bounding box 801 according to the local response diagram. The target tracking device may output an output bounding box according to the predicted position offset and size offset.

For example, the target tracking device may process a local response diagram using a third neural network, and predict a position offset and a size offset of an output bounding box. The third neural network may be different from the above-mentioned first neural network and second neural network. Here, the prediction result of the output bounding box may include location information and size information of the target bounding box. Hereinafter, the process of outputting the output bounding box based on the local response diagram may be referred to as a self-adaptive prediction process.

9 is an exemplary diagram of self-adaptive prediction according to an embodiment.

The target tracking device may process the local response diagram (S) 901 using the convolutional neural networks 902 and 903 . The target tracking device may output an offset D 905 through the offset prediction 904 . The target tracking apparatus may perform self-adaptive prediction 907 based on the first stage prediction result P _{1 906 and the offset D 905 . A second stage prediction result P 2} 908 may be output as a result of the self-adaptive prediction 907 .

The target tracking device may predict an offset D = (dx, dy, dw, dh), and the offset includes a position offset and a magnitude offset. For example, the position offset may be a coordinate offset between the center position coordinates of the output bounding box of the second stage and the center position coordinates of the temporary bounding box of the first stage, and the size offset is a predetermined value between the output bounding box of the second stage and the predetermined position offset. It may be a size offset between the bounding boxes.

The target tracking apparatus may obtain the prediction result of the output bounding box of the second stage according to the predicted position offset and the size offset. When the sum of the absolute values of the coordinate offsets is greater than a preset threshold, the target tracking apparatus may output the prediction result of the temporary bounding box of the first stage as the prediction result of the output bounding box of the second stage.

When the sum of the absolute values of the coordinate offsets is less than or equal to a preset threshold, the target tracking device adds the predicted position offset and the center position of the temporary bounding box of the first stage, and adds the size of the predetermined bounding box and the predicted size offset A prediction result of the output bounding box of the second stage may be obtained.

For example, the prediction result of the temporary bounding box of the first stage is P1 = (x1, y1, w1, h1), and the size of the predefined bounding box is (w0, h0) (width w0, height h0) In this case, the prediction result of the output bounding box of the second stage may be P2=(x1+dx, y1+dy, w0+dw, h0+dh).

10 is an exemplary diagram of a second stage operation of a method for tracking a target according to an embodiment.

및 갱신된 제2 깊이 특징

을 획득할 수 있다. 타겟 추적 장치는 제1 깊이 특징

및 갱신된 제2 깊이 특징

을 컨벌루션 네트워크(1003)에 입력할 수 있다. 타겟 추적 장치는 블록 상과(1004)을 수행할 수 있다. 타겟 추적 장치는 간섭 억제를 수행하고 로컬 서브 응답 다이어그램을 융합할 수 있다. 타겟 추적 장치는 로컬 서브 응답 다이어그램(1006)을 획득할 수 있다. 타겟 추적 장치는 자가 적응 예측(1007)을 수행할 수 있다. 타겟 추적 장치는 제2 스테이지의 예측 결과 P₂를 출력할 수 있다. The target tracking device provides a first depth feature through a first stage.

and an updated second depth feature.

can be obtained. The target tracking device has a first depth feature

and an updated second depth feature.

may be input to the convolutional network 1003 . The target tracking device may perform block phase 1004 . The target tracking device may perform interference suppression and fuse local sub-response diagrams. The target tracking device may obtain a local sub-response diagram 1006 . The target tracking device may perform self-adaptive prediction 1007 . The target tracking apparatus may output _{the prediction result P 2 of the second stage.}

11 is an exemplary diagram for network training according to an embodiment.

The target tracking device may track the target object using a cascade network (including a first neural network, a second neural network, and a third neural network). Cascade networks can be trained using multiple supervisory signals. Here, the multi-supervision signal includes a global response diagram, a local response diagram, and a target bounding box.

The steps performed in the following training process may be similarly applied in some way in the inference process. Multiple supervisory signals can be used to optimize the loss value of the loss function. Multi-supervised signals can be used to learn the parameters of the network through iterative cyclic learning.

In the learning process using the multiple supervision signals, the learning apparatus may perform the first stage tracking 1101 on the template image 1101 and the search area image 1102 to obtain the global response diagram 1103 . In the global response diagram, it may be set to +1 when the distance from the central position is less than a specific threshold, and set to -1 when the distance from the central position is greater than a specific threshold. The learning device may output a coarse prediction box 1105 based on the global response diagram 1103 . The learning device may output the shared feature 1104 based on the first stage tracking 1102 result and the clipped coarse prediction box 1105 .

The learning device may proceed with the second stage tracking 1108 . The learning apparatus may obtain a segmentation result (a segmentation algorithm or manually obtained) for a target on the search area image. The learning apparatus may obtain the supervision signal of the local response diagram 1109 by performing distance transformation on the segmentation result and numerically normalizing the distance transformation map.

The learning apparatus may obtain an accurate prediction result through the self-adaptive prediction 1106 . In the learning process, the global response diagram, the local response diagram 1109 and the target bounding box can be used as supervisory signals, and through iterative cyclic learning, the loss function can be optimized to learn the parameters of the cascade network.

For example, the learning apparatus may acquire temporary bounding box information by inputting an image pair (including a template image and a search region image) extracted from the same video sequence into the neural network of the first stage. The learning apparatus may calculate the loss (Loss0) between the predicted value and the actual value of the global response diagram using binary cross entropy.

The learning apparatus may obtain the local response diagram 1109 by inputting the shared feature 1104 of the first stage to the neural network of the stage 2 based on the coarse prediction result. The learning apparatus may obtain an accurate prediction result of the second stage based on the local response diagram. The learning apparatus may measure the loss (Loss1) of the predicted value and the actual value of the local response diagram using KL divergence (Kullback-leibler divergence). The learning apparatus may measure the loss (Loss2) of the accurately predicted bounding box and the actual box using the L1 distance. The learning apparatus may learn the parameters of the neural network by optimizing the loss (Loss=Loss0+(a1)*Loss1+(a2)*Loss2). Here, a1 and a2 represent the weight of the loss, respectively.

12 shows a comparison of the effect difference between the block correlation method and the global correlation method and the block correlation method combining interference suppression according to an embodiment.

Referring to FIG. 12 , a result 1203 of interference suppression performed in a global correlation response diagram 1201 , a block correlation response diagram 1202 , and a block correlation response diagram are shown. The target tracking apparatus may effectively extract detailed information of a target by performing interference suppression based on block correlation to further improve tracking accuracy.

13 is a diagram illustrating a configuration of an apparatus for estimating a target according to an embodiment.

According to one embodiment, the target tracking device comprises at least one processor.

The processor may obtain a first depth feature from the target area image, and obtain a second depth feature from the search area image. The processor may obtain a global response diagram between the first depth feature and the second depth feature. The processor may obtain temporary bounding box information based on the global response diagram. The processor may acquire the updated second depth feature by updating the second depth feature based on the temporary bounding box information. The processor may obtain a plurality of local feature blocks based on the first depth feature. The processor may obtain a local response diagram based on the plurality of local feature blocks and the updated second depth feature. The processor may obtain the output bounding box information based on the local response diagram.

As described above, a target tracking method and a target tracking apparatus according to an embodiment have been described with reference to FIGS. 1 to 13 . It should be understood that each operation of the device shown in FIG. 13 may be implemented in software, hardware, firmware, or any combination thereof to perform a specific function. For example, the operation of the target estimation apparatus may be implemented by a special integrated circuit, pure software code, or a module combining software and hardware. For example, the target tracking device may be, but is not limited to, a PC computer, a tablet device, a personal digital assistant, a smart phone, a web application, or other device capable of executing program commands.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and a software application running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment, or are known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those of ordinary skill in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

obtaining a first depth feature from the target area image and obtaining a second depth feature from the search area image;
obtaining a global response diagram between the first depth feature and the second depth feature;
obtaining temporary bounding box information based on the global response diagram;
acquiring an updated second depth feature by updating the second depth feature based on the temporary bounding box information;
obtaining a plurality of local feature blocks based on the first depth feature;
obtaining a local response diagram based on the plurality of local feature blocks and the updated second depth feature; and
obtaining output bounding box information based on the local response diagram;
How to track your target. According to claim 1,
The obtaining of the plurality of local feature blocks includes:
obtaining the plurality of local feature blocks by dividing the first depth feature or a third depth feature additionally extracted from the first depth feature,
Obtaining the local response diagram comprises:
obtaining the local response diagram based on a fourth depth feature additionally extracted from the updated second depth feature or a correlation between the second depth feature and the plurality of local feature blocks;
How to track your target. 3. The method of claim 2,
Obtaining the local response diagram based on the correlation comprises:
obtaining a plurality of local sub-response diagrams based on the correlation between each of the plurality of local feature blocks and the second depth feature or the fourth depth feature; and
synthesizing the plurality of local sub-response diagrams to obtain the local sub-response diagram,
How to track your target. 4. The method of claim 3,
The step of synthesizing the plurality of local sub-response diagrams to obtain the local response diagram comprises:
classifying the plurality of local feature blocks into target feature blocks or background feature blocks; and
Comprising the step of synthesizing the local sub-response diagram based on the classification result to obtain the local response diagram,
How to track your target. 5. The method of claim 4,
The classification step is
classifying the plurality of local feature blocks into a target feature block or a background feature block based on an overlap ratio of the temporary bounding box and each of the plurality of local feature blocks,
How to track your target. 4. The method of claim 3,
The output bounding box information includes a coordinate offset between the center position coordinates of the temporary bounding box and the center position coordinates of the output bounding box included in the temporary bounding box information, and a size offset between the size of the output bounding box and a preset size. ,
The step of obtaining the output bounding box information comprises:
When the sum of the absolute values of the coordinate offsets is greater than a threshold value, outputting the temporary bounding box information as the output bounding box information,
When the sum of the absolute values of the coordinate offsets is less than or equal to a threshold value, a result of summing the center position of the temporary bounding box and the coordinate offset and a result of summing the size of the temporary bounding box and the size offset are displayed as the output bounding box information output as,
How to track your target. 7. The method of claim 6,
The step of obtaining the temporary bounding box information comprises:
The coordinates having the greatest correlation in the global response diagram of the current frame are output as the coordinates of the center position of the temporary bounding box of the current frame, and the size of the output bounding box estimated in the previous frame is the size of the temporary bounding box of the current frame. output as size,
How to track your target. 3. The method of claim 2,
obtaining the plurality of local feature blocks by dividing the first depth feature or a third depth feature additionally extracted from the first depth feature,
dividing the first depth feature or the third depth feature so that the plurality of local feature blocks do not overlap, partition the plurality of local feature blocks so that they overlap, or partition based on a preset block distribution,
How to track your target. A computer program stored in a computer-readable recording medium in combination with hardware to execute the method of any one of claims 1 to 8. at least one processor;
The processor is
obtain a first depth feature from the target area image, obtain a second depth feature from the search area image,
obtain a global response diagram between the first depth feature and the second depth feature,
obtaining temporary bounding box information based on the global response diagram;
obtaining an updated second depth feature by updating the second depth feature based on the temporary bounding box information;
obtaining a plurality of local feature blocks based on the first depth feature,
obtain a local response diagram based on the plurality of local feature blocks and the updated second depth feature,
obtaining output bounding box information based on the local response diagram,
target tracking device.

Images