KR20240045055A - Method and electronic device for object tracking - Google Patents

Abstract

In this disclosure, a method for tracking at least one object is provided. A method of tracking at least one object according to an embodiment of the present disclosure includes obtaining an image, extracting a feature map for performing a plurality of tasks related to object tracking from the image, and using the extracted feature map. extracting location information indicating the location of at least one object, an identification feature enabling identification of the at least one object, and an angle of the body direction toward which the body of the at least one object faces, and the location of the at least one object. The method may include tracking at least one object based on the information, identifying features, and body orientation angle.

Description

METHOD AND ELECTRONIC DEVICE FOR OBJECT TRACKING}

Embodiments of the present disclosure relate to a method for tracking an object, an electronic device for tracking an object, and a computer-readable recording medium on which a program for performing the method for tracking an object is recorded on a computer.

In the field of computer vision, research on object tracking technology is actively underway. Object tracking technology is a technology that detects at least one object in an image or image sequence acquired by an electronic device and simultaneously tracks the path of each of the at least one object. In order to track at least one object simultaneously, a process of detecting at least one object and matching at least one detected object with at least one object that was previously being tracked is required.

During object tracking, occlusion may occur or the object may leave the camera's field of view and then reappear. Occlusion includes occlusion that occurs when one part of an object obscures another part of the object, occlusion that occurs between objects, and occlusion that occurs due to structures in the background.

According to one aspect of the present disclosure, a method for tracking at least one object is provided. The method includes acquiring an image. The method includes extracting a feature map from an image to perform a plurality of tasks related to object tracking. The method uses the extracted feature map to extract location information indicating the location of at least one object, an identification feature enabling identification of at least one object, and the angle of the body direction toward which the body of at least one object faces. It includes steps to: The method includes tracking at least one object based on location information, identifying features, and body orientation angle of the at least one object.

According to one aspect of the present disclosure, an electronic device for tracking at least one object is provided. The electronic device includes a communication interface, a memory storing at least one instruction, and at least one processor executing the at least one instruction stored in the memory. The at least one processor may acquire an image by executing the at least one instruction. The at least one processor may extract a feature map for performing a plurality of tasks related to object tracking from an image by executing the at least one instruction. By executing the at least one instruction, the at least one processor generates location information indicating the location of at least one object using the extracted feature map, an identification feature enabling identification of at least one object, and at least one The angle of the body direction toward which the object's body faces can be extracted. By executing the at least one instruction, the at least one processor may track at least one object based on location information, identification characteristics, and body direction angle of the at least one object.

According to one aspect of the present disclosure, a computer-readable recording medium is provided on which a program for tracking at least one object and performing any one of the above and below-described methods on a computer is recorded.

FIG. 1 is a diagram illustrating a method by which an electronic device tracks an object using object tracking information according to an embodiment of the present disclosure.
Figure 2 is a flowchart showing a method by which an electronic device tracks an object according to an embodiment of the present disclosure.
Figure 3 is a flowchart showing a process of tracking an object using an object tracking model according to an embodiment of the present disclosure.
Figure 4 is an exemplary diagram showing training image data of an object tracking model according to an embodiment of the present disclosure.
FIG. 5A is a diagram illustrating a method of extracting location information, identification features, and body direction angle of an object using each head of an object tracking model according to an embodiment of the present disclosure.
FIG. 5B is a diagram illustrating a method of extracting location information of an object using a detection head of an object tracking model according to an embodiment of the present disclosure.
Figure 6 is a diagram showing object detection information and object tracking information according to an embodiment of the present disclosure.
Figure 7 is a flowchart showing a data association process according to an embodiment of the present disclosure.
FIG. 8A is a flowchart showing a first data association process according to an embodiment of the present disclosure.
FIG. 8B is an exemplary diagram illustrating a first data association process according to an embodiment of the present disclosure.
Figure 9 is a flowchart showing a second data association process according to an embodiment of the present disclosure.
Figure 10 is a flowchart showing a third data association process according to an embodiment of the present disclosure.
FIG. 11A is a flowchart showing a tracking information management process according to an embodiment of the present disclosure.
FIG. 11B is a diagram illustrating a process of updating tracking information of an object according to an embodiment of the present disclosure.
FIG. 12A is a flowchart illustrating a process for updating the tracking state of an object according to an embodiment of the present disclosure.
FIG. 12B is a flowchart illustrating a process of updating the tracking state of an object according to an embodiment of the present disclosure.
Figure 13 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.
Figure 14 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.

In the present disclosure, the expression “at least one of a, b, or c” refers to “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b and c", or variations thereof.

The terms used in the present disclosure have selected general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but this may vary depending on the intention or precedents of those skilled in the art, the emergence of new technologies, etc. In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant description. Therefore, the terms used in this disclosure should be defined based on the meaning of the term and the overall content of this disclosure, rather than simply the name of the term.

Singular expressions may include plural expressions, unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as generally understood by a person of ordinary skill in the technical field described herein. Additionally, terms including ordinal numbers, such as 'first' or 'second', used in this specification may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When it is said that a part "includes" a certain element throughout the specification, this means that, unless specifically stated to the contrary, it does not exclude other elements but may further include other elements. Additionally, terms such as “unit” and “module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software.

Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may consist of one or multiple processors. At this time, one or more processors may be a general-purpose processor such as a CPU, AP, or DSP (Digital Signal Processor), a graphics-specific processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-specific processor such as an NPU. One or more processors control input data to be processed according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, when one or more processors are dedicated artificial intelligence processors, the artificial intelligence dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

Predefined operation rules or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that the basic artificial intelligence model is learned using a large number of learning data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform the desired characteristics (or purpose). It means burden. This learning may be performed on the device itself that performs the artificial intelligence according to the present disclosure, or may be performed through a separate server and/or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

An artificial intelligence model may be composed of multiple neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and neural network calculation is performed through calculation between the calculation result of the previous layer and the plurality of weights. Multiple weights of multiple neural network layers can be optimized by the learning results of the artificial intelligence model. For example, a plurality of weights may be updated so that loss or cost values obtained from the artificial intelligence model are reduced or minimized during the learning process. Artificial neural networks may include deep neural networks (DNN), for example, Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the examples described above.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement the present invention. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present disclosure in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification. In addition, the reference numerals used in each drawing are only for explaining each drawing, and the different reference numerals used in each of the different drawings are not intended to indicate different elements.

Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

FIG. 1 is a diagram illustrating a method by which an electronic device tracks an object using object tracking information according to an embodiment of the present disclosure.

Referring to FIG. 1, an electronic device 2000 according to an embodiment of the present disclosure may track at least one object using object tracking information. In one embodiment, tracking an object means that the electronic device 2000 acquires an image in real time, detects an object in the acquired image, and tracks the path of the object. For example, the electronic device 2000 may acquire an image in real time, detect an object every frame, and track the path of the object. The acquisition time of each image may be recorded in the electronic device 2000 in units of frames. For example, each image may include a timestamp, and the electronic device 2000 may record the acquisition time of the image based on the timestamp. According to one embodiment, when a plurality of objects are detected in an image acquired in real time, the electronic device 2000 may simultaneously track the paths of each object. At this time, the electronic device 2000 may match detected objects with previously tracked objects and update the tracking information of the objects.

In one embodiment, the tracking information of the object may include, but is not limited to, the object class, the ID of the object, the path of the object, the list of identifying characteristics of the object, and the last matching time of the object. In one embodiment, the last matching time of an object may mean the time of acquisition of an image at which the object was last matched with one of the objects being tracked. A detailed description of the information included in the object tracking information will be described later with reference to FIG. 6. The electronic device 2000 may store a tracking database in memory. The tracking database according to one embodiment may include tracking information on at least one object being tracked by the electronic device 2000.

An object according to one embodiment may be an object whose identification features are distinguished depending on the direction of the body. For example, the object class may include, but is not limited to, people, vehicles, and animals. For example, if the class of the object is a person, the front, side, and back of the person are distinguished, and identification features for the front, side, and back can also be distinguished accordingly.

In one embodiment, an identification feature refers to a feature vector that can identify each object within the same class. For example, the electronic device 2000 may detect objects from images acquired in real time and extract identification features for the detected objects. At this time, identification features for the same object extracted by the electronic device 2000 from images may have a high degree of similarity. For example, identification features for the same object may have a cosine similarity close to 1. On the other hand, identification features for different objects may have low similarity.

Referring to FIG. 1, the electronic device 2000 may detect the first object 110 with ID 5. The electronic device 2000 according to one embodiment may identify the first object 110 using the identification feature list 130 of the first object 110 included in the tracking information of the first object 110. . In one embodiment, the identification feature list 130 refers to a list including at least one identification feature corresponding to each of the latest identification feature of an object and a representative angle of the body direction. In one embodiment, the representative angle refers to angles having a predetermined angle interval set to extract the angle of the body direction using a classification method when detecting an object. For example, when the electronic device 2000 classifies angles in the body direction into 10, representative angles in the body direction may be set at intervals of 36 degrees. Hereinafter, for convenience of explanation, an example will be given where the electronic device 2000 divides the angles in the body direction into four and the representative angles in the body direction are set to 0 degrees, 90 degrees, 180 degrees, and 270 degrees. .

For example, the identification feature list may include all of the identification features corresponding to each representative angle, or may include only some of the identification features corresponding to each representative angle. While the first object 110 is being tracked by the electronic device 2000, if the first object 110 whose body direction angle is 90 degrees is not detected, the identification feature list 130 of the first object 110 may not include an identifying feature corresponding to 90 degrees. However, if the first object 110 whose body direction angle is 90 degrees is detected later, the electronic device 2000 places the extracted identification feature at 90 degrees in the identification feature list 130 of the first object 110. Corresponding identification features can be updated.

In one embodiment, when detecting the first object 110, the electronic device 2000 may detect the angle of the body direction of the first object 110. For example, in the case of the first object 110 shown in FIG. 1, the body direction angle may be detected as 180 degrees. The electronic device 2000 according to an embodiment may use the extracted identification feature as the latest identification feature in the identification feature list 130 of the first object 110 and the detected body direction angle of the first object 110. Identification characteristics can be updated. For example, when the body direction angle of the first object 110 is detected as 180 degrees, the electronic device 2000 selects the latest identification feature from the identification feature list 130 of the first object 110 as the extracted identification feature. And the identification feature corresponding to 180 degrees of the first object 110 may be updated.

According to one embodiment, the first object 110 may disappear off the screen after being detected by the electronic device 2000 in the acquired image. The first object 110 may disappear off the screen and then reappear on the screen after a certain period of time. When the first object 110 reappears on the screen, the first object 110 may reappear with a body direction angle that is different from the body direction angle at the last detection time of the first object 110. When the direction of the body of the first object 110 changes, the appearance of the first object 110 may change. If the direction of the body of the first object 110 is not considered, the identification characteristics of the first object 110 extracted from the image obtained as the appearance changes even though it is the same object, and the 1 The similarity between the identification features of the object 110 may be low, and there may be difficulty in tracking the object.

The electronic device 2000 according to one embodiment may detect the second object 120. In this case, the electronic device 2000 may detect the angle of the body direction of the second object 120. For example, in the case of the second object 120 shown in FIG. 1, the body direction angle may be detected as 0 degrees. The electronic device 2000 may compare the extracted identification features of the second object 120 with the identification features corresponding to 0 degrees among the identification feature list included in each tracking information of the objects being tracked. If the extracted identification feature of the second object 120 has the highest similarity to the identification feature corresponding to 0 degrees in the identification feature list 130 of the first object 110, the electronic device 2000 selects the second object ( 120 is determined to be the first object 110, and tracking information of the first object 110 may be updated. For example, the electronic device 2000 may update the latest identification feature and the identification feature corresponding to 0 degrees among the identification feature list 130 of the first object 110 with the identification feature extracted from the acquired image.

As described above, if identification features according to the body direction angle are stored for each object and identification features corresponding to the same body direction angle are compared when tracking an object, various effects can be obtained, including the effect of increasing the identification rate of the object. You can. Hereinafter, a method by which the electronic device 2000 tracks an object will be described in detail.

Figure 2 is a flowchart showing a method by which an electronic device tracks an object according to an embodiment of the present disclosure.

In step S210, the electronic device 2000 may acquire an image. The electronic device 2000 according to one embodiment can capture an image using a camera included in the electronic device 2000 and acquire the image in real time. The camera may be installed inside the electronic device 2000 or connected externally. Additionally, the electronic device 2000 according to one embodiment may acquire images in real time from an external device through a communication interface included in the electronic device 2000. For example, the electronic device 2000 may receive images acquired from an external camera in real time through a communication interface.

According to one embodiment, the acquired image may include at least one object to be tracked. Additionally, the acquired image may include objects of different classes.

In step S220, the electronic device 2000 may extract a feature map from the image. A feature map according to one embodiment may be a feature map for performing a plurality of tasks related to object tracking. According to one embodiment, the electronic device 2000 may extract a feature map from an image through a backbone network of an object tracking model. In one embodiment, the object tracking model is a multi-task artificial intelligence model for tracking objects, and may be composed of a backbone network, a detection head, an identification head, and a body direction head. The backbone network and Each of the heads may be composed of at least one layer. A detailed description of the object tracking model will be described later with reference to FIG. 3.

In step S230, the electronic device 2000 may extract location information of at least one object, an identification feature of at least one object, and an angle of the body direction of at least one object using the extracted feature map. In one embodiment, the location information of an object may mean a bounding box that represents the location and size of the object in a rectangular shape within an acquired image. According to one embodiment, the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object are respectively the detection head, the identification head, and the object tracking model according to the embodiment. It can be extracted by the body direction head.

According to one embodiment, the identification feature of at least one object and the angle of the body direction of at least one object may be extracted based on the location information of the at least one object. A detailed description of a method of extracting the identification feature and body direction angle of at least one object based on the location information of the at least one object will be described later with reference to FIG. 5A.

In step S240, the electronic device 2000 tracks at least one object based on the extracted location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object. You can. According to one embodiment, tracking the at least one object based on the extracted location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object includes data association. (Data Association) process and tracking information management process may be included.

In one embodiment, data correlation refers to a process in which the electronic device 2000 compares each of at least one detected object with objects that were previously being tracked and matches it with one of the objects that were previously being tracked. In one embodiment, matching two objects means that the electronic device 2000 determines that the two objects are the same object. The electronic device 2000 may perform a data association process based on the extracted location information of at least one object, the identification characteristic of at least one object, and the angle of the body direction of at least one object.

According to one embodiment, the electronic device 2000 may perform a first data association or a second data association process depending on whether the object to be compared with the object detected in the data association process is in an active tracking state. , if the two objects do not match each other, a third data association process can be performed. In one embodiment, an object in an active tracking state means an object that has been matched within a preset period from the time of image acquisition among the objects being tracked. In other words, it means an object whose period from the last matching time of the object to the time of image acquisition is within a preset period. Conversely, an object in an inactive tracking state means an object that was not matched within a preset period from the time of image acquisition among the objects being tracked. A detailed description of the data correlation process will be described later with reference to FIGS. 7 to 10.

According to one embodiment, the electronic device 2000 may perform a tracking information management process after the data association process. The electronic device 2000 may update the list of identification features and the last matching time included in the tracking information, or may start tracking a new object. Additionally, the electronic device 2000 may determine whether the object is in an active tracking state based on the last matching time point, update the state to an inactive tracking state, or end tracking the object. A detailed description of the tracking information management process will be described later with reference to FIGS. 11 and 12.

Figure 3 is a flowchart showing a process of tracking an object using an object tracking model according to an embodiment of the present disclosure.

Referring to FIG. 3 , the electronic device 2000 may track an object using the object tracking model 320.

In step S310, the electronic device 2000 may input the acquired image into the object tracking model 320. The electronic device 2000 may store and use the object tracking model 320 in memory, or may use the object tracking model 320 stored in an external server. When the electronic device 2000 uses the object tracking model 320 stored in an external server, the electronic device 2000 may transmit the acquired image to the external server through a communication interface. The external server may input the image acquired from the electronic device 2000 to the object tracking model 320 and output output data including location information, identification features, and body direction angle of at least one object. The electronic device 2000 may receive output data of the object tracking model 320 from an external server. Hereinafter, for convenience of explanation, the electronic device 2000 stores the object tracking model 320 in the memory and uses the object tracking model 320 stored in the memory as an example.

The object tracking model 320 according to one embodiment may be composed of a backbone network 322, a detection head 324, an identification head 326, and a body direction head 328. The object tracking model 320 is a multi-task artificial intelligence model for object tracking, and can perform object location detection tasks, object identification tasks, and object body direction detection tasks in parallel. Each of the backbone network 322 and the head may be composed of at least one layer.

According to one embodiment, the backbone network 322 may extract a feature map from the image. The backbone network 322 may have a CNN structure including a plurality of layers. The extracted feature map can be used for object location detection tasks, object identification tasks, and object body direction detection tasks. That is, the extracted feature map can be input data for the detection head 324, the identification head 326, and the body direction head 328. By using a feature map rather than an image as input data in each head of the object tracking model 320, the accuracy and processing speed of tasks performed in each head can be increased. The backbone network 322 can divide the image into a plurality of channels and downsample it through convolution layers and pooling layers with a plurality of filters to be suitable for use in each task. For example, the backbone network 322 can extract a feature map by downsampling the image to 1/4. In this case, when the size of the image is (W, H), the size of the feature map may be (W/4, H/4).

According to one embodiment, detection head 324 may perform an object location detection task. That is, the detection head 324 can extract location information of at least one object from the extracted feature map. Additionally, identification head 326 may perform object identification tasks. That is, the identification head 326 may extract identification features of at least one object from the extracted feature map. Additionally, the body direction head 328 may perform an object body direction detection task. That is, the body direction head 328 can extract the body direction angle of at least one object from the extracted feature map.

일 때, 히트맵 헤드의 출력 데이터의 크기는

일 수 있다. 이 때, 마지막 차원은 히트맵 스코어에 대응될 수 있다. According to one embodiment, detection head 324 may include a heatmap head, a bounding box size head, and a center offset head. The heatmap head may extract a heatmap indicating the center position of at least one object from the feature map. The heatmap has a heatmap score corresponding to each pixel of the heatmap, and may have a peak heatmap score at the center position on the feature map of at least one object. For example, the size of the feature map is

When , the size of the output data of the heatmap head is

It can be. At this time, the last dimension may correspond to the heatmap score.

일 수 있다. 이 때, 마지막 차원은 각각 경계 상자의 x축 길이 및 y축 길이에 대응될 수 있다. According to one embodiment, the bounding box size head may extract the x-axis length and y-axis length of the bounding box corresponding to each pixel of the feature map from the feature map. In one embodiment, the bounding box size may include the x-axis length and y-axis length of the bounding box. At this time, the x-axis length and y-axis length of the bounding box corresponding to the center position on the feature map of at least one object may be the x-axis length and y-axis length of the bounding box of at least one object. For example, the size of the feature map is When , the size of the output data of the bounding box size head is

It can be. At this time, the last dimension may correspond to the x-axis length and y-axis length of the bounding box, respectively.

일 수 있다. 이 때, 마지막 차원은 각각 x축의 중심 오프셋 및 y축의 중심 오프셋에 대응될 수 있다.According to one embodiment, the center offset head may extract the center offset corresponding to each pixel of the feature map from the feature map. In one embodiment, the center offset may be an offset for obtaining the center position on an image of at least one object. At this time, the center offset corresponding to the center position on the feature map of at least one object may be the center offset of at least one object. For example, the size of the feature map is When , the size of the output data of the center offset head is

It can be. At this time, the last dimension may correspond to the center offset of the x-axis and the center offset of the y-axis, respectively.

일 수 있다. 이 때,

는 특징맵의 각 픽셀에 대응하는 식별 임베딩(embedding) 벡터의 크기일 수 있다. 예를 들어, 128개의 기준을 가지고 객체의 식별 특징을 추출하는 경우, 식별 헤드(326)는 128개의 필터를 가지는 합성곱 레이어를 포함할 수 있다. 이 경우, 식별 헤드(326)의 출력 데이터의 크기는

일 수 있다. 식별 헤드(326)는 식별 임베딩(embedding) 벡터들을 추출할 수 있다. 일 실시예에서, 식별 임베딩 벡터란, 식별 헤드(326)의 출력 데이터에 포함되는 데이터로, 특징맵의 임의의 픽셀에 대응하는 식별 특징을 의미한다. 즉, 일 실시예에서, 식별 임베딩 벡터와 식별 특징은 동일한 의미로 사용될 수 있다. 이 때, 적어도 하나의 객체의 특징맵 상에서의 중심 위치에 대응하는 식별 임베딩 벡터가 적어도 하나의 객체의 식별 특징일 수 있다.According to one embodiment, the identification head 326 may include a convolution layer having a plurality of filters. The size of the feature map, which is the input data of the identification head 326, is When , the size of the output data of the identification head 326 is

It can be. At this time,

May be the size of the identification embedding vector corresponding to each pixel of the feature map. For example, when extracting identification features of an object using 128 criteria, the identification head 326 may include a convolution layer with 128 filters. In this case, the size of the output data of the identification head 326 is

It can be. The identification head 326 may extract identification embedding vectors. In one embodiment, the identification embedding vector is data included in the output data of the identification head 326 and means an identification feature corresponding to an arbitrary pixel of the feature map. That is, in one embodiment, identification embedding vector and identification feature may be used with the same meaning. At this time, an identification embedding vector corresponding to the center position on the feature map of at least one object may be an identification feature of at least one object.

일 수 있다. 이 때,

는 특징맵의 각 픽셀에 대응하는 몸통 방향 임베딩 벡터의 크기일 수 있다. 예를 들어, 대표 각도가 0도, 90도, 180도, 270도라면, 몸통 방향 헤드(328)는 4개의 필터를 가지는 합성곱 레이어를 포함할 수 있다. 합성곱 레이어의 각 필터는 대표 각도 각각에 대응될 수 있다. 이 경우, 몸통 방향 헤드(328)의 출력 데이터의 크기는

일 수 있다. 몸통 방향 헤드(328)는 몸통 방향 임베딩 벡터들을 추출할 수 있다. 일 실시예에서, 몸통 방향 임베딩 벡터란, 몸통 방향 헤드(328)의 출력 데이터에 포함되는 데이터로, 특징맵의 임의의 픽셀에 대응된다. 이 때, 적어도 하나의 객체의 특징맵 상에서의 중심 위치에 대응하는 몸통 방향 임베딩 벡터가 적어도 하나의 객체의 몸통 방향 임베딩 벡터일 수 있다. 객체의 몸통 방향 임베딩 벡터는 대표 각도 각각에 대응하는 원소들로 이루어지고, 원소들 각각의 값은 대표 각도 각각에 대한 스코어일 수 있다.According to one embodiment, the body direction head 328 may include a convolution layer having as many filters as the number of representative angles. The size of the feature map, which is the input data of the body direction head 328, is When , the size of the output data of the body direction head 328 is

It can be. At this time,

May be the size of the body direction embedding vector corresponding to each pixel of the feature map. For example, if the representative angles are 0 degrees, 90 degrees, 180 degrees, and 270 degrees, the body direction head 328 may include a convolution layer with four filters. Each filter of the convolution layer may correspond to each representative angle. In this case, the size of the output data of the body direction head 328 is

It can be. The body direction head 328 can extract body direction embedding vectors. In one embodiment, the body direction embedding vector is data included in the output data of the body direction head 328 and corresponds to an arbitrary pixel of the feature map. At this time, the body direction embedding vector corresponding to the center position on the feature map of at least one object may be the body direction embedding vector of at least one object. The object's body direction embedding vector consists of elements corresponding to each representative angle, and the value of each element may be a score for each representative angle.

일 수 있다. 히트맵 헤드는 각 클래스에 대응되는 히트맵을 추출할 수 있다. 또한, 식별 헤드(326)의 출력 데이터의 크기는

일 수 있다. 식별 헤드(326)는 각 클래스 별로 특징맵의 각 픽셀에 대응하는 식별 임베딩 벡터들을 추출할 수 있다. 또한, 몸통 방향 헤드(328)의 출력 데이터의 크기는

일 수 있다. 몸통 방향 헤드(328)는 각 클래스 별로 특징맵의 각 픽셀에 대응하는 몸통 방향 임베딩 벡터들을 추출할 수 있다. According to one embodiment, when there are multiple classes of objects tracked by the object tracking model 320, the output data of the detection head 324, the identification head 326, and the body direction head 328 are vectors in the last dimension. The number of elements may be increased or dimensions may be added. For example, there are C classes of objects in the acquired image, and the size of the feature map that is the input data for each head is When , the size of the output data of the heatmap head included in the detection head 324 is

It can be. The heatmap head can extract the heatmap corresponding to each class. Additionally, the size of the output data of the identification head 326 is

It can be. The identification head 326 may extract identification embedding vectors corresponding to each pixel of the feature map for each class. In addition, the size of the output data of the body direction head 328 is

It can be. The body direction head 328 can extract body direction embedding vectors corresponding to each pixel of the feature map for each class.

When there are multiple classes of an object, the electronic device 2000 according to one embodiment may identify the class of the object based on the heatmap scores of the heatmap corresponding to each class extracted from the heatmap head. For example, the heatmap head may extract a heatmap having a peak heatmap score only at the center location of at least one object corresponding to each class. The electronic device 2000 determines at least one object corresponding to each class based on a pixel having a peak heatmap score in the heatmap corresponding to each class, and determines the center position of each object on the feature map. You can decide.

The electronic device 2000 according to an embodiment extracts information from the identification head 326 and the body direction head 328 corresponding to the same class based on the central position of at least one object corresponding to each class on the feature map. The identification features and body direction angle of each object can be extracted from the identified identification embedding vectors and body direction embedding vectors. Hereinafter, for convenience of explanation, an example will be given where there is only one class of object included in the acquired image.

In step S330, the electronic device 2000 may perform an output merging process. That is, the electronic device 2000 may merge output data output from the detection head 324, the identification head 326, and the body direction head 328. The electronic device 2000 extracts location information of at least one object from the detection head 324 and outputs it from the identification head 326 and the body direction head 328 based on the extracted location information of the at least one object. An identification feature of at least one object and an angle of the body direction of at least one object may be extracted from the identified identification embedding vectors and body direction embedding vectors.

A detailed description of the output data output from each head and the output merging process will be described later with reference to FIG. 5A.

In step S340, the electronic device 2000 may perform a data association process. That is, the electronic device 2000 may match the object with one of the previously tracked objects based on the location information, identification feature, and body direction angle of at least one object.

In step S350, the electronic device 2000 may perform a tracking information management process. That is, the electronic device 2000 may update the list of identification features and the last matching time included in the tracking information, or may start tracking a new object. Additionally, the electronic device 2000 may determine whether the object is in an active tracking state based on the last time it was detected, update the object to an inactive tracking state, or end tracking the object.

Figure 4 is an exemplary diagram showing training image data of an object tracking model according to an embodiment of the present disclosure.

An object tracking model according to one embodiment may be learned with a training image dataset containing various annotations. The object tracking model may be learned by the electronic device 2000 or by an external server. When an object tracking model is learned by the electronic device 2000, the electronic device 2000 may receive a training image data set from an external database through a communication interface. When the object tracking model is learned by an external server, the object tracking model may be stored in the external server, or the electronic device 2000 may receive the object tracking model from the external server through a communication interface and store it. Hereinafter, for convenience of explanation, the object tracking model learned by the electronic device 2000 will be described as an example.

According to one embodiment, training image data used to learn an object tracking model may include annotations on the object's class, ID, bounding box information, segmentation information, and body direction angle. . However, the training image data is not limited to this and may include more annotations.

According to one embodiment, the class of an object may represent the type of the object. For example, the object's class can be person, vehicle, animal, etc. The bounding box information of an object may include the x and y coordinates of each vertex of the bounding box. Segmentation means dividing each object within an image. The segmentation information of an object includes the x and y coordinates of the boundary of the object measured in certain units. The angle of the body direction of the object may be labeled as a representative angle that is closest to the angle of the body direction of the actual object. The training image data set may include at least one object corresponding to each representative angle.

일 때, 객체의 학습 이미지 데이터 상에서의 중심 위치는

이며, 객체의 특징맵 상에서의 중심 위치는

일 수 있다. 이는, 객체의 특징맵 상에서의 중심 위치를 특징맵 상에서 정수의 x, y 좌표로 나타내기 위함이다. According to one embodiment, the center position of the object on the feature map can be obtained through the bounding box coordinates of the object in training image data. For example, the feature map is extracted by downsampling the image to 1/4, and the bounding box coordinates on the object's training image data are

When , the center position of the object on the training image data is

, and the center position on the feature map of the object is

It can be. This is to indicate the center position of the object on the feature map as integer x and y coordinates on the feature map.

일 수 있다. 객체의 중심 오프셋은 학습 이미지 데이터의 객체의 경계 상자 좌표 및 객체의 특징맵 상에서의 중심 위치를 통해 구해질 수 있다. 예를 들어, 특징맵은 이미지가 1/4로 다운샘플링되어 추출된 것이고, 객체의 경계 상자 좌표가 일 때, 객체의 중심 오프셋은

일 수 있다. 이는 객체의 이미지 상에서의 정확한 중심 위치를 구하기 위함이다.Likewise, the bounding box size of an object can be obtained through the bounding box coordinates of the object in the training image data. For example, if the object's bounding box coordinates are When the object's bounding box size is

It can be. The center offset of the object can be obtained through the object's bounding box coordinates in the training image data and the center position on the object's feature map. For example, the feature map is extracted by downsampling the image to 1/4, and the bounding box coordinates of the object are When , the center offset of the object is

It can be. This is to obtain the exact center position on the image of the object.

According to one embodiment, the identification characteristics of the object may be determined based on the object's segmentation information, the object's color combination, the object's bounding box information, etc. in the training image data. The electronic device 2000 may train an object tracking model so that among the identification features extracted by the identification head, the identification features between the same objects have high similarity, and the identification features between different objects have low similarity.

According to one embodiment, the angle of the object's body direction may be determined based on the object's segmentation information, object's bounding box information, etc. in training image data. For example, the appearance of an object may be distinguished by representative angle. The electronic device 2000 may train an object tracking model to predict the body-direction angle of the body-direction head of the object based on the external appearance of the object distinguished for each representative angle.

According to one embodiment, the electronic device 2000 calculates loss values for each of the object's heatmap, the object's bounding box size, the object's center offset, the object's identification feature, and the object's body direction angle. And, the object tracking model can be trained so that the total loss value is minimized. In the case of the object's heatmap, the object's bounding box size, and the object's center offset, the electronic device 2000 may set the loss value to be smaller as the difference between the ground-truth and the predicted value is smaller.

According to one embodiment, for identification features of objects, a fully-connected layer and a softmax function may be used for learning. As a result of inputting the predicted identification features into the fully connected layer, the loss value can be set so that the higher the probability of the same object, the smaller the loss value. In the case of the body direction angle of the object, the loss value can be set so that the larger the value of the element corresponding to the representative angle, which is the body direction angle of the object, in the object's body direction embedding vector, the smaller the loss value.

According to one embodiment, the training image data set may include objects corresponding to different classes. For example, a training image data set may include people and vehicles as objects to be tracked. The object tracking model includes a heatmap head, an identification head, and a body direction head, a heatmap corresponding to each class, an identification embedding vector corresponding to each pixel of the feature map for each class, and a heatmap corresponding to each pixel of the feature map for each class. The body direction embedding vector can be learned to be extracted. For example, the heatmap head may include a layer with as many filters as the number of classes, and each filter may be learned to extract a heatmap corresponding to each class. The electronic device 2000 may set the loss value to be smaller as the heatmap corresponding to each class has a larger heatmap score at the center position of at least one object corresponding to each class.

FIG. 5A is a diagram illustrating a method of extracting location information, identification features, and body direction angle of an object using each head of an object tracking model according to an embodiment of the present disclosure.

Referring to FIG. 5A , the electronic device 2000 may merge output data from each head of the object tracking model to extract the object's location information, identification features, and body direction angle. Figure 5a shows an example in which three objects are detected in the image, but more or fewer objects may be detected in the image.

According to one embodiment, the detection head 324 of the object tracking model may be composed of a heatmap head, a bounding box size head, and a center offset head. Each head included in the detection head 324 may use a feature map extracted from the backbone network of the object tracking model as input data. The heatmap head, bounding box size head, and center offset head can extract heatmap 510, bounding box size data 520, and center offset data 530, respectively. The heatmap 510 has the same size as the feature map and may have a heatmap score corresponding to each pixel of the heatmap 510. Additionally, the heatmap 510 may have a peak heatmap score at the center position on the feature map of at least one object. The heatmap head may include a max pooling layer to obtain the peak heatmap score of the heatmap 510. According to one embodiment, after applying max pooling, the heatmap head may extract the top k (eg, 100) pixels among the peak heatmap scores in descending order of heatmap score. At this time, k may be a preset value.

According to one embodiment, the bounding box size and center offset of at least one object may be extracted based on the center position on the feature map of at least one object. In the bounding box size data 520 and the center offset data 530, the bounding box size and center offset corresponding to the center position on the feature map of at least one object may be the bounding box size and center offset of the at least one object. .

According to one embodiment, the electronic device 2000 detects at least one object based on the center position on the feature map of the at least one object, the bounding box size of the at least one object, and the center offset of the at least one object. Location information can be extracted. A detailed description of the method for extracting location information of at least one object will be described with reference to FIG. 5B.

According to one embodiment, the identification head 326 of the object tracking model may extract identification embedding vectors 540. Each pixel of the feature map may have a corresponding identification embedding vector. For example, if the identification head 326 is composed of a convolution layer with 128 filters, the identification embedding vector may be a vector with 128 elements. The electronic device 2000 may extract an identification feature of at least one object based on the central position of the feature map of the at least one object obtained from the heat map head 510 of the detection head 324. The electronic device 2000 may determine an identification embedding vector corresponding to the center position on the feature map of at least one object as an identification feature of at least one object.

According to one embodiment, the body direction head 328 of the object tracking model may extract body direction embedding vectors 550. Each pixel of the feature map may have a corresponding body direction embedding vector. For example, if the body direction head 328 is composed of a convolution layer with four filters, the body direction embedding vector may be a vector with four elements. The electronic device 2000 may extract a body direction embedding vector of at least one object based on the center position of the feature map of the at least one object. The electronic device 2000 may determine a body direction embedding vector corresponding to the center position on the feature map of at least one object as the body direction embedding vector of at least one object.

The electronic device 2000 according to an embodiment may extract the angle of the body direction of at least one object based on the element having the maximum value among the elements of the body direction embedding vector of the at least one object. The electronic device 2000 may determine a representative angle corresponding to an element with the maximum value among elements of the body direction embedding vector of at least one object as the body direction angle of at least one object.

FIG. 5B is a diagram illustrating a method of extracting location information of an object using a detection head of an object tracking model according to an embodiment of the present disclosure.

Referring to FIG. 5B, the electronic device 2000 according to one embodiment generates object location information ( 560) can be extracted.

The electronic device 2000 according to one embodiment may extract the heatmap 510 through a heatmap head and obtain the center position 570 on the feature map of the object from the extracted heatmap 510. Additionally, the electronic device 2000 may extract bounding box size data 520 through the bounding box size head. The electronic device 2000 may determine the x-axis length and y-axis length of the bounding box corresponding to the center position 570 on the feature map of the object as the bounding box size 560 of the object. Additionally, the electronic device 2000 may extract center offset data 530 through a center offset head. The electronic device 2000 may determine the center offset corresponding to the center position 570 on the feature map of the object as the center offset 580 of the object. The electronic device 2000 may determine the center position 590 on the image of the object based on the center position 570 on the acquired feature map of the object and the center offset 580 of the object.

The electronic device 2000 may determine location information of an object based on the center position 590 of the object in the image and the bounding box size 560 of the object. The electronic device 2000 uses the object's location information as a bounding box centered on the center position 590 on the image of the object and having the x, y coordinate length of the object's bounding box size 560 as the x, y coordinate length. can be decided.

According to one embodiment, the electronic device 2000 performs Non-Maximum Suppression (NMS) on bounding boxes determined based on the center position 590 on the image of the object and the bounding box size 560 of the object. You can. The electronic device 2000 may remove a bounding box corresponding to a pixel having a heatmap score less than or equal to a preset reference heatmap score (eg, 0.3) among the top k pixels extracted based on the heatmap score. .

Figure 6 is a diagram showing object detection information and object tracking information according to an embodiment of the present disclosure.

In one embodiment, object detection information 610 is information obtained when the electronic device 2000 detects an object in an acquired image, and may include output data output through an object tracking model. Object detection information 610 according to one embodiment may include an object class, a bounding box of the object, a heatmap score, an identification feature of the object, and an angle of the body direction of the object, but is not limited thereto.

In one embodiment, the object tracking information 620 is information about the object being tracked by the electronic device 2000, including the object class, object ID, object path, object identification feature list, and the last matching point of the object. It may include, but is not limited to this. For example, the object tracking information 620 may include the tracking status of the object. According to one embodiment, the last matching point of an object may be stored in units of frames.

Figure 7 is a flowchart showing a data association process according to an embodiment of the present disclosure.

In step S710, the electronic device 2000 according to one embodiment may compare at least one detected object with previously tracked objects. The electronic device 2000 may compare each detected at least one object with all previously tracked objects. The electronic device 2000 may create a pair consisting of one of the at least one detected object and one of the previously tracked objects and compare the two objects. The electronic device 2000 may compare each of the at least one detected object with all previously tracked objects and match it with the object with the highest similarity.

In step S720, the electronic device 2000 may determine whether each of the previously tracked objects is in an inactive tracking state. For example, the electronic device 2000 may acquire images in real time and detect at least one object in the images. For example, if the previously tracked object p has not been matched for more than 30 frames from the last matching time of object p based on the image acquisition time, the electronic device 2000 may determine the object p to be in an inactive tracking state. . Conversely, if the image acquisition time is within 30 frames from the last matching time of object p, the electronic device 2000 may determine object p to be in an active tracking state. The electronic device 2000 may perform a first data association (S730) when comparing at least one detected object with an object in an inactive tracking state, and perform a second data association when comparing the at least one detected object with an object in an active tracking state. Data correlation (S740) can be performed.

In step S730, when comparing at least one detected object with an object in an inactive tracking state, the electronic device 2000 may perform a first data association process. For example, if the detected object q and the previously tracked object p correspond to the same object, and the object p is in an inactive tracking state, the body direction angles of object q and object p may be different. For example, if object p disappears from the screen after the last matching point and reappears 30 frames after the last matching point, the angle of the body direction of the reappeared object p may be different. As the angle of the body direction changes, the identification features of object q and object p may have low similarity even though they are the same object. Accordingly, the electronic device 2000 may consider not only the identification characteristics of the object but also the angle of the body direction of the object in the first data association process. A detailed description of the first data association process will be described with reference to FIGS. 8A and 8B.

In step S740, when comparing at least one detected object with an object in an active tracking state, the electronic device 2000 may perform a second data association process. For example, if the detected object q and the previously tracked object p correspond to the same object, and the object p is in an active tracking state, the body direction angles of object q and object p may be similar. For example, if object p is matched and tracked for every frame in images acquired in real time, the body direction angle cannot vary significantly because the movement of object p is continuous. In this case, the electronic device 2000 may only consider the identification characteristics of the object in the second data association process.

In step S750, the electronic device 2000 may determine whether the same object exists among existing tracked objects for each of the at least one detected object. According to one embodiment, the electronic device 2000 may determine that the two objects are the same object when the similarity between the identification features of the two objects according to the first data association or the second data association is greater than or equal to a predetermined reference value. The electronic device 2000 may perform a third data association (S760) on an object that does not match existing tracked objects among at least one detected object. For example, when the similarity between the identification characteristics of object q and each of the previously tracked objects according to the first data association or second data association of the detected object q and the previously tracked objects is all less than a predetermined reference value, The electronic device 2000 may perform third data association (S760) with respect to object q.

In step S760, the electronic device 2000 may perform a third data association process for an object that does not match existing tracked objects among at least one detected object. The electronic device 2000 detects a detected object among the previously tracked objects based on IoU (Intersection over Union) between the bounding box (bbox) of each of the at least one detected object and the existing tracked objects. It can be determined whether an object identical to each of at least one object exists. The IoU between bounding boxes is an indicator that determines the degree to which two bounding boxes overlap, and corresponds to the value obtained by dividing the area where two bounding boxes overlap by the combined area of the two bounding boxes. For example, if the IoU between the bounding boxes of the detected object q and the previously tracked object p is greater than or equal to a predetermined reference value, the electronic device 2000 may determine that object q and object p are the same object. Alternatively, the electronic device 2000 may select objects for which the IoU between the detected object q and the bounding box of each of the previously tracked objects is greater than or equal to a predetermined reference value. The electronic device 2000 may determine that the object with the largest IoU value among the selected objects is the same object as object q.

According to one embodiment, the electronic device 2000 performs a first data association or second data association process on both the at least one detected object and the objects that are already being tracked, and each of the detected at least one object For , objects whose similarity between identification features is greater than or equal to a predetermined reference value can be selected. The electronic device 2000 may determine that the object with the highest similarity among the selected objects corresponding to each of the at least one detected object is the same object as each of the corresponding at least one detected object.

FIG. 8A is a flowchart showing a first data association process according to an embodiment of the present disclosure.

In step S810, the electronic device 2000 according to an embodiment may compare at least one detected object with objects in an inactive tracking state among objects that are currently being tracked. Since step S810 corresponds to step S710, detailed description is omitted.

에 대응하는 객체 p의 식별 특징

이 존재하는지 여부를 판단할 수 있다. 는 대표 각도 중 하나에 해당할 수 있다. 예를 들어, 객체 p가 추적되는 동안, 몸통 방향의 각도가 인 상태로 탐지된 적이 없는 경우, 객체 p의 식별 특징 목록에는 가 존재하지 않을 수 있다. 전자 장치(2000)는 가 존재하지 않는 경우, 와 가장 가까운 대표 각도를 결정(S830)할 수 있고, 가 존재하는 경우, 객체 q의 식별 특징

과 간 유사도를 계산(S850)할 수 있다. 이 때, t는 이미지 획득 시점을 의미한다.In step S820, the electronic device 2000 determines whether, for each pair of the detected object and the object being tracked, an identification feature corresponding to the angle of the body direction of the detected object exists in the identification feature list of the object being tracked. You can judge whether or not. For example, when the electronic device 2000 compares a detected object q with an existing object p that is being tracked, the electronic device 2000 includes the angle of the body direction of object q in the list of identifying features of object p.

The identifying feature of the object p corresponding to

You can determine whether it exists or not. may correspond to one of the representative angles. For example, while object p is being tracked, the angle of the body direction is If it has never been detected, the list of identifying features of object p contains may not exist. Electronic devices (2000) If does not exist, The representative angle closest to can be determined (S830), If exists, the identifying feature of object q

class The similarity between the two can be calculated (S850). At this time, t refers to the time of image acquisition.

를 결정할 수 있다. 예를 들어, 대표 각도가 0도, 90도, 180도, 270도이고, 가 90도인 경우, 는 0도 또는 180도 일 수 있다. 전자 장치(2000)는 0도 및 180도에 대응하는 식별 특징이 존재하는지 여부를 판단하고, 식별 특징이 존재하는 대표 각도를 로 결정할 수 있다. In step S830, for each of the pairs of the detected object and the object being tracked, for pairs in which the identification feature corresponding to the angle of the body direction of the detected object does not exist in the identification feature list of the object being tracked, the electronic device (2000) can determine the representative angle closest to the angle of the body direction of the detected object. For example, in the list of identification features of an existing tracked object p, there is an identification feature corresponding to the angle of the body direction of the detected object q. If does not exist, the electronic device 2000 The representative angle closest to

can be decided. For example, representative angles are 0 degrees, 90 degrees, 180 degrees, and 270 degrees, If is 90 degrees, can be 0 degrees or 180 degrees. The electronic device 2000 determines whether an identification feature corresponding to 0 degrees and 180 degrees exists, and selects a representative angle at which the identification feature exists. can be decided.

If both representative angles have an identifying feature, comparison may be made with one of the two identifying features or with both identifying features. Hereinafter, for convenience of explanation, comparison with one of two identification features will be described as an example. If no identifying feature exists for both representative angles, the electronic device 2000 Starting from the representative angle close to the can be decided.

와 에 대응하는 객체 p의 식별 특징

간 유사도를 계산할 수 있다. 예를 들어, 전자 장치(2000)는 와 간 코사인 유사도를 계산할 수 있다. In step S840, for each pair of a detected object and an object being tracked, the electronic device 2000 generates an identification feature corresponding to a representative angle that is closest to the body direction angle of the detected object among the identification feature list of the object being tracked. And, the similarity between the identification features of the detected object can be calculated. For example, when the electronic device 2000 compares a detected object q with an existing object p that is being tracked, the electronic device 2000

and The identifying feature of the object p corresponding to

The similarity between the two can be calculated. For example, electronic device (2000) and The inter-cosine similarity can be calculated.

와

간 유사도를 계산할 수 있다. 예를 들어, 전자 장치(2000)는 와 간 코사인 유사도를 계산할 수 있다.In step S850, for pairs of a detected object and an object being tracked, for pairs in which an identification feature corresponding to the angle of the body direction of the detected object exists in the identification feature list of the object being tracked, the object being tracked The similarity between the identification feature corresponding to the body direction of the detected object and the identification feature of the detected object from the list of identification features can be calculated. For example, when comparing a detected object q with an existing object p that is being tracked, the electronic device 2000

and

The similarity between the two can be calculated. For example, electronic device (2000) and The inter-cosine similarity can be calculated.

FIG. 8B is an exemplary diagram illustrating a first data association process according to an embodiment of the present disclosure.

The electronic device 2000 according to one embodiment may compare the object q (810) detected in the first data association process with the object p that is already being tracked. At this time, the angle of the body direction of the detected object q (810) may be 90 degrees. There may not be an identification feature corresponding to 90 degrees in the identification feature list 820 of object p. In this case, the electronic device 2000 may determine the representative angle closest to 90 degrees as 0 degrees or 180 degrees. The electronic device 2000 may calculate the similarity between the identification feature corresponding to 0 degrees or 180 degrees and the identification feature of object q (810). According to one embodiment, the electronic device 2000 may calculate the similarity between both identification features corresponding to 90 degrees and 180 degrees and the identification feature of object q (810).

Figure 9 is a flowchart showing a second data association process according to an embodiment of the present disclosure.

In step S910, the electronic device 2000 according to an embodiment may compare at least one detected object with objects in an active tracking state among objects that are currently being tracked. Since step S910 corresponds to step S710, detailed description is omitted.

와 객체 p의 최신 식별 특징

간 유사도를 계산할 수 있다. 는 객체 p가 마지막으로 매칭된 시점에서의 객체 p의 식별 특징일 수 있다. 예를 들어, 전자 장치(2000)는 와 간 코사인 유사도를 계산할 수 있다. In step S920, the electronic device 2000 may calculate the similarity between the identification characteristics of at least one detected object and the latest identification characteristics of each of the previously tracked objects. For example, when the electronic device 2000 compares a detected object q with an existing object p that is being tracked, the electronic device 2000

and the latest identifying features of object p

The similarity between the two can be calculated. may be an identification feature of object p at the time when object p was last matched. For example, electronic device (2000) and The inter-cosine similarity can be calculated.

Figure 10 is a flowchart showing a third data association process according to an embodiment of the present disclosure.

In step S1010, the electronic device 2000 according to one embodiment may compare at least one detected object with previously tracked objects.

In step S1020, the electronic device 2000 may calculate the IoU between the bounding box (bbox) of at least one detected object and each of the previously tracked objects. For example, when the electronic device 2000 compares a detected object q with an existing object p that is being tracked, the electronic device 2000 may calculate the IoU between the bounding boxes (bbox) of object q and object p. . Since the description of step S1020 is the same as the description of step S760, detailed description is omitted.

FIG. 11A is a flowchart showing a tracking information management process according to an embodiment of the present disclosure.

Referring to FIG. 11A , the electronic device 2000 according to an embodiment may update tracking information of objects that are already being tracked or may initiate tracking of at least one detected object.

In step S1110, the electronic device 2000 may compare at least one detected object with previously tracked objects. Since step S1110 corresponds to step S710, detailed description is omitted.

In step S1120, the electronic device 2000 may perform a data association process between at least one detected object and previously tracked objects. Since the data correlation process has been described with reference to FIGS. 7 to 10, detailed description will be omitted.

In step S1130, the electronic device 2000 may determine, for each of the at least one detected object, whether the same object exists among objects that are already being tracked. For example, when the electronic device 2000 compares an object q detected in a data association process with an object p that is already being tracked, the electronic device 2000 may determine whether object q and object p are the same object. You can. If object q and object p are matched during the data association process, the electronic device 2000 may update tracking information of object p (S1140). The electronic device 2000 may determine whether an object matching object q exists not only for object p but also for all previously tracked objects. If there is no object matching object q, the electronic device 2000 may start tracking object q (S1150).

In step S1140, the electronic device 2000 may update tracking information of objects that match at least one detected object among objects that are already being tracked. For example, if object q detected in the data correlation process matches object p that is already being tracked, the electronic device 2000 may update tracking information of object p based on the detection information of object q. A detailed description of the process of updating tracking information of object p will be described with reference to FIG. 11B.

을 객체 q의 최신 식별 특징 및 객체 q의 몸통 방향의 각도

에 해당하는 식별 특징으로 추가할 수 있다. 전자 장치(2000)는 이미지 획득 시점인 t를 객체 q의 마지막 매칭 시점으로 결정할 수 있다.In step S1150, the electronic device 2000 may initiate tracking of at least one detected object that does not match all of the previously tracked objects. For example, if the detected object q does not match all of the previously tracked objects, the electronic device 2000 may start tracking object q. For example, the electronic device 2000 may store tracking information about object q in a tracking database. The electronic device 2000 may determine the ID of object q as q and start tracking the path of object q. The electronic device 2000 adds the extracted identification features of object q to the list of identification features of object q.

the latest identifying features of object q and the angle of body orientation of object q

It can be added as an identification feature corresponding to . The electronic device 2000 may determine t, the image acquisition time, as the last matching time of object q.

FIG. 11B is a diagram illustrating a process of updating tracking information of an object according to an embodiment of the present disclosure.

Referring to FIG. 11B, when object q (1110) and object p match, the electronic device 2000 may update tracking information of object p based on detection information of object q (1110). For example, the identification feature corresponding to 90 degrees may not exist in the identification feature list 1120 of object p before update. If the angle of the body direction of object q (1110) is 90 degrees, and if object q (1110) and object p are matched, the electronic device 2000 updates the latest identification feature of object p and the identification feature corresponding to 90 degrees with the identification feature of object q (1110). You can. Accordingly, all identification features corresponding to each representative angle may be present in the updated identification feature list 1130 of object p.

The electronic device 2000 may update not only the identification feature list 1120 of object p, but also other information included in the tracking information of object p. The electronic device 2000 may update the path of object p based on the detected location information of object q (1110). Additionally, the electronic device 2000 may update the image acquisition time t, which is the detection time point of object q (1110), to the last matching time point of object p.

FIG. 12A is a flowchart illustrating a process for updating the tracking state of an object according to an embodiment of the present disclosure.

The electronic device 2000 according to one embodiment may update the tracking status of objects that are already being tracked during the tracking information management process. The electronic device 2000 may update to an inactive tracking state or end tracking the object based on the unmatched period from the last matching time.

In step S1210, the electronic device 2000 may set objects in an active tracking state among existing tracked objects as tracking status update targets.

In step S1220, the electronic device 2000 may determine whether a first period has elapsed from the time when each of the objects being tracked was last matched, based on the image acquisition time. That is, the electronic device 2000 may determine whether each of the objects being tracked has not been matched during the first period from the last matching point. The first period may be a preset period that serves as a standard for the active tracking state and inactive tracking state of the object. For example, the electronic device 2000 may set the first period to 30 frames. The electronic device 2000 may update the tracking status of objects for which a first period of time has elapsed from the last matching point among the objects being tracked to an inactive tracking status (S1230). The electronic device 2000 may not update the tracking status of objects for which the first period has not yet elapsed from the last matching point among the objects being tracked.

In step S1230, the electronic device 2000 may update the tracking status of objects for which a first period has elapsed from the last matching point among the objects being tracked to an inactive tracking state, based on the image acquisition time. For example, when the first period is set to 30 frames, among the objects being tracked, for objects that have not been matched for more than 30 frames from the last matching time, the electronic device 2000 may update the tracking state to an inactive tracking state.

FIG. 12B is a flowchart illustrating a process of updating the tracking state of an object according to an embodiment of the present disclosure.

In step S1240, the electronic device 2000 may set objects in an inactive tracking state among objects that are already being tracked as tracking status update targets.

In step S1250, the electronic device 2000 may determine whether a second period has elapsed from the time when each of the objects being tracked was last matched, based on the image acquisition time. That is, the electronic device 2000 may determine whether each of the objects being tracked has not been matched for a second period from the last matching point. The second period is a preset period that serves as a standard for the inactive tracking state of the object and the end of object tracking, and may be longer than the first period. For example, the electronic device 2000 may set the second period to 50 frames. The electronic device 2000 may terminate tracking of objects for which a second period of time has elapsed since the last matching point among the objects being tracked (S1240). The electronic device 2000 may not end tracking of objects for which the second period has not yet elapsed from the last matching point among the objects being tracked.

In step S1260, the electronic device 2000 may end tracking objects for which a second period has elapsed since the last matching time among the objects being tracked, based on the image acquisition time. When the electronic device 2000 decides to end tracking an object, the objects for which tracking has ended may not be compared with objects detected in images acquired later.

FIG. 13 is a block diagram illustrating the configuration of an electronic device 2000 according to an embodiment of the present disclosure.

Referring to FIG. 13 , an electronic device 2000 according to an embodiment may include at least one processor 2100, a memory 2200, and a communication interface 2300.

At least one processor 2100 may control operations or functions performed by the electronic device 2000 by executing instructions or programmed software modules stored in the memory 2200. At least one processor 2100 may be comprised of hardware components that perform arithmetic, logic, input/output operations, and signal processing.

At least one processor 2100 may include, for example, a Central Processing Unit, a microprocessor, a Graphics Processing Unit, Application Specific Integrated Circuits (ASICs), or Digital Signal Processors (DSPs). , DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), Application Processors, Neural Processing Units, or hardware structures specialized for processing artificial intelligence models. It may consist of at least one of the designed artificial intelligence processors, but is not limited to this. Each processor constituting at least one processor 2100 may be a dedicated processor for performing a certain function.

At least one processor 2100 executes one or more instructions stored in the memory 2200, thereby enabling the electronic device 2000 to control overall operations for tracking at least one object using an object tracking model. there is.

The memory 2200 may store instructions, data structures, and program code that can be read by at least one processor 2100. Operations performed by at least one processor 2100 may be implemented by executing instructions or codes of a program stored in the memory 2200.

The memory 2200 includes flash memory type, hard disk type, multimedia card micro type, and card type memory (for example, SD or XD memory, etc.). Non-volatile memory that includes at least one of ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. and volatile memory such as RAM (Random Access Memory) or SRAM (Static Random Access Memory).

The memory 2200 according to one embodiment may store a tracking database so that the electronic device 2000 can track at least one object.

The communication interface 2300 can perform wired or wireless communication with other devices or networks. The communication interface 2300 may include a communication circuit or communication module that supports at least one of various wired and wireless communication methods. For example, the communication interface 2300 may include wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, WFD (Wi-Fi Direct), and infrared. Communication (IrDA, infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), Wibro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol) , Data communication between the electronic device 2000 and other devices may be performed using at least one of data communication methods including Wireless Gigabit Alliances (WiGig) and RF communication.

The communication interface 2300 according to one embodiment may transmit an image acquired through the camera 2500 to an external server where the object tracking model is stored. The external server can input the received image into the object tracking model and output output data. The communication interface 2300 may receive output data from an external server. Additionally, the communication interface 2300 may receive an object tracking model or a training image data set used for learning an object tracking model from an external device.

Figure 14 is a block diagram showing the configuration of an electronic device according to an embodiment of the present disclosure.

The electronic device 2000 may further include a display 2400 and a camera 2500 in addition to the components shown in FIG. 13 .

Contents that overlap with those described in FIG. 13 for at least one processor 2100 and the memory 2200 will be omitted hereinafter. A tracking database and an object tracking model may be stored in the memory 2200. At least one processor 2100 may load an object tracking model from the memory 2200 and execute it.

The display 2400 includes an output interface that provides information or images, and may further include an input interface that receives input. The output interface may include a display panel and a controller that controls the display panel, such as an Organic Light Emitting Diode (OLED) display, an Active-Matrix Organic Light-Emitting Diode (AM-OLED) display, a Liquid Crystal Display (LCD), etc. It can be implemented in various ways. The input interface can receive various types of input from the user and may include at least one of a touch panel, keypad, and pen recognition panel. The display 2400 may be provided in the form of a touch screen that combines a display panel and a touch panel, and may be implemented in a flexible or foldable manner. The display 2400 according to one embodiment may output the paths of objects being tracked in real time.

The camera 2500 is a hardware module that acquires images. The camera 2500 can capture images or videos. The camera 2500 may include at least one camera module and, depending on the specifications of the electronic device 2000, may support functions such as macro, depth of field, telephoto, wide angle, and ultra wide angle.

At least one processor 2100 according to an embodiment of the present disclosure may acquire an image by executing at least one instruction stored in the memory 2200. At least one processor 2100 may extract a feature map for performing a plurality of tasks related to object tracking from an image by executing at least one instruction stored in the memory 2200. At least one processor 2100 executes at least one instruction stored in the memory 2200, thereby generating location information indicating the location of at least one object and identifying at least one object using the extracted feature map. Identification features and the angle of the body direction toward which the body of at least one object faces can be extracted. At least one processor 2100 executes at least one instruction stored in the memory 2200 to track at least one object based on the position information, identification feature, and body direction angle of the at least one object. You can.

At least one processor 2100 according to an embodiment executes at least one instruction stored in the memory 2200, thereby extracting a feature map using an object tracking model, location information, and identification features of at least one object. , and an operation to extract body direction information may be performed. The object tracking model is a multi-task artificial intelligence model that performs multiple tasks related to object tracking, including a backbone network that extracts a feature map, a detection head that extracts location information of at least one object, and a backbone network that extracts the location information of at least one object. It may include an identification head for extracting identification features and a body direction head for extracting the angle of the body direction of at least one object. Each of the backbone network and heads may be composed of at least one layer.

At least one processor 2100 according to an embodiment may identify the class of at least one object by executing at least one instruction stored in the memory 2200.

At least one processor 2100 according to an embodiment executes at least one instruction stored in the memory 2200, thereby creating at least one heatmap indicating the center position on the feature map of at least one object using the feature map. The bounding box size of the object and the center offset for obtaining the center position in the image of at least one object can be extracted. At least one processor 2100 may obtain the center position of at least one object on the feature map in the heat map by executing at least one instruction stored in the memory 2200. At least one processor 2100 executes at least one instruction stored in the memory 2200 to determine the center position of the at least one object on the feature map, the bounding box size of the at least one object, and the center offset of the at least one object. Based on this, location information of at least one object can be extracted. The heatmap may have a heatmap score corresponding to each pixel of the heatmap, and may have a peak heatmap score at the center position on the feature map of at least one object.

At least one processor 2100 according to an embodiment may extract identification features using a feature map by executing at least one instruction stored in the memory 2200. At least one processor 2100 extracts an identification feature of at least one object from the identification features based on the central position of the at least one object on the feature map by executing at least one instruction stored in the memory 2200. can do.

At least one processor 2100 according to an embodiment may extract body direction embedding vectors using a feature map by executing at least one instruction. The at least one processor 2100 executes at least one instruction stored in the memory 2200 to determine the body direction of the at least one object from the body direction embedding vectors based on the center position of the feature map of the at least one object. Embedding vectors can be extracted. At least one processor 2100 executes at least one instruction stored in the memory 2200 to determine the body direction of the at least one object based on the element having the maximum value among the elements of the body direction embedding vector of the at least one object. The angle of can be extracted. The body direction embedding vector consists of elements corresponding to each representative angle of the body direction, and the value of each element may be a score for each representative angle of the body direction.

At least one processor 2100 according to an embodiment executes at least one instruction, thereby generating at least an identification feature corresponding to the angle of the body direction of at least one object included in the identification feature list of each of the objects being tracked. The identifying characteristics of one object can be compared. At least one processor 2100 executes at least one instruction stored in the memory 2200, so that when there is no identification feature corresponding to the angle of the body direction of the at least one object in the identification feature list, the body of the at least one object The identification feature of at least one object may be compared with the identification feature corresponding to the representative angle closest to the direction angle. At least one processor 2100 may track at least one object based on the comparison result by executing at least one instruction stored in the memory 2200. The identification feature list may include at least one identification feature corresponding to each of the latest identification features and a representative angle of the body direction.

At least one processor 2100 according to an embodiment executes at least one instruction, so that when at least one object matches at least one of the objects being tracked, the latest information included in the identification feature list of the at least one object The identification feature and the identification feature corresponding to the angle of the body direction of the at least one object may be updated as the identification feature of the at least one object.

At least one processor 2100 according to an embodiment executes at least one instruction to generate location information of at least one object when comparing it with an object that was matched within a preset period from the time of image acquisition among the objects being tracked. And based on the identification characteristics of the at least one object, comparing the at least one object with an object that was matched within a preset period from the time of image acquisition, and comparing the at least one object with the object that was not matched within a preset period from the time of image acquisition among the objects being tracked. When comparing with an object, based on the location information of at least one object, the identification feature of at least one object, and the angle of the body direction of at least one object, an object that was not matched within a preset period from the time of image acquisition At least one object can be compared. At least one processor 2100 may track at least one object based on the comparison result by executing at least one instruction.

A method of tracking at least one object according to an embodiment of the present disclosure may include acquiring an image. A method of tracking at least one object may include extracting a feature map for performing a plurality of tasks related to object tracking from an image. A method of tracking at least one object uses an extracted feature map to provide location information indicating the location of at least one object, identification features that enable identification of the at least one object, and information about where the body of the at least one object is facing. It may include extracting the angle of the body direction. A method of tracking at least one object may include tracking at least one object based on location information, an identification feature, and a body direction angle of the at least one object.

According to one embodiment, in a method of tracking at least one object, extracting a feature map, extracting location information, identification features, and body direction angle of at least one object are performed through an object tracking model. You can. The object tracking model is a multi-task artificial intelligence model that performs multiple tasks related to object tracking, including a backbone network that extracts a feature map, a detection head that extracts location information of at least one object, and a backbone network that extracts the location information of at least one object. It may include an identification head for extracting identification features and a body direction head for extracting the angle of the body direction of at least one object. Each of the backbone network and heads may be composed of at least one layer.

According to one embodiment, in a method of tracking at least one object, the object tracking model includes an annotation on a class indicating the type of the object, an annotation on the ID of the object, and a bounding box indicating the position and size of the object in a rectangle. It may be learned with an image dataset that includes an annotation for (bounding box) information, an annotation for segmentation information indicating the boundary of the object, and an annotation for the angle of the body direction toward which the body of the object faces.

According to one embodiment, in a method of tracking at least one object, extracting location information of at least one object using the extracted feature map may include identifying the class of the at least one object. there is.

According to one embodiment, in a method of tracking at least one object, the step of extracting location information of at least one object using a feature map includes determining the center position of the at least one object on the feature map using the feature map. It may include extracting a heatmap representing a heatmap, a bounding box size of at least one object, and a center offset for obtaining the center position on the image of at least one object. Extracting location information of at least one object using the feature map may include obtaining the center position of the at least one object on the feature map from the heat map. The step of extracting location information of at least one object using the feature map includes the center position on the feature map of the at least one object, the bounding box size of the at least one object, and the center offset of the at least one object. Thus, the method may include extracting location information of at least one object. The heatmap may have a heatmap score corresponding to each pixel of the heatmap, and may have a peak heatmap score at the center position on the feature map of at least one object.

According to one embodiment, in a method of tracking at least one object, extracting identification features of at least one object using a feature map may include extracting identification features using a feature map. Extracting the identification feature of at least one object using the feature map may include extracting the identification feature of the at least one object from the identification features based on the central position of the at least one object on the feature map. You can.

According to one embodiment, in a method of tracking at least one object, the step of extracting the angle of the body direction of at least one object using a feature map includes extracting body direction embedding vectors using the feature map. May include steps. The step of extracting the angle of the body direction of the at least one object includes extracting the body direction embedding vector of the at least one object from the body direction embedding vectors based on the center position on the feature map of the at least one object. can do. The step of extracting the angle of the body direction of the at least one object includes extracting the angle of the body direction of the at least one object based on the element having the maximum value among the elements of the body direction embedding vector of the at least one object. can do. The body direction embedding vector consists of elements corresponding to each representative angle of the body direction, and the value of each element may be a score for each representative angle of the body direction.

According to one embodiment, in a method of tracking at least one object, based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object, the at least one object The tracking step may include determining whether an identification feature corresponding to the angle of the body direction of at least one object exists in the identification feature list of each of the objects being tracked.

The step of tracking the at least one object based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object includes the body of the at least one object in the identification feature list. If there is an identification feature corresponding to the angle of direction, the identification feature corresponding to the angle of the body direction of at least one object included in the identification feature list is compared with the identification feature of the at least one object, and at least one object is included in the identification feature list. If there is no identification feature corresponding to the angle of the body direction of the object, comparing the identification feature of the at least one object with the identification feature corresponding to the representative angle closest to the angle of the body direction of the object. You can.

Tracking the at least one object based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object may include: tracking the at least one object based on the comparison result; It may include the step of tracking. The identification feature list may include at least one identification feature corresponding to each of the latest identification feature of the object and the representative angle of the body direction.

A method of tracking at least one object according to an embodiment includes, when the at least one object matches at least one of the objects being tracked, the latest identification feature included in the identification feature list of the at least one object and at least one It may include updating an identification feature corresponding to the angle of the body direction of the object to an identification feature of at least one object.

According to one embodiment, in a method of tracking at least one object, based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object, the at least one object The tracking step includes obtaining an image based on the location information of at least one object and the identification feature of at least one object when comparing the objects among the being tracked objects with those that were matched within a preset period from the time of image acquisition. When comparing at least one object with an object that was matched within a preset period from the point of view and comparing it with an object that was not matched within a preset period from the point of image acquisition among the objects being tracked, location information of at least one object, at least It may include comparing the at least one object with an object that was not matched within a preset period from the time of image acquisition, based on the identification characteristic of one object and the angle of the body direction of the at least one object.

The step of tracking the at least one object based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object includes: tracking the at least one object based on the comparison result; May include tracking steps.

A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' simply means that it is a tangible device and does not contain signals (e.g. electromagnetic waves). This term refers to cases where data is semi-permanently stored in a storage medium and temporary storage media. It does not distinguish between cases where it is stored as . For example, a 'non-transitory storage medium' may include a buffer where data is temporarily stored.

According to one embodiment, methods according to various embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. A computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store or between two user devices (e.g. smartphones). It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable app) is stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server. It can be temporarily stored or created temporarily.

Claims (20)

In a method of tracking at least one object,
acquiring an image;
extracting a feature map for performing a plurality of tasks related to object tracking from the image;
Using the extracted feature map, location information indicating the location of the at least one object, an identification feature enabling identification of the at least one object, and an angle of a body direction toward which the body of the at least one object faces are extracted. steps; and
A method comprising tracking the at least one object based on location information, identifying features, and body orientation angle of the at least one object. According to paragraph 1,
The step of extracting the feature map, the step of extracting the location information, identification feature, and body direction angle of the at least one object are performed through an object tracking model,
The object tracking model is a multi-task artificial intelligence model that performs a plurality of tasks related to object tracking, and includes a backbone network for extracting the feature map, a detection head for extracting location information of the at least one object, and An identification head for extracting identification features of at least one object and a body direction head for extracting an angle of the body direction of the at least one object,
The method, wherein each of the backbone network and the heads consists of at least one layer. According to any one of claims 1 and 2,
The object tracking model includes an annotation on the class indicating the type of object, an annotation on the ID of the object, an annotation on bounding box information indicating the position and size of the object in a rectangle, and an annotation on bounding box information indicating the boundary of the object. A method that is learned with an image dataset that includes an annotation for segmentation information and an annotation for the angle of the body direction toward which the body of the object faces. According to any one of claims 1 to 3,
The step of extracting location information of the at least one object using the extracted feature map includes:
A method comprising: identifying a class of the at least one object. According to any one of claims 1 to 4,
The step of extracting location information of at least one object using the feature map includes:
Using the feature map, a heatmap indicating the central position of the at least one object on the feature map, a bounding box size of the at least one object, and a central position of the at least one object on the image are obtained. extracting a centroid offset;
Obtaining a central position of the at least one object on the feature map from the heat map;
Extracting location information of the at least one object based on a center position of the at least one object on the feature map, a bounding box size of the at least one object, and a center offset of the at least one object. It further includes,
The heatmap has a heatmap score corresponding to each pixel of the heatmap, and has a peak heatmap score at a central position on the feature map of the at least one object. According to any one of claims 1 to 5,
The step of extracting identification features of the at least one object using the feature map includes:
extracting identification features using the feature map;
Extracting an identifying feature of the at least one object from the identifying features based on a central location of the at least one object on the feature map. According to any one of claims 1 to 6,
The step of extracting the angle of the body direction of the at least one object using the feature map includes:
extracting body direction embedding vectors using the feature map;
extracting a body direction embedding vector of the at least one object from the body direction embedding vectors based on a central position of the at least one object on the feature map;
A step of extracting the angle of the body direction of the at least one object based on the element having the maximum value among the elements of the body direction embedding vector of the at least one object,
The method wherein the body direction embedding vector is composed of the elements corresponding to each representative angle of the body direction, and the value of each element is a score for each representative angle of the body direction. According to any one of claims 1 to 7,
Tracking the at least one object based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object,
determining whether an identification feature corresponding to a body direction angle of the at least one object exists in an identification feature list of each of the objects being tracked;
If the identification feature list includes an identification feature corresponding to the angle of the body direction of the at least one object, the identification feature corresponding to the angle of the body direction of the at least one object included in the identification feature list and the at least one Compare the identifying characteristics of objects,
If there is no identification feature in the identification feature list that corresponds to the angle of the body direction of the at least one object, identification corresponding to the representative angle closest to the angle of the body direction of the at least one object included in the identification feature list. comparing features and identifying characteristics of the at least one object;
Based on the comparison result, tracking the at least one object,
The method, wherein the identification feature list includes at least one identification feature corresponding to each of the latest identification feature of the object and the representative angle of the body direction. According to any one of claims 1 to 8,
The above method is,
When the at least one object matches at least one of the objects being tracked, identification corresponding to the latest identification feature included in the identification feature list of the at least one object and the angle of the body direction of the at least one object. The method further comprising updating a feature with an identifying feature of the at least one object. According to any one of claims 1 to 9,
Tracking the at least one object based on the location information of the at least one object, the identification feature of the at least one object, and the angle of the body direction of the at least one object,
When comparing among the objects being tracked with an object that was matched within a preset period from the time of image acquisition, based on the location information of the at least one object and the identification characteristic of the at least one object, the time of image acquisition Compare the at least one object with an object that was matched within a preset period of time,
When comparing among the objects being tracked with an object that was not matched within the preset period from the time of image acquisition, location information of the at least one object, identification characteristics of the at least one object, and the at least one object Comparing the at least one object with an object that was not matched within a preset period from the time of image acquisition, based on the angle of the body direction;
A method comprising tracking the at least one object based on the comparison result. In an electronic device (2000) for tracking at least one object,
communication interface 2300;
A memory 2200 that stores at least one instruction; and
At least one processor (2100) executing the at least one instruction stored in the memory (2200),
The at least one processor 2100 executes the at least one instruction,
acquire the image,
Extracting a feature map for performing a plurality of tasks related to object tracking from the image,
Using the extracted feature map, location information indicating the location of the at least one object, an identification feature enabling identification of the at least one object, and an angle of a body direction toward which the body of the at least one object faces are extracted. do,
An electronic device (2000) that tracks the at least one object based on location information, identification features, and body direction angle of the at least one object. According to clause 11,
The operation of extracting the feature map, the operation of extracting the location information, identification feature, and body direction information of the at least one object are performed through an object tracking model,
The object tracking model is a multi-task artificial intelligence model that performs a plurality of tasks related to object tracking, including a backbone network for extracting the feature map, a detection head for extracting location information of the at least one object, and An identification head for extracting identification features of at least one object and a body direction head for extracting an angle of the body direction of the at least one object,
The electronic device (2000), wherein each of the backbone network and the heads consists of at least one layer. According to any one of claims 11 to 12,
The object tracking model includes an annotation on the class indicating the type of object, an annotation on the ID of the object, an annotation on bounding box information indicating the position and size of the object in a rectangle, and an annotation on bounding box information indicating the boundary of the object. An electronic device (2000), which is trained with an image dataset that includes annotations for segmentation information and annotations for the angle of the body direction toward which the body of the object faces. According to any one of claims 11 to 13,
The at least one processor 2100 executes the at least one instruction,
An electronic device (2000) that identifies a class of the at least one object. According to any one of claims 11 to 14,
The at least one processor 2100 executes the at least one instruction,
Using the feature map, a heatmap indicating the central position of the at least one object on the feature map, a bounding box size of the at least one object, and a central position of the at least one object on the image are obtained. extract the centroid offset,
Obtaining the central position of the at least one object on the feature map from the heat map,
Extracting location information of the at least one object based on the center position of the at least one object on the feature map, the bounding box size of the at least one object, and the center offset of the at least one object,
The heatmap has a heatmap score corresponding to each pixel of the heatmap, and has a peak heatmap score at a central position on the feature map of the at least one object. . According to any one of claims 11 to 15,
The at least one processor 2100 executes the at least one instruction,
Extract identification features using the feature map,
An electronic device (2000) that extracts an identification feature of the at least one object from the identification features based on a central position of the at least one object on the feature map. According to any one of claims 11 to 16,
The at least one processor 2100 executes the at least one instruction,
Extract body direction embedding vectors using the feature map,
extracting a body direction embedding vector of the at least one object from the body direction embedding vectors, based on the central position of the at least one object on the feature map;
Extracting the angle of the body direction of the at least one object based on the element having the maximum value among the elements of the body direction embedding vector of the at least one object,
The electronic device (2000) wherein the body direction embedding vector is composed of the elements corresponding to each representative angle of the body direction, and the value of each element is a score for each representative angle of the body direction. According to any one of claims 11 to 17,
The at least one processor 2100 executes the at least one instruction,
Determine whether an identification feature corresponding to the angle of the body direction of the at least one object exists in the identification feature list of each object being tracked,
If the identification feature list includes an identification feature corresponding to the angle of the body direction of the at least one object, the identification feature corresponding to the angle of the body direction of the at least one object included in the identification feature list and the at least one Compare the identifying characteristics of objects,
If there is no identification feature in the identification feature list corresponding to the angle of the body direction of the at least one object, the identification feature corresponding to the representative angle closest to the angle of the body direction of the at least one object and the at least one object Compare the identifying characteristics of
Based on the comparison result, track the at least one object,
The electronic device (2000) wherein the identification feature list includes at least one identification feature corresponding to each of the latest identification feature and a representative angle of the body direction. According to any one of claims 11 to 18,
The at least one processor 2100 executes the at least one instruction,
When the at least one object matches at least one of the objects being tracked, identification corresponding to the latest identification feature included in the identification feature list of the at least one object and the angle of the body direction of the at least one object. An electronic device (2000) that updates features with identification features of the at least one object. A computer-readable recording medium on which a program for performing the method of any one of claims 1 to 10 is recorded on a computer.

Images