KR20220079428A - Method and apparatus for detecting object in video - Google Patents

Abstract

A method and apparatus for detecting a target object that can be applied to fields such as artificial intelligence, object tracking, object detection, and image processing are disclosed. An object is detected from a frame image of a video including a plurality of frame images based on a target template set including one or more target templates. By using a set of target templates, the accuracy of object detection and tracking is improved.

Description

METHOD AND APPARATUS FOR DETECTING OBJECT IN VIDEO

The disclosure below relates to a method and apparatus for tracking and detecting an object in a video.

Visual object tracking is one of the important research fields in computer vision technology. Object tracking technology is generally used to track a moving object, such as a person, an animal, an aircraft, a car, or the like.

In the process of tracking a moving object, a target object to be tracked may be indicated in a first frame (also referred to as an initial frame) of a video. The object tracking algorithm performs continuous tracking of the target object in subsequent frames of the video, and provides location information of the target object within the frame.

For object tracking, template information associated with the target object is extracted based on the target object indicated in the first frame. Within a search region of subsequent video frames, a degree of matching between a plurality of other candidate regions and the template information is calculated, and the most matching candidate region is the location of the target object is determined by

According to one embodiment, a method for detecting an object from a frame image of a video comprising a plurality of frame images is provided. The method includes, based on a target template set, detecting the object from the frame image, and outputting information regarding the detected object. include The target template set includes one or more target templates. each of the one or more target templates includes information of the object in a respective frame image determined to include the object among previous frame images of the frame image of the video. do.

According to an embodiment, the target template set may further include an initial target template. The initial target template may include information on the object in the frame image determined by the user as including the object among the frame images of the video. Alternatively, the initial target template may include a separate image independent of the video and including the object.

According to an embodiment, the detecting of the object from the frame image comprises integrating an image feature of an image region corresponding to each of the target templates included in the target template set to integrate the target fusion feature. determining an integrated target feature; acquiring one or more target candidate areas determined to include the object in the frame image based on the target fusion feature; and the one or more target candidate areas. It may include determining one target area from the target candidate area.

According to an embodiment, the obtaining of the one or more target candidate regions includes determining a plurality of search regions within the frame image, and extracting image features of each of the plurality of search regions to obtain a search region obtaining a search area feature, calculating a correlation between each search area feature of the plurality of search areas and the target fusion feature, and based on the correlation, the plurality of search areas and determining the one or more target candidate regions.

According to an embodiment, the method may further include updating the target template set. The updating of the target template set includes calculating a similarity of the target region and the target fusion feature of the target template set, and if the similarity is less than a threshold value, the target region as a target template It may include adding (add) to the target template.

According to an embodiment, the detecting of the object from the frame image includes: obtaining one or more target candidate areas determined to include the object in the frame image; It may include determining one target area from the target candidate area. Also, the method may further include updating the target template set when the target area satisfies a predetermined condition.

According to an embodiment, the updating of the target template set includes calculating a similarity of each of all target templates of the target template set and the target region, and all of the similarities are less than a threshold value. In this case, the method may include adding the target region as a target template to the target template.

According to an embodiment, the detecting of the object from the frame image may include detecting the object from the frame image based on the target template set and an interference template set. . The interference template set includes one or more interference templates. Each of the one or more interference templates includes information about an interference object that interfered with the detection of the object among previous frame images of the frame image of the video.

According to an embodiment, the detecting of the object from the frame image includes one or more target candidate regions determined to include the object in the frame image based on the target template set and the interference template set. The method may include obtaining a target candidate area, and determining one target area from the one or more target candidate areas.

According to an embodiment, the determining of one target region from the one or more target candidate regions includes, for each of the one or more target candidate regions, each of the target candidate regions and each target template of the set of target templates; calculating a matching degree of , for each of the one or more target candidate regions, calculating a matching degree of each of the target candidate regions and each interference template in the set of interference templates; , based on the matching degree of each of the target candidate areas with each target template of the target template set and the matching degree of each of the target candidate areas with each interference template of the interference template set, each of the target candidate areas calculating a target matching degree of , and determining one target area among the one or more target candidate areas based on each target matching degree of the target candidate area.

According to an embodiment, the calculating of each target matching degree of the target candidate area includes an average or median value of matching degrees between each of the target candidate areas and each target template of the target template set, and/or and calculating each target matching degree of the target candidate area based on an average or median value of matching degrees between each of the target candidate areas and each interference template of the interference template set.

According to an embodiment, the obtaining of the one or more target candidate regions includes: determining, based on the target template set, an integrated target feature; based on the interference template set, an interference fusion feature. determining an integrated inteference feature; and obtaining the one or more target candidate regions from the frame image based on the target fusion feature and the interference fusion feature.

According to an embodiment, the determining of the target fusion characteristic based on the target template set includes integrating image characteristics of an image region corresponding to each of the target templates included in the target template set to integrate the target determining a fusion characteristic. The determining of the interference fusion characteristic based on the interference template set may include: determining the interference fusion characteristic by integrating image characteristics of an image region corresponding to each of the interference templates included in the interference template set may include

According to an embodiment, the determining of the target fusion characteristic based on the target template set may include determining the target fusion characteristic based on all target templates included in the target template set. . The determining of the interference fusion characteristic based on the interference template set may include determining the interference fusion characteristic based on all interference templates included in the interference template set.

According to an embodiment, the method may further include updating the interference template set. According to an example, the updating of the interference template set includes adding some or all of the target candidate regions other than the target region among the one or more target candidate regions as an interference template to the interference template. may include According to another example, the updating of the interference template set may include calculating a similarity of some or all of the interference fusion characteristic of the interference template set and the other target candidate region, and a part of the other target candidate region or The method may include adding a target candidate region in which the similarity is less than a threshold value among all the interference templates to the interference template.

According to embodiments, since one or more target templates are included in the target template set, information included in the target template set is richer, and it is possible to avoid using only the first frame image to detect the target object. That is, since the target template set records various characteristic information that may appear of the target object, information of the target object can be described more comprehensively, thereby improving the accuracy of target object detection.

1 is a diagram for describing an object tracking method according to an embodiment.
2 is a diagram for describing an object tracking method according to an embodiment.
3 is a flowchart of an object tracking method according to an embodiment.
4 is a diagram for explaining memory matching of an object tracking method according to an embodiment.
FIG. 5 is a diagram for describing a light-weight object detection method according to an exemplary embodiment.
6 is a view for explaining a simplified object detection method (light-weight object detection method) according to another embodiment.
7 is a diagram for describing memory update according to an exemplary embodiment.
8 is a block diagram of an object detecting apparatus according to an exemplary embodiment.
9 is a block diagram of an electronic device for object detection according to an embodiment.

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implemented form is not limited to the specific embodiments disclosed, and the scope of the present disclosure includes changes, equivalents, or substitutes included in the technical idea described in the embodiments.

Although terms such as “first” or “second” may be used to describe various elements, these terms should only be construed for the purpose of distinguishing one element from another. For example, a “first component” may be termed a “second component” and similarly a “second component” may also be termed a “first component”.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present disclosure, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, action, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present disclosure. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

When tracking a target object, changes in lighting, changes in scale, background interference, out of field of view, in-plane rotation, low resolution, fast motion, motion blur, occlusion ), characteristic information of the tracked target object may be changed. This may reduce the accuracy of object identification and/or object tracking. Embodiments disclosed herein provide an object detection method capable of solving at least one of the above problems. These methods may be executed by various electronic devices, which may be a terminal device, a server, or the like.

In one embodiment, there is provided a method for detecting and/or tracking an object from a frame image of a video comprising a plurality of frame images provided

The method detects and/or tracks an object from a frame image based on a target template set, and outputs information regarding the detected object.

The method may be used to track and detect a target object in a video, or may be used to detect a target object from an arbitrary frame image, but is not limited thereto.

A source of video to which embodiments can be applied is not limited. For example, the video source may be a movie, or may be a video captured by a visible light camera, such as a CCTV image, a monitoring image, a live video of a sporting event, or the like.

The object detection method according to an embodiment may be applied to visual object tracking. Visual object tracking, as one of the important computer vision research tasks, has various practical applications in daily life. For example, object tracking/detection according to an embodiment may include finding a target object (eg, a suspect, a missing child, a missing elderly person, a missing animal, etc.) from a monitoring image, or locating a target player from a live video of a sporting event. ) can be used to In addition, the object tracking/detection according to an embodiment may be used to determine a path traveled by the vehicle from a monitoring image, or may be used to track a small aircraft in flight. As another example, object tracking/detection according to an embodiment may be used in a smart phone to help a camera focus, generate a video effect, analyze a moving object, and the like. In the object tracking/detection according to the embodiments, a scenario applied according to actual needs may be determined, but is not limited thereto.

When applied to video tracking and/or detection, a target template set includes one or more target templates. Each of the one or more target templates corresponds to one of previous frame images of the corresponding frame image of the video. Each of the one or more target templates includes information about the object in a respective frame image determined to include a target object among previous frame images of that frame image of the video. . The object information may be an image area in which the target object is located.

The target object may be understood as an object to be tracked and/or detected, such as a moving person, animal, or means of transportation (car, aircraft, etc.). A target object to be tracked and/or detected may be selected according to actual needs, without any limitation herein. Optionally, the object to be tracked in the video (ie, the target object) may be determined by a user's indication, or may be determined by an image of a known object. For example, the user may determine the target object by pointing to an object in a video frame in which the target object first appears in the video, and the image area selected by the user in this way will be used as an initial target template of the target object. can Alternatively, the initial target template may be an image of a known object. For example, when searching for a specific person, a photo of the person may be used as an initial target template.

The target template set is stored in a storage device such as a memory or a hard disk. A storage device for storing a set of target templates is referred to herein as a "target memory". The target template included in the target template set may include an image region corresponding to the detection result of the target object in the detected frame image of the video (ie, the region in which the target object is located in each frame image). For example, the target template may be an image area in which the target object is located in the frame image. That is, the image region including the target object in the frame image may be used as the target template. Of course, according to an embodiment, the target template stored in the target template set may be feature information of a corresponding image area. For example, the target template may be image feature information representing an image region extracted from the image. That is, feature information extracted from the image region including the target object in the frame image may be used as the target template. In addition, the target template included in the target template set may be location information of the target object detected in each frame image.

Also, the target template set may include an initial target template. The initial target template may be information of the target object in the frame image determined by the user as including the target object among frame images of the video. For example, the user may determine an initial target template from a frame image in which the target object first appears in the video. A user may indicate a target object to be tracked in a frame image in which the target object first appears, or may designate an image area including the target object. The image region designated in this way may be used as an initial target template, and feature information of the image region may be used as an initial target template.

In an embodiment, the initial target template may be obtained from a separate image independent of a video to be tracked and including the target object. For example, when tracking and/or identifying a specific person, a picture of the person or image characteristic information of the picture of the person may be used as an initial target template.

The target template set may further include, in addition to the initial target template, a target template corresponding to other frame images in which the target object has already been detected. For example, in the video, a target template corresponding to at least one frame image among previous frame images of a frame image in which a target object is to be detected/tracked may be further included.

For example, when the subject frame image is the second frame image, the previous frame image is the first frame image. When it is determined that the target object is included in the first frame image, the target template set may include information (a first target template) about an image region including the target object in the first frame image.

When the corresponding image is the third frame image, previous frame images include the first frame image and the second frame image. When it is determined that the target object is included in the first frame image and the second frame image, the target template set includes information about an image area (target area) including the target object in the first frame image (the first target template) and information about an image region including a target object in the second frame image (a second target template).

In an embodiment, for the frame image, all of the target templates corresponding to all previous frame images of the frame image may be included in the target template set.

According to an embodiment, the target template set may not include target templates corresponding to all frame images determined to include the target object among previous frame images, but only some target templates. For example, with respect to the corresponding frame image, only a target template satisfying a specific condition among all previous frame images of the corresponding frame image may be included in the target template set. The specific condition may be configured according to an embodiment.

For example, whether the new target template is included in the target template set may be determined according to similarity, matching degree, etc. with target templates already stored in the target template set. For example, if the degree of similarity or matching is less than a threshold value, a new target template may be included in the target template set. By doing so, a target template similar to target templates already included in the target template set (a target template having a similarity higher than the threshold value) may not be added to the target template set. By not adding a target template similar to existing target templates, the amount of subsequent data calculation can be effectively reduced.

A target template set used when detecting a target object in a corresponding frame image may be different depending on the corresponding frame image. For example, the target template set used when detecting the target object in the 50th frame image may be different from the target template set used when detecting the target object in the 200th frame image. Similarly, the number of cartridge templates included in the set of target templates used may be different.

When tracking and/or detecting a target object in a video, detection results of the target object may be obtained by performing detection of the target object on the frame image, through the target template set, on the frame image. . The detection result may be an image area determined to include the target object in the frame image, image feature information of the image area, or location information of the target object in the frame image. . In addition, the detection result may further include their reliability. Reliability may be used to process the detection result, and the range of values may be [0, 1].

According to an embodiment, detection and/or tracking of a target object in a frame image is performed through a target template set, and a detection result of the target object in the frame image is obtained. According to this method, since the target template set includes not only the initial target template but also the target template corresponding to the image region where the target object is located in previous frame images, the information included in the target template set becomes richer. That is, due to the diversity of target templates included in the target template set, information that can be used for target object detection becomes richer, and the accuracy of the detection result is also improved.

According to an embodiment, detecting the object from the frame image includes detecting the object from the frame image based on a target template set and an interference template set.

The interference template set includes one or more interference templates. Each of the one or more interference templates includes information about an interference object that interfered with the detection of the object among previous frame images of the frame image of the video.

The interfering object may also be referred to as an interfering object, that is, an object capable of interfering with the detection of a target object. An interfering object may be an object, texture (non-object) or shape that is (visually) similar to the target object being tracked in the frame image of the video.

The interference template set is stored in a storage device such as a memory or a hard disk. A storage device storing a set of interference templates is referred to herein as "interference memory". The interference template included in the interference template set may include an image region (ie, a region in which the interference object is located in the frame image) including the interference object that interferes with detection of the target object in frame images of the video. For example, the interference template may be an image region in which the interference object corresponding to the target object is located in the frame image. That is, the image region including the interference object in the frame image may be used as the interference template. Alternatively, the interference template may include an image region other than the image region (target region) determined to include the target object in the frame image. Of course, according to an embodiment, the interference template stored in the interference template set may be feature information of the image region including the interference object. For example, the interference template may be image feature information representing an image region, extracted from the image. That is, feature information extracted from an image region including an interference object in each frame image may be used as an interference template. Also, the interference template included in the interference template set may be location information of the interference object detected in the frame image.

In one embodiment, the interference template set includes interference templates corresponding to the target object. Each of the interference templates is an image region in which the interference object of the target object is located in previous image frames of the subject frame image.

For example, when the corresponding frame image is the third frame image, the previous frame images include the first frame image and the second frame image. When the target object is selected by the user in the first frame image, the initial target template is an image area selected by the user from among the first frame images. Also, the interference object may be selected from the first frame image by the user's selection. In this case, information about the image region corresponding to the selected interference object may be added to the interference template set. Optionally, the detection of the interfering object corresponding to the target body may not proceed with respect to the first frame image. That is, the detection of the interfering object from the first frame may not proceed.

When a target object is detected with respect to the second frame image, a plurality of target candidate regions are obtained, and a portion of target candidate regions (image regions) other than the target region determined to include the target object among the plurality of target candidate regions. Alternatively, all of the target object may be determined as the image region in which the interfering object is located. It can be added to the interference template set as an interference template.

When the target object is detected for the third frame image, the target template (target template obtained from the first frame and target template obtained from the second frame) included in the target template set and the interference templates included in the interference template set (interference template obtained from the second frame image) may be used. Also, when a target object is detected for the third frame image, an additional interference template may be obtained from the third frame image by the same method.

In an embodiment, when detecting a target object from a frame image, both information of a target template set and an interference template set may be considered. That is, a target object having a high similarity to the target template set and a low similarity to the interference template set may be detected.

Information on the interference object may include, for example, an image area determined to contain the interference object in the previous frame image, image feature information of the image area, or interference in the previous frame image. It may be location information of an object.

A target object in a video has various changes due to lighting transformations, rotation of the target object, fast movement, motion blur, occlusion, deformation, and background interference, and the like. According to embodiments, the target memory (or target template set) remembers these changes of the target object, so that the target object can be more accurately identified and tracked.

Further, according to embodiments, since the interference memory (or the interference template set) stores image characteristics of the interference object that interfere with the detection of the target object, the influence of the interference object on the detection of the target object can be eliminated or reduced .

This method of detecting a target object by integrating the target memory (or target template set) and the interference memory (or interference template set) can not only comprehensively reflect various characteristics of the target object, but also By eliminating or reducing the negative impact, the accuracy of the identification and/or tracking of the target object can be greatly improved.

In an embodiment, the detecting of the object from the frame image includes: obtaining one or more target candidate areas determined to include the object in the frame image, and the one or more target candidates and determining one target area from the area.

The target candidate area may be referred to as a candidate area indicating a possible position of the target object in the frame image, an anchor region, an anchor box, or an anchor. The location and size of the target candidate area may vary.

In an embodiment, when detecting a target object in an image frame, one or more target candidate regions may be obtained from the image frame by using a set of target templates.

In one embodiment, the step of obtaining the one or more target candidate areas comprises: determining a plurality of search areas within the frame image; extracting image features of each of the plurality of search areas; extracting to obtain a search area feature, calculating a correlation between each search area feature of the plurality of search areas and the target fusion feature, and based on the correlation and determining the one or more target candidate areas among the plurality of search areas.

For example, a plurality of search area boxes may be obtained from a subject frame image. For example, a plurality of search area boxes may be set around an area determined to include the target object in the previous frame. According to an embodiment, all of the images in the search area boxes may be target candidate areas. According to another embodiment, the image of each search area box may be matched one by one with each target template of the target template set. Alternatively, the image of each search area box may be matched with an integration template obtained by integrating all target templates of the target template set. After the matching, one or more search areas having a high matching degree are selected as target candidate areas. Then, a target area is selected from among the target candidate areas, and a detection result of the target object is obtained based on the target area.

It may be understood that embodiments may be implemented by a previously trained neural network model.

1 illustrates a neural model implementing an object tracking method according to an embodiment.

In FIG. 1 , a video including, by way of example, a sports scene of an athlete is provided. The target object you want to track in the video is a specific athlete (you might call it a "target player"). A current frame image 110 is a frame image in a video for which a target athlete is to be detected. In the embodiment of FIG. 1 , when a user finds a target object (target athlete) during playback of a video, the user indicates the target object in this frame image. Then, information of a part of this frame image (this part includes a target object) is used as an initial target template 120 .

The memory pool 130 includes a target memory 131 . The target memory 131 is a storage space for storing a target template set. The memory pool may be implemented as a memory, a hard disk (HDD), a flash memory, or the like, but is not limited thereto. The target template set stores target templates corresponding to frame images before the initial target template 120 and the current frame image 110 .

In one embodiment, in order to detect a target object in the current frame image 110 , the current frame image 110 is input to a backbone network 140 . The backbone network 140 is a network for extracting image features using a neural network, and MobileNet, ResNet, Xception Network, etc. may be used, but is not limited thereto. The backbone network 140 receives the current frame image 110 , and outputs an image feature 145 corresponding to the current frame image 110 . For example, the backbone network 140 may output W x H x C as the image feature 145 of the current frame image 110 . W x H x C is a feature map, where W and H represent the width and height of the feature map, respectively, and C represents the number of channels (that is, the number of features) of the feature map. .

According to another embodiment, a plurality of search areas are determined within the current frame image 110 . The search area is an area of a partial image within the current frame image 110 . Each of the plurality of search areas is input to a backbone network 140 . The backbone network 140 outputs a search area feature 145 for each of the plurality of search areas.

When the target templates stored in the target template set are images, target templates of the target template set stored in the target memory 131 are input to the backbone network 150 to obtain one or more target candidate regions. The backbone network 150 is a network for extracting image features using a neural network, and MobileNet, ResNet, Xception Network, etc. may be used, but is not limited thereto. The backbone network 150 may be the same as or different from the backbone network 140 . The backbone network 150 receives the target templates and outputs image features corresponding to the target templates to a recall network 163 .

According to another embodiment, the target templates stored in the target template set may be features of the corresponding image. In this case, since it is not necessary to extract the image features corresponding to each target template through the backbone network 150 , the target template is directly input to the recall network 163 .

According to an embodiment, the processor 160 may include a recall network 163 , a correlation calculator 166 , and an anchor processor 169 . The recall network 163 , a correlation calculator 166 , and an anchor processor 169 are conceptual classifications according to functions, and do not necessarily have to be physically separately implemented. For example, recall network 163 , correlation calculater 166 , and anchor processor 169 may all be other functions processed by processor 160 , as described above. They are separated for convenience.

The recall network 163 integrates an image feature of an image region corresponding to each of the target templates included in the target template set, and outputs an integrated target feature. The recall network 163 is a network that fuses image features input using a neural network, and a feature fusion convolutional neural network (FFCNN) or the like may be used, but is not limited thereto.

A correlation calculator 166 calculates a correlation between the image feature 145 of the current frame image 110 and the target fusion feature. Alternatively, the correlation calculator 166 calculates a correlation between the search area feature and the target fusion feature of a plurality of search areas in the current frame image 110 . . Based on the calculated correlation, one or more target candidate areas (also referred to as "candidate anchors") among a plurality of search areas are determined.

The correlation generated by the correlation calculator 166 is input to the anchor processor 169 . The anchor processor 169 may output confidence and/or regression results for each of one or more target candidate regions (candidate anchors). In one embodiment, the regression result may include a regression position (x, y, w, h). Here, x and y may represent abscissa and ordinate of a specific point of a corresponding target candidate area. For example, x and y may be the abscissa and ordinate of an arbitrary vertex in the target candidate region, such as a center point of the target candidate region, a vertex of an upper left corner, and the like. Also, w and h may represent the width and height of the target candidate area.

According to the confidences and/or regression results of the target candidate regions, one target candidate region (final anchor) 170 is selected from among the plurality of target candidate regions (candidate anchors). do.

Based on the confidence of the selected target candidate region (final anchor) 170 and/or the regression result, the position of the target object in the current frame image 110 and the corresponding detection confidence ( detection confidence) is obtained. In this way, information regarding the detected target object is output.

Also, according to the tracking/detection result corresponding to the current frame image 110 , the target memory 131 is updated. For example, the detected target candidate region (final anchor) 170 is added to the target memory as a new target template. In this way, the target memory 131 is updated, and the updated target memory 131 is used to detect and/or track the target object of the next frame image.

According to an embodiment, detecting the target object from the frame image includes detecting the target object from the frame image based on a target template set and an interference template set. The interference template set includes one or more interference templates. Each of the one or more interference templates includes information about an interference object that interferes with the detection of the object among previous frame images of a subject frame image of the video. include

According to an embodiment, detecting the target object from the frame image includes: obtaining one or more target candidate regions based on a set of target templates; based on a set of interference templates, one target from the one or more target candidate regions determining an area, and detecting a target object based on the target area.

Optionally, when detecting a target object from the current frame image, one or more target candidate regions corresponding to the current frame image may be obtained according to the target template set and the interference template set, and the specific process is as follows.

Through the backbone network, image features are extracted from the target template set and the interference template set in the memory pool. The recall network obtains an integration template that fuses the features from the extracted image features. When detecting a target object in the current frame image, by searching several areas of the current frame image, several search area boxes are obtained. Then, each search area box is matched with the fusion template, and one or more target candidate areas with a high degree of matching are obtained. Then, one target area is finally selected from among the one or more target candidate areas, and a detection result of a target object corresponding to the current frame image is obtained based on the selected target area.

Optionally, in the target tracking in the neural network model, in addition to using the target memory, an interference memory may be used, and the specific process is as follows.

Step 1, a first frame image is selected from a video sequence.

Step 2, the user indicates the target in the first frame image. According to an indication, an image feature of an image region in which a target object is located is extracted from the first frame image. From the extracted image features, an initial target template is obtained. The initial target template is added to the set of target templates. The interference template set is empty.

Step 3, the second frame image of the video sequence is selected. Based on the initial target template included in the target template set, the target position of the target object in the second frame image is predicted, and the prediction result is output.

Step 4, based on the prediction result, tracking information such as the target position of the target object and the prediction reliability in the second frame image is output.

Step 5, it is determined whether the prediction result of the second frame image satisfies a memory update condition. If satisfied, the target template set is updated, otherwise it is not updated. For example, if satisfied, a target template corresponding to the second frame image may be added to the target template set.

Step 6, the third frame image of the video sequence is selected. At this time, the target template set and the interference template set are loaded from the memory pool according to the target template set and the interference template set.

Step 7, based on the target template set and the interference template set, a target position of a target object in a subsequent frame image is predicted, and a prediction result is output. Then, the next frame proceeds in the same way.

In order to describe the above method in more detail, it will be described with reference to the example shown in FIG. 2 .

2 illustrates a neural model implementing an object tracking method according to an embodiment.

In FIG. 2 , a target object to be tracked is a specific athlete (which may be referred to as a “target player”). A current frame image 210 is a frame image in a video for which a target athlete is to be detected. In the embodiment of FIG. 2 , when a user finds a target object (target athlete) during playback of a video, the user indicates the target object in this frame image. Then, information of a part of this frame image (this part includes a target object) is used as an initial target template 220 .

The memory pool 230 includes a target memory 231 and an interference memory 232 . The target memory 131 is a storage space for storing a target template set. The interference memory 232 is a storage space for storing an interference template set.

The target template set stores target templates corresponding to frame images before the current frame image 210 . A target template is information about a target object in frame images of a video. For example, the target template may be an image area including a target object or feature information of the image area. The target template set may further store the initial target template 220 .

The interference template set stores interference templates corresponding to frame images before the current frame image 210 . The interference template is information about an interference object that has prevented detection of a target object in frame images of a video. For example, the interference template may be an image region including an interference object, or may be feature information of the image region.

In an embodiment, in order to detect a target object in the current frame image 210 , all or a part (search area) of the current frame image 210 is input to a backbone network 240 . The backbone network 240 may extract a search area feature using a neural network. The search area feature is an image feature corresponding to the search area. For example, the backbone network 240 may output W x H x C as the search area feature 245 corresponding to the search area of the current frame image 210 . W x H x C is a feature map, where W and H represent the width and height of the feature map, respectively, and C represents the number of channels (that is, the number of features) of the feature map. .

To obtain one or more target candidate regions, the target template set stored in the target memory 231 of the memory pool 230 and the interference template set stored in the interference memory 232 are input to the backbone network 250 . The backbone network 250 extracts image features corresponding to each target template in the target template set, and extracts image features corresponding to each interference template in the interference template set. These image features are input to the recall network 263 .

According to an embodiment, the processor 260 may include a recall network 263 , a correlation calculator 266 , and an anchor processor 269 . The recall network 263 , a correlation calculator 266 , and an anchor processor 269 are conceptually divided according to functions, and do not necessarily have to be physically separately implemented. For example, recall network 263 , correlation calculater 266 , and anchor processor 269 may all be other functions processed by processor 260 , as described in They are separated for convenience.

The recall network 263 integrates an image feature of an image region corresponding to each of the target templates included in the target template set, and outputs an integrated target feature. Also, the recall network 263 outputs an integrated interference feature by integrating an image feature of an image region corresponding to each of the interference templates included in the interference template set. The recall network 263 is a network that fuses image features input using a neural network, and a feature fusion convolutional neural network (FFCNN) may be used, but is not limited thereto.

According to an embodiment, an integrated target feature is determined by integrating an image feature of an image region corresponding to each of the target templates included in the target template set. Then, based on the target fusion characteristic, one or more target candidate areas determined to include the object in the frame image are obtained. One target area is determined from the one or more target candidate areas.

According to one embodiment, correlation calculater 266 , each of a search area feature 245 corresponding to each of a plurality of search areas of current frame image 210 and a target A correlation of a target feature kernel (eg, a target fusion feature) is calculated. According to the degree of correlation, one or more candidate anchors (target candidate areas) among the search areas are determined. For example, K candidate anchors (target candidate areas) having large correlation values are selected and recorded as Top-K candidate anchors (target candidate areas).

According to one embodiment, correlation calculater 266 also provides each of the search area features 245 corresponding to each of the search areas and an interference feature kernel. Calculate the correlation of (eg, interference fusion features). According to the degree of correlation, one candidate anchor (target candidate area) having the lowest degree of correlation (or matching degree) among the search areas is selected. The selected candidate anchor (target candidate area) is recorded as a Bottom-1 candidate anchor.

The correlation generated by the correlation calculator 266 is input to the anchor processor 269 . The anchor processor 269 may output confidence and/or regression results for each of one or more target candidate regions (candidate anchors). In one embodiment, the regression result may include a regression position (x, y, w, h). Here, x and y may represent abscissa and ordinate of a specific point of a corresponding target candidate area. Based on this, the position and reliability of the target object in the current frame image 210 may be determined.

According to the confidences and/or regression results of the target candidate regions, one target candidate region (final anchor) 270 is selected from among the plurality of target candidate regions (candidate anchors). do.

In addition, it is also possible to update the target memory 231 and the interference memory 232 according to the tracking result. For example, the detected target candidate region (final anchor) 270 may be added to the target memory 231 as a new target template. In addition, some or all of the plurality of target candidate regions (candidate anchors) excluding the finally selected target candidate region (final anchor) 270 may be added to the interference memory 232 . . In an embodiment, a candidate anchor having the lowest degree of matching with the final anchor 270 may be added to the interference memory 232 .

3 is a flowchart of an object tracking method according to FIG. 2 . Referring to FIG. 3 , an object tracking method according to FIG. 2 is described.

In step 310, the target template set and the interference template set are read from the target memory 231 and the interference memory 232, respectively.

In step 320, the recall network 263 integrates an image feature of an image region corresponding to each of the target templates included in the target template set to form an integrated target feature ( Outputs a “target feature kernel” 330 . In addition, the recall network 263 integrates an image feature of an image region corresponding to each of the interference templates included in the interference template set to form an integrated interference feature (“interference feature kernel”). (also called "interference feature kernel)") (340).

In step 350 , correlation calculater 266 , each of the search area features 245 corresponding to each of a plurality of search areas of the current frame image 210 and the target Calculate the correlation of the fusion feature (target feature kernel). Further, a correlation calculator 266 is configured to perform a correlation between each of the search area features 245 corresponding to each of the search areas and an interference fusion feature (interference feature kernel). to calculate

In operation 360 , one or more target candidate regions among the search regions are determined according to the correlation calculation result of operation 350 .

In step 370, one target candidate region (final anchor) of one or more target candidate regions (candidate anchors) according to regression results and/or confidences of the one or more target candidate regions. ) (270) is selected. That is, one target area determined to include the target object in the current frame image 210 is determined. This is understood as the same operation as determining the position of the target object in the current frame image 210 .

In step 380 , the target memory 231 and the interference memory 232 are updated according to the detection result.

4 is a diagram for explaining memory matching of an object tracking method according to an embodiment.

As shown in FIG. 4 , the memory pool 430 includes a target memory 431 and an interfering memory 432 . The target memory 431 stores a target template set. The interference memory 432 stores an interference template set. The target template set stored in the target memory 431 of the memory pool 430 and the interference template set stored in the interference memory 432 are input to the backbone network 440 .

The backbone network 440 extracts image features corresponding to each target template in the target template set. The backbone network 440 extracts image features corresponding to each interference template in the interference template set.

The recall network 450 integrates 451 image features corresponding to target templates received from the backbone network 440 to form an integrated target feature ( 453) is created. The recall network 450 integrates 455 image features corresponding to the interference templates received from the backbone network 440 to form an integrated intereference feature 457 ) is created.

A correlation calculator 460 is configured to calculate a correlation ( ) of a target fusion feature 453 with each of a search area feature 445 corresponding to each of a plurality of search areas of the current frame image. correlation) is calculated. Further, a correlation calculator 266 is configured to correlate each of the search area features 445 corresponding to each of the plurality of search areas with the coherent fusion feature 457 . to calculate

The correlation generated by the correlation calculator 460 is input to the anchor processor 470 . The anchor processor 470 may output one or more anchors based on the degree of matching using, for example, a neural network. The anchor processor 470, based on the correlation between the target fusion feature 453 and each of the search region features 445 corresponding to each of the search regions output by the correlation calculation unit 460, is configured for the K anchors. (search areas) can be selected. For example, according to the matching degree based on the correlation, K anchors (search areas) having a high matching degree may be selected. K is an integer greater than or equal to 1 and may be selected according to an embodiment. For example, K may be 3. The selected K anchors are stored as Top-K anchors 475 . The Top-K anchor 475 becomes the target candidate area.

According to an embodiment, the correlation calculator 480 is interference fusion with each of the Top-K anchors 475 (or feature information corresponding to each of the Top-K anchors 475 ). Calculate the correlation of features 457 . The degree of matching between the target candidate regions 475 and the interference fusion feature 457 is calculated by the correlation calculator 480 . One target candidate region (candidate anchor) having the lowest degree of matching with the interference fusion feature 457 among the target candidate regions 475 is selected as the Bottom-1 anchor. This selected Bottom-1 anchor is determined as the target area 490 of the target object. The target area corresponds to a position in the current frame image where the target object is determined to be located.

Based on the confidence of the selected target area 490 and/or the regression result, a detection confidence for the position of the target object in the current frame image is calculated.

It can be understood that the configurations shown in FIGS. 1, 2 and 4 are exemplary for the purpose of explanation. Embodiments are not limited to these configurations. For example, the target templates stored in the target memory of the memory pool of FIGS. 1 , 2 and 4 and the interference templates stored in the interference memory are exemplary and may be updated according to subsequent frames of the video. The difference between the embodiment of Fig. 1 and the embodiment of Fig. 2 is that the embodiment of Fig. 1 uses a target memory (or target templates) to detect the target object, and the embodiment of Fig. 2 uses the embodiment of Fig. 2 to detect the target object. It is to use a target memory (or target templates) and an interference memory (or interference templates) together.

In embodiments, a target object may be detected based on a target memory and/or an interfering memory. Using these embodiments, when the target object moves, it is possible to more comprehensively record various changes of the target object, and effectively improve the detection accuracy.

According to an embodiment, the determining of one target region from the one or more target candidate regions comprises: for each of the one or more target candidate regions, each of the target candidate regions and each target template of the set of target templates calculating a matching degree, for each of the one or more target candidate areas, calculating a matching degree between each of the target candidate areas and each interference template in the set of interference templates; Based on the matching degree of each of the target candidate areas with each target template of the target template set and the matching degree of each of the target candidate areas with each interference template of the interference template set, each of the target candidate areas calculating a target matching degree, and determining one of the one or more target candidate areas based on the target matching degree of each of the target candidate areas.

In this embodiment, the matching degree between each target candidate area and each target template is obtained, which can be recorded as the first matching degree. For any target candidate region, a matching degree between the target candidate region and each interference template of the interference template set is calculated, and a plurality of matching degrees between the target candidate region and the interference templates can be obtained. The average value (or median value) of the plurality of matching degrees is selected as the matching degree between the target candidate area and the interference template. A matching degree between each target candidate area and each interference template may also be calculated, and this may be recorded as a second matching degree. By subtracting the value of the first matching degree and the value of the second matching degree in a 1:1 correspondence, respectively, a first target matching degree may be calculated. A target candidate region having the largest value of the first target matching degree is identified as a target region.

According to an embodiment, a target candidate area having a high degree of matching with the target memory and a low degree of matching with the interfering memory may be identified as the target area according to the matching degree. According to this, it is possible to select an object that is as close as possible to the target templates for the currently tracked target object, and at the same time to select an object that is least affected by the interference templates. Accordingly, the accuracy of detection of the target object is increased.

In an embodiment, the detection result of the target object may be obtained by using each target template of the target template set and each interference template of the interference template. Specifically, one or more target candidate regions may be obtained by one-to-one matching of each target template of the target template set and each interference template of the interference template with each of a plurality of search region boxes corresponding to a portion in a frame image, respectively. Then, one target area corresponding to the target object may be selected from the one or more target candidate areas, and a detection result of the target object may be obtained based on the target area.

The above is only an example, and the present embodiment is not limited thereto.

FIG. 5 is a diagram for describing a light-weight object detection method according to an exemplary embodiment.

는 타겟 메모리(531)에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 특징 벡터를 나타낸다(denote). 또한, 간섭 템플릿 세트에 포함되는 간섭 템플릿들의 각각은 대응하는 이미지 영역의 특징 벡터(feature vector)이다. 즉, 도 5에서,

는 간섭 메모리에 저장된 간섭 템플릿 세트 내 i번째 간섭 템플릿의 특징 벡터를 나타낸다.As shown in FIG. 5 , a memory pool 530 includes a target memory 531 and an interference memory 532 . The target memory 531 stores a target template set. The interference memory 532 stores an interference template set. In this embodiment, each of the target templates included in the target template set is a feature vector of a corresponding image region. That is, in Fig. 5,

denotes a feature vector of the i-th target template in the target template set stored in the target memory 531 (denote). In addition, each of the interference templates included in the interference template set is a feature vector of a corresponding image region. That is, in Fig. 5,

denotes a feature vector of the i-th interference template in the interference template set stored in the interference memory.

In the embodiment of FIG. 5 , the target template corresponding to the first frame image is not stored in the target memory 531 and may be separately input. By processing the template image 520 extracted from the first frame image through the backbone network 550 and the adjustment layer 553 , an initial target template 556 is obtained.

According to an embodiment, the template image 520 extracted from the first frame image may be an image of a search area (image area) determined by a user to include a target object among frame images of a video. According to another embodiment, the first frame image is an image including a target object, and may be a separate image independent of the video. The target template, as described above, represents the image features of the corresponding image.

A plurality of search areas are selected from the current frame image 510 . The plurality of search areas may be areas obtained by dividing the current frame image 510 into a plurality of pieces. A plurality of search areas may partially overlap each other. When the target area of the target object is determined in the previous frame, a plurality of search areas may be determined in the current frame based on the location of the target area of the previous frame. By processing each of the plurality of search areas through the backbone network 540 and an adjustment layer 543 , a search area feature 546 corresponding to the search area is obtained.

The backbone networks 540 and 550 are networks for extracting image features using a neural network, and MobileNet, ResNet, Xception Network, etc. may be used, but is not limited thereto. In FIG. 5 , feature information output by the backbone network is adjusted by the adjustment layers 553 and 543 , but may be omitted according to embodiments. The adjustment layers 553 and 543 may extract only some of the feature information output by the backbone network or adjust the value of the feature information.

A correlation calculator 560 calculates a correlation between the search area feature 546 corresponding to the search area and an initial target template 556 . For example, the correlation calculator 560 may calculate the correlation using a depthwise correlation method.

The correlation generated by the correlation calculator 560 is input to the anchor processor 570 . The anchor processor 570 selects K anchors (search areas) having a high matching degree as the Top-K anchors 575 according to the matching degree based on the correlation. The selected anchor (search area) is identified as a target candidate area.

For each of the K target candidate regions 575 , a score of Equation 1 is calculated, for example by a computing function 580 .

는 타겟 메모리에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 특징 벡터를 나타낸다(denote).

는 간섭 메모리에 저장된 간섭 템플릿 세트 내 i번째 간섭 템플릿의 특징 벡터를 나타낸다.

는 K개의 타겟 후보 영역들(Top-K 앵커들) 중 k번째 타겟 후보 영역(앵커)의 특징 벡터를 나타낸다.

는

와

의 상관도를 계산한 매칭값을 나타낸다.

는 타겟 메모리에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 가중치를 나타낸다.

는 간섭 메모리에 저장된 간섭 템플릿 세트 내 i번째 간섭 템플릿의 가중치를 나타낸다. m1은 타겟 템플릿 세트에 포함된 타겟 템플릿들의 개수이고, m2는 간섭 템플릿 세트에 포함된 간섭 템플릿들의 개수이다.here,

denotes a feature vector of the i-th target template in the target template set stored in the target memory (denote).

denotes a feature vector of the i-th interference template in the interference template set stored in the interference memory.

denotes a feature vector of a k-th target candidate region (anchor) among K target candidate regions (Top-K anchors).

Is

Wow

Shows the matching value calculated by the correlation of .

denotes the weight of the i-th target template in the target template set stored in the target memory.

denotes the weight of the i-th interference template in the interference template set stored in the interference memory. m1 is the number of target templates included in the target template set, and m2 is the number of interference templates included in the interference template set.

Then, the target candidate region having the largest score calculated by Equation 1 is selected as the target region (final anchor) 590 . This can be expressed by Equation (2).

Through Equations 1 and 2, a target candidate area most similar to the target templates included in the target template sets and least similar to the interference templates included in the interference template sets is selected as the target area (final anchor) .

The number of target templates m1 and the number of interference templates m2 may vary according to a current frame image for identifying a target object.

According to an embodiment, the sum of the weights of all target templates in the target template set may be 1, and the weight of the initial target template may be set to be the highest. However, the method of setting the weight of the target templates is not limited thereto, and may be set in various ways.

According to an embodiment, the sum of weights of all interference templates in the interference template set is 1, and the weight of each interference template may be set according to a distance between an image region in which each interference template is located and an image region in which a target object is located. However, the method of setting the weight of the interference templates is not limited thereto, and may be set in various ways.

According to an embodiment, after extracting an image feature (initial target template) corresponding to the first frame image, the initial target template may be stored in the target memory 531 . Then, thereafter, the initial target template may be loaded from the target memory 531 and used.

6 is a view for explaining a simplified object detection method (light-weight object detection method) according to another embodiment.

는 타겟 메모리(631)에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 특징 벡터를 나타낸다(denote).As shown in FIG. 6 , the memory pool 630 includes only a target memory 631 and does not include an interference memory. The target memory 531 stores a target template set. In this embodiment, each of the target templates included in the target template set is a feature vector of a corresponding image region. That is, in Fig. 6,

denotes a feature vector of the i-th target template in the target template set stored in the target memory 631 (denote).

A plurality of search areas are selected from the current frame image 610 . Each of the plurality of search areas is processed through a backbone network 640 and an adjustment layer 643 , and a search area characteristic 646 corresponding to the search area is obtained.

An initial image 620 including the target object is processed through a backbone network 650 and an adjustment layer 653 , and image features 656 corresponding to the initial image 620 are obtained. This image feature 656 is used as an initial target template.

A correlation calculator 660 calculates a correlation between the search area feature 646 corresponding to the search area and an initial target template 656 . The correlation generated by the correlation calculator 660 is input to the anchor processor 670 . The anchor processor 670 selects K anchors (search areas) with high correlation as Top-K anchors 675 based on the correlation. The selected anchor (search area) is identified as a target candidate area.

For each of the K target candidate regions 675 , the score of Equation 3 is calculated by, for example, by a computing function 680 .

는 타겟 메모리에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 특징 벡터를 나타낸다(denote).

는 K개의 타겟 후보 영역들(Top-K 앵커들) 중 k번째 타겟 후보 영역(앵커)의 특징 벡터를 나타낸다.

는

와

의 상관도를 계산한 매칭값을 나타낸다.

는 타겟 메모리에 저장된 타겟 템플릿 세트 내 i번째 타겟 템플릿의 가중치를 나타낸다. m1은 타겟 템플릿 세트에 포함된 타겟 템플릿들의 개수이다.here,

denotes a feature vector of the i-th target template in the target template set stored in the target memory (denote).

denotes a feature vector of a k-th target candidate region (anchor) among K target candidate regions (Top-K anchors).

Is

Wow

Shows the matching value calculated by the correlation of .

denotes the weight of the i-th target template in the target template set stored in the target memory. m1 is the number of target templates included in the target template set.

Then, the target candidate region (Top-1 anchor) having the largest score calculated by Equation 3 is selected as the target region (final anchor) 690 . This can be expressed by Equation (4).

According to an embodiment, the initial target template may be replaced with a fusion template of all or some of the target target templates included in the target template set, unlike the invention described above.

The method according to an embodiment determines an integrated target feature by integrating an image feature of an image region corresponding to each of the target templates included in the target template set, and Based on the target fusion characteristic, one or more target candidate areas determined to include the object in the frame image may be acquired. Then, one target area is determined from the one or more target candidate areas.

The obtaining of the one or more target candidate areas may include: determining a plurality of search areas within the frame image; extracting image features of each of the plurality of search areas to obtain a search area feature obtaining, calculating a correlation between a feature of each search region of the plurality of search regions and the target fusion feature, and based on the correlation, the at least one target candidate among the plurality of search regions It may include determining the area.

In an embodiment, when determining a target area from one or more target candidate areas, the following method may be used. For an arbitrary target candidate area, a matching degree with each target template of the target template set is calculated, to obtain a plurality of matching degrees. The average value of the plurality of matching degrees may be taken as the matching degree between the corresponding target candidate region and the target template. Finally, a target candidate area having the highest matching degree among each target candidate area is taken as the final target area.

According to an embodiment, when the final target area satisfies a predetermined condition, the method may further include updating the target template set.

The updating of the target template set includes: calculating a similarity of a final target region and a target fusion characteristic of the target template set; and if the similarity is less than a threshold, the final target region as a target template. It may include adding to the target template.

The updating of the target template set includes calculating a similarity of each of all target templates of the target template set and a final target area, and when all of the similarities are less than a threshold value, the final target area It may include adding (add) to the target template as a target template.

Optionally, a similarity between the final target region and each target template in the set of target templates may be calculated. When all of the calculated similarities are equal to or less than a set threshold (eg, 0.9, etc.), a target template corresponding to the final target area may be added to the target template set. Alternatively, if the fusion similarity (eg, the average value of the calculated similarities) corresponding to the calculated similarity is less than or equal to a set threshold (eg, 0.9, etc.), the target template corresponding to the final target region may be added to the target template set.

According to this, if the final target area is similar to target templates included in the existing target template set, it is not included in the target template set. This is because the final target area similar to the target templates already included does not significantly improve the performance of the existing target template set. In addition, since the target template set can include the latest characteristics of the target object through the update of the target template set, the integrity of the target template set is improved.

According to an embodiment, the method may further include updating the interference template set.

In an embodiment, a part or all of the target candidate regions other than the final target region among one or more target candidate regions may be added to the interference template as an interference template. For example, an image region (or an image feature of this image region) corresponding to an anchor other than the Bottom-1 anchor among Top-K anchors may be selected as the interference template.

According to an alternative embodiment, for some or all of the target candidate regions other than the final target region among the one or more target candidate regions, a degree of similarity with the interference fusion characteristic of the interference template set is calculated. A target candidate region in which the calculated similarity is less than the threshold may be added to the interference template as the interference template. According to this, when a candidate interference area selected to be added to the interference template is similar to the interference templates included in the existing interference template set, it is not included in the interference template set. This is because the candidate interference region similar to the interference templates already included does not significantly improve the performance of the existing interference template set.

According to an embodiment, a target area in which a target object is located and a candidate interference area (image area) in which a possible interference object with respect to the target object is located among the frame images are obtained. For a candidate interference region in which a possible interference object is located, a first degree of similarity between the candidate interference region and the target region and a second similarity between the candidate interference region and each interference template in the interference template set are determined. Whether to update the interference template set is determined based on the first and second similarities with respect to the candidate interference region in which each possible interference object is located.

According to an embodiment, with respect to an image region (candidate interference region) in which any possible interfering object is located, a first similarity between the image region and the target region (the first similarity indicates a distance between the image region and the target region) may be determined), and a second degree of similarity between the corresponding image region and each interference template among the interference template sets (the second similarity may indicate a similarity between the corresponding image region and each interference template) may be determined. Then, based on the first degree of similarity and the second degree of similarity, it is possible to determine a candidate interference region to be included in the interference template set from among the candidate interference regions. For example, a candidate interference region having a large first similarity and a small second similarity may be selected as a candidate interference region to be included in the interference template set.

Hereinafter, the update of the target memory and/or the interference memory will be described with reference to the drawings.

7 is a diagram for describing memory update according to an exemplary embodiment.

As shown in FIG. 7 , the finally selected Bottom-1 anchor 790 is information on the image area in which the target object is determined to be located in the current frame image. That is, the Bottom-1 anchor 790 is the target area (final anchor). The Bottom-1 anchor 790 may be an image region or a feature map indicating image features for the image region, but is not limited thereto.

The detection result of the target object (eg, the Bottom-1 anchor 790 ) and the target templates of the target template set stored in the target memory 731 are compared. It is determined whether a similarity between the detection result of the target object and the target template set is less than a threshold value. In one example, a similarity of the target area 790 and each of all target templates of the target template set is calculated. In another example, a target fusion characteristic of a set of target templates and a similarity of the target region are calculated. In other words, it is determined how similar the detected target area 790 is to target templates already stored in the target memory 731 . If the similarity is less than a threshold (eg, 90%), the target template set stored in the target memory 731 is updated based on the detected target area 790 . For example, the detected target area 790 is added to the target template set as a new target template. If the similarity is greater than or equal to a threshold (eg, 90%), the target memory 731 is not updated.

Among the Top-K candidate anchors 765 , the Bottom-K2 candidate anchors 795 among the remaining anchors except for the Bottom-1 anchor 790 selected as the target area are selected as candidate interference anchors. . In order to find the Bottom-1 anchor 790 among the Top-K candidate anchors 765 , the matching degree for the Top-K candidate anchors 765 is calculated using the target template set and the interference template set. The anchor having the greatest matching degree among the Top-K candidate anchors 765 is selected as the Bottom-1 anchor 790 . Then, K2 anchors having a matching degree of a small score among the Top-K candidate anchors 765 are determined as the Bottom-K2 candidate anchors 795 .

The distance between each of the candidate interference anchors 795 and the position of the detected target object (target area) in the current frame image is calculated. It is determined whether the distance between the candidate interference anchor 795 and the location of the target object is greater than a threshold degree. If the distance is greater than the threshold, this candidate interference anchor may be included in the interference object 740 . If the distance is less than the threshold, this candidate interfering anchor is considered not an interfering object, and information about this candidate interfering anchor is discarded.

The interference templates of the selected interference object 740 and the interference template set stored in the interference memory 732 are compared. It is determined whether the similarity between the interference object 740 and the interference template set is less than a threshold value. In one example, a similarity of the interference object 740 and each of all interference templates of the interference template set is calculated. In another example, the similarity of the interference object 740 and the interference fusion characteristic of the interference template set is calculated. In other words, it is determined to what extent the detected interference object 740 is similar to the interference templates already stored in the interference memory 732 . If the similarity is less than a threshold (eg, 90%), the interference template set stored in the interference memory 732 is updated based on the detected interference object 740 . For example, the detected interference object 740 is added to the interference template set as a new interference template. If the similarity is greater than or equal to a threshold (eg, 90%), the interference memory 731 is not updated.

According to an embodiment, maintenance and update of the target template set and the interference template set may be performed according to the detection result of the target object. That is, by keeping the target memory and the interference memory up to date, it is possible to provide richer and more diverse information for detection of the next target object, and to improve detection accuracy.

Table 1 shows the case in which neither the target memory nor the coherence memory is used (the case where neither the target template set and the coherence template set are used) (case 1), when only the target memory is used (only the target template set is used) (case 2), the success rate and accuracy of object detection/tracking in the case where both the target memory and the interference memory are used (when both the target template set and the interference template set are used) (case 3).

As described in Table 1, when only the target memory is used (when only the target template set is used) (case 2), and when both the target memory and the interference memory are used (when both the target template set and the interference template set are used) ( In all case 3), it can be seen that the success rate and accuracy are improved compared to the case in which neither the target memory nor the interference memory is used (when neither the target template set nor the interference template set is used) (case 1). In addition, the case where both the target memory and the interference memory were used (when both the target template set and the interference template set were used) (case 3) was compared with the case where only the target memory was used (only the target template set was used) (case 2). , it can be seen that the accuracy is improved. FIG. 8 is a block diagram of an apparatus for detecting an object according to an exemplary embodiment.

The object detection apparatus 800 may be implemented as an electronic device including a memory and a processor. The object detection apparatus 800 includes a target object processing module 850 that detects an object from a frame image of a video comprising a plurality of frame images. ) may be included.

The object detection apparatus 800 detects the object from the frame image based on a target template set, and outputs information regarding the detected object. can

The object detection apparatus 800 determines an integrated target feature by integrating an image feature of an image region corresponding to each of the target templates included in the target template set, and the target fusion Based on the feature, one or more target candidate areas determined to include the object in the frame image may be acquired. The object detection apparatus 800 may determine one target area from the one or more target candidate areas.

The object detection apparatus 800 may determine a plurality of search areas within a frame image, and extract image features of each of the plurality of search areas to obtain a search area feature. The object detection apparatus 800 calculates a correlation between a feature of each of the search areas of the plurality of search areas and the target fusion feature, and based on the correlation, the at least one target among the plurality of search areas. Candidate regions can be determined.

The object detection apparatus 800 may calculate a similarity between the target fusion feature of the target template set and the target region. When the similarity is less than a threshold value, the object detection apparatus 800 may update the target template set by adding the target region as a target template to the target template.

The object detection apparatus 800 may calculate similarity of each of all target templates of the target template set and the detected target area. The object detection apparatus 800 may update the target template set by adding the target region as a target template to the target template when all of the similarities are less than a threshold value.

The object detection apparatus 800 may acquire one or more target candidate areas determined to include the object in the frame image, based on the target template set and the interference template set. The object detection apparatus 800 may determine one target area from the one or more target candidate areas.

The object detection apparatus 800 may calculate, for each of the one or more target candidate areas, a matching degree between each of the target candidate areas and each target template of the target template set. The object detection apparatus 800 may calculate, for each of the one or more target candidate regions, a matching degree between each of the target candidate regions and each interference template of the interference template set. The object detection apparatus 800 is configured to: based on a matching degree between each of the target candidate areas and each target template of the target template set and a matching degree between each of the target candidate areas and each interference template of the interference template set , it is possible to calculate each target matching degree of the target candidate region. The object detection apparatus 800 may determine one target area among the one or more target candidate areas based on the respective target matching degrees of the target candidate areas. Here, when calculating the target matching degree of each of the target candidate areas, the object detection apparatus 800 includes an average or median value of matching degrees between each of the target candidate areas and each target template of the target template set, and / Alternatively, each target matching degree of the target candidate area may be calculated based on an average value or a median value of matching degrees between each of the target candidate areas and each interference template of the interference template set.

The object detection apparatus 800 may determine an integrated target feature based on the target template set, and may determine an integrated inteference feature based on the interference template set. The object detection apparatus 800 may obtain the one or more target candidate regions from the frame image based on the target fusion characteristic and the interference fusion characteristic. For example, the object detection apparatus 800 calculates a similarity of some or all of the interference fusion characteristics of the interference template set and the other target candidate regions, and the similarity among some or all of the other target candidate regions is a threshold value. A smaller target candidate area may be added to the interference template as an interference template.

9 is a block diagram of an electronic device for object detection according to an embodiment.

The electronic device 900 according to an embodiment may include a processor 910 , a network interface 940 , and a storage 950 . Also, the electronic device 910 may further include a user interface 930 and at least one communication bus 920 . The communication bus 920 is used to realize the connection and communication between these components. The user interface 930 may include a display and a keyboard, and optionally, may further include a standard wired/wireless interface. The network interface 940 may include a standard wired interface and/or a wireless interface (eg, a Wi-Fi interface). The storage device 950 may include a high-speed RAM memory, and may include at least one disk memory or a non-volatile memory such as a flash memory. The storage device 950 may also be a removable storage device that can be detached from the electronic device 900 . The storage device 950 may store an operating system, a network communication module, a user interface module, and a device control application program. The network interface 940 may provide a network communication function. The user interface 930 is used to provide an input interface for a user. The processor 910 may be used to invoke a device control application stored in the storage 950 .

In some implementations, the electronic device 900 may execute the method described with reference to FIGS. 1 to 7 through each built-in function module. For specific details, refer to the implementation method provided previously, which will not be repeated further.

According to an embodiment, at least one module among a plurality of modules may be implemented through the AI model. Functions related to AI may be performed by non-volatile memory, volatile memory, and processors. A processor may include one or more processors. At this time, the one or more processors are general-purpose processors such as central processing units (CPUs), application processors (APs), or pure graphics processing units such as graphics processing units (GPUs), visual processing units (VPUs), and/or neural processing units ( It may be an AI-only processor such as an NPU).

The one or more processors control processing of input data according to predefined operating rules or artificial intelligence (AI) models stored in non-volatile and volatile memory. The one or more processors provide predefined operating rules or artificial intelligence models through training or learning. Here, providing through training means obtaining an AI model having predefined operation rules or desired characteristics by applying a learning algorithm to a plurality of training data. Learning may be performed in the device itself on which AI according to the embodiment is performed, and/or may be implemented by a separate server/system.

An AI model may include multiple neural network layers. Each layer has a plurality of weight values, and calculation of one layer is performed using a calculation result of a previous layer and a plurality of weights of the current layer. Examples of neural networks include convolutional neural networks (CNN), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep trust neural networks (DBNs), bidirectional recurrent neural networks (BRDNNs), generative antagonistic networks ( GAN) and deep Q networks.

The learning algorithm is a method of enabling, allowing, or controlling a predetermined target device (eg, a robot) to be determined or predicted by training a predetermined target device (eg, a robot) using a plurality of learning data. Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

The AI model may use the image data as input data of the AI model to obtain a detection result of a target object in an arbitrary one-frame image of a video. The AI model can be obtained through training. Here, "obtain by training" means training a basic AI model with a large number of training data through a training algorithm to obtain a predefined operation rule or AI model configured to perform a desired feature (or purpose). means that An AI model may include multiple neural network layers. Each of the plurality of neural network layers includes a plurality of weight values, and the neural network calculation is performed by calculation between the calculation result of the previous layer and the plurality of weight values.

The embodiments described above may be applied to visual understanding. Visual understanding is a technology for recognizing and processing like human vision, for example, object recognition, object tracking, image search, human recognition, scene recognition, 3D reconstruction/positioning, or Includes image enhancement.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and a software application running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those of ordinary skill in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (18)

A method of detecting an object from a frame image of a video comprising a plurality of frame images, the method comprising:
detecting the object from the frame image based on a set of target templates; and
outputting information about the detected object
including,
The target template set comprises:
One or more target templates
including,
Each of the one or more target templates,
information of the object in each frame image determined to include the object among previous frame images of the frame image of the video
How to include.
According to claim 1,
The target template set comprises:
initial target template
further comprising,
The initial target template is
information of the object in a frame image determined by the user to include the object among the frame images of the video; or
A separate image that is independent of the video and contains the object
How to include.
According to claim 1,
The step of detecting the object from the frame image,
determining a target fusion characteristic by fusing image characteristics of an image region corresponding to each of the target templates included in the target template set;
obtaining one or more target candidate regions determined to include the object in the frame image based on the target fusion characteristic; and
determining one target area from the one or more target candidate areas;
How to include.
4. The method of claim 3,
Obtaining the one or more target candidate regions includes:
determining a plurality of search regions within the frame image;
obtaining a search area feature by extracting image features of each of the plurality of search areas;
calculating a correlation between a search area feature of each of the plurality of search areas and the target fusion feature; and
determining the at least one target candidate area among the plurality of search areas based on the correlation
How to include.
4. The method of claim 3,
updating the target template set
further comprising,
Updating the target template set includes:
calculating a similarity between the target fusion feature and the target region of the target template set; and
adding the target region as a target template to the target template when the similarity is less than a threshold
How to include.
4. The method of claim 3,
The step of detecting the object from the frame image,
obtaining one or more target candidate regions determined to include the object in the frame image; and
determining one target area from the one or more target candidate areas;
including,
The method is
updating the target template set when the target area satisfies a predetermined condition;
How to include more.
7. The method of claim 6,
Updating the target template set includes:
calculating a similarity of each of all target templates of the target template set and the target region; and
adding the target region as a target template to the target template when all of the similarities are less than a threshold value;
How to include.
According to claim 1,
The step of detecting the object from the frame image,
detecting the object from the frame image based on the target template set and the interference template set;
including,
The interference template set,
One or more clash templates
including,
Each of the one or more interference templates,
Information about an interfering object that interfered with the detection of the object among previous frame images of the frame image of the video
How to include.
9. The method of claim 8,
The step of detecting the object from the frame image,
obtaining one or more target candidate regions determined to include the object in the frame image based on the target template set and the interference template set; and
determining one target area from the one or more target candidate areas;
How to include.
10. The method of claim 9,
Determining one target region from the one or more target candidate regions includes:
calculating, for each of the one or more target candidate regions, a matching degree between each of the target candidate regions and each target template of the set of target templates;
calculating, for each of the one or more target candidate regions, a matching degree between each of the target candidate regions and each interference template in the set of interference templates;
Based on the matching degree of each of the target candidate areas with each target template of the target template set and the matching degree of each of the target candidate areas with each interference template of the interference template set, each of the target candidate areas calculating a target matching degree; and
determining one target area among the one or more target candidate areas based on respective target matching degrees of the target candidate areas;
How to include.
11. The method of claim 10,
Calculating each target matching degree of the target candidate region comprises:
an average or median value of matching degrees between each of the target candidate regions and each target template of the target template set, or
The average or median value of matching degrees between each of the target candidate regions and each interference template of the interference template set
calculating each target matching degree of the target candidate area based on
How to include.
10. The method of claim 9,
Obtaining the one or more target candidate regions includes:
determining a target fusion characteristic based on the set of target templates;
determining an interference fusion characteristic based on the interference template set; and
obtaining the one or more target candidate regions from the frame image based on the target fusion characteristic and the interference fusion characteristic;
How to include.
13. The method of claim 12,
Determining a target fusion characteristic based on the target template set comprises:
determining the target fusion characteristic by fusing image characteristics of an image region corresponding to each of the target templates included in the target template set;
including,
Determining an interference fusion characteristic based on the interference template set comprises:
determining the interference fusion characteristic by fusing image characteristics of an image region corresponding to each of the interference templates included in the interference template set;
How to include.
13. The method of claim 12,
Determining a target fusion characteristic based on the target template set comprises:
determining the target fusion characteristic based on all target templates included in the target template set;
including,
Determining an interference fusion characteristic based on the interference template set comprises:
Determining the interference fusion characteristic based on all interference templates included in the interference template set
How to include.
13. The method of claim 12,
updating the interference template set
further comprising,
Updating the interference template set comprises:
adding some or all of the target candidate regions other than the target region among the one or more target candidate regions as an interference template to the interference template;
How to include.
13. The method of claim 12,
updating the interference template set
further comprising,
Updating the interference template set comprises:
calculating a similarity of some or all of the interference fusion feature and other target candidate regions of the interference template set; and
adding a target candidate region having a similarity smaller than a threshold value among some or all of the other target candidate regions as an interference template to the interference template;
How to include.
In an electronic device,
including memory and a processor;
The memory stores a computer program,
The processor is configured to implement the method of claim 1 when the computer program is executed.
electronic device.
In the non-transitory computer-readable recording medium,
storing a computer program, the computer program being executed by a processor to implement the method of claim 1 ,
A non-transitory computer-readable recording medium.

Images