KR102174656B1 - Apparatus and method for recognizing activity and detecting activity area in video - Google Patents

Abstract

In the present invention, only the action label is learned using the annotated learning video, thereby reducing the time and cost burden for acquiring the learning video, and performing action localization by accurately extracting the action region by recognizing the action of the object included in the video. It is possible to provide an apparatus and method for recognizing a possible video action and detecting an action area.

Description

Video action recognition and action area detection device and method {APPARATUS AND METHOD FOR RECOGNIZING ACTIVITY AND DETECTING ACTIVITY AREA IN VIDEO}

The present invention relates to an apparatus and method for recognizing a video motion and detecting a motion section, and to an apparatus and method for recognizing a motion of an object and extracting a motion region from a video.

Recognizing an action of an object included in a video and extracting an action area is essential in various video usage fields such as video surveillance, video summary, and video caption. Various technologies for detecting objects in video have been disclosed, and the technique for recognizing the action of an object has also made great progress. However, accurately extracting the location of an action is due to various reasons including the diversity of actions and complex background. There have been limitations.

Accordingly, various studies have recently been conducted to perform action localization that extracts an action region of an object from a video using an artificial neural network learned by a deep learning technique. By using the deep learning technique, the performance of the action localization task for video was greatly improved.

In the existing deep learning technique, artificial neural networks are trained in a fully supervised learning method. Therefore, it is required that the ground truth label for the action boundary of the object in the training video be fully annotated during learning.

However, manually annotating the boundaries for each action in a video is very inefficient in terms of time and cost. In addition, there is a problem that the boundary of each action can be subjectively determined according to an operator, and thus an artificial neural network can be learned incorrectly.

Korean Patent Registration No. 10-1900237 (Registered on September 13, 2018)

It is to provide an apparatus and method for recognizing a video action and detecting an action region that can be learned based on a weakly-supervised learning method using only a simple action label that is easy to obtain annotated learning video.

Another object of the present invention is to provide a video action recognition and action region detection apparatus and method capable of performing action localization on a video through learning through weak guidance learning.

Another object of the present invention is to provide a video action recognition and action region detection apparatus and method capable of accurately extracting an action region of an object from a video.

The video action recognition and action region detection apparatus according to an embodiment of the present invention for achieving the above object searches for a bounding box that is an area containing a predetermined object in each of a plurality of frames of a video according to a previously learned pattern estimation method. And an object tubelet acquisition unit that connects a corresponding bounding box in a plurality of frames to generate an object tubelet; A tublet adjuster configured to obtain a tube by adjusting the size of a plurality of bounding boxes of the object tube by pre-learning a pattern estimation method in a weak supervised learning method using an action learning video with an action label annotated; A feature map acquisition unit configured to generate a feature map by extracting features of the tubelet image by time-averaging pooling of the plurality of optimal bounding boxes of the tubelet and transforming it into a tubelet image, and extracting features of the tubelet image according to a previously learned pattern estimation method; An action weight acquisition unit for acquiring an action weight from the feature map and weighting a corresponding feature map to obtain a weighted feature map; And an action class score indicating a level corresponding to each of the plurality of action classes for which the weighted feature map is determined, and selecting an action corresponding to the tublet according to the action class score, and an optimal bounding box included in the tublet. An action recognition and area determination unit that outputs location information of the user; Includes.

The action recognition and region determination unit is configured to include an artificial neural network and calculates the action class score (λ ⁿ (c)) for the n-th tubelet (P _n ) among N (N is a natural number) tubelets included in the video. Equation

(Where α ⁿ is the action weight of, y ⁿ is the feature map, and w _T (c, d) is used in the action class classifier to identify the specified action class (c ∈ {1, ..., C}). As the corresponding d-th element, it can be obtained according to the weight of the computation layer of the artificial neural network).

The action recognition and region determination unit selects an action class score that is equal to or greater than a predetermined reference action class score among the action class scores, outputs an action class corresponding to the selected action class score as an action of the object, and corresponds to the selected action class score. Position information of the optimal bounding box of the tube can be output.

The action recognition and area determination unit outputs one action class with the highest action class score according to a predetermined setting, or outputs one action class score greater than or equal to the reference action class score when there are a plurality of action class scores equal to or greater than the reference action class score for the same tubelet. You can print multiple action classes that appear together.

The video action recognition and action region detection apparatus learns a ring map based on an action learning video in which only an action label is annotated, the tubelet adjustment unit, the feature map acquisition unit, the action weight acquisition unit, and the action recognition and region determination unit. For learning department; The learning unit further includes, in response to the action learning video, the action weight acquisition unit 150 adds a weighted feature map output for all action tublets (P _n ) to obtain a video feature map, and The video action class score is obtained from the feature map, and the difference between the video action class score and the action label of the action learning video is acquired as an action loss and back propagated to perform weak guidance learning.

The object tublet acquisition unit searches for a bounding box, which is an area containing a predetermined object in each of a plurality of frames of a video, according to a previously learned pattern estimation method, and generates an object score indicating the probability that an object to be detected exists in each bounding box. Acquired together, and using a bounding box whose object score is equal to or greater than a predetermined reference object score, the learning unit generates an object tutorial, and the learning unit obtains the object tubelet acquisition unit by applying an object learning video to which only the object label is annotated. By acquiring the difference between the obtained object score and the object label annotated in the object learning video as an object loss and backpropagating, the object tubelet acquisition unit may learn the weak map.

A video action recognition and action area detection apparatus and method according to another embodiment of the present invention for achieving the above object includes a bounding box that is an area including an object specified in each of a plurality of frames of a video according to a previously learned pattern estimation method. Searching, and generating an object tubelet by connecting corresponding bounding boxes in a plurality of frames;

Obtaining a tube by adjusting sizes of a plurality of bounding boxes of the object tube according to a pattern estimation method learned by a weak supervised learning method using an action learning video in which an action label is annotated; Transforming a plurality of optimal bounding boxes of the tubelet into a tubelet image by time-averaging pooling, and extracting features of the tubelet image according to a previously learned pattern estimation method to generate a feature map; Acquiring an action weight from the feature map and weighting it to a corresponding feature map to obtain a weighted feature map; And an action class score indicating a level corresponding to each of the plurality of action classes for which the weighted feature map is determined, and selecting an action corresponding to the tublet according to the action class score, and an optimal bounding box included in the tublet. Outputting the location information of; Includes.

Accordingly, the apparatus and method for recognizing a video action and detecting an action region according to an embodiment of the present invention can reduce a time and cost burden for acquiring a training video by learning using a training video in which only an action label is annotated. In addition, it is possible to recognize an action of an object included in a video and accurately extract an action area to perform action localization.

1 illustrates a schematic structure of an apparatus for recognizing a video action and detecting an action region according to an embodiment of the present invention.
FIG. 2 shows a detailed configuration of a feature map acquisition unit of FIG. 1.
3 shows an example of an action learning video for weak guidance learning.
FIG. 4 shows an example of a bounding box whose size is adjusted by the tubelet controller of FIG. 1.
5 and 6 illustrate an example of a result of performing action localization in the apparatus for recognizing a video action and detecting an action region according to the present embodiment.
7 shows a method of recognizing a video action and detecting an action region according to an embodiment of the present outbreak.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the implementation of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings. However, the present invention may be implemented in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless specifically stated to the contrary. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean units that process at least one function or operation, which is hardware, software, or hardware. And software.

1 shows a schematic structure of an apparatus for recognizing a video action and detecting an action region according to an embodiment of the present invention, and FIG. 2 shows a detailed configuration of an object tubelet acquisition unit of FIG. 1.

Referring to FIG. 1, the apparatus for recognizing a video action and detecting an action region according to the present embodiment includes an object tubelet acquisition unit 110, a tubelet adjustment unit 120, a dimensional conversion unit 130, and a feature extraction unit 140. , An action weight acquisition unit 150, an action recognition and area determination unit 160, and a learning unit 170.

The object tubelet acquisition unit 110 acquires a video for which action localization is to be performed, and generates an object tubelet by detecting a region including an object in a plurality of frames of the acquired video. The object tubelet acquisition unit 110 includes an artificial neural network in which a pattern estimation method is learned in advance according to a weakly-supervised method, and detects a predetermined object in a plurality of frames of a video, and An object tublet is created by connecting the area in which the object is detected in the frame.

Referring to FIG. 2, the object tubelet acquisition unit 110 may include a video providing unit 111, a frame grouping unit 112, an object detection unit 113, and a tubelet generation unit 114.

The video provider 111 acquires a video composed of a plurality of frames for which action localization is to be performed. Here, action localization is to classify the action area of at least one object included in the video, and means recognizing the action of the object, and extracting the area including the action of the recognized object within each frame.

The frame grouping unit 112 includes frames (f _t , t = {1, ..., T}) in units of a predetermined number (here, T (T is a natural number)) in a video including a plurality of consecutive frames. ) Are grouped and classified. In general, video contains a large number of frames, and objects rarely remain the same in all frames included in the video. In addition, there are cases in which all frames of a video cannot be obtained at once, such as streaming. Accordingly, the frame grouping unit 112 groups the videos in units of a predetermined number of frames so that objects included in the video can be easily detected. Here, the number of frames to be grouped may be variously set, but may be grouped in units of eight frames as an example.

The object detection unit 113 detects objects included in a plurality of frames grouped according to the learned pattern estimation method, including an artificial neural network in which the pattern estimation method is learned in advance, and an object region in which the detected object appears in the plurality of frames. Detect the bounding box for (B = {B ₁ , ..., B _T }).

Here, the object detection unit 113 may be learned in advance by designating an object to be detected. In the present embodiment, it is assumed that the video action recognition and action region detection apparatus recognizes a person's action and detects the action region. Accordingly, the object detection unit 113 detects a region including a person in a plurality of grouped frames.

The object detection unit 113 may detect an object area in which an object appears in a plurality of grouped frames f _t as a rectangular bounding box B, and output the coordinates of the detected bounding box B. have.

Here, the object detection unit 113 may be implemented as convolutional neural networks, for example, and may be learned according to a weak supervised learning method for convenience of learning.

When the object detection unit 113 is trained in a fully supervised learning method, the object detection performance is excellent, but it requires a large amount of training videos in which the verification data label for the boundary of the object, that is, the bounding box B, is completely annotated. . In addition, since the training video annotated with the verification data label is basically acquired by hand, it is not easy to acquire the training video.

In contrast, in the present embodiment, the object detection unit 113 learns a ring map based on an object learning video in which only simple object labels are annotated.

An object learning video in which only a simple object label is annotated refers to a video in which only an object label is provided for the entire video without additional annotation on the object area. For example, in the object learning video in the present embodiment, only the label of an object such as a person, dog, cat, goat, etc. is provided as an annotation, and a separate annotation is not provided for an object area in which the object appears.

In the weak instructional learning using a video provided only with a simple object label, since it is not necessary to manually annotate the boundary of an object, a large amount of video for object learning can be produced quickly and easily at low cost.

Wherein not less than the object detection section 113 is the game reference object acquired for each boundary box (B) is an object score (h ^*) represents the probability of an object to be detected together, and the obtained object score (h ^*) is Basis Only normal bounding box (B) can be determined and output. This is to improve the reliability of object detection by the object detection unit 113.

)을 추출하고, 추출된 객체 특징(x_n)로부터 수학식 1에 따라 프레임별 객체 스코어(hⁿ)를 획득할 수 있다.The object detection unit 113 detects the object feature (x _n ∈) of the bounding box B _n detected in the n-th frame f _n among a plurality of frames f _t according to a previously learned pattern estimation method.

), and an object score h ^{n for each} frame according to Equation 1 from the extracted object feature x _n .

(Wh _h (d) is a d-th element corresponding to an object classifier (w _h ) for identifying a designated object, and represents the weight of the computational layer (eg, convolutional layer) of the artificial neural network.)

The object detection unit 113 applies an average pooling and a sigmoid function to each frame object score h ⁿ obtained for each of a plurality of frames f _t , Accordingly, it is possible to obtain an object score (h ^* ) that is a probability that an object exists in the bounding box (B) of a plurality of frames (f _t ).

Meanwhile, when a plurality of objects are included in a plurality of frames, the object detection unit 113 may detect regions according to various combinations of the plurality of objects. For example, a plurality of objects appear in a video, and the plurality of objects may appear spaced apart from each other or may appear adjacent to each other or overlapped in a partial area. Accordingly, the object detection unit 113 detects objects separated from each other as separate object areas, and objects with adjacent or overlapping areas are detected as separate object areas for each object, as well as adjacent or overlapped objects. The included object area can also be detected. Here, the object detection unit 113 may be configured to additionally detect an object area including an object when at least a part of the object area for each object overlaps due to an adjacent or overlapping arrangement of objects.

The object detection unit 113 detects a bounding box (B = {B ₁ , ..., B _T }) corresponding to each of the _T frames (f ₁ , ..., f _T ), and When a plurality of object regions are detected in a frame (f _t ), the number of boundary regions (B _t ⁿ , where {n = 1, ..., N}) corresponding to each object region (N) can be detected. have.

When an object region is detected by the object detection unit 113, the tubelet generator 114 generates an object tublet by connecting the object regions detected for the same object in a plurality of frames. That is, the tubelet may be obtained as a set of bounding boxes B for a region including the same object in a plurality of grouped frames.

The tubelet generator 114 is a bounding box (B _t ^m , B _t-1 ⁿ ) (where m, n ∈ (1, ..., N) in two consecutive frames (f _t-1 , f _t ). }) is obtained, a link score (E _link ) between the bounding box (B _t ^m ) and the bounding box (B _t-1 ⁿ ) is obtained according to Equation 3.

(Where h(B _t ⁿ ) is an object score representing the probability that the object specified in the bounding box will be included, and E _feat (B _t ⁿ , B _t-1 ⁿ ) is the bounding box (B _t ⁿ ) by the L ₂ -norm function. ) And the normalized features of the bounding box (B _t-1 ⁿ ), and E _IoU (B _t ^m , B _t-1 ⁿ ) is the bounding box (B _t ^m ) and the bounding box (B _t-1 ⁿ ) ⁿ ) represents the result of measuring Union of IoU (Intersection of Union) as an overlap score between, and β ₁ and β ₂ are parameters that control feature similarity and weight for overlapping scores, respectively.)

The link score E _link has similar characteristics of an object in two consecutive frames (f _t-1 and f _t ), and has a larger value as the area where the object appears overlaps, so that it is strongly connected.

And in order to generate a tublet for the n-th object in the frame (f _t ), by configuring a connection path having an index (π _t (n)) according to the path equation 4, the object tublet represented by equation 5 Produces (O ⁿ ).

(Here, l ∈ {1, ..., N} and t ∈ {2, ..., T}.)

However, since various methods of acquiring an object tube have been previously disclosed, in some cases, the object tube acquiring unit 110 may be learned in advance in a conventional manner to generate an object tube (O ⁿ ).

The tubelet controller 120 also receives the object tubelet O ⁿ obtained from the object tubelet acquisition unit 110 including an artificial neural network in which the pattern estimation method is learned in advance, and receives the applied object tubelet O ⁿ ) Adjust the size of the bounding boxes (B) in each. That is, the size of each of the object tubelets O ⁿ is adjusted to obtain a tubelet P _n .

As described above, when the object tubelet acquisition unit 110 is weakly supervised learning, it is possible to very easily acquire a training video, but the detection performance of the bounding box B may be lower than that of the fully supervised learning method. That is, the bounding box B does not exactly correspond to the object region in which the object appears, and may be extracted including an unnecessary region. Accordingly, the tublet control unit 120 removes the unnecessary area from the bounding box B of the object tublet so that the bounding box B accurately designates only the object area.

The tubelet adjustment unit 120 reduces the offset for the width (u _t ⁿ ) and the height (v _t ⁿ ) based on the center position of the n-th bounding box (B _t ⁿ ) of the t-th frame (f _t ). Adjust the size of the bounding box (B).

)를 획득한다.In more detail, the tublet controller 120 adjusts the width (∇u _t ⁿ ) and the height (v _t ⁿ ) for the width (u _t ⁿ ) of the bounding box (B _t ⁿ ) according to the previously learned pattern estimation method. to obtain a controlled height (∇v _t ^n), and an optimum bounding box size is adjusted as shown in equation 6, according to the obtained control width (∇u _t ⁿ⁾ and adjusting the height (∇v _t ⁿ⁾ (

).

The tubular adjustment unit 120 includes a convolutional neural network composed of a plurality of convolutional layers and at least one activation function layer (here, for example, ReLU), and the adjustment width (∇u _t ⁿ ) and adjustment The height (∇v _t ⁿ ) can be obtained.

3 shows an example of an action learning video for weak supervised learning, and FIG. 4 shows an example of a bounding box whose size is adjusted in the tube control unit of FIG. 1.

After the object tublet acquisition unit 110 has learned about the weak map by the object learning video, the tube control unit 120 includes the object tube obtained from the video for the action learning With the authorization, the ring map can be additionally learned. Here, the action learning video is simply a video to which an action label is annotated, and as an example, as shown in FIG. 3, a video including a single action to which an action label such as diving, golf, ice dancing, or fencing is annotated.

)를 나타내고 있다. 도 4에 도시된 바와 같이, 경계 박스(B_t ⁿ)는 객체가 나타나는 영역에 대해 상대적으로 큰 영역으로 검출되어 여백이 포함되는 반면, 최적 경계 박스(

)는 객체의 영역에 매우 타이트하게 설정되었음을 알 수 있다.In FIG. 4, a bounding box (B _t ⁿ ) detected by the object tubelet acquisition unit 110 in a plurality of consecutive frames and an optimal bounding box adjusted by the tubelet controller 120 (

). As shown in FIG. 4, the bounding box (B _t ⁿ ) is detected as a relatively large area with respect to the area in which the object appears, and a margin is included, whereas the optimal bounding box (

You can see that) is set very tightly in the area of the object.

)들에 대해 시간축을 기준으로 시간 평균 풀링(time average pooling)을 수행하여, 다수의 최적 경계 박스(

)를 포함하는 3차원의 튜블릿(P_n) 각각을 2차원의 튜블릿 이미지로 변환한다. The dimensional transform unit 130 includes a plurality of optimal bounding boxes of each of the tubelets P _n

) On the basis of the time axis by performing time average pooling,

Each of the three-dimensional tublets (P _n ) including) is converted into a two-dimensional tublet image.

)을 획득한다.The feature extraction unit 140 receives the tubelet image, extracts the features of the tubelet image according to a previously learned pattern estimation method, and extracts a feature map (y ⁿ ∈

).

Action characterized by the weight obtaining unit 150 obtains the action weight (α ⁿ⁾ in the characteristic map obtained from the feature extraction unit 140 according to the pre-training pattern estimation method, corresponding to the obtained action weight map (y ⁿ ) To obtain a weighted feature map (α ⁿ y ⁿ ).

Here, the action weight α ⁿ is a weight representing the action level, that is, motion of the object in the tubular image. The action weight acquisition unit 150 acquires the action weight α ^{n and} weights it to the feature map y ⁿ , although the object tubelet acquisition unit 110 detects an object and generates the object tube O ⁿ . This is because even if acquired, if there is no movement in the object of the object tubelet O ⁿ , it is meaningless in action localization that detects the action area of the object.

The dimensional conversion unit 130 and the feature extraction unit 140 may be integrated into a feature map acquisition unit.

The action recognition and area determination unit 160 receives the weighted feature map α ⁿ y ⁿ and classifies it into at least one action class from among a plurality of predefined action classes according to a previously learned pattern estimation method. The action recognition and region determination unit 160 may also be implemented as an artificial neural network.

The action recognition and region determination unit 160 first includes an action class score (λ ⁿ⁾ indicating a level corresponding to each of a plurality of predefined action classes in which a weighted feature map (α ⁿ y ⁿ ) corresponding to each of the tubelets (P _n ) is determined. (c) = {λ ⁿ (1), ..., λ ⁿ (C)}) is obtained according to Equation 7.

)에 대응하는 d번째 요소로서 인공 신경망의 연산 레이어(예를 들면 컨볼루션 레이어)의 가중치를 나타낸다.)(Where w _T (c, d) is the action class classifier (w _T ∈) to identify the specified action class (c ∈ {1, ..., C}).

), which represents the weight of the computational layer (for example, the convolutional layer) of the artificial neural network.)

는 n번째 튜블릿(P_n)의 클래스(c)에 대한 연관성을 나타내는 분류 스코어로서 수학식 8와 같이 표현될 수 있다.In Equation 7

Is a classification score indicating the association with the class (c) of the n-th tubelet (P _n ), and may be expressed as in Equation 8.

이다.)(Where s ⁿ = [s ⁿ (1), ..., s ⁿ (C)] ^T ∈

to be.)

Equation 7 can be expressed as Equation 9 by Equation 8.

That is, the action class score (λ ⁿ (c)) for the class (c) of the n-th tubelet (P _n ) is obtained as an action weight (α ⁿ ) and a classification score (S ⁿ ), as shown in Equation 9.

)와 액션 클래스(c)를 획득하여 액션 로컬라이제이션의 결과로 출력한다. 즉 액션의 종류와 함께 비디오에서 액션이 나타난 객체 영역을 출력한다.When the action class score (λ ⁿ (c)) for a plurality of action classes (c) of each of the tubelets (P _n ) is obtained, the action recognition and region determination unit 160 is an action class equal to or greater than a predetermined reference action class score. A score (λ ⁿ (c)) is selected, and a tubelet (P _n ) corresponding to the selected action class score (λ ⁿ (c)) is extracted as an action tubelet. And the optimal bounding box of the extracted action tube (

) And action class (c) are obtained and output as the result of action localization. That is, the object area where the action appears in the video is output along with the type of action.

In this case, one tubelet P _n may have an action class score λ ⁿ (c) greater than or equal to the reference action class score for a plurality of action classes c. That is, there may be a case where one tubelet P _n corresponds to a plurality of action classes. In this case, the action recognition and region determination unit 160 outputs one action class (c) having the highest action class score (λ ⁿ (c)) according to a predetermined setting, or a plurality of actions that appear above the reference action class score. All of the action classes (c) of can be displayed.

The learning unit 170 is a configuration for learning a weak map of the action recognition and action region detection apparatus, and is added only when learning is performed, and may be omitted after learning.

The learning unit 170 first learns the object tubelet acquisition unit 110 using the object learning video, and then, uses the object tubelet acquisition unit 110 and the action learning video, which has been learned on the weak map. The adjusting unit 120, the dimensional transforming unit 130, the feature extracting unit 140, the action weight obtaining unit 150, and the action recognition and region determining unit 160 may be taught a ring map.

The learning unit 170 receives an object score (h ^* ) indicating a probability that an object to be detected exists in the bounding box B obtained by applying the object learning video to the object tubelet acquisition unit 110. In addition, the difference between the object score (h ^* ) and the object label annotated in the object learning video is acquired as an object loss, and back propagated to the object tubelet acquisition unit 110 to learn the object tubelet acquisition unit 110 ring map. . In this case, the learning unit 170 may obtain the object loss by applying it to a known function such as a standard multi-label cross entropy loss as an example.

When the object tubelet acquisition unit 110 learns weak guidance, the learning unit 170 applies the action learning video to the object tubelet acquisition unit 110, and the action weight acquisition unit 150 applies all the action tubelets (P). _n) all of the weighting characteristic map (α ⁿ y ⁿ⁾ that is output as shown in equation 10 for the addition to obtain a video feature map (y ^*) representing a feature for the action in the tube wavelet video levels.

In addition, the learning unit 170 obtains a video action class score λ(c) from the video feature map (y ^* ) according to Equation 11, similar to Equation 7.

When the video action class score λ(c) is obtained, the learning unit 170 acquires the difference between the video action class score λ(c) and the action label of the action learning video as an action loss and backpropagates it, The tubelet adjustment unit 120, the dimensional conversion unit 130, the feature extraction unit 140, the action weight acquisition unit 150, and the action recognition and region determination unit 160 may be taught a ring map. Here, the learning unit 170 may obtain the action loss by applying it to a known function such as a standard multi-label cross entropy loss, for example.

5 and 6 illustrate an example of a result of performing action localization in the apparatus for recognizing a video action and detecting an action region according to the present embodiment.

In Figure 5 (a) and (b) show the results of performing action localization for basketball and ice dancing, respectively, and in Figure 6 (a) to (d) are action localization for diving, soccer, basketball and cycle, respectively. Shows the result of performing. In FIGS. 5 and 6, in order to compare the performance of the apparatus for recognizing a video action and detecting an action region according to the present embodiment, a ground truth label previously performed manually or the like is displayed together.

As shown in Figs. 5 and 6, it can be seen that the video action recognition and action region detection apparatus according to the present embodiment can accurately extract the region in which the action of the object occurs even though the learning is performed by the weak guidance learning method. have.

7 shows a method of recognizing a video action and detecting an action region according to an embodiment of the present outbreak.

Referring to FIGS. 1 to 6, the video action recognition and action region detection method of FIG. 7 will be described. First, the learning unit 170 uses an object learning video to which an object label is annotated, and the object tube acquisition unit 110 The weak map is learned (S12). The learning unit 170 obtains the difference between the object score (h ^* ) and the object label output in response to the object learning video by the object tubelet acquisition unit 110 as an object loss, and sends the object tubelet acquisition unit 110 to By backpropagating, the object tubelet acquisition unit 110 can be trained.

Thereafter, the learning unit 170 uses the learned object tublet acquisition unit 110 and the action learning video in which the action label of the object is annotated, and the tube control unit 120, the dimensional conversion unit 130, and the feature extraction unit ( 140), the action weight acquisition unit 150 and the action recognition and area determination unit 160 are learned about a weak map (S12).

The learning unit 170 obtains a video feature map (y ^* ) according to Equation 10 from the weighted feature map (α ⁿ y ⁿ ) output for all action tublets (P _n ) from the action weight acquisition unit 150 Then, a video action class score (λ(c)) is obtained from the video feature map (y ^* ). In addition, the difference between the obtained video action class score λ(c) and the action label of the video for action learning may be acquired as an action loss and backpropagated to perform weak guidance learning.

After the learning is performed, the video action recognition and action region detection device receives the video for which action localization is to be performed, and the object tubelet acquisition unit 110 whose pattern estimation method is weakly learned includes an object specified in the video. The object tubelet O ⁿ is obtained by detecting the bounding box B, which is the selected area (S21). In this case, the number of object tubelets O ⁿ obtained may vary according to the number of objects included in the video.

)를 갖는 튜블릿(P_n)을 획득한다.The tubelet control unit 120 determines the object tube (O ⁿ ) as shown in Equation 6 according to a weak map-learned pattern estimation method for each bounding box (B _t ⁿ ) of the obtained object tube (O ⁿ ). By adjusting the size of the bounding box (B), the optimal bounding box (

A tubelet (P _n ) with) is obtained.

)들에 대해 시간축을 기준으로 시간 평균 풀링을 수행하여, 튜블릿 이미지로 변환한다 (S23).When the tubelet P _n is obtained, the dimensional transform unit 130 performs a plurality of optimal bounding boxes for each of the tubelets P _n .

) Are subjected to time average pooling based on the time axis, and converted into a tubelet image (S23).

In addition, according to the previously learned pattern estimation method, the feature extraction unit 140 extracts the features of the tubelet image to obtain a feature map (y ⁿ ), and the action weight acquisition unit 150 obtains an action weight (α ⁿ ) from the feature map. ) Is obtained and applied to the corresponding feature map (y ⁿ ) to obtain a weighted feature map (α ⁿ y ⁿ ) (S24).

The action recognition and region determination unit 160 includes an action class score (λ ⁿ ()) indicating a level corresponding to each of a plurality of predetermined action classes in which a weighted feature map (α ⁿ y ⁿ ) corresponding to each of the tubelets (P _n ) is determined. c) = {λ ⁿ (1), ..., λ ⁿ (C)}) is obtained as in Equation 9 (S25).

)와 액션 클래스(c)를 획득하여 출력한다(S26).And the obtained action class score (λ ⁿ (c)) of the exchanger tube for selecting a specified standard action class scores than action class score (λ ⁿ (c)), and corresponding to the selected action class score (λ ⁿ (c)) The bullet (P _n ) is extracted as an action tube. With this, the optimal bounding box of the extracted action tube (

) And the action class (c) are obtained and output (S26).

The method according to the present invention may be implemented as a computer program stored in a medium for execution on a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may also include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and ROM (Read Dedicated memory), RAM (random access memory), CD (compact disk)-ROM, DVD (digital video disk)-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those of ordinary skill in the art will appreciate that various modifications and other equivalent embodiments are possible therefrom.

Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

110: object tublet acquisition unit 120: tublet regression unit
130: dimensional transformation unit 140: feature extraction unit
150: action weight acquisition unit 160: action recognition and area determination unit
170: learning department

Claims (12)

An object tublet that searches for a bounding box that is an area containing a predetermined object in each of a plurality of frames of a video according to a pre-learned first pattern estimation method, and creates an object tube by connecting the corresponding bounding box in a plurality of frames. Acquisition unit;
A tubelet adjusting unit for obtaining a tube by adjusting the size of a plurality of bounding boxes of the object tube by pre-learning a second pattern estimation method in a weak supervised learning method using an action learning video with an action label annotated;
A feature map for generating a feature map by transforming a plurality of bounding boxes of which the size of the tublet is adjusted into a tublet image by time-averaging pooling, and extracting features of the tublet image according to a previously learned third pattern estimation method Acquisition unit;
An action weight acquisition unit for acquiring an action weight from the feature map and weighting a corresponding feature map to obtain a weighted feature map; And
An action class score representing a level corresponding to each of a plurality of action classes in which the weighted feature map is determined is calculated, an action corresponding to a tublet is selected according to the action class score, and the size included in the tublet is An action recognition and area determination unit outputting position information of the adjusted bounding box; Video action recognition and action area detection device comprising a. The method of claim 1, wherein the tubular adjustment unit
Adjusted width (∇u _t ⁿ ) and height (v _t ⁿ ) for each width (u _t ⁿ ) of a plurality of bounding boxes (B _t ⁿ ) of the object tube according to the second pattern estimation method Obtain (∇v _t ⁿ ), and from the obtained adjustment width (∇u _t ⁿ ) and adjustment height (∇v _t ⁿ )

The bounding box whose size is adjusted according to (

A video action recognition and action area detection device to acquire). The method of claim 1, wherein the action recognition and area determination unit
The action class score (λ ⁿ (c)) for the n-th tubelet (P _n ) among N (N is a natural number) tubelets configured including an artificial neural network and included in the video
Equation

(Where α ⁿ is the action weight of, y ⁿ is the feature map, and w _T (c, d) is used in the action class classifier to identify the specified action class (c ∈ {1, ..., C}). As the corresponding d-th element, it represents the weight of the computational layer of the artificial neural network.)
Video action recognition and action area detection device acquired according to. The method of claim 1, wherein the action recognition and area determination unit
Among the action class scores, an action class score equal to or greater than a predetermined reference action class score is selected, the action class corresponding to the selected action class score is output as an action of the object, and the size of the tube corresponding to the selected action class score is adjusted. A video action recognition and action area detection device that outputs location information of a bounding box. The method of claim 4, wherein the action recognition and area determination unit
When there are multiple action class scores that are equal to or greater than the base action class score for the same tubelet, one action class with the highest action class score is output according to a predetermined setting, or multiple action classes that appear above the base action class score are combined. Output video action recognition and action area detection device. The apparatus of claim 1, wherein the video action recognition and action area detection device
A learning unit for learning a weak map based on an action learning video in which only an action label is annotated, the tubelet control unit, the feature map acquisition unit, the action weight acquisition unit, and the action recognition and region determination unit; Including more,
The learning unit
In response to the action learning video, the action weight acquisition unit 150 adds weighted feature maps output for all action tublets (P _n ) to obtain a video feature map, and obtains a video action class score from the video feature map. A video action recognition and action region detection device that acquires and performs weak supervised learning by acquiring the difference between the video action class score and the action label of the action learning video as an action loss and backpropagating it. The method of claim 6, wherein the object tubelet acquisition unit
According to the first pattern estimation method, a bounding box, which is an area containing a predetermined object in each of a plurality of frames of a video, is searched, an object score indicating the probability of the existence of an object to be detected in each bounding box is obtained, and the obtained An object tublet is created using a bounding box having an object score equal to or greater than a predetermined reference object score,
The learning unit
An object learning video in which only an object label is annotated is applied, and the difference between the object score obtained by the object tubelet acquisition unit and the object label annotated in the object learning video is acquired as an object loss and backpropagated, so that the object tube A video action recognition and action region detection device that learns weak guidance of the acquisition unit. Searching for a bounding box, which is an area including an object, in each of a plurality of frames of a video according to a pre-learned first pattern estimation method, and connecting the corresponding bounding boxes in the plurality of frames to generate an object tube;
Obtaining a tube by adjusting the size of a plurality of bounding boxes of the object tube according to a second pattern estimation method learned by a weak supervised learning method using an action learning video with an action label annotated;
Generating a feature map by performing time-averaged pooling of the bounding box of which the size of the tubelet is adjusted to convert it into a tubelet image, and extracting features of the tubelet image according to a previously learned third pattern estimation method;
Acquiring an action weight from the feature map and weighting it to a corresponding feature map to obtain a weighted feature map; And
An action class score representing a level corresponding to each of a plurality of action classes in which the weighted feature map is determined is calculated, an action corresponding to a tublet is selected according to the action class score, and the size included in the tublet is Outputting position information of the adjusted bounding box; Video action recognition and action area detection method comprising a. The method of claim 8, wherein outputting the location information comprises:
The action class score (λ ⁿ (c)) for the n-th tubelet (P _n ) among N (N is a natural number) tubelets configured including an artificial neural network and included in the video
Equation

(Where α ⁿ is the action weight of, y ⁿ is the feature map, and w _T (c, d) is used in the action class classifier to identify the specified action class (c ∈ {1, ..., C}). As the corresponding d-th element, it represents the weight of the computational layer of the artificial neural network.)
Video action recognition and action area detection method obtained according to. The method of claim 8, wherein outputting the location information comprises:
Selecting an action class score that is equal to or greater than a predetermined reference action class score among the action class scores;
Outputting an action class corresponding to the selected action class score as an action of the object; And
Outputting location information of a bounding box in which a size of a tube corresponding to the selected action class score is adjusted; Video action recognition and action area detection method comprising a. The method of claim 8, wherein the video action recognition and action region detection method
Learning a medicine map based on the action learning video in which only the action label is annotated; Including more,
The step of learning the medicine map
Acquiring a video feature map (y ^* ) by adding weighted feature maps (α ⁿ y ⁿ ) output to all acquired action tubelets (P _n ) in response to the action learning video;
Obtaining a video action class score (λ(c)) from the video feature map (y ^* ); And
Acquiring a difference between the video action class score λ(c) and the action label of the video for action learning as an action loss and backpropagating; Video action recognition and action area detection method comprising a. The method of claim 8, wherein generating the object tublet comprises:
Searching for a bounding box that is an area including a predetermined object in each of a plurality of frames of a video according to the first pattern estimation method;
Acquiring an object score indicating a probability that an object to be detected exists in each bounding box together: And
Generating an object tubelet using a bounding box in which the obtained object score is equal to or greater than a predetermined reference object score; Including,
The step of learning the medicine map
An object learning video in which only an object label is annotated is applied, and the difference between the object score obtained by the object tubelet acquisition unit and the object label annotated in the object learning video is acquired as an object loss and backpropagated, so that the object tube Learning an acquiring unit; Video action recognition and action area detection method further comprising.

Images