KR102357000B1 - Action Recognition Method and Apparatus in Untrimmed Videos Based on Artificial Neural Network - Google Patents

Abstract

According to the present invention, a processor receives an analysis target image, selects some frame images from the analysis target image for each preset section based on a time domain, and recognizes a place and an action from the selected frame image. An artificial neural network-based unrefined artificial neural network that finds in which frame and in which spatial region a place and a behavior appear with an artificial neural network that is learned from clip unit place and behavior information from unrefined video by labeling the selected frame image with feature values according to and behavior A method and apparatus for recognizing behavior in a video are disclosed.

Description

{Action Recognition Method and Apparatus in Untrimmed Videos Based on Artificial Neural Network}

The present invention relates to a method and apparatus for recognizing a behavior, and more particularly, to a method and apparatus for recognizing a behavior in an unrefined video based on an artificial neural network.

Recently, deep learning has been widely used to solve various problems in the field of computer vision. However, in order to use a deep learning-based system, a large amount of data required for learning is required. In addition, in order to build a system suitable for use in real situations, it is necessary to learn from data similar to real situations. skill to do However, since the data to be used for learning must provide accurate information, it must be created manually, and since unrefined video contains a lot of unnecessary frames, it is difficult to find the frame in which the place and action appear in order to construct unrefined video data. It takes a lot of time. In addition, it is very labor intensive to indicate the place and the zone where the action takes place in each frame. On the other hand, labeling the location and action of a clip in units of clips (approximately 5 second intervals) requires relatively little labor.

The present invention is a method and apparatus for recognizing behavior in an artificial neural network-based unrefined video, in which a processor receives an analysis target image, selects some frame images from the analysis target image for each preset section based on a time domain, and It is an artificial neural network that recognizes places and behaviors in the selected frame image and labels the recognized place and behavior values on the selected frame image and learns from the clip unit place and behavior information from the unrefined video. The purpose is to find out in which spatial domain it is appearing.

In addition, it uses a single artificial neural network that derives areas of place and action together, and uses a multitasking method that solves two or more different tasks to identify areas of places and actions that are highly related to each other. There is another purpose to derive together.

Other objects not specified in the present invention may be additionally considered within the scope that can be easily inferred from the following detailed description and effects thereof.

In order to solve the above problem, in the behavior recognition method according to an embodiment of the present invention, a processor receives an analysis target image, and selects some frame images from the analysis target image for each preset section based on a time domain and recognizing a place and an action in the selected frame image, and labeling a feature value according to the recognized place and action on the selected frame image.

Here, the step of recognizing a place and an action in the selected frame image includes extracting a first feature tensor for place recognition of the selected frame image using a first convolutional neural network and using a second convolutional neural network. and extracting a second feature tensor for object recognition of the selected frame image.

Here, the second convolutional neural network is trained on an object recognition dataset similar to the behavior information to be recognized.

Here, the step of recognizing a place and an action in the selected frame image includes performing an operation using an attention function to extract the first feature tensor in the spatiotemporal domain as a first feature vector having a preset size, and The method further includes extracting the second feature tensor in the space-time domain as a second feature vector having a preset size by performing an operation using an attention function.

Here, the step of recognizing a place and an action in the selected frame image includes: performing a class transformation operation that transforms the first feature vector or the second feature vector so that the operation is possible through dimensional transformation, and the first feature The method further includes extracting a combined feature vector by performing a transformer operation using a multitask transformer unit for extracting a feature corresponding to a space-time domain by adding the vector and the second feature vector on which the class transformation operation has been performed.

In this case, the step of performing the transformer operation using the multitask transformer unit includes: a query input unit receiving the first feature vector, and a fully connected feature value according to the input first feature vector and a predetermined connection weight generating, generating a convolutional feature value through convolutional transformation of the selected frame image, performing a matrix multiplication operation of the fully connected feature value and the convolutional feature value, and the fully connected feature value and performing normalization by summing the convolutional feature values on which the matrix multiplication operation has been performed.

Here, the transformer operation includes a first transformer operation to a third transformer operation, and the step of recognizing a place and an action in the selected frame image is to fully connect the combined feature vector extracted by performing the third transformer operation. The method further includes classifying places and actions using a fully-connected layer.

A behavior recognition apparatus according to an embodiment of the present invention includes an image acquisition unit for acquiring an analysis target image including behavior information to be recognized from the outside, a memory for storing one or more instructions, and one or more instructions stored in the memory. and a processor, wherein the processor recognizes a place and an action from the analysis target image based on an artificial neural network.

Here, the processor includes the steps of selecting a part of a frame image from the analysis target image for each preset section based on a time domain in the analysis target image, recognizing a place and an action in the selected frame image, and the recognized place and labeling the feature value according to the action on the selected frame image.

Here, the processor includes: extracting a first feature tensor for place recognition of the selected frame image using a first convolutional neural network; and a second method for object recognition of the selected frame image using a second convolutional neural network. 2 The step of extracting the feature tensor is performed.

Here, the second convolutional neural network is trained on an object recognition dataset similar to the behavior information to be recognized.

Here, the processor performs an operation using an attention function to extract the first feature tensor in the space-time domain as a first feature vector of a preset size, and an attention function. and extracting the second feature tensor in the space-time domain as a second feature vector having a preset size.

Here, the processor performs, by the processor, a class transformation operation for transforming the first feature vector or the second feature vector to be operable through dimensional transformation, and the first feature vector and the second feature vector performing the class transformation operation. A step of extracting a combined feature vector is performed by performing a transformer operation using a multitask transformer unit for summing two feature vectors and extracting a feature corresponding to a space-time domain.

Here, the processor receives the first feature vector, generates a fully connected feature value according to the input first feature vector and a predetermined connection weight, and convolves the selected frame image through convolutional transformation. Generating a convolution feature value, performing a matrix multiplication operation of the fully connected feature value and the convolution feature value, and adding the convolution feature value obtained by performing the matrix multiplication operation with the fully connected feature value Steps to perform normalization are performed.

Here, the transformer operation includes a first transformer operation to a third transformer operation, and the processor uses a fully-connected layer for the combined feature vector extracted by performing the third transformer operation to categorize places and actions.

As described above, according to the embodiments of the present invention, a processor receives an analysis target image, selects some frame images from the analysis target image for each preset section based on a time domain, and selects a frame image from the selected frame image. It is an artificial neural network that recognizes places and actions and labels feature values according to the recognized places and actions on the selected frame image, and learns from the unrefined video clip unit location and behavior information. You can find out if it's showing up.

In addition, it uses a single artificial neural network that derives areas of place and action together, and uses a multitasking method that solves two or more different tasks to identify areas of places and actions that are highly related to each other. can be derived together.

Even if it is an effect not explicitly mentioned herein, the effects described in the following specification expected by the technical features of the present invention and their potential effects are treated as if they were described in the specification of the present invention.

1 is a block diagram of a behavior recognition apparatus according to an embodiment of the present invention.
2 is a diagram for explaining a processor of a behavior recognition apparatus according to an embodiment of the present invention.
3 is a diagram illustrating an artificial neural network structure of a behavior recognition apparatus and method according to an embodiment of the present invention.
4 is an example showing the labeling of the behavior recognition apparatus and method according to an embodiment of the present invention.
5 is a diagram illustrating an MTx operation structure of a behavior recognition apparatus and method according to an embodiment of the present invention.
6 is a diagram illustrating an AQPr operation structure of a behavior recognition apparatus and method according to an embodiment of the present invention.
7 shows experimental results using the behavior recognition apparatus and method according to an embodiment of the present invention.
8 to 10 are flowcharts showing a behavior recognition method according to an embodiment of the present invention.

Hereinafter, a method and apparatus for recognizing a behavior in an artificial neural network-based unrefined video according to the present invention will be described in more detail with reference to the drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly explain the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves.

The present invention relates to a method and apparatus for recognizing behavior in an artificial neural network-based unrefined video.

1 is a block diagram of a behavior recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 1 , a behavior recognition apparatus 1 according to an embodiment of the present invention includes a processor 10 , an image acquisition unit 20 , a memory 30 , and an I/O interface 40 .

The behavior recognition device 1 according to an embodiment of the present invention is a device for recognizing a behavior in an unrefined video, and is a QKV (Query, Key, Value) used in a Multitask Transformer Network, which is a type of memory network among deep learning structures. ) concept to use an algorithm to find out in which frame and in which spatial domain the place and action are appearing with an artificial neural network that is learned from clip unit location and behavior information from unrefined video.

The behavior recognition apparatus 1 according to an embodiment of the present invention utilizes an artificial neural network learned with place and behavior information labeled in units of clips in an unrefined video to find specific spatiotemporal regions in which places and actions appear respectively. The purpose.

The processor 10 recognizes a place and an action from the analysis target image based on an artificial neural network.

After learning with data having a small amount of information, the processor 10 of the behavior recognition device 1 according to an embodiment of the present invention learns to generate more detailed and specific information. Weakly supervised learning method learning) is used. A typical example of weakly supervised learning used in deep learning is CAM (Class Activation Map). In this method, after learning the artificial neural network with data labeled at the image-level, the result is derived at the pixel-level. When learning, the artificial neural network structure consists of a GAP (Global Average Pooling) and FC (Fully Connected) layer, the composition of the last layer that classifies the feature tensor extracted through the convolution filter. make up Then, the feature tensor passes through the GAP and is pooled into a spatial domain to become a feature vector, and since the FC layer classifies this feature vector, learning is performed at the image-level. When deriving a result at the pixel-level, it proceeds to a structure in which GAP is removed from the artificial neural network structure. In this case, since the feature tensor extracted through the convolution filter goes through the FC layer as it is, each of the feature vectors existing in each spatial region of the feature tensor is classified through the FC layer, so that each spatial region (pixel-level) classification is performed. be able to

In addition, there is also a T-CAM (Temporal Class Activation Map) method in which the method of extending information in the spatial domain is changed to the temporal domain. In this method, the GAP, which was pooled in the spatial domain in the artificial neural network structure that was trained in the image-level, is pooled in the temporal domain so that it is learned in the video-level ( pooling), and then removes the GAP when deriving a result in the same way as CAM so that it is classified every time.

In addition, CAM is commonly used in weakly supervised learning methods using deep learning. The processor 10 of the behavior recognition apparatus 1 according to an embodiment of the present invention transmits video-level information in spatio-temporal method using a weakly supervised learning method that has many characteristics different from those of CAM. Use the method of specifying (or localizing, localization) as a region. The processor 10 of the behavior recognition apparatus 1 according to an embodiment of the present invention performs an artificial neural network structure based on a multitask transformer network. This Multitask Transformer Network is an artificial neural network structure in which the Action Transformer Network classifies only the action in the video, and expands the artificial neural network structure so that the scene and the action are classified together. .

The Action Transformer Network is an artificial neural network structure that has been changed so that what the Transformer Network was used for natural language processing (language translation) can be applied to video. This Transformer Network is an artificial neural network structure that utilizes the QKV (Query, Key, Value) concept among the structures of the Memory Network to enable high-performance natural language processing.

In the case of CAM, the artificial neural network structure is the structure of the Convolution Filter - GAP - FC, and the structure of the last GAP-FC must be used. Therefore, there are many limitations in the design of the artificial neural network structure, and it is difficult to expect high performance from the image (video)-level classification to be learned. Image(video)-level classification with low performance inevitably has low localization performance in the spatiotemporal domain, which should be finally done. However, in the case of the QKV concept, it can be used anywhere in the artificial neural network structure, and this high degree of freedom in design derives high performance and has been widely used recently. In addition, since the artificial neural network structure is different when learning and deriving results (the GAP that existed during learning is removed when deriving results), it is difficult to expect good performance from CAM. In the case of the weakly supervised learning method using the QKV concept, good performance can be expected in the case of a well-trained artificial neural network because an artificial neural network with the same structure is used for learning and for deriving results.

The image acquisition unit 20 acquires an analysis target image including behavior information to be recognized from the outside. When an analysis target image is acquired through a separate input unit, a place and an action in the analysis target image are recognized through a processor.

The memory 30 may store programs (one or more instructions) for processing and controlling the processor 10 .

The I/O interface 40 is a device capable of mounting a connection medium capable of connecting a system or equipment, and in the present invention, an image acquisition unit and a processor are connected.

2 is a diagram for explaining a processor of a behavior recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 2 , the processor 10 of the behavior recognition apparatus 1 according to an embodiment of the present invention includes a place recognition feature extraction unit 100 , an object recognition feature extraction unit 200 , an AQPr operation unit 300 , It includes a CCM calculating unit 400 , an MTx calculating unit 500 , and a feature value classifying unit 600 .

The artificial neural network structure used in the processor 10 of the behavior recognition apparatus 1 according to an embodiment of the present invention will be described in detail with reference to FIG. 3 below.

The processor 10 may be divided into a plurality of modules according to functions, and functions may be performed by one processor.

The processor 10 recognizes a place and an action from the analysis target image based on an artificial neural network. Specifically, a frame selected for each preset section in the analysis target image is selected, a place and an action are recognized in the selected frame, and a characteristic value according to the recognized place and action is labeled for each selected frame.

The place recognition feature extraction unit 100 extracts a first feature tensor for place recognition of the selected frame by using a first convolutional neural network.

The object recognition feature extraction unit 200 extracts a second feature tensor for object recognition of the selected frame using a second convolutional neural network.

Here, the second convolutional neural network is trained on an object recognition dataset similar to the behavior information to be recognized.

The AQPr calculating unit 300 includes a first AQPr calculating unit 300a and a second AQPr calculating unit 300b. The first AQPr calculating unit 300a extracts the first feature tensor in the space-time domain as a first feature vector having a preset size by performing an operation using an attention function.

The second AQPr operation unit 300b extracts the second feature tensor in the space-time domain as a second feature vector having a preset size by performing an operation using an attention function.

The CCM operation unit 400 performs a class transformation operation so that the operation is possible through dimensional transformation of the first feature vector or the second feature vector.

The MTx operation unit 500 performs a transformer operation using a multitask transformer unit for extracting a feature corresponding to a space-time domain by adding the first feature vector and the second feature vector on which a class transformation operation has been performed to obtain a combined feature vector extract

The MTx operation unit 500 includes a first MTx operation unit to a third MTx operation unit to perform a first transformer operation to a third transformer operation, and the feature value classification unit 600 performs the third transformer operation and the extracted combination Classify places and behaviors using feature vectors as fully-connected layers.

3 is a diagram illustrating an artificial neural network structure of a behavior recognition apparatus and method according to an embodiment of the present invention.

A behavior recognition apparatus and method according to an embodiment of the present invention proposes a method of finding a space-time region in which a place and an action appear in a weakly supervised learning method in an unrefined video.

The Multitask Transformer Network, which is an artificial neural network structure used in the processor 10 of the behavior recognition device 1 according to an embodiment of the present invention, uses the QKV concept, and learns by using the QKV concept. It learns from video-level place and behavior information, but the result derives which place and behavior correspond to in the space-time domain.

In addition, the present invention uses a single artificial neural network that derives the domains of places and actions together. This method of solving two or more different tasks is called multitasking. Since places and actions, which are each task, are highly related to each other, it is better than using an artificial neural network that is independent of places and actions. Better performance can be expected.

As shown in FIG. 3 , the multitask transformer network used in the processor 10 of the behavior recognition device 1 according to an embodiment of the present invention receives a segment, which is a certain section of an unrefined video, as an input. Receive and output the location and behavior of the segment as output. In the case of segment, which is an input, if the entire unrefined video is used as an input at once, 32 RGB images at an interval of 6.4 fps, that is, 32 frames for 5 seconds due to the limitation of the memory of the graphic card to be used for parallel operation for deep learning acceleration is entered as input. Therefore, the Multitask Transformer Network of FIG. 3 outputs what kind of place and action the corresponding 5-second segment represents, and shows what the place and action are at 5-second intervals in the entire unrefined video.

Learning of the Multitask Transformer Network follows the stochastic gradient descent (SGD) method commonly used in deep learning, and it is preferable that the CoVieW 2019 dataset be used as the data set used for learning to experimentally demonstrate the performance of the present invention. . The CoVieW 2019 dataset is a dataset with three labels: a scene, an action, and an import score of how important the 5-second segment is in the video at 5-second intervals in the unrefined video. . An example of labeling in one unrefined video in the CoVieW 2019 dataset is shown in FIG. 4 below.

A detailed look at the structure of the artificial neural network of FIG. 3 is as follows. First, by using the Places365 2D CNN and ImageNet 2D CNN trained on the Places365 and ImageNet datasets in the place recognition feature extraction unit 100 and the object recognition feature extraction unit 200, respectively, the segment frame features are extracted. The 2D CNN used at this time uses ResNet18, which receives H×W×3 RGB (3-channel) images as (height×width×RGB channels) as input and H/32×W/32×512 as output. It is an artificial neural network (2D CNN) that outputs size feature tensors. In the present invention, 32×224×224×3 is input as a segment (time length×height×width×RGB channel) and features of size 32×7×7×512 are extracted from each 2D CNN.

Here, Places365 2D CNN is trained in Places365, a place recognition dataset, so it extracts place-related features, and ImageNet 2D CNN is trained in ImageNet, an object recognition dataset, so it extracts object-related features. In the case of ImageNet 2D CNN, we will finally recognize the behavior. The reason why we learned on the object recognition dataset rather than the 2D CNN trained on the behavior recognition dataset is that the 2D CNN is a 2-Dimensional CNN that uses images rather than videos as input. In order to receive , it is necessary to use a dataset consisting of images, but since there is no behavior recognition dataset consisting of images, a 2D CNN trained on the object recognition dataset most similar to behavior recognition was used. And, the reason why 2D CNN was trained and used in other datasets in advance is that the CoVieW 2019 dataset provides a total of 1500 unrefined videos and provides a relatively small amount of data to learn the deep learning structure, resulting in an overfitting problem. 2D CNN is pre-trained in Places365 (about 8 million images) and ImageNet (about 1.2 million images), which provide a relatively large amount of data, so that general features can be extracted.

으로 나타낼 때, AQPr에서의 수학식 1과 같이 진행된다.After that, as shown in FIG. 3, the feature (convolution feature) extracted by the 2D CNN is input to AQPr (Attentional Query Processor) implementing the first AQPr operation unit 300a and the second AQPr operation unit 300b as input, and MTx It is also used as an input of MTx (Multitask Transformer units) implementing the operation unit 500 . 32×7×7×512 extracted with 2D CNN

When expressed as , it proceeds as in Equation 1 in AQPr.

이 AQPr의 출력이고,

는 sigmoid function이고,

는 학습되는 weight (trainable parameter)이고 InstanceNorm은 Instance Normalization으로 이 역시 학습되는 weight를 포함한 정규화 (normalization) 방식 중 하나이다. 따라서, AQPr을 거치면 32Х7Х7Х512의 크기의 시공간 영역의 특징 텐서가 512 크기의 특징 벡터로 추출되며, 이 특징 벡터와 도 3의 CCM 연산부(400)를 구현하는 CCM (class conversion matrix) 에서 추출된 특징 벡터가 더해져서 MTx의 query 입력으로 사용된다.here,

This is the output of AQPr,

is a sigmoid function,

is a trained weight (trainable parameter), and InstanceNorm is Instance Normalization, which is also one of the normalization methods including the trained weight. Accordingly, through AQPr , a feature tensor of a spatiotemporal domain with a size of 32Х7Х7Х512 is extracted as a feature vector with a size of 512. is added and used as a query input for MTx.

, 출력을

, n은 특징 벡터의 채널 (도 3에서 AQPr의 출력과 더해지는 부분은 n=512 이며, 나머지 "MTx->concat->"과 더해지는 부분은 n=256이다.) 이라 하였을 때, CCM에서의 연산은 수학식 2와 같이 진행된다.In the case of the CCM of Figure 3, the input

, the output

, n is the channel of the feature vector (in FIG. 3, the part added to the output of AQPr is n=512, and the part added to the remaining "MTx->concat->" is n=256 ), the operation in CCM is performed as in Equation 2.

는 ReLU function이고,

는 학습되는 weight이다. here

is the ReLU function,

is the weight to be learned.

And the MTx operation is performed as shown in FIG. 5 below.

In FIG. 3 , a solid line indicates “a place where a gradient is propagated when learning”, and a gray dotted line indicates “a place where a gradient does not propagate when learning”. In other words, since the gradient is not propagated in the CCM, which is the part where the sharing of features (information) of place and behavior occurs, the MTx learns only about the place or behavior, which is the task it is in charge of. That is, in FIG. 3 , the MTx indicated by the upper line with a plain pattern have expertise in the place, and the MTx indicated by the hatched lower line have the specialization in the action.

If the Multitask Transformer Network of FIG. 3 is further explained, as mentioned briefly above, information about places and actions is shared in CCM, and AQPr serves to aggregate feature tensors in the spatiotemporal domain into one feature vector (feature aggregation). , and MTx finds the region to be viewed in memory (feature vector extracted with 2D CNN) through the operation of query features and matrix multiplication (each proceeds with inner product operation in the spatio-temporal region) as shown in the operation in FIG. In order to extract the features corresponding to the space-time domain, it is extracted through matrix multiplication operation once more. That is, the first matrix multiplication is to find a space-time region to be viewed, and the second matrix multiplication is to extract features corresponding to the space-time region. And the rear layer norm, FC, etc. have the same role as a general multi-layer perceptron (MLP), and play a role of embedding features in a high-dimensional manner.

In the first Matrix Multiplication, the spatiotemporal region that best matches the feature (Query, 1x512, 1x128 through FC, which is Query Embedding) containing the information to be found related to place/behavior recognition (Memory 32x7x7x512, 32x7x7x128 when subjected to Key Embedding) To find, we extract the 32x7x7x1 features through Matrix Multiplication to obtain the probability that a place/action exists in the spatiotemporal domain. In the second Matrix Multiplication, the 32x7x7x1 and Value Embedding extracted above are used to extract the place/action features corresponding to the spatiotemporal domain 32x7x7. The extracted 32x7x7x128 and 1x128 size features are extracted through Matrix Multiplication.

In other words, the Query (1x512, 1x128 if going through FC) is a feature about the place/action previously extracted (entered into the input of MTx), and the feature extracted from Key Embedding (32x7x7x128) is the time-space extracted to match the Query. It is a feature, and the feature extracted from Value Embedding (32x7x7x128) is a feature for each space-time domain for action/place.

4 is an example showing the labeling of the behavior recognition apparatus and method according to an embodiment of the present invention.

Learning of the Multitask Transformer Network follows the stochastic gradient descent (SGD) method commonly used in deep learning, and the CoVieW 2019 dataset is used as the dataset used for learning to experimentally demonstrate the performance of the present invention. The CoVieW 2019 dataset is a dataset with three labels: a scene, an action, and an import score of how important the 5-second segment is in the video at 5-second intervals in the unrefined video. . An example of labeling in one unrefined video in the CoVieW 2019 dataset is shown in FIG.

5 is a diagram illustrating an MTx operation structure of a behavior recognition apparatus and method according to an embodiment of the present invention.

The MTx operation unit 500 extracts a combined feature vector by adding the first feature vector and the second feature vector on which the class conversion operation is performed, and performing a transformer operation using a multitask transformer unit.

The multitask transformer unit of the MTx operation unit 500 includes a query input unit 510 , a memory input unit 520 , and at least one fully-connected layer 530 .

The MTx calculator 500 receives the first feature vector through the query inputter 510 and generates a fully connected feature value according to the input first feature vector and a predetermined connection weight.

In addition, a convolutional feature value is generated through convolutional transformation of the selected frame image input to the memory input unit 520 .

A matrix multiplication operation of the fully connected feature value and the convolutional feature value is performed, and normalization is performed by adding the fully connected feature value and the convolution feature value on which the matrix multiplication operation is performed.

In FIG. 5 , a query implementing the query input unit 510 is an input coming from the left side of the MTx block of FIG. 3 , and Memory is an input coming from above or below the MTx block. FC stands for fully connected layer, 1x1x1 Conv stands for 1x1x1 Convolution, and Layer Norm is Layer Normalization, which is one of the normalization methods including learned weights, similar to the aforementioned InstanceNorm. Softmax represents the softmax function. Dropout was used to alleviate the overfitting problem.

는 matrix multiplication 연산자를,

는 element-wise sum (일반적인 덧셈) 연산자를 나타낸다. 그리고 빨간색 화살표의 경우, 특별한 연산이 따로 있는 것이 아니며, 추후 설명할 본 발명의 약 지도 학습법에서 사용할 특징 텐서가 빨간색 화살표에서 추출된 특징 텐서를 사용할 것임을 나타낸다. 도 5의 MTx 에서 최종적으로 출력되는 맨 오른쪽 화살표에선,

의 크기를 갖는 특징 벡터가 추출이 된다.

is the matrix multiplication operator,

represents the element-wise sum (normal addition) operator. And, in the case of the red arrow, there is no special operation, and it indicates that the feature tensor extracted from the red arrow will be used as the feature tensor to be used in the weakly supervised learning method of the present invention, which will be described later. In the rightmost arrow finally output from MTx of FIG. 5,

A feature vector with a size of is extracted.

크기 벡터가 단순히 연결되어

의 크기가 됨을 나타낸다.Thereafter, as shown in FIG. 3, features extracted from two MTx are subjected to “concat”, which is an abbreviation for concatenation,

The magnitude vectors are simply concatenated

indicates the size of

In addition, the feature value classification unit 600 extracts the MTx stacked by two in FIG. 3 a total of three times, and the feature vector is finally input to one fully connected layer (FC), and through this, a scene, Categorize actions.

The Multitask Transformer Network of FIG. 3 is an artificial neural network that classifies the location and behavior of a 5-second segment (video clip), and only requires information on the corresponding location and behavior every 5 seconds of the video when learning. And, the artificial nerve learned in this way is learned so that it can find out the place and the action well, even if the person does not give information about which spatiotemporal region to see in the arrow part of FIG. 5 . Therefore, the feature tensor extracted from this red arrow will indicate the spatiotemporal region in which places and actions appear, and since learning information for this purpose is not given, if this information is used to express the regions indicated by places and actions, it is a weakly supervised learning method. (Because it expresses the space-time domain in which the place and action appear after learning only with the relatively easy information, such as behavior and place information every 5 seconds)

, query가 FC를 거쳐 추출된 Query Embedding 특징 벡터를

, memory가 1x1x1 Conv를 거쳐 추출된 Key Embedding 특징 텐서를

으로 표현할 때 수학식 3과 같이 계산된다 (T는 time, H는 height, W는 width으로 처음에 설명한 T=32, H=7, W=7이 사용된다)For an accurate representation, the feature tensor extracted from the arrow

, the query embedding feature vector extracted through FC

, the key embedding feature tensor extracted through 1x1x1 conv

It is calculated as in Equation 3 when expressed as

으로 각각 상수값(scalar) 를 하나씩 갖게 되며, 이는 시공간 영역의 장소 또는 행동이 나타나는 정도를 띄게 된다.Here, softmax is a softmax function, so that the total sum of the feature tensor Y is 1. Thus, the space-time domain

Each of them has a constant value (scalar), which shows the extent to which a place or action in the space-time domain appears.

를 bilinear interpolation 방식으로 높이와 너비를 32배 늘려준 후, 원본 이미지 크기로 맞춰주고 visualization을 위해 값이 낮은 곳은 파란색, 값이 높은 곳은 빨간색으로 표현하면 하기 도 7과 같은 결과를 얻을 수 있다.However, this T×H×W=32×7×7 is the size extracted from the 2D CNN, which is 1/32 times larger than the height (H) and width (W) of the original image. Therefore, this

After increasing the height and width by 32 times using the bilinear interpolation method, adjust the original image size and express low values in blue and high values in red for visualization, the result shown in FIG. 7 can be obtained.

Finally, in the present invention, the importance of each 32 (time) × 7 (height) × 7 (width) output as a Softmax result in the MTx intended in the present invention is set to the original size of 32 (time) × 224 (height) × 224 (width). To increase the size of , a bilinear interpolation method is used.

Since CNN uses pooling to reduce the height to 1/2 the size 5 times, it extracts features that are 1/32 times larger than the height of the input image. Therefore, the importance of every 32×7×7, which is a feature used in the present invention, is not compressed in the temporal direction, but is compressed by 1/32 times only in the spatial direction. Since there is no change in size in the time direction, when viewed only from the perspective of one frame, 224×224, each cell of the 7×7 feature is responsible for the size of 32×32 in a 224×224 image. In order to visualize this nicely, the 7×7 feature is increased to 224×224, and the interpolation method only needs to increase the 7×7 size feature to 224×224 regardless of what is used. We adopt the bilinear ineterpolation method, which is the method used in CAM, a related weak-supervised learning method.

6 is a diagram illustrating an AQPr operation structure of a behavior recognition apparatus and method according to an embodiment of the present invention.

으로 나타낼 때, AQPr에서의 상기 수학식 1과 같이 진행된다.As shown in FIG. 3, the feature (convolution feature) extracted by the 2D CNN is input to the AQPr (Attentional Query Processor) and is also used as an input to the MTx (Multitask Transformer units). 32×7×7×512 extracted with 2D CNN

When expressed as , it proceeds as in Equation 1 above in AQPr.

Through AQPr, a feature tensor of a space-time domain with a size of 32×7×7×512 is extracted as a feature vector with a size of 512 , and this feature vector and the feature vector extracted from the CCM (class conversion matrix) of FIG. It is used as a query input for MTx.

7 shows experimental results using the behavior recognition apparatus and method according to an embodiment of the present invention.

7 shows experimental results for two segments (video clips). The input video represents the original image, the scene represents the space-time domain of the place, and the action represents the space-time domain of the action. Among the two segments, in the above experimental result, it can be seen that the place is spread out over most of the area, and the area is concentrated only on the person for the action. It can be seen that the original segment represents the concert place and the performance action well. For the segment result below, it can be seen that the areas where places and actions appear are concentrated only in some specific times, and the indicated areas are also concentrated in a specific area. In this case, when a place or action is remarkably well displayed in a specific region of a specific frame, the result of concentrating the space-time domain only in the specific frame appears. Therefore, it can be seen that cycling, an action, showed well to cyclists, and in the case of park, which is a place, the space-time domain did not appear in the indoor space, and it was confirmed that the area was well expressed only outdoors.

8 to 10 are flowcharts showing a behavior recognition method according to an embodiment of the present invention.

Referring to FIG. 8 , in the behavior recognition method according to an embodiment of the present invention, the processor receives an analysis target image, and selects some frame images from the analysis target image for each preset section based on a time domain. (S100), recognizing a place and action in the selected frame image (S200), and labeling a feature value according to the recognized place and action on the selected frame image (S300).

Referring to FIG. 9 , in the step S200 of recognizing a place and an action in a selected frame, a first feature tensor for recognizing a place in the selected frame is extracted using a first convolutional neural network in step S210.

In step S220, a second feature tensor for object recognition of the selected frame is extracted using a second convolutional neural network.

Here, the second convolutional neural network is trained on an object recognition dataset similar to the behavior information to be recognized.

In step S230, an operation using an attention function is performed to extract the first feature tensor in the space-time domain as a first feature vector having a preset size.

In step S240, an operation using an attention function is performed to extract the second feature tensor in the space-time domain as a second feature vector having a preset size.

In step S250, a class transformation operation for transforming the first feature vector or the second feature vector to be operable through dimensional transformation is performed.

In step S260, the combined feature vector is extracted by performing a transformer operation using a multitask transformer unit for extracting a feature corresponding to a space-time domain by adding the first feature vector and the second feature vector on which the class transformation operation has been performed.

In step S260, the transformer operation includes a first transformer operation to a third transformer operation,

In step S270, the combined feature vector extracted by performing the third transformer operation is classified into places and behaviors using a fully-connected layer.

Referring to FIG. 10 , in the step S260 of performing the transformer operation using the multitask transformer unit, a first feature vector is received in step S261, and a pulley according to the input first feature vector and a predetermined connection weight Create a connected feature value.

In step S262, the selected frame image is subjected to convolutional transformation to generate a convolutional feature value.

In step S263, a matrix multiplication operation of the fully connected feature value and the convolution feature value is performed.

In step S264, normalization is performed by adding the fully connected feature value and the convolution feature value on which the matrix multiplication operation is performed.

The behavior recognition apparatus and method according to an embodiment of the present invention require a lot of labor for a person to find all the space-time regions in which a place and an action occur in an unrefined video. It is relatively much easier to label . In the present invention, using the QKV concept, we propose a weakly supervised learning method to find the spatiotemporal region in which the place and action in the unrefined video are displayed through the location and action labeling of the video unit. In the present invention, by using a single artificial neural network that derives the spatiotemporal domain of place and action, a strong artificial neural network through the relationship between place and action can learn, so better performance is expected compared to an artificial neural network that derives only the result of place or action. can

The above description is only one embodiment of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to implement it in a modified form without departing from the essential characteristics of the present invention. Therefore, the scope of the present invention is not limited to the above-described embodiments, and should be construed to include various embodiments within the scope equivalent to the content described in the claims.

Claims (15)

receiving, by a processor, an analysis target image and selecting a part of a frame image from the analysis target image for each preset section based on a time domain;
recognizing a place and an action in the selected frame image; and
Including; labeling a feature value according to the recognized place and action on the selected frame image;
Recognizing a place and an action in the selected frame image comprises:
extracting a first feature tensor for location recognition of the selected frame image using a first convolutional neural network;
extracting a second feature tensor for object recognition of the selected frame image using a second convolutional neural network;
Using a CCM (class conversion matrix) operator to share the characteristic information of the place and the behavior without propagating a gradient,
An MTx belonging to the first group intensively learns the place by using a multitask transformer unit (MTx) operation unit, and the MTx belonging to the second group intensively learns the behavior on the behavior recognition method. delete According to claim 1,
The second convolutional neural network is
A behavior recognition method, characterized in that it is learned from an object recognition dataset similar to the behavior information to be recognized. According to claim 1,
Recognizing a place and an action in the selected frame image comprises:
extracting the first feature tensor in the space-time domain as a first feature vector having a preset size by performing an operation using an attention function; and
Behavior recognition method further comprising; extracting the second feature tensor of the space-time domain as a second feature vector of a preset size by performing an operation using an attention function. 5. The method of claim 4,
Recognizing a place and an action in the selected frame image comprises:
performing a class transformation operation for transforming the first feature vector or the second feature vector so that the operation is possible through dimensional transformation; and
extracting a combined feature vector by performing a transformer operation using a multi-task transformer unit for extracting a feature corresponding to a space-time domain by summing the first feature vector and the second feature vector on which a class transformation operation has been performed; Behavior recognition method comprising: 6. The method of claim 5,
The step of performing a transformer operation using the multi-task transformer unit includes:
receiving the first feature vector by a query input unit and generating a fully connected feature value according to the input first feature vector and a predetermined connection weight;
generating a convolutional feature value through convolutional transformation of the selected frame image;
performing a matrix multiplication operation of the fully connected feature value and the convolution feature value; and
and performing normalization by summing the fully connected feature value and the convolutional feature value on which the matrix multiplication operation has been performed. 6. The method of claim 5,
The transformer operation includes a first transformer operation to a third transformer operation,
Recognizing a place and an action in the selected frame image comprises:
and classifying places and actions using a fully-connected layer with the combined feature vector extracted by performing the third transformer operation. an image acquisition unit for acquiring an analysis target image including behavior information to be recognized from the outside;
a memory storing one or more instructions; and
Including; a processor that executes one or more instructions stored in the memory;
The processor recognizes a place and an action from the analysis target image based on an artificial neural network,
selecting, by the processor, a part of a frame image from the analysis target image for each preset section based on a time domain in the analysis target image;
recognizing a place and an action in the selected frame image; and
performing a labeling of a feature value according to a recognized place and an action on the selected frame image;
The processor may include: extracting a first feature tensor for location recognition of the selected frame image using a first convolutional neural network; and
extracting a second feature tensor for object recognition of the selected frame image using a second convolutional neural network;
The processor uses a class conversion matrix (CCM) operation unit to share characteristic information of the location and the behavior without propagating a gradient,
An MTx belonging to the first group intensively learns the place by using a multitask transformer unit (MTx) operation unit, and the MTx belonging to the second group intensively learns the action on the behavior recognition apparatus. delete delete 9. The method of claim 8,
The second convolutional neural network is
Behavior recognition apparatus, characterized in that it is learned from an object recognition dataset similar to the behavior information to be recognized. 9. The method of claim 8,
extracting, by the processor, the first feature tensor of the space-time domain as a first feature vector having a preset size by performing an operation using an attention function; and
Behavior recognition apparatus comprising: performing an operation using an attention function to extract the second feature tensor in the space-time domain as a second feature vector having a preset size; 13. The method of claim 12,
performing, by the processor, a class transformation operation for transforming the first feature vector or the second feature vector so that the operation is possible through dimensional transformation; and
performing a transformer operation using a multi-task transformer unit for extracting a feature corresponding to a space-time domain by summing the first feature vector and the second feature vector on which a class transformation operation has been performed to extract a combined feature vector; Behavior recognition device, characterized in that. 14. The method of claim 13,
receiving, by the processor, the first feature vector and generating a fully connected feature value according to the input first feature vector and a predetermined connection weight;
generating a convolutional feature value through convolutional transformation of the selected frame image;
performing a matrix multiplication operation of the fully connected feature value and the convolution feature value; and
and performing normalization by summing the fully connected feature value and the convolutional feature value on which the matrix multiplication operation has been performed. 14. The method of claim 13,
The transformer operation includes a first transformer operation to a third transformer operation,
and the processor classifies a place and a behavior using a fully-connected layer for the combined feature vector extracted by performing the third transformer operation;

Images