KR20210040604A - Action recognition method and device - Google Patents

Abstract

The present invention relates to an action recognition method and device using an unsupervised learning method. An action recognition device according to one embodiment of the present invention may comprise: an input module for receiving analysis target image and reference image; a feature extraction module that extracts features of the received analysis target image and the received reference image in frame units using an artificial neural network, and extracts the core features that are some of the extracted features, respectively; and a verification module that calculates the similarity between the extracted key features, and outputs the reference image having the greatest similarity among reference images for which the similarity is calculated. Therefore, the present invention has the effect of being capable of classifying an input image by analyzing a previously unlearned action.

Description

Behavior recognition method and device {Action recognition method and device}

The present invention relates to a method and apparatus for recognizing actions, and more particularly, to a method and apparatus for recognizing actions using an unsupervised learning method.

Surveillance systems are becoming an essential element of modern society to minimize damage to people and assets in preparation for crimes such as assault and theft and unexpected situations. However, since it is a reality that it is impossible for a monitor to check the camera image by monitoring the surveillance system one by one, a surveillance system capable of detecting objects in the surveillance area by analyzing the acquired image in real time was required. Based on this demand, a deep learning-based behavior recognition method that can classify data derived from a human neural network algorithm has recently attracted attention.

Recognizing human behavior and behavior is an important skill in developing surveillance systems. In the past few years, a number of methods have been proposed for behavior recognition methods. Conventional behavior recognition technology is provided with multiple learning images showing behaviors based on supervised learning among deep learning methods, and extracts features from the learning images through supervised learning, so that the input image is added to the category of the learning image. It was possible to classify.

However, these methods can only classify predefined actions, so the detectable actions are very limited. Since human behavior is diverse and complex, the application of this cognitive model is bound to be limited. In addition, to add an action targeting classification, a lot of training image data for the action is required, so when a new action is added, retraining the model and changing the structure of the model are required, which complicates the process. there was.

Accordingly, it is possible to recognize behaviors that have not been learned in advance, and when adding new behaviors, the relearning process is simplified and there is no need to change the structure of the model, and a new method of behavior recognition is required.

H. Wang and C. Schmid. Action recognition with improved trajectories. In Proceedings of the IEEE international con?ference on computer vision, pages 3551-3558, 2013. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pages 568-576, 2014. L. Sun, K. Jia, D.-Y. Yeung, and B. E. Shi. Human action recognition using factorized spatio-temporal convolutional networks. In Proceedings of the IEEE International Con?ference on Computer Vision, pages 4597-4605, 2015. L. Wang, Y. Qiao, and X. Tang. Mofap: A multi-level representation for action recognition. International Journal of Computer Vision, 119(3):254-271, 2016. Z. Lan, Y. Zhu, A. G. Hauptmann, and S. Newsam. Deep local video feature for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1-7, 2017. A. Diba, V. Sharma, and L. Van Gool. Deep temporal linear encoding networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 2329-2338, 2017. J. Zhu, Z. Zhu, and W. Zou. End-to-end video-level representation learning for action recognition. In 2018 24th International Conference on Pattern Recognition (ICPR), pages 645-650. IEEE, 2018.

An object of the present invention is to solve the above-described problem, and an object of the present invention is to provide a method and apparatus for recognizing a behavior using an unsupervised learning method.

In addition, an object of the present invention is to provide an action recognition method and apparatus capable of classifying an action that has not been previously learned.

The objects of the present invention are not limited to the above-described objects, and other objects not mentioned will be clearly understood from the following description.

The behavior recognition method according to an embodiment of the present invention includes the steps of: (a) receiving an analysis target image and a first reference image from among a plurality of reference images; (b) extracting features of the analysis target image and features of the first reference image in a frame unit using an artificial neural network; (c) extracting features of the extracted image to be analyzed and core features that are some of the features of the first reference image; And (d) calculating a degree of similarity between the extracted core features.

After the step (d), (e) receiving a second reference image different from the first reference image among the plurality of reference images; (f) calculating a similarity between the features of the analysis target image and the features of the second reference image: and (g) a reference image having a larger calculated similarity among the first and second reference images. It may further include the step of outputting.

After the step (g), calculating a similarity between one of the remaining images among the plurality of reference images and features of the analysis target image, and outputting a reference image having the greatest similarity. Can include.

After the step (d), the steps (b), (c) and (d) are repeated for the other reference images except for the first reference image to obtain a reference image having the largest similarity among the calculated similarities. It may further include the step of outputting.

Prior to the step (a), the training step further includes a training step, wherein the training step includes an anchor image including behavior information, a positive image including behavior information identical to that of the anchor image, and the anchor image Receiving a negative image including behavior information different from the one; Calculating a similarity between the anchor image and the positive image and a similarity between the anchor image and the negative image; And calculating a loss function using the calculated similarities.

In the step (a), the analysis target image may be input from the outside, and the reference image may be input from a database unit in which the reference image is previously stored.

The step (b) may include generating a plurality of feature maps for each frame; Blocking feature maps having the same size among the feature maps; And extracting features by receiving the output values of feature maps of all layers arranged before the feature map of each layer included in one block.

The step (c) may include extracting key features using a K-means clustering algorithm.

The step (d) may include calculating a similarity by selecting only core features belonging to a section in which an action occurs in the analysis target image and the first reference image among the extracted core features.

The computer-readable recording medium according to an embodiment of the present invention may record a program for performing the above-described method on a computer.

A behavior recognition apparatus according to an embodiment of the present invention includes an input module for receiving an analysis target image and a reference image; A feature extraction module for extracting features of the received analysis target image and features of the input reference image in a frame unit using an artificial neural network, and extracting core features, which are some of the extracted features, respectively; And a verification module that calculates a similarity between the extracted core features and outputs a reference image having the largest similarity among the reference images for which the similarity is calculated.

It may further include a database unit for storing the reference image.

The anchor image and the positive calculated from an anchor image including behavior information, a positive image including behavior information identical to the anchor image, and a negative image including behavior information different from the anchor image A training module that receives a similarity between images and a similarity between the anchor image and the negative image, calculates a loss function using the received similarities, and trains the behavior recognition method of claim 1.

The input module may receive the analysis target image from an external source, and may receive the reference image from a database unit.

The feature extraction module generates a plurality of feature maps for each frame, blocks feature maps of the same size among the feature maps, and a feature map of each layer included in one block is Features can be extracted by receiving output values of feature maps of all layers previously arranged.

The feature extraction module may extract core features using a K-means clustering algorithm.

The verification module may calculate a similarity by selecting only core features belonging to a section in which an action occurs in the analysis target image and the reference image among the extracted core features.

The method for recognizing an action according to an embodiment of the present invention has an effect that it is possible to classify an input image by analyzing an action that has not been previously learned.

In addition, the method for recognizing an action according to an embodiment of the present invention has an effect that an unmanned monitoring system is possible since the action recognition apparatus can learn the actions of an object by itself through the method.

In addition, the method for recognizing an action according to an embodiment of the present invention has an effect that it is possible to minimize cost because it is not necessary to change the structure of a model when adding new actions.

The effects of the present invention are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a diagram illustrating an apparatus for recognizing an action according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating an example of extracting a feature of an image in a frame unit by a feature extraction module of the behavior recognition apparatus according to an embodiment of the present invention.
3 is a diagram illustrating an example of extracting core features from a feature extraction module of the behavior recognition apparatus according to an embodiment of the present invention.
FIG. 4 is a diagram exemplarily illustrating selecting only key features belonging to a section in which an action occurs in an image among key features extracted from a verification module of the behavior recognition apparatus according to an embodiment of the present invention.
5 is a diagram illustrating a training process of an apparatus for recognizing an action according to an embodiment of the present invention.
6 is a diagram illustrating a flow chart of a method for recognizing an action according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments of the present invention disclosed in this specification or application are exemplified only for the purpose of describing the embodiments according to the present invention, and the embodiments according to the present invention may be implemented in various forms. And should not be construed as being limited to the embodiments described in this specification or application.

Since the embodiments according to the present invention can be modified in various ways and have various forms, specific embodiments are illustrated in the drawings and will be described in detail in the present specification or application. However, this is not intended to limit the embodiments according to the concept of the present invention to a specific form of disclosure, it should be understood to include all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

In the present specification, terms such as first and/or second are used only for the purpose of distinguishing one component from other components. That is, it is not intended to limit the components by the terms.

Components, features, and steps referred to in the present specification as'comprise' means the existence of the corresponding components, features, and steps, and is intended to exclude one or more other components, features, steps, and equivalents thereof. This is not.

Unless otherwise specified and stated in the singular form in the specification, plural forms are included. That is, the components and the like mentioned in the present specification may mean the presence or addition of one or more other components.

Unless otherwise defined, all terms used in this specification, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. to be.

That is, terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings of the context of the related technology, and should be interpreted as ideal or excessively formal meanings unless explicitly defined in this specification. It doesn't work.

Hereinafter, the present invention will be described in detail by describing a preferred embodiment of the present invention with reference to the accompanying drawings.

1 is a diagram illustrating an apparatus for recognizing an action according to an embodiment of the present invention.

Referring to FIG. 1, the behavior recognition apparatus 10 according to an embodiment of the present invention includes an input module 100, a database unit 200, a feature extraction module 300, a verification module 400, and a training module 500. ) Can be included.

The behavior recognition apparatus 10 may recognize the behavior of an object from various images including real-time images using a convolutional neural network (CNN). The convolutional neural network may be a set of algorithms that extract various information from an image and recognize an object's behavior from an image based on the extracted information.

The behavior recognition device 10 may be an imaging device such as a camera, CCTV, or black box, but is not limited thereto, and may include an imaging device or a computing device capable of receiving an image by communicating with an imaging device. For example, the behavior recognition device 10 may be a smart phone, a tablet PC, a PC, a smart TV, a mobile phone, a navigation device, an IoT device, a home appliance, or the like.

The input module 100 may receive an analysis target image and a reference image.

The analysis target image is an image input from the outside of the behavior recognition device 10, and is an image in which the behavior recognition device 10 attempts to recognize the behavior of an object acting inside the image. The image to be analyzed is, for example, input in real time to the input module 100 through a photographing camera of the behavior recognition device 10, or by using the input module 100 of the behavior recognition device 10 as a pre-made video clip. Can be entered through.

The reference image may be compared with the image to be analyzed to perform a role of allowing the behavior recognition device 10 to recognize a behavior of an object acting within the image to be analyzed. The reference image may be input to the input module 100 from the database unit 200 in which reference images of various categories are stored.

As described above, the database unit 200 is a type of storage space in which reference images of various categories can be stored.

The database unit 200 may transmit a reference image to the input module 100 and may receive a reference image of a new category from the outside. The user may add a new reference image by inputting the reference image to the database unit 200.

In the conventional case, in order to add a new classification target behavior to the behavior recognition apparatus, a plurality of image data for the classification target behavior had to be input and learned. However, the behavior recognition apparatus 10 according to an embodiment of the present invention, It is possible to classify as a new classification target behavior by adding a reference image of a new category to the database unit 200 and calculating a similarity to an image to be analyzed by extracting features of the reference image. In other words, re-learning and structural changes of the behavior recognition method are not required, and classification is possible even if the behavior has not been previously learned.

The feature extraction module 300 may extract the received features of the analysis target image and the features of the reference image in a frame unit, and extract core features among the extracted features.

The feature extraction module 300 may receive an analysis target image and a reference image received by the input module 100 from the input module 100. The feature extraction module 300 may use an artificial neural network model to extract features from an analysis target image and a reference image.

For example, the feature extraction module 300 may use a deep neural network model such as DenseNET. The deep neural network model may have a structure including an input layer, a convolution layer, a pooling layer, and an output layer. In the case of using a deep neural network model such as DenseNET, low-dimensional features can be better preserved by reducing the gradient loss problem that occurs during feature extraction and by accumulating the features extracted in the extraction process in a chain. By reducing the number, the amount of computation can be reduced, and high learning performance can be achieved even with relatively little data.

2 is a diagram illustrating a process of extracting features by the behavior recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 2, it can be seen that the behavior recognition apparatus 10 extracts features of an input image in units of frames using a deep neural network model.

The feature extraction module 300 may extract features in a frame unit from the received image. That is, as many features as the number of frames included in each image can be extracted. Since the number of frames included for each input image is different, the number of extracted features also varies for each input image.

One frame of an image is input to a deep neural network model, and a plurality of feature maps may be generated through a convolutional layer and a pooling layer. Feature maps such as size among the feature maps are blocked, and output values of feature maps of all layers disposed before the feature map of each layer included in one block may be input.

The deep neural network model can be used to extract features from the input image data. The deep neural network model may include a plurality of layers and blocks. Each of the layers may receive input data and may process the input data to generate output data. That is, the deep neural network model can generate a feature map by convolving the frame of the input image through the convolution layer, and sample it through the pooling layer, and the feature map of each layer included in one block is The features of the image can be specified by receiving the output values of the feature maps of all the previously arranged layers and accumulating the features.

The feature map arranged in the lower layer is a feature map that is output at the beginning of the feature extraction process, and a feature of a low level, for example, a feature such as a gradient, may be extracted. The feature maps arranged in the upper layer accumulate information from the feature maps of the lower layer to extract more complex features than the feature maps of the lower layer, such as human facial expressions and silhouettes.

Conventionally, the feature map received only the output value of the feature map of the lower layer adjacent thereto. However, in the behavior recognition apparatus 10 according to an embodiment of the present invention, the feature map of each layer is a feature map of all the previously placed layers. Since the output value can be repeatedly input from, the feature is further specified by passing through the layers and blocks of the deep neural network model in the extraction process, and information loss due to the specification of the feature can be prevented. Therefore, more sophisticated feature extraction than the prior art is possible.

An algorithm for extracting features from a frame of an input image may be expressed as Equation 1 below.

는 특징 추출 함수,

는 영상의 i번째 프레임,

는 특징 추출 함수의 파라미터,

는 영상의 i번째 프레임으로부터 추출된 특징을 의미한다.here

Is the feature extraction function,

Is the i-th frame of the image,

Is the parameter of the feature extraction function,

Denotes a feature extracted from the i-th frame of the image.

The feature extraction module 300 may extract core features from features extracted from a frame of an image. In order to reduce the amount of computation of the behavior recognition apparatus 10 and increase memory efficiency, core features may be extracted from features extracted from all frames of an input image. The feature extraction module 300 may extract core features, which are some of the features extracted using a K-means clustering algorithm.

FIG. 3 is a diagram exemplarily illustrating extraction of core features, which are some of the features extracted in units of frames, by the behavior recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 3, it can be seen that core features are extracted among features extracted in units of frames of an image.

The feature extraction module 300 may extract core features from among features extracted using a K-means clustering algorithm. The K-means clustering algorithm can be expressed as Equation 2 below.

는 영상의 j번째 프레임으로부터 추출된 특징,

는 K-평균 클러스터링 centroid의 집합을 이루는 핵심 특징을 의미한다(

).Where argmin is the argmin function,

Is the feature extracted from the j-th frame of the image,

Denotes the key features that make up the set of K-means clustering centroids (

).

로서 귀결된다.That is, the features finally extracted from the input image

It boils down to.

Core features extracted using the K-means clustering algorithm may be stored in a first storage unit (not shown) of the feature extraction module 300. The first storage unit (not shown) may be, for example, a storage device such as a hard drive (HDD), but is not limited thereto. The extracted core features are stored, so that the stored data can be used if necessary in a later behavior recognition process.

The verification module 400 calculates the similarity between the extracted key features of the analysis target image and the extracted key features of the reference image, and outputs a reference image having the maximum similarity among the analyzed target image and the reference images whose similarity is calculated. can do.

The verification module 400 may receive the core features extracted by the feature extraction module 300 from the feature extraction module 300. The verification module 400 may calculate a similarity between the analysis target image and the reference image using the received core features. When calculating the similarity between the core features of the image to be analyzed and the core features of the reference image, the verification module 400 may calculate the similarity by selecting only core features belonging to a section in which an action occurs in the image among the extracted core features. .

FIG. 4 is a diagram exemplarily illustrating selecting only key features belonging to a section in which an action occurs in an image among key features extracted from a verification module of the behavior recognition apparatus according to an embodiment of the present invention.

Referring to FIG. 4, it can be seen that among the core features of the two images, only core features belonging to a section in which an action occurs in the image is considered when calculating the similarity.

In the process of extracting a core feature in the feature extraction module 300, time information of an image may be lost. In addition, since the number of frames is different for each image and the section in which the action occurs is different, it is difficult to directly compare the similarity. Accordingly, it is possible to calculate the similarity between the images by selecting only the core features belonging to the section in which the action occurred in the input image among the extracted core features.

In the matrix shown in FIG. 4, a portion marked with '1' means sections to be considered when calculating the similarity. That is, when calculating the similarity between images having different playback times, only the core features belonging to the section in which the action occurred in each image is selected and used for calculating the similarity.

The algorithm used when measuring the similarity in the verification module 400 may be expressed as Equation 3 below.

는 벡터의 크기를 계산하기 위한 함수,

는 Heaviside 함수,

는 유사도 수준에 대한 하이퍼파라미터(hyperparameter)를 의미한다.S is the similarity function,

Is a function for calculating the size of a vector,

Heaviside function,

Means a hyperparameter for the level of similarity.

The verification module 400 may output a reference image having a maximum similarity. The behavior recognition apparatus 10 according to an embodiment of the present invention may calculate a similarity between an analysis target image input from an external source and a plurality of reference images included in the database unit 200. That is, after calculating the similarity with respect to one reference image, the process of calculating the similarity with respect to other reference images stored in the database unit 200 may be repeatedly performed.

The calculated similarity may be stored in a second storage unit (not shown) of the verification module 400. For example, the second storage unit (not shown) may be a storage device such as a hard drive, but is not limited thereto. Since the calculated similarity is stored in the second storage unit (not shown), the stored data can be used if necessary in a later behavior recognition process.

The verification module 400 may calculate a similarity for each reference image, store it in a second storage unit (not shown), and output a reference image having a maximum similarity among the stored similarities. That is, the behavior of the object inside the image to be analyzed may be classified into a category of the reference image having the maximum similarity.

The algorithm for outputting the reference image having the maximum similarity by the verification module 400 may be expressed as Equation 4 below.

는 분석대상 영상에서 추출된 특징,

는 데이터베이스부의 i번째 기준 영상에서 추출된 특징을 의미한다.Here, argmax is an argmax function, i is an index of a reference image previously stored in the database unit, and S is a similarity function of Equation 3,

Is the feature extracted from the image to be analyzed,

Denotes a feature extracted from the i-th reference image of the database unit.

As described above, the behavior recognition apparatus 10 according to an embodiment of the present invention inputs a new reference image including the behavior into the behavior recognition apparatus 10 even in the case of a previously unlearned behavior, and from the reference image. It is possible to classify a new behavior without relearning the behavior recognition method through the process of extracting features and calculating the similarity between images. In addition, since features are extracted using a deep neural network model, more sophisticated feature extraction is possible.

5 is a diagram illustrating a training process of an apparatus for recognizing an action according to an embodiment of the present invention.

Referring to FIG. 5, after receiving a kind of training sample in the training process of the behavior recognition apparatus 10 according to an embodiment of the present invention, it may be confirmed that the behavior recognition method is trained through a specific algorithm.

The training module 500 may train a behavior recognition method based on a training sample received by the behavior recognition apparatus 10 according to an embodiment of the present invention.

The training module 500 includes an anchor video clip, a positive video clip, and a negative video clip, which are training samples previously input by the behavior recognition device 10 according to an embodiment of the present invention. After each feature is extracted from, the similarity between the extracted features is calculated, and the behavior recognition method can be trained using this.

The positive image refers to an image belonging to the same category as the anchor image, and the negative image refers to an image belonging to a different category from the anchor image. For example, when the anchor image includes an act of playing tennis, the positive image may also include an act of playing tennis. Conversely, the negative image may include an act belonging to a different category than the anchor image, for example, lifting a dumbbell.

After receiving a training sample, the behavior recognition apparatus 10 according to an embodiment of the present invention may extract features of images included in the training sample and calculate a similarity between the images. That is, the behavior recognition method can be trained by calculating the similarity between the anchor image and the positive image, and the anchor image and the negative image, and calculating a loss function using the calculated two similarities. The above training sample input, feature extraction, and similarity calculation process may be performed through the input module 100, the feature extraction module 300, and the verification module 400.

The algorithm used in the training process performed in the training module 500 may be expressed as Equation 5 below.

는 앵커 영상,

는 포지티브 영상,

은 네거티브 영상,

은 minimum dissimilarity를 의미한다.Where L is the loss function, max is the max function, S is the similarity function of Equation 3,

Is the anchor video,

The positive video,

Silver negative video,

Means minimum dissimilarity.

6 is a diagram illustrating a flow chart of a method for recognizing an action according to an embodiment of the present invention.

6, the behavior recognition method according to an embodiment of the present invention includes receiving an analysis target image and a first reference image from among a plurality of reference images (S602), features of the analysis target image and a first reference image. Extracting the features of the image in units of frames (S603), extracting the features of the extracted analysis target image and the core features that are some of the features of the first reference image (S604), and between the extracted core features It may include a step of calculating the similarity (S605). In addition, the step S606 may further include outputting a reference image having the largest similarity among the calculated similarities by repeating steps S603 to S605 with respect to the other reference images except the first reference image. In addition, it may further include the step of training the behavior recognition method (S601).

In the step of training the behavior recognition method (S601), a training sample, that is, an anchor image including behavior information, a positive image including behavior information identical to the anchor image, and a negative image including behavior information different from the anchor image, are input. In this step, the similarity between the anchor image and the positive image and the similarity between the anchor image and the negative image are calculated, and a loss function is calculated and trained using the calculated similarity.

As described above, after receiving the anchor, positive, and negative images, which are training samples, and extracting respective features, the similarity is calculated, and the training module 500 may train the behavior recognition method using the algorithm of Equation 5 above. Unlike the prior art, it is not necessary to relearn the behavior recognition method and change the structure in order to add a new classification target behavior after training.

In step S602 of receiving an analysis target image and a first reference image from among the plurality of reference images, an analysis target image and a first reference image are input to the input module 100 of the behavior recognition apparatus 10.

An image to be analyzed may be input from the outside, and refers to an image in which the behavior recognition device 10 attempts to recognize the behavior of an object included in the image. The first reference image may be input from the database unit 200 in which a plurality of reference images are previously stored, and compared with the analysis target image, the behavior recognition device 10 can recognize the behavior of the object acting inside the analysis target image. It can play a role of enabling.

In the step of extracting the features of the analysis target image and the features of the first reference image in frame units (S603), the features of each image are extracted from the analysis target image and the first reference image received by the input module 100 in a frame unit. This is the extraction step.

The feature extraction module 300 may receive an image input from the input module 100. The feature extraction module 300 may extract features in units of frames of each image using an artificial neural network, for example, a deep neural network model such as DenseNET. Feature maps can be generated from an input frame using a deep neural network model to extract features.

Each of the extracted features of the analysis target image and the core features, which are some of the features of the first reference image, is extracted by using a K-means clustering algorithm among the features of the image extracted through an artificial neural network. This is the step of extracting the key features of.

The feature extraction module 300 may extract features of each image using a deep neural network model, and then extract some of the core features using a K-means clustering algorithm among the extracted features.

The step of calculating the similarity between the extracted core features (S605) is a step of calculating the similarity between the core features extracted by the feature extraction module 300.

The verification module 400 may receive information on key features extracted by the feature extraction module 300. The verification module 400 may calculate a similarity between core features of each received image. Each image has a different section in which an action occurs, and in order to compensate for the loss of temporal information in the video during the core feature extraction process, the similarity can be calculated by selecting only the core features belonging to the section in which the action occurs.

The step (S606) of outputting a reference image having the largest similarity among the calculated similarities by repeating steps S603 to S605 for the rest of the reference images except the first reference image (S606) is a first reference previously stored in the database unit 200. After the input module 100 receives the reference images other than the image, steps S603 to S605 are repeated, and the reference image having the largest similarity among the calculated similarities is output.

The behavior recognition apparatus 10 according to an embodiment of the present invention calculates the similarity of each of the plurality of reference images stored in the database unit 200 with the analysis target image in order to recognize the behavior of the object inside the behavior analysis target image. Can be calculated. Therefore, after completing the calculation of the similarity between the analysis target image and the first reference image, the input module 100 receives the remaining reference images to calculate the similarity between the other reference image and the analysis target image, except for the first reference image, Can be performed repeatedly. After repeating the above steps, the verification module 400 may output a reference image having the largest similarity among the calculated similarities to classify the behavior of the object inside the image to be analyzed.

In the method of recognizing an action according to another embodiment of the present invention, the step of receiving an analysis target image and a first reference image from among a plurality of reference images (S602), and (S603), extracting features of the extracted image to be analyzed and core features that are some of the features of the first reference image (S604), and calculating similarity between the extracted core features ( S605), receiving a second reference image different from the first reference image among a plurality of reference images, calculating a similarity between the features of the analysis target image and the features of the second reference image, a first reference image And outputting a reference image having a higher calculated similarity among the second reference images, and calculating a similarity between one of the remaining images among a plurality of reference images and features of the analysis target image, thereby having the greatest similarity It may include the step of outputting a reference image having a.

The method for recognizing an action according to an embodiment of the present invention may be implemented in the form of a computer-readable recording medium in which a program to be executed on a computer is recorded. The computer-readable recording medium may be any available medium that can be accessed by a computer, and may include both volatile and nonvolatile media, and removable and non-removable media. Further, the computer-readable recording medium may include a computer storage medium. Computer storage media may include both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

Exemplary modules, steps, or a combination thereof according to the embodiments described herein may be performed by electronic hardware (digital design designed by coding, etc.), software (various types of applications including program instructions), or a combination thereof. Can be implemented. Which form of hardware and/or software is implemented may vary according to design constraints imposed on the user terminal.

One or more of the configurations described herein may be stored in a memory as computer program instructions, which computer program instructions may execute the methods described herein centered on a digital signal processor. Connection examples between configurations specified with reference to the accompanying drawings in the present specification are merely exemplary, and at least some of them may be omitted, and conversely, not only these configurations but also additional configurations may be further included.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to describe it, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

10: behavior recognition device
100: input module
200: database unit
300: feature extraction module
400: verification module
500: training module

Claims (17)

(a) receiving an analysis target image and a first reference image from among a plurality of reference images;
(b) extracting features of the analysis target image and features of the first reference image in a frame unit using an artificial neural network;
(c) extracting features of the extracted image to be analyzed and core features that are some of the features of the first reference image; And
(d) calculating a degree of similarity between the extracted core features,
How to recognize behavior. The method of claim 1,
After step (d),
(e) receiving a second reference image different from the first reference image among the plurality of reference images;
(f) calculating a similarity between the features of the analysis target image and the features of the second reference image: and
(g) further comprising the step of outputting a reference image having a larger calculated similarity among the first and second reference images,
How to recognize behavior. The method of claim 2,
After the step (g),
The method further comprising calculating a similarity between one of the remaining images among the plurality of reference images and features of the analysis target image, and outputting a reference image having the greatest similarity,
How to recognize behavior. The method of claim 1,
After step (d),
The step of repeating steps (b), (c) and (d) with respect to the other reference images excluding the first reference image, further comprising outputting a reference image having the largest similarity among the calculated similarities,
How to recognize behavior. The method of claim 1,
Before step (a),
It further includes a training step,
The training step,
Receiving an anchor image including behavior information, a positive image including behavior information identical to the anchor image, and a negative image including behavior information different from the anchor image;
Calculating a similarity between the anchor image and the positive image and a similarity between the anchor image and the negative image; And
Computing a loss function using the calculated similarities,
How to recognize behavior. The method of claim 1,
In step (a),
The analysis target image is input from the outside,
The reference image is input from a database unit in which the reference image is previously stored,
How to recognize behavior. The method of claim 1,
The step (b),
Generating a plurality of feature maps for each frame;
Blocking feature maps having the same size among the feature maps; And
Including the step of extracting features by receiving the output values of the feature maps of all the layers arranged before the feature map of each layer included in one block,
How to recognize behavior. The method of claim 1,
The step (c),
Including the step of extracting key features using a K-means clustering (K-means clustering) algorithm,
How to recognize behavior. The method of claim 1,
The step (d),
Comprising the step of calculating a similarity by selecting only the core features belonging to the section in which the action occurred in the analysis target image and the first reference image among the extracted key features,
How to recognize behavior. A computer-readable recording medium recording a program for performing the method according to any one of claims 1 to 9 on a computer. An input module for receiving an analysis target image and a reference image;
A feature extraction module for extracting features of the received analysis target image and features of the input reference image in a frame unit using an artificial neural network, and extracting core features, which are some of the extracted features, respectively; And
Comprising a verification module that calculates the similarity between the extracted core features, and outputs a reference image having the largest similarity among the reference images for which the similarity is calculated,
Behavior recognition device. The method of claim 11,
Further comprising a database unit for storing the reference image,
Behavior recognition device. The method of claim 11,
The anchor image and the positive calculated from an anchor image including behavior information, a positive image including behavior information identical to the anchor image, and a negative image including behavior information different from the anchor image Receiving the similarity between the images and the similarity between the anchor image and the negative image,
Further comprising a training module for training the behavior recognition method of claim 1 by calculating a loss function using the received similarities,
Behavior recognition device. The method of claim 11,
The input module,
Receiving the image to be analyzed from the outside,
Receiving the reference image from the database unit,
Behavior recognition device. The method of claim 11,
The feature extraction module,
All layers in which a plurality of feature maps are generated for each frame, feature maps having the same size among the feature maps are blocked, and feature maps of each layer included in one block are placed before it To extract features by receiving the output values of the feature maps of,
Behavior recognition device. The method of claim 11,
The feature extraction module,
Extracting key features using K-means clustering algorithm,
Behavior recognition device. The method of claim 11,
The verification module,
Among the extracted core features, calculating similarity by selecting only core features belonging to a section in which an action occurs in the analysis target image and the reference image,
Behavior recognition device.

Images