KR20230017126A - Action recognition system based on deep learning and the method thereof - Google Patents

Abstract

A deep learning-based behavior recognition system according to an embodiment of the present invention comprises: a data preprocessing unit that groups overlapping motion classes in a video image data set, extracts frames from each video image, adjusts the frames to a frame size suitable for a deep learning pipeline (DLPL) CNN learning model, and pre-trains the deep learning pipe line (DLPL) CNN learning model; a transfer learning unit that trains and fine-tunes the deep learning pipeline (DLPL) CNN learning model pre-trained in the data preprocessing unit by applying an additional data set; a deep feature extraction unit that extracts high-dimensional deep features by learning frame-level spatial information from a visual data stream by applying the pre-trained deep learning pipeline (DLPL) CNN learning model fine-tuned in the transfer learning unit; an encoder unit that compresses the high-dimensional deep features extracted from the deep feature extraction unit into a low-dimensional feature map; and an adjustment module unit that learns temporal information from the feature map compressed in the encoder unit and repeatedly fine-tunes and trains the model by applying a changed part of novel video image data to the previously trained model. Accordingly, action recognition errors can be reduced through behavioral analysis that can accurately differentiate similar human actions and postures.

Description

Deep learning-based action recognition system and method {ACTION RECOGNITION SYSTEM BASED ON DEEP LEARNING AND THE METHOD THEREOF}

The present invention relates to a deep learning-based action recognition system and method, and in particular, provides accurate differentiation of action recognition by extracting and learning in-depth features through a spatio-temporal framework for similar or partially overlapping action recognition in long image data. It is about a deep learning-based action recognition system and method that can

Generally, human action recognition tasks involve identifying various activities performed by humans in a video data stream. Human action recognition mainly focuses on questions such as 'what action is detected and recognized' or 'where is the action performed in the video data stream', and the video data stream contains spatial and temporal information of the action A lot of hidden information helpful for motion recognition is extracted, including spatio-temporal information and their relationships.

Human action recognition can be divided into several categories such as human-to-human actions, human-object interactions, human body movements, sub-activities or atomic actions, gestures, group activities, events and actions. Atomic motions or activities such as walking and running are less complex and easier to recognize. However, activities such as cooking are complex and difficult to recognize as a combination of many sub-activities.

In other words, most video data currently contains a lot of hidden information and patterns that can be used for HAR (Human Action Recognition). HAR can be applied in many areas such as behavior analysis, intelligent video surveillance, robot vision, etc. For example, some of the applications can be applied to fraud detection, suspicious or abnormal behavior detection, elderly behavior monitoring, intelligent video surveillance, human-computer interaction systems and robot vision.

However, occlusion, perspective changes and lighting present problems that make HAR work more difficult. In addition, similar tasks or some overlapping parts exist in some task classes, so there are many problems that have not been resolved as the main cause that contributes the most to misclassification.

In order to solve the aforementioned problem, a framework is extracted and learned from overlapping behavioral patterns in long image data using a CNN algorithm, and fine-tuning is performed using a pre-trained deep learning pipeline (DLPL) model. To provide a deep learning-based behavior recognition system and its method.

A deep learning-based action recognition system according to an embodiment of the present invention groups overlapping action classes in a data set of video images, and extracts frames from each video image to fit a Deep Learning Pipeline (DLPL) CNN learning model. A data pre-processing unit for pre-learning by adjusting the frame size; a transfer learning unit for learning and fine-tuning by applying an additional data set to a deep learning pipeline (DLPL) CNN learning model previously trained in the data pre-processing unit; a deep feature extraction unit for extracting high-dimensional deep features by learning frame-level spatial information from a visual data stream by applying a pretrained deep learning pipeline (DLPL) CNN learning model finely tuned in the transfer learning unit; an encoder unit for compressing the high-dimensional deep features extracted by the deep feature extraction unit into a low-dimensional feature map; and an adjustment module unit that learns temporal information from the feature map compressed by the encoder unit and repeatedly fine-tunes and learns by applying a modified part of new video image data to a previously learned model.

Here, in particular, the deep learning pipeline (DLPL) CNN learning model includes DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19 and Xception, and uses any one of these pre-trained CNN learning models. there is.

In particular, the transfer learning unit removes the last classification layer of the pre-trained deep learning pipeline (DLPL) CNN model and adds a new layer for a new data set to learn and refine.

Here, in particular, the deep feature extraction unit removes the last fully connected layer (FCL) from the Deep Learning Pipeline (DLPL) CNN training model and adds some additional layers to learn spatial patterns and relationships in the visual data stream for the data set, , finally, the output before the fully connected SoftMax layer is provided to the next network to extract deep features.

In particular, the encoder unit is characterized in that it learns a minimum representation of the input data using a deep autoencoder, reconstructs the original input data into an output closest to the original input data, and outputs it as a low-dimensional feature map.

In particular, the adjustment module unit is characterized in that it uses Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN) to learn a long-term temporal context.

In addition, the deep learning-based action recognition method according to an embodiment of the present invention groups overlapping action classes in a data set of video images, extracts frames from each video image, and uses a Deep Learning Pipeline (DLPL) CNN learning model. pre-learning by adjusting the frame size to fit; learning and fine-tuning by applying an additional data set to the pre-trained Deep Learning Pipeline (DLPL) CNN learning model; extracting high-dimensional deep features by learning frame-level spatial information from a visual data stream by applying the fine-tuned pretrained deep learning pipeline (DLPL) CNN learning model; compressing the high-dimensional deep features extracted by the deep feature extraction unit into a low-dimensional feature map; and learning time information from the compressed feature map and repeatedly fine-tuning and learning by applying a modified part of new video image data to a previously learned model.

Here, in particular, the deep learning pipeline (DLPL) CNN learning model includes DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19 and Xception, and uses any one of these pre-trained CNN learning models. there is.

Here, in particular, in the learning and fine-tuning step, the last classification layer of the pretrained deep learning pipeline (DLPL) CNN model is removed, and a new layer is added for a new data set to learn and fine-tune. there is.

Here, in particular, the step of extracting the deep features spatially from the visual data stream for the data set by removing the last Fully Connected Layer (FCL) from the pre-trained Deep Learning Pipeline (DLPL) CNN training model and adding some additional layers. It is characterized by learning patterns and relationships, and finally extracting deep features by providing the output before the fully connected SoftMax layer to the next network.

Here, in particular, in the step of compressing the low-dimensional feature map, the minimum expression for the input data is learned using a deep autoencoder, and the output closest to the original input data is reconstructed and output as a low-dimensional feature map. has its characteristics.

Here, in particular, in the step of fine-tuning and learning, a long short-term memory (LSTM) and a recurrent neural network (RNN) are used to learn a long-term temporal context.

According to an embodiment disclosed in the present invention, there is an effect of reducing behavioral recognition errors through behavioral analysis that can accurately differentiate them from similar behaviors and postures of human behavior.

1 is a diagram schematically showing the configuration of a deep learning-based behavior recognition system according to an embodiment of the present invention.
2 schematically illustrates a spatial-temporal human action recognition (HAR) framework according to an embodiment of the present invention;
3 schematically shows the architecture of a tip autoencoder of the present invention.
Figure 4 schematically shows the architecture of the RNN model of the present invention.
5 schematically illustrates the architecture of the LSTM model of the present invention;
6 is a diagram illustrating a process of iterative fine-tuning for applying changes to a new video in the adjustment module unit of the present invention;
7 is a flowchart of a deep learning-based behavior recognition method according to an embodiment of the present invention.

Since the present invention can have various changes and various embodiments, specific embodiments will be described in detail with reference to the drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. Like reference numerals have been used for like elements throughout the description of each figure.

Terms such as first, second, A, and B may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The term and/or includes a combination of a plurality of related items or any one of a plurality of related items.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Throughout the specification and claims, when a part includes a certain component, it means that it may further include other components, not excluding other components unless otherwise stated.

1 is a diagram schematically showing the configuration of a deep learning-based action recognition system according to an embodiment of the present invention, and FIG. 2 is a schematic diagram of a space-time human action recognition (HAR) framework according to an embodiment of the present invention. It is a drawing shown as

As shown in FIG. 1, the deep learning-based behavior recognition system 100 includes a data pre-processing unit 110, a transfer learning unit 120, a deep feature extraction unit 130, an encoder unit 140, and an adjustment module unit. (150).

As shown in FIG. 2, the data pre-processing unit 110 groups overlapping motion classes in a data set of video images and extracts frames from each video image to fit a Deep Learning Pipeline (DLPL) CNN learning model. Pre-learning is performed by adjusting the frame size.

More specifically, the Deep Learning Pipeline (DLPL) CNN training model may include DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19, and Xception, and pre-learning is performed using them.

For example, data pre-processing can group work classes that have similar parts or some overlapping jobs in the UCF-101 data set. For the UCF-101 dataset, the total number of groups containing nested actions is 18, and each group can consist of action classes ranging from 2 to 5. After grouping the overlapping motion classes, frames are extracted from each video and then reconstructed according to the shape of the input required for each of the seven pre-trained CNN models. At this time, the frame may be readjusted to dimensions 224, 224, 3. For InceptionV3 and Xception among Deep Learning Pipeline (DLPL) CNN training models, frames can be rescaled to dimension (299, 299, 3) for DenseNet201, ResNet101V2, ResNet152V2, VGG16 and VGG19 pre-trained CNN models.

The transfer learning unit 120 learns and fine-tunes by applying an additional data set to the deep learning pipeline (DLPL) CNN training model previously trained in the data preprocessing unit 110.

More specifically, the transfer learning unit 120 removes the last classification layer of the pretrained deep learning pipeline (DLPL) CNN model and adds a new layer to a new data set to learn and fine-tune.

In other words, transfer learning is a technique in machine learning that learns useful information and hidden patterns in one specific domain and then uses them to pass them on to other related domains. can be used for further generalization. Here, training a model for one task can be a starting point for training other models for related tasks, and fine-tuning these pre-trained models in transfer learning is very fast with less computation compared to training a model from scratch. . That said, another benefit of using transfer learning is that the pretrained model has already been trained on very large data sets using many optimization strategies over a long period of time, requiring only a small amount of data. Among many other strategies using transfer learning, one of them is to remove the last classification layer, add a new layer, and then train and fine-tune the pretrained model and additional layers added at the end of the new dataset.

The deep feature extraction unit 130 learns frame-level spatial information from the visual data stream by applying the pre-trained deep learning pipeline (DLPL) CNN training model finely tuned in the transfer learning unit 120 to obtain high-dimensional deep features. will extract In other words, the deep feature extraction unit 130 removes the last fully connected layer (FCL) from the Deep Learning Pipeline (DLPL) CNN training model and adds some additional layers to determine spatial patterns and spatial patterns in the visual data stream for the data set. The relationship is learned, and finally the output before the fully connected SoftMax layer is fed to the next network to extract deep features.

More specifically, video data contains an enormous amount of useful information hidden in the visual data stream. Here, color intensity changes, edges and shapes, motion and texture patterns, and long-term temporal context information are included.

Therefore, in order to take advantage of transfer learning using the above information, we use 7 pre-trained CNN models: DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19 and Xception.

These CNN models are used to extract deep features at the frame level learning spatial patterns and relationships in visual data streams. Additionally, a CNN model can be trained on a data set of about 1.2 million color images for 1000 different classes.

Here, the deep feature in the deep feature extractor 130 is a feature of a high-dimensional space representing both local and global features of spatial information in a video frame.

At the end of the pre-trained CNN model for extracting these deep features, we remove the last few fully connected layers (FCL) and add some additional layers so that the model can be trained on the data set. Then, after adding a layer at the end of the network, the pre-trained CNN model can be fine-tuned and used for motion recognition tasks in videos.

In addition, in order to extract deep features in the deep feature extraction unit 130, an output before the last completely connected SoftMax layer may be captured and provided to the next network for expression learning. For example, for a (5 x 5) kernel-sized convolutional layer, the output is of dimension (150 x 150) for 200 feature maps and the same pooling with a stride value of 1 can be used. For an image of size (150, 100, 3) and an RGB image with 3 color channels, the overall parameters could be (5 x 5 x 3 + 1) x 200 = 15,200.

All 200 feature maps consist of (150 x 100) neurons, and with (5 x 5 x 3) = 73 inputs, the sum of the weights for all neurons is calculated. For 32-bit floating point values, for each instance in the data set, total (200 x 150 x 100 x 32) = 96 million bits (12 MB). During the prediction phase, one layer releases RAM when the calculations for the next layer are complete, but for training purposes, all calculations performed in the forward pass are retained for the backward pass.

Meanwhile, in order to detect the location of a specific object in a 2D image, a weight function can be expressed as Equation 1 below.

[Equation 1]

where () represents the location of time and () is the convolution kernel. The position of an object can be calculated with a simple moving average (SMA) using convolution. The kernel for SMA can be expressed as Equation 2 below.

[Equation 2]

Here, the window size is a simple moving average. A complex image of a 2D image can be calculated as in Equation 3 below.

[Equation 3]

Here, is a 2D image on which convolution is performed and is a 2D smoothing kernel. The output of a single neuron in the convolutional layer can be calculated as in Equation 4 below.

[Equation 4]

Here, for the feature map, i of the convolutional layer is i _, , , with ^th rows and ^th columns. is a vertical stride, is a horizontal stride, is a field, and is a field. is the width of the receiving field. Also, is the number of the previous layer of the feature map, , _, is the neuron output of the previous layer for i ^th row and ^th and feature map or channel. Bias is denoted by _, , , are denoted by the connection weights between neurons in feature map and feature map k for the rows and columns of layer .

Figure 3 is a diagram schematically showing the architecture of the tip autoencoder of the present invention, Figure 4 is a diagram schematically showing the architecture of the RNN model of the present invention, Figure 5 is a schematic diagram of the architecture of the LSTM model of the present invention FIG. 6 is a diagram showing a process of iterative fine adjustment to apply changes to a new video in the adjustment module unit of the present invention.

The encoder unit 140 compresses the high-dimensional deep features extracted by the deep feature extraction unit into a low-dimensional feature map. Here, the encoder unit 140 learns a minimum representation of the input data using a deep autoencoder, reconstructs the original input data into an output closest to the original input data, and outputs the low-dimensional feature map.

More specifically, as shown in FIG. 3 , the encoder unit 140 may be a deep autoencoder, which is a neural network capable of reconstructing an output as close to an original input as possible based on an expression of input data. Here, the reconstruction process of the encoder unit 140 proceeds according to a learning process rather than a storage process that generates an output from input data in an unsupervised manner. That is, it can be composed of a neural network of this architecture with multiple hidden layers.

The encoder unit 140 may be composed of two parts as an autoencoder, and the encoder unit 140 serves to learn the smallest representation of input data that can be additionally used by the decoder, and is generated by the decoder. With the help of an internal representation, the output closest to the original input data is reconstructed. An internal representation generated by the encoder in the encoder unit 140 is defined as a latent space, and the latent space includes a probability distribution generated by the encoder supplied to the decoder. Here, the goal of the decoder is to generate output data in the latent space and generate samples from a probability distribution close to the original input, enabling the use of autoencoders for generative modeling. In general, the latent space will contain useful features in dimensions smaller than the original input. Learning these small dimensions allows us to use them to obtain the most salient features given the training data.

The adjustment module unit 150 learns time information from the feature map compressed by the encoder unit 140 and repeatedly fine-tunes and learns by applying a modified part of new video image data to a previously learned model. Here, the adjustment module unit 150 uses Long Short-Term Memory (LSTM) and Recurrent Neural Network (RNN) to learn the long-term temporal context.

More specifically, as shown in FIG. 4 , the recurrent neural network (RNN) model of the adjustment module unit 150 is a neural network and uses previous data to learn time information from a frame sequence. Here, the output of the previous step is passed to the next step, and the 'memory' of the RNN keeps track of the previously learned content. At this time, the parameters are less complex as we perform similar operations for each input in all hidden layers using the same parameters, and we keep the weights and biases the same for all layers, resulting in transforming independent activations into dependent activations. can Then, you enter the network one step at a time. The current state is computed using both the current input and the previous network state. The current internal state, called the hidden state, becomes 1 at the next time step. Compute the output to the last state after competing similarly for all time steps and then compare it to the original output to calculate the error rate. This error is back-propagated into the network to update the weights and biases for the next iteration. Through the whole mechanism of the RNN described above, it is possible to learn temporal features from the frame sequence of the visual data stream.

In addition, as shown in FIG. 5 , the Long Short-Term Memory (LSTM) model of the adjustment module unit 150 is an algorithm model for solving a gradient loss or exploding gradient problem in an RNN model. LSTM iterations will keep track of previously learned knowledge and exclude irrelevant data. A vector named internal cell state is then preserved by every LSTM repeat unit. LSTM uses a four-gate forget-gate that serves to 'forget' the previous amount of data using the sigmoid activation function. The input-gate or update-gate stores the internal cell state and uses the sigmoid activation function to account for the amount of information. The inputmodulation-gate or relevance plays a role of modulating the information on which the input-gate operates by using the Hyperbolic-Tangent activation function. The output gate. plays a role in generating an output from the internal cell state and uses a sigmoid activation function.

And, the adjustment module unit 150 performs fine adjustment through an iterative learning model for a new data stream.

Specifically, as shown in Figure 6, the fine-tuning module architecture is such that due to the changing nature of video data over time, previously trained models cannot adopt changes in video data and work on new data streams. cannot be learned correctly. Therefore, the framework recognises changes in the surrounding environment by repeatedly updating itself with new data. At this time, in order to repeatedly retrain the learning model, a data stream having a predicted reliability score greater than a specific threshold may be used to adopt various conditions in the surrounding environment. Deployment settings and user requirement thresholds can be selected to account for environmental impact. Once enough data is collected, the previously trained model is fine-tuned so that the model adapts to changes in the environment of the new data. The trained model can then be applied to many other areas, such as monitoring patient activity in video for real-time surveillance, and detecting fraud and anomalous behavior.

7 is a flowchart of a deep learning-based behavior recognition method according to an embodiment of the present invention.

As shown in FIG. 7, the deep learning-based action recognition method according to an embodiment of the present invention first groups overlapping action classes in a data set of video images, extracts frames from each video image, and deep learning pipe A step of pre-learning by adjusting the frame size suitable for the line (DLPL) CNN learning model is performed (S710).

Here, the deep learning pipeline (DLPL) CNN learning model includes DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19, and Xception, and it is preferable to use any one of these pre-trained CNN learning models.

For example, data pre-processing can group work classes that have similar parts or some overlapping jobs in the UCF-101 data set. For the UCF-101 dataset, the total number of groups containing nested actions is 18, and each group can consist of action classes ranging from 2 to 5. After grouping the overlapping motion classes, frames are extracted from each video and then reconstructed according to the shape of the input required for each of the seven pre-trained CNN models. At this time, the frame may be readjusted to dimensions 224, 224, 3. For InceptionV3 and Xception among Deep Learning Pipeline (DLPL) CNN training models, frames can be rescaled to dimension (299, 299, 3) for DenseNet201, ResNet101V2, ResNet152V2, VGG16 and VGG19 pre-trained CNN models.

Then, transfer learning and fine-tuning are performed by applying an additional data set to the pretrained deep learning pipeline (DLPL) CNN learning model (S720).

More specifically, the transfer learning and fine-tuning step removes the last classification layer of the pre-trained Deep Learning Pipeline (DLPL) CNN model and adds a new layer for a new data set to learn and fine-tune it.

In other words, transfer learning is a technique in machine learning that learns useful information and hidden patterns in one specific domain and then uses them to pass them on to other related domains. can be used for further generalization. Here, training a model for one task can be a starting point for training other models for related tasks, and fine-tuning these pre-trained models in transfer learning is very fast with less computation compared to training a model from scratch. . That said, another benefit of using transfer learning is that the pretrained model has already been trained on very large data sets using many optimization strategies over a long period of time, requiring only a small amount of data. Among many other strategies using transfer learning, one of them is to remove the last classification layer, add a new layer, and then train and fine-tune the pretrained model and additional layers added at the end of the new dataset.

Subsequently, a step of extracting high-dimensional deep features by learning frame-level spatial information from the visual data stream by applying the finely tuned pretrained deep learning pipeline (DLPL) CNN learning model is performed (S730). Here, deep feature extraction removes the last Fully Connected Layer (FCL) from the above pre-trained Deep Learning Pipeline (DLPL) CNN training model and adds some additional layers to determine spatial patterns and relationships in visual data streams for a dataset. Finally, the output before the fully connected SoftMax layer is fed to the next network to extract deep features.

These CNN models are used to extract deep features at the frame level learning spatial patterns and relationships in visual data streams. Additionally, a CNN model can be trained on a data set of about 1.2 million color images for 1000 different classes.

Here, the deep feature is a feature of a high-dimensional space representing both local and global features of spatial information in a video frame.

At the end of the pre-trained CNN model for extracting these deep features, we remove the last few fully connected layers (FCL) and add some additional layers so that the model can be trained on the data set. Then, after adding a layer at the end of the network, the pre-trained CNN model can be fine-tuned and used for motion recognition tasks in videos.

Next, a step of compressing the extracted high-dimensional deep features into a low-dimensional feature map is performed (S740). Here, in the step of compressing into the low-dimensional feature map, a minimum representation of the input data is learned using a deep autoencoder, and an output closest to the original input data is reconstructed and output as a low-dimensional feature map.

More specifically, a deep autoencoder, which is a neural network capable of reconstructing an output as close as possible to an original input by leaning on the expression of input data, can be applied to output as a low-dimensional feature map. Here, the encoder reconstruction process proceeds according to a learning process rather than a memorization process that generates an output from input data in an unsupervised manner. That is, it can be composed of a neural network of this architecture with multiple hidden layers.

Such a deep autoencoder can consist of two parts, one serving to learn the smallest representation of the input data that is further usable by the decoder, and the output closest to the original input data with the help of the internal representation produced by the decoder. will reconstruct The internal representation produced by the encoder is defined as the latent space, and the latent space contains the probability distribution generated by the encoder that is fed to the decoder. Here, the goal of the decoder is to generate output data in the latent space and generate samples from a probability distribution close to the original input, enabling the use of autoencoders for generative modeling. In general, the latent space will contain useful features in dimensions smaller than the original input. Learning these small dimensions allows us to use them to obtain the most salient features given the training data.

Subsequently, a step of learning time information from the compressed feature map and repeatedly fine-tuning and learning by applying a modified part of new video image data to a previously learned model is performed (S750). Here, long short-term memory (LSTM) and recurrent neural network (RNN) are used to learn a long-term temporal context through the fine-tuning learning.

A Recurrent Neural Network (RNN) model is a neural network that uses previous data to learn temporal information from a sequence of frames. Here, the output of the previous step is passed to the next step, and the 'memory' of the RNN keeps track of the previously learned content. At this time, the parameters are less complex as we perform similar operations for each input in all hidden layers using the same parameters, and we keep the weights and biases the same for all layers, resulting in transforming independent activations into dependent activations. can Then, you enter the network one step at a time. The current state is computed using both the current input and the previous network state. The current internal state, called the hidden state, becomes 1 at the next time step. Compute the output to the last state after competing similarly for all time steps and then compare it to the original output to calculate the error rate. This error is back-propagated into the network to update the weights and biases for the next iteration. Through the whole mechanism of the RNN described above, it is possible to learn temporal features from the frame sequence of the visual data stream.

In addition, the LSTM (Long Short-Term Memory) model is an algorithm model for solving a gradient loss or exploding gradient problem in an RNN model. LSTM iterations will keep track of previously learned knowledge and exclude irrelevant data. A vector named internal cell state is then preserved by every LSTM repeat unit. LSTM uses a four-gate forget-gate that serves to 'forget' the previous amount of data using the sigmoid activation function. The input-gate or update-gate stores the internal cell state and uses the sigmoid activation function to account for the amount of information. The inputmodulation-gate or relevance plays a role of modulating the information on which the input-gate operates by using the Hyperbolic-Tangent activation function. The output gate. plays a role in generating an output from the internal cell state and uses a sigmoid activation function.

It then proceeds to fine-tune through iterative learning models for new data streams.

Specifically, the trained model for fine-tuning is concerned that, due to the changing nature of video data over time, a previously trained model will not be able to adopt changes in the video data and will not be able to correctly learn tasks from new data streams. do. Therefore, the framework recognises changes in the surrounding environment by repeatedly updating itself with new data. At this time, in order to repeatedly retrain the learning model, a data stream having a predicted reliability score greater than a specific threshold may be used to adopt various conditions in the surrounding environment. Deployment settings and user requirement thresholds can be selected to account for environmental impact. Once enough data is collected, the previously trained model is fine-tuned so that the model adapts to changes in the environment of the new data. The trained model can then be applied to many other areas, such as monitoring patient activity in video for real-time surveillance, and detecting fraud and anomalous behavior.

An embodiment of the present invention may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, or other transport mechanism, and includes any information delivery media.

Although the methods and systems of the present invention have been described with reference to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

110: data pre-processing unit
120: transfer learning unit
130: deep feature extraction unit
140: encoder unit
150: adjustment module unit

Claims (13)

A data pre-processing unit that groups overlapping motion classes in a data set of video images, extracts frames from each video image, and adjusts the frame size to fit a deep learning pipeline (DLPL) CNN learning model for pre-learning;
a transfer learning unit for learning and fine-tuning by applying an additional data set to a deep learning pipeline (DLPL) CNN learning model previously trained in the data pre-processing unit;
a deep feature extraction unit for extracting high-dimensional deep features by learning frame-level spatial information from a visual data stream by applying a pretrained deep learning pipeline (DLPL) CNN learning model finely tuned in the transfer learning unit;
an encoder unit for compressing the high-dimensional deep features extracted by the deep feature extraction unit into a low-dimensional feature map; and
A deep learning-based action recognition system comprising an adjustment module unit that learns time information from the feature map compressed by the encoder unit and repeatedly fine-tunes and learns by applying a modified part of new video image data to a previously learned model.
According to claim 1,
The deep learning pipeline (DLPL) CNN learning model includes DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19 and Xception, and uses any one of these pre-trained CNN learning models Based on deep learning, characterized in that behavioral recognition system.
According to claim 1,
The transfer learning unit removes the last classification layer of the pretrained deep learning pipeline (DLPL) CNN model and adds a new layer for a new data set to learn and fine-tune. Deep learning-based action recognition system, characterized in that .
According to claim 1,
The deep feature extraction unit removes the last fully connected layer (FCL) from the Deep Learning Pipeline (DLPL) CNN training model and adds some additional layers to learn spatial patterns and relationships in the visual data stream for the data set, and finally A deep learning-based action recognition system characterized by extracting deep features by providing the output before the fully connected SoftMax layer to the next network.
According to claim 1,
The encoder unit learns a minimum representation of the input data using a deep autoencoder, reconstructs the original input data into an output closest to the original input data, and outputs it as a low-dimensional feature map. .
According to claim 1,
The adjustment module unit uses a long short-term memory (LSTM) and a recurrent neural network (RNN) to learn a long-term temporal context.
Grouping overlapping motion classes in a data set of video images, extracting frames from each video image, adjusting the frame size to fit a deep learning pipeline (DLPL) CNN learning model, and pre-learning;
Transfer learning and fine-tuning by applying an additional data set to the pre-trained Deep Learning Pipeline (DLPL) CNN learning model;
extracting high-dimensional deep features by learning frame-level spatial information from a visual data stream by applying the fine-tuned pretrained deep learning pipeline (DLPL) CNN learning model;
compressing the high-dimensional deep features extracted by the deep feature extraction unit into a low-dimensional feature map; and
Deep learning-based action recognition method comprising the step of learning temporal information from the compressed feature map and repeatedly fine-tuning and learning by applying a modified part of new video image data to a previously learned model.
According to claim 7,
The deep learning pipeline (DLPL) CNN learning model includes DenseNet201, InceptionV3, ResNet101V2, ResNet152V2, VGG16, VGG19 and Xception, and uses any one of these pre-trained CNN learning models Based on deep learning, characterized in that Behavior Recognition Method.
According to claim 7,
In the transfer learning and fine-tuning step, the last classification layer of the pre-trained Deep Learning Pipeline (DLPL) CNN model is removed and a new layer is added for the new data set to learn and fine-tune Deep learning based behavioral recognition method.
According to claim 7,
Extracting the deep features involves removing the last Fully Connected Layer (FCL) from the pre-trained Deep Learning Pipeline (DLPL) CNN training model and adding some additional layers to the dataset for a spatial pattern and relationship in the visual data stream. Deep learning-based action recognition method characterized by learning and finally providing the output before the fully connected SoftMax layer to the next network to extract deep features.
According to claim 7,
Characterized in that in the step of compressing into the low-dimensional feature map, a minimum expression for the input data is learned using a deep autoencoder, and the output is reconstructed into an output closest to the original input data and output as a low-dimensional feature map Deep learning-based behavior recognition method.
According to claim 7,
Deep learning-based action recognition method, characterized in that using a long short-term memory (LSTM) and a recurrent neural network (RNN) to learn the long-term temporal context in the fine-tuning and learning step.
A computer-readable recording medium recording a program for performing the method according to any one of claims 7 to 12 on a computer.

Images