KR20230036249A - Anomaly recognition method and system based on lstm - Google Patents

Abstract

The present invention relates to an anomaly recognition method and system based on LSTM. The method includes the steps of: receiving a plurality of consecutive image frames to be determined for anomaly recognition; generating a feature vector representing characteristics of the plurality of image frames by providing the received image frames to a trained lightweight CNN model; and recognizing an abnormal activity corresponding to the plurality of image frames by providing the generated feature vector to a residual attention-based LSTM.

Description

LSTM-based anomaly recognition method and system {ANOMALY RECOGNITION METHOD AND SYSTEM BASED ON LSTM}

The present invention relates to an LSTM-based anomaly recognition method and system, and more specifically, to a method and system for recognizing anomaly activity included in an image using a lightweight CNN and a new type of LSTM.

For a safe smart city environment, it is required to recognize abnormal activities such as assaults, crimes, and road accidents in images captured through surveillance cameras. However, since such abnormal activities in real life are diverse, complex, and do not occur frequently, it is difficult for any system to accurately recognize the abnormal activities.

Meanwhile, for public safety, it is important to recognize abnormal activities by analyzing images captured by surveillance cameras in real time. However, most surveillance cameras installed in public support only a recording function and do not support a real-time monitoring function. Accordingly, in order to recognize an abnormal activity, an expert must directly view the recorded video and determine whether an abnormal activity has occurred. In this way, when an expert makes a direct judgment, the time required to recognize the occurrence of an abnormal activity increases rapidly.

The present invention provides an LSTM-based abnormality recognition method, a computer program and a system (device) stored in a recording medium to solve the above problems.

The present invention can be implemented in a variety of ways, including a method, system (device) or computer program stored on a readable storage (recording) medium.

According to an embodiment of the present invention, an LSTM-based anomaly recognition method performed by at least one processor includes the steps of receiving a plurality of consecutive image frames that are subject to determination of anomaly recognition, the received plurality of image frames Generating feature vectors representing features of a plurality of image frames by providing them to the learned lightweight CNN model and recognizing abnormal activities corresponding to the plurality of image frames by providing the generated feature vectors to a residual attention-based LSTM includes

According to an embodiment of the present invention, a lightweight CNN includes convolution blocks distinguishable by depth. The convolution block includes a per-depth convolution layer associated with a 3 x 3 filter and a per-point convolution layer associated with a 1 x 1 filter for merging the values filtered by the per-depth convolution layer.

According to one embodiment of the present invention, a lightweight CNN includes an extension layer for extending the number of channels of input data.

According to an embodiment of the present invention, the LSTM based on residual attention includes a normalization layer for normalizing feature vectors to reduce training time.

According to an embodiment of the present invention, the LSTM based on residual attention is associated with a self-attention layer. The self-attention layer generates situational awareness vectors for continuous features of a plurality of image frames based on the feature vectors.

According to one embodiment of the present invention, the LSTM based on residual attention is associated with a soft max layer for classifying abnormal activities. Recognizing the deviant activity includes recognizing the deviant activity by providing the generated feature vector to a residual attention-based LSTM and inputting the output data to the softmax layer.

According to one embodiment of the present invention, the residual attention-based LSTM is associated with an Adam optimization function and a cross entropy loss function.

A computer program stored in a computer-readable recording medium is provided to execute the above-described LSTM-based anomaly recognition method according to an embodiment of the present invention in a computer.

An abnormality recognition system according to an embodiment of the present invention includes a communication module, a memory, and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory. At least one program receives a plurality of contiguous image frames to be judged for abnormality recognition, provides the received plurality of image frames to a learned lightweight CNN model, and generates feature vectors representing characteristics of the plurality of image frames. and instructions for recognizing abnormal activity corresponding to a plurality of image frames by providing the generated feature vector to a residual attention-based LSTM.

In various embodiments of the present invention, the anomaly recognition system can recognize anomalies occurring in real time with high accuracy using a lightweight CNN and a new type of LSTM.

By using the self-attention layer in various embodiments of the present invention, the LSTM based on residual attention can efficiently recognize abnormal activity by receiving only one input such as a feature vector, unlike other models that require multiple inputs. .

Abnormal activity recognition performance can be maximized by applying the Adam optimization function and the cross entropy loss function in various embodiments of the present invention.

In various embodiments of the present invention, the loss of gradient problem in which the weight for each parameter is modified as learning progresses can be effectively solved by using the residual.

In various embodiments of the present invention, the residual attention-based LSTM can recognize abnormal activity in real time with less computing power.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned are clear to those skilled in the art (referred to as "ordinary technicians") from the description of the claims. will be understandable.

BRIEF DESCRIPTION OF THE DRAWINGS Embodiments of the present invention will be described with reference to the accompanying drawings described below, wherein like reference numbers indicate like elements, but are not limited thereto.
1 is a diagram showing an example of an anomaly recognition system according to an embodiment of the present invention.
2 is an exemplary diagram showing the internal configuration of a lightweight CNN according to an embodiment of the present invention.
3 is an exemplary diagram showing the internal configuration of an LSTM based on residual attention according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a feature extraction process of LSTM based on residual attention according to an embodiment of the present invention.
5 is an exemplary confusion matrix showing a prediction result for abnormal activity of an LSTM based on residual attention according to an embodiment of the present disclosure.
6 is a flowchart illustrating an example of an anomaly recognition method based on LSTM according to an embodiment of the present invention.
7 is a block diagram showing the internal configuration of an abnormality recognition system according to an embodiment of the present invention.

Hereinafter, specific details for the implementation of the present invention will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present invention, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding elements are given the same reference numerals. In addition, in the description of the following embodiments, overlapping descriptions of the same or corresponding components may be omitted. However, omission of a description of a component does not intend that such a component is not included in an embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the following embodiments in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and can be implemented in various different forms, only these embodiments make the present invention complete and the scope of the invention to those skilled in the art. It is provided only for complete information.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in this specification have been selected from general terms that are currently widely used as much as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the related field, the emergence of new technologies, and the like. In addition, in a specific case, there is also a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, not simply the name of the term.

Expressions in the singular number in this specification include plural expressions unless the context clearly dictates that they are singular. Also, plural expressions include singular expressions unless the context clearly specifies that they are plural. When it is said that a certain part includes a certain component in the entire specification, this means that it may further include other components without excluding other components unless otherwise stated.

In the present invention, the terms "comprise", "comprising" and the like may indicate that features, steps, operations, elements and/or components are present, but may be used when such terms include one or more other functions, It is not excluded that steps, actions, elements, components, and/or combinations thereof may be added.

In the present invention, when a specific element is referred to as being “coupled”, “combined”, “connected”, or “reactive” to any other element, the specific element is directly bonded to, combined with, and/or other elements. or may be linked or reacted, but is not limited thereto. For example, one or more intermediate components may exist between certain components and other components. Also, in the present invention, “and/or” may include each of one or more items listed or a combination of at least a part of one or more items.

1 is a diagram showing an example of an abnormality recognition system 100 according to an embodiment of the present invention. The anomaly recognition system 100 may refer to a system for receiving an image captured by an image sensor (eg, a surveillance camera, etc.) and recognizing whether an abnormal activity has occurred based on the received image. For example, the abnormality recognition system 100 may be included inside an image sensor or may be associated with one or more image sensors. As shown, the anomaly recognition system 100 may include a lightweight convolutional neural network (CNN) 102 and a long short-term memory (LSTM) 104 based on residual attention. .

According to an embodiment, the lightweight CNN 102 included in the anomaly recognition system 100 may receive a plurality of consecutive image frames 110 to be judged for anomaly recognition. For example, the plurality of consecutive image frames 110 may be 30 consecutive image frames extracted from an image, but are not limited thereto. In addition, the lightweight CNN 102 may refer to a lightweight artificial neural network in order to reduce the amount of computation, computing power required for computation, and computation time. For example, the lightweight CNN 102 may include a MobileNet in which a huge convolutional layer is replaced with a depth-wise distinguishable convolution block.

The anomaly recognition system 100 may generate a feature vector 120 representing characteristics of the plurality of image frames 110 by providing the received plurality of image frames 110 to the learned lightweight CNN model. there is. In other words, the lightweight CNN 102 may receive a plurality of image frames 110 extracted from an image and generate feature vectors 120 corresponding to the plurality of image frames 110 . For example, a plurality of sets of image frames may be extracted from one image, and the lightweight CNN 102 may generate a corresponding feature vector for each of the plurality of sets of image frames. Here, the feature vector 120 may be a vector generated based on a feature point included in each of a plurality of images, and the feature point may be extracted based on a corner or edge in the image. However, it is not limited thereto.

The abnormality recognition system 100 may recognize abnormal activities corresponding to the plurality of image frames 110 by providing the generated feature vector 120 to the residual attention-based LSTM 104 (130). In other words, the residual attention-based LSTM 104 receives the feature vector 120 from the lightweight CNN 102, and based on the information included in the feature vector 120, an ideal image corresponding to a plurality of image frames 110 is generated. activity can be recognized. Here, the residual attention-based LSTM 104 is a residual LSTM associated with a self-attention layer, and performs operations on continuous information (eg, a plurality of consecutive image frames, etc.) It may be an artificial neural network for That is, the residual attention-based LSTM 104 can recognize whether an abnormal activity has occurred by considering information included in a plurality of consecutive image frames 110 rather than a single image frame based on the feature vector 120. there is.

In FIG. 1 , the abnormality recognition system 100 has been described in detail as recognizing whether or not an abnormal activity has occurred, but is not limited thereto, and the abnormality recognition system 100 may recognize whether any kind of abnormal activity has occurred. With this configuration, the abnormality recognition system 100 can recognize abnormal activities occurring in real time with high accuracy using the lightweight CNN 102 and the new type of LSTM 104.

2 is an exemplary diagram showing the internal configuration of a lightweight CNN 220 according to an embodiment of the present invention. As described above, the lightweight CNN 220 includes a plurality of sets of image frames 210 (eg, a plurality of image frames (i), a plurality of image frames (i+1), a plurality of image frames (N), etc. ) and generate feature vectors 230 for each set of image frames. For example, the lightweight CNN 220 may be associated with MobileNetV2 or an artificial neural network created based on MobileNetV2.

According to an embodiment, the lightweight CNN 220 may include a distinguishable convolution block for each depth in which a convolution layer is replaced. This convolution block may include a convolution layer for each depth associated with a 3 × 3 filter and a convolution layer for each point associated with a 1 × 1 filter for merging values filtered by the convolution layer for each depth. . That is, the lightweight CNN 220 can be lightweight by replacing a huge convolution layer with such a convolution block, and thus can operate quickly with little computing power.

According to one embodiment, the lightweight CNN 220 may include an expansion layer for extending the number of channels of input data. For example, the extension layer may transfer data by extending the number of channels before data is transferred to the depth-specific convolution layer. In other words, an enhancement layer can generate more output channels than a given number of input channels. Also, the point-by-point convolution layer may be part of a projection layer. That is, the point-by-point convolution layer may transfer data by projecting a number of channels to a smaller number of channels.

The lightweight CNN 220 may include a bottleneck residual block. This bottleneck residual block may reduce the number of parameters used to generate the feature vector 230 or reduce the number of matrix multiplications. In addition, the lightweight CNN 220 may include a ReLU used as an activation function and a normalization layer (eg, a batch normalization layer). Additionally, the lightweight CNN 220 may include a global average pooling layer to reduce overfitting problems.

3 is an exemplary diagram showing the internal configuration of the LSTM 310 based on residual attention according to an embodiment of the present invention. As described above, the residual attention-based LSTM 310 may receive the feature vector 230 and recognize the abnormal activity 330 based on the received feature vector 230 . For example, an LSTM is a type of recurrent neural network (RNN) and may include a memory cell controlled by an input gate and a forget gate.

According to one embodiment, the cell state may be changed by the forget gate or modified by the input data. Here, the forget gate may determine the amount of information and/or data that must be passed from the previous memory to the next time step. That is, determining the extent to which information and/or data stored in the previous memory is necessary for an operation or the like at a next time step, the determined amount of information and/or data can be transferred. Also, the input gate may determine the amount of new information and/or data to be input into the memory cell. Using such an input gate and a forget gate, LSTM can effectively deal with the problem of long-term and short-term dependencies of sequential information. Such an LSTM can be determined as in Equation 1 below.

,

,

,

는 히든 상태(hidden state)에 관한 이전 시간 단계의 정보를 제어하고 모니터링하는데 사용되는 파라미터일 수 있으며,

,

,

,

는 현재 입력 시간 단계의 가중치 메트릭(matrices)에 적용되는 파라미터일 수 있다. 또한, 위 첨자 s는 입력 시퀀스 정보를 나타내고, 아래 첨자 t는 시간 단계 정보를 나타낼 수 있다.

는 시그모이드 함수에 사용될 수 있으며,

는 요소별 곱셈을 뜻하는 Hadamard product 연산자일 수 있다. 이와 같은 구성에 의해, LSTM은 각 시간 단계의 반복 모듈 간의 상호작용을 통해 이전 단계에서 얻은 정보가 다음 단계에서도 지속되도록 할 수 있으며, 그에 따라, 긴 기간의 의존성(long-term dependencies) 문제를 해결할 수 있다.here,

,

,

,

May be a parameter used to control and monitor the information of the previous time step about the hidden state,

,

,

,

may be a parameter applied to weight metrics of the current input time step. In addition, superscript s may indicate input sequence information, and subscript t may indicate time step information.

can be used for the sigmoid function,

may be a Hadamard product operator meaning element-by-element multiplication. With this configuration, LSTM can allow the information obtained in the previous step to continue in the next step through the interaction between the iterative modules of each time step, thereby solving the problem of long-term dependencies. can

According to an embodiment, the LSTM 310 based on residual attention may include a normalization layer. When information is normalized by the normalization layer, the training time of the residual attention-based LSTM 310 can be reduced. For example, normalization may be performed by Equation 2 below.

은 i번째 뉴런의 LSTM(310)의 각 레이어의 히든 상태일 수 있으며,

및

는 활성화 함수 f에 대한 입력 시퀀스를 리스케일(rescale)하기 위해 사용되는 훈련 가능한 가중치일 수 있다. 또한, t는 시간 단계(time step)를 나타낼 수 있다. 예를 들어, 오버 피팅을 감소시키기 위해, 잔차 어텐션 기반의 LSTM(310)의 각 레이어에 0.5의 탈락 임계값(dropout threshold)이 적용될 수 있다.here,

may be a hidden state of each layer of the LSTM 310 of the ith neuron,

and

may be the trainable weights used to rescale the input sequence for the activation function f. Also, t may represent a time step. For example, in order to reduce overfitting, a dropout threshold of 0.5 may be applied to each layer of the residual attention-based LSTM 310 .

According to an embodiment, the residual attention-based LSTM 310 may be associated with a self-attention layer. In this case, the self-attention layer may generate a context-aware vector for continuous features of a plurality of image frames based on the feature vector 230 . Here, the situational awareness vector may be a vector including sequential features (eg, correlation between features, etc.) of a plurality of consecutive image frames. By using such a self-attention layer, the residual attention-based LSTM 310 can efficiently recognize abnormal activity by receiving only one input such as the feature vector 230, unlike other models that require multiple inputs. there is.

According to one embodiment, the residual attention-based LSTM 310 is associated with a softmax layer 320 for classifying anomalous activity, and the softmax layer 320 is associated with the residual attention-based LSTM 310 Abnormal activity can be recognized based on the information output by For example, the residual attention-based LSTM 310 and/or the soft max layer 320 may recognize abnormal activity by Equation 3 below.

,

,

,

는 어텐션 가중치

에 따른 프레임 특징

에 대하여 학습된 파라미터이고,

는 이상 활동과 연관된 스코어를 나타낼 수 있다. 또한,

는 소프트 맥스 레이어(320)로부터 획득된 이상 활동의 확률을 나타낼 수 있다. 다시 말해, 잔차 어텐션 기반의 LSTM(310)에 의해 추출된 복수의 이미지 프레임에 대한 정보는 해당 이미지 프레임이 이상 활동 및/또는 정상 활동을 포함하는지 여부를 식별하는데 사용될 수 있으며, 소프트 맥스 레이어(320)는 잔차 어텐션 기반의 LSTM(310)에 의해 출력된 정보를 기초로 최종적인 분류 및/또는 예측을 통해 이상 활동을 인식할 수 있다. 또한, 잔차 어텐션 기반의 LSTM(310)은 Adam 최적화(optimization) 함수 및 크로스 엔트로피(cross-entropy) 손실 함수와 연관될 수 있다. 즉, Adam 최적화 함수 및 크로스 엔트로피 손실 함수가 적용됨으로써, 이상 활동 인식 성능이 극대화될 수 있다.here,

,

,

,

is the attention weight

Frame characteristics according to

is a parameter learned for

may represent a score associated with an anomalous activity. also,

may represent the probability of an abnormal activity obtained from the soft max layer 320. In other words, information on a plurality of image frames extracted by the residual attention-based LSTM 310 may be used to identify whether the corresponding image frame includes abnormal activity and/or normal activity, and the soft max layer 320 ) may recognize abnormal activity through final classification and/or prediction based on the information output by the residual attention-based LSTM 310. In addition, the residual attention-based LSTM 310 may be associated with an Adam optimization function and a cross-entropy loss function. That is, by applying the Adam optimization function and the cross entropy loss function, abnormal activity recognition performance can be maximized.

,

,

,

은 각 시간 단계의 새로운 입력으로서, 복수의 이미지 프레임으로부터 추출된 특징 벡터의 적어도 일부를 나타낼 수 있다. 즉, 잔차 어텐션 기반의 LSTM은 반복적으로 실행되며, 각 시간 단계에서 새로운 입력을 수신하여 처리할 수 있다. 이 경우, 각 시간 단계에서의 LSTM은 서로 연관될 수 있으며, 이전 시간 단계의 정보 중 적어도 일부가 다음 시간 단계에서 이용될 수 있다.4 is an exemplary diagram illustrating a feature extraction process 400 of an LSTM based on residual attention according to an embodiment of the present invention. In the example shown,

,

,

,

is a new input at each time step, and may represent at least some of feature vectors extracted from a plurality of image frames. That is, the residual attention-based LSTM is repeatedly executed, and can receive and process new inputs at each time step. In this case, LSTMs at each time step may be correlated with each other, and at least some of the information of the previous time step may be used at the next time step.

According to an embodiment, an LSTM based on residual attention may be associated with a self-attention layer. When associated with the self-attention layer, the LSTM based on residual attention can effectively utilize previous information even when the length of the time step is long, and the LSTM based on residual attention and/or the self-attention layer can effectively utilize the information extracted from multiple image frames. Elements that are related to each other among consecutive pieces of information may be determined. In this way, features associated with abnormal activities of a plurality of image frames extracted by the residual attention-based LSTM may be concatenated and/or processed and transmitted to the softmax layer.

Also, the LSTM may be a residual based LSTM. Here, the residual characterizes the sequential information of the top layer and can be used to reconstruct the layer by finding a residual function given as an input layer. For example, residual function learning may be determined by Equation 4 below.

는 입력 데이터일 수 있으며,

는 레이어의 결과적인 순차 정보 벡터일 수 있다. 또한,

는 관련된 레이어에서 학습된 잔차를 나타낼 수 있다. 이러한 잔차 학습에 의해 레이어들 사이의 효과적인 훈련을 위한 숏컷(shortcut) 함수가 생성될 수 있다. 이와 같은 구성에 의해, 잔차를 이용함으로써 학습이 진행되면서 각 파라미터에 대한 가중치가 변형되는 기울기 소실(vanishing gradient) 문제가 효과적으로 해결될 수 있다.here,

can be input data,

may be the resulting sequential information vector of the layer. also,

may represent the residual learned in the relevant layer. A shortcut function for effective training between layers may be generated by such residual learning. With this configuration, a vanishing gradient problem in which weights for each parameter are modified while learning progresses can be effectively solved by using the residuals.

5 is an exemplary confusion matrix showing prediction results for abnormal activity of LSTM based on residual attention according to an embodiment of the present disclosure. In the illustrated example, the residual attention-based LSTM predicts anomalous activity (eg whether an anomalous activity occurs and/or the type of anomalous activity that has occurred) based on a plurality of image frames using the UCF-Crime dataset. can As shown, the residual attention-based LSTM can predict attacks, explosions, fighting, normals, road accidents, etc. with high accuracy using a plurality of image frames. can

Additionally, the performance of LSTMs based on residual attention can be measured higher than other types of LSTMs. In this regard, the performance of each of LSTM, bidirectional LSTM, residual LSTM and LSTM based on residual attention was evaluated. As shown in Table 1 below, performance evaluation was performed using the UCF-Crime dataset, UMN and Avenue datasets.

Model dataset Recall (%) accuracy(%) F1 score (%) AUC (%) LSTM UCF-Crime dataset 86 74 77 88 BD-LSTM 79 84 76 87 Residual LSTM 91 78 82 95 LSTM based on residual attention 78 87 81 96 LSTM UMN 87 77 81 86 BD-LSTM 88 81 84 88 Residual LSTM 94 95 94 96 LSTM based on residual attention 98 98 98 98 LSTM Avenue 91 93 92 91 BD-LSTM 94 95 94 94 Residual LSTM 93 94 94 94 LSTM based on residual attention 98 99 99 98

As shown in Table 1, the residual attention-based LSTM performed better in most indicators than other conventional LSTMs. In addition, as shown in Table 2 below, the efficiency of LSTM based on residual attention has increased compared to other conventional models.

Model Time Complexity (seconds) Model size (MB) parameter
(Millions) FLOPs (Mega) VGG-16 - 528 138 VGG-19 - 549 143 FlowNet - 638.5 162.49 DEARESt - 1187.5 305.49 LSTM based on residual attention 0.263 12.8 3.3 618.3

In other words, as shown in Table 2, the residual attention-based LSTM showed high performance in all aspects of time complexity, model size, parameters, and FLOPs. Accordingly, the residual attention-based LSTM can generate abnormalities in real time with less computing power. activity can be recognized. In addition, as shown in Table 3 below, the prediction accuracy of LSTM based on residual attention increased compared to other conventional models.

Model accuracy(%) UCF-Crime UMN Avenue VGG-16 72.66 - - VGG-19 71.66 - - FlowNet 71.33 - - DEARESt 76.66 - - Nandedkar and Bansod - 96.99 - Kyung Joo Cheoi - 96.50 90.18 Al-Dhamari et al. - 97.44 - LSTM based on residual attention 78.43 98.20 98.80

That is, the residual attention-based LSTM can recognize abnormal activity with higher accuracy than other conventional models using less computing power.

6 is a flowchart illustrating an example of an anomaly recognition method 600 based on LSTM according to an embodiment of the present invention. The LSTM-based anomaly recognition method 600 may be performed by a processor (eg, at least one processor of an anomaly recognition system). As shown, in the LSTM-based anomaly recognition method 600, a processor may receive a plurality of consecutive image frames to be judged for anomaly recognition (S610).

The processor may generate a feature vector representing characteristics of the plurality of image frames by providing the received plurality of image frames to the learned lightweight CNN model (S620). Here, the lightweight CNN may include a distinguishable convolution block by depth, and the convolution block is a convolution layer by depth associated with a 3 x 3 filter and a 1 x 1 for merging the values filtered by the convolution layer by depth. 1 Convolution layer for each point associated with the filter may be included. In addition, the lightweight CNN may include an extension layer for extending the number of channels of input data.

The processor may recognize abnormal activities corresponding to a plurality of image frames by providing the generated feature vectors to the residual attention-based LSTM (S630). For example, an LSTM based on residual attention may include a normalization layer to normalize feature vectors to reduce training time. In addition, LSTM based on residual attention may be associated with a self-attention layer. Here, the self-attention layer may generate context vectors for continuous features of a plurality of image frames based on the feature vectors.

According to one embodiment, the LSTM based on residual attention may be associated with a soft max layer for classifying abnormal activity. In this case, the processor may recognize the abnormal activity by providing the generated feature vector to the residual attention-based LSTM and inputting output data to the softmax layer. Here, the LSTM based on residual attention may be associated with an Adam optimization function and a cross entropy loss function.

7 is a block diagram showing the internal configuration of an abnormality recognition system 700 according to an embodiment of the present invention. The anomaly recognition system 700 may include a memory 710 , a processor 720 , a communication module 730 and an input/output interface 740 . As shown in FIG. 7 , the anomaly recognition system 700 may be configured to communicate information and/or data through a network using a communication module 730 .

Memory 710 may include any non-transitory computer readable storage medium. According to one embodiment, the memory 710 is a non-perishable mass storage device (permanent mass storage device) such as random access memory (RAM), read only memory (ROM), disk drive, solid state drive (SSD), flash memory, and the like. mass storage device). As another example, a non-perishable mass storage device such as a ROM, SSD, flash memory, disk drive, etc. may be included in the anomaly recognition system 700 as a separate permanent storage device separate from memory. Also, an operating system and at least one program code may be stored in the memory 710 .

These software components may be loaded from a computer-readable recording medium separate from the memory 710 . Recording media readable by such a separate computer may include recording media directly connectable to the anomaly recognition system 700, for example, a floppy drive, disk, tape, DVD/CD-ROM drive, memory card, etc. It may include a computer-readable recording medium. As another example, software components may be loaded into the memory 710 through the communication module 730 rather than a computer-readable recording medium. For example, at least one program may be loaded into the memory 710 based on a computer program installed by files provided by developers or a file distribution system that distributes application installation files through the communication module 730. can

The processor 720 may be configured to process commands of a computer program by performing basic arithmetic, logic, and input/output operations. Commands may be provided to a user terminal (not shown) or other external system by the memory 710 or the communication module 730 .

The communication module 730 may provide a configuration or function for the anomaly recognition system 700 to communicate with an external device and/or a user terminal (not shown) through a network, and the anomaly recognition system 700 may communicate with an external system. (For example, a separate cloud system, etc.) may provide configuration or functionality for communication. For example, control signals, commands, data, etc. provided under the control of the processor 720 of the anomaly recognition system 700 pass through the communication module 730 and the network to the user terminal and/or the communication module of the external system. It may be transmitted to a terminal and/or an external system.

In addition, the input/output interface 740 of the anomaly recognition system 700 is connected to the anomaly recognition system 700 or means for interface with a device (not shown) for input or output that the anomaly recognition system 700 may include. can be In FIG. 7 , the input/output interface 740 is illustrated as an element separately configured from the processor 720 , but is not limited thereto, and the input/output interface 740 may be included in the processor 720 . The anomaly recognition system 700 may include more components than those shown in FIG. 7 . However, there is no need to clearly show most of the prior art components.

The processor 720 of the anomaly recognition system 700 may be configured to manage, process, and/or store information and/or data received from a plurality of user terminals and/or a plurality of external systems.

The above-described methods and/or various embodiments may be realized with digital electronic circuits, computer hardware, firmware, software, and/or combinations thereof. Various embodiments of the present invention may be performed by a data processing device, eg, one or more programmable processors and/or one or more computing devices, or as a computer readable recording medium and/or a computer program stored on a computer readable recording medium. can be implemented The above-described computer programs may be written in any form of programming language, including compiled or interpreted languages, and may be distributed in any form, such as a stand-alone program, module, or subroutine. A computer program may be distributed over one computing device, multiple computing devices connected through the same network, and/or distributed over multiple computing devices connected through multiple different networks.

The methods and/or various embodiments described above may be performed by one or more processors configured to execute one or more computer programs that process, store, and/or manage any function, function, or the like, by operating on input data or generating output data. can be performed by For example, the method and/or various embodiments of the present invention may be performed by a special purpose logic circuit such as a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), and the method and/or various embodiments of the present invention may be performed. Apparatus and/or systems for performing the embodiments may be implemented as special purpose logic circuits such as FPGAs or ASICs.

The one or more processors executing the computer program may include a general purpose or special purpose microprocessor and/or one or more processors of any kind of digital computing device. The processor may receive instructions and/or data from each of the read-only memory and the random access memory, or receive instructions and/or data from the read-only memory and the random access memory. In the present invention, components of a computing device performing methods and/or embodiments may include one or more processors for executing instructions, and one or more memory devices for storing instructions and/or data.

According to one embodiment, a computing device may exchange data with one or more mass storage devices for storing data. For example, a computing device may receive/receive data from and transfer data to a magnetic or optical disc. A computer-readable storage medium suitable for storing instructions and/or data associated with a computer program includes semiconductor memory devices such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable PROM (EEPROM), and flash memory devices. Any type of non-volatile memory may be included, but is not limited thereto. For example, computer readable storage media may include magnetic disks such as internal hard disks or removable disks, magneto-optical disks, CD-ROM and DVD-ROM disks.

To provide interaction with a user, a computing device includes a display device (eg, a cathode ray tube (CRT), a liquid crystal display (LCD), etc.) It may include a pointing device (eg, a keyboard, mouse, trackball, etc.) capable of providing input and/or commands to, but is not limited thereto. That is, the computing device may further include any other type of device for providing interaction with a user. For example, a computing device may provide any form of sensory feedback to a user for interaction with the user, including visual feedback, auditory feedback, and/or tactile feedback. In this regard, the user may provide input to the computing device through various gestures such as visual, voice, and motion.

In the present invention, various embodiments may be implemented in a computing system including a back-end component (eg, a data server), a middleware component (eg, an application server), and/or a front-end component. In this case, the components may be interconnected by any form or medium of digital data communication, such as a communication network. For example, the communication network may include a local area network (LAN), a wide area network (WAN), and the like.

A computing device based on the example embodiments described herein may be implemented using hardware and/or software configured to interact with a user, including a user device, user interface (UI) device, user terminal, or client device. can For example, the computing device may include a portable computing device such as a laptop computer. Additionally or alternatively, the computing device may include personal digital assistants (PDAs), tablet PCs, game consoles, wearable devices, internet of things (IoT) devices, virtual reality (VR) devices, AR (augmented reality) device, etc. may be included, but is not limited thereto. A computing device may further include other types of devices configured to interact with a user. Further, the computing device may include a portable communication device (eg, a mobile phone, smart phone, wireless cellular phone, etc.) suitable for wireless communication over a network, such as a mobile communication network. A computing device communicates wirelessly with a network server using wireless communication technologies and/or protocols such as radio frequency (RF), microwave frequency (MWF) and/or infrared ray frequency (IRF). It can be configured to communicate with.

The various embodiments herein, including specific structural and functional details, are exemplary. Accordingly, embodiments of the present invention are not limited to those described above and may be implemented in various other forms. In addition, terms used in the present invention are for describing some embodiments and are not construed as limiting the embodiments. For example, the singular and the above may be construed to include the plural as well, unless the context clearly dictates otherwise.

In the present invention, unless defined otherwise, all terms used in this specification, including technical or scientific terms, have the same meaning as commonly understood by a person of ordinary skill in the art to which such concept belongs. . In addition, terms commonly used, such as terms defined in a dictionary, should be interpreted as having a meaning consistent with the meaning in the context of the related technology.

Although the present invention has been described in relation to some embodiments in this specification, various modifications and changes can be made without departing from the scope of the present invention that can be understood by those skilled in the art. Moreover, such modifications and variations are intended to fall within the scope of the claims appended hereto.

100: anomaly recognition system
102: lightweight CNN
104: LSTM based on residual attention
110: image frame
120: feature vector
130: Anomalous Activity Recognition

Claims (9)

An LSTM-based anomaly recognition method performed by at least one processor,
Receiving a plurality of continuous image frames that are subject to determination of abnormality recognition;
providing the received plurality of image frames to a trained lightweight convolutional neural network (CNN) model to generate feature vectors representing characteristics of the plurality of image frames; and
recognizing an abnormal activity corresponding to the plurality of image frames by providing the generated feature vector to a residual attention-based long short-term memory (LSTM);
Including, LSTM-based anomaly recognition method.
According to claim 1,
The lightweight CNN includes convolutional blocks that can be distinguished depth-wise,
The convolution block is a depth-wise convolution layer associated with a 3 × 3 filter and a point-wise associated with a 1 × 1 filter for merging values filtered by the depth-wise convolution layer. -wise) LSTM-based anomaly recognition method including a convolutional layer.
According to claim 1,
The lightweight CNN includes an expansion layer for extending the number of channels of input data, an LSTM-based anomaly recognition method.
According to claim 1,
Wherein the residual attention-based LSTM includes a normalization layer for normalizing the feature vector to reduce training time.
According to claim 1,
The residual attention-based LSTM is associated with a self-attention layer,
wherein the self-attention layer generates a context-aware vector for continuous features of the plurality of image frames based on the feature vector.
According to claim 1,
The residual attention-based LSTM is associated with a Softmax layer for classifying abnormal activity,
Recognizing the abnormal activity,
recognizing the abnormal activity by providing the generated feature vector to the residual attention-based LSTM and inputting output data to the softmax layer;
Including, LSTM-based anomaly recognition method.
According to claim 1,
The residual attention-based LSTM is associated with an Adam optimization function and a cross-entropy loss function, an LSTM-based anomaly recognition method.
A computer program stored in a computer-readable recording medium to execute the LSTM-based anomaly recognition method according to any one of claims 1 to 7 on a computer.
As an anomaly recognition system,
communication module;
Memory; and
at least one processor connected to the memory and configured to execute at least one computer readable program contained in the memory
including,
The at least one program,
Receiving a plurality of continuous image frames to be judged for abnormality recognition;
Providing the received plurality of image frames to a learned lightweight CNN model to generate feature vectors representing characteristics of the plurality of image frames;
and instructions for recognizing an abnormal activity corresponding to the plurality of image frames by providing the generated feature vector to a residual attention-based LSTM.

Images