KR20230095845A - Video anomaly detection method and apparatus therefor - Google Patents

Abstract

A video anomaly detection method and apparatus are disclosed. The apparatus for detecting an anomaly in video includes a predictive frame generating unit for generating a prediction frame in which spatial and temporal characteristics of previous frames are reflected by applying a pre-learned attention-based autoencoder model to each video frame; and an anomaly situation detector detecting an anomaly event by calculating an anomaly score for the prediction frame.

Description

Video anomaly detection method and apparatus therefor}

The present invention relates to a video anomaly detection method and apparatus therefor.

Anomaly detection in video surveillance is a popular research area in computer vision due to its diverse applications such as traffic accident detection, criminal activity detection or illegal activity detection. However, it is difficult to detect abnormal activity in the vast majority of normal situations.

Since it takes a lot of time and is inefficient for a person to directly view and analyze such a large amount of surveillance images, an automatic anomaly detection system for analyzing and detecting abnormal events in surveillance images is essential.

The goal of detecting frame anomalies in video is to identify frames that contain other spatial and motion information, and for anomaly detection models trained to learn only the normal distribution of typical events, cannot distinguish events or activities that do not appear to be anomalies. There are downsides.

The prior art uses a two-stream network comprising a spatial stream and a temporal stream, but it has the disadvantage of requiring additional calculations for optical flow extraction.

Another conventional technique uses a recurrent neural network such as Variational LSTM, but this has a disadvantage in that the model becomes more complicated as the number of stack layers increases.

An object of the present invention is to provide a video anomaly detection method and apparatus therefor.

In addition, the present invention is to provide a video anomaly detection method and apparatus capable of real-time video anomaly detection by utilizing spatial-related features and time-related features using an attention-based auto-encoder.

According to one aspect of the present invention, a video anomaly detection device is provided.

According to an embodiment of the present invention, a predictive frame generating unit generating prediction frames in which spatial and temporal characteristics of previous frames are reflected by applying a pre-learned attention-based auto-encoder model to each video frame; and an anomaly detection unit configured to detect an anomaly event by calculating an anomaly score for the prediction frame.

The attention-based autoencoder model includes a deep convolutional neural network module for extracting feature maps having multiple resolutions from each of the video frames; a spatial branching module generating a spatially related feature map by aggregating feature maps extracted respectively corresponding to a plurality of input frames through the deep convolutional neural network module; a time branching module generating a time-related feature map by performing a shift operation on a part of the feature map of the current frame output from the deep convolutional neural network module and combining it with a previous frame; and an encoder module including a combining module generating a combined feature map by combining the spatial feature map and the temporal feature map.

The attention-based auto-encoder model may include a plurality of decoding blocks for restoring the combined feature map or a feature map of a previous decoding block by deconvolution; and a decoder module including a plurality of channel attention modules disposed after each decoding block and applying a channel attention operation to an output value of each decoding block, wherein the decoder module has the same output value as the output value of each channel attention module. After being combined with the feature map of the deep convolutional neural network module that extracts with spatial resolution, it can be upsampled and transmitted to the next decoding block.

The attention-based autoencoder model may be pre-learned with normal situation frames.

The ideal score is calculated using the following equation,

이며, I는 실제 프레임을 나타내고,

는 예측 프레임을 나타내며, N는 프레임의 행과 열의 픽셀 개수를 나타내며,

는

의 최대값을 나타내고,

과

는 비디오 시퀀스에서 PSNR의 최소값과 최대값을 각각 나타낸다. here,

, I denotes a real frame,

denotes the prediction frame, N denotes the number of pixels in the row and column of the frame,

Is

represents the maximum value of

class

Represents the minimum and maximum values of PSNR in a video sequence, respectively.

According to another aspect of the present invention, a video anomaly detection method may be provided.

According to an embodiment of the present invention, (a) applying each video frame to a pre-learned attention-based auto-encoder model to generate a prediction frame in which spatial and temporal characteristics of previous frames are reflected; and (b) detecting an anomaly event by calculating an anomaly score for the prediction frame.

The step (a) includes an encoding step, wherein the encoding step includes extracting a plurality of feature maps having multi-spatial resolutions by applying each of the video frames to a deep convolutional neural network module; By passing the feature maps finally output from the deep convolutional neural network module to two branches, feature maps extracted respectively corresponding to a plurality of input frames through the deep convolutional neural network module through the first branch are aggregated and space generating a related feature map, shifting a part of the feature map of the current frame output from the deep convolutional neural network module through a second branch, and combining it with a previous frame to generate a temporal feature map; and generating a combined feature map by combining the spatial feature map and the temporal feature map.

The step (a) further includes a decoding step, which is located between a plurality of decoders having a top-down hierarchical structure, and reduces the dimension by applying a 1 x 1 convolution operation to the output value of the decoder located at the previous stage, and then ReLU. An activation function is applied, a 1 x 1 convolution operation is applied to generate a first intermediate result value, a global pooling operation is applied to the first intermediate result value, and a dimension is reduced by a 1 x 1 convolution operation applied. After applying the ReLU activation function, applying the 1 x 1 convolution operation, and then applying the sigmoid function, a second intermediate result value is generated, the first intermediate result value and the second intermediate result value are multiplied element by element, and then the generating a residual channel attention map by summing the output values of the decoders located at the previous stage; and outputting a deconvolution operation result by applying a result of element sum operation of the feature map having the same spatial resolution output from the deep convolutional neural network module and the residual channel attention map to a decoder, The highest layer decoder may perform deconvolution on the combined feature map.

Prior to step (a), the method may further include pre-learning the attention-based autoencoder model using normal situation frames.

By providing a video anomaly detection method and apparatus according to an embodiment of the present invention, it is possible to detect video anomalies in real time by utilizing spatial-related features and temporal-related features using an attention-based autoencoder model.

1 is a block diagram schematically illustrating an apparatus for detecting an anomaly in video according to an embodiment of the present invention;
2 is a diagram illustrating an attention-based autoencoder model according to an embodiment of the present invention;
3 is a diagram illustrating a time branching module according to an embodiment of the present invention;
4 is a diagram for explaining a residual channel attention module according to an embodiment of the present invention;
5 is a result of comparing performance according to spatial, temporal, and attention component configurations for a representative anomaly detection data set according to an embodiment of the present invention.
6 is a result of comparing the networks with the best performance in RESNET, which is the main axis in encoder design.
7 is a comparison result of abnormal video event detection performance according to an embodiment of the present invention and the conventional one.
8 is a flowchart illustrating a video anomaly detection method according to an embodiment of the present invention.

Singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some of the steps It should be construed that it may not be included, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or action, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing an apparatus for detecting an anomaly in video according to an embodiment of the present invention, FIG. 2 is a diagram showing an attention-based auto-encoder model according to an embodiment of the present invention, and FIG. It is a diagram shown to explain a time branching module according to an embodiment of the present invention, FIG. 4 is a diagram shown to explain a residual channel attention module according to an embodiment of the present invention, and FIG. This is a result of comparing the performance according to spatial, temporal and attention component configurations for a representative anomaly detection data set according to the embodiment. FIG. 7 is a result of comparing video anomaly event detection performance according to an embodiment of the present invention and the conventional one.

Referring to FIG. 1 , an apparatus 100 for detecting an anomaly in video according to an embodiment of the present invention includes a learning unit 110, a predicted frame generation unit 120, an anomaly detection unit 130, a memory 140, and a processor ( 150).

The learning unit 110 is a means for learning an attention-based auto-encoder model using a test data set. The operation of the attention-based auto-encoder module will be described in more detail in the prediction frame generator 120.

The predictive frame generating unit 120 has an attention-based auto-encoder model, and is a means for generating a predicted frame after inputting a video sequence (frames) to the pre-learned attention-based auto-encoder model.

The detailed structure of this attention-based autoencoder model is as shown in FIG. 2 . The attention-based autoencoder model includes an encoder module 210 and a decoder module 250.

The encoder module 210 extracts a feature map by analyzing input frames.

As shown in FIG. 2 , the encoder module 210 includes a deep convolutional neural network module 212 , a spatial divergence module 214 , a temporal divergence module 216 and a combination module 218 .

The deep convolutional neural network module 212 may extract a plurality of feature maps having different resolutions from t frame sequences, respectively. For convenience, they will be referred to as a first feature map, a second feature map, and a third feature map. That is, the deep convolutional neural network module 212 may extract a plurality of feature maps having a hierarchical structure and different resolutions.

The final feature map extracted from the deep convolutional neural network module 212 may be input to the spatial branching module 214 and the temporal branching module 216, respectively. The remaining feature maps may be transferred to decoder blocks of the same resolution level of the decoder module 250, respectively.

The spatial divergence module 214 may analyze the feature map output from the deep convolutional neural network module 212 to extract spatial features of objects existing in space. That is, the spatial branching module 214 may extract a spatially related feature map by concatenating extracted feature maps of input frames over several input frames. That is, the spatial branching module 214 may aggregate the feature maps extracted from the deep convolutional neural network model 212 over several frames. In order to reduce the computational complexity, the number of channels can be reduced since the aggregated feature map contains a large number of channels by applying a 1 x 1 convolution to the combined feature maps.

The temporal branching module 216 applies temporal shifting to utilize temporal information in the video anomaly detection process. That is, the time branching module 216 extracts a time-related feature map by applying a shift operation to the feature map output from the deep convolutional neural network module 212 .

This will be described in more detail with reference to FIG. 3 .

As shown in Figure 3, time branching module 216 moves a portion of the channel to the next frame. At this time, the remaining part of the channel may be maintained as it is. Accordingly, the feature map of the current frame may be combined with the feature map of the previous frame.

의 시간 관련 특징맵은 수학식 1과 같이 계산될 수 있다. Thus, the input feature map

The time-related feature map of can be calculated as in Equation 1.

는 시프트 연산을 나타낸다. here,

represents a shift operation.

)으로 구성되는 것을 가정하기로 한다. 현재 프레임의 채널 일부는 다음 프레임으로 시프트될 수 있다. 즉, 프레임

의 채널 일부는 프레임

의 일부로 대체될 수 있다. Referring to FIG. 3, the input feature map is composed of four feature maps (

) is assumed to be composed of. Some of the channels in the current frame may be shifted to the next frame. i.e. the frame

Some of the channels in the frame

may be replaced by a part of

The combining module 218 combines the spatial-related feature maps and temporal-related feature maps of the spatial branching module 214 and the temporal branching module 216 and transmits the combined feature map to the decoder module 250 .

If this is expressed as an equation, it is the same as equation (2).

는 시간 관련 특징맵을 나타내고,

는 공간 관련 특징맵을 나타낸다. here,

Represents a time-related feature map,

denotes a spatially related feature map.

The final feature map output from the combining module 218 is transferred to the decoder module 250.

The decoder module 250 is a means for generating a prediction frame by applying deconvolution to the final feature map transmitted from the encoder module 210. The decoder module 250 has a plurality of decoder blocks 252a to 252c having a top-down hierarchical structure, and each decoder block combines the feature map extracted from the encoder module having the same spatial resolution with the result of the previous decoder block, You can deconvolve it. This will be described in more detail below.

The output of the encoder module 210 is used as the input of the decoder module 250.

The combined feature map may be passed to the decoder module 250 to reconstruct detailed information and spatial resolution of the prediction frame.

The decoder module 250 includes a plurality of decoder blocks 252a to 252c having different resolutions and a plurality of channel attention modules 254a to 254c.

Each decoder block 252a to 252c may be composed of a series of blocks including a deconvolution layer, a batch normalization layer, and a ReLU activation function.

In order to utilize the channel relationship of the feature map, channel attention modules 254a to 254c may be disposed after each deconvolution layer. The output feature map of the channel attention module is combined with the corresponding low-level feature map extracted by the deep convolutional neural network model with the same spatial resolution. The combined feature map can be used in the next decoding block.

That is, the feature map extracted through the deep convolutional neural network model may be composed of a plurality of feature maps having different resolutions. For convenience, they will be referred to as a first feature map, a second feature map, and a third feature map. The finally output third feature map is delivered to the spatial branching module 214 and the temporal branching module 216, and after adding the spatial feature map and the temporal feature map, respectively, they can be combined and delivered to the uppermost decoder block. Thereafter, the decoder block deconvolves the combined feature map and transmits the deconvolution to the first channel attention module, and channel attention may be applied through the first channel attention module.

After channel attention is applied through the first channel attention module, it may be combined with the second feature map transmitted through the deep convolutional neural network model and transmitted to the second decoder block.

)에 채널 어텐션을 적용한다. In this way, the feature map output from the channel attention module is combined with the low-level feature map extracted from the deep convolutional neural network model having the same spatial resolution, and then passed to the next decoder block, which deconverts it for upsampling. You can volute. For the channel dependency of the feature map, the channel attention module is used. For example, 'Squeeze-and-Exciation' applies global average pooling, whereas CBAM uses average pooling and maximum pooling to obtain per-channel statistics. The Channel Attention module contains two convolutional layers instead of two fully connected layers. After each deconvolution layer, the feature map (through the channel attention modules 254a to 254c)

) to apply channel attention.

)은 수학식 3과 같이 계산될 수 있다. Output feature map (

) can be calculated as in Equation 3.

는 채널 어텐션을 나타내고,

는 요소별 곱셈(element-wise product)를 나타낸다. here,

represents the channel attention,

represents an element-wise product.

)으로 전달된다. 채널 종속성을 이용하기 위해 전역 평균 풀링이 특징맵(F)에 적용된다. 전역 평균 풀링의 출력은 C값을 갖는 벡터 v이다. 그런 다음 1 x 1 컨볼루션 연산을 적용하여 축소 비율 r로 차원을 줄인 다음 ReLU 활성화 함수(

)를 적용하고, 채널 차원을 포함하는 두번째 1 x 1 컨볼루션을 복구한다. The output of each deconvolution layer is the input feature map of the channel attention module (

) is transmitted to Global average pooling is applied to the feature map (F) to exploit channel dependencies. The output of global average pooling is a vector v with C values. A 1 x 1 convolution operation is then applied to reduce the dimensionality by a reduction factor r, followed by a ReLU activation function (

) and recovers a second 1 x 1 convolution containing the channel dimension.

If this is expressed as an equation, it is equivalent to Equation 4.

,

,

는 각각 두 컨볼루션 레이어의 가중치와 시그모이드 함수를 나타낸다. here,

,

,

denotes the weight and sigmoid function of the two convolutional layers, respectively.

According to another embodiment of the present invention, the channel attention modules 254a to 254c may be residual channel attention modules. For example, we found that the residual channel attention module provided better results in training attention-based autoencoder models for some large-scale test datasets.

과 같이 주어지는 경우, 잔여 채널 어텐션 블록은 수학식 5와 같이 계산될 수 있다. In the Residual Channel Attention module, the channel attention is placed after the two 3 x 3 turnvolution layers just before the residual concatenation. The ReLU activation is placed between two convolutional layers, as shown in FIG. 4 . If the input feature map is

When given as , the residual channel attention block can be calculated as in Equation 5.

와

는 각각 입력 특징맵과 출력 특징맵을 나타내고,

는 채널 어텐션을 나타내며, X는 수학식 6과 같이 획득될 수 있다. here,

and

denotes an input feature map and an output feature map, respectively,

Represents channel attention, and X can be obtained as in Equation 6.

는 ReLU 활성화 함수를 나타낸다. here,

represents the ReLU activation function.

들로부터 예측 프레임(

)을 예측하는 것을 목표로 한다. 각 프레임은 많은 픽셀로 구성되고, 각 픽셀에는 강도가 있기 때문에, 강도와 기울기에 대한 제약 조건은 예측 오류를 최소화하는데 중요한 요소가 될 수 있다. 따라서, RGB 공간의 모든 픽셀의 유사성은 수학식 7과 같이 실제 프레임과 예측된 프레임 사이의 픽셀 값의 차이를 비교하는 강조 제약 조건에 의해 보장될 수 있다. This attention-based autoencoder model is a sequence of input frames.

Predicted frames from (

) is aimed at predicting Since each frame consists of many pixels and each pixel has an intensity, constraints on intensity and gradient can be important factors in minimizing prediction errors. Therefore, the similarity of all pixels in RGB space can be guaranteed by the emphasis constraint condition comparing the pixel value difference between the actual frame and the predicted frame as shown in Equation 7.

는 예측된 프레임을 나타내며,

는 norm 연산을 나타낸다.

거리 채택에 의해 발생되는 잠재적인 흐림(blur)를 처리하고, 더 선명한 비디오 프레임을 획득하기 위히 기울기 제약 조건이 추가된다. where I denotes a real frame,

denotes a predicted frame,

represents the norm operation.

A gradient constraint is added to account for potential blur caused by distance adoption and to obtain a sharper video frame.

The loss function can be calculated by the difference between the gradients along two spatial dimensions, as shown in Equation 8.

Here, i and j respectively represent pixel coordinates.

Multi-Scale Structural Similarity (MS-SSIM) may be used to measure structural similarity (SSIM). MS-SSIM has been proposed to evaluate image quality at different resolutions.

The loss function including strength, slope and MS-SSIM constraints is shown in Equation 9.

는 각각 손실들 사이의 가중치 발란스를 위한 계수이다. here,

is a coefficient for weight balance between each loss.

)를 도출하여 이상 이벤트를 감지한다.The anomaly detection unit 130 calculates anomaly scores of predicted frames generated by the attention-based autoencoder model.

) to detect abnormal events.

) 사이의 차이를 측정하는 것으로, PSNR(Peak Signal to Noise Ratio)를 이용하여 계산될 수 있다. The anomaly score is the difference between the actual frame (I) and the predicted frame (

), which can be calculated using PSNR (Peak Signal to Noise Ratio).

PSNR is calculated as in Equation 10.

는

의 최대값을 나타낸다. PSNR 값이 높을수록 프레임의 품질이 좋다는 것을 의미한다. 즉, 실측 프레임과 예측 프레임의 차이가 작다. Here, N represents the number of rows and columns of the frame (i.e., the number of pixels in rows and columns),

Is

represents the maximum value of The higher the PSNR value, the better the frame quality. That is, the difference between the measured frame and the predicted frame is small.

After the PSNRs of all frames are normalized to the range [0,1], an anomaly score S(t) for each frame can be calculated using Equation 11. An ideal score can be calculated as shown in Equation 11 using the PSNR value.

과

은 각각 주어진 비디어 시퀀스에서 PSNR값의 최소값과 최대값을 나타낸다. 예측 프레임의 이상 점수는 임계값에서 프레임이 정상인지 비정상인지를 나타낸다. here,

class

denotes the minimum and maximum values of PSNR values in a given video sequence, respectively. The abnormality score of the predicted frame indicates whether the frame is normal or abnormal at the threshold.

The memory 140 is a means for storing program codes for performing a video anomaly detection method using an auto-encoder model based on channel attention according to an embodiment of the present invention.

The processor 150 includes internal components (eg, the learning unit 110, the prediction frame generation unit 120, and the anomaly detection unit 130) of the video anomaly detection apparatus 100 according to an embodiment of the present invention. , the memory 140, etc.).

5 is a result of comparing performance according to configurations of spatial, temporal, and attention components for a representative anomaly detection data set according to an embodiment of the present invention. As shown in FIG. 5 , it can be seen that the maximum performance is 97.4%, 86.7%, and 73.6% when the three components are included.

6 is a result of comparing networks with the best performance in RESNET, which is the main axis in designing an encoder. As shown in Figure 6, it can be seen that the performance is the best when WiderResNet38 is configured as a backbone network.

7 is a diagram comparing video anomaly detection results according to the prior art and an embodiment of the present invention.

As shown in FIG. 7 , it can be seen that the video anomaly detection method according to an embodiment of the present invention shows AUC performance of 97.4%, 86.7%, and 73.6%, and is superior to the prior art.

8 is a flowchart illustrating a video anomaly detection method according to an embodiment of the present invention.

In step 810, the apparatus 100 for detecting an anomaly in video applies each video frame to the pre-learned attention-based autoencoder model to generate a prediction frame in which spatial and temporal characteristics of previous frames are reflected. Here, it is assumed that the attention-based autoencoder model is pretrained with frames including normal events.

The attention-based autoencoder model extracts a plurality of feature maps having multi-spatial resolution by applying each video frame to the deep convolutional neural network module, and delivers the feature maps finally output from the deep convolutional neural network module to two branches. By doing so, a spatially related feature map is generated by aggregating feature maps extracted respectively corresponding to a plurality of input frames through the deep convolutional neural network module through a first branch, and the deep convolutional neural network module through a second branch. A part of the feature map of the current frame output from is shifted and combined with the previous frame to generate a time-related feature map, and a combined feature map is obtained by combining the space-related feature map and the time-related feature map. can create

In addition, the attention-based autoencoder model is located between a plurality of decoders having a top-down hierarchical structure, reduces the dimensionality by applying a 1 x 1 convolution operation to the output value of the decoder located in the previous stage, and then applies the ReLU activation function and A x 1 convolution operation is applied to generate a first intermediate result value, a global pooling operation is applied to the first intermediate result value, a 1 x 1 convolution operation is applied to reduce the dimensionality, and a ReLU activation function is applied; After applying the 1 x 1 convolution operation, a second intermediate result value is generated by applying the sigmoid function, and after element-by-element multiplication of the first intermediate result value and the second intermediate result value, the decoder located at the previous stage A residual channel attention map is generated by summing the output value, and the deconvolution operation result is applied to the decoder by element summing the feature map having the same spatial resolution output from the deep convolutional neural network module and the residual channel attention map. You can perform the step of outputting. At this time, the highest layer decoder among the decoders may perform a deconvolution operation on the combined feature map.

Since the attention-based auto-encoder model is the same as that described with reference to FIGS. 1 to 4, a detailed description thereof will be omitted.

In step 815, the video anomaly detection apparatus 100 detects an anomaly event by calculating an anomaly score for the predicted frame.

Devices and methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in computer readable media. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - Includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, etc. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

Claims (10)

a predictive frame generating unit that applies each video frame to a pre-learned attention-based auto-encoder model to generate a predicted frame in which spatial and temporal characteristics of previous frames are reflected; and
and an abnormal situation detector configured to detect an abnormal event by calculating an anomaly score for the predicted frame.
According to claim 1,
The attention-based autoencoder model,
a deep convolutional neural network module extracting feature maps having multiple resolutions from each of the video frames;
a spatial branching module generating a spatially related feature map by aggregating feature maps extracted respectively corresponding to a plurality of input frames through the deep convolutional neural network module;
a time branching module generating a time-related feature map by performing a shift operation on a part of the feature map of the current frame output from the deep convolutional neural network module and combining it with a previous frame; and
and an encoder module including a combining module generating a combined feature map by combining the spatial feature map and the temporal feature map.
According to claim 2,
The attention-based autoencoder model,
a plurality of decoding blocks for restoring the combined feature map or the feature map of a previous decoding block by deconvolution; and
Further comprising a decoder module disposed after each decoding block and including a plurality of channel attention modules for applying a channel attention operation to an output value of each decoding block,
The decoder module
Video anomaly detection device characterized in that the feature map of the deep convolutional neural network module extracted having the same spatial resolution as the output value of each channel attention module is combined with the feature map, and then upsampled and transmitted to the next decoding block.
According to claim 1,
The video anomaly detection device, characterized in that the attention-based autoencoder model is pre-learned with normal situation frames.
According to claim 1,
The video anomaly detection device, characterized in that the anomaly score is calculated using the following equation.

here,

, I denotes a real frame,

denotes the prediction frame, N denotes the number of pixels in the row and column of the frame,

Is

represents the maximum value of

class

Represents the minimum and maximum values of PSNR in a video sequence, respectively.
(a) generating prediction frames in which spatial and temporal features of previous frames are reflected by applying each video frame to a pre-learned attention-based auto-encoder model; and
(b) detecting an anomaly event by calculating an anomaly score for the predicted frame.
According to claim 1,
Step (a) includes an encoding step,
The encoding step is
extracting a plurality of feature maps having multi-spatial resolution by applying each of the video frames to a deep convolutional neural network module;
By passing the feature maps finally output from the deep convolutional neural network module to two branches, feature maps extracted respectively corresponding to a plurality of input frames through the deep convolutional neural network module through the first branch are aggregated and space generating a related feature map, shifting a part of the feature map of the current frame output from the deep convolutional neural network module through a second branch, and combining it with a previous frame to generate a temporal feature map; and
and generating a combined feature map by combining the spatial feature map and the temporal feature map.
According to claim 7,
The step (a) further includes a decoding step,
It is located between a plurality of decoders having a top-down hierarchical structure, and after reducing the dimension by applying a 1 x 1 convolution operation to the output value of the decoder located in the previous stage, applying a ReLU activation function and applying a 1 x 1 convolution operation, After generating a first intermediate result value, applying a global pooling operation to the first intermediate result value, applying a 1 x 1 convolution operation to reduce the dimension, applying a ReLU activation function, and applying a 1 x 1 convolution operation Afterwards, a second intermediate result value is generated by applying the sigmoid function, the first intermediate result value and the second intermediate result value are multiplied element by element, and then summed with the output value of the decoder located at the previous stage to generate a residual channel attention map. generating; and
Applying an element sum operation result of a feature map having the same spatial resolution output from the deep convolutional neural network module and the residual channel attention map to a decoder and outputting a deconvolution operation result,
A top layer decoder among the decoders performs a deconvolution operation on the combined feature map.
According to claim 1,
Before step (a),
The video anomaly detection method further comprising pretraining the attention-based autoencoder model using normal situation frames.
A computer-readable recording medium recording program codes for performing the method according to any one of claims 6 to 9.

Images