KR20230095365A - Image analysis apparatus for detecting abnormal event, and method thereof - Google Patents

Abstract

An image analysis apparatus for detecting abnormal behavior according to the present invention includes a pre-processing module for generating image data for analysis by degrading original data, which is image data captured at high resolution, to low resolution; a detection neural network module that selects a local area in which abnormal behavior exists from the image data for analysis as an abnormal behavior detection area; a local data extraction module extracting local data for a region corresponding to the abnormal behavior detection region selected by the detection deep neural network module from the original data; It may be configured to include; a classification neural network module that determines whether or not there is an abnormal behavior by taking the local data as an input.

Description

Image analysis apparatus for detecting abnormal event, and method thereof}

The present invention relates to a technique for detecting an abnormal behavior in an image collected from a camera.

Surveillance cameras are used for a variety of purposes, including industrial, educational, medical, traffic control, and control purposes, and are effective in preventing and deterring crime, facilitating detection and arrest of criminals, reducing the fear of crime recognized by citizens, and in major areas. Its effects, such as supplementing the limited police manpower following the installation of E, have been empirically proven.

Recently, video data collected from CCTVs, drones, smartphones, etc. are applied to video analysis algorithms to analyze abnormal events of objects included in video data. Although technology for preventing accidents by detecting abnormal behavior from image data is in the limelight, high computational power is required to process a vast amount of image data. In particular, as the quality and quantity of image data that can be collected from CCTVs, drones, and smartphones rapidly increase, the performance of small embedded devices is insufficient to process the vast amount of image data captured at high resolution, so high-performance computing resources are inevitably needed. There is a difficulty in that the image must be transmitted to a server having a post-processing or a person must directly analyze the image.

On the other hand, the technology of analyzing images through deep neural network learning has been continuously developed since surpassing human accuracy in 2015, and is being usefully used to solve real industrial and social problems. However, since the development of such deep neural network technology requires higher computing power, it is difficult to use it in low-end embedded devices.

An object of the present invention is to provide an image analysis device and method capable of detecting abnormal behavior in high-resolution image data even in a low-end embedded environment.

The objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the description below.

An image analysis apparatus for detecting abnormal behavior according to the present invention includes a pre-processing module for generating image data for analysis by degrading original data, which is image data captured at high resolution, to low resolution; a detection neural network module that selects a local area in which abnormal behavior exists from the image data for analysis as an abnormal behavior detection area; a local data extraction module extracting local data for a region corresponding to the abnormal behavior detection region selected by the detection deep neural network module from the original data; It may be configured to include; a classification neural network module that determines whether or not there is an abnormal behavior by taking the local data as an input.

The present invention may further include a detection learning module for training the detection neural network module, and when the detection learning module outputs a latent vector indicating a region in which abnormal behavior exists in the input image data, the detection neural network module outputs a latent vector. , characterized in that the detection neural network module is trained so that the difference between the latent vector and a preset expected value is minimized.

In addition, the present invention may further include a classification learning module for training the classification neural network module, wherein the classification learning module generates an output value indicating whether or not abnormal behavior is included in the image data input to the classification neural network module. Upon calculation, it is characterized in that the classification neural network module is trained so that the difference between the output value and a preset expected value is minimized.

In another aspect, an image analysis method for detecting abnormal behavior according to the present invention includes generating image data for analysis by degrading original data, which is image data captured at high resolution, by a preprocessing module to a low resolution, and generating image data for analysis by a detection neural network module. selecting a local area where abnormal behavior exists from the analysis image data as an abnormal behavior detection area; and a local data extraction module corresponding to the abnormal behavior detection area selected from the original data by the detection deep neural network module. It may be configured to include a step of extracting local data for, and a step of determining whether or not there is an abnormal behavior by a classified neural network module using the local data as an input.

The method according to the present invention may further include a detection learning step of training the detection neural network module, wherein the detection learning step is a step of generating deterioration data obtained by degrading input high resolution image data to a low resolution by a preprocessing module. ; outputting, by a detection neural network module, a latent vector indicating a region in which an abnormal behavior exists in the deterioration data; and learning the detection neural network module so that the difference between the latent vector and a preset expected value is minimized by the detection learning module.

In addition, the method according to the present invention may further include converting, by the preprocessing module, the deteriorated data into a plurality of deteriorated data through data augmentation.

In addition, the method according to the present invention may further include a classification learning step of training the classification neural network module, wherein the classification learning step indicates whether or not abnormal behavior is included in the image data input to the classification neural network module. Calculating an output value; The classification learning module may include training the classification neural network module such that a difference between the output value output by the classification neural network module and a preset expected value is minimized.

In another aspect, the present invention may provide a computer readable recording medium on which a program for executing the above-described image analysis method for detecting abnormal behavior in a computer is recorded.

An image analysis apparatus and method for detecting abnormal behavior according to the present invention implements a deep neural network for detecting abnormal behavior in a low-end embedded device to solve the problem of performance degradation caused when analyzing high-resolution image data, thereby detecting abnormal behavior. For this purpose, it is possible to detect abnormal behavior in real time without involving high-performance computing devices and manpower, thereby contributing to the establishment of a social safety net.

1 is a schematic diagram illustrating a configuration of an image analysis device for detecting abnormal behavior based on a deep neural network and an image analysis method for detecting abnormal behavior according to an embodiment of the present invention.
2 is a schematic diagram illustrating a pre-processing process of original data according to an embodiment of the present invention.
3 is a schematic diagram illustrating a learning process of a detection neural network module according to an embodiment of the present invention.
4 is a schematic diagram illustrating a learning process of a classified neural network module according to an embodiment of the present invention.

Prior to the detailed description of the present invention, the terms or words used in this specification and claims described below should not be construed as being limited to a common or dictionary meaning, and the inventors should use their own invention in the best way. It should be interpreted as a meaning and concept corresponding to the technical idea of the present invention based on the principle that it can be properly defined as a concept of a term for explanation. Therefore, the embodiments described in this specification and the configurations shown in the drawings are only the most preferred embodiments of the present invention, and do not represent all of the technical ideas of the present invention, so various equivalents that can replace them at the time of the present application. It should be understood that there may be water and variations.

In addition, terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In addition, terms such as "include" or "having" described in this specification are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or the It should be understood that the above does not preclude the possibility of the presence or addition of other features, numbers, steps, operations, components, parts, or combinations thereof.

In addition, terms such as “… unit”, “… unit”, and “module” described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. there is.

Also, "a or an", "one", "the" and similar words in the context of describing the invention (particularly in the context of the claims below) indicate otherwise in this specification. may be used in the sense of including both the singular and the plural, unless otherwise clearly contradicted by the context.

In addition, embodiments within the scope of the present invention include computer readable media having or conveying computer executable instructions or data structures stored on the computer readable media. Such computer readable media can be any available media that can be accessed by a general purpose or special purpose computer system. By way of example, such computer readable media may be in the form of RAM, ROM, EPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, or computer executable instructions, computer readable instructions or data structures. physical storage media, such as, but not limited to, any other medium that can be used to store or convey any program code means that is used and that can be accessed by a general purpose or special purpose computer system. In the following description and claims, computer readable instructions include, for example, instructions and data that cause a general purpose or special purpose computer system to perform a particular function or group of functions. Computer executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, or even source code.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. At this time, it should be noted that the same components in the accompanying drawings are indicated by the same reference numerals as much as possible. In addition, detailed descriptions of well-known functions and configurations that may obscure the gist of the present invention will be omitted. For the same reason, some components in the accompanying drawings are exaggerated, omitted, or schematically illustrated, and the size of each component does not entirely reflect the actual size.

First, an image analysis apparatus and method for detecting abnormal behavior based on a deep neural network according to an embodiment of the present invention will be described. 1 is a schematic diagram illustrating a configuration of an image analysis device for detecting abnormal behavior based on a deep neural network and an image analysis method for detecting abnormal behavior according to an embodiment of the present invention, and FIG. 2 is a diagram according to an embodiment of the present invention. Figure 3 is a schematic diagram explaining the preprocessing process of original data, Figure 3 is a schematic diagram explaining the learning process of the detection neural network module according to an embodiment of the present invention, Figure 4 is a schematic diagram explaining the learning process of the classification neural network module according to an embodiment of the present invention It is an explanatory diagram.

As shown in FIG. 1, the image analysis apparatus for detecting abnormal behavior according to the present invention is a pre-processing module for generating image data D2 for analysis by degrading original data D1, which is image data captured at high resolution, to low resolution ( 10), a detection neural network module 20 that selects a local area in which abnormal behavior exists from the analysis image data D2 as an abnormal behavior detection area D3, and the detection deep neural network from the original data D1 A local data extraction module 40 for extracting local data D4 for an area corresponding to the abnormal behavior detection area selected by the module 20, and the local data D4 as an input to determine whether or not there is an abnormal behavior It may be configured to include a classified neural network module 30 that discriminates.

Here, the image data means a moving image or still image captured in high resolution through a camera such as a CCTV or a smartphone. "Abnormal behavior" can be defined as an abnormal behavior among behaviors made up of various behaviors, including the type of object (person or object), the number of objects (one person or two or more), and detailed movements (wielding, Kicking, pushing, pulling, throwing, falling, etc.) In the present invention, even in a low-noise embedded environment (eg, drone, cctv, smartphone, etc.) including a camera and a graphic processing unit (GPU), in order to detect whether abnormal behavior exists in high-resolution images captured by a camera, It is intended to train and utilize deep neural networks.

The performance of an arithmetic device required by a deep neural network module in detecting abnormal behavior in image data is generally proportional to the resolution (size) of an image given as an input. Therefore, the high-resolution image provided according to the performance of the camera through the pre-processing module 10 is degraded into a low-resolution image that can be operated by a low-performance arithmetic unit in a low-specification embedded device. The detection neural network module 20 is trained through the degraded low-resolution image and used to detect abnormal behavior.

2 shows a process of generating degraded data L2 by degrading original data L1, which is image data captured at high resolution, to low resolution for learning of the detection neural network module 20. As shown in FIG. 2, when original data L1 having a resolution of, for example, 2K or higher is input, the preprocessing module 10 generates degraded data L2 degraded to a low resolution (eg, 640×480). In addition, the pre-processing module 10 adds noise to the degraded image data through data augmentation, inverts it, rotates it, or cuts out a part of it during learning of the detection neural network module 20. By modifying the method, a plurality of deterioration data (L2) can be generated and used for learning. Through this, it is possible to compensate for the degradation of detection performance due to data deterioration by increasing the amount of data used for learning.

Next, as shown in FIG. 3, the detection neural network module 20 detects an area in which deviant behavior exists (ie, an anomalous behavior detection area) within deterioration data given as an input. Here, the detection neural network module 20 may include a plurality of layers divided into an input layer, a plurality of hidden layers, and an output layer. Each of the plurality of layers includes a plurality of operations to which weights (W) are applied. The input layer and the plurality of hidden layers are preferably connected in a non-linear structure. In the case of the hidden layer, it may have a form of a convolutional layer, a residual layer, a fully connected layer, or the like, depending on the characteristics of input data. The combination of these hidden layers can use various neural network models (RCNN, YOLO, etc.) for the purpose of object detection in an image known in the related art. The output layer takes the output of the previous hidden layer as an input and outputs a latent vector in which a local area (abnormal behavior detection area) in image data where abnormal behavior exists is expressed in a vector form.

For example, the detection neural network module 20 divides the image of the input data into grid cells of S×S (S is a natural number), and the area where the object exists (bounding box, B) and the probability of abnormal behavior existing within the bounding box ( Create a latent vector representing the class probability, C). Here, the bounding box B includes X, Y coordinates and horizontal and vertical size information. That is, the latent vector includes positional information about an area where an abnormal behavior exists within the input image data and probability information where an abnormal behavior exists within the area. For example, if a "448×448" size image is input, a "7×7" grid is applied, the latent vector becomes a shape of "7×7×30", and each cell of "7×7" In , a vector with a length of 30 has information on the bounding box B of each cell and the class probability C classified.

In the learning process, the detection neural network module 20 improves the accuracy of the deviant behavior detection area through learning. and correct answer data with probability values) to train the detection neural network module 20 to minimize the difference. That is, the latent vector output by the detection neural network module 20 for the deterioration data L2 input as training data is deep in the direction in which the output value of the loss function is lowered so that the local region L3 including the abnormal behavior is represented. This is done by adjusting the weights (W) of the neural network (the process of adjusting the weights in this way is called 'learning'). Finally, the detection neural network module 20 is trained to output a latent vector representing a local area in which abnormal behavior exists in the input degradation data.

Meanwhile, as shown in FIG. 4, the classified neural network module 30 is trained to calculate an output value indicating whether or not abnormal behavior is included in the high-resolution image data L4. Similar to the detection neural network module 20 described above, the classified neural network module 30 may include an input layer, a plurality of hidden layers, and an output layer. Additionally, the size of image data given as input to the classified neural network module 30 may be different, and a convolution layer, a pooling layer, and the like may be further included to process inputs of different sizes. In this way, the detection neural network module 20 may be composed of a plurality of layers, and each of the plurality of layers may perform a plurality of operations to which a weight w is applied. The output layer constituting the classified neural network module 30 receives the output of the previous hidden layer as an input and calculates an output value indicating whether or not abnormal behavior is included in the image data L4. For example, it may be output in the form of a real value representing whether or not there is an abnormal behavior, such as "1" if there is an abnormal behavior and "0" if there is no abnormal behavior. In the classification learning module 30a, the direction in which the output value of the loss function that takes the output value output from the classification neural network module 30 and the preset expected value as input decreases, that is, the difference between the output value of the classification neural network module 30 and the expected value is the minimum. The classified neural network module 30 is trained by adjusting the weight w of the deep neural network constituting the classified neural network module 30 so that

Returning to FIG. 1 again, through the image analysis device using the well-learned detection neural network module 20 and the classified neural network module 30, it is possible to detect abnormal behavior in high-resolution image data in a low-end embedded device without performance degradation. .

That is, as shown in FIG. 1, when original data D1, which is image data captured at high resolution, is input, the preprocessing module 10 degrades it to a low resolution to generate image data D2 for analysis. The generated image data for analysis D2 is input to the detection neural network module 20, and the detection neural network module 20 detects abnormal behavior in the local area where abnormal behavior exists from the image data for analysis D2 deteriorated to low resolution. area (D3). Specifically, the detection neural network module 20 outputs a latent vector indicating a region in which abnormal behavior exists. For example, the latent vector represents a coordinate value for a local area in which an abnormal behavior exists in the deteriorated analysis image data D2, and based on this coordinate value, the local data extraction module 40 performs the high-resolution original data D1 ), local data D4 for an area corresponding to the abnormal behavior detection area D3 is extracted. Here, the local data D4 is composed of a part of high-resolution original data corresponding to an area selected by the detection neural network module 20 as an area in which abnormal behavior exists from the data D2 degraded to a low resolution. Using the extracted high-resolution local data D4 as an input, the classified neural network module 30 calculates an output value indicating whether or not there is an abnormal behavior, and determines whether or not the behavior is abnormal according to the calculated output value.

Conventionally, in order to analyze high-resolution image data and determine whether or not it is an abnormal behavior, it was inevitable to transmit the data to a server supporting high-end computing resources and process it ex post. However, according to the present invention, since abnormal behavior in high-resolution image data can be detected without performance degradation even in a low-specification embedded environment, it is possible to detect abnormal behavior in real time through low-specification computing resources such as CCTVs or drones, thereby enabling a social safety net. can contribute to building

Meanwhile, the methods according to the embodiments of the present invention described above may be implemented in a program form readable by various computer means and recorded on a computer-readable recording medium. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks ( magneto-optical media), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program commands may include high-level language wires that can be executed by a computer using an interpreter, as well as machine language wires such as those produced by a compiler.

As set forth above, this specification contains many specific implementation details, but these should not be construed as limiting on the scope of any invention or claimables, but rather may be specific to a particular embodiment of a particular invention. It should be understood as a description of the features in Certain features that are described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable subcombination. Further, while features may operate in particular combinations and are initially depicted as such claimed, one or more features from a claimed combination may in some cases be excluded from that combination, and the claimed combination is a subcombination. or sub-combination variations.

The embodiments of the present invention present the best mode of the present invention and provide examples to illustrate the present invention and to enable those skilled in the art to make and use the present invention. The specification thus prepared does not limit the invention to the specific terms presented. Therefore, although the present invention has been described in detail with reference to the above-described examples, those skilled in the art may make modifications, changes, and modifications to the present examples without departing from the scope of the present invention. Therefore, the scope of the present invention should not be defined by the described examples, but by the claims.

Claims (10)

A pre-processing module for generating image data for analysis by degrading original data, which is image data captured at high resolution, to low resolution;
a detection neural network module that selects a local area in which abnormal behavior exists from the image data for analysis as an abnormal behavior detection area;
a local data extraction module extracting local data for a region corresponding to the abnormal behavior detection region selected by the detection deep neural network module from the original data;
A video analysis device for detecting abnormal behavior, comprising: a classified neural network module that determines whether or not there is an abnormal behavior by taking the local data as an input.
According to claim 1,
Further comprising a detection learning module for learning the detection neural network module,
The detection learning module,
Characterized in that, when the detection neural network module outputs a latent vector representing a region in which an abnormal behavior exists in the input image data, the detection neural network module is trained so that a difference between the latent vector and a preset expected value is minimized. Video analysis device for detecting abnormal behavior.
According to claim 2,
The video analysis device for detecting abnormal behavior, characterized in that the latent vector includes positional information about a region in which abnormal behavior exists in the input image data and probability information in which abnormal behavior exists in the region.
According to claim 1,
Further comprising a classification learning module for learning the classification neural network module,
The classification learning module,
Characterized in that, when the classification neural network module calculates an output value indicating whether or not an abnormal behavior is included in the input image data, the classification neural network module is trained so that a difference between the output value and a preset expected value is minimized. Image analysis device for motion detection.
Generating image data for analysis by a pre-processing module by degrading original data, which is image data captured at high resolution, to low resolution;
selecting, by a detection neural network module, a local area in which an abnormal behavior exists from the image data for analysis as an abnormal behavior detection area;
extracting, by a local data extraction module, local data for an area corresponding to the abnormal behavior detection area selected by the detection deep neural network module from the original data;
An image analysis method for detecting abnormal behavior, comprising: a classification neural network module determining whether the local data is an abnormal behavior by using the local data as an input.
According to claim 5,
Further comprising a detection learning step of learning the detection neural network module,
In the detection learning step,
generating deterioration data obtained by degrading input high resolution image data to low resolution by a preprocessing module;
outputting, by a detection neural network module, a latent vector indicating a region in which an abnormal behavior exists in the deterioration data;
characterized in that it comprises; learning the detection neural network module so that the difference between the latent vector and a preset expected value is minimized by the detection learning module;
Image analysis method for detecting abnormal behavior.
According to claim 6,
The video analysis method for detecting abnormal behavior, characterized in that the latent vector includes positional information about a region in which an abnormal behavior exists in the input degradation data and probability information in which an abnormal behavior exists in the region.
According to claim 5,
Characterized in that the pre-processing module further comprises the step of converting the degradation data into a plurality of degradation data through data augmentation,
Image analysis method for detecting abnormal behavior.
According to claim 5,
Further comprising a classification learning step of training the classification neural network module,
The classification learning step,
calculating, by the classification neural network module, an output value indicating whether an abnormal behavior is included in the input image data;
Characterized in that it comprises, the classification learning module learning the classification neural network module such that the difference between the output value output by the classification neural network module and a preset expected value is minimized by the classification learning module.
Image analysis method for detecting abnormal behavior.
A computer-readable recording medium on which a program for executing the image analysis method for detecting an abnormal behavior according to any one of claims 5 to 9 in a computer is recorded.

Images