KR20220060587A - Method, apparatus and system for detecting abnormal event - Google Patents

Abstract

The present invention relates to a method, apparatus, and system for detecting an abnormal event to quickly detect abnormal events even in a real-time environment. According to one embodiment of the present invention, the abnormal event detection method detects an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system. The method comprises the following steps of: acquiring a 3D depth image and Internet of Things (IoT) sensor data in real time around a predetermined place; using a learned event recognition model learned through the training data including the input data for the 3D depth image and the result data on whether an abnormal event occurs or the type of the abnormal event according to a deep learning technique to recognize whether an abnormal event for the object in the image occurs on the basis of the acquired 3D depth image; and using IoT sensor data to determine whether a corresponding abnormal event actually occurs when the abnormal event is recognized.

Description

Abnormal event detection method, device and system {METHOD, APPARATUS AND SYSTEM FOR DETECTING ABNORMAL EVENT}

The present invention relates to a method, apparatus and system for detecting an abnormal event, and more particularly, to a method, apparatus and system for detecting an abnormal event during video monitoring.

The video surveillance system is a system for protecting lives and property, such as crime prevention and disaster monitoring, based on the analysis of images collected from CCTV. These video surveillance systems recognize objects to be monitored, such as people and vehicles, from the collected images, analyze their behavior patterns, and when abnormal events (eg, intrusion, fire, violence, theft, speculation, etc.) occur, It is a system that delivers information about this to the monitor through various warning devices.

However, most conventional video surveillance systems (hereinafter, referred to as “prior art”) use only RGB images to detect abnormal events. Accordingly, the prior art has a problem in that it is difficult to accurately and quickly detect a corresponding abnormal event in a complex environment or a low-illuminance environment such as a night scene.

In addition, in the prior art, various pre-processing, such as object extraction through background removal, must be essentially performed when analyzing an image for detecting an abnormal event. Accordingly, the prior art has a problem in that it is difficult to apply to a real-time environment because the detection of an abnormal event is inevitably delayed due to the pre-processing.

KR 10-1690050 B

In order to solve the problems of the prior art as described above, it is an object of the present invention to provide a method, apparatus and system capable of quickly and accurately detecting an abnormal event during video monitoring in a real-time environment.

However, the problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

An abnormal event prevention method according to an embodiment of the present invention for solving the above problems is performed in an electronic device, and an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of an image monitoring system is detected. A detection method comprising: acquiring a three-dimensional depth image and Internet of Things (IoT) sensor data in real time around a specific place; Through training data including input data for 3D depth image and result data for abnormal event or abnormal event type, respectively, using an event recognition model that has been previously learned according to a deep learning technique, Recognizing whether there is an abnormal event on the object in the image based on the 3D depth image; and if recognized as an abnormal event, determining whether the corresponding abnormal event actually occurs using IoT sensor data.

The event recognition model may include a convolution layer each including a filter for recognizing an abnormal event and a filter for recognizing a human joint.

The event recognition model may be a model in which the weight of the convolution layer is updated to reduce an error between the position of the human joint in the current image frame and the position of the human joint of the abnormal event by using the human joint information.

The event recognition model may be a model in which the weight of the convolutional layer is updated so that a total loss according to the loss for abnormal event classification and the loss for joint recognition is minimized.

The IoT sensor data may be data measured on the state of the person in the IoT sensor possessed by the person in the image.

The IoT sensor may be included in a wearable device that is in contact with, attached to, worn, or inserted into a body part of the person.

The IoT sensor may be any one of a heart rate sensor, an electrocardiogram sensor, an oxygen sensor, a skin conduction sensor, and a skin temperature sensor.

In the recognizing step, an abnormal event may be recognized by dividing the continuous 3D depth image acquired in real time by a specific time unit according to a sliding window.

The abnormal event detection method according to an embodiment of the present invention may further include generating notification data when it is determined that the abnormal event has actually occurred.

An abnormal event detection apparatus according to an embodiment of the present invention is an apparatus for detecting an abnormal event with respect to an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, and a 3D depth image around a specific place. and a communication unit configured to receive Internet of Things (IoT) sensor data in real time; a storage unit for storing an event recognition model previously learned according to a deep learning technique through learning data including input data for a three-dimensional depth image and result data on whether an abnormal event or an abnormal event type is present; and a control unit controlling the detection of an abnormal event using the information received in the communication unit and the information stored in the storage unit.

The control unit can recognize whether an abnormal event for an object in the image is based on the received 3D depth image using the stored event recognition model, and when recognized as an abnormal event, the corresponding IoT sensor data is used It can be determined whether an abnormal event actually occurs.

An image monitoring system according to an embodiment of the present invention includes: a three-dimensional depth camera for photographing a three-dimensional depth image; IoT sensor for measuring IoT (Internet of Things) sensor data by being located around the shooting position of the three-dimensional depth camera; and a controller configured to receive a 3D depth image and sensor data in real time, and control detection of an abnormal event with respect to an object in the real-time 3D depth image.

The control unit uses an event recognition model previously learned according to a deep learning technique through learning data including input data for a three-dimensional depth image and result data on whether an abnormal event or an abnormal event type is present, respectively. , it is possible to recognize whether an abnormal event for an object in the image is based on the received 3D depth image, and when it is recognized as an abnormal event, it is possible to determine whether the abnormal event actually occurs using the received sensor data .

The present invention configured as described above has an advantage in that it is possible to quickly detect an abnormal event even in a real-time environment because an abnormal event is analyzed through a three-dimensional depth image without a pre-processing process.

In addition, the present invention has an advantage in that it is possible to accurately detect an abnormal event even in a low-illuminance environment such as at night by considering human joint information and IoT sensor data along with a three-dimensional depth image.

In addition, the present invention has the advantage of being installed in a security-vulnerable environment and interworking with the video surveillance system, thereby ensuring personal and property protection and preventing crime.

The effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

1 shows a block diagram of a video surveillance system 10 according to an embodiment of the present invention.
2 shows an example of a general RGB image and a 3D depth image.
3 is a block diagram showing a detection apparatus 400 according to an embodiment of the present invention.
4 is a block diagram showing the control unit 450 in the detection apparatus 400 according to an embodiment of the present invention.
5 shows the learning process (S101 to S104) of the event recognition model.
6 shows an example of the structure of an event recognition model learned based on a deep neural network (DNN).
7 shows an abnormal event detection process ( S201 to S205 ) using the learned event recognition model.
8 shows an example of a case in which a sliding window is applied to an event recognition model learned based on a deep neural network (DNN).

The above object and means of the present invention and its effects will become more apparent through the following detailed description in relation to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form as the case may be, unless otherwise specified in the phrase. In this specification, terms such as “include”, “provide”, “provide” or “have” do not exclude the presence or addition of one or more other components other than the mentioned components.

In this specification, terms such as “or” and “at least one” may indicate one of the words listed together, or a combination of two or more. For example, “or B” and “at least one of B” may include only one of A or B, or both A and B.

In the present specification, descriptions according to “for example” and the like may not exactly match the information presented, such as recited properties, variables, or values, tolerances, measurement errors, limits of measurement accuracy, and other commonly known factors The embodiments of the present invention according to various embodiments of the present invention should not be limited by effects such as modifications including .

In this specification, when it is described that a certain element is 'connected' or 'connected' to another element, it may be directly connected or connected to the other element, but other elements may exist in between. It should be understood that there may be On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

In this specification, when it is described that a certain element is 'on' or 'in contact with' another element, it may be directly in contact with or connected to the other element, but another element may exist in the middle. It should be understood that On the other hand, when it is described that a certain element is 'directly on' or 'directly' of another element, it may be understood that another element does not exist in the middle. Other expressions describing the relationship between the elements, for example, 'between' and 'directly between', etc. may be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various components, but the components should not be limited by the above terms. In addition, the above terms should not be construed as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first component' may be referred to as a 'second component', and similarly, a 'second component' may also be referred to as a 'first component'.

Unless otherwise defined, all terms used herein may be used with meanings commonly understood by those of ordinary skill in the art to which the present invention pertains. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows a block diagram of a video surveillance system 10 according to an embodiment of the present invention.

The video surveillance system 10 according to an embodiment of the present invention is a system for protecting life and property, such as crime prevention and disaster monitoring. That is, the video monitoring system 10 is based on the collected real-time 3D depth image and sensor data, an abnormal event (ie, a person or object appearing in the image) for the object to be monitored in the 3D depth image (that is, the image). It is a system that detects a situation different from normal, crisis situation, emergency situation, etc.) and delivers a notification to a supervisor through the warning device 300 when a corresponding abnormal event occurs. In particular, when the object to be monitored is a person, the abnormal event may be a situation in which the person's behavior is strange and the person's identity is in jeopardy (eg, fight, collapse, etc.).

As shown in FIG. 1 , the video surveillance system 10 may include a three-dimensional depth camera 100 , an Internet of Things (IoT) sensor 200 , a warning device 300 , and a detection device 400 . can

2 shows an example of a general RGB image and a 3D depth image.

The 3D depth camera 100 is installed in the vicinity of a monitoring target object and captures and collects a 3D depth image of the monitoring object, and transmits the measured 3D depth image to the detection device 400 . That is, referring to FIG. 2 , since the 3D depth image includes information about the depth of each pixel on the captured screen, it includes image information different from the RGB image obtained from a general optical camera. That is, the RGB image includes information on the luminance of RGB for each pixel in the captured screen. That is, according to the present invention, since the 3D depth image is used, there is an advantage in that it is possible to accurately detect an abnormal event even in a low-illuminance environment such as at night, compared to the case of using an RGB image. For example, the 3D depth image may be obtained by a moiré technique, a stereo imaging technique, or laser measurement, but is not limited thereto.

The IoT sensor 200 is located in the vicinity of the 3D depth camera 100 to detect a monitoring object or various information about its surroundings, and generates sensor data and transmits it to the detection device 400 . The IoT sensor 200 is a sensor (hereinafter, referred to as a “first sensor”) that detects information about the surrounding object to be monitored, or is possessed by a person in the image, such as a wearable device, to receive information about the person. It may be a sensing sensor (hereinafter referred to as a “second sensor”). In this case, “possessed” may refer to being in contact with, attached to, worn, or inserted into a body part of a person in the image.

For example, the wearable device may be an electronic glove, electronic glasses, head-mounted-device (HMD), electronic garment, electronic bracelet, electronic necklace, electronic accessory, smart watch, or smart glass (smart watch). glass), but is not limited thereto.

For example, the first sensor may include, but is not limited to, an illuminance sensor, an illuminance sensor, a humidity sensor, a leakage sensor, a vibration sensor, or a geomagnetic sensor. In addition, the second sensor may be a heart rate sensor, an electrocardiogram (ECG) sensor, an oxygen sensor (measuring oxygen in the blood), a skin conduction sensor (measuring the current response of the skin, capable of measuring the amount of sweating), or a skin temperature sensor (measuring user temperature) and the like, but is not limited thereto.

The warning device 300 is a device that generates corresponding notification information according to notification data transmitted from the detection device 400 and delivers it to a supervisor when the detection device 400 finally determines that an abnormal event has occurred. For example, the notification information may be visual information (occurrence of a warning screen) or auditory information (occurrence of a warning sound), but is not limited thereto.

3 is a block diagram showing a detection apparatus 400 according to an embodiment of the present invention.

The detection device 400 is a device that detects whether an abnormal event has occurred for an object in an image based on information collected in real time from the 3D depth camera 100 and the IoT sensor 200, and is capable of computing It may be an electronic device or a computing network.

For example, the electronic device includes a desktop personal computer (PC), a laptop personal computer (PC), a tablet personal computer (PC), a netbook computer, a workstation, and a personal digital assistant (PDA). , a smartphone (smartphone), a smart pad (smartpad), or a mobile phone (mobile phone), etc., but is not limited thereto.

As shown in FIG. 3 , the detection device 400 may include an input unit 410 , a communication unit 420 , a display 430 , a memory 440 , and a control unit 450 .

The input unit 410 generates input data in response to input of various users (monitor, etc.), and may include various input means. For example, the input unit 410 may include a keyboard, a keypad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), a menu button (menu button) and the like may be included, but is not limited thereto.

The communication unit 420 is configured to communicate with other devices. For example, the communication unit 420 is 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energe (BLE), near field communication (NFC), Wireless communication such as Wi-Fi communication may be performed or wired communication such as cable communication may be performed, but is not limited thereto. In particular, the communication unit 420 may receive a 3D depth image in real time from the 3D depth camera 100 , and may receive sensor data in real time from the IoT sensor 200 . Also, the communication unit 420 may transmit notification data to the warning device 300 .

The display 430 displays various image data on a screen, and may be configured as a non-emissive panel or a light emitting panel. For example, the display 430 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or an electronic paper display, etc., but is not limited thereto. In addition, the display 430 may be implemented as a touch screen or the like in combination with the input unit 410 .

The memory 440 stores various types of information necessary for the operation of the detection device 400 . The storage information may include, but is not limited to, a received 3D depth image, received sensor data, an event recognition model, and program information related to an abnormal event detection method to be described later. For example, the memory 440 may be a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type depending on the type of the memory 440 . ), a Sagneto-optical media type, a multimedia card micro type, a flash memory type, a read only memory type, or a random access memory type), but is not limited thereto. In addition, the memory 440 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system according to its purpose/location, but is not limited thereto.

The controller 450 may perform various control operations of the detection device 400 . That is, the controller 450 may control the execution of an abnormal event detection method to be described later, and the remaining components of the detection device 400 , that is, the input unit 410 , the communication unit 420 , the display 430 , and the memory 440 . You can control actions such as For example, the controller 450 may include a processor that is hardware or a process that is software that is executed in the corresponding processor, but is not limited thereto.

4 is a block diagram showing the control unit 450 in the detection apparatus 400 according to an embodiment of the present invention. 5 and 7 are flowcharts of an abnormal event detection method according to an embodiment of the present invention. That is, FIG. 5 shows the learning process (S101 to S104) of the event recognition model, and FIG. 7 shows the abnormal event detection process (S201 to S205) using the learned event recognition model.

The control unit 450 controls the execution of the abnormal event detection method according to an embodiment of the present invention, and as shown in FIG. 4 , the learning unit 451 , the model analysis unit 452 , and the sensor analysis unit 453 . and a notification generator 454 . For example, the learning unit 451 , the model analysis unit 452 , the sensor analysis unit 453 , and the notification generation unit 454 are a hardware configuration of the control unit 450 or a process that is software performed by the control unit 450 . may be, but is not limited thereto.

On the other hand, the event recognition model is a deep learning model learned according to a deep learning technique through training data of input data (or input data and output data pair). For example, deep learning techniques include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, etc. can, but is not limited thereto.

The event recognition model includes a plurality of layers and has a function on the relationship between input data and output data. That is, when input data is input to the event recognition model, output data according to a corresponding function may be output.

The event recognition model expresses the relationship between input data and output data in multiple layers (ie, layers), and these multiple representation layers are also referred to as “neural networks”. Each layer in the neural network consists of at least one filter, and each filter has a matrix of weights. That is, each element (pixel) in the matrix of the corresponding filter may correspond to a weight value. Such filters may have a two-dimensional or three-dimensional matrix. That is, the two-dimensional matrix has a total of h'×w' (where h' and w' are natural numbers) number of weighting elements, which is also referred to as a “dimensional tensor”. In addition, the 3D matrix has a total of h×w×d weight elements (where h, w, and d are natural numbers), which is also referred to as a “dimensional tensor”. If m filters of a three-dimensional matrix (h×w×d) are provided in one layer (m is a natural number greater than or equal to 2), it has a total of h×w×d×n weighting elements, This is sometimes referred to as a “dimensional tensor”.

Hereinafter, the abnormal event detection method according to the present invention will be described in more detail.

<Learning process>

First, referring to FIG. 5 , the learning process according to the present invention may include steps S101 to S104 and may be controlled by the learning unit 451 . That is, the learning unit 451 may perform a learning process of generating an event recognition model, which is a deep learning model for detecting abnormal events, using the 3D depth image and human joint information obtained from the 3D image camera 100 . there is.

That is, in S101, a 3D depth image is acquired from the 3D imaging camera 100, and human joint information about a person displayed in the 3D depth image is acquired. In this case, the human joint information is obtained through image processing on the corresponding 3D depth image, and is information on the joint position of the person on the corresponding screen, and may be image information.

For example, human joint information may be acquired using various human joint extraction algorithms. Alternatively, the human joint information may be acquired using a deep learning model learned through a deep learning technique. In this case, the deep learning model is to be learned according to the deep learning technique using the input data of the 3D depth image including the person and the learning data including the result data on the joint information of the person in the 3D depth image. can When a three-dimensional depth image is input, the deep learning model learned in this way may generate human joint information about it as a result value.

In S102, after classifying the acquired 3D depth image and human joint information for each abnormal event, training data GT (Ground-Truth) for data learning is generated. That is, the training data may include input data for the 3D depth image and result data on whether or not an abnormal event is performed or the type of the abnormal event as a pair. In addition, each pair of learning data may be matched with related human joint information.

In S103, learning according to the deep learning technique is performed using the generated learning data. In particular, by updating the network weights using human joint information matched with the learning data in addition to the learning data during learning, the accuracy of recognizing (classifying) abnormal events can be further improved.

In S104, an event recognition model that is a deep learning model may be generated as a result of the learning.

6 shows an example of the structure of an event recognition model learned based on a deep neural network (DNN).

For example, as shown in FIG. 6 , the event recognition model may be a deep learning model in which learning is performed using a 3D depth image and human joint information based on a deep neural network (DNN). That is, in S103, deep learning may be performed by inputting a three-dimensional depth image (eg, spatio-temporal analysis) and the human joint information image (eg, spatio analysis) to the DNN.

In particular, the DNN includes a convolution layer for performing joint recognition differently from the conventional behavior recognition model. That is, the convolutional layer includes a filter for recognizing (classifying) abnormal events (hereinafter referred to as “first filter”) and a filter for recognizing (classifying) human joints (hereinafter referred to as “second filter”). include each.

At this time, in order to improve the accuracy of recognizing abnormal events, the network weight of the first filter is updated so that the error between the correct action (event) and the inferred action (event) is minimized during learning, as well as in the current image frame. The network weight of the second filter may be updated so that an error between the joint position of the object (person) and the correct joint position is minimized. That is, the weight of the second filter may be updated to reduce an error between the human joint position in the current image frame and the human joint position of the abnormal event using the human joint information.

The three-dimensional depth image (V) includes a plurality of consecutive frames (F), which are V={F ₁ , F ₂ , ... F _N } (however, N is a natural number greater than or equal to 2). In this case, in the event recognition model, the loss for the recognition of the abnormal event, that is, the classification loss ( _LC ) of the abnormal event may be expressed as in Equation (1) below.

(1)

(One)

However, in Equation (1), x _k is the key-point logits, z _k is the key-point targets, q is the positive weight, and σ is the sigmoid function. function) respectively.

In addition, in the event recognition model, the loss (key-point loss) (L _K ) for joint recognition can be expressed as the following Equation (2).

(2)

(2)

However, in Equation (2), y _i ' is the class ground truth i _th element of one hot vector, and y _i is the i _th logit element from the network. network) respectively.

In addition, in the event recognition model, the total loss (L) according to L _C and L _K can be expressed as Equation (3) below.

(3)

(3)

However, in Equation (3), w _c represents a weight of L _c (classification loss), and w _k represents a weight of L _k (key-point loss), respectively.

That is, the event recognition model can be optimized by updating the network weight of each filter of the convolutional layer so that the total loss (L) according to the loss (L _C ) and the loss (L _K ) for recognizing an anomaly is minimized. can The event recognition model optimized as described above is stored in the memory 440 and may be used later in the abnormal event detection process.

However, the learning process according to S101 to S104 described above may be performed in a device other than the detection device 400 . In this case, the learning process according to S101 to S104 may be performed in the corresponding other device to optimize the event recognition model. Thereafter, the learned event recognition model may be transmitted from another device to the detection device 400 through the communication unit 420 and stored in the memory 440 of the detection device 400 .

<Anomalous event detection process>

Next, referring to FIG. 7 , the abnormal event detection process according to the present invention may include S201 to S205 , and is controlled by the model analyzer 452 , the sensor analyzer 453 , and the notification generator 454 . can be That is, in the abnormal event detection process, an abnormal event for the 3D depth image and IoT sensor data generated in the real-time environment is detected using the event recognition model generated as a result of the learning process. However, the human joint information used in learning the event recognition model is not required in the abnormal event detection process. This is because human joint information is not included in the learning data during abnormal event learning. That is, the learning data includes a 3D depth image excluding information on human joints. However, the human joint information for a specific abnormal event is only used to update the network weight when learning the abnormal event. Accordingly, in the event recognition model that has been trained, the network weight according to the joint information of the human body is already reflected.

In S201, a real-time 3D depth image and IoT sensor data are received and input.

In S202, the model analysis unit 452 detects an abnormal behavior based on the abnormal event recognition model. That is, by inputting the received 3D depth image to the abnormal event recognition model, it is possible to recognize whether an abnormal event occurs according to the result value.

In S203, the sensor analysis unit 453 checks (determines) whether the abnormal event actually occurs by using IoT sensor data when it is recognized as an abnormal event in S202. That is, IoT sensor data is not used to detect the type of abnormal event, but is used to check whether the abnormal event recognized by the abnormal event model actually occurred.

For example, the IoT sensor 200 of the heart rate sensor possessed by the person in the screen may be utilized. In this case, when S202 recognizes an abnormal event called “fighting”, if the IoT sensor data of the heart rate sensor shows a normal pattern, it is determined that the abnormal event of “fighting” has not actually occurred, and the corresponding IoT sensor data If an abnormal pattern is shown, it can be determined that an abnormal event of “fighting” has actually occurred.

In addition, when S202 recognizes an abnormal event of “fall”, if the IoT sensor data of the heart rate sensor shows a normal pattern, it is determined that the abnormal event of “fall” has not actually occurred, and the corresponding IoT sensor data shows a sudden increase in heart rate or decrease in heart rate If an abnormal pattern such as such is shown, it can be determined that the “falling down” event has actually occurred.

However, in S204, it is finally determined whether an abnormal event has occurred. That is, when the abnormal event is recognized through the event recognition model and the occurrence of the corresponding abnormal event is confirmed through the IoT sensor data, it is determined that the abnormal event has occurred, and S205 is performed. If the abnormal event is not recognized through the event recognition model or the occurrence of the abnormal event is not confirmed through the IoT sensor data, the process returns to S201 and repeats the above-described process for the following real-time image and IoT sensor.

8 shows an example of a case in which a sliding window is applied to an event recognition model learned based on a deep neural network (DNN).

In particular, steps S201 to S204 may be performed by grafting a sliding window technology, as shown in FIG. 8 . That is, the sliding window is used to analyze the start and end of an abnormal event, and is a technology for dividing consecutive 3D depth image frames into specific time units and detecting abnormal events for each divided unit. In this case, the search time and efficiency can be further improved.

In S205 , when it is determined that the abnormal event has actually occurred, the notification generator 454 may generate notification data and transmit it to the warning device 300 . Of course, in addition to the warning device 300 , the notification information according to the corresponding notification data may be transmitted to the monitor using the display 430 or other output unit provided in the detection device 400 itself.

The present invention configured as described above can be applied to a real-time environment because it is possible to detect an anomaly more quickly in an e2e (end to end) structure without an image pre-processing process.

Meanwhile, in the case of a model trained according to a machine learning technique based on only an image as in the prior art, since only an image is considered, the accuracy of detecting an abnormal event may be lowered. Accordingly, the present invention can recognize an abnormal event more accurately than in the prior art through the complex sensor data-based abnormal event recognition technology at night and in a complex environment. That is, the present invention considers human joint information and IoT sensor data as well as a 3D depth image, so that it is possible to accurately detect an abnormal event even in a low-illuminance environment such as at night.

In particular, the network weight of the first filter of the convolutional layer is updated by learning using learning data based on a 3D depth image, but the network weight of the second filter of the convolutional layer is updated using human joint information during learning. update As a result, the learned event recognition model may further improve the accuracy of abnormal event recognition. The event recognition model learned in this way can generate a result value for event recognition (classification) beyond that by inputting only a three-dimensional depth image without information on human joints thereafter. That is, a 3D depth image and its joint information are required for learning, but only a 3D depth image is required when using the learned event recognition model. Therefore, according to the present invention, it is possible to quickly and accurately detect an abnormal event during video monitoring in a real-time environment. In particular, the present invention is installed in a security-vulnerable environment and interlocked with the video surveillance system, so that abnormal events can be detected more accurately and quickly than before, even in low-light environments and complex event situations such as at night, to ensure personal safety and property, and crime can be prevented.

In the detailed description of the present invention, although specific embodiments have been described, various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and their equivalents.

100: 3D depth camera 200: IoT sensor
300: warning device 400: detection device
410: input unit 420: communication unit
430: display unit 440: memory
450: control unit 451: learning unit
452: model analysis unit 453: sensor analysis unit
454: notification generator

Claims (11)

A method of detecting an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, which is performed in an electronic device, the method comprising:
acquiring a 3D depth image and Internet of Things (IoT) sensor data in real time around a specific place;
Through training data including input data for 3D depth image and result data for abnormal event or abnormal event type, respectively, acquired using an event recognition model learned according to deep learning technique Recognizing whether there is an abnormal event on the object in the image based on the 3D depth image; and
when it is recognized as an abnormal event, determining whether the abnormal event actually occurs using IoT sensor data;
How to include.
According to claim 1,
The event recognition model includes a convolution layer each including a filter for abnormal event recognition and a filter for human joint recognition.
3. The method of claim 2,
The event recognition model is a model in which the weight of the convolution layer is updated to reduce an error between the position of the human joint in the current image frame and the position of the human joint of the abnormal event by using the human joint information.
4. The method of claim 3,
The event recognition model is a model in which the weight of the convolutional layer is updated so that the total loss according to the loss for abnormal event classification and the loss for joint recognition is minimized.
According to claim 1,
The IoT sensor data is data measured on the state of the person in the IoT sensor possessed by the person in the image.
6. The method of claim 5,
The IoT sensor is a method included in a wearable device that is in contact with, attached to, worn, or inserted into a body part of the person.
6. The method of claim 5,
The IoT sensor is any one of a heart rate sensor, an electrocardiogram sensor, an oxygen sensor, a skin conduction sensor, or a skin temperature sensor.
According to claim 1,
The step of recognizing is a method of recognizing whether an abnormal event exists by dividing continuous 3D depth images acquired in real time into specific time units according to a sliding window.
According to claim 1,
When it is determined that the abnormal event has actually occurred, the method further comprising the step of generating notification data.
A device for detecting an abnormal event on an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, comprising:
a communication unit configured to receive a 3D depth image and Internet of Things (IoT) sensor data around a specific place in real time;
a storage unit for storing an event recognition model previously learned according to a deep learning technique through learning data including input data for a three-dimensional depth image and result data on whether an abnormal event or an abnormal event type is present; and
A control unit for controlling the detection of an abnormal event by using the information received in the communication unit and the information stored in the storage unit;
The control unit is
Using the stored event recognition model, based on the received 3D depth image, it recognizes whether there is an abnormal event on the object in the image,
When recognized as an abnormal event, a device that determines whether the abnormal event actually occurs by using the received IoT sensor data.
a three-dimensional depth camera for taking a three-dimensional depth image;
IoT sensor for measuring IoT (Internet of Things) sensor data by being located around the shooting position of the three-dimensional depth camera; and
A control unit for receiving a 3D depth image and sensor data in real time, and controlling the detection of an abnormal event for an object in the real-time 3D depth image;
The control unit is
Through the training data including input data for the 3D depth image and result data on the presence or absence of abnormal events or abnormal event types, respectively, the received event recognition model according to the deep learning technique is used. Recognizes whether there is an abnormal event on the object in the image based on the 3D depth image,
When an abnormal event is recognized, an image monitoring system that determines whether the abnormal event actually occurs using the received sensor data.

Images