KR102484198B1 - Method, apparatus and system for detecting abnormal event - Google Patents

Abstract

The present invention relates to an abnormal event detection method, apparatus and system. An abnormal event detection method according to an embodiment of the present invention is a method of detecting an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, which is performed in an electronic device, and is performed around a specific place. Acquiring a 3D depth image and IoT (Internet of Things) sensor data in real time; Obtained using an event recognition model pre-trained according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an anomalous event or the type of an anomalous event, respectively. Recognizing whether there is an abnormal event for an object in the image based on the 3D depth image; and determining whether the corresponding abnormal event has actually occurred using IoT sensor data when it is recognized as an abnormal event.

Description

Abnormal event detection method, apparatus and system {METHOD, APPARATUS AND SYSTEM FOR DETECTING ABNORMAL EVENT}

The present invention relates to a method, apparatus, and system for detecting an abnormal event, and more particularly, to a method, apparatus, and system for detecting an abnormal event during video monitoring.

The video surveillance system is a system for protecting life and property, such as crime prevention and disaster monitoring, based on analysis of images collected from CCTVs. These video surveillance systems recognize objects to be monitored, such as people and vehicles, from collected images, analyze their behavior patterns, and when abnormal events (e.g., intrusion, fire, violence, theft, dumping, etc.) occur, It is a system that delivers information about this to the monitor through various warning devices.

However, most of the conventional video monitoring systems (hereinafter, referred to as “prior art”) detect abnormal events using only RGB images. Accordingly, the prior art has a problem in that it is difficult to accurately and promptly detect a corresponding abnormal event in a complex environment and a low light environment such as a night scene.

In addition, in the prior art, various preprocessing such as object extraction through background removal must be necessarily performed when analyzing an image for detecting an abnormal event. Accordingly, the prior art has a problem in that it is difficult to apply to a real-time environment because detection of an abnormal event is inevitably delayed due to preprocessing.

KR 10-1690050 B

In order to solve the problems of the prior art as described above, an object of the present invention is to provide a method, apparatus, and system capable of quickly and accurately detecting abnormal events during video surveillance in a real-time environment.

However, the problem to be solved by the present invention is not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the description below. There will be.

An abnormal event prevention method according to an embodiment of the present invention for solving the above problems is performed in an electronic device and detects an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system. A detection method comprising: obtaining a 3D depth image and IoT (Internet of Things) sensor data in real time around a specific place; Obtained using an event recognition model pre-trained according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an anomalous event or the type of an anomalous event, respectively. Recognizing whether there is an abnormal event for an object in the image based on the 3D depth image; and determining whether the corresponding abnormal event has actually occurred using IoT sensor data when it is recognized as an abnormal event.

The event recognition model may include a convolution layer each including a filter for recognizing an abnormal event and a filter for recognizing a human body joint.

The event recognition model may be a model in which weights of a convolution layer are updated to reduce an error between a position of a human body joint in a current image frame and a position of a human body joint in an abnormal event by using human joint information.

The event recognition model may be a model in which weights of a convolution layer are updated such that a total loss according to a loss for abnormal event classification and a loss for joint recognition is minimized.

The IoT sensor data may be data measured for the state of the person in the video by the IoT sensor possessed by the person.

The IoT sensor may be included in a wearable device that contacts, attaches to, wears, or is inserted into a body part of a corresponding person.

The IoT sensor may be any one of a heart rate sensor, an electrocardiogram sensor, an oxygen sensor, a skin conduction sensor, or a skin temperature sensor.

In the step of recognizing, it is possible to recognize whether or not there is an abnormal event by dividing continuous 3D depth images acquired in real time by a specific time unit according to a sliding window.

The abnormal event detection method according to an embodiment of the present invention may further include generating notification data when it is determined that an abnormal event has actually occurred.

An abnormal event detection apparatus according to an embodiment of the present invention is a device for detecting an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, and includes a 3D depth image around a specific place. And IoT (Internet of Things) communication unit for receiving sensor data in real time; a storage unit for storing an event recognition model pre-learned according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an anomalous event is present or a type of an anomalous event, respectively; and a control unit controlling the detection of an abnormal event using the information received by the communication unit and the information stored in the storage unit.

The control unit may recognize whether or not an abnormal event occurs for an object in the image based on the received 3D depth image by using the stored event recognition model, and when it is recognized as an abnormal event, the corresponding IoT sensor data is used. It is possible to determine whether an abnormal event has actually occurred.

An image monitoring system according to an embodiment of the present invention includes a 3D depth camera for capturing a 3D depth image; An IoT sensor located around a shooting position of a 3D depth camera to measure Internet of Things (IoT) sensor data; and a control unit receiving the 3D depth image and sensor data in real time and controlling the detection of an abnormal event for an object in the real time 3D depth image.

The control unit uses an event recognition model pre-learned according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an abnormal event occurs or the type of an abnormal event, respectively. , Based on the received 3D depth image, it is possible to recognize whether there is an abnormal event for the object in the image, and when it is recognized as an abnormal event, it is possible to determine whether the corresponding abnormal event has actually occurred using the received sensor data. .

Since the present invention configured as described above analyzes an abnormal event through a 3D depth image without a pre-processing process, it has an advantage of quickly detecting an abnormal event even in a real-time environment.

In addition, the present invention has an advantage in that an abnormal event can be accurately detected even in a low-light environment such as at night by considering human joint information and IoT sensor data together with a 3D depth image.

In addition, the present invention has the advantage of guaranteeing personal safety and property and preventing crime by installing in a security weak environment and interworking with a video surveillance system.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 shows a block configuration diagram of a video surveillance system 10 according to an embodiment of the present invention.
2 shows an example of a general RGB image and a 3D depth image.
3 shows a block diagram of a detection device 400 according to an embodiment of the present invention.
Figure 4 shows a block diagram of the control unit 450 in the detection device 400 according to an embodiment of the present invention.
5 shows a learning process (S101 to S104) of an event recognition model.
6 shows an example of the structure of an event recognition model learned based on a deep neural network (DNN).
7 shows an abnormal event detection process (S201 to S205) using the learned event recognition model.
8 shows an example of a case where a sliding window is applied to an event recognition model learned based on a deep neural network (DNN).

The above objects and means of the present invention and the effects thereof will become clearer through the following detailed description in conjunction with the accompanying drawings, and accordingly, those skilled in the art to which the present invention belongs can easily understand the technical idea of the present invention. will be able to carry out In addition, in describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms in some cases unless otherwise specified in the text. In this specification, terms such as "comprise", "have", "provide" or "have" do not exclude the presence or addition of one or more other elements other than the mentioned elements.

In this specification, terms such as “or” and “at least one” may represent one of the words listed together, or a combination of two or more. For example, "or B" and "at least one of B" may include only one of A or B, or may include both A and B.

In this specification, descriptions following "for example" may not exactly match the information presented, such as cited characteristics, variables, or values, and tolerances, measurement errors, limits of measurement accuracy and other commonly known factors It should not be limited to the embodiments of the present invention according to various embodiments of the present invention with effects such as modifications including.

In this specification, when a component is described as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as 'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

In the present specification, when an element is described as being 'on' or 'in contact with' another element, it may be in direct contact with or connected to the other element, but another element may be present in the middle. It should be understood that On the other hand, if an element is described as being 'directly on' or 'directly in contact with' another element, it may be understood that another element in the middle does not exist. Other expressions describing the relationship between components, such as 'between' and 'directly between', can be interpreted similarly.

In this specification, terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. In addition, the above terms should not be interpreted as limiting the order of each component, and may be used for the purpose of distinguishing one component from another. For example, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'.

Unless otherwise defined, all terms used in this specification may be used in a meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless explicitly specifically defined.

Hereinafter, a preferred embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

1 shows a block configuration diagram of a video surveillance system 10 according to an embodiment of the present invention.

The video monitoring system 10 according to an embodiment of the present invention is a system for protecting life and property such as crime prevention and disaster monitoring. That is, the video monitoring system 10 is based on the collected real-time 3D depth image and sensor data, and an abnormal event (i.e., a person or object appearing in the image) of the object to be monitored in the 3D depth image (i.e., person or object appearing in the image). It is a system that detects an unusual situation, a crisis situation, an emergency situation, etc.) and delivers a notification to the supervisor through the warning device 300 when an abnormal event occurs. In particular, when the object to be monitored is a person, the abnormal event may be that the behavior of the person is strange, and the person's personal status may be endangered (eg, fight, fall, etc.).

As shown in FIG. 1 , the video monitoring system 10 may include a 3D depth camera 100, an Internet of Things (IoT) sensor 200, a warning device 300, and a detection device 400. can

2 shows an example of a general RGB image and a 3D depth image.

The 3D depth camera 100 is a device installed around an object to be monitored to capture and collect a 3D depth image of the object to be monitored, and transmits the measured 3D depth image to the detection device 400 . That is, referring to FIG. 2 , since a 3D depth image includes depth information for each pixel in a photographed screen, it includes image information different from an RGB image obtained by a general optical camera. That is, the RGB image includes information about RGB luminance and the like for each pixel in the captured screen. That is, according to the present invention using a 3D depth image, there is an advantage in that an abnormal event can be accurately detected even in a low-light environment such as at night compared to the case of using an RGB image. For example, a 3D depth image may be obtained by a moiré technique, a stereo imaging technique, or laser measurement, but is not limited thereto.

The IoT sensor 200 is located around the 3D depth camera 100 to sense a monitoring object or various information about its surroundings, and generates sensor data and transmits it to the detection device 400. The IoT sensor 200 is a sensor (hereinafter, referred to as a “first sensor”) that detects information about the surroundings of a monitoring object, or a wearable device possessed by a person in an image to obtain information about the person. It may be a sensor (hereinafter referred to as “second sensor”) that senses. At this time, “possessed” may mean contact, attachment, wear, or insertion into a part of the body of the person in the video.

For example, wearable devices include electronic gloves, electronic glasses, head-mounted-devices (HMDs), electronic clothing, electronic bracelets, electronic necklaces, electronic appcessories, smart watches, or smart glasses. glass), etc., but is not limited thereto.

For example, the first sensor may include an illuminance sensor, an illuminance sensor, a humidity sensor, a leak sensor, a vibration sensor, or a geomagnetic sensor, but is not limited thereto. In addition, the second sensor may be a heart rate sensor, an electrocardiogram (ECG) sensor, an oxygen sensor (measuring oxygen in the blood), a skin conduction sensor (measuring current response of the skin, measuring the degree of sweating), or a skin temperature sensor (measuring user temperature). etc., but is not limited thereto.

The warning device 300 is a device that generates corresponding notification information according to notification data transmitted from the detection device 400 when the detection device 400 finally determines that an abnormal event has occurred and delivers it to the supervisor. For example, this notification information may be visual information (generating a warning screen) or auditory information (generating a warning sound), but is not limited thereto.

3 shows a block diagram of a detection device 400 according to an embodiment of the present invention.

The detection device 400 is a device that detects whether an abnormal event occurs for an object in an image based on information collected in real time from the 3D depth camera 100 and the IoT sensor 200, and is capable of computing. It can be an electronic device or a computing network.

For example, the electronic device includes a desktop personal computer, a laptop personal computer, a tablet personal computer, a netbook computer, a workstation, and a personal digital assistant (PDA). , a smart phone (smartphone), a smart pad (smartpad), or a mobile phone (mobile phone), etc., but is not limited thereto.

As shown in FIG. 3 , the detection device 400 may include an input unit 410, a communication unit 420, a display 430, a memory 440, and a control unit 450.

The input unit 410 generates input data in response to inputs of various users (such as a supervisor) and may include various input means. For example, the input unit 410 may include a keyboard, a key pad, a dome switch, a touch panel, a touch key, a touch pad, and a mouse. (mouse), menu button (menu button), etc. may be included, but is not limited thereto.

The communication unit 420 is a component that communicates with other devices. For example, the communication unit 420 may perform 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), Bluetooth, bluetooth low energy (BLE), near field communication (NFC), Wireless communication such as WiFi communication or wired communication such as cable communication may be performed, but is not limited thereto. In particular, the communication unit 420 may receive a real-time 3D depth image from the 3D depth camera 100 and real-time sensor data from the IoT sensor 200 . Also, the communication unit 420 may transmit notification data to the warning device 300 .

The display 430 displays various image data on a screen, and may be composed of a non-emissive panel or a light-emitting panel. For example, the display 430 may include a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, and a micro electromechanical system (MEMS). mechanical systems) display, or electronic paper (electronic paper) display, etc. may be included, but is not limited thereto. Also, the display 430 may be combined with the input unit 410 and implemented as a touch screen or the like.

The memory 440 stores various information necessary for the operation of the detection device 400. The stored information may include, but is not limited to, a received 3D depth image, received sensor data, an event recognition model, and program information related to an abnormal event detection method described later. For example, the memory 440 may be of a hard disk type, a magnetic media type, a compact disc read only memory (CD-ROM), or an optical media type according to its type. ), Sagneto-optical media type, Sultimedia card micro type, flash memory type, read only memory type, or RAM type (random access memory type), etc., but is not limited thereto. In addition, the memory 440 may be a cache, a buffer, a main memory, an auxiliary memory, or a separately provided storage system depending on its purpose/location, but is not limited thereto.

The controller 450 may perform various control operations of the detection device 400 . That is, the control unit 450 may control the execution of an abnormal event detection method to be described later, and the remaining components of the detection device 400, that is, the input unit 410, the communication unit 420, the display 430, and the memory 440 etc. can be controlled. For example, the controller 450 may include, but is not limited to, a processor that is hardware or a process that is software that is executed in the corresponding processor.

Figure 4 shows a block diagram of the control unit 450 in the detection device 400 according to an embodiment of the present invention. 5 and 7 are flowcharts of an abnormal event detection method according to an embodiment of the present invention. That is, FIG. 5 shows the learning process (S101 to S104) of the event recognition model, and FIG. 7 shows the abnormal event detection process (S201 to S205) using the learned event recognition model.

The control unit 450 controls the execution of the abnormal event detection method according to an embodiment of the present invention, and as shown in FIG. 4, the learning unit 451, the model analysis unit 452, and the sensor analysis unit 453 And it may include a notification generating unit 454. For example, the learning unit 451, the model analysis unit 452, the sensor analysis unit 453, and the notification generation unit 454 are hardware components of the control unit 450 or software processes executed by the control unit 450. It may be, but is not limited thereto.

On the other hand, the event recognition model is a deep learning model learned according to a deep learning technique through training data of input data (or a pair of input data and output data). For example, deep learning techniques include Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Q-Networks, etc. It can be done, but is not limited thereto.

An event recognition model includes a plurality of layers and has a function for a relationship between input data and output data. That is, when input data is input to the event recognition model, output data according to a corresponding function may be output.

The event recognition model expresses the relationship between input data and output data in multiple layers (ie, layers), and these multiple expression layers are also referred to as “neural networks”. Each layer in the neural network is composed of at least one filter, and each filter has a matrix of weights. That is, each element (pixel) in the matrix of the corresponding filter may correspond to a weight value. These filters can have a 2D or 3D matrix. That is, the two-dimensional matrix has a total of h'×w' (where h' and w' are natural numbers) weight elements, which are also referred to as “dimensional tensors”. In addition, the 3D matrix has a total of h×w×d (where h, w, and d are natural numbers) weight elements, which are also referred to as “dimensional tensors”. If there are a plurality of m filters of a three-dimensional matrix (h × w × d) in one layer (m is a natural number of 2 or more), a total of h × w × d × n weight elements are provided, This is also referred to as a “dimensional tensor”.

Hereinafter, an abnormal event detection method according to the present invention will be described in detail.

<Learning process>

First, referring to FIG. 5 , the learning process according to the present invention may include S101 to S104 and may be controlled by the learning unit 451 . That is, the learning unit 451 may perform a learning process of generating an event recognition model, which is a deep learning model for detecting abnormal events, using the 3D depth image and human body joint information acquired from the 3D video camera 100. there is.

That is, in S101, a 3D depth image is obtained from the 3D image camera 100, and human body joint information for a person appearing in the 3D depth image is obtained. In this case, the human body joint information is obtained through image processing of the corresponding 3D depth image, and may be information about joint positions of a person on a corresponding screen, and may be image information.

For example, human body joint information may be acquired using various human body joint extraction algorithms. Alternatively, the human body joint information may be acquired using a deep learning model learned through a deep learning technique. In this case, the deep learning model is trained according to a deep learning technique using training data including input data of a 3D depth image including a person and result data on the human body joint information of the person in the 3D depth image. can When a 3D depth image is input, the deep learning model trained in this way may generate human joint information for the 3D depth image as a result value.

In operation S102, after classifying the acquired 3D depth image and human body joint information for each abnormal event, learning data GT (Ground-Truth) for data learning is generated. That is, the training data may include input data for a 3D depth image and result data for whether or not an abnormal event occurs or the type of an abnormal event associated with the input data in pairs. In addition, human body joint information related thereto may be matched to each pair of learning data.

In S103, learning is performed according to a deep learning technique using the generated training data. In particular, by updating network weights using body joint information matched therewith in addition to learning data during learning, the accuracy of event recognition (classification) can be further improved.

In S104, an event recognition model, which is a deep learning model, may be generated as a learning result.

6 shows an example of the structure of an event recognition model learned based on a deep neural network (DNN).

For example, the event recognition model, as shown in FIG. 6 , may be a deep learning model in which learning is performed using a 3D depth image and human body joint information based on a deep neural network (DNN). That is, in S103, deep learning may be performed by inputting a 3D depth image (eg, spatio-temporal analysis) and the human joint information image (eg, spatio analysis) to the DNN.

In particular, the DNN includes a convolution layer for performing joint recognition unlike conventional action recognition models. That is, the convolution layer includes a filter for recognizing (classifying) abnormal events (hereinafter referred to as “first filter”) and a filter for recognizing (classifying) human joints (hereinafter referred to as “second filter”). includes each

At this time, in order to improve the accuracy of recognizing the abnormal event, during learning, not only the network weight of the first filter is updated so that the error between the correct action (event) and the inferred action (event) is minimized, but also in the current image frame. Network weights of the second filter may be updated so that an error between the joint position of the object (human) and the correct joint position is minimized. That is, weights of the second filter may be updated to reduce an error between the position of the human body joints in the current image frame and the position of the human body joints in the abnormal event by using the human body joint information.

The 3D depth image V includes a plurality of consecutive frames F, which are V={F ₁ , F ₂ , . . . It can be expressed as F _N } (provided that N is a natural number of 2 or more). In this case, in the event recognition model, the loss for abnormal event recognition, that is, the abnormal event classification loss (classification loss) (L _C ) can be expressed as in Equation (1) below.

(1)

(One)

However, in equation (1), x _k is a key-point logits, z _k is a key-point target, q is a positive weight, and σ is a sigmoid function function), respectively.

In addition, in the event recognition model, the loss of joint recognition (key-point loss) (L _K ) can be expressed as in Equation (2) below.

(2)

(2)

However, in equation (2), y _i 'is the class ground truth i _th element of one hot vector, y _i is the i th logit element of the network (i _th logit element from network) respectively.

In addition, in the event recognition model, the total loss (L) according to L _C and L _K can be expressed as Equation (3) below.

(3)

(3)

However, in Equation (3), w _c represents the weight of L _c (classification loss), and w _k represents the weight of L _k (key-point loss).

That is, the event recognition model is optimized by updating the network weight of each filter of the convolution layer so that the total loss (L) according to the loss for abnormal event recognition (L _C ) and the loss for joint recognition (L _K ) is minimized. can The optimized event recognition model is stored in the memory 440 and can be used in a later abnormal event detection process.

However, the learning process according to S101 to S104 described above may be performed in another device other than the detection device 400. In this case, the learning process according to S101 to S104 is performed in the other device, so that the event recognition model can be optimally learned. Thereafter, the learned event recognition model may be transmitted from another device to the detection device 400 through the communication unit 420 and stored in the memory 440 of the detection device 400 .

<Abnormal event detection process>

Next, referring to FIG. 7 , the abnormal event detection process according to the present invention may include steps S201 to S205, and is controlled by the model analyzer 452, sensor analyzer 453, and notification generator 454. It can be. That is, in the abnormal event detection process, an abnormal event for the 3D depth image and IoT sensor data occurring in the real-time environment is detected using the event recognition model generated as a result of the learning process. However, the human body joint information used in learning the event recognition model is not required in the abnormal event detection process. This is because human body joint information is not included in learning data when learning an abnormal event. That is, the learning data includes a 3D depth image excluding human joint information. However, human body joint information for a specific abnormal event is only used to update network weights when learning an abnormal event. Accordingly, the event recognition model for which learning has been completed is in a state in which network weights according to human joint information are already reflected.

In S201, real-time 3D depth image and IoT sensor data are received and input.

In S202, the model analyzer 452 detects an abnormal behavior based on the abnormal event recognition model. That is, by inputting the received 3D depth image to the abnormal event recognition model, it is possible to recognize whether or not an abnormal event occurs according to the result value.

In S203, when the sensor analysis unit 453 is recognized as an abnormal event in S202, it checks (determines) whether the corresponding abnormal event actually occurs using IoT sensor data. That is, IoT sensor data is not used to detect the type of abnormal event, but is used to check whether the abnormal event recognized by the abnormal event model has actually occurred.

For example, the IoT sensor 200 of the heart rate sensor possessed by the person in the screen may be utilized. In this case, when an abnormal event of “fight” is recognized in S202, if the IoT sensor data of the heart rate sensor shows a normal pattern, it is determined that the abnormal event of “fight” has not actually occurred, and the IoT sensor data is such as a rapid rise in heart rate. If an abnormal pattern is shown, it can be determined that the abnormal event of “fight” actually occurred.

In addition, when recognizing the abnormal event of "fall" in S202, if the IoT sensor data of the heart rate sensor shows a normal pattern, it is determined that the abnormal event of "fall" has not actually occurred, and the IoT sensor data indicates a rapid rise in heart rate or a decrease in heart rate. If an abnormal pattern such as the above is shown, it can be determined that the “fall” event has actually occurred.

However, in S204, it is finally determined whether an abnormal event has occurred. That is, when the abnormal event is recognized through the event recognition model and the occurrence of the corresponding abnormal event is confirmed through the IoT sensor data, it is determined that the abnormal event has occurred, and S205 is performed. If the abnormal event is not recognized through the event recognition model or the occurrence of the abnormal event is not confirmed through the IoT sensor data, the process returns to S201 and repeats the above-described process for the next real-time video and IoT sensor.

8 shows an example of a case where a sliding window is applied to an event recognition model learned based on a deep neural network (DNN).

In particular, as shown in FIG. 8 , S201 to S204 may be performed by grafting a sliding window technique. That is, the sliding window is for analyzing the start and end of an abnormal event, and is a technique for detecting an abnormal event for each divided unit by dividing frames of a continuous 3D depth image into specific time units. In this case, the search time and efficiency can be further improved.

In S205 , when it is determined that the abnormal event has actually occurred, the notification generation unit 454 may generate and transmit notification data to the warning device 300 . Of course, in addition to the warning device 300, the notification information according to the corresponding notification data may be delivered to the supervisor using the display 430 provided in the detection device 400 itself or another output unit.

The present invention configured as described above can detect abnormal events more quickly in an end-to-end (e2e) structure without image pre-processing, and thus can be applied to a real-time environment.

On the other hand, in the case of a model trained according to a machine learning technique based on only images as in the prior art, since only images are considered, the accuracy of event detection may be lowered. Accordingly, the present invention can more accurately recognize an abnormal event than in the prior art through an abnormal event recognition technology based on complex sensor data at night and in a complex environment. That is, according to the present invention, human body joint information and IoT sensor data are considered together with the 3D depth image, it is possible to accurately detect an abnormal event even in a low-light environment such as at night.

In particular, the network weight of the first filter of the convolution layer is updated by learning using training data based on the 3D depth image, and the network weight of the second filter of the convolution layer is updated using human body joint information during learning. update As a result, the learned event recognition model can further improve the accuracy of abnormal event recognition. The event recognition model learned in this way can generate a result value for event recognition (classification) even if only a 3D depth image is input thereafter without human body joint information. That is, while a 3D depth image and its joint information are required during learning, only a 3D depth image is required when using the learned event recognition model thereafter. Accordingly, the present invention can quickly and accurately detect abnormal events during video monitoring in a real-time environment. In particular, the present invention is installed in a security vulnerable environment and interlocks with a video surveillance system, enabling detection of abnormal events more accurately and quickly than before even in low-light environments such as nighttime and complex event situations, thereby guaranteeing personal safety and property crime can be prevented.

In the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention is not limited to the described embodiments, and should be defined by the following claims and equivalents thereof.

100: 3D depth camera 200: IoT sensor
300: warning device 400: detection device
410: input unit 420: communication unit
430: display unit 440: memory
450: control unit 451: learning unit
452: model analysis unit 453: sensor analysis unit
454: notification generating unit

Claims (11)

A method of detecting an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, performed in an electronic device, comprising:
Acquiring a 3D depth image and IoT (Internet of Things) sensor data in real time around a specific place;
Obtained using an event recognition model pre-trained according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an anomalous event or the type of an anomalous event, respectively. Recognizing whether there is an abnormal event for an object in the image based on the 3D depth image; and
When it is recognized as an abnormal event, determining whether the corresponding abnormal event actually occurs using IoT sensor data;
The event recognition model is learned based on the input data for the 3D depth image, and the network weight is updated using additional information about the human body joints of the person in the 3D depth image of the input data,
The recognizing is a method of recognizing whether or not the abnormal event occurs only with the acquired 3D depth image without additional information about the body joints of the person in the acquired 3D depth image.
According to claim 1,
The event recognition model includes a convolution layer each including a filter for recognizing an abnormal event and a filter for recognizing human body joints in the 3D depth image.
According to claim 2,
The method of claim 1 , wherein the event recognition model is a model in which weights of a convolution layer are updated to reduce an error between a position of a human body joint in a current image frame and a position of a human body joint in an abnormal event using human body joint information.
According to claim 3,
The event recognition model is a model in which weights of convolution layers are updated such that a total loss according to a loss for abnormal event classification and a loss for joint recognition is minimized.
According to claim 1,
The IoT sensor data is data measured for the state of the person in the IoT sensor possessed by the person in the video.
According to claim 5,
The IoT sensor is included in a wearable device that is contacted, attached, worn, or inserted into a part of the person's body.
According to claim 5,
The IoT sensor is any one of a heart rate sensor, an electrocardiogram sensor, an oxygen sensor, a skin conduction sensor, or a skin temperature sensor.
According to claim 1,
The step of recognizing is a method of recognizing whether or not there is an abnormal event by dividing continuous 3D depth images acquired in real time by a specific time unit according to a sliding window.
According to claim 1,
The method further comprising generating notification data when it is determined that the abnormal event has actually occurred.
A device for detecting an abnormal event for an object in a real-time 3D depth image captured by a 3D depth camera of a video surveillance system, comprising:
A communication unit for receiving a 3D depth image and IoT (Internet of Things) sensor data in real time around a specific place;
a storage unit for storing an event recognition model pre-learned according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an anomalous event is present or a type of an anomalous event, respectively; and
A control unit controlling the detection of an abnormal event using the information received from the communication unit and the information stored in the storage unit;
The controller uses the stored event recognition model to recognize whether or not there is an abnormal event for an object in the image based on the received 3D depth image, and when it is recognized as an abnormal event, the corresponding abnormal event is detected using the received IoT sensor data. determine whether the actual occurrence of
The event recognition model is learned based on the input data for the 3D depth image, and the network weight is updated using additional information about the human body joints of the person in the 3D depth image of the input data,
wherein the control unit recognizes whether or not the abnormal event occurs only with the received 3D depth image without additional information about the body joints of the person in the received 3D depth image.
a 3D depth camera that captures a 3D depth image;
An IoT sensor located around the shooting position of the 3D depth camera to measure Internet of Things (IoT) sensor data; and
A control unit for receiving the 3D depth image and sensor data in real time and controlling the detection of an abnormal event for an object in the real-time 3D depth image;
The control unit uses an event recognition model pre-learned according to a deep learning technique through learning data including input data for a 3D depth image and result data on whether or not an abnormal event occurs or the type of an abnormal event, respectively. , Based on the received 3D depth image, whether or not an abnormal event is recognized for the object in the image, and when it is recognized as an abnormal event, it is determined whether the corresponding abnormal event actually occurs using the received sensor data,
The event recognition model is learned based on the input data for the 3D depth image, and the network weight is updated using additional information about the human body joints of the person in the 3D depth image of the input data,
The video surveillance system of claim 1 , wherein the control unit recognizes whether or not the abnormal event occurs only with the received 3D depth image without additional information about the body joints of the person in the received 3D depth image.

Images