KR102260123B1 - Apparatus for Sensing Event on Region of Interest and Driving Method Thereof - Google Patents

Abstract

The present invention relates to a local event detecting device and a method of driving the device, wherein the local event detecting device according to an embodiment of the present invention exists in an area of interest in a store or public place monitored by a plurality of photographing devices. A storage unit that stores the customer's status-related metadata, and the customer's abnormality based on the customer's behavior appearing in the area of interest obtained by analyzing the captured images provided by a plurality of shooting devices and the stored customer's status-related metadata The control unit may include a controller that determines the state and notifies the manager when the congestion index of the region of interest satisfies a specified condition in the determined abnormal state.

Description

Apparatus for Sensing Event on Region of Interest and Driving Method Thereof

The present invention relates to an event detection device in a local area and a method of driving the device, and more particularly, in a store such as an unmanned convenience store, or in a public place such as a subway, a notification from a manager based on an abnormal state of a customer and a congestion index accordingly It relates to an event detection device in a region specialized in a region having a specific purpose, such as giving a message, and a method of driving the device.

As social crimes increase day by day and science and technology develop together, surveillance systems using cameras such as CCTV are rapidly improving for crime prevention or monitoring such as traffic jams. Recently, surveillance personnel have developed into a technology that provides intelligent notifications beyond simple monitoring. In addition, it is not an exaggeration to say that the camera image technology is in the midst of a flood as the collection of big data is urgently required for efficient monitoring in cases of traffic jams.

In this regard, in the related art, most of the monitoring devices using images relate to the detection of events occurring within a large area. Since the surveillance device in a small area has borrowed a surveillance device in a large area (eg, using PTZ cameras to track objects and various event detection programs, etc.), the facility system is also complicated, and the actual event monitoring within a special purpose Because it is different from the standard (eg, special circumstances for each installation region), there are many cases where the actual event cannot be detected, so there is a problem of low utility.

Korean Patent Publication No. 10-2015-0100141 (2015.09.02) Korean Patent Publication No. 10-1699479 (2017.01.18) Korean Patent Publication No. 10-2016-0077932 (2016.07.04)

In an embodiment of the present invention, for example, in a store such as an unmanned convenience store or a public place such as a subway, an event detection within a region specialized in a region with a specific purpose, such as giving a notification from the manager based on the customer's abnormal state and the resulting congestion index An object of the present invention is to provide an apparatus and a method of driving the apparatus.

A local event detection device according to an embodiment of the present invention includes a storage unit for storing status-related metadata of a customer existing in an area of interest of a store or public place monitored by a plurality of photographing devices, and the plurality of photographing devices. The customer's abnormal state is determined based on the stored customer's state-related metadata and the customer's behavior appearing in the region of interest obtained by analyzing the captured image provided by , and congestion of the region of interest in the determined abnormal state and a control unit that notifies the manager when the index satisfies the specified condition.

The controller may determine the abnormal state based on a result of measuring an aspect ratio of an object with respect to the customer's behavior extracted from the captured image.

The controller may measure the time the customer stays in the region of interest and determine the abnormal state when a specified time interval is exceeded.

The controller classifies the customer-related object in the captured image by concurrently applying a rule-based first method and a deep learning-based second method for recognizing an object extracted from the captured image, and selects the classified object. Mistakes can be classified based on the traced trace results.

The storage unit further includes and stores as the metadata, behavior data related to a predetermined behavior of a customer existing in the region of interest according to the six-and-a-half principle, and the controller compares the analysis result of the captured image and the stored behavior data. Thus, it is possible to determine the abnormal state of the customer.

The control unit analyzes a captured image of a first camera that captures the first region of the region of interest to determine some actions of the six-fold principle, and analyzes a captured image of a second camera of the region of interest to perform other than the partial actions. The rest of the actions can be judged.

The controller may collect event or accident-related event data of another region of interest having the same characteristic, store it in the storage unit, and further determine the abnormal state based on the event data of the other region of interest.

The storage unit stores metadata related to the state of the customer at the unmanned convenience store, and the control unit determines abnormal purchase based on the customer's behavior of purchasing a product at the unmanned convenience store and the stored metadata. to notify the customer.

In addition, in the driving method of the driving method of the local event detection device according to the embodiment of the present invention, the state-related metadata of the customer existing in the ROI of a store or public place monitored by a plurality of photographing devices is stored in the storage unit. storing, and, by the controller, the customer's abnormal state based on the customer's behavior appearing in the region of interest obtained by analyzing the captured images provided by the plurality of photographing devices and the stored metadata related to the customer's state and notifying a manager when the congestion index of the region of interest satisfies a specified condition in the determined abnormal state.

The determining may include determining the abnormal state based on a result of measuring an aspect ratio of an object with respect to the customer's behavior extracted from the captured image.

The determining may include determining the abnormal state when the customer stays in the region of interest is measured and a specified time interval is exceeded.

The driving method includes the steps of classifying the customer-related object in the captured image by concurrently applying a rule-based first method and a deep learning-based second method to recognize an object extracted from the captured image, and the classification The method may further include classifying a misleading object based on a tracking result of tracking one object.

The storing may include further including action data related to a predetermined action according to the six-and-lower principle of the customer existing in the region of interest and storing it as the metadata, and the determining step includes the analysis result of the captured image and the storage An abnormal state of the customer may be determined by comparing one behavior data.

The determining may include analyzing a photographed image of a first camera that captures the first region of the region of interest to determine some of the six-and-lower principles, and analyzing an image captured by a second camera of the region of interest to analyze the partial action Other actions can be judged.

The driving method further includes collecting event or accident-related event data of another region of interest having the same characteristics and storing the event data in the storage unit, and the determining step is based on the event data of the other region of interest. The abnormal state may be further determined.

The storing may include storing metadata related to the customer's status at the unmanned convenience store, and the notifying may include, based on the customer's behavior of purchasing a product at the unmanned convenience store and the stored metadata. An abnormal purchase may be determined and notified to the customer.

According to an embodiment of the present invention, by providing an image analysis algorithm specialized for an area of interest or region with a specific purpose, such as an unmanned convenience store, it is possible to extract an event by tracking an object through an image with a small equipment even in a small store. .

For example, in the case of a store such as an unmanned convenience store, the store management may be facilitated by determining an abnormal state of the customer, such as a customer falling down or an abnormal behavior of a drunk customer, and notifying the store owner based on this.

In addition, by acquiring and using data related to an abnormal state of a customer occurring in a plurality of stores, a new state of a customer, that is, an event or an event occurrence of an accident, is detected and notified to the manager, so that a prompt response related thereto can be performed.

1 is a view showing an event detection system within a region according to an embodiment of the present invention;
Figure 2 is a diagram showing the region of interest of Figure 1,
3 is a view illustrating an image screen of the first photographing device;
4 is a view illustrating an image screen of a second photographing device;
5 is a diagram for explaining a first scenario according to an embodiment of the present invention;
6 is a diagram for explaining a second scenario according to an embodiment of the present invention;
7 is a diagram for explaining a third scenario according to an embodiment of the present invention;
8 to 11 are diagrams for explaining an abnormal behavior analysis solution function according to an embodiment of the present invention;
12 is a view showing an object detection classification type according to an embodiment of the present invention;
13 is a block diagram illustrating the detailed structure of the event detection device in the region of FIG. 1 or FIG. 2;
14 is a block diagram illustrating the detailed structure of the event detection unit within the region of FIG. 13;
15 is a block diagram illustrating a detailed structure of an event detection device within a region according to another embodiment of the present invention;
16 is a block diagram illustrating the detailed structure of an event detection device in a region according to another embodiment of the present invention, and a view summarizing the functions of the components 15;
17 is a flowchart illustrating a driving process of an event detecting device in an area according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing an event detection system within a region according to an embodiment of the present invention, FIG. 2 is a diagram schematically showing the region of interest of FIG. 1, FIG. 3 is a diagram illustrating an image screen of the first photographing device, FIG. 4 is a diagram illustrating an image screen of a second photographing device, FIG. 5 is a diagram for explaining a first scenario according to an embodiment of the present invention, and FIG. 6 is a diagram for explaining a second scenario according to an embodiment of the present invention , FIG. 7 is a diagram for explaining a third scenario according to an embodiment of the present invention, FIGS. 8 to 11 are diagrams for explaining an abnormal behavior analysis solution function according to an embodiment of the present invention, and FIG. 12 is the present invention It is a diagram showing an object detection classification type according to an embodiment.

1 and 2, the local event detection system 90 according to the embodiment of the present invention includes a photographing device 100, a communication network 110, a local event detection device (or an abnormal behavior detection device, anomaly behavior analysis device) 120 and part or all of the manager terminal device 130 .

Here, “including some or all” means that some components such as the communication network 110 are omitted, so that the photographing device 100 and the local event detection device (hereinafter, event detection device) 120 communicate directly (eg, : P2P communication), or some or all of some components such as the event detection device 120 are integrated into a network device (eg, a wireless switching device, etc.) constituting the communication network 110, etc. As meaning, it is described as including all in order to help a sufficient understanding of the invention.

The region of interest 95 means a public place such as a store or subway, such as an unmanned convenience store, according to an embodiment of the present invention, and the photographing device 100 is installed in the region of interest 95 to provide various types of objects, above all, customers. In other words, it monitors the actions (or actions) of people. More precisely, it may be appropriate when a manager or supervisor is not located nearby. The photographing apparatus 100 includes a fixed camera such as a CCTV or IP camera. For example, in the case of an unmanned convenience store as shown in FIG. 2 , the photographing device 100 includes the first photographing device 101 installed in the area of the first door (eg, the front door) and the first photographing apparatus installed in the area of the second entrance (eg, the side door). 2 may include a photographing device 103 . The number of photographing apparatuses 100 installed in the region of interest 95 may be variously changed according to the number or arrangement of shelves to be monitored, for example, vending machines (or vending machines) 97 .

As shown in FIG. 2 , a vending machine 97 for selling products is installed in an area of interest 95 such as an unmanned convenience store, and of course, the vending machine 97 performs payment through a payment method to withdraw the product. will be able Various types of vending machines 97 are installed in the area of interest 95, which is an unmanned convenience store, as shown in FIG. 2, in addition to that, a coffee machine may be installed, and a service device such as a microwave oven may also be installed. As such, when the vending machines 97 are installed for product sales, the respective photographing devices 100 are installed in the area of interest 95 to monitor customers in the unmanned convenience store. When the imaging device 100 is installed, it is preferable to monitor all regions of the region of interest 95 as much as possible.

3 and 4 show images captured by the first photographing device 101 and the second photographing device 103 installed in the region of interest 95, which is an unmanned convenience store, respectively. The first imaging device 101 and the second imaging device 103 are preferably installed on the ceiling to facilitate monitoring of the region of interest 95 and to reduce the risk of product damage by intruders. Of course, it can be seen that each captured image photographed by the first photographing device 101 and the second photographing device 103 is monitored by the manager terminal device 130, and the region of interest ( 95) After synthesizing and storing the status information about my situation in the form of augmented reality on the screen, it can be provided to the manager. For example, as for the status information, information related to the number of people in the current store, residence time, and congestion index (or congestion level) is displayed.

The first photographing apparatus 101 and the second photographing apparatus 103 according to the embodiment of the present invention transmit the photographed image to the event detection device 120 of FIG. 1 by photographing the designated area of jurisdiction, but the first photographing The device 101 may be installed at an appropriate location to detect abnormal behavior of the number of people and customers in the area of interest 95 , and the second imaging device 103 may be installed at an appropriate location for detecting abnormal behavior in the monitoring area. have. Of course, when the first photographing device 101 transmits the photographed image to the event detection device 120 , the device identification information is also transmitted. Through this, the event detection device 120 determines the number of people with respect to the received first photographing device 101 . and anomaly detection analysis. On the other hand, it can be seen that only abnormal behavior detection is analyzed through the captured image provided from the second photographing device 103 . As can be seen in FIG. 2 , the monitoring area has some overlapping areas, but is different from each other.

The communication network 110 may be different depending on how the system designer operates the system, but when the event detection device 120 is configured as shown in FIG. 2 in each ROI 95 in FIG. 1 , that is, When the event detection device 120 is operated locally, it can be connected to the event detection device 120 simply by a LAN line for direct communication, but the system 90 is used as a global as shown in FIG. 1 . In case of operation, a private communication network operated by a telecommunication company may be used. Of course, in the case of FIG. 2, a wired type LAN line is illustrated, but since a wireless LAN may be used, it is not particularly limited to any one type.

The communication network 110 includes both wired and wireless communication networks. For example, a wired/wireless Internet network may be used or interlocked as the communication network 110 . Here, the wired network includes an Internet network such as a cable network or a public telephone network (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, Evolved Packet Core (EPC), Long Term Evolution (LTE), and Wibro networks. meaning to include Of course, the communication network 110 according to the embodiment of the present invention is not limited thereto, and may be used, for example, in a cloud computing network under a cloud computing environment, a 5G network, and the like. For example, when the communication network 110 is a wired communication network, the access point in the communication network 110 can connect to a switching center of a telephone company. or by connecting to various repeaters such as Base Station Transmission (BTS), NodeB, and e-NodeB to process data.

The communication network 110 may include an access point. Here, the access point includes a small base station such as a femto or pico base station that is often installed in a building. Femto or pico base stations are classified according to the maximum number of accessable imaging devices 100 and the like within the region of interest 95 in the classification of small base stations. Of course, the access point may include a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the photographing device 100 and the like. The access point may use TCP/IP or Real-Time Streaming Protocol (RTSP) for wireless communication. Here, short-distance communication may be performed in various standards such as Bluetooth, Zigbee, infrared, radio frequency (RF) such as ultra high frequency (UHF) and very high frequency (VHF), and ultra-wideband communication (UWB) in addition to Wi-Fi. Accordingly, the access point extracts the location of the data packet, designates the best communication path for the extracted location, and transmits the data packet to the next device, for example, the event detection device 120 along the designated communication path. The access point may share several lines in a general network environment, and includes, for example, a router, a repeater, and a repeater.

The event detecting device 120 detects an event occurring within the ROI 95 . More precisely, it detects anomalies. However, it will not be particularly limited to the concept of such terms. The event detecting device 120 analyzes the captured image provided from the photographing device 100 in the region of interest 95 , and for example, when a plurality of photographing devices 100 are installed in the region of interest 95 , the first photographing device Through the captured image of 101, the number of people and the abnormal behavior detection analysis can be performed, and only the abnormal behavior detection and analysis can be performed through the captured image of the second photographing device 103 . Of course, this is an example, and for example, the captured images provided from the plurality of photographing apparatuses 100 may be set to analyze different events. In other words, the first captured image may be used to detect the occurrence of an event or accident, and the second captured image may be used to determine only the number of people. Alternatively, when the customer's behavior in the region of interest 95 is set as a scenario, an operation of dividing and analyzing each behavior of the corresponding scenario may be performed. When the customer's behavior scenario is followed in the order of A, B, C, and D, the first shot image is used to analyze the behavior of A and B, and the second shot image is used to analyze the behavior of C and D .

Above all, the event detection device 120 according to an embodiment of the present invention performs abnormal behavior detection more accurately rather than simply an event, and for this purpose, it is possible to monitor a state centered on a person. In this process, the event detection device 120 detects a situation in which one purchasing customer falls to the floor while purchasing a product from the vending machine 97 after entering the store or after purchasing, as shown in FIG. 5 , or purchases when viewed in FIG. 6 . After two customers enter the store and purchase a product from the vending machine 97, one of them collapses, or as shown in FIG. 7, one purchasing customer leaves the store within 20 seconds after entering the store. can In the case of FIGS. 5 and 6 , it can be seen that a long stay detection (eg, Long stay: 30 second stay condition) and collapse occurred as an event detection. The number of customers currently staying in the store, the total number of visiting customers, and the average residence time may be stored in the DB 120a as metadata. In addition, although the case of FIG. 6 is not significantly different from that of FIG. 5 , event detection for a congestion alarm in an area may be additionally performed. On the other hand, in the case of FIG. 7 , it is determined that no event has occurred, but related data, that is, metadata related to the number of customers currently staying in the store, the total number of visiting customers, and the average residence time are stored.

The event detecting device 120 may analyze the captured image provided from the photographing device 100 in the region of interest 95 and mix the analysis result with the corresponding photographed image and store it in the DB 120a. Here, in order to store as metadata, it is preferable to store additional information such as time information together. Accordingly, when the past image stored in the DB 120a is checked again, the additional information may be usefully used. For example, if a past image is searched for with a keyword of collapse or a date is input, the image may be easily searched. Alternatively, if average residence time can be a search condition, congestion index can also be a search condition.

Above all, it is preferable that the event detecting device 120 according to an embodiment of the present invention detects an event by monitoring a person's behavior as much as the monitoring target focuses on a person. For example, in the case of an unmanned convenience store, an action such as a person falling down or a drunk person falling down may occur frequently. In other words, the event detection device 120 sets an event limited to an unmanned convenience store or detects an event based on the data of the surrounding area of interest 95 having the same characteristics, such as through CCTV. It can be seen that it is differentiated from the prior art of filming a large area and analyzing an event. For example, consider an unmanned clothing store. It is clear that customers in clothing stores will behave differently from customers in convenience stores. In this way, by setting and predicting the customer's behavior for the region of interest 95 with different characteristics, and furthermore, by utilizing the data of the neighboring region of interest 95 having the same characteristic, the event of the region of interest 95 having a specific character is detected. be able to expand.

In this regard, the event detecting device 120 according to the embodiment of the present invention may be configured in various ways according to the purpose of installation or operation. For example, as shown in FIG. 2 , the operation is performed in a local manner in which operation is limited to a specific region of interest 95 . In this case, a program for performing an operation according to an embodiment of the present invention may be installed in the event detecting device 120 and executed. In addition, the local method of FIG. 2 is extended to interwork with the surrounding event detection device 120 . For example, to interwork with the event detection device 120 within a designated communication radius. This will expand the event. Alternatively, as shown in FIG. 1 , in order to provide a general service, it is connected to the communication network 110 operated by a private communication company to operate the event detection device 120 , or to combine the configuration of FIG. 2 . In other words, as shown in FIG. 1 , while the first event detecting device is operated in the region of interest 95 , the general purpose second event detecting device is operated in parallel. It is possible to perform role division or selectively operate a first or a second event detecting device. For example, when the communication network 110 is in a normal state, the second event detection device that provides the general service of FIG. 1 operates, but when a communication failure of the communication network 110 occurs, the local-based first event detection device operates will do As described above, since the system can operate in various ways according to the intention of the system designer, the embodiment of the present invention will not be particularly limited to any one form.

The event detecting device 120 of FIGS. 1 and 2 shows the number of customers currently staying in the region of interest, as well as customers outside the region of interest, in order to detect abnormal behaviors of customers, that is, people appearing in the region of interest (ROI) as shown in FIG. 8 . The number is not counted, but the total number of visitors in the area of interest, the average customer stay time, and the congestion index value are measured. If the customer stays longer than the normal time it takes to pass through the area of interest, the congestion index rises, and when the congestion level exceeds the standard, an event alarm may be generated. Of course, it is desirable to generate such an alarm at all times, as long as the alarm is not stopped, regardless of whether the administrator is monitoring or not.

In addition, as shown in FIG. 9 , the event detecting device 120 measures an aspect ratio of a customer within an ROI. That is, by measuring the width and height of a person, an abnormal score is calculated from the image frame, and when the accumulated threshold is exceeded, it is determined as a collapse, and a collapse event, that is, the phrase “abnormal” can be popped up. In this way, the event detecting device 120 can detect a fall in a horizontal direction, a fall in a vertical direction, and a fall regardless of the direction. It can be used for various purposes such as falling, fainting, and drunken detection.

Furthermore, the event detecting device 120 performs an event detection operation for a long-time stay of a visitor in a store as shown in FIG. 10 . When a customer enters the area of interest, a time timer is activated, and when the set time is exceeded, an object box of a specific color (eg pink) is displayed and an alarm is given by popping up the phrase "Long Stay". The long dwell event time can be set to 30 seconds, as experienced with POCs, eg demo tests, etc. This time can of course be varied. In addition, the event detection device 120 may perform various functions, such as detecting a motion winner wandering, detecting a customer interested in purchasing a product, and the like. Through the detection of customers interested in purchasing a product, it may be possible to judge the customer trend for the product.

In addition, the event detection device 120 may detect a congestion/accident detection event in the region of interest as shown in FIGS. 11A and 11B . It calculates the normal time it takes customers to pass through the area of interest, and when the normal time is exceeded, the congestion index rises, and when the threshold is reached, abnormal congestion occurs. In (a) of FIG. 11, the stagnation index is shown as 1.28, and the residence time is shown as 27.6 seconds. Such an operation may be useful in detecting an escalator accident in a subway station, as in FIG. 11(b). In addition, it could be used to detect customer queues at checkout counters and congestion in stores or to detect accidents.

So far, the event detecting device 120 has looked at detecting an abnormal behavior centered on a person while explaining a store or public place such as an unmanned convenience store as an example, but the event detecting device 120 according to an embodiment of the present invention is of interest Region 95 can be used to detect events in various places. For example, as more cases of accompanying dogs in public places increase, detection of abnormal behavior by people and their dogs can be made. In addition, there may be cases of leaving lost items in an indoor environment such as a subway. In this way, the event detection device 120 can easily detect an event by detecting an abnormal behavior specific to the region of interest 95 . For example, it is assumed in FIG. 1 that the region of interest A is an unmanned convenience store and the region of interest B is a public place. The two ROIs may have different anomaly detection operations. Accordingly, the event detecting device 120 may match and store the detection type or shape with the identification information of the ROI 95 and then perform different detection operations based on the matching. In other words, the different detection types may be different from the operation of classifying the object in the acquired image, the process of tracking the classified object may be simplified, and the events to be detected through this may be different.

In this way, as shown in FIG. 12 and <Table 1>, assuming that 80 or more objects can be detected, the event detecting device 120 can detect only the objects related to the region of interest 95. have. For example, it is unnecessary to classify objects such as two-wheeled vehicles and trucks to detect abnormal behavior (or behavior) of animals in a zoo. However, in the image processing process, for example, after extracting an object except for the background, all objects not related to animals may be classified or deleted. Of course, the classified objects may be used for learning.

On the other hand, the event detecting device 120 according to the embodiment of the present invention uses only a small number of video devices (e.g., a camera) due to the specificity of the place (e.g., unmanned convenience store, etc.) where the image detecting device (e.g., server) is installed. After first setting a scenario based on the six-fourth principle (who/when/where/what/how/why), set the basic event according to each of the six-fourth principles, and also set new unrecorded new events that occur in other places with the same specificity. By learning the event and applying it in real time at the currently installed place, perfect video surveillance in a small area is achieved. For example, the event detection device 120 sets a scenario for the primary event detection criteria according to the six-and-a-half principle (eg, an unmanned convenience store). For example, it may be who (gender/age, etc.), what time of visit and departure, at which stand, what item, with card, purchased. This is the standard that customers usually visit a convenience store and pay for the product after purchasing it. For example, a small number of fixed cameras are used to detect object tracking and video events, and the primary (or first) camera detects and tracks only events related to who/what time/card/purchase history among the six principles of camera shooting. Create and use object code. On the other hand, in the captured image of the secondary (or second) camera, an object using a tracking object code detects only an event about a certain shelf/something. In this way, the event detecting device 120 can easily extract an event by tracking an object through an image with a small device in a small store.

It is preferable that the manager terminal device 130 of FIG. 1 can monitor the region of interest 95 remotely as much as possible. It is desirable that the manager of a small store rather than the device of the monitoring center, more precisely, in the case of an unmanned store, can remotely monitor the situation of the store and be notified of the event from time to time. Therefore, the manager terminal device 130 is a desktop computer, laptop computer, tablet PC, smart phone, smart TV and peripheral devices connected thereto that can display images, artificial intelligence (AI) speaker, and domestic S company's Galaxy Gear and It may include various devices such as the same wearable device.

The manager terminal device 130 can access the event detection device 120 of FIGS. 1 and 2 at any time to check a specific situation, and when there is an event notification, it can take a quick action after examining the corresponding situation. Alternatively, it is possible to periodically check the situation of the store through categories such as the congestion level or stay time using keywords such as past images at a specific time, and perform appropriate actions for accident prevention or product placement through the congestion level. have. For example, during the time period, the manager may directly visit the store to manage and supervise the store.

13 is a block diagram illustrating the detailed structure of the local event detecting unit of FIG. 1 , and FIG. 14 is a block diagram illustrating the detailed structure of the local event detecting unit of FIG. 13 .

As shown in FIG. 13 , the local event detecting device (hereinafter, event detecting device) 120 of FIG. 1 according to an embodiment of the present invention includes a communication interface unit 1300 , a controller 1310 , and a local event detecting device. Some or all of the unit 1320 and the storage unit 1330 are included.

Here, "including some or all" means that some components such as the storage unit 1330 are omitted or some components such as the local event detection unit 1320 are other components such as the control unit 1310. It means that it can be integrated into and configured, and it will be described as including all in order to help a sufficient understanding of the invention.

The communication interface unit 1300 communicates with the photographing apparatus 100 such as a CCTV camera and an IP camera installed in the region of interest 95 of FIG. 1 , for example, and receives a photographed image. In the process of receiving the photographed image, identification information of the photographing apparatus 100 may be received.

In addition, the communication interface unit 1300 may be capable of performing wired/wireless communication with the photographing device 100 , and may include a wired/wireless communication module for this purpose. In the process of performing communication, operations such as encoding/decoding, muxing/demuxing, and scaling may be performed, and in the case of FIG. 2 , a separate encoding or decoding operation may be omitted. Since encoding/decoding and muxing/demuxing are obvious to those skilled in the art, further descriptions will be omitted.

The controller 1310 is in charge of overall control operations of the communication interface unit 1300 constituting the event sensing device 120 , the local event sensing unit 1320 , and the storage unit 1330 . The control unit 1310 may temporarily store the captured image provided from the communication interface unit 1300 in the storage unit 1330 and then provide it to the local event detection unit 1320, and in the local event detection unit 1320 The generated metadata is classified and stored in the DB 120a of FIG. 1 .

In addition, the control unit 1310 receives, for example, event-related data of an area of interest (eg, unmanned convenience store A and unmanned convenience store B) having the same characteristics according to the request of the intra-region event detection unit 1320, and stores it in the storage unit 1330. After saving, it can be used when detecting an event. When the event detecting device 120 is operated in a local manner, a specific event detecting device 120 may be designated, but event-related data of an area of interest having the same characteristics based on the communication radius to more accurately reflect the local characteristics. can be collected and used. For example, convenience stores in factory areas may have similar customer bases. Of course, it is also possible to collect and use data in units of administrative districts such as dongs or gu units.

The local event detection unit 1320 may be an event, that is, an event or an accident, but more precisely, it can be viewed as detecting an abnormal behavior. In particular, human-centered analysis can be made, and for this purpose, the aspect ratio of a person can be measured. Normally, the length of the bell should be large, but if it is in the opposite direction, the local event detection unit 1320 detects that an abnormal situation has occurred for the customer, and continuous measurement can be made over several frames of images. . For example, a general customer or drunken person may temporarily fall and then stand up again. At this time, the aspect ratio of the surrounding people is measured together or the number of people is measured to calculate the congestion index for the abnormal behavior. Of course, additional information obtained through such image analysis, more precisely, information such as the number of customers in the area of interest of the camera, residence time, congestion index, etc. is continuously displayed on the corresponding screen, and can be synthesized and stored in the image of the corresponding screen. The standard of the aspect ratio may be the last scan line in the input unit frame image or the image of the corresponding scan line. Alternatively, the image may be an image corresponding to the first scan line of the input unit frame image.

In addition, the intra-region event detection unit 1320 detects an abnormal behavior in the region of interest, more precisely, in the region of interest, based on the program data set at the time of initial installation of the device, and then detects the abnormal behavior in the surrounding area. Abnormal behavior may be detected or detected based on data collected in the region of interest 95 . Of course, the collected data is data of the region of interest 95 having the same characteristics, and furthermore, it may be more accurate to reflect the data of the same region. The same area has been cited as an example of a factory area or an administrative district. However, since region and region of interest may be used interchangeably, the concept of such terms is not particularly limited. That is, the region can be substituted for the region of interest.

In order to increase accuracy, the local event detection unit 1320 may concurrently perform rule-based object detection and classification and artificial intelligence deep learning-based object detection and classification. For example, in the case of a department store, where object congestion increases in the image, object classification may not be accurate if objects overlap each other, and in this case, false tracking may occur. to prevent erroneous detection. By parallelizing deep learning in this way, it is possible to reduce erroneous detection of objects. Since the embodiment of the present invention is human-centered detection, it is possible to detect a person as an object as much as possible and classify various types of human objects. In the embodiment of the present invention, since face detection is not stored and used in the form of a template to distinguish people as in the prior art, it can be seen that the entire body shape is recognized or detected.

Above all, the local event detecting unit 1320 may classify the object based on the feature point of the object of the image without performing separate object detection through the human object. Of course, in this case, it may be possible to analyze the image by copying or capturing an image of the corresponding part based on the feature point. In addition, if the aspect ratio of the extracted object or the copy, that is, the captured object image is measured and the aspect ratio is different from the normal state, it is determined as an abnormal state, and continuous tracking can be performed. monitoring their subsequent actions. As a result of the subsequent behavior monitoring, if the image congestion in the relevant area increases or the congestion continues for a long time due to the abnormal behavior of people gathering, the local event detection unit 1320 continuously compares the preset data and An event is generated when a condition is satisfied. Here, "generating an event" means that a phrase that notifies an abnormal state is popped up on the image so that the currently monitoring manager can see it, or notifying the event to the manager's terminal device 130 of the manager. can mean that Since the notification may be made in various forms, it will not be specifically limited to any one form.

In this way, the intra-region event detection unit 1320 installs a program specialized for the specific region of interest 95 and uses it, so that it is simple in terms of hardware or software, and the overall system is simplified, so that image data processing can be performed quickly. Of course, it is possible to periodically update the corresponding software program in the form of firmware online, or it may be possible to update by writing the program in the form of an EEPROM offline.

In addition, the local event detection unit 1320 may perform event detection based on a scenario by analyzing captured images received from different photographing devices 100 in the region of interest 95 . In other words, when a customer enters an unmanned convenience store, the payment method is put into the vending machine 97 to purchase a product, and after payment is made, the customer leaves the area, but as a result of the analysis of the video, if this normal scenario is not followed, abnormal It is judged as an action and a notification is given to the manager or the customer. Notifications to customers may also be provided through local speakers. In the case of the manager, a pop-up is made by synthesizing a phrase of an abnormal state in the corresponding image through the manager terminal device 130 or the like.

Of course, in this process, the local event detecting unit 1320 may derive different events by analyzing the respective captured images received from different photographing devices 100 . In other words, based on the scenario, some actions of the scenario are analyzed with the captured image of the first photographing device 101 , and the remaining actions are analyzed with the captured image of the second photographing device 103 , excluding some actions in the scenario. As such, the local event detection unit 1320 may operate as a division of labor system in image analysis.

14 is an event detection unit 1320' within a region according to another embodiment of the present invention for detecting a new event by collecting data of a neighboring region of interest 95 having the same characteristics and performing image processing based on a scenario. ) is shown. The event detection unit 1320 ′ of FIG. 14 includes a first camera image receiving unit 1300 , a first camera event determining unit 1401 , an object tracking unit 1402 , a second camera image receiving unit 1403 , and a second camera event Determination unit 1404, product DB 1405, settlement unit 1406, event recording unit 1407, event alarm unit 1408, scenario event DB 1409, new event DB 1410, new event collection unit 1411 , some or all of the new event analysis unit 1412 , and the new event generation unit 1413 may be included.

Here, "including some or all" means that some of the components constituted by hardware, software, or a combination thereof may be omitted or may be configured by being integrated with each other, and in an embodiment of the present invention, the invention In order to help you understand fully, it is explained as including everything.

As mentioned above, the first photographing device 101, that is, the photographed image of the first camera and the photographed image of the second camera, which is the second photographing device 103, are the respective image receiving units 1400 and 1403 and the event determining unit 1401, 1404), image analysis and event determination are performed. The object tracker 1402 may perform a tracking operation on the object classified in the captured image. For example, through object tracking, it may be determined that a customer enters a store, stays for a while, and then leaves. The object DB 1405 may store data related to various types of objects, and may be used to determine the type of object from an image analysis result. It can be used to judge the product selected by the customer. The settlement unit 1406 may process information about purchased goods and payment methods. For example, in the case of a credit card, it can be executed by including a payment module. The event recording unit 1407 may store, for example, an image of a part related to an event determined to be in an abnormal state, and the event alarm unit 1408 may perform an operation related to which event alarm is to be given. For example, it is enough to provide an event alarm of an output terminal that matches input data, such as a lookup table (LUT). In the scenario event DB 1408, for example, action scenarios that may occur in an unmanned convenience store and a subway escalator are different from each other, and thus event information by these actions and the corresponding actions may be stored.

In addition, the new event collection unit 1411 may collect event-related data of an adjacent region of interest 95 having the same characteristic or the same characteristic and store it in the new event DB 1410 . Here, the DB means a storage unit, but may be a DB that works separately. The new event analysis unit 1412 determines a new event based on data stored in the new event DB 1410 . For example, a new event may be detected based on collected data of an event that was not detected in the ROI A. For example, if it is impossible to analyze an event for a drunken when setting up an initial program, it is a good example that an event analysis for a drunken becomes possible through newly collected data. The new event generating unit 1413 may perform an operation of determining a new event based on the analysis result of the new event analyzing unit 1412 . For example, if an abnormal behavior of drunkenness is detected in a new event based on the data collected from the ROI A, but no abnormal behavior is detected as a result of using it in the ROI B, it will not be created as a new event. In this way, by accumulating data and using the data, it becomes possible to detect anomalies specialized for a specific region of interest 95 .

It exemplifies how a specific scenario works. Person A enters the store. When an image A is received through camera 1, the main server, that is, the event detection device 120, designates an object code using characteristic information about a person's gender/age/height/face shape. And the main server confirms the arrival time of A. disappears from camera 1. The main server checks whether A appears in camera 2 through the characteristic information on the object code (if it is not confirmed, it is determined that a specific event has occurred). When A appears in Camera 2, the main server checks which stand for how long and what items A has picked up. When A disappears from camera 2, the main server checks whether it appears in camera 1 through the characteristic information on the object code (if it is not confirmed, it is determined that a specific event has occurred). The main server detects whether camera A stops at the checkout when camera 1 appears (event occurs if the checkout does not stop even after pickup / no event occurs if there is no pickup). The main server can check whether the picked up item and the calculated price match when A pays for the item at the checkout in Camera 1.

Continue to see how the new event works. New event means that when an unexpected event is detected in a place with the same purpose, the image is collected and analyzed as a new event to be used as a detection standard for another area. For example, if an object collapses in a store in another area and an event of sudden collapse occurs by analyzing the video, the main server determines that it is a new event if the object in the video decreases in height and does not grow taller again or there is no movement It is applied to the event DB and immediately applied as an event standard in other regions.

The storage unit 1330 of FIG. 13 stores various information or data processed under the control of the controller 1310 . Here, information means additional information, and data means real data such as image data. For example, time information of a captured image according to an embodiment of the present invention is additional information. On the other hand, the captured image becomes real data. Meta data corresponds to the sum of various additional information related to the actual data as described above. The storage unit 1330 may provide data necessary for the operation of the local event detection unit 1320 .

Meanwhile, the controller 1310 of FIG. 13 may include a CPU and a memory as another embodiment of the present invention, and the CPU and the memory may be configured as a single chip. According to this configuration, the CPU copies the program stored in the local event detecting unit 1320 of FIG. 3 to the memory when the device is initially driven or if necessary, by selectively copying the program of the local event detecting unit 1320. By executing it after loading, the data operation processing speed can be increased quickly. The CPU includes a control circuit, an arithmetic unit (ALU), an instruction interpreter and a registry, and the like. The control circuit performs actual control operations, the arithmetic unit performs the operation of binary bit information, and the instruction interpreter uses a high-level language such as an interpreter or compiler. It performs operations such as converting machine language and machine language into high-level language, and the registry can be involved in software data storage.

15 is a block diagram illustrating a detailed structure of a local event detecting apparatus according to another embodiment of the present invention, and FIG. 16 is a block diagram illustrating a detailed structure of an intra-regional event detecting apparatus according to another exemplary embodiment of the present invention. It is a diagram showing the functions of the components of the diagram and 15 in a summary.

As shown in FIG. 15 , the local event detection device 120 ′ according to another embodiment of the present invention includes a preprocessor 1500 , an object recognition unit 1510 , an object tracking unit 1520 , and an object classification unit ( 1530), the filter unit 1540, the optimization unit 1550, the metadata processing unit 1560, and some or all of the event unit 1570, wherein "including some or all" has the same meaning as above. Do. The above components may be configured by hardware, software, or a combination thereof.

When a captured image is received from the photographing apparatus 100 of the region of interest 95 of FIG. 1 , the pre-processing unit 1500 performs a pre-processing operation of the image. The preprocessing may perform an operation such as converting a color image into a gray scale image or removing noise.

The object recognition unit 1510 recognizes which object corresponds to an object extracted from the pre-processed image, for example, by applying it to a template. The recognizer may provide a detection result. For example, if you are walking upright and have hands and feet, you can be recognized as a person. In particular, in the case of humans, when shooting with a surrounding background with trees, misrecognition may occur, so rule-based object recognition and deep learning-based object recognition can be performed together in this recognition process. For example, if it is determined that the object is continuously detected in several frames of an image, it is an advantage of deep learning to finally recognize it as an object through learning. In this way, the accuracy of object recognition can be improved by combining rule-based and deep learning-based object recognition.

Also, it can be seen that the object tracking unit 1520 determines the movement of the object while tracking the object recognized through the object recognition unit 1510 along the image frame according to time change. In this process, the object tracking unit 1520 may store the motion of the object in the form of coordinate values.

The object classification unit 1530 and the (false positive) filter unit 1540 classify the object based on the tracking result of the object tracking unit 1520, and filter out the classified object through the filter unit 1540. Detected objects can be discarded.

The optimizer 1550 may, for example, perform an operation of adapting the installation program to the region of interest 95 of FIG. 1 . That is, an operation specialized for the region of interest 95 is performed.

The metadata processing unit 1560 may generate metadata based on the set object, action, and event, and store it in the DB 120a of FIG. 1 together with the received captured image. Of course, the finally stored image is the first received color image, and as described above with reference to FIGS. 8 to 11 , various additional information or metadata may be synthesized and stored in the color image. In other words, after matching and storing the metadata, when the corresponding image is provided to the manager terminal device 130 of FIG. 1 , it is displayed on the screen together.

The event unit 1570 may store an object, an action of the object, and data related to an event of the action, or perform an operation of determining an event based on the data.

Since the case of FIG. 15 has been reviewed from the viewpoint of image processing, it may be understood as a component constituting the intra-region event detection unit 1320 of FIG. 13 . Therefore, in the embodiment of the present invention, it will not be particularly limited to any one form in the form of configuration.

As can be seen in FIG. 16 , the local event detection device 120 ″ according to another embodiment of the present invention includes a (communication) interface unit 1600 , an image processing unit 1610 , a management unit 1630 , and an intelligent engine. It may include some or all of the portion 1640 .

Here, the intelligent engine unit 1640 includes the object recognition unit 1510, the object tracking unit 1520, the object classification unit 1530, the (false detection) filter unit 1540 and the optimization unit 1550 of FIG. 15 and Some or all of the metadata processing unit 1560 and the event unit 1570 may be included, or the corresponding components may be operated.

For details regarding the (communication) interface unit 1600, the image processing unit 1610, the management unit 1630, and the intelligent engine unit 1640 of FIG. 16, the event detection device 120 of FIG. 1 and the event detection of FIG. Since it has been sufficiently described through the device 120', the contents thereof will be used instead.

As can be seen in <Table 2>, the event detection device 120, 120', 120" according to an embodiment of the present invention includes a system setting function, an object detection function, a crowd tracking, a real-time region of interest (ROI) analysis, It can perform event detection, event alarm, camera calibration, and recording functions, etc. Here, in the embodiment of the present invention, multi-object tracking, multi-object detection and classification, object height measurement, etc. can be performed for crowd tracking. By measuring the height of an object, it is possible to detect actions such as falling down, etc. In addition, through real-time analysis of the region of interest, object counting, total number of visits within the region, average residence time, and congestion index are extracted and recorded in the captured image. It can be displayed in a pop-up form.

In addition, the event detection device (120, 120', 120") can measure a straight line distance or a movement distance, etc. to detect an event, for example, to determine the stagger of a drunken person, and the event alarm is displayed for a long time stay, abnormal, or An alarm for abnormal congestion can be provided, for example, for distance measurement, the movement distance per hour can be measured using the time information measured through a timer, or the distance using the coordinates of drunk people located at two points in the video It can also be measured with a formula.

In addition to the above, the event detecting apparatuses 120, 120', and 120" can perform various operations, but since detailed information has been sufficiently described above, those contents are substituted.

17 is a flowchart illustrating a driving process of an event detecting device within a region according to an embodiment of the present invention.

Referring to FIG. 17 together with FIG. 1 for convenience of explanation, the event detecting device 120 according to an embodiment of the present invention provides a method of detecting a customer present in an area of interest of a store or a public place monitored by a plurality of photographing devices 100 . State-related metadata is stored (S1700).

In addition, the event detection device 120 detects the abnormal state of the customer based on the customer's behavior appearing in the region of interest obtained by analyzing the captured images provided by the plurality of photographing devices 100 and the stored metadata related to the customer's state. In the determined abnormal state, if the congestion index of the region of interest satisfies the specified condition, the manager is notified (S1710).

In addition to the above, the event detecting device 120 may perform various operations, and since the other detailed information has been sufficiently described above, those contents are substituted.

On the other hand, even though it has been described that all components constituting the embodiment of the present invention are combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the object of the present invention, all the components may operate by selectively combining one or more. In addition, all of the components may be implemented as one independent hardware, but a part or all of each component is selectively combined to perform some or all functions of the combined components in one or a plurality of hardware program modules It may be implemented as a computer program having Codes and code segments constituting the computer program can be easily deduced by those skilled in the art of the present invention. Such a computer program is stored in a computer-readable non-transitory computer readable media, read and executed by the computer, thereby implementing an embodiment of the present invention.

Here, the non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, not a medium that stores data for a short moment, such as a register, cache, memory, etc. . Specifically, the above-described programs may be provided by being stored in a non-transitory readable recording medium such as a CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

In the above, preferred embodiments of the present invention have been illustrated and described, but the present invention is not limited to the specific embodiments described above, and it is common in the technical field to which the present invention pertains without departing from the gist of the present invention as claimed in the claims. Various modifications may be made by those having the knowledge of, of course, and these modifications should not be individually understood from the technical spirit or perspective of the present invention.

95: area of interest 97: vending machine
100: photographing device 110: communication network
120, 120', 120": (in the region) event detection device 130: administrator terminal device
1300: communication interface unit 1310: control unit
1320, 1320': local event detection unit 1330: storage unit
1500: preprocessor 1510: object recognition unit
1520: object tracking unit 1530: object classification unit
1540: (false detection) filter unit 1550: optimization unit
1560: metadata processing unit 1570: event unit
1600: interface unit 1610: image processing unit
1620: management unit 1630: intelligent engine unit

Claims (16)

a storage unit for storing status-related metadata of customers existing in an area of interest in a store or a public place monitored by a plurality of photographing devices; and
The customer's abnormal state is determined based on the customer's behavior appearing in the region of interest obtained by analyzing the captured images provided by the plurality of photographing devices and the stored customer's state-related metadata, and when the determined abnormal state is Including; a control unit that notifies a manager when the congestion index of the region of interest satisfies a specified condition;
The storage unit stores, as the metadata, further including behavior data related to a preset behavior according to the six-four principle of the customer existing in the region of interest,
The control unit determines the abnormal state of the customer by comparing the analysis result of the captured image and the stored behavior data,
The control unit analyzes a captured image of a first camera that captures the first region of the region of interest to determine some actions of the six-fold principle, and analyzes an image captured by a second camera of the region of interest to perform other than the partial actions. judging the rest of the
The control unit collects event or accident-related event data of another region of interest having the same characteristics, stores it in the storage unit, and further determines the abnormal state based on the event data of the other region of interest. . According to claim 1,
The control unit may be configured to determine the abnormal state based on a result of measuring an aspect ratio of an object with respect to the customer's behavior extracted from the captured image. According to claim 1,
The controller is configured to measure the length of time the customer stays in the region of interest to determine the abnormal state when a specified time interval is exceeded. According to claim 1,
The control unit classifies customer-related objects in the captured image by concurrently applying a rule-based first method and a deep learning-based second method for recognizing an object extracted from the captured image, , a local event detection device for classifying misleading objects based on the tracking result of tracking the classified objects. delete delete delete According to claim 1,
The storage unit stores metadata related to the state of the customer in the unmanned convenience store,
The control unit determines an abnormal purchase based on the stored meta data and the customer's behavior of purchasing a product at the unmanned convenience store, and notifies the customer of an event detecting device in the region. Storing, in a storage unit, metadata related to a customer's status existing in an area of interest in a store or a public place monitored by a plurality of photographing devices; and
The control unit determines the abnormal state of the customer based on the stored metadata related to the customer's state and the behavior of the customer appearing in the region of interest obtained by analyzing the captured images provided by the plurality of photographing devices, Notifying an administrator when the congestion index of the region of interest satisfies a specified condition in an abnormal state;
The saving step is
Storing as the metadata, further including behavior data related to a predetermined behavior according to the six-legged principle of the customer existing in the region of interest,
The determining step is
determining an abnormal state of the customer by comparing the analysis result of the captured image and the stored behavior data; and
The first camera for capturing the first region of the ROI is analyzed to determine some of the six-and-lower principles, and the second camera for the ROI is analyzed to determine the remaining actions other than the partial action. including;
Further comprising; collecting event or accident-related event data of another region of interest having the same characteristics and storing the data in the storage unit;
The determining step is
A driving method of a method of driving an event detecting device within a region to further determine the abnormal state based on the event data of the other region of interest. 10. The method of claim 9,
The determining step is
A method of driving an event detection device in a region to determine the abnormal state based on a result of measuring an aspect ratio of an object for the customer's behavior extracted from the captured image. 10. The method of claim 9,
The determining step is
A method of driving an event detecting device in an area for measuring the time the customer stayed in the region of interest and determining the abnormal state when a specified time interval is exceeded. 10. The method of claim 9,
classifying a customer-related object in the captured image by concurrently applying a rule-based first method and a deep learning-based second method for recognizing an object extracted from the captured image; and
Classifying the wrong object based on the tracking result of tracking the classified object;
Driving method of the event detection device in the region further comprising. delete delete delete 10. The method of claim 9,
The saving step is
Stores metadata related to the customer's status at the unmanned convenience store
The step of notifying
A method of driving a local event detection device for determining an abnormal purchase based on the customer's behavior of purchasing a product at the unmanned convenience store and the stored metadata and notifying the customer.

Images