KR102584708B1 - System and Method for Crowd Risk Management by Supporting Under and Over Crowded Environments - Google Patents

Abstract

The present invention relates to a crowd risk management system and method that supports underpopulated and overcrowded environments. The crowd risk management system that supports underpopulated and overcrowded environments according to an embodiment of the present invention includes a filming device installed in a place where crowds are concentrated. By analyzing the captured video, when the crowd density is lower than the standard value, the movement direction and abnormal behavior of the crowd are detected through object detection and tracking of the detected object, and when the crowd density is higher than the standard value, the movement of the crowd in the captured video is detected. A crowd risk management video analysis device that tracks optical flow and determines crowd overcrowding and abnormal flow to generate events, and crowd count information is provided from the crowd risk management video analysis device to show crowd density and crowd flow on a map. and a control device that displays risk information.

Description

Crowd risk management system and method for supporting under and overcrowded environments {System and Method for Crowd Risk Management by Supporting Under and Over Crowded Environments}

The present invention relates to a crowd risk management system and method that supports depopulated and overcrowded environments. More specifically, the present invention relates to a crowd risk management system and method that supports depopulated and overcrowded environments. More specifically, the present invention relates to a crowd risk management system and method that uses, for example, a deep learning-based object detection and tracking function, a deep learning-based crowd coefficient detection function, and an optical-based flow extraction function. In the case of an environment, the object detection and tracking function is used to detect events for individual objects or an object tracking-based crowd counting function is used. When the environment is converted to an overcrowded environment, the density-based crowd counter and flow of objects, rather than individual objects, are extracted to detect the crowd. It relates to a crowd risk management system and method that supports overcrowded and overcrowded environments to determine risk situations related to the flow of people.

Object detection and tracking technology is a widely used technology in video security. Additionally, the accuracy of technology is increasing through deep learning techniques. Deep learning object detection technology is a method of detecting objects by learning the type of object to be detected. As these object detection technologies become more advanced, efforts to accurately detect and track objects in crowded places can be seen through the MOT Challenge, and numerous algorithms are being attempted. However, in this object detection and tracking method, the object detection performance is inevitably lowered when the object to be detected is obscured or part of the body is obscured. Additionally, when the crowd density is so high that only the head is exposed or the exposed head is obscured, the existing object tracking method is bound to be difficult.

Conventionally, research is being actively conducted on convolutional neural networks, autoencoders (AEs), and recurrent neural networks (RNNs) for crowd behavior recognition. Some papers show that the higher the crowd density, the more difficult it is to find high-level meaning. According to the paper, in the case of crowd density, a density map is extracted. The paper defines crowd density and behavior as three levels each.

At the low level of crowd density and behavior detection, it is viewed as simply estimating the location of people. For example, in situations where crowd density is high, the degree of atomic flow is recognized, or in places where object density is low, it is the degree to which a clear posture (e.g., sitting, standing, walking) is recognized. In other words, it is an attitude that can be estimated from one scene. The temporal range of action recognition judgment is at the level of making judgments based on a video of about a few seconds.

The Medium level is defined as the level that identifies groups and estimates clustering and group trajectories for each group. It is about tracking and clustering of multiple objects and swarms of people, and is a very difficult area, so it is an area that is being studied a lot. The temporal range of behavior recognition can be viewed as the level that defines behavior from minutes to hours.

In the case of high level, there is semantic understanding by group, clustering by action, and detection of action. Semantic understanding can be said to be a part of judging whether something is dangerous through the crowd density flow analysis results, which were previously analyzed. In the case of behavior recognition, it is the stage of recognizing behavior on a time or date basis, and can be said to be at a level where changes in two or more types of behavior can be detected.

If you look at the information published in the papers, you can see that the technology basis for behavior recognition when density is low and detection technology when density is high is different. For example, the model that shows the best performance in the ShanghaiTech A dataset is P2PNet. As the title of the paper suggests, this network can intuitively learn and evaluate by using GT as a point and prediction as a point, and has excellent crowd counting performance (see Rethinking Counting and Localization in Crowds: A Purely Point-Based Framework) ). Additionally, GauNet is the model with the best performance on the ShanghaiTech B and JHU-CROWD++ datasets. This model extracts a density map based on CNN to estimate coefficients and calculates the coefficients, and adopts a method of replacing the convolution layer with a Gaussian kernel to reduce annotation (tagging) errors.

Meanwhile, Canon has developed and has a solution for counting crowds based on artificial intelligence (AI). This solution has been launched as a product called Crowd People Counter in Mileston's Xprotect, providing a solution to count thousands of people within seconds. In the case of this solution, the measurement area is manually set to obtain the unit density. This is a crowd management system developed by NEC and analyzes based on density rather than individual detection. Extract crowd density and flow and detect abnormal crowd behavior based on this.

However, both Canon and NEC companies only offer functions that use crowd counters alone, and recently, there has been a demand for a method to prevent risks in advance by predicting the density of crowds using this function.

Korean Patent Publication No. 10-2017-0075445 (2017.07.03) Korean Patent Publication No. 10-2016-0093253 (2016.08.08) Korea Patent Publication No. 10-2020-0022149 (2020.03.03) Korean Patent Publication No. 10-2022-0056399 (2022.05.26)

A Short Review of Deep Learning Methods for Understanding Groups and Crowd Activities, IEEE, 2018 Rethinking Counting and Localization in Crowds: A Purely Point-Based Framework, ICCV2021 Rethinking Spatial Invariance of Convolutional Networks for Object Counting, CVPR2022 Single-image crowd counting via multi-column convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 589-597, 2016.

An embodiment of the present invention detects events for individual objects using, for example, a deep learning-based object detection and tracking function, a deep learning-based crowd coefficient detection function, and an optical-based flow extraction function, and in the case of a depopulated environment, an object detection and tracking function. It uses an object tracking-based crowd counting function, and when it switches to an overcrowded environment, it extracts the density-based crowd counter and flow of objects rather than individual objects to determine risk situations related to the crowd flow, supporting both under- and overcrowded environments. The purpose is to provide risk management systems and methods.

The crowd risk management system that supports under- and overcrowded environments according to an embodiment of the present invention analyzes captured images from a recording device installed in a place where crowds are concentrated, and detects objects and detects objects when the crowd density is lower than the standard value as a result of the analysis. Detects the movement direction and abnormal behavior of the crowd through tracking, and when the crowd density is higher than the standard value, optical flow related to the movement of the crowd in the captured video is tracked to determine whether the crowd is overcrowded and abnormal flow. It includes a crowd risk management video analysis device that determines and generates an event, and a control device that receives crowd count information of the crowd from the crowd risk management video analysis device and displays crowd density, crowd flow, and risk information on a map. .

The crowd risk management video analysis device includes a crowd coefficient estimation unit that analyzes the captured video to estimate the crowd coefficient, and an optical flow extraction unit that extracts the crowd flow based on the optical flow of the crowd coefficient extracted from the crowd coefficient estimation unit. , and a crowd flow event detection unit that determines the crowd density per unit area and the abnormal flow of the crowd based on the tracking result of the detected object, the estimated crowd coefficient, and the extracted optical flow.

The crowd coefficient estimation unit uses a crowd coefficient estimator based on a density map of an artificial intelligence deep learning method to estimate the coefficient of the crowd, or expresses the location of the crowd as points and estimates this using a deep learning method to determine the density. You can use any method to get it.

The optical flow extraction unit may extract the optical flow using the density map or crowd points obtained from the crowd coefficient estimation unit as a reference point for detecting the optical flow as a starting point.

The crowd flow event detector may determine whether a flow collision or congestion situation occurs at a density higher than a reference value as the abnormal flow of the crowd.

In addition, the crowd risk management method supporting under- and over-crowded environments according to an embodiment of the present invention involves a crowd risk management video analysis device analyzing video footage from a filming device installed in a place where crowds are concentrated, and as a result of the analysis, the crowd density is When the crowd density is lower than the standard value, the movement direction and abnormal behavior of the crowd are detected through object detection and tracking of the detected object, and when the crowd density is higher than the standard value, the optical flow related to the movement of the crowd in the captured video is tracked to detect overcrowding of the crowd. A step of generating an event by determining presence and abnormal flow, and a step of a control device receiving crowd count information of the crowd from the crowd risk management video analysis device and displaying crowd density, crowd flow, and risk information on a map. Includes.

The step of generating the event includes the crowd coefficient estimation unit of the crowd risk management video analysis device analyzing the captured video to estimate the crowd coefficient, and the optical flow extraction unit of the crowd risk management video analysis device performing the crowd coefficient. Extracting a crowd flow based on the optical flow of the crowd coefficient extracted by an estimation unit, and a crowd flow event detection unit of the crowd risk management video analysis device, tracking results of the detection object, the estimated crowd coefficient, and the crowd flow event detection unit. It may include the step of determining crowd density per unit area and abnormal flow of the crowd based on the extracted optical flow.

The step of estimating the coefficient is performed by using a crowd coefficient estimator based on a density map of an artificial intelligence deep learning method to estimate the coefficient of the crowd, or by expressing the location of the crowd as a point and estimating it using a deep learning method. A method of obtaining density by estimating density can be used.

In the step of extracting the crowd flow, the optical flow may be extracted using the density map or crowd points obtained from the crowd coefficient estimation unit as a reference point for detecting the optical flow as a starting point.

The step of determining the abnormal flow may determine whether a flow collision or congestion situation occurs at a density higher than a reference value as the abnormal flow of the crowd.

According to an embodiment of the present invention, it is possible to detect high-level behavioral events such as falls and acts of violence that may occur in a crowded environment by using a generally installed CCTV, and by extracting the density and abnormal flow of the crowd in an overcrowded place. Since you can find out whether there is an abnormality in the crowd's behavior, you can prevent safety accidents by informing the crowd in advance of dangers, or quickly notify them when they occur so that quick action can be taken.

1 is a diagram showing a crowd safety management system for overpopulated and overcrowded environments according to an embodiment of the present invention;
Figure 2 is a block diagram illustrating the detailed structure of the crowd risk management video analysis device and control device of Figure 1;
Figure 3 is an example of counting a crowd using an object detection method or analyzing the flow through the trajectory of an object;
Figure 4 shows an example of a function to estimate the crowd coefficient based on density and an example of a function to estimate the crowd coefficient by the point estimation method.
Figure 5 is an example of a function in which the area of a user-specified area is automatically calculated as actual measurement when camera calibration is completed;
Figure 6 is an example of displaying the moving direction and moving speed of an object as an Optical Flow Field;
Figure 7 is an example of clustering and classifying crowd flows into large flows.
Figure 8 is a flowchart for determining the crowd count frequency and whether to detect a flow based on the currently extracted crowd count value;
Figure 9 is a diagram showing the role of each video analysis engine by crowd density;
Figure 10 is an example of a control system screen that maps to a crowd density map and monitors the crowd coefficient value;
Figure 11 is a block diagram illustrating another detailed structure of the crowd risk management video analysis device of Figure 1, and
Figure 12 is a flowchart of the driving process of the crowd risk management video analysis device of Figure 1.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

1 is a diagram showing a crowd safety management system for overcrowded and overcrowded environments according to an embodiment of the present invention.

As shown in Figure 1, the crowd safety management system 90 for under- and overcrowded environments according to an embodiment of the present invention includes a photographing device 100, a communication network 110, and a crowd risk management video analysis device (or crowd risk management system). management device) 120, and may further include an edge device or a control device 130 provided adjacent to or included in the photographing device 100.

Here, “including part or all” means that some components such as edge devices are omitted to form a crowd safety management system 90 for under- and overcrowded environments, or to configure a crowd risk management video analysis device 120. This means that some or all of the components can be integrated and configured into a network device (e.g., wireless exchange device, etc.) constituting the communication network 110, and all are included to facilitate a sufficient understanding of the invention. Explain.

The imaging device 100 may generally include a pre-installed CCTV, but in the embodiment of the present invention, it is not limited to this and may further include various devices such as an additionally installed IP camera or artificial intelligence (AI) camera. there is. Of course, the imaging device 100 may include various types of devices, such as a PTZ camera capable of controlling PTZ (Pan, Tilt, Zoom) as well as a fixed type.

Above all, the imaging device 100 may include a module for image analysis inside, or may be additionally provided with an edge device for image analysis adjacent to it. Of course, the image analysis device can receive the captured image provided from the image capture device 100 without any additional compression and perform an image analysis operation. By primarily or preliminary video analysis using the edge device, it will be possible to reduce traffic in the communication network 110 and prevent load from occurring.

The communication network 110 includes both wired and wireless communication networks. For example, a wired or wireless Internet network may be used or linked as the communication network 110. Here, the wired network includes Internet networks such as cable networks and public switched telephone networks (PSTN), and the wireless communication network includes CDMA, WCDMA, GSM, EPC (Evolved Packet Core), LTE (Long Term Evolution), and Wibro networks. It means including. Of course, the communication network 110 according to an embodiment of the present invention is not limited to this, and can be used as an access network for a next-generation mobile communication system to be implemented in the future, for example, a cloud computing network in a cloud computing environment, a 5G network, etc. For example, if the communication network 110 is a wired communication network, the access point within the communication network can connect to the telephone company's exchange office, etc., but in the case of a wireless communication network, data is processed by connecting to the SGSN or GGSN (Gateway GPRS Support Node) operated by the communication company, or Data can be processed by connecting to various repeaters such as BTS (Base Transceiver Station), NodeB, and e-NodeB.

Communication network 110 may also include access points. Access points include small base stations such as femto or pico base stations that are often installed in buildings. Here, femto or pico base stations are classified according to the maximum number of imaging devices 100, etc., that can be connected in the classification of small base stations. Of course, the access point may include a short-distance communication module for performing short-distance communication such as Zigbee and Wi-Fi with the photographing device 100, etc. Access points can use TCP/IP or RTSP (Real-Time Streaming Protocol) for wireless communication. Here, in addition to Wi-Fi, short-range communication can be performed using various standards such as Bluetooth, Zigbee, infrared (IrDA), RF (Radio Frequency) such as UHF (Ultra High Frequency) and VHF (Very High Frequency), and ultra-wideband communication (UWB). You can. Accordingly, the access point extracts the location of the data packet, specifies the best communication path for the extracted location, and forwards the data packet along the designated communication path to the next device, such as the crowd risk management video analysis device 120. . Access points can share multiple lines in a typical network environment and include, for example, routers, repeaters, and repeaters.

The crowd risk management video analysis device 120 may be configured to include a server or DB 120a to form a crowd risk management system that can be used in both underpopulated and overcrowded environments, for example, in a CCTV environment. The crowd risk management video analysis device 120 detects crowd flow and abnormal behavior in a depopulated environment based on an object detector if the crowd is judged to be less than a certain crowd through a crowd counter, and if the crowd density is measured to be very overcrowded, the crowd coefficient Through engine and flow extraction, it is determined whether there is a risk of overcrowding and whether there is an abnormality in the crowd flow. Accordingly, when the crowd risk management video analysis device 120 is determined to be an abnormal situation, it operates to quickly transmit the situation to the control system to enable quick initial response. In addition, the crowd risk management video analysis device 120 can operate to prevent risks in advance by predicting the density of the crowd based on the generated crowd coefficient information.

More specifically, the crowd risk management video analysis device 120 according to an embodiment of the present invention analyzes the captured video and counts the crowd by detecting objects in the captured video, that is, in an environment with few people, or detects the object through the trajectory of the object. The flow can be analyzed through tracking. Additionally, behavior-based events can be detected for each detected individual object. On the other hand, the crowd risk management video analysis device 120 estimates the coefficient of the crowd in the overcrowded environment by using a density-based crowd coefficient estimator using a deep learning method when people in the captured video are in an overcrowded environment or estimates the crowd coefficient in the video. The density is obtained by expressing the location as a point and estimating it using a deep learning method to obtain the density. In addition, the crowd risk management video analysis device 120 extracts the optical flow, and for this purpose, extracts the crowd flow using the density map or crowd points extracted in the crowd coefficient extraction process. Next, the crowd risk management video analysis device 120 can generate an event by determining whether there is overcrowding and abnormal flow using object and flow information extracted through deep learning object detection tracking, crowd coefficient estimation, and optical flow extraction processes.

Above all, the crowd risk management video analysis device 120 according to an embodiment of the present invention plays an important role in determining whether to use the crowd flow information by the object detector or the crowd flow information by the crowd counter through the crowd coefficient extraction unit. do. For example, if the crowd counting value obtained from the crowd counter exceeds a certain pre-specified density value, the crowd counting cycle is adjusted upward (e.g., increasing the counting measurement cycle, etc.) and optical-based flow detection is started. Additionally, if the crowd count density of people is higher than the risk density K, the event due to object detection can be stopped.

The control device 130 receives information related to crowd counting from one or more crowd risk management video analysis devices 120, displays crowd density, flow, and risk information on a map, and analyzes the information within the video for each CCTV camera video. Displays crowd density and event alarms within the video. The control device 130 can configure a map-based control device (or first control device) for the former operation, and can also configure a video control-based control device (or second control device) for the latter operation. there is. The control device 130 may include a server or a computer managed by control personnel.

Figure 2 is a block diagram illustrating the detailed structure of the crowd risk management video analysis device and control device of Figure 1, Figure 3 is an example of counting a crowd using an object detection method or analyzing the flow through the trajectory of an object, and Figure 4 is an example. An example of the function to estimate the crowd coefficient based on density and an example of the function to estimate the crowd coefficient using the point estimation method are also shown in Figure 5. A function in which the area of the area specified by the user is automatically calculated as actual measurement when camera calibration is completed. , FIG. 6 is an example showing the moving direction and moving speed of an object as an Optical Flow Field, FIG. 7 is an example showing clustering and classifying the crowd flow into large flows, and FIG. 8 is the currently extracted crowd coefficient. A flowchart that determines the crowd count frequency and flow detection based on the value, Figure 9 is a diagram showing the role of each video analysis engine for each crowd density, and Figure 10 is a control system that maps to the crowd density map and monitors the crowd count value. This is an example of the screen.

As shown in Figure 2, the crowd risk management video analysis device 120 of Figure 1 according to an embodiment of the present invention includes an image input unit 200, a deep learning-based object detection and tracking unit 210, and a crowd coefficient estimation unit ( 220), an optical flow extraction unit 230, an object trajectory-based event detection unit 240, and a part or all of the crowd flow event detection unit 250.

Here, “including part or all” means that the crowd risk management video analysis device 120 is configured by omitting some components such as the object trajectory-based event detection unit 240, or the object trajectory-based event detection unit 240 and This means that some of the same components can be integrated into other components, such as the crowd flow event detection unit 250, and will be described as all-inclusive to facilitate a sufficient understanding of the invention.

The above image input unit 200, deep learning-based object detection and tracking unit 210, crowd coefficient estimation unit 220, optical flow extraction unit 230, object trajectory-based event detection unit 240, and crowd flow event detection unit 250. ) may be configured by hardware, software, or a combination thereof.

The video input unit 200 corresponds to a part of the video analysis device 120 that receives real-time video from the imaging device 100. Of course, it is also possible to receive captured images already stored in a storage medium. The operation of detecting and extracting pedestrians in a relatively sparse environment is performed through the deep learning-based object detection and tracking unit 210, which detects and tracks objects using a deep learning method in the image obtained from the image input unit 200.

The deep learning-based object detection and tracking unit 210 detects an object using a deep learning method and then determines whether or not it is the same object in the (video) frame and tracks it. The purpose is to apply it in an environment where the crowd density is rare, and when the size of the object is very small, increase the input image used for inference or utilize a network with a deep network depth. In addition, an object tracking algorithm is needed to extract the flow of detected objects, which usually uses the SORT (Simple Object Realtime Tracking) algorithm. To increase tracking performance, a Deep Sort method using deep learning features is used, or tracking is performed using a template matching method. Perform. This can be an important factor for events based on the trajectories of individual objects in environments where object density is very low.

Figure 3 is an example of a tracking result by object detection. Figure 3 (a) shows an example of tracking by object detection (e.g. object box) in a plaza, (b) is an example of object tracking on a campus, and (c) and (d) show an example of tracking by object detection. Each example of a trajectory display is shown. The first and second images, such as (a) and (b), are the results of object detection. Detection performance is guaranteed when the pedestrian's full body is sufficient. The remaining images, such as (c) and (d), are images that display the object tracking results in the form of a trajectory. Through this trajectory information, the movement direction and flow of the object can be detected.

The crowd coefficient estimation unit 220 estimates the coefficient of the crowd in an overcrowded environment using a density-based crowd coefficient estimator using a deep learning method, and also expresses the location of the crowd in the video as points and estimates this using a deep learning method to estimate the density. The density is obtained by estimating the obtained method.

In other words, the crowd coefficient estimation unit 220 is responsible for extracting or estimating the crowd coefficient in not only underpopulated but also overcrowded environments. This crowd coefficient is suitable for environments with very high crowd density, as shown in Figure 4. Since it is usually trained to count even when occluded with the head as GT (Ground Truth), it is robust in complex environments with high crowd density. In the case of crowd density, an estimated density image or point list is calculated, and the number and density of the crowd can be calculated based on this information. In the case of the Mecca pilgrimage image and the playground image, as shown in Figures 4 (a) to (f), this is the result of obtaining a density map from the input image. In addition, the estimated coefficients are also displayed on the left, that is, (b), (d), and (f). The third image (e) in Figure 4 is the result of estimating the crowd based on points. In Figure 4, (a) is the Mecca pilgrimage image (original + heat map), (b) is the density image (heat map), (c) is the playground image (original + heat map), (d) is the playground image (heat map) ), (e) represents the crowd image (GT: 760), and (f) represents the point-based estimation image, respectively.

Crowd density is expressed as the number of people per square meter. Therefore, it is necessary to map the area of interest specified in the image to real space. This makes it possible to map individual objects to real space through camera calibration information, and through this, density calculation is possible. Figure 5 is a diagram for explaining a method of measuring the actual size and position of an object from a single camera. Since various methodologies for camera calibration are already known in addition to the literature presented in the prior art, further explanation will be omitted. The second or bottom picture of FIG. 5 shows that the area is automatically calculated by specifying area A and area B in the image for which camera calibration has been completed.

In an embodiment of the present invention, the crowd coefficient estimation unit (or extraction unit) 220 plays an important role in determining whether to use crowd flow information by an object detector or crowd flow information by a crowd counter. As in steps S800 to S820 of FIG. 8, when the crowd count value obtained from the crowd counter exceeds a certain pre-specified density value (or reference value), the crowd count cycle is adjusted upward and optical-based flow detection begins. Additionally, if the crowd coefficient density of people is higher than the risk density K, the event by object detection can be stopped.

The optical flow extraction unit 230 receives a series of multiple (video) frames and estimates the flow of the crowd. In order to extract the optical flow, the density map or crowd points extracted from the crowd coefficient estimation unit 220 are input and the crowd flow is extracted.

More specifically, the optical flow extraction unit 230 extracts optical flow from the input image. When density is very high, detection and tracking of individual objects is difficult. It is very difficult to detect objects in the Mecca pilgrimage video of Figure 4. Therefore, the density map or points obtained from the crowd coefficient estimation unit 220 are used as the reference point for detecting the optical flow, and then the optical flow is obtained. Figure 6 is an example diagram of Optical Flow. The first video (a) is a video of three people moving in different directions to aid understanding. If you extract this input image as an Optical Flow Field image, you can obtain the image on the right (b). Additionally, marathon images (c) and (d) are ground truth images for the optical flow field images. Figure 7 shows the crowd flow clustered into large flows.

The object trajectory-based event detection unit 240 extracts behavior-based events for each detected individual object. That is, the object trajectory-based event detection unit 240 detects the event by trajectory detection by object tracking. Events based on the object's trajectory include the object wandering, passing the boundary line, and events based on object speed measurement. Additionally, if the size of the object is large enough for recognition, events such as falls and violence can also be detected. In other words, the behavior of individual objects is detected when the crowd density is relatively low. Event detection can detect events through comparison with event-related reference data stored on a rule basis, but since events can also be detected based on deep learning of artificial intelligence, in an embodiment of the present invention, any one There will be no particular limitation on the form.

The crowd flow event detection unit 250 receives object and flow information extracted from the deep learning-based object detection and tracking unit 210, the crowd coefficient estimation unit 220, and the optical flow extraction unit 230 to determine whether there is overcrowding and abnormal flow. Make a decision and generate an event. That is, the crowd flow event detection unit 250 detects events using object detection and flow information obtained from the deep learning-based object detection and tracking unit 210, the crowd coefficient estimation unit 220, and the optical flow extraction unit 230. The events that can be detected here include events about overcrowding and events about abnormal flow. In the case of an overcrowding event, when the crowd density appears to be very high (above the standard value), it can be judged as a dangerous situation. In the case of abnormal flow, when the density is above a certain level (or above the standard value), a situation of flow collision or stagnation is considered an abnormal flow. You can judge.

Figure 9 shows the role and scope of each engine according to the density of the crowd. The object detection and tracking engine is a deep learning-based object detection and tracking unit 210 and an object trajectory-based event detection unit 240, and is performed when the crowd density is relatively low. In the case of the crowd coefficient extraction engine, it monitors at a low cycle in normal times, but when the crowd coefficient exceeds a certain threshold, it performs analysis at a higher cycle, and the optical flow analysis engine is activated when switched to the concentrated crowd coefficient mode to detect the flow of the crowd. will be performed. Here, the engine refers to a program that performs core and essential functions in the computer field. A program that coordinates the overall operation of a series of programs that work together for a single purpose or performs a central function within an application program. The crowd flow event detection unit (or analysis unit) 250 functions to generate and manage crowd flow-related events by integrating all analysis results.

As the control device 130 of FIG. 1, the map-based monitoring system (or device) 131 of FIG. 2 collects crowd count-related information (e.g., crowd density, flow, risk) from one or more crowd risk management video analysis devices 120. etc.) and displays crowd density, flow, and risk information on the map, and the video control-based monitoring system (or device) 132 reports crowd density within the video and event alarms within the video for each CCTV camera video. Display.

Figure 10 is a diagram explaining the function of displaying information received from a video analysis device on a map-based control screen. It displays the crowd density map or the location of objects by mapping them to points, allowing the direction and flow of movement on the map to be known. It also shows the function of displaying a graph so that you can see changes in crowd density at each major point. Embodiments of the present invention focus on being able to simultaneously display events related to crowd density while also displaying events occurring in a crowd density environment.

Figure 11 is a block diagram illustrating another detailed structure of the crowd risk management video analysis device of Figure 1.

As shown in FIG. 11, the crowd risk management video analysis device 120' according to another embodiment of the present invention includes a communication interface unit 1100, a control unit 1110, a crowd risk management unit 1120, and a storage unit 1130. ) includes part or all of.

Here, "including part or all" means that the crowd risk management video analysis device 120' is configured by omitting some components such as the storage unit 1130, or some components such as the crowd risk management unit 1120. This means that it can be integrated and configured with other components such as the control unit 1110, and is explained as being fully included to facilitate a sufficient understanding of the invention.

The communication interface unit 1100 communicates with the imaging device 100 and the control device 130 via the communication network 110 of FIG. 1, respectively. The communication interface unit 1100 can perform operations such as modulation/demodulation and encoding/decoding during communication, and since this is obvious to those skilled in the art, further description will be omitted.

The communication interface unit 1100 may receive captured images provided from a photographing device 100 such as CCTV and provide the captured images to the control unit 1110. Additionally, the communication interface unit 1100 may notify the control device 130 of an event under the control of the control unit 1110 when an abnormal flow of crowds is detected in the filming area of the photographing device 100. In other words, the communication interface unit 1100 provides crowd count-related information according to the video analysis results and allows the control device 130 to display crowd density, flow, and risk information on the map. Additionally, the communication interface unit 1100 may perform related operations in the control device 130 to display the crowd density within the image and the event alarm within the image for each CCTV camera image.

The control unit 1110 is responsible for the overall control operation of the communication interface unit 1100, crowd risk management unit 1120, and storage unit 1130 of FIG. 11. Typically, the control unit 1110 temporarily stores the image data of the captured image provided by the communication interface unit 1100 in the storage unit 1130, retrieves it, and provides it to the crowd risk management unit 1120 for image analysis. Additionally, the control unit 1110 may operate in conjunction with the crowd risk management unit 1120 for crowd risk management. In other words, the crowd risk management unit 1120 may provide the relevant judgment result to the control unit 1110 when it is determined that the area being filmed by the imaging device 100 is overcrowded and is in a dangerous situation. Accordingly, the control unit (1110) 1110) controls the communication of the communication interface unit 110 to transmit crowd count-related information to the control device 130 of FIG. 1, such as sending crowd density, flow, and risk information on the map to the monitor of the control device 130. It can be expressed.

The crowd risk management unit 1120 analyzes captured images, such as CCTV, and performs actions to manage crowd risk. Of course, the crowd risk management unit 1120 according to an embodiment of the present invention can detect objects based on deep learning in captured images and track the detected objects in a depopulated environment where people are not overcrowded, rather than only managing crowd risks. , events can be detected based on the tracking results of the object and appropriate actions can be taken. For example, it is possible to detect high-level behavioral events such as falls and acts of violence that can occur in a depleted environment. Detection of these events is done by pre-storing standard data related to event detection based on rules and comparing them with the corresponding standard data. Events can be detected, and it may also be possible to detect events through learning by applying an artificial intelligence program.

In addition, in the process of analyzing the filmed video, the crowd risk management department (1120) determines whether the crowd density is overcrowded beyond a certain crowd through a crowd counter and determines whether there is a risk due to crowd overcrowding and whether there is an abnormality in the crowd flow through the crowd counting engine and flow extraction. can be determined. Of course, here, the engine can be seen as executing a program to determine whether there is a risk from overcrowding and whether there is an abnormality in the crowd flow. To this end, in an embodiment of the present invention, when a crowd is dense, the optical flow of the moving direction and moving speed of an object is determined. Of course, movement speed can be seen as measured through changes in the image. In the case of an overcrowding event, when the crowd density appears to be much higher than the standard value, it can be judged as a dangerous situation and notified to the control center so that appropriate measures can be taken. In the case of abnormal flow, when the density is above a certain level, flow collision or congestion can occur. can be judged as an abnormal flow and appropriate measures can be taken.

The storage unit 1130 can temporarily store various types of data processed under the control of the control unit 1110. The storage unit 1130 may temporarily store image data of the captured image provided by the imaging device 100 of FIG. 1 and then provide the image data to the crowd risk management unit 1120 for image analysis.

In addition to the above, the communication interface unit 1100, control unit 1110, crowd risk management unit 1120, and storage unit 1130 of FIG. 11 can perform various operations, and other details have been sufficiently explained previously. I want to replace it with fields.

The communication interface unit 1100, control unit 1110, crowd risk management unit 1120, and storage unit 1130 of FIG. 11 according to an embodiment of the present invention are composed of hardware modules that are physically separated from each other, but each module has an internal It will be possible to store software to perform the above operations and execute it. However, the software is a set of software modules, and each module can be formed of hardware, so there will be no particular limitation on the configuration of software or hardware. For example, the storage unit 1130 may be hardware, such as storage or memory. However, since it is possible to store information through software (repository), the above content will not be specifically limited.

Meanwhile, as another embodiment of the present invention, the control unit 1110 may include a CPU and memory, and may be formed as a single chip. The CPU includes a control circuit, an arithmetic unit (ALU), an instruction interpretation unit, and a registry, and the memory may include RAM. The control circuit performs control operations, the operation unit performs operations on binary bit information, and the command interpretation unit includes an interpreter or compiler, which can convert high-level language into machine language and machine language into high-level language. , the registry may be involved in software data storage. According to the above configuration, for example, at the beginning of the operation of the crowd risk management video analysis device 120', data operation is performed by copying the program stored in the crowd risk management unit 1120, loading it into memory, that is, RAM, and then executing it. Processing speed can be quickly increased. In the case of deep learning models, they can be loaded into GPU memory rather than RAM and executed by accelerating the execution speed using GPU.

Figure 12 is a flow chart of a crowd risk management method according to an embodiment of the present invention.

For convenience of explanation, referring to FIG. 12 together with FIGS. 1 and 2, the crowd risk management video analysis device 120 of FIG. 1 analyzes the captured video of the photographing device 100 installed in a place where crowds are concentrated and provides analysis results. When the crowd density is lower (or lower) than the standard value, the movement direction and abnormal behavior of the crowd are detected through object detection and tracking of the detection object, and when the crowd density is higher (or higher than the standard value), the movement of the crowd in the captured video is detected. By tracking the optical flow related to the crowd, it determines whether there is overcrowding and abnormal flow and generates an event (S1200).

The crowd risk management video analysis device 120 can estimate the crowd coefficient using a density-based crowd coefficient estimator using a deep learning method to estimate the crowd coefficient in the captured video, and can also estimate the crowd coefficient in the video by using a heat map. You can obtain the actual density using this method, and furthermore, you can estimate the crowd coefficient in various ways, such as by expressing the position of the crowd in the video as a point and estimating it using a deep learning method to obtain the density.

In addition, the crowd risk management video analysis device 120 estimates the crowd flow using a series of multiple frames of the captured video to determine the crowd flow within the captured video, and uses the crowd coefficient estimation unit to extract the optical flow. The crowd flow is extracted using the extracted density map or crowd points. Obtain the optical flow using the extracted density map or crowd points as the starting point.

Furthermore, the crowd risk management video analysis device 120 detects events using object detection and flow information obtained from the deep learning-based object detection and tracking unit, crowd coefficient estimation unit, and optical flow extraction unit, and in the case of overcrowded events, crowd If the density appears to be very high above the standard value, for example, it can be identified as the density per unit area, and in the case of abnormal flow, when the density is above a certain level, that is, above the standard value, flow collision or stagnation situation can be judged as abnormal flow and an event can be generated. .

Meanwhile, the control device 130 receives crowd count information of the crowd from the crowd risk management video analysis device 120 and displays crowd density, crowd flow, and risk information on the map (S1210). Of course, crowd counting information can be seen to include information such as crowd density, crowd flow, and risk. Additionally, the control device 130 may be able to display the information separately by photographing device 100 when displaying it on the map based on the corresponding crowd count information.

In addition to the above, the crowd risk management video analysis device 120 and control device 130 of FIGS. 1 and 2 can perform various operations, and other details have been sufficiently explained previously, so these will be replaced.

Meanwhile, even though all the components constituting the embodiment of the present invention are described as being combined or operating in combination, the present invention is not necessarily limited to this embodiment. That is, as long as it is within the scope of the purpose of the present invention, all of the components may be operated by selectively combining one or more of them. In addition, although all of the components may be implemented as a single independent hardware, a program module in which some or all of the components are selectively combined to perform some or all of the combined functions in one or more pieces of hardware. It may also be implemented as a computer program with . The codes and code segments that make up the computer program can be easily deduced by a person skilled in the art of the present invention. Such computer programs can be stored in non-transitory computer readable media and read and executed by a computer, thereby implementing embodiments of the present invention.

Here, a non-transitory readable recording medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. . Specifically, the above-described programs may be stored and provided on non-transitory readable recording media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, etc.

In the above, preferred embodiments of the present invention have been shown and described, but the present invention is not limited to the specific embodiments described above, and may be used in the technical field to which the invention pertains without departing from the gist of the invention as claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be understood individually from the technical idea or perspective of the present invention.

100: photographing device 110: communication network
120, 120': Crowd risk management video analysis device 130: Control device
1100: Communication interface unit 1110: Control unit
1120: Crowd risk management department 1130: Storage department

Claims (10)

By analyzing video footage from a recording device installed in a place where crowds are dense, when the crowd density is lower than the standard value as a result of the analysis, the movement direction and abnormal behavior of the crowd are detected through object detection and tracking of the detection object, and the crowd density is below the standard value. A crowd risk management video analysis device that tracks the optical flow related to the movement of the crowd in the captured video when the crowd is higher and determines whether the crowd is overcrowded and abnormal flow to generate an event; and
A control device that receives the crowd count information of the crowd from the crowd risk management video analysis device, displays crowd density, crowd flow, and risk information on a map, and displays it in a graph so that changes in crowd density can be seen at each point; Including,
The crowd risk management video analysis device,
A deep learning-based object detection and tracking unit that detects an object in the captured image using a deep learning method and then determines whether the object is the same in the video frame of the captured image to track the detected object;
an object trajectory-based event detection unit that detects the movement direction of the crowd and events of abnormal behavior by detecting a trajectory by tracking the detection object when the crowd density is lower than the reference value;
The crowd coefficient is estimated by analyzing the captured video, the location of the crowd within the captured video is expressed as a point, and the expressed point is estimated using a deep learning method to obtain a density map. Crowd coefficient estimator to obtain;
an optical flow extraction unit that extracts a crowd flow based on the optical flow of the crowd coefficient extracted by the crowd coefficient estimation unit; and
A crowd flow event detection unit that determines the crowd density per unit area and the abnormal flow of the crowd based on the tracking result of the detected object in the deep learning-based object detection and tracking unit, the estimated crowd coefficient, and the extracted optical flow. Contains,
The optical flow extraction unit extracts the optical flow using the density map or crowd points indicating the density obtained from the crowd coefficient estimation unit as a reference point for detecting the optical flow as a starting point,
The crowd flow event detection unit determines whether a flow collision or congestion situation occurs at a density higher than a reference value as the abnormal flow of the crowd,
The crowd coefficient estimation unit,
When the crowd counting value obtained from the crowd counter exceeds the specified standard value, the crowd counting cycle is adjusted upward to increase the counting measurement cycle, and optical-based flow detection begins. If the density of crowd counting people is higher than the risk density K, the object is detected. A crowd risk management system that supports under- and overcrowded environments, stopping events caused by . delete delete delete delete A crowd risk management video analysis device analyzes video footage from a recording device installed in a place where crowds are concentrated, and when the crowd density is lower than the standard value, the crowd's movement direction and abnormal behavior are detected through object detection and tracking of the detected object. Detecting, when the crowd density is higher than a reference value, tracking optical flow related to the movement of the crowd in the captured image to determine whether the crowd is overcrowded and abnormal flow to generate an event; and
The control device receives the crowd count information of the crowd from the crowd risk management video analysis device, displays crowd density, crowd flow, and risk information on the map, and displays it in a graph so that changes in crowd density can be seen at each point. Including the step of:
The step of generating the event is,
A deep learning-based object detection and tracking unit of the crowd risk management video analysis device detects an object in the captured image using a deep learning method, and then determines whether the object is the same in a video frame of the captured image to track the detected object;
Detecting, by the object trajectory-based event detection unit of the crowd risk management video analysis device, a movement direction of the crowd and an event of abnormal behavior by detecting a trajectory by tracking the detection object when the crowd density is lower than a reference value;
The crowd coefficient estimation unit of the crowd risk management video analysis device analyzes the captured video to estimate the crowd coefficient, expresses the location of the crowd in the captured video as a point, and estimates the expressed point using a deep learning method. Obtaining the density by estimating it by obtaining a density map;
Extracting, by the optical flow extraction unit of the crowd risk management video analysis device, a crowd flow based on the optical flow of the crowd coefficient extracted by the crowd coefficient estimation unit; and
The crowd flow event detection unit of the crowd risk management video analysis device determines the crowd density per unit area based on the tracking result of the detected object in the deep learning-based object detection and tracking unit, the estimated crowd coefficient, and the extracted optical flow. Including: determining the abnormal flow of the crowd,
The step of extracting the crowd flow is,
Extracting the optical flow using the density map or crowd points indicating the density obtained from the crowd coefficient estimation unit as a reference point for detecting the optical flow as a starting point,
The step of determining the abnormal flow is,
As the abnormal flow of the crowd, determine whether a flow collision or congestion situation occurs at a density higher than the reference value,
When the crowd count value obtained from the crowd counter exceeds the specified reference value, the crowd count estimation unit starts optical-based flow detection by upwardly adjusting the crowd count period to increase the count measurement period, and the density of crowd count people increases to the risk density K. stopping the event caused by the object detection if it is higher than
Crowd risk management methods that support overcrowded and overcrowded environments, including: delete delete delete delete