KR102673628B1 - Method and apparatus for monitoring crowd density based on map cell for situational awareness in digital twin - Google Patents

Abstract

A map cell-based density monitoring method and device for digital twin situation awareness are disclosed. A map cell-based density monitoring method for digital twin situation awareness according to an embodiment of the present invention includes the steps of collecting map information of a specific area, and collecting the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in the specific area. A step of collecting surveillance camera information and surveillance shooting data, dividing the road area included in the shooting coverage of each surveillance camera, and subdividing the divided road area into road cells of a set size, and pre-learned Using a people counting video analysis algorithm, calculating the number of people within the shooting coverage of each surveillance camera based on the shooting data of each surveillance camera, and calculating the number of people within the shooting coverage of each surveillance camera within the shooting coverage of each surveillance camera Calculating the average number of road crowds by dividing by the number of road cells; If the shooting coverage of surveillance cameras overlaps, calculating the number of crowds by road cell by correcting the average number of crowds by road cells in the overlapping area; and calculating the number of crowds by road cell by calculating the average number of crowds by road cells. It may include a step of deriving a crowd density distribution map in a specific area based on the number, and a step of monitoring the crowd density situation in a specific area based on the derived crowd density distribution map.

Description

Map cell-based crowd density monitoring method and device for digital twin situational awareness {METHOD AND APPARATUS FOR MONITORING CROWD DENSITY BASED ON MAP CELL FOR SITUATIONAL AWARENESS IN DIGITAL TWIN}

The present invention utilizes people counting technology to recognize crowd density situations based on map cells for digital twin situation awareness, and monitors crowd density based on map cells for digital twin situation awareness. It relates to methods and devices.

In general, a digital twin is a realization of machines, equipment, objects, etc. from the real world in a virtual world in a computer. An object (twin) identical to the real thing is created in a virtual space and verified through various mock tests (simulations). It refers to the technique of trying. This digital twin technology can improve accuracy as it collects vast amounts of information through the Internet of Things (IoT).

Using digital twin technology, you can monitor the status of equipment and systems in the virtual world, identify maintenance points, and make improvements. In addition, the risk of accidents can be reduced by predicting various situations that may occur during operation to verify safety or prevent unexpected accidents.

On the other hand, if the crowd density increases beyond the critical point, the possibility of a safety accident increases dramatically, so it is necessary to set standards for accepting floating populations within a limited area and closely manage them.

According to one study, the risk of an accident can be judged based on how many people are standing on a piece of land measuring 1m horizontally and vertically, that is, 1m ² (square meter). For example, 1-2 people per 1m ² can be judged as a very relaxed and free movement situation, and 3 people can be judged as a bit crowded but not too crowded. In addition, in the case of 4 people per 1m ² , the distance between people becomes narrower, but it can be judged as a situation where personal space, which is an area very close to the body, is not invaded. However, when the number of people exceeds 5, physical contact begins to increase among the crowd, so it may be safe when you are a spectator or audience watching a performance, but if there is a situation where people push each other, it can be judged to be a critical point where it starts to become a problem. there is.

And when it reaches 6 people per ^1m2 , it can be judged that the situation starts to become dangerous. In these situations, there is a lot of physical contact, it becomes difficult for each person to maintain a wide posture, and people can easily lose control of their own movements, such as falling down.

However, even if there is a density standard as above, it may not be easy to apply it in the actual field to determine the level of risk. For example, at first glance, there may not seem to be a big difference between 4 people per 1 m 2 and 6 people per 1 m ² .

Accordingly, there is a need for research on how to properly determine density based on various criteria other than just the number of people. In addition, as problems with passive security surveillance systems continue to arise, the need for automated and intelligent crowd density measurement that supports real-time management and supervision of crowds in public places is increasing.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known art disclosed to the general public before filing the application for the present invention.

Korean Patent Publication No. 10-2022-0162351 (2022.12.08) Korean Patent Publication No. 10-1556693 (2015.09.23)

Therefore, the technical problem that the present invention aims to solve is to recognize the crowd density situation based on map cells by utilizing people counting technology for digital twin situation recognition.

In addition, the technical problem that the present invention seeks to solve is to monitor the number of crowds and crowd movement trends analyzed based on people counting technology from surveillance cameras in the entire area in connection with map information of a specific area by mapping them to map cells. .

The object of the present invention is not limited to the problems mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood through the following description and will be more clearly understood through examples of the present invention. Additionally, it will be appreciated that the objects and advantages of the present invention can be realized by means and combinations thereof as indicated in the patent claims.

A map cell-based crowd density monitoring method for digital twin situation awareness according to an aspect of the present invention includes the steps of collecting map information of a specific area, and collecting the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in the specific area. A step of collecting surveillance camera information and surveillance shooting data, dividing the road area included in the shooting coverage of each surveillance camera, and subdividing the divided road area into road cells of a set size, and pre-learned Using a people counting video analysis algorithm, calculating the number of people within the shooting coverage of each surveillance camera based on the shooting data of each surveillance camera, and calculating the number of people within the shooting coverage of each surveillance camera within the shooting coverage of each surveillance camera Calculating the average number of road crowds by dividing by the number of road cells; If the shooting coverage of surveillance cameras overlaps, calculating the number of crowds by road cell by correcting the average number of crowds by road cells in the overlapping area; and calculating the number of crowds by road cell by calculating the average number of crowds by road cells. It may include a step of deriving a crowd density distribution map in a specific area based on the number, and a step of monitoring the crowd density situation in a specific area based on the derived crowd density distribution map.

In addition, another method for implementing the present invention, another system, and a computer-readable recording medium storing a computer program for executing the method may be further provided.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to an embodiment of the present invention, one task of the embodiment of the present disclosure is to recognize the crowd density situation based on map cells by utilizing people counting technology to recognize the digital twin situation, thereby pre-empting the crowd density situation. Monitoring can be made possible.

In addition, according to an embodiment of the present invention, the number of crowds and crowd movement trends analyzed based on people counting technology from surveillance cameras in the entire area can be monitored by mapping them to map cells in connection with map information of a specific area. Through this, the crowd density situation can be monitored at all times to enable disaster safety response in response to overcrowding.

In addition, according to an embodiment of the present invention, it is possible to calculate the crowding risk by determining not only the number of crowds in a specific area but also the speed of crowd movement in a specific area, thereby enabling more accurate monitoring of the crowding risk.

In addition, according to an embodiment of the present invention, the accuracy of situation analysis can be improved by performing a correction process by reflecting the results analyzed from a plurality of surveillance cameras installed in a specific area in the map cell (road cell).

In addition, according to an embodiment of the present invention, by specifying a color for a map cell and enabling monitoring, it is possible to more intuitively identify density and risk, thus improving user satisfaction and situational awareness accuracy.

In addition, according to an embodiment of the present invention, the ease of situation awareness monitoring can be improved by providing LOD (Level of Detail) for situation awareness using the concept of a cell group, which is a collection of map cells.

In addition, according to an embodiment of the present invention, based on digital twin technology, it is possible to enable situation awareness using various sensors and enable situation awareness through real-time analysis of surveillance camera images.

The effects of the present invention are not limited to those mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically showing a crowd density monitoring system according to an embodiment.
Figure 2 is a block diagram schematically showing a crowd density monitoring device according to an embodiment.
Figure 3 is an example diagram to explain the process of calculating the number of crowds for each road cell according to an embodiment.
Figure 4 is an example diagram for explaining the process of calculating the number of crowds for each road cell in an overlapping area according to an embodiment.
Figure 5 is an example diagram showing the crowd density distribution as a heat map according to an embodiment.
Figure 6 is an example diagram to explain a process of managing density by grouping road cells according to an embodiment.
Figure 7 is an example diagram illustrating a process of calculating crowd movement speed for each road cell according to an embodiment.
Figure 8 is an example diagram showing the crowd speed distribution as a heatmap according to an embodiment.
Figure 9 is an example diagram showing the crowd density risk distribution map as a heat map according to an embodiment.
Figure 10 is a flowchart to explain a crowd density monitoring method according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail together with the accompanying drawings.

However, the present invention is not limited to the embodiments presented below, but may be implemented in various different forms, and should be understood to include all conversions, equivalents, and substitutes included in the spirit and technical scope of the present invention. . The embodiments presented below are provided to ensure that the present invention is complete and to fully inform those skilled in the art of the present invention of the scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may obscure the gist of the present invention, the detailed description will be omitted.

The terms used in this application are only used to describe specific embodiments and are not intended to limit the invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof. Terms such as first, second, etc. may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

Hereinafter, embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components are assigned the same drawing numbers and duplicate descriptions thereof are omitted. I decided to do it.

1 is a diagram schematically showing a crowd density monitoring system according to an embodiment.

Referring to FIG. 1 , the crowd density monitoring system 1 may include a crowd density monitoring device 100, a user terminal 200, a server 300, and a network 400.

The crowd density monitoring system 1 of one embodiment provides situational awareness based on map cells linked to map information in digital twin technology. In addition, the crowd density monitoring system 1 of one embodiment utilizes people counting technology through various sensors (e.g., multiple surveillance cameras) installed in a specific area to monitor the crowd density situation, such as the number of crowds and crowd movement trends analyzed. It is mapped to a map cell to enable monitoring.

In other words, the crowd density monitoring system 1 of one embodiment can be said to be a map cell-based crowd density monitoring system for digital twin situation awareness.

In general, in digital twin technology, it is important to use various sensors to recognize the situation. For example, surveillance cameras included in various sensors can be used to enable situational awareness through real-time analysis.

That is, in one embodiment, the crowd density monitoring system 1 can enable real-time situation awareness through a plurality of surveillance cameras installed in a specific area, and in particular, utilizes video analysis-based people counting technology to map cells. The mapped crowd density can make it possible to perceive the situation more intuitively.

In addition, the crowd density monitoring system 1 of one embodiment can improve the accuracy of situation analysis by reflecting the results analyzed from a plurality of surveillance cameras installed in a specific area and correcting them in the map cells. By assigning colors to cells and enabling monitoring, it is possible to identify density and risk more intuitively, thereby improving user satisfaction and situational awareness accuracy.

Meanwhile, in one embodiment, a map cell means that a specific area is divided into cells of a specific size in connection with map information. In one embodiment, in particular, roads including all areas where people can move are divided into cells, so hereafter: It can be written as a road cell. In one embodiment, map cell and road cell may have the same meaning.

The crowd density monitoring system (1) of one embodiment relates to a crowd density situation recognition technology in urban areas using surveillance camera people counting technology, and is based on a conventional technology for estimating the number of crowds by analyzing surveillance camera images. By linking with map information in urban areas, it is possible to identify crowd movement trends analyzed from surveillance cameras throughout the area. Through this, the crowd density monitoring system 1 can constantly check the crowd density situation and enable disaster safety response according to overcrowding.

Meanwhile, in one embodiment, users access an application or website implemented on the user terminal 200, request a crowd density monitoring service, and use (confirm) the service in the monitoring interface provided by the crowd density monitoring device 100. ), and you can enter (set) detailed settings for monitoring crowd density (for example, risk coefficient, etc.).

In one embodiment, the user terminal 200 may receive a crowd density monitoring service through an authentication process after accessing a crowd density monitoring application or website. The authentication process may include, but is not limited to, authentication of entering user information such as membership registration, authentication of a user terminal, etc., but is not limited to this, and accesses a link transmitted from the crowd density monitoring device 100 and/or the server 300. The authentication process may be performed simply by doing so.

These user terminals 200 include desktop computers, smartphones, laptops, tablet PCs, smart TVs, mobile phones, personal digital assistants (PDAs), laptops, media players, micro servers, and global positioning system (GPS) devices operated by the user. , e-book terminals, digital broadcasting terminals, navigation devices, kiosks, MP3 players, digital cameras, home appliances, and other mobile or non-mobile computing devices, but are not limited thereto.

Additionally, the user terminal 200 may be a wearable terminal such as a watch, glasses, hair band, or ring equipped with a communication function and data processing function. The user terminal 200 is not limited to the above-described content, and any terminal capable of web browsing may be used without limitation.

Meanwhile, in one embodiment, the crowd density monitoring system 1 may be implemented by the crowd density monitoring device 100 and/or the server 300. In other words, the crowd density monitoring device 100 may be partially or entirely implemented as the server 300.

That is, the crowd density monitoring device 100 may be implemented in the server 300, where the server 300 is a server for operating the crowd density monitoring system 1 including the crowd density monitoring device 100 or the crowd density monitoring device 100. It may be a server that implements part or all of the monitoring device 100.

In one embodiment, the server 300 collects surveillance camera information and surveillance shooting data including map information of a specific area, the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in a specific area, and collects surveillance camera information and surveillance shooting data for each surveillance camera. The road area included in the shooting coverage is subdivided into road cells of a set size, the number of crowds and the average number of road crowds within the shooting coverage of each surveillance camera are calculated, and the average number of road crowds in the overlapping area is corrected to calculate the number of crowds for each road cell. A crowd density monitoring device for the overall process, such as calculating the crowd density distribution in a specific area based on the calculated number of crowds for each road cell, and monitoring the crowd density situation in a specific area based on the derived crowd density distribution map. It may be a server that controls the operation of (100).

In addition, the server 300 manages, for example, not only the configuration of the crowd density monitoring device 100, but also new registration of related devices, network environment, etc. to facilitate service provision and management in the crowd density monitoring platform. It may be a server in charge of operating various devices so that they can be performed properly.

Additionally, the server 300 may be a database server that provides data for operating the crowd density monitoring device 100. Additionally, the server 300 may include a web server, an application server, or a deep learning network provision server.

Additionally, the server 300 may include a big data server and an AI server required to apply various artificial intelligence algorithms, and a calculation server that performs calculations of various algorithms.

Additionally, in this embodiment, the server 300 may include the servers described above or may be networked with these servers. That is, in this embodiment, the server 300 may include the above-mentioned web server and AI server or may be networked with these servers.

Meanwhile, in the crowd density monitoring system 1, the crowd density monitoring device 100 and the server 300 may be connected by a network 400. These networks 400 include, for example, wired networks such as local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), and integrated service digital networks (ISDNs), or wireless LANs, CDMA, Bluetooth, and satellite communications. It may cover wireless networks such as, but the scope of the present invention is not limited thereto. Additionally, the network 400 may transmit and receive information using short-range communication and/or long-distance communication.

Additionally, the network 400 may include connections of network elements such as hubs, bridges, routers, switches, and gateways. Network 400 may include one or more connected networks, including public networks such as the Internet and private networks such as secure corporate private networks, such as a multi-network environment. Access to network 400 may be provided through one or more wired or wireless access networks. Furthermore, the network 400 may support an IoT (Internet of Things) network and/or 5G communication that exchanges and processes information between distributed components such as objects.

Figure 2 is a block diagram schematically showing a crowd density monitoring device according to an embodiment.

Referring to FIG. 2 , the crowd density monitoring device 100 may include a communication interface 110, a user interface 120, a memory 130, and a processor 140.

The communication interface 110 may provide a communication interface necessary to provide transmission and reception signals between external devices in the form of packet data in conjunction with the network 400. Additionally, the communication interface 110 may be a device that includes hardware and software necessary to transmit and receive signals such as control signals or data signals through wired or wireless connections with other network devices.

This communication interface 110 can support various object intelligence communications (IoT (internet of things), IoE (internet of everything), IoST (internet of small things), etc.), M2M (machine to machine) communication, V2X ( It can support vehicle to everything communication (D2D) communication and D2D (device to device) communication.

That is, the processor 140 can receive various data or information from an external device connected through the communication interface 110, and can also transmit various data or information to the external device. And, the communication interface 110 may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, and an NFC module.

The user interface 120 may include an input interface through which user requests and commands for the crowd density monitoring platform provided by the crowd density monitoring device 100 are input. For example, a user may request to receive a crowd density monitoring service through the user interface 120.

In addition, the user interface 120 provides detailed settings for calculating the operation of the crowd density monitoring device 100 (e.g., crowd density distribution, crowd movement speed distribution, crowd density risk, etc.), parameters and learning conditions for the people counting video analysis algorithm. It may include an input interface through which user requests and commands for controlling changes (changes, etc.) are input.

Additionally, the user interface 120 may include an output interface through which a monitoring screen for the crowd density monitoring platform is output (see FIGS. 5, 8, and 9). That is, the user interface 120 can output results according to user requests and commands. The input interface and output interface of this user interface 120 may be implemented in the same interface.

Meanwhile, in one embodiment, the crowd density monitoring device 100 may include a surveillance camera (not shown) as a component, but the surveillance camera is configured separately and is located from the separate surveillance camera through the communication interface 110. Surveillance camera information including shooting coverage and identification information and shooting data can be collected.

In addition, in one embodiment, the surveillance camera may be a CCTV (Closed-circuit Television), which is a closed-circuit camera installed nearby for safety purposes such as crime prevention, surveillance, and fire prevention, but is not limited thereto, and may be a general camera, depth camera, or thermal camera. It may include a video camera, etc.

In addition, in one embodiment, it is explained that people counting is performed by analyzing images collected through surveillance cameras, but various other people counting technologies can be used. That is, in addition to surveillance cameras, various people counting devices or sensors (for example, object detection sensors, radar sensors, lidar sensors, etc.) can be applied.

The memory 130 stores various information necessary for controlling (computing) the operation of the crowd density monitoring device 100 and/or the server 300, and can store control software, and includes a volatile or non-volatile recording medium. can do.

The memory 130 is connected to one or more processors 140 by an electrical or internal communication interface and, when executed by the processor 140, causes the processor 140 to control the crowd density monitoring device 100. ) Codes can be saved.

Here, the memory 130 may include a non-transitory storage medium such as magnetic storage media or flash storage media, or a temporary storage medium such as RAM, but is within the scope of the present invention. is not limited to this. This memory 130 may include internal memory and/or external memory, volatile memory such as DRAM, SRAM, or SDRAM, one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, Non-volatile memory, such as NAND flash memory, or NOR flash memory, SSD. It may include a flash drive such as a compact flash (CF) card, SD card, Micro-SD card, Mini-SD card, Xd card, or memory stick, or a storage device such as an HDD.

Additionally, information related to an algorithm for performing learning according to the present invention may be stored in the memory 130. In addition, various information necessary within the scope of achieving the purpose of the present invention may be stored in the memory 130, and the information stored in the memory 130 may be updated as it is received from a server or external device or input by the user. It may be possible.

The processor 140 may control the overall operation of the crowd density monitoring device 100. Specifically, the processor 140 is connected to the configuration of the crowd density monitoring device 100 including the memory 130, and operates the crowd density monitoring device 100 by executing at least one command stored in the memory 130. can be controlled overall.

The processor 140 may be implemented in various ways. For example, the processor 140 may include an application specific integrated circuit (ASIC), an embedded processor, a microprocessor, hardware control logic, a hardware finite state machine (FSM), and a digital signal processor. Processor, DSP).

The processor 140 is a type of central processing unit and can control the operation of the crowd density monitoring device 100 by running control software mounted on the memory 130. Processor 140 may include all types of devices capable of processing data. Here, 'processor' may mean, for example, a data processing device built into hardware that has a physically structured circuit to perform a function expressed by code or instructions included in a program.

Figure 3 is an example diagram for explaining a process for calculating the number of crowds for each road cell according to an embodiment, Figure 4 is an example diagram for explaining a process for calculating the number of crowds for each road cell in an overlapping area according to an embodiment, and Figure 5 is an example diagram for explaining the process for calculating the number of crowds for each road cell in an overlapping area. This is an example diagram showing the crowd density distribution as a heat map according to an embodiment, and Figure 6 is an example diagram to explain the process of managing density by grouping road cells according to an embodiment.

The processor 140 may collect map information for a specific area. In one embodiment, the processor 140 may collect map information from a map information database server that stores map information in a database. At this time, the processor 140 may receive map information of the corresponding area from the map information database server according to the requested location. Map information may include, for example, all information that can be found on a map, such as topographical maps of the entire country and administrative district information, and may include 2-dimensional and 3-dimensional map information.

For example, in one embodiment, the map information database server may be a GIS (geographic information system) server. Therefore, in the map information database server, geospatial information of the entire country can be digitized and created into a digital map, which can be used for disaster, environment, facility, national space management and administrative services through various information and communication technologies.

In one embodiment, GIS refers to an integration of computer hardware and software, geographic data, and human resources designed to effectively collect, store, update, coordinate, analyze, and express all forms of geographically referenceable information, and It can be said to be a computer system that collects data describing a location and makes it available. In addition, GIS stores, extracts, and stores information by linking topographic information that spatially expresses location and non-geometric attribute information that explains and supplements its form and function to graphics and database management functions for reference of various earth surface information. It is an information system-related technology that supports users through management and analysis. It is also a comprehensive analysis tool that expresses the spatial relationships of maps by adding characteristic (attribute) information of geographical information.

The processor 140 may collect surveillance camera information including the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in a specific area, and shooting data from the surveillance cameras. In one embodiment, shooting data may be collected from a surveillance camera in real time, or shooting data may be collected according to a set schedule, but the present invention is not limited to this and may change depending on settings.

The processor 140 may distinguish road areas included in the shooting coverage of each of the plurality of surveillance cameras, and subdivide the divided road areas into road cells of a set size. For example, the size of the road cell may be set to ^60cm

That is, in one embodiment, the processor 140 may determine the number of road cells in the road area included in the shooting coverage of each of the plurality of surveillance cameras, for example, in the shooting coverage of surveillance camera A among the plurality of surveillance cameras. The number of road cells included can be expressed as Total Road[A], that is, TR[A].

The processor 140 may generate a polygon on the captured data of the surveillance camera based on the captured data of the surveillance camera and distinguish road objects among objects in the captured data. In one embodiment, various methods for classifying objects in an image other than the method of generating polygons may be applied.

The processor 140 may use a previously learned people counting video analysis algorithm to calculate the number of crowds within the shooting coverage of each surveillance camera based on the shooting data of each surveillance camera. In one embodiment, the number of people in surveillance camera A among a plurality of surveillance cameras can be expressed as Total People[A], that is, TP[A].

In one embodiment, the previously learned people counting image analysis algorithm may be implemented in the processor 140, but only the results may be received after people are counted in a separately configured device or server. That is, the processor 140 can receive a video (or image) of a crowd, recognize the crowd in the video (or image), and count the number. At this time, the people counting method may be configured as a program and installed and executed in the spectator density monitoring device 100, or may be configured and implemented as an electronic circuit such as an ASIC (application-specific semiconductor). Alternatively, the processor 140 may receive only the processed results from a dedicated device that exclusively processes the task of recognizing crowds from images and counting their number, or from the surveillance camera itself capable of such tasks. Meanwhile, an image is composed of consecutive frames in time. One frame has one image. Additionally, a video may have one frame (or image). In other words, the video may be a single image.

In one embodiment, the people counting method may have various algorithmic approaches depending on the installation angle or distance of the camera. It can be broadly divided into methods using semantic information and methods using machine learning. Methods that use semantic information may include depth map, template matching, and histogram analysis.

In one embodiment, surveillance camera images are used, so they can have various variants depending on changes in shooting angle or shooting distance, and since pedestrian images are recorded, the characteristics of the crowd are learned manually to solve the people counting problem. Methods for creating a recognition model may be applied.

That is, the pre-learned people counting algorithm in one embodiment may be a machine learning method, and the machine learning method may include a method using manual features and a method using a convolutional neural network. For example, widely known hand-crafted features such as HOG (histogram of oriented gradient), color feature, and canny edge are used, and area, perimeter, and boundary edges are used. A method using linear features such as edges can also be applied. Additionally, methods such as using a CNN (convolutional neural network) structure may be applied.

Referring to FIG. 3, the processor 140 may calculate the average number of road crowds by dividing the number of people within the shooting coverage of each surveillance camera by the number of road cells within the shooting coverage of each surveillance camera. For example, the processor 140 may calculate the average road crowd number by dividing the crowd number TP[A] of surveillance camera A among the plurality of surveillance cameras by the number TR[A] of road cells. In one embodiment, the average number of road crowds of surveillance camera A among a plurality of surveillance cameras can be expressed as Average Road People[A], that is, ARP[A].

For example, if TR[A] is 7.5 and TP[A] is 30 people, ARP[A] may be 4. And for each road cell, the RP value reflecting the cell size may be as follows. RP[2][0]=2, RP[3][0]=4, RP[4][0]=4, RP[5][0]=4, RP[6][0]=2, RP[2][1]=2, RP[3][1]=4, RP[4][1]=4, RP[5][1]=4, RP[6][1]=0.

Meanwhile, referring to FIG. 4 , in one embodiment, when the shooting coverages of surveillance cameras overlap, the processor 140 may calculate the number of crowds for each road cell by correcting the average number of road crowds in the overlapping area. In one embodiment, the number of crowds per road cell may mean the number of crowds per road cell corrected from the average number of crowds on the road, and the number of crowds per road cell of surveillance camera A among the plurality of surveillance cameras is Road People[A], that is, RP[ It can be expressed as [A].

At this time, the processor 140 uses the average value of the average number of road crowds of the first surveillance camera (surveillance camera A) whose shooting coverage overlaps among the plurality of surveillance cameras and the average number of crowds on the road of the second surveillance camera (surveillance camera B). The number of crowds for each road cell in the overlapping area can be corrected. For example, in the case of an overlapping area, the corrected crowd number RP[A/B] for each road cell can be calculated by calculating (ARP[A]+ARP[B])/2.

For example, the processor 140 calculates TR[A], TP[A], ARP[A], TR[B], TP[B], and ARP[B] to calculate the RP in overlapping areas. For the RP cells (RP[4][0], RP[5][0], RP[6][0], RP[4][1], RP[5][1]), each obtained value The road cell value can be calculated as the average value of the values.

That is, in one embodiment, in order to correct accuracy, result values from a plurality of surveillance cameras can be corrected using an average value based on road cells.

Meanwhile, in one embodiment, there may be an area that is not included in the shooting coverage of a plurality of surveillance cameras in a specific area, and the number of crowds in the area can be expressed as RP[0]. In this case, when calculating the total number of crowds in the area, areas that are not included in the coverage can be excluded. However, it is not limited to this, and methods such as calculating the average value of surrounding surveillance cameras in the area may be applied.

Referring to FIG. 5, the processor 140 may derive a crowd density distribution map in a specific area based on the calculated crowd number RP for each road cell. And the processor 140 can monitor the crowd density situation in a specific area based on the crowd density distribution map derived above.

At this time, in one embodiment, the crowd density distribution may be displayed as a heat map on the map of a specific area based on map information of the specific area (see FIG. 5). Additionally, in one embodiment, for risk and disaster response, density information can be visualized in various ways, such as being mapped on a map, tabulated in numbers, or displayed in a graph, so that one can quickly and accurately prepare for or respond to high density situations. can do.

That is, in one embodiment, the processor 140 calculates the total number of road crowds by adding the number of crowds for each road cell of all road cells in a specific area, and if the total number of road crowds is greater than or equal to the set total number of crowds, the crowd in the specific area This can signal a high density situation. For example, the total number of road crowds can be expressed as TRP (Total Road People), where TRP is the sum of RP. In one embodiment, it may be determined that the higher the TRP, the higher the crowd density in the area and thus a situation requiring vigilance. Therefore, in one embodiment, when the RP value of a specific region is high, the RP value can be induced to be distributed to the entire region.

Additionally, in one embodiment, the processor 140 may notify that the crowd density in a specific road cell is high when the number of crowds per road cell in a specific road cell in a specific area is greater than or equal to the set number of crowds in each road cell.

Also, referring to FIG. 6, in one embodiment, the processor 140 may generate a road cell set by grouping road cells in a specific area according to set criteria. Then, the processor 140 calculates the number of road crowds for each road cell set by adding the number of crowds for each road cell within the road cell set, and if the number of road crowds for each road cell set is greater than the set crowd number, the crowd density of the road cell set is high. It can be announced that That is, in one embodiment, the concept of RG (Road Group), a set of road cells, can be introduced to manage the relevant area by dividing it by region.

That is, in one embodiment, the processor 140 may provide a Level of Detail (LOD) for situational awareness by utilizing the RG concept, which is a collection of road cells. Accordingly, the processor 140 can apply LOD technology to visualize 3D GIS data more efficiently and at a faster speed. LOD technology is a technology that gradually expresses the precision and resolution of terrain, images, and 3D objects depending on the distance from the user's viewpoint. As a result, it is possible to improve the speed of data visualization and minimize the user's hardware use by visualizing data with a smaller volume than the actual data. Meanwhile, in one embodiment, notification means sending various alarms to the crowd density monitoring device 100. This may include notifying a dangerous situation through methods (image, vibration, sound, etc.) or displaying a density distribution map on the screen, and may also include providing an alarm directly at the site.

Figure 7 is an example diagram for explaining a process of calculating crowd movement speed for each road cell according to an embodiment, and Figure 8 is an example diagram showing a crowd speed distribution chart as a heat map according to an embodiment.

Referring to FIG. 7, the processor 140 may use a previously learned people counting video analysis algorithm to calculate the movement speed of the crowd within the shooting coverage of each surveillance camera based on the shooting data of each surveillance camera. That is, in one embodiment, based on the people counting video analysis algorithm as described above, objects (people) in the video can be detected and the movement speed of the objects can be determined. In one embodiment, the crowd movement speed of surveillance camera A among a plurality of surveillance cameras can be expressed as People Speed[A], that is, PS[A].

Additionally, the processor 140 may calculate the average road crowd movement speed by dividing the crowd movement speed within the shooting coverage of each surveillance camera by the number of road cells within the shooting coverage of each surveillance camera. For example, the processor 140 may calculate the average road crowd movement speed by dividing the crowd movement speed PS[A] of surveillance camera A among the plurality of surveillance cameras by the number of road cells TR[A]. In one embodiment, the average road crowd movement speed of surveillance camera A among the plurality of cameras can be expressed as Average Road People Speed [A], that is, ARPS [A].

The processor 140 may calculate the crowd movement speed for each road cell in the same manner as the crowd number calculation process for each road cell described above. That is, the same method can be applied even when the shooting coverage of a plurality of surveillance cameras overlaps. Accordingly, the processor 140 can calculate the crowd movement speed for each road cell by correcting the average road movement speed in the overlapping area. In one embodiment, the crowd movement speed for each road cell may mean the crowd movement speed for each road cell corrected from the average road crowd movement speed, and the crowd movement speed for each road cell of the A surveillance camera among the plurality of surveillance cameras may be Road People Speed [A ], that is, it can be expressed as RPS[A].

At this time, the processor 140 determines the average road crowd movement speed of the first surveillance camera (surveillance camera A) whose shooting coverage overlaps among the plurality of surveillance cameras and the average road crowd movement speed of the second surveillance camera (surveillance camera B). The average value can be used to correct the crowd movement speed for each road cell in the overlapping area. For example, in the case of an overlapping area, the corrected crowd movement speed RPS[A/B] for each road cell can be calculated by calculating (ARPS[A]+ARPS[B])/2.

For example, for each road cell, the RPS value reflecting the cell size may be as follows. RPS[2][0]=PS[A], RPS[3][0]=PS[A], RPS[4][0]=PS[A], RPS[5][0]=PS[A] ], RPS[6][0]=PS[A], RPS[2][1]=PS[A], RPS[3][1]=PS[A], RPS[4][1]=PS [A], RPS[5][1]=PS[A].

Referring to FIG. 8, the processor 140 may derive a crowd speed distribution map in a specific area based on the calculated crowd movement speed for each road cell. And the processor 140 can monitor the crowd movement speed situation in a specific area based on the crowd speed distribution map derived above.

At this time, in one embodiment, the crowd speed distribution may be displayed as a heat map on the map of a specific area based on map information of the specific area (see FIG. 8).

Figure 9 is an example diagram showing the crowd density risk distribution map as a heat map according to an embodiment.

Meanwhile, in one embodiment, it is provided to monitor the crowd speed distribution, but it may be difficult to determine a crowding risk situation based only on the speed of the crowd. Accordingly, in one embodiment, in order to monitor a potential risk area due to large crowds and slow speed, the above-described crowd density distribution map based on the number of crowds and the crowd speed distribution map based on crowd movement speed are combined to create a crowd density risk distribution map in a specific area. can be derived and provided for monitoring.

At this time, the processor 140 may calculate the risk of crowding according to the number of crowds (RP) for each road cell and the crowd movement speed (RPS) for each road cell based on Equation 2 below. Here, k is a risk coefficient, which can be set in advance and can be changed or set by the user depending on the situation of a specific area or various situations.

In other words, as shown in Equation 2, the risk of crowd crowding in each road cell can be understood as the correlation between the number of crowds for each road cell, RP, and the crowd movement speed for each road cell, RPS. It can be seen that it tends to be directly proportional to RP and inversely proportional to RPS. there is.

In one embodiment, if the crowd density risk in a specific area is greater than or equal to a threshold, the processor 140 may notify that the crowd density in the specific area is a dangerous situation.

In addition, as described above, the processor 140 combines the crowd density distribution and the crowd speed distribution in a specific area to derive a crowd density risk distribution in a specific area, and based on the map information of the specific area, maps the specific area. The crowd density risk distribution can be displayed as a heat map on the screen (see Figure 9).

Meanwhile, in another embodiment, the processor 140 generates a direction map based on each shooting data captured by a plurality of surveillance cameras, and connects the generated plurality of direction maps to each other to move the crowd throughout a specific area. You can also construct a direction map layer that represents direction.

The processor 140 can combine the configured direction map layer with map information of a specific area to provide monitoring so that the flow of crowds in a specific area can be intuitively easily identified.

Here, the direction map can be set to one entrance (for example, the nearest subway station exit for entering a specific area, a designated entrance to a specific area, etc., which is judged to have a lot of crowds) and at least one exit. It consists of a group of movement lines with a certain point (e.g., exits of a subway station close to a specific area, designated exits in a specific area, etc.), and can be set from a specific point (entrance) to a number of other points (exits) in a specific area. ), you can intuitively visualize the direction of the crowd movement lines that diverge, and you can also intuitively visualize the density.

Meanwhile, in one embodiment, the monitoring interface screen divided into road cells such as distribution chart, risk level, heat map, etc. is briefly shown in two dimensions in Figures 3 to 9, but this is for convenience of explanation. When a monitoring interface divided into road cells, such as a heat map, is mapped and displayed on a 3D map, it can be provided in 3D.

Figure 10 is a flow chart to explain a crowd density monitoring method according to an embodiment.

As shown in FIG. 10, in step S100, the processor 140 collects map information of a specific area. In one embodiment, the processor 140 may collect map information from a map information database server that stores map information in a database.

In step S200, the processor 140 collects surveillance camera information including the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in a specific area, and shooting data from the surveillance cameras. In one embodiment, shooting data may be collected from a surveillance camera in real time, or shooting data may be collected according to a set schedule, but the present invention is not limited to this and may change depending on settings.

In step S300, the processor 140 divides the road area included in the shooting coverage of each surveillance camera and subdivides the divided road area into road cells (TR) of a set size. That is, in one embodiment, the processor 140 may determine the number of road cells in the road area included in the shooting coverage of each of the plurality of surveillance cameras, for example, in the shooting coverage of surveillance camera A among the plurality of surveillance cameras. The number of road cells included can be expressed as Total Road[A], that is, TR[A].

And in step S410, the processor 140 derives a crowd density distribution map in a specific area based on the number of crowds for each road cell. At this time, the processor 140 calculates the number of crowds (TP) within the shooting coverage of each surveillance camera using a previously learned people counting video analysis algorithm, calculates the average number of people on the road (ARP), and calculates the average number of people on the road in the overlapping area. By correcting the crowd number, the crowd number (RP) for each road cell can be calculated.

More specifically, the processor 140 may calculate the average number of road crowds by dividing the number of people within the shooting coverage of each surveillance camera by the number of road cells within the shooting coverage of each surveillance camera. When the shooting coverages of the surveillance cameras overlap, the processor 140 may calculate the number of crowds for each road cell by correcting the average number of road crowds in the overlapping area. At this time, the processor 140 uses the average value of the average number of road crowds of the first surveillance camera (surveillance camera A) whose shooting coverage overlaps among the plurality of surveillance cameras and the average number of crowds on the road of the second surveillance camera (surveillance camera B). The number of crowds for each road cell in the overlapping area can be corrected.

In step S420, the processor 140 monitors the crowd density situation in a specific area based on the crowd density distribution map. At this time, in one embodiment, the crowd density distribution may be displayed as a heat map on the map of a specific area based on map information of the specific area (see FIG. 5).

Meanwhile, after step S300, in step S510, the processor 140 derives a crowd speed distribution map in a specific area based on the crowd movement speed for each road cell. At this time, the processor 140 calculates the crowd movement speed (TPS) within the shooting coverage of each surveillance camera using a previously learned people counting video analysis algorithm, calculates the average road crowd movement speed (ARPS), and calculates the overlap area. By correcting the average road crowd movement speed, the crowd movement speed (RPS) for each road cell can be calculated.

More specifically, the processor 140 may calculate the average road crowd movement speed by dividing the crowd movement speed within the shooting coverage of each surveillance camera by the number of road cells within the shooting coverage of each surveillance camera. The processor 140 may calculate the crowd movement speed for each road cell in the same manner as the crowd number calculation process for each road cell described above. That is, the same method can be applied even when the shooting coverage of a plurality of surveillance cameras overlaps. Accordingly, the processor 140 can calculate the crowd movement speed for each road cell by correcting the average road movement speed in the overlapping area.

In step S520, the processor 140 monitors the crowd movement speed situation in a specific area based on the crowd speed distribution map. At this time, in one embodiment, the crowd speed distribution may be displayed as a heat map on the map of a specific area based on map information of the specific area (see FIG. 8).

And after steps S420 and S520, in step S600, the processor 140 combines the crowd density distribution map and the crowd speed distribution map in a specific area to derive a crowd density risk distribution map in a specific area. At this time, the processor 140 may calculate the risk of crowd crowding according to the number of crowds (RP) for each road cell and the crowd movement speed (RPS) for each road cell. The risk of crowd crowding in each road cell can be determined by the correlation between the crowd number RP for each road cell and the crowd movement speed RPS for each road cell, and may tend to be directly proportional to RP and inversely proportional to RPS.

Afterwards, in step S700, the processor 140 monitors the risk of crowding in a specific area.

In one embodiment, in order to monitor potential risk areas with large crowds and slow speeds, the crowd density distribution map based on the number of crowds described above and the crowd speed distribution map based on crowd movement speed are combined to create a crowd density risk distribution map in a specific area. It can be derived and provided for monitoring.

That is, as described above, the processor 140 combines the crowd density distribution map and the crowd speed distribution map of a specific area to derive a crowd density risk distribution map of a specific area, and based on the map information of the specific area, maps the specific area. The crowd density risk distribution can be displayed as a heat map on the screen (see Figure 9).

Additionally, in one embodiment, if the crowd density risk in a specific area is greater than or equal to a threshold, the processor 140 may notify that the crowd density in the specific area is a dangerous situation.

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the media includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM. , RAM, flash memory, etc., may include hardware devices specifically configured to store and execute program instructions.

Meanwhile, the computer program may be specially designed and constructed for the present invention, or may be known and usable by those skilled in the art of computer software. Examples of computer programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification (particularly in the claims) of the present invention, the use of the term “above” and similar referential terms may refer to both the singular and the plural. In addition, when a range is described in the present invention, it includes the invention to which individual values within the range are applied (unless there is a statement to the contrary), and each individual value constituting the range is described in the detailed description of the invention. It's the same.

Unless there is an explicit order or statement to the contrary regarding the steps constituting the method according to the invention, the steps may be performed in any suitable order. The present invention is not necessarily limited by the order of description of the above steps. The use of any examples or illustrative terms (e.g., etc.) in the present invention is merely to describe the present invention in detail, and unless limited by the claims, the scope of the present invention is limited by the examples or illustrative terms. It doesn't work. Additionally, those skilled in the art will appreciate that various modifications, combinations, and changes may be made according to design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

1: Crowd density monitoring system 100: Crowd density monitoring device
110: communication interface 120: user interface
130: memory 140: processor
200: user terminal 300: server
400: Network

Claims (24)

A map cell-based crowd density monitoring method for digital twin situation awareness that allows recognizing the crowd density situation in a specific area based on a map cell for digital twin situation awareness, in which at least part of each step is performed by a processor, comprising:
(A) collecting map information of a specific area;
(B) collecting surveillance camera information including the location, shooting coverage, and identification information of a plurality of surveillance cameras installed in the specific area, and shooting data of the surveillance cameras;
(C) dividing road areas included in the shooting coverage of each of the surveillance cameras and subdividing the divided road areas into road cells of a set size;
(D) Calculate the number of crowds within the shooting coverage of each of the surveillance cameras based on the shooting data of each of the surveillance cameras using a previously learned people counting video analysis algorithm, and calculate the number of crowds within the shooting coverage of each of the surveillance cameras. The average number of road crowds is calculated by dividing by the number of road cells within the shooting coverage of each surveillance camera, and when the shooting coverages of the surveillance cameras overlap, the average number of road crowds in the overlapping area is corrected to calculate the number of crowds for each road cell, Deriving a crowd density distribution map of the specific area based on the calculated number of crowds for each road cell, and monitoring the crowd density situation of the specific area based on the derived crowd density distribution map;
(E) Calculate the movement speed of the crowd within the shooting coverage of each surveillance camera based on the shooting data of each surveillance camera using the previously learned people counting video analysis algorithm, and calculate the movement speed of the crowd within the shooting coverage of each surveillance camera The average road crowd movement speed is calculated by dividing the movement speed by the number of road cells within the shooting coverage of each of the surveillance cameras, and when the shooting coverages of the surveillance cameras overlap, the average road crowd movement speed in the overlapping area is corrected for each road cell. Calculating crowd movement speed, deriving a crowd speed distribution map of the specific area based on the calculated crowd movement speed for each road cell, and monitoring the crowd movement speed situation of the specific area based on the derived crowd speed distribution chart. ;
(F) Combining the crowd density distribution and the crowd speed distribution in the specific area to derive the crowd density risk distribution in the specific area, and drawing the crowd density on the map of the specific area based on the map information of the specific area. It includes the step of displaying the risk distribution as a heat map,
In step (D),
Road cells in the specific area are grouped according to set criteria to create a road cell set, the number of crowds for each road cell within the road cell set is added to calculate the number of road crowds for each road cell set, and the road for each road cell set is calculated. If the number of crowds is greater than the set number of crowds, it further includes a step of notifying that the crowd density of the road cell set is high,
In step (F),
Based on Equation 1 below, a map cell-based map cell for digital twin situation awareness calculates the crowd density risk according to the number of crowds (RP, Road People) for each road cell and the crowd movement speed (RPS, Road People Speed) for each road cell. Crowd density monitoring method.
[Equation 1]

Here, k is the risk coefficient. According to paragraph 1,
The crowd density situation monitoring in step (D) is,
A map cell-based crowd density monitoring method for digital twin situation awareness that displays the crowd density distribution on a map of the specific area as a heat map, based on map information of the specific area. According to paragraph 1,
In step (C),
A map cell-based crowd density monitoring method for digital twin situation awareness that generates polygons on the captured data of the surveillance camera based on the captured data of the surveillance camera and distinguishes road objects among objects in the captured data. According to paragraph 1,
Calculating the number of crowds for each road cell by correcting the average number of road crowds in the overlapping area in step (D),
Digital twin situation recognition, which corrects the number of crowds for each road cell in the overlapping area with the average value of the average number of road crowds of the first surveillance camera whose shooting coverage overlaps among the plurality of surveillance cameras and the average number of street crowds of the second surveillance cameras. A map cell-based crowd density monitoring method for. According to paragraph 1,
In step (D),
Map cell-based crowd density monitoring for digital twin situation recognition, further comprising notifying that the crowd density of the specific road cell is high if the crowd number of each road cell of a specific road cell in the specific area is greater than the set number of crowds of each road cell. method. According to paragraph 1,
In step (D),
calculating the total number of road crowds by adding the number of crowds for each road cell of all road cells in the specific area; and
If the total number of road crowds is greater than the set total number of crowds, further comprising notifying that the crowd density in the specific area is high,
Map cell-based crowd density monitoring method for digital twin situational awareness. According to paragraph 1,
The crowd movement speed situation monitoring in step (E) is,
A map cell-based crowd density monitoring method for digital twin situation awareness that displays crowd speed distribution as a heat map on a map of the specific area, based on map information of the specific area. According to paragraph 1,
In step (F),
A map cell-based crowd density monitoring method for digital twin situation awareness, further comprising the step of notifying that the crowd density in the specific area is a dangerous situation if the crowd density risk in the specific area is greater than a threshold. delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete delete