KR20220061312A - Deep learning based building management system with rail robot device - Google Patents

Abstract

An objective of the present invention is to protect equipment of a building and minimize the entry of people into the space in which the equipment is positioned to protect a manager from the risk of accidents. A building monitoring system using a rail robot device (200) moving along a rail previously installed on a ceiling comprises: the rail (100) previously designed on the ceiling side based on the position of equipment to be monitored; the rail robot device (200) having a real image camera module (210) and a thermal image camera module (220), using the rail (100) as a track to run in accordance with a preset rule, acquiring a real image and a thermal image of a target area from the real image camera module (210) and the thermal image camera module (220) respectively, and having a communication module (230) transmitting the acquired real image and the acquired thermal image wirelessly or in a wired manner; and an integrated server (300) analyzing the real image and the thermal image transmitted from the communication module (230) of the rail robot device (200) by a preset artificial intelligence method. The integrated server (300) uses a deep neural network (DNN) among artificial intelligence methods to analyze the real image and the thermal image, and determines at least one among the smoke generation status, flooding determination status, fire occurrence status, flame occurrence status, and intruder occurrence status of a target area of a building through the real image and the thermal image.

Description

Deep learning based building management system with rail robot device

The present invention relates to a deep learning-based building management system equipped with a rail robot device.

In buildings such as houses, apartments, shopping malls, commercial buildings, factories, and large warehouses, various on-site equipment, such as various electrical equipment, gas equipment, air conditioning equipment, water supply equipment, communication equipment, security equipment, air conditioning equipment, and various switching equipment of them are installed In order to check the operating state of these equipment, it has been conventional for an administrator to directly inspect the field equipment with the naked eye or to install and monitor a stationary camera capable of photographing the field equipment.

However, there are uneconomical problems such as a very long observation time and an increase in the number of people required for management when the manager directly inspects the field equipment with the naked eye. In the case of using a fixed camera, the number of necessary cameras increases as the type or number of field equipment to be observed increases, which is also very uneconomical. There is a fatal problem that cannot immediately respond to an event occurring in a blind spot.

In addition, the observation area that can be photographed with a single fixed camera also has a limitation depending on the resolution or the distance to the field equipment, so more precise and more accurate observation has been difficult.

On the other hand, in recent years, as image analysis technology develops, various image analysis technologies are applied to manage event situations in buildings and to cope with them.

Intelligent video analysis technology is a technology that automatically detects abnormal behaviors and objects by analyzing video information, and sends an alert to the manager. there is. When a predefined event occurs using this image analysis, an alarm is created and notified to the manager. Instead of continuously monitoring all images, when an alarm occurs, the user sees the corresponding screen to judge and respond to the situation in real time. do.

As such, when intelligent image analysis technology is used, the management of the building is easy, and compared to judging with the human eye, it is possible to diagnose the event situation more accurately. By quickly recognizing and responding to the situation, damage can be minimized.

As a prior art for managing buildings using intelligent image analysis technology, Korean Patent Registration No. 10-2096175 is disclosed. The prior art is a 'ceiling rail-type IoT-based monitoring robot device', which takes real and thermal infrared camera images, transmits them to an artificial intelligence server, and uses a convolutional neural network (CNN) to A configuration for determining whether abnormality is disclosed. However, the prior art does not disclose an image analysis method optimized for a specific disaster situation (eg, flood, water leak, smoke generation, fire, spark generation), but uses a convolutional neural network (CNN), so many images and tasks There is a disadvantage in that it is not economical in terms of cost as time is involved.

Accordingly, recently, not a stationary camera, but a mobile camera is used to receive an image captured in real time, and the image is analyzed using deep learning. The need for an image analysis technology optimized for each occurrence is increasing.

Korean Patent Registration No. 10-2096175

The present invention has been devised to solve the problems of the prior art described above.

The present invention is a technology for protecting a manager from the risk of an accident by minimizing a person's entry into the space at the same time as protecting these facilities, since many electric devices or facilities are provided in the space where the facilities of the building are located. would like to propose

In addition, according to the present invention, when a supervisor directly monitors a space in which a facility of a building is located, there is a deviation depending on the supervisor, for example, it is very inaccurate for a person to check a small amount of water in relation to a water leak. We would like to propose a technology that can detect accidents from an early stage.

The present invention is a building monitoring system using a rail robot device 200 that moves along a rail installed in advance on a ceiling. Based on the location of a facility to be monitored, the rail 100 designed in advance on the ceiling side; As the rail robot device 200 having a real image camera module 210 and a thermal image camera module 220, it is operated according to a preset rule with the rail 100 as a track, and the real image camera module ( 210) and a communication module 230 for acquiring a real image and a thermal image of the target area from the thermal imaging camera module 220, respectively, and transmitting the acquired real image and thermal image wirelessly or by wire This is provided, the rail robot device 200; and an integrated server 300 for analyzing the real image and thermal image transmitted from the communication module 230 of the rail robot device 200 in a preset artificial intelligence method; Including, the integrated server 300 uses a deep neural network (DNN) of the artificial intelligence method to analyze a real image and a thermal image, but through the real image and thermal image , to provide a building monitoring system for determining at least one of smoke occurrence, flood judgment, fire occurrence, flame occurrence, and intruder occurrence in a target area of a building.

In addition, the present invention is a method of using the above-described building monitoring system, wherein the integrated server 300 determines whether smoke occurs in a target area through a real image frame constituting a real image image, a first determination module (310); Further comprising, (A1) a step of inputting a real image frame for learning to the first judgment and learning unit 312, wherein the real image frame for learning is divided into predetermined blocks, a non-smoking area and a smoke area After being input in this divided state, the step of learning the acting area by a preset method (S110); (A2) the step of transmitting the real image from the rail robot device 200 to the first determination module 310 (S120); (A3) A step of dividing, by the first determination module 310, a real image frame constituting the real image into predetermined blocks, using the acting area learned in the step (A1), dividing an area corresponding to a smoke area among the preset blocks into block groups, and then generating an edge portion for the block group (S130); (A4) when a plurality of block groups are formed in the real image frame in the step (A3), comparing edge portions generated for each block group (S140); (A5) when the edges of the first and second block groups overlap each other in step (A4), merging the first and second block groups to create a smoke area (S150); and (A6) repeating steps (A3) to (A5) for the first to Nth real image frames included in the real image (S160); It provides a method comprising:

In addition, the present invention is a method of using the above-described building monitoring system, wherein the integrated server 300 determines whether a flood occurs in a target area through a real image frame constituting a real image image, a second determination module (320); Further comprising, (B1) a step of inputting a real image frame for learning to the second judgment learning unit 322, wherein the real image frame for learning is divided into predetermined blocks, and a normal region and a flood region are After being input in a divided state, the step of learning the flood zone by a preset method (S210); (B2) the step of transmitting the real image image from the rail robot device 200 to the second determination module 320 (S220); (B3) dividing the real image frame constituting the real image into predetermined blocks by the second determination module 320, using the acting area learned in the step (B1), dividing an area corresponding to a flood area among the preset blocks into block groups, and then generating an edge portion for the block group (S230); (B4) In the step (B3), when a plurality of block groups are formed in the real image frame, the edges generated for each block group are compared, but each of the first block group and the second block group generating a flood zone by merging the first and second block groups when the edges overlap (S240); and (B5) calculating the area of the flood zone and the amount of water generated in the step (B4) by a preset method (S250); It provides a method comprising:

In addition, the present invention is a method of using the above-described building monitoring system, as a third determination module 330 for determining whether an event has occurred through a real image frame constituting a real image, wherein the occurrence of the event occurs in the target area. a third determination module 330; Further comprising, (C1) a step in which a real image image frame for learning is input to the third judgment learning unit 332, wherein the real image image frame for learning is a deep learning-based deep neural network model learning performed step (S310); (C2) by the step (C1), the step of forming a deep neural network model for the occurrence of the event (S320); (C3) a step of transmitting a real image from the rail robot device 200 to the third determination module 330, wherein the real image is input to the deep neural network model generated in the step (C2) ( S330); And (C4) the deep neural network model, for the real image input in the step (C3), by forming the nodes of the final output layer differently, determining whether a fire, flame, or intruder occurs (S340) ; In the event occurrence, the range of the region of interest is set to be small in the order of intruder occurrence, fire occurrence, and flame generation, and in step (C4), the smaller the range of the region of interest is set, the smaller the node value of the final output layer provides a larger set, method.

In addition, (C5) comparing the real image determined as the occurrence of the event and the thermal image acquired from the thermal imaging camera module 220, the region determined as the occurrence of the event and the deterioration of the real image determining whether the event has occurred by matching the temperature information for each area on the screening (S350); may further include.

In addition, the deep neural network model, based on the network structure of YOLO (You Only Look Once), can be learned through the real image frame for learning.

In addition, the rail robot device 200 further includes a position sensor 240 , and the position information obtained from the position sensor 240 is transmitted to the integrated server 300 through the communication module 230 . In the integrated server 300, drawing information of the target area, design information of the rail, and location information of facilities provided in the target area are input in advance, and the rail robot device 200 on the drawing information. a mapping module 340 for mapping movement information of ; Further comprising, the building monitoring system, the display unit 400 for visually displaying the movement information of the rail robot device 200 transmitted from the mapping module 340; It further includes, through the display unit 400, the movement information of the rail robot device 200, whether or not smoke is generated in the target area, whether a flood is judged, whether a fire occurs, whether a flame is generated, and whether an intruder occurs. At least one may be provided.

In addition, the rail robot device 200 is coupled to be movable on the rail 100 and receives power from the power supply unit 120 provided in the rail 100, the driving unit 250; and a photographing unit 260 provided with the real image camera module 210 and the thermal image camera module 220; A proximity sensor 252 is provided in front and rear of the driving unit 250, and the photographing unit 260 is configured to be rotatable 360 degrees with respect to the horizontal direction, and the photographing angle in the vertical direction. may be configured to be adjustable.

In addition, the communication module 230 of the rail robot device 200 is configured to be wirelessly connected to the personal terminal 500 at a predetermined distance within the target area, and the building monitoring system is the rail robot device 200 . A repeater unit 600 relaying the transmission of the real image and thermal image obtained from the integrated server 300 is provided, and the personal terminal 500 is provided with a wireless network By being connected to the network, it may be configured to transmit and receive information from the integrated server 300 .

In addition, the thermal imaging camera module 220 uses the temperature information for each area of the target area in a preset manner, and determines whether a fire or flame occurs in the target area, whether an abnormality occurs in the facility, and whether an intruder occurs. can do.

The present invention as described above has the following effects.

According to the present invention, many electric devices or equipment are provided in the space where the facilities of the building are located, and at the same time, it is possible to protect the facilities and, at the same time, by minimizing the entry of people into the space, it is possible to protect the manager from the risk of an accident. .

In addition, when the observer directly monitors the space where the facility of the building is located, there is a deviation depending on the observer. accidents can be detected.

1 is a schematic schematic diagram of a building monitoring system according to the present invention.
2 is a block diagram showing the overall configuration of a building monitoring system according to the present invention.
3 is a perspective view schematically illustrating a state in which a rail robot device is coupled to a rail in the building monitoring system according to the present invention.
4 (a) and (b) schematically show the driving part of the rail robot device.
5 is a block diagram showing the overall configuration of the rail robot device in the building monitoring system according to the present invention.
6 schematically illustrates a case in which an event situation occurs in the building monitoring system according to the present invention.
7 is a flowchart of a method for determining whether smoke is generated using the building monitoring system according to the present invention.
8 is a conceptual diagram schematically illustrating a process of determining whether smoke is generated using a deep neural network in FIG. 7 .
9 is a flowchart of a method for determining whether a flood occurs using the building monitoring system according to the present invention.
FIG. 10 is a conceptual diagram schematically illustrating a process of determining whether a flood occurs using a deep neural network in FIG. 9 .
11 is a flowchart of a method of determining whether an event situation (fire, flame, intruder) has occurred using the building monitoring system according to the present invention.
12 is a conceptual diagram schematically illustrating a process of determining whether an event situation occurs using the YOLO V3 Model, which is an example of an artificial intelligence method in FIG. 11 .

Hereinafter, a building monitoring system according to the present invention will be described with reference to the drawings. The present invention is a technology applicable to all regardless of the type of building, and is a technology for increasing the accuracy and speed of judgment through unmanned technology. For example, specify in advance that it can be applied to all places, such as underground power outlets, substation facilities, medical facilities, controlled areas, and railway stations, where it is difficult for a manager to directly reside and monitor. The term 'target area' used below means an area that can be monitored using a rail robot device to be described later. For example, when rails and rail robot devices are installed in an underground facility room, it means the entire space of the underground facility room. A facility to be monitored is located in a work space within the target area, and the target area and the monitoring target are conceptually separated.

Construction of the building monitoring system

1 and 2, a building monitoring system according to the present invention will be described.

The building monitoring system is largely composed of a rail 100 , a rail robot device 200 , an integrated server 300 , and a display unit 400 .

The rail 100 is designed based on the location of the facility to be monitored, and serves as a track on which the rail robot device 200 travels. The rail 100 is preferably installed on the ceiling side of the target area. However, even if it is not a ceiling, it may be configured to be coupled to an already installed pipe or the like, and any location is possible as long as the monitoring of the target area is possible. As an example, the rail 100 may be configured to have an inclination of up to 25 degrees, the radius of curvature may be 1000mm, and the maximum length of the rail 100 may be less than 1Km.

The rail robot device 200 is a device for monitoring a target area while driving the rail 100 unattended. It is operated according to a preset rule, and an automatic patrol mode in which sequential patrol monitoring is performed according to the setting of an administrator may be performed. In addition, it is possible to set the patrol section and control the movement speed. The rail robot device 200 includes a real image camera module 210 and a thermal image camera module 220 .

The real image camera module 210 provides a real image by photographing the target area in real time. For example, the resolution of the real image camera module 210 may be 1920 * 1080, 10 Hz. The thermal imaging camera module 220 may monitor fire and abnormal signs of equipment through temperature analysis, and at the same time monitor the heat of the human body. For example, a thermal image detector with a 17um pitch detects infrared rays having a wavelength of 7.5 to 13.5um, and a temperature can be visually expressed by a change in color according to the intensity of heat. The applied resolution may be 336*254, 10Hz.

In addition, the rail robot device 200 further includes a communication module 230 . The communication module 230 acquires a real image and a thermal image of the target area from the real image camera module 210 and the thermal image camera module 220, respectively, and transmits the obtained real image and thermal image wirelessly or It is configured to transmit over the wire. In this case, the communication module 230 may be configured as a Wi-Fi module, and may wirelessly transmit a real image image and a thermal image image to the personal terminal 500 of the manager or user. In addition, the communication module 230 is configured to transmit and receive image information with the repeater unit 600 to be described later, and is configured to transmit a real image and a thermal image to the integrated server 300 .

The rail robot device 200 may further include a voice communication module (not shown), and it is preferable that the voice communication module is configured to enable voice communication with the central control room and/or the manager. In the target area (eg, the basement), when the use of the personal terminal 500 of the operator is limited, this is a function that can be very usefully used.

With reference to FIG. 5, the rail robot device 200 will be described in more detail, and FIG. 5 shows the configurations of the rail robot device 200, and is classified according to each function.

The rail robot device 200 is divided into a driving unit 250 and a photographing unit 260 .

The driving unit 250 may be divided into a power supply unit 251 , a sensor group 252 , a motor group 253 , a main board module 254 , and a battery group 255 . In the main board module 254, a predetermined operation processing is performed, and it is configured to transmit image information to the repeater unit 600, and at the same time is configured to input a control command through the repeater unit 600, and according to the control command , the operation of the rail robot device 200 is controlled. In addition, the main board module 254 may further include a schedule storage unit (not shown). In the schedule storage unit, information on the operating time and operating method of the rail robot device 200 may be stored in advance, or may be modified by an administrator. Based on the information stored in the schedule storage unit, the rail robot device 200 is driven.

The rail robot device 200 is mounted on the rail 256 through a roller 256, and the roller 100 is a four-wheel drive method. can In addition, the motor group 253 may apply a high-performance BLDC motor and a driving driver, and as an example, a motor of 75W output and a 24V/3.6A driving driver may be applied. Through this, the rail robot device 200 can be run at a speed of 1 m/sec, but can be configured to be variable running.

In addition, a proximity sensor 252 may be provided at the front and rear of the driving unit 250 of the rail robot device 200 , respectively. Through the proximity sensor 252, the rail robot device 200 can automatically stop when reaching the end of the rail 100, and automatically stop when there is an object in the middle of the rail 100 in the process of driving can be configured.

A position sensor 240 is built in the rail robot device 200 . The position sensor includes a high-performance encoder and barcode, so that the real-time position is calculated through the encoder, the absolute position is calculated through the barcode, and then this position information is transmitted through the communication module 230 to the integrated server ( 300) is configured to transmit.

The integrated server 300 is configured to analyze the real image and thermal image transmitted from the communication module 230 of the rail robot device 200 in a preset artificial intelligence method.

Specifically, the integrated server 300 includes first to third determination modules 310 , 320 , 330 and a mapping module 340 . Here, the building monitoring system according to the present invention uses a deep neural network (DNN) and/or a convolutional neural network (CNN), which are artificial intelligence methods, and thus requires prior learning. Accordingly, the first to third determination modules 330 include first to third determination learning units 312 , 322 , and 332 corresponding to each other.

The integrated server 300 determines whether smoke occurs in the target area of the building, whether a flood occurs, whether a fire occurs, whether a flame occurs, and whether an intruder occurs through the real image and thermal image transmitted from the communication module 230 . can be done, which will be described later.

In the integrated server 300, the drawing information of the target area, the design information of the rail (the arrangement information of the rail), and the location information of the equipment provided in the target area are input in advance, and the movement of the rail robot device 200 on the drawing information and a mapping module 340 for mapping information. That is, through the mapping module 340, the movement of the rail robot device 200 is generated in real time. Since the movement information of the rail robot device 200 is accumulated and stored in the integrated server 300 , an administrator or a user can load the movement information of a desired time from the integrated server 300 . At this time, the building monitoring system further includes a display unit 400 that visually displays the movement information of the rail robot device 200 transmitted from the mapping module 340, and through the display unit 400, the manager to the user, movement information of the rail robot device 200, whether smoke occurs in the target area, whether a flood is determined, whether a fire occurs, whether a flame occurs, and whether an intruder occurs. The display unit 400 may be provided in a separate space, the central control room of the building, and may provide an alarm to the manager through the display unit 400 . However, the above information may be provided to the personal terminal 500 of an administrator or a user, and the personal terminal 500 may perform the function of the display unit 400 .

As described above, the display unit 400 diagrams the installed rail 100 , and may display the current position of the rail robot device 200 and the last state value of individual facilities. In addition, the temperature change over the last hour is provided as a graph, and the maximum temperature and average temperature on the display unit 400 are also recorded and displayed.

In addition, information on equipment to be monitored is further stored in the integrated server 300 . Specifically, the range of the normal temperature is individually set for each facility, and the specific method required for monitoring may be different depending on the characteristics of the facility. Information on height, etc. is set to be optimized for each facility.

The integrated server 300 may further include a report generation module (not shown). The report generation module provides a report to the manager according to a preset format based on the automatic operation record of the rail robot device 200 . In addition, the integrated server 300 may further include a self-diagnosis module (not shown). Through self-diagnosis of the rail robot device 200, for each location of the rail 100, the power supply and communication status are recorded and graphed and provided to the manager, and a specific location with poor power supply and communication status is automatically operated can be set to avoid. As an example, when the rail robot device 200 is in a communication disabled position during operation, it may be designed to quickly escape from the corresponding position and then wait for the next control command.

Meanwhile, for the convenience of managers and users (or workers), the building monitoring system according to the present invention may be configured to be wirelessly connected to their personal terminals 500 .

First, the communication module 230 built into the rail robot device 200 is configured to be wirelessly connected to the personal terminal 500 at a predetermined distance within the target area. As described above, the Wi-Fi method may be connected, but the connection method is not limited thereto, and a short-distance communication method may be applied. At this time, the short-distance communication method is Wi-Fi (Wireless-Fidelity), Bluetooth Low Energy (BLE), RFID (Radio Frequency Identification), infrared communication (IrDA, Infrared Data Association), UWB (Ultra-Wideband), and may be configured to use at least one of ZigBee technologies. In addition, wireless communication includes at least one of code division multiple access (CDMA), frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), and single carrier frequency division multiple access (SCFDMA) technologies. It may be configured to use one.

In addition, when the target area is located in the basement of a building or is not connected to an external wireless network such as an LTE network, the personal terminal 500 may be connected to the wireless network through the repeater unit 600 . That is, when the personal terminal 500 itself is difficult to connect to the external wireless network, it means that the wireless connection is activated via the repeater unit 600 .

Referring to FIG. 6 , a case in which an event situation occurs is schematically illustrated. Here, the event situation is a comprehensive concept that includes all of the occurrence of smoke, flood judgment, fire, spark, intruder, and flood in the target area.

6 shows an example according to the present invention. The first and second facilities are located in the target area, and in consideration of the location of the facilities, the rail 100 is designed on the ceiling side of the target area. The rail robot device 200 automatically performs patrol with the rail 100 as a track, and captures a real image and a thermal image and transmits it to the integrated server 300 . When it detects that a fire has occurred in the first facility, in order to more accurately determine the state of the fire, it approaches a preset adjacent location, and transmits information about it to the integrated server 300 and the personal terminal 500 in real time. . Of course, in this process, a siren or a separate alarm may be operated to inform the manager of the occurrence of an emergency. In addition, this situation may be provided through the display unit 400 located in the integrated control room.

How to use a building monitoring system

A process of determining whether smoke is generated using a building monitoring system will be described with reference to FIGS. 7 and 8 .

The integrated server 300 includes a first determination module 310 that determines whether smoke occurs in the target area through the real image frame constituting the real image, and the first determination module 310 determines the first determination The model learned by the learning unit 312 is applied.

The method includes steps S110 to S160.

Step S110 is a step in which a real image image frame for learning is input to the first judgment learning unit 312, wherein the real image image frame for learning is divided into predetermined blocks, and a non-smoking region and a smoke region are separated. After being input as a state, it is a step in which the acting area is learned by a preset method.

At this time, the first determination module 310 uses the image information for learning and testing. Given image information, the moving foreground (in this case, smoke) is first separated from the fixed background for each image frame. In this case, it is preferable to use the Gaussian mixing-based background/foreground segmentation algorithm implemented in OpenCV as cv::BackgroundSubtractorMOG2. The first determination module 310 considers the image of the image information, and by repeating each, separates the foreground and divides the foreground into contours by applying cv::findContoures. Each contour is divided into 40 x 40 image blocks. Blocks containing smoke and non-smoke materials are collected separately. The DNN was developed as shown in FIG. 8, and the DNN is trained with blocks.

Step S120 is a step in which a real image is transmitted from the rail robot device 200 to the first determination module 310 .

Step (S130) is a step of dividing the real image frame constituting the real image image into predetermined blocks by the first determination module 310. In the step (S110), the learned acting area is used. Thus, it is a step of dividing an area corresponding to a smoke area among the preset blocks into block groups, and then generating an edge part for the block group.

Step S140 is a step of comparing edge portions generated for each block group when a plurality of block groups are formed in the real image frame in step S130.

Step S150 is a step of generating a smoke region by merging the first and second block groups when the edges of the first and second block groups respectively overlap in the step S140.

Step S160 is a step of repeatedly performing steps S130 to S150 with respect to the first to Nth real image frames included in the real image.

A process of determining whether smoke is generated using a building monitoring system will be described with reference to FIGS. 9 and 10 .

The integrated server 300 further includes a second determination module 320 that determines whether a flood occurs in the target area through the real image frame constituting the real image, and the second determination module 320 includes: The model learned by the second judgment learning unit 322 is applied.

The method includes steps S210 to S250.

Step S210 is a step in which a real image image frame for learning is input to the second judgment learning unit 322, wherein the real image image frame for learning is divided into predetermined blocks, and a normal region and a flood region are separated. After being input as , it is a step in which the flood zone is learned by a preset method.

Step S220 is a step in which the real image image is transmitted from the rail robot device 200 to the second determination module 320 .

Step (S230) is a step of dividing the real image frame constituting the real image image into predetermined blocks by the second determination module 320. In the step (B1), the learned acting area is used. Thus, it is a step of dividing an area corresponding to a flood area among the preset blocks into block groups, and then generating an edge portion for the block group.

In step S240, when a plurality of block groups are formed in the real image frame in step S230, edge portions generated for each block group are compared, but the first block group and the second block group are compared. When each edge portion overlaps, the first and second block groups are merged to create a flood area.

Step S250 is a step S250 of calculating the area of the flood zone and the amount of water generated in step S240 by a preset method.

To elaborate a little more, the flood judgment is largely divided into two stages. In the first step, the bottom area is detected in the real image, and in the second step, water is detected only in the bottom area. Unlike the first determination module 310, the second determination module 320 operates on a still image, so a real image in the form of a video clip (meaning a moving picture) is not required. For learning and judgment, each real image frame is divided into blocks of a size of 32 x 32.

For the floor detection part, a large amount of floor images and images of parts except for the floor image are collected, and then the DNN is trained as part of the image set. As in the case of the first determination module 310, when a connected block is found and the number of elements in the group is very small, the group is excluded. This helps to eliminate the erroneous detection floor block. Finally, we apply a water-sensing classifier to each block group and calculate the approximate amount of water on the floor. The configuration of the second determination module 320 is the same as that of sensing the floor area, which is illustrated in FIG. 10 . However, the second determination module 320 is learned with a set of various image blocks, such as a block containing water and a block containing an empty floor. Filtering a group of blocks containing a small number of blocks can give an appropriate result.

The present invention may further include a fourth determination module (not shown) for determining the leak.

The fourth determination module operates on video clips similarly to the first determination module 310 . Initially, the moving part (here the water flow) is separated from the stationary part. However, unlike the case of the first determination module 310, it is not easy to detect moving water. The reason is that the smoke normally tries to spread, but at low velocities the water flow creates a very thin beam that is difficult to detect. To this, one-dimensional convolution is applied in the temporal domain of the image sequence, and then two-dimensional convolution is applied in the spatial domain of the image. Here, it is preferable that the one-dimensional convolution layer encodes a motion pattern of falling water, and the two-dimensional convolution layer encodes an image of the falling water.

The integrated server 300 is a third determination module 330 that determines whether an event has occurred through the real image frame constituting the real image, and the event occurrence determines whether a fire occurs in the target area, whether a flame occurs, and whether an intruder The generator further includes a third determination module 330 .

The method includes steps S310 to S350.

Step S310 is a step in which a real image image frame for learning is input to the third judgment learning unit 332, wherein the real image image frame for learning is a step in which learning of a deep learning-based deep neural network model is performed am.

Step S320 is a step in which a deep neural network model for whether the event has occurred is formed by the step S310.

Step (S330) is a step in which a real image is transmitted from the rail robot device 200 to the third determination module 330, and the real image is input to the deep neural network model generated in the step (S320). is a step

Step S340 is a step in which the deep neural network model determines whether a fire, flame, or intruder occurs by forming different nodes of the final output layer with respect to the real image input in step S330.

In operation S340 , as the range of the ROI is set smaller, the node value of the final output layer may be set larger.

Step S350 is a step of comparing the real image determined as the occurrence of the event with the thermal image acquired from the thermal imaging camera module 220, and the region determined as the occurrence of the event among the real image images and the deterioration It is a step of determining whether the event has occurred by matching the temperature information for each area on the screen.

Here, the deep neural network model is configured to learn through the real image frame for learning based on the network structure of YOLO (You Only Look Once).

Specifically, a single model is used for fire, flame and intruder detection. Two factors are considered while choosing a model. The first factor is to detect very fast, and the second factor is to detect small objects such as sparks and large objects such as intruders such as small amounts of fire. Through this, YOLO V3 was decided as a common model for spark and intruder detection. For learning, it is necessary to capture about 500 images containing various types of fires, flames and people. Labeling and bounding boxes are created for each image, about 400 images are randomly selected for training, and the remaining images are selected for validation. The above has been tested by the applicant, and it took about 3 days with a standard computer with a CUDA supported GPU.

The key to YOLO (You Only Look Once) is that it can recognize objects with one pass. The goal of this is to use the system on a moving camera and detect anomalies in real time. Modules are combined with a larger system that receives live streams from all cameras. Periodically, a large set of images is passed to the detection module, which is configured to check for anomalies, raise an alert if a suspicious event is found and, accordingly, the administrator to take the necessary action.

12 shows the architecture of the YOLO-V3 model. As shown in Fig. 12, products are extracted with three different scales. As an example, a larger object, such as an intruder, is detected by the output generated at node 82, and a smaller object, such as a flame or small amount of fire, is detected by the output generated at node 106.

Even with pre-trained weights for the convolutional layer, the training time of YOLO is very long. But once trained, detection works really fast. Experimentally, on a standard desktop computer with a Titan X GPU, YOLO can process about 45 frames per second. Detectors can be applied to images captured at 15 frames per second. When using a camera in which an object is moving within a nearby tunnel, high detection rates are required to detect anomalies, which results in a very short duration in the captured image, taken as a result. This system works best with ultra-high-resolution cameras, but testing on images captured with regular IP cameras also showed satisfactory results. And obviously, performance could be further improved if training and test samples were taken using cameras similar to these.

In the above, the present specification has been described with reference to the embodiments shown in the drawings so that those skilled in the art can easily understand and reproduce the present invention, but these are merely exemplary, and those skilled in the art can make various modifications and equivalent other modifications from the embodiments of the present invention. It will be appreciated that embodiments are possible. Therefore, the protection scope of the present invention should be defined by the claims.

100: rail
200: rail robot device
210: real image camera module
220: thermal imaging camera module
230: communication module
240: position sensor
250: driving unit
260: shooting department
300: integrated server
310: first determination module
320: second determination module
330: third determination module
340: mapping module
400: display unit
500: personal terminal
600: relay donation

Claims (10)

As a building monitoring system using a rail robot device 200 that moves along a rail installed in advance on the ceiling,
The rail 100 designed in advance on the ceiling side based on the location of the facility to be monitored;
As the rail robot device 200 having a real image camera module 210 and a thermal image camera module 220, it is operated according to a preset rule with the rail 100 as a track, and the real image camera module ( 210) and a communication module 230 for acquiring a real image and a thermal image of the target area from the thermal imaging camera module 220, respectively, and transmitting the acquired real image and thermal image wirelessly or by wire This is provided, the rail robot device 200; and
Including; and the integrated server 300 for analyzing the real image and thermal image transmitted from the communication module 230 of the rail robot device 200 in a preset artificial intelligence method;
The integrated server 300,
Analyze real image and thermal image using Deep Neural Network (DNN) among artificial intelligence methods,
Determining at least one of whether smoke occurs in a target area of a building, whether a flood is determined, whether a fire occurs, whether a flame occurs, and whether an intruder occurs through the real image and thermal image,
Building monitoring system.
A method of using the building monitoring system according to claim 1, comprising:
The integrated server 300,
a first determination module 310 that determines whether or not smoke is generated in the target area through the real image frame constituting the real image; further comprising,
(A1) A step of inputting a real image frame for learning to the first judgment learning unit 312, wherein the real image frame for learning is divided into predetermined blocks, and a non-smoking region and a smoke region are separated. After being input, the step of learning the acting area by a preset method (S110);
(A2) the step of transmitting the real image from the rail robot device 200 to the first determination module 310 (S120);
(A3) A step of dividing, by the first determination module 310, a real image frame constituting the real image into predetermined blocks, using the acting area learned in the step (A1), dividing an area corresponding to a smoke area among the preset blocks into block groups, and then generating an edge portion for the block group (S130);
(A4) when a plurality of block groups are formed in the real image frame in step (A3), comparing edge portions generated for each block group (S140);
(A5) when the edges of the first and second block groups overlap each other in step (A4), merging the first and second block groups to create a smoke area (S150); and
(A6) repeating steps (A3) to (A5) for the first to Nth real image frames included in the real image (S160); containing,
Way.
A method of using the building monitoring system according to claim 1, comprising:
The integrated server 300 includes, through a real image frame constituting a real image, a second determination module 320 that determines whether a flood occurs in a target area; further comprising,
(B1) A step of inputting a real image image frame for learning to the second judgment learning unit 322, wherein the real image image frame for learning is divided into predetermined blocks, and is input in a state in which a normal region and a flood region are separated After that, by a preset method, the step of learning the flood zone (S210);
(B2) the step of transmitting the real image image from the rail robot device 200 to the second determination module 320 (S220);
(B3) dividing the real image frame constituting the real image into predetermined blocks by the second determination module 320, using the acting area learned in the step (B1), dividing an area corresponding to a flood area among the preset blocks into block groups, and then generating an edge portion for the block group (S230);
(B4) In the step (B3), when a plurality of block groups are formed in the real image frame, the edges generated for each block group are compared, but each of the first block group and the second block group generating a flood zone by merging the first and second block groups when the edges overlap (S240); and
(B5) calculating the area of the flood zone and the amount of water generated in the step (B4) by a preset method (S250); containing,
Way.
A method of using the building monitoring system according to claim 1, comprising:
A third determination module 330 for determining whether an event has occurred through a real image frame constituting a real image, wherein the event occurrence is a third determination whether a fire occurs in a target area, whether a flame occurs, and an intruder module 330; further comprising,
(C1) a step in which a real image frame for learning is input to the third judgment learning unit 332, wherein the real image frame for learning is a step in which learning of a deep learning-based deep neural network model is performed (S310) );
(C2) by the step (C1), the step of forming a deep neural network model for the occurrence of the event (S320);
(C3) a step of transmitting a real image from the rail robot device 200 to the third determination module 330, wherein the real image is input to the deep neural network model generated in the step (C2) ( S330); and
(C4) The deep neural network model, for the real image input in the step (C3), by forming the nodes of the final output layer differently, determining whether a fire, a spark, and an intruder occurs (S340); includes,
In the event occurrence, the range of the region of interest is set small in the order of intruder occurrence, fire occurrence and flame generation,
The step (C4) is,
The smaller the range of the region of interest is set, the larger the node value of the final output layer is set.
Way.
5. The method according to claim 4,
(C5) comparing the real image determined as the occurrence of the event and the thermal image acquired from the thermal imaging camera module 220, in the region determined as the occurrence of the event in the real image and in the thermal image determining whether the event has occurred by matching the temperature information for each area (S350); further comprising,
Way.
5. The method according to claim 4,
The deep neural network model, based on the network structure of YOLO (You Only Look Once), learns through the real image frame for learning,
Way.
The method according to claim 1,
The rail robot device 200,
Further comprising a position sensor 240, the position information obtained from the position sensor 240 is transmitted to the integrated server 300 through the communication module 230,
The integrated server 300,
Drawing information of the target area, design information of the rail, and location information of facilities provided in the target area are input in advance, and a mapping module 340 for mapping movement information of the rail robot device 200 on the drawing information. ); further comprising,
The building monitoring system,
a display unit 400 for visually displaying movement information of the rail robot device 200 transmitted from the mapping module 340; further comprising,
Through the display unit 400,
Providing at least one of movement information of the rail robot device 200, whether smoke is generated in the target area, whether a flood is determined, whether a fire occurs, whether a flame is generated, and whether an intruder occurs,
Building monitoring system.
The method according to claim 1,
The rail robot device 200,
a driving unit 250 coupled to be movable on the rail 100 and receiving power from a power supply unit 120 provided on the rail 100; and
a photographing unit 260 provided with the real image camera module 210 and the thermal image camera module 220; is composed of
A proximity sensor 252 is provided at the front and rear of the driving unit 250,
The photographing unit 260 is configured to be rotatable by 360 degrees with respect to the horizontal direction, and configured to be able to adjust the photographing angle in the vertical direction,
Building monitoring system.
The method according to claim 1,
The communication module 230 of the rail robot device 200,
It is configured to be wirelessly connected to the personal terminal 500 at a predetermined distance within the target area,
The building monitoring system,
A repeater unit 600 for relaying the transmission to the integrated server 300 of the real image and thermal image acquired from the rail robot device 200 is provided,
The personal terminal 500,
Through the repeater unit 600, by being connected to a wireless network network, configured to transmit and receive information from the integrated server 300,
Building monitoring system.
The method according to claim 1,
The thermal imaging camera module 220,
Using the temperature information for each area of the target area in a preset manner, determining whether a fire occurred, whether a flame occurred, whether an abnormality occurred in the facility, and whether an intruder occurred in the target area,
Building monitoring system.

Images