KR20230080522A - Fire monitoring system using artificial intelligence and method thereof - Google Patents

Abstract

The present invention relates to a control method of a fire monitoring system using artificial intelligence, wherein a real image and a thermal image are received from a robot, the real image is analyzed to detect at least one of smoke and flame, and the smoke and A detection area corresponding to at least one of the flames is extracted from a real image, at least one object included in the detection area is individually identified, and a first area corresponding to the detection area is analyzed in the thermal image. The main point is to extract the maximum temperature of the detection area and determine the fire by comparing the individual permissible temperature of the identified object with the maximum temperature.

Description

Fire monitoring system using artificial intelligence and its control method {Fire monitoring system using artificial intelligence and method thereof}

The present disclosure relates to a fire monitoring system and a control method thereof, and specifically, to a fire monitoring system using artificial intelligence and a control method thereof.

In recent years, the frequency of large-scale fires has steadily increased, and public interest in fire and safety has increased. In the past case, a large number of people died and many people were injured due to a large fire. In particular, keeping the golden time in the event of a fire is applied as a very important factor, but recent fire cases in which human casualties have occurred have in common that sprinklers do not work or initial reporting and response are delayed, leading to large-scale fires. In addition, due to the malfunction of the heat detector in summer and the manager's lack of awareness about the fire receiver, there are frequent cases in which the fire receiver is turned off.

It shows the number of fire dispatches and the number of misplaced dispatches, and the number of occurrences tends to increase every year. In addition, about 60% of the total number of mobilization cases may not be mobilized to the site where firefighting power is needed due to the dispatch of the wrong person. Therefore, maintenance and inspection must be performed accurately to reduce malfunction of the detector and ensure that the fire detector functions normally in the event of a fire. there is.

In the case of the prior art, a method of detecting smoke or flame by detecting a change in a previous image or a subsequent image using a still camera or detecting a fire by detecting a temperature change in a fixed area was used. However, in this case, there is a problem in that the accuracy of fire detection is lowered in an environment where the camera itself moves.

An object of the present invention is to accurately identify a monitoring target and identify fire occurrence by using a safety temperature database for each target in order to reduce malfunction cases of the existing temperature change detection method for fire detection and color filter use method for flame detection. It is to provide a fire monitoring system and its control method.

An object of the present invention is a fire monitoring system that detects the occurrence of a fire by using a thermal image that measures the temperature of an object area where a flame is detected in order to reduce the inability to accurately detect a flame caused by a camera mounted on a moving robot, and the same It is to provide a control method.

An object of the present invention is to provide a fire monitoring system capable of accurately detecting the location of a fire by dividing an object into a plurality of regions based on a real image and a thermal image, and checking the temperature of each region, and a control method therefor. there is.

Other objects and advantages of the present disclosure can be understood by the following description, and will be more clearly understood by the embodiments of the present disclosure. Furthermore, it will be readily apparent that the objects and advantages of the present disclosure may be realized by means of the instrumentalities and combinations thereof set forth in the claims.

According to an embodiment of the present disclosure, a control method of a fire monitoring system using artificial intelligence includes receiving a real image and a thermal image from a mobile robot; detecting at least one of smoke and flame by analyzing the real image; extracting a detection area corresponding to at least one of the smoke and the flame from a real image; individually identifying at least one object included in the sensing area; extracting a maximum temperature of the sensing area by analyzing a first area corresponding to the sensing area in the thermal image; and determining a fire by comparing the individual permissible temperature of the identified object with the maximum temperature.

According to an embodiment of the present invention, in order to reduce malfunction cases of the existing temperature change detection method for fire detection and color filter use method for flame detection, a monitoring target is accurately identified and a safety temperature database for each target is used. User convenience can be improved because fire occurrence can be identified more accurately.

According to another embodiment of the present invention, in order to reduce the inability to accurately detect flames caused by a camera mounted on a moving robot, a thermal image measuring the temperature of an object area where a flame is detected is used to more accurately determine the occurrence of a fire. Therefore, user convenience can be improved.

According to another embodiment of the present invention, an object can be divided into a plurality of regions based on a real image and a thermal image, and the location of a fire can be more accurately detected by checking the temperature for each region, thereby effectively suppressing a fire. can

Effects obtainable in the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below. will be.

1 is a diagram schematically illustrating a system for monitoring and removing an anomaly of an object according to an embodiment of the present invention.
2 is a block diagram of a functional configuration of a driving robot.
3 is a perspective view of a traveling robot.
4 is a perspective view illustrating a traveling robot traveling along a track.
5 is a block diagram of a functional configuration of a control server.
6 is a flowchart illustrating a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention.
7 to 9 are diagrams illustrating operations implemented according to a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention.
10 is a flowchart illustrating a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention.
11 to 15 are diagrams illustrating operations implemented according to a method for monitoring and removing an abnormal object according to another embodiment of the present invention.
16 is a diagram showing the configuration of a fire monitoring system according to an embodiment of the present invention.
17 is a diagram illustrating a control method of a fire monitoring system according to an embodiment of the present invention.
18 is a diagram showing the structure of a CNN according to an embodiment of the present invention.
19 is a diagram illustrating flame, smoke, and haze used for fire detection according to an embodiment of the present invention.
20 is a diagram illustrating a process of detecting flames and smoke using a CNN according to an embodiment of the present invention.
21 is a diagram illustrating a process of identifying an object using YOLOv2, according to an embodiment of the present invention.
22 is a diagram illustrating an example of identifying an object using YOLOv2 according to an embodiment of the present invention.
23 is a diagram illustrating an example of identifying an object using YOLOv2 according to an embodiment of the present invention.
24 is a diagram illustrating an example of subdividing an area of an identified object according to an embodiment of the present invention.
25 is a diagram illustrating an example of detecting a maximum temperature for each region in a thermal image according to an embodiment of the present invention.
26 is a diagram illustrating an example of executing a fire alarm by comparing a maximum temperature for each object with a safety temperature stored in a DB according to an embodiment of the present invention.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present disclosure. However, the present disclosure may be implemented in many different forms and is not limited to the embodiments described herein.

In describing the embodiments of the present disclosure, if it is determined that a detailed description of a known configuration or function may obscure the gist of the present disclosure, a detailed description thereof will be omitted. And, in the drawings, parts irrelevant to the description of the present disclosure are omitted, and similar reference numerals are attached to similar parts.

In the present disclosure, components that are distinguished from each other are intended to clearly explain each characteristic, and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated to form a single hardware or software unit, or a single component may be distributed to form a plurality of hardware or software units. Accordingly, even such integrated or distributed embodiments are included in the scope of the present disclosure, even if not mentioned separately.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment composed of a subset of components described in one embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to the components described in various embodiments are also included in the scope of the present disclosure.

In the present disclosure, components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, an embodiment composed of a subset of components described in one embodiment is also included in the scope of the present disclosure. In addition, embodiments including other components in addition to the components described in various embodiments are also included in the scope of the present disclosure.

In the present disclosure, terms such as first and second are used only for the purpose of distinguishing one element from another, and do not limit the order or importance of elements unless otherwise specified. Accordingly, within the scope of the present disclosure, a first component in one embodiment may be referred to as a second component in another embodiment, and similarly, a second component in one embodiment may be referred to as a first component in another embodiment. can also be called

When a component of the present disclosure is referred to as “connected” or “connected” to another component, it may be directly connected or connected to the other component, but there may be other components in the middle. It should be understood that it may be On the other hand, when a component is referred to as “directly connected” or “directly connected” to another component, it should be understood that no other component exists in the middle.

In addition, in the present disclosure, the description of each drawing may be applied to different drawings unless one drawing showing an embodiment of the present disclosure corresponds to another drawing and an alternative embodiment.

Hereinafter, the present disclosure will be described in more detail with reference to the drawings.

Referring to FIGS. 1 to 5 , a system for monitoring and removing an abnormal object according to an embodiment of the present invention will be described.

1 is a diagram schematically illustrating a system for monitoring and removing an anomaly of an object according to an embodiment of the present invention. 2 is a block diagram of the functional configuration of the driving robot, and FIG. 3 is a perspective view of the driving robot.

The system (10; hereinafter referred to as 'the system') for monitoring and removing an abnormal object is monitoring whether or not a dangerous situation occurs in at least one object disposed in the space, and if a dangerous situation is detected, the driving robot 300 is placed in danger. It is a system that removes or solves the abnormal situation by moving to the abnormal occurrence point 40 where the situation is caused. Hereinafter, the dangerous situation is explained by exemplifying a fire, but it may be an intrusion or infiltration of a person or a suspicious object in space security, and may be various abnormal (or abnormal) situations that may occur in the space without being limited thereto.

The system 10 includes sensing devices 100 and 200, a driving robot 300 and a track device 400, and a control server.

The sensing device is installed at a location that is considerably spaced apart from or close to the plurality of objects 30 in the space 20 and regularly monitors whether an abnormal situation such as a fire occurs. The sensing devices 100 and 200 may be a fixed camera 100 disposed at a spaced-apart position or a proximity sensing device 200 having an optical fiber temperature sensor for detecting the temperature along the surface of the object 30 so as to be installed at a close position. can

The fixed camera 100 may be located in a place where images of a plurality of objects 30 are acquired in a space 20 where a plurality of objects 30 such as a plurality of various equipment exist. The fixed camera 100 acquires an image of the object 30 and transmits it to the control server 500, and the image of the object 30 determines whether the object is abnormal, such as a fire, and also determines the fire occurrence point 40. It may be the sensing data used to make the decision. A detailed description of the calculation of the location of the fire occurrence point 40 and the nearest trajectory using the fixed camera 100 will be described later. The fixed camera 100 can be rotated up and down in a certain angular range around a horizontal axis parallel to the ground, as well as rotated left and right in a certain angular range around a vertical axis perpendicular to the ground. Accordingly, the fixed camera 100 (100) is not only possible to take a wide range of images, but also, when a plurality of them are provided, the shooting ranges overlap with each other, so that it is possible to more quickly check whether the object 30 is abnormal. In addition, the fixed camera 100 contributes to more accurately determining abnormality by acquiring an enlarged image of the object 30 having a suspicious abnormality by having a zoom in-out function. The fixed camera 100 may be a general CCTV or PTZ camera.

The proximity sensing device 200 may be located in a facility in which it is difficult to obtain an image of the fixed camera 100 within the space 20 . For example, in the case of electrical system facilities such as pipes and/or electrical wires stacked in multiple layers in the vertical direction of the ground, the fixed camera 100 for electrical wires disposed inside facilities or bundled electrical wires in the middle layer It is practically difficult to detect a fire in its early stages. In order to quickly determine whether the object has an abnormal high temperature or a fire, the optical fiber temperature sensor of the proximity sensing device 200 is mounted along the surface of the object 30, which is difficult to detect by the fixed camera 100, and It can detect the temperature of (30) and whether or not there is a fire. The optical fiber temperature sensor may be made of a material capable of optical communication with the control server 500 through an optical fiber, and the optical fiber temperature sensor may be set by the control server 500 to have location data specified at predetermined intervals along the extension direction. can Accordingly, when the optical fiber temperature sensor is disconnected due to a fire of the object 30, optical communication is performed only at the disconnected abnormal occurrence point 40 and the control server 500, so the control server 500 is an optical communication possible point. An abnormal occurrence point 40 of the object 30 may be calculated based on location data corresponding to . When the anomaly occurrence point 40 is calculated, the closest trajectory position based thereon may be calculated by the control server 500.

The driving robot 300 travels along the first track 402 of the track device and periodically monitors whether or not a fire occurs in the plurality of objects 30 together with the detection device. In addition, when an abnormality of the object 30 is detected by the sensing device, the driving robot 300 is controlled to be provided as the minimum distance position from the abnormality occurrence point 40 of the object 30 and checks the abnormality of the object Remove. In addition, the driving robot 300 may detect an abnormality of the object 30 before the sensing devices 100 and 200 and transmit the abnormality to the control server 500. ), posture control of the moving or traveling robot 300 may be executed so as to provide a position at the minimum distance from the anomaly occurrence point 40 while moving to a position closest to the trajectory. As another example, when the driving robot 300 detects an abnormality first, the driving robot 300 transmits the detection data it has acquired from the object 30 to the control server 500, and the control server 500 When it is determined that the object 30 has an abnormality, the closest trajectory position may be calculated in conjunction with the sensing devices 100 and 200 .

Referring to FIGS. 2 and 3 , the driving robot 300 includes a driving unit 302, a sensing unit 304, an anomaly removing unit 306, a pan-tilt adjusting unit 308, and a transceiving unit 310. , a memory 312, an alarm unit 314, a power supply unit 316, and a processor 304.

The drive unit 302 attaches a motor, a drive wheel mechanically connected thereto, and the like to the main body 328 so that the driving robot 300 can travel along the first and second tracks 402 and 408 set by the track device. In addition, a height-adjustable lift capable of moving the driving robot 300 in a vertical direction may be provided in order to improve aiming accuracy for removing abnormality of the driving robot 300 . The height-adjustable lift may have a pantograph-type joint structure between the main body 328 of the traveling robot 300 and the support or a plurality of hydraulic cylinders combined in a vertical direction.

The sensing unit 304 is coupled to the main body 328 through the support 330 and the support assembly 332, and includes a visual camera 320 and an object 30 that acquire an image related to the real image of the object 30. In addition to the thermal imaging camera 318 that detects heat, a sensor module 322 including various sensors such as a gas sensor, a temperature sensor, a sound recognition sensor, an ultraviolet sensor, a position sensor, and a current/voltage meter may be included. there is.

The sensor 304 is activated in normal monitoring to check whether the object 30 is abnormal, and when an abnormal occurrence, for example, a fire in the object 30 is confirmed, the trajectory the driving robot 300 is closest to. When the location is reached, the fire point 40 is photographed to contribute to calculating the minimum distance required to control the driving robot 300.

The sensing unit 304 can be controlled to be rotatable in the left and right directions and up and down directions by the pan-tilt adjusting unit 308. Accordingly, in the process of moving from the nearest track position to the fire occurrence point 40, a predetermined The fire occurrence point 40 of the object 30 is located in the image acquired by the detection unit 304 such that it is perpendicular between the detection unit 304 pan-tilted at a fixed angle of and the center point of the anomaly occurrence point 40. A case may be determined as the minimum distance location. The minimum distance location may be determined based on the maximum temperature of the fire detected by the thermal imaging camera 318 and the central point of the fire obtained by the visual imaging camera 320 . Determination of the minimum distance position and detailed description of position/posture control will be described later.

The abnormal removal unit 306 is coupled to the main body 328 through the support body 330 and the support assembly 332, and may be mounted together with the detection unit 304 on the support assembly 332, for example. The anomaly removal unit 306 includes a fire extinguisher 324 for injecting a fire extinguishing agent, for example, an aerosol fire extinguishing agent, and an injection inducer 326 for injecting the fire extinguishing agent to the fire occurrence point 40 when the object 30 is abnormal. can include Although not shown in the drawing, the injection inducer 326 may be any location that smoothly guides the fire extinguishing agent injection around the fire extinguisher 324, and may be implemented in a fan shape. When the fire extinguishing agent is an aerosol type, if it is mainly sprayed to the surrounding fire rather than the center of the fire, an upward air current may be generated due to the aerosol fire extinguishing agent, and the fire may not be extinguished smoothly. Therefore, the injection inducer 326 may guide the direction of the fire extinguishing agent spray so that the fire extinguishing agent is sprayed toward the center of the fire derived based on the detected data of the visible camera 320, the thermal image camera 318, the gas sensor, and the temperature sensor. can

The anomaly removal unit 306 is operated when the mobile robot 300 is moved and/or posture controlled so as to be provided at the minimum position distance, and the fire occurrence point 40 considering the fire center point and the like by the pan-tilt adjustment unit 308 After being aimed at, the fire extinguishing agent can be sprayed.

As shown in FIG. 3, the pan tilt adjusting unit 308 is provided between the main body 328 and the support 330 and controls the anomaly removing unit 306 to rotate in the left and right directions along a vertical axis. The fan driving adjusting unit 334 and A tilt driving adjustment unit 336 may be provided between the support body 330 and the support assembly 332 to control the anomaly removal unit 306 to rotate in the vertical direction along a horizontal axis.

The sensing unit 304 and the anomaly removing unit 306 are mounted on the integral support assembly 332 and can rotate in a predetermined direction synchronized by the control of the pan/tilt adjusting unit 308 . According to this, since the minimum distance location and fire center point determined based on the detection data of the sensing unit 304 are also shared from the point of view of the anomaly removal unit 306, the anomaly removal unit 306 moves to the minimum distance location without additional adjustment. Movement/position can be controlled, and the fire can be quickly extinguished by finely aiming in the current state. In another example, the sensing unit 304 and the anomaly removing unit 306 may be controlled by different pan/tilt adjusting units. In this case, the anomaly removing unit 306 controls the position and It can be controlled based on posture.

The transmission/reception unit 310 transmits various controls of the driving robot 300, detection data of the detection unit 304 related to whether the object 30 is abnormal, data related to removal of anomaly, such as an image of a fire suppression process, etc. to a control server ( 500) and a module that transmits and receives. The transmitting/receiving unit 310 may perform wired/wireless communication with the control server 500, and may be configured with WI-FI, Bluetooth, and ZigBee communication modules as short-range communication in the case of wireless communication.

The memory 312 controls the driving robot 300 and configures the data received from the control server 500, for example, location data such as the coordinates of the entire space 20 including the locations of a plurality of objects, and the system 10. The location data of the devices 100 to 400, the location data of the anomaly occurrence point 40, and the trajectory location data closest to the anomaly occurrence point 40 may be stored.

When an abnormality occurs, the alarm unit 314 not only visually and/or audibly propagates the abnormal occurrence situation within the space 20 , but also may propagate the current situation in various ways according to the abnormal occurrence situation.

The power supply unit 316 may include a battery to enable independent operation and function of the driving robot 300, and even if power supply is cut off in the space 20 due to an abnormality such as fire, the driving robot 300 itself Position calculation, movement, posture control, spraying, etc. to approach the fire occurrence point 40 can be independently performed. The driving robot 300 may move to a charging station and charge the power supply unit 316 when the battery remaining amount is less than a reference value.

The processor 304 controls the functions of the above-described members of the driving robot 300 and processes data transmitted and received with the control server 500.

4 is a perspective view illustrating a traveling robot traveling along a track.

4 and 11 in relation to the track device, the track device includes a first track 402 and a second track 408, and the traveling robot 300 has first and second tracks 402 , 408) may further include switching branching units 410 provided to be mutually changeable. Here, the second track 408 may be installed to extend at least one of a different height and a different direction from the first track 402 . In the embodiment of FIG. 11 , only the first and second trajectories 402 and 408 are illustrated, but a plurality of trajectories can be connected to the first trajectory 402 according to the location of an object in the space 20 and the construction environment. A switching branch 410 may be installed for each site.

The first and second tracks 402 and 408 may be connected to and fixed to a support frame 406 located at a predetermined location in the space 20 through periodically installed rods 404 . According to the settings of the control server 500, the first track 402 is designated so that it can be used for both the normal monitoring of the driving robot 300 and the driving route when an abnormality occurs, and the second track 408 is used when an abnormality occurs. In the case where it is calculated as the closest orbit position to the corresponding point, it can be designated to be used as an auxiliary driving route when an abnormality occurs. In another example, according to the setting of the control server 500, all of the first and second tracks 402 and 408 may be used as normal monitoring and driving routes in case of an abnormality. If the second trajectory 408 extending in a height and/or direction different from the first trajectory 402 is used as a route for normal monitoring, accuracy of normal monitoring and rapid confirmation of occurrence of abnormality can be exerted.

When an abnormal occurrence is determined by the control server 500 and the abnormal occurrence point 40 is calculated and the nearest orbit position is determined as a predetermined position of the second orbit, the mobile robot 300 moves to the first orbit 402 In a driving situation, it can be transferred from the first trajectory to the nearest trajectory on the second trajectory through the switching branch 410 . For smooth trajectory change, the first and second trajectories 402 and 408 adjacent to the transition branch 410 may be disposed at the same height as the transition branch 410 . Accordingly, when the driving robot 300 enters the switching branch 410 from the first track 402, the switching branch 410 rotates so that the driving robot 300 can move toward the second track. It contributes to being connected with the second orbit 408.

Meanwhile, when the driving robot 300 implements WI-FI communication with the control server 500 through the transceiver 310, the first track 402 uses a CRA (Cable type WI-FI Radial Antenna) cable. can include

CRA may mean a cable type WI-FI radial antenna. For example, the CRA may be a cable equipped with an antenna capable of communicating by radiating radio waves to the outside. For example, a slot (or home or area) may exist in the CRA to enable communication. In this case, the slot may serve as an antenna, and a different frequency may be selected for radio waves emitted for communication based on the length or slope of the slot. For example, the frequency at which the driving robot 300 operates may be a frequency for a local area network, and the slot may be designed to use the above-described frequency band. That is, a slot for the CRA may be set in a cable in consideration of frequency and may serve as an antenna. More specifically, the CRA may refer to a transmission line used to transmit radio frequency power by connecting a transmitter and a transmission antenna or between a reception antenna and a receiver. In this case, the cable may transmit the radio signal obtained from the CRA to a server or other device through a transmission line. Alternatively, the cable may transmit a signal generated from a server or other device through a transmission line and emit it as a radio wave at the CRA. As an example, the CRA may be a leaky coaxial cable, but is not limited thereto.

The traveling robot 300 may move along the provided cables. For example, the weight (or load) of the driving robot 300 may be designed to be movable on a cable. That is, the traveling robot 300 can move through a general track without installing a separate track. On the other hand, the driving robot 300 can move the cable equipped as a general track through the drive unit 302 . At this time, the driving robot 300 may perform wireless communication through the CRA in a normal track.

As another example, the driving robot 300 may travel along the CRA and the support line (or steel line) in consideration of equipment or devices included in the driving robot 300 . At this time, the traveling robot 300 may be driven by being connected to a certain steel wire like a cable car. As another example, when a device with a large load is mounted on the driving robot 300, the driving robot 300 may travel using a profile (eg aluminum or steel) to which the CRA is mounted (or inserted) as a track. . That is, the driving robot 300 can travel through a track that can withstand a large load as a CRA insertion type track.

In another example, all of the first and second tracks 402 and 408 may be installed to include all of the CRA cables.

In this way, the CRA cable included in the track has flame resistance, and thus, communication failure due to a fire can be suppressed. Also, in the past, the driving robot 300 could transmit data by contacting wires and communication lines. However, the driving robot 300 may travel in an environment where human access is difficult, such as an underground facility or high temperature and high pressure. At this time, in the case where the driving robot 300 transmits data through a contact-type track as in the past, there is a limit to implementing the driving robot 300 considering the above environment. Considering this point, the driving robot 300 may be designed to transmit information obtained from the sensor 304 to a server (or CMS) through wireless communication through CRA.

5 is a block diagram of a functional configuration of a control server.

The control server 500 controls the switching and changing of the sensing devices 100 and 200, the driving robot 300 and the trajectory, as well as the detection data received from the sensing devices 100 and 200 and the driving robot 300. Based on this, it is possible to process whether or not an abnormality has occurred, various location data related to the abnormal occurrence point 40, and the like, and control the behavior of the driving robot 300 during the anomaly removal process.

The control server 500 may include a transceiver 502 , a memory 504 , an analysis unit 506 , a location calculation unit 508 and a processor 510 in detail.

The transceiver 502 may include a module for exchanging data with the sensing devices 100 and 200 and the driving robot 300 in a defined communication method, and when the driving robot 300 communicates with WI-FI , The transmission/reception unit 502 may communicate with the driving robot 300 using a CRA cable built into the first track 402 .

The memory 504 may store various data as well as programs for operating the control server 500 . The stored data may store location data, images obtained by the sensing device and the driving robot 300 during normal or fire suppression processes, and sensing data such as temperature, presence of gas, sound, and the like.

The location data includes location data such as coordinates of the entire space 20 including location information of a plurality of objects 30, location data of the devices 100 to 400 constituting the system 10, and anomaly occurrence point 40. ), orbital position data closest to the anomaly occurrence point 40, and the like. The coordinates of the entire space are based on at least one of the three-dimensional space information obtained by the architectural drawing and the three-dimensional scanner (for example, Lidar scanner) based on construction of the space 20 in which the object 30 is placed. can be built

The analysis unit 506 uses the model learned through the machine learning method to detect devices 100 and 200, a real image camera 320, a thermal image camera 318, a gas sensor, a temperature sensor, a sound recognition sensor, and a location sensor. It can be determined by analyzing whether or not a fire has occurred in the object 30 detected from the lamp. In the case of the fixed camera 100, the analysis unit 506 analyzes the RGB pixels constituting the image of the object 30 to analyze the color and movement type, and the thermal imaging camera 318, the gas sensor, It is possible to analyze and determine whether a fire has occurred by comprehensively analyzing the temperature distribution of the object 30 obtained from the sound recognition sensor, detection information of a specific gas generated in case of fire, and detection information of sound caused in case of fire.

According to the machine learning function, the analysis unit 506 may determine whether the object is abnormal through a training data set. In this case, the training data set may be past input information stored in the memory 504 and a model generated by machine learning. That is, the analysis unit 506 may learn past input information through machine learning to create a model for determining abnormalities, and based on this, compare real-time mouth detection data received from the detection devices 100 and 200 to abnormality can be judged.

Through this, the analysis unit 506 can determine whether the object 30 is abnormal, and can provide information on whether or not the object 30 is abnormal to the driving robot 300 . The driving robot 300 may determine whether additional input information for the object 30 is required based on information received from the control server 500 . When the driving robot 300 determines that additional input information is needed, the driving robot 300 may acquire more specific information about the target and transmit it to the control server 500 . The control server 500 may finally determine whether or not the object 30 is abnormal based on more specific information about the object 30 , and store and provide information thereon. In addition, the model may be updated by re-learning additional information on the above-described abnormalities.

The location calculation unit 508 may determine the nearest trajectory location based on the anomaly occurrence point 40 of the object where the anomaly is detected by the sensing devices 100 and 200 .

In the case of the fixed camera 100, the location calculation unit 508 includes the location data of the entire space 20, the angle of view of the object 30 on fire within the image obtained from the fixed camera 100, and the fixed camera ( Location data (coordinates) of the fire point 40 may be calculated based on the location data of 100).

In the case of a proximity sensing device 200 equipped with a fiber optic temperature sensor. The location calculation unit 508 may calculate location data where an optical fiber temperature sensor installed on the object 30 is disconnected due to a fire as the fire occurrence point 40 of the object 30 .

In addition, the location calculation unit 508 searches for the closest trajectory among the first and second trajectories 402 and 408 from the above-described fire occurrence point 40, and the degeneration point among the consecutive coordinates on the searched trajectory. and the nearest orbit position can be calculated. In addition, the position calculation unit 508 determines that the sensing angle at which the sensing unit 304 of the driving robot 300 faces the center point of the fire when the driving robot 300 reaches the closest trajectory position is the pan-tilt adjusting unit. When the fire center point is located at a point corresponding to the angle in the image of the detector 304 so that the angle is 90 degrees or 270 degrees while being varied by 308, the minimum distance position may be determined. According to the position of the minimum distance determined by the control server 500, the driving robot 300 may be moved to a location and controlled at an attitude. In this embodiment, the minimum distance position is exemplified as being calculated by the control server 500, but the driving robot 300 due to power cutoff due to power outages in the space 20 or poor communication with the control server 500. The minimum position distance is autonomously determined through the above-described process, and thus position movement and attitude control can be performed. Even if it is not the above-described abnormal situation, the driving robot 300 may be set to calculate and determine the minimum position distance without intervention of the control server 500 according to the user's settings.

The processor 510 controls the functions of the above-described members of the control server 500, processes data transmitted and received from the sensing devices 100 and 200, the driving robot 300, and the track device 400, and also processes each device. control the operation of

Hereinafter, a method for monitoring and removing an abnormal object according to another embodiment of the present invention will be described with reference to FIGS. 1 to 9 . For convenience of understanding, an abnormality of an object is exemplarily described as a fire.

6 is a flowchart illustrating a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention. 7 to 9 are diagrams illustrating operations implemented according to a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention.

First, in normal times, the detection devices 100 and 200 monitor whether a fire occurs in a plurality of objects 30 in the space 20 (S105). The sensing device may be a proximity sensing device 200 including a fixed camera 100 and an optical fiber temperature sensor. In normal object detection, not only the above-described detection device, but also the driving robot 300 detects whether or not a fire has occurred for each object 30 along the first track 402 under the direction of the control server 500 or autonomously. This detection is to obtain various sensing data obtained from the visible image camera 320, the thermal image camera 318, a temperature sensor, a gas sensor, an ultraviolet sensor, a sound recognition sensor, and the like. As another example, when a plurality of objects are vertically arranged as in FIG. 11 , the sensing range of the driving robot 300 is an object adjacent to the second trajectory 408 disposed below the first trajectory 402 ( 30), the control server 500 may utilize all of the first and second trajectories 402 and 408 as normal monitoring routes. Accordingly, the monitoring route of the driving robot 300 can be set in various ways, and can be organized as an efficient route as much as possible. For example, the monitoring path includes a first trajectory 402, a transition branch 410, a rightward movement in the second trajectory 408, a leftward movement at the end point of the second trajectory 408, a transition branch 410, It may be in the order of movement to the first trajectory 402 that is not driven. An embodiment of the operation of the driving robot 300 in relation to the first and second trajectories 402 and 408 disposed at different heights described in other examples is described in FIGS. 10 to 15, and in this embodiment, the object 30 on fire ), only the first track 402 composed of a single layer is installed around the driving robot 300 to extinguish a fire on the first track 402.

Next, when an abnormality, that is, a fire occurs in the object 30 at a specific location, the sensing devices 100 and 200 transmit sensing data related to the object 30 on fire to the control server 500, The server 500 analyzes the sensing data and determines the occurrence of a fire as an object abnormality (S110).

The analysis unit 506 of the control server 500 uses the machine learning method to detect devices 100 and 200, a real image camera 320, a thermal image camera 318, a gas sensor, a temperature sensor, It may be determined by analyzing whether a fire occurs in the object 30 detected by a sound recognition sensor, a location sensor, or the like. In the case of the fixed camera 100, the analysis unit 506 analyzes the color and movement type based on the RGB pixels constituting the image of the object 30, and the thermal imaging camera 318, the gas sensor, It is possible to analyze and determine whether a fire has occurred by comprehensively analyzing the temperature distribution of the object 30 obtained from the sound recognition sensor, detection information of a specific gas generated in case of fire, and detection information of sound caused in case of fire. In the case of the proximity detection device 200, it may be determined whether a specific object 30 has a fire in consideration of the temperature and disconnection of the optical fiber temperature sensor.

Next, the location calculation unit 508 of the control server 500 calculates the abnormal occurrence point 40 from the sensing data obtained from the sensing devices 100 and 200 (S115).

In the case of the fixed camera 100, the location calculation unit 508 includes the location data of the entire space 20, the angle of view of the object 30 on fire within the image obtained from the fixed camera 100, and the fixed camera ( Location data (coordinates) of the fire point 40 may be calculated based on the location data of 100). Referring to FIG. 7 , each object 30 disposed in the space 20 using the angle of view with the object 30 on fire recognized in the image of the left fixed camera 100 and the location data of the fixed camera 100 ) The three-dimensional coordinates that are in contact with the location data of the fire can be calculated as the location data of the fire object 30, that is, the fire point 40.

In the case of a proximity sensing device 200 equipped with a fiber optic temperature sensor. The location calculation unit 508 may calculate location data where an optical fiber temperature sensor installed on the object 30 is disconnected due to a fire as the fire occurrence point 40 of the object 30 .

Next, as shown in FIG. 7, the position calculation unit 508 searches for a trajectory closest to the fire occurrence point 40, and among the consecutive coordinates on the searched first trajectory 402, the trajectory closest to the fire occurrence point The position (arrow in FIG. 7) is calculated (S120).

Subsequently, the control server 500 instructs the driving robot 300 to travel on the first track 402, and the driving robot 300 is transferred to the nearest track position along the first track 402 ( S125).

Next, the moving robot 300 is moved to a position and posture controlled so as to provide a minimum distance from the abnormal occurrence point 40, that is, the fire occurrence point 40 (S130).

Referring to FIGS. 8 and 9 , the calculation of the minimum distance position is described. When the position calculation unit 508 reaches the nearest trajectory position (corresponding to point 1 in FIGS. 8 and 9 ), the position calculation unit 508 , while the detection angle at which the detection unit 304 of the driving robot 300 faces the central point of the fire is varied by the pan-tilt adjustment unit 308, the angle is 90 degrees or 270 degrees (corresponding to two points in FIGS. 8 and 9). ), a case where the central point of the fire is located at a point corresponding to the angle in the image 340 of the detector 304 (for example, the central region of the image as shown in FIG. 9) may be determined as the minimum distance location. The center point of the fire may be estimated by, for example, the lower center point of the flame acquired from the image of the visual image camera 320 and the highest temperature point found from the temperature distribution image obtained from the thermal image camera 318. According to the position of the minimum distance determined by the control server 500, the driving robot 300 may be moved to a location and controlled at an attitude. Positional movement may be, for example, movement of the driving robot 300 from one point to two points on the first trajectory 402 of FIG. 8 . Posture control is the posture of the sensing unit 304, such as adjusting the sensing direction and angle of the sensing unit 304 by the pan/tilt adjusting unit 308 according to autonomous control of the driving robot 300 or instructions from the control server 500. may be to control In this embodiment, the minimum distance location is exemplified as being calculated by the control server 500, but the driving robot 300 due to power cut off due to power outages in the space 20 or poor communication with the control server 500. The minimum position distance can be autonomously determined through the above-described process, and thus position movement and attitude control can be performed. Even if it is not the above-described abnormal situation, the driving robot 300 may be set to calculate and determine the minimum position distance without intervention of the control server 500 according to the user's setting.

Next, the driving robot 300 aims and sprays the fire extinguisher 324 and the spray inducer 326 to eliminate abnormality, that is, to suppress the fire, and notifies the surroundings of the fire occurrence situation by the alarm unit 314, Sensing data including images related to the fire suppression situation is transmitted to the control server 500 (S135).

After the driving robot 300 is provided at the minimum distance position, it acquires and analyzes the fire-related detection data of the sensing unit 304 in real time, and controls the pan-tilt driving unit 302 according to the analysis result to control the central point of the fire just before the fire extinguishing agent is sprayed. Aim the fire extinguisher 324 toward and spray the fire extinguishing agent. At the same time, the blowing direction of the fan constituting the injection inducer 326 may also be controlled so that the fire extinguishing agent is injected toward the center of the fire.

In addition, for efficient fire suppression, the driving robot 300 drives a height-adjustable lift according to the center point of the fire analyzed by the image obtained from the sensor 304 so that the fire extinguisher 324 is more accurately aimed at the center point of the fire. The spray height of (324) can be adjusted.

According to the present embodiment, after moving the driving robot to a track position closest to the abnormal occurrence point 40 determined by detection through the sensing device, the driving robot detects the anomaly occurrence point 40 and By being precisely controlled to provide the minimum distance position of , it is possible to efficiently remove the abnormal situation of danger occurring to the object.

Hereinafter, a method for monitoring and removing an abnormal object according to another embodiment of the present invention will be described with reference to FIGS. 1 to 5 and 10 to 15 .

This embodiment relates to a case in which multi-layered tracks are installed according to the vertically arranged objects 30 different from those of FIG. 6 . Accordingly, this embodiment is different from the embodiment of FIG. 6 in steps S120 and S125, but is substantially the same as steps S105 to S115 and S130 and S135 in FIG. Descriptions of the steps of the same foregoing will be omitted.

10 is a flowchart illustrating a method for monitoring and removing an anomaly of an object according to another embodiment of the present invention. 11 to 15 are diagrams illustrating operations implemented according to a method for monitoring and removing an abnormal object according to another embodiment of the present invention. For reference, FIG. 11 is a schematic diagram showing the track device when the system 10 is viewed from the side, and FIGS. 12 to 15 show the track device when the system 10 is viewed from the top to the bottom. it is a schematic

First, when a fire is detected and the control server 500 determines the fire occurrence point 40 of the object 30 on fire, the location calculation unit 508 calculates a fire among the first and second tracks 402 and 408. The trajectory closest to the point of origin 40 is searched, and the location of the trajectory closest to the point of occurrence of metaplasia among consecutive coordinates on the searched trajectory is calculated (S205).

In the search for the nearest trajectory, the most suitable trajectory is selected in consideration of the monitoring range of the sensing unit 304 and the possible injection range of the anomaly removal unit 306 for each detailed position on each trajectory. For convenience of understanding, in this embodiment, in a situation where the driving robot 300 normally travels on the first track 402, the fire point 40 is adjacent to a specific position on the second track 408, A description will be given assuming that a specific position is determined as the nearest orbital position. According to this, as shown in FIG. 11, since the fire occurrence point 40 is closest to the specific position of the second track 408, the specific position is determined as the closest track position.

Next, when the specific location of the second trajectory 408 is determined as the closest trajectory location, as shown in FIGS. 12 and 13, the driving robot 300 moves from the first trajectory 402 to the second trajectory 408. It is moved to the conversion branching unit 410 for moving and converting (S210).

Subsequently, the switching branch 410 is rotated under the control of the control server 500, etc., so that the traveling path of the traveling robot 300 is connected to the second trajectory 408, and the traveling robot 300 is connected to the second trajectory ( 408) is changed and moved (S215).

Continuing, the driving robot 300, like step S130, moves the location from the closest trajectory position of the second trajectory 408 so as to provide the minimum distance from the abnormal occurrence point 40, that is, the fire occurrence point 40 And posture control may be made (S220).

In this embodiment, it has been mainly described that the second trajectory is determined in the step of calculating the closest trajectory position before determining the minimum distance location. In another embodiment, the control server 500 analyzes sensing data such as the sensing devices 100 and 200 to determine a specific position on the first track 402 as the nearest track position, but the driving robot 300 autonomously Alternatively, by exchanging data with the control server 500 or receiving instructions, the minimum distance location may be determined as the position most adjacent to the abnormal occurrence point in the second trajectory 408 instead of the first trajectory 402 and transferred. In this case, the driving robot 300 is the second most adjacent to the detection range of the detection unit 304, the injection range of the anomaly removal unit 306, and the anomaly occurrence point 40 at a specific position of the first trajectory 402. The minimum distance position can be determined by considering all the corresponding positions of the trajectory 408 and the like. According to this, taking the urban situation of FIG. 11 as an example, the driving robot 300 reversely moves from the position of the first track 402 adjacent to the fire occurrence point 40 to the switching branch 410, and the switching branch ( 410) is controlled to be provided to a specific location of the second trajectory 408 determined as the closest trajectory location and/or the minimum distance location.

According to the present embodiment, the driving robot 300 is transported to a track closest to a fire occurrence point among multi-layered tracks to suppress an anomaly such as a fire, thereby maximizing fire suppression efficiency. Furthermore, since there is no need to excessively extend the fire extinguisher 324 downward by the height-adjustable lift, shaking of the fire extinguisher 324 or the like due to strong injection pressure is suppressed, and suppression efficiency can be further improved.

Exemplary methods of this disclosure are presented as a series of operations for clarity of explanation, but this is not intended to limit the order in which steps are performed, and each step may be performed concurrently or in a different order, if desired. In order to implement the method according to the present disclosure, other steps may be included in addition to the exemplified steps, other steps may be included except for some steps, or additional other steps may be included except for some steps.

In addition, the fire monitoring system and its control method will be described in detail with reference to FIGS. 16 to 26 below. However, those skilled in the art may supplement and interpret FIGS. 16 to 26 or modify the embodiment with reference to FIGS. 1 to 15 described previously.

16 is a diagram showing the configuration of a fire monitoring system according to an embodiment of the present invention.

Referring to FIG. 16 , the fire monitoring system 1000 includes a mobile robot 100, a control server 200, and an external device 300.

The mobile robot 100 includes a real image camera 110, a thermal image camera 120, a communication unit 130, a controller 140, and a memory 150.

The real image camera 110 means a camera that captures real images.

The thermal imaging camera 120 refers to a camera that captures thermal images.

The mobile robot 100 captures a real image with the real image camera 110 and captures the thermal image with the thermal image camera.

The communication unit 130 transmits and receives data with an external device and the control server 200 .

The controller 140 controls the real image camera 110 , the thermal image camera 120 , the communication unit 150 , and the memory 150 .

The memory 150 stores the captured real image and thermal image.

Here, the capture cycle of the real image is different from the capture cycle of the thermal image. A capture cycle of a real image may be faster than a capture cycle of a thermal image.

This is because the thermal image does not change significantly over time, but the real image image undergoes a greater change due to flames and smoke when a fire occurs.

The mobile robot 100 includes a mobile robot that moves along a rail.

The control server 200 includes a communication unit 210, a memory 220, a control unit 230, and a display 240.

The communication unit 210 transmits and receives data with the mobile robot 100 and the external device 300 .

The memory 220 may be a database and stores safety temperature information for each object.

The control unit 230 performs overall control logic of the fire monitoring system.

The controller 230 receives a real image and a thermal image from the robot, analyzes the real image, detects at least one of smoke and flame, and converts a sensing area corresponding to at least one of the smoke and flame into a real image. extracting from an image, individually identifying at least one object included in the sensing area, analyzing a first area corresponding to the sensing area in the thermal image, extracting a maximum temperature of the sensing area, and identifying the A fire is determined by comparing the individual permissible temperature of the object with the maximum temperature.

The control unit 230 determines a fire when the maximum temperature is higher than the individual permissible temperature of the object.

The control unit 230 determines that it is not a fire when the maximum temperature is lower than the individual permissible temperature of the object.

The controller 230 detects at least one of smoke and flame by using a CNN in the real image.

Specifically, the controller 230 stores the real image as a real image in chronological order, converts the stored real image into a black and white image, image-processes the converted black and white image, and performs image processing on the black and white image. Contours are extracted from, learning is performed using the extracted image contours as an input to the CNN, and at least one of smoke and flame is detected based on the learning result.

The controller 230 individually identifies the at least one object using YOLOv2.

Specifically, the control unit 230 extracts the real image screen into individual images, divides the extracted individual images into grids of a predetermined size, and the number of bounding boxes designated in a predefined shape around the center of the divided grids. predicts , determines whether the bounding box includes the at least one object using reliability calculation, and identifies the at least one object according to the determination result.

The control unit 230 determines a fire by comparing the allowable temperature for each object with the maximum temperature by referring to a database including permissible temperature information for each object.

When there is no permissible temperature information for each object in the database, the controller 230 requests permissible temperature information for each object from the external device 300, and when there is permissible temperature information for each object matched to the external device 300, it retrieves the permissible temperature information for each object. can receive

The display 240 displays a graphic image according to a control command from the controller 230 .

The external device 300 may be a device that stores allowable temperature information for each object and safety temperature information for each object.

17 is a diagram illustrating a control method of a fire monitoring system according to an embodiment of the present invention. The present invention is carried out by means of a fire monitoring system (1000).

First, a real image and a thermal image are received from the robot (S1710).

At least one of smoke and flame is detected by analyzing the real image (S1720).

A detection area corresponding to at least one of smoke and flame is extracted from the real image (S1730).

At least one object included in the sensing area is individually identified (S1740).

The highest temperature of the sensing area is extracted by analyzing the first area corresponding to the sensing area in the thermal image (S1750).

A fire is determined by comparing the individual permissible temperature of the identified object with the maximum temperature (S1760).

18 is a diagram showing the structure of a CNN according to an embodiment of the present invention.

CNN is a deep learning technology, and it is a model that mimics the structure of the human optic nerve processing vision information. CNN classifies information by extending low-dimensional information to high-dimensional information in order to easily classify information from an input image. Basically, a CNN is composed of a convolutional layer, a pooling layer or sub-sampling layer, and a classification layer as shown in FIG. 18.

CNN automatically extracts the features of the recognition target from the learning image by alternately performing the convolutional layer and the pooling layer. The convolution layer can extract features while considering the characteristics of the image by performing a convolution operation while moving the learned kernel one by one on the input data coming from the previous layer. In addition, the integration layer serves to reduce the number of dimensions of the feature map generated through the convolution layer, and there are methods such as max pooling and average pooling. In order to acquire stimulating information as a feature, maximum pooling is often used. As such, after the convolution layer and the pooling layer are alternately performed several times, features having main pattern information are automatically extracted, and based on this, the classification layer finally outputs the image recognition result.

19 is a diagram illustrating flame, smoke, and haze used for fire detection according to an embodiment of the present invention. 19 includes FIGS. 19(a), 19(b) and 19(c).

19(a) means a flame used for fire detection.

19(b) means smoke used for fire detection.

19(c) means the haze used for fire detection.

According to the present invention, the controller 230 detects at least one of flame, smoke, and haze by analyzing a real image using a CNN.

20 is a diagram illustrating a process of detecting flames and smoke using a CNN according to an embodiment of the present invention. The present invention is carried out by means of a fire monitoring system (1000).

As shown in FIG. 20, the real image is stored as a real image in chronological order (S2010).

Specifically, the controller 230 stores the real picture images in the buffer in chronological order.

The stored real image is converted into a black and white image (S2020).

Specifically, the controller 230 subtracts the previous image from the current image to convert the real image into a black and white image.

Image processing is performed on the converted black and white image (S2030).

An outline is extracted from the image-processed black-and-white image (S2040).

Specifically, the controller 230 compares the colors of flame and smoke based on the outline.

The controller 230 detects the extension of the outline.

The control unit 230 executes at least one of contour merging and separation estimation.

Learning is performed using the extracted image contour as an input to the CNN (S2050).

Based on the learning result, at least one of smoke and flame is sensed (S2060).

According to the present invention, smoke and flame can be more accurately detected using CNN learning.

21 is a diagram illustrating a process of identifying an object using YOLOv2, according to an embodiment of the present invention.

A real picture screen is extracted as an individual image (S2110).

The extracted individual images are divided into grids of a predetermined size (S2120). Here, the predetermined size means an arbitrary size that is not fixed.

According to another embodiment, the extracted individual images may be divided into grids of the same size.

The number of bounding boxes designated in a predefined shape centered on the center of the divided grid is predicted (S2130).

It is determined whether the bounding box includes the at least one object (S2140). For example, whether a bounding box includes at least one object is determined based on reliability.

For example, if the reliability is 80% or higher, it is determined that the bounding box contains at least one object.

According to the determination result, the at least one object is identified (S2150).

22 is a diagram illustrating an example of identifying an object using YOLOv2 according to an embodiment of the present invention.

As shown in FIG. 22 , a screen 2200 includes a first object 2210 , a second object 2220 and a third object 2230 .

YOLO (You Only Look Once) is a model used in the field of object detection, which divides a prediction image into n x n grid cells and predicts one object in each cell. And, it refers to an algorithm that determines the location and size of an object through a preset number of boundary claps. YOLOv2 is an upgraded algorithm of YOLO.

The first region 10 includes a first object 2210 .

The second area 20 includes a second object 2220 .

The third area 30 includes a third object 2230 .

Here, the first object 2210 may be a person, the second object 2220 may be a bench, and the third object 2230 may be a cell phone 2230.

23 is a diagram illustrating an example of identifying an object using YOLOv2 according to an embodiment of the present invention.

Referring to FIG. 23 , a real image 2300 includes a first object 2310 and a second object 2320 .

For example, the first object 2310 may be the first machine as a result of identification using YOLOv2. In this case, the reliability may be 99.9%.

The second object 2320 may be a second machine as a result of identification using YOLOv2. In this case, the reliability may be 99.9%.

Also, the controller 230 may analyze the real image 2300 to detect the flame 5 .

24 is a diagram illustrating an example of subdividing an area of an identified object according to an embodiment of the present invention.

Referring to FIG. 24 , a real image 2300 includes a first object 2310 and a second object 2320 .

The first object 2310 may be divided into a first area 10 , a second area 20 , and a third area 30 .

The second object 2320 may be divided into a first area 40 , a second area 50 , and a third area 60 .

Also, the controller 230 may analyze the real image 2300 to detect the flame 5 of the first area 40 of the second object 2320 .

According to the present invention, individual objects included in the real image 2300 can be divided into a first area, a second area, and a third area, so that when a fire occurs, it is possible to more accurately detect which area the fire has occurred in.

25 is a diagram illustrating an example of detecting a maximum temperature for each region in a thermal image according to an embodiment of the present invention.

Referring to FIG. 25 , a thermal image 2500 includes a first object 2310 and a second object 2320 .

A thermal imaging camera is a camera that detects infrared energy, that is, heat, and converts it into a visible image, and a thermal image refers to an image captured by a thermal imaging camera.

The first object 2310 may be divided into a first area 10 , a second area 20 , and a third area 30 .

Here, the maximum temperature of the first region 10 is 25 degrees, the maximum temperature of the second region 20 is 35 degrees, and the maximum temperature of the third region 30 is 65 degrees.

Accordingly, the maximum temperature of the first object 2310 is 65 degrees.

The second object 2320 may be divided into a first area 40 , a second area 50 , and a third area 60 .

Here, the maximum temperature of the first region 40 is 98 degrees, the maximum temperature of the second region 50 is 35 degrees, and the maximum temperature of the third region 60 is 25 degrees.

Accordingly, the maximum temperature of the second object 2310 is 98 degrees.

Also, the controller 230 may analyze the real image 2300 to detect the flame 5 of the first area 40 of the second object 2320 .

According to the present invention, individual objects included in the thermal image 2500 may be divided into a first area, a second area, and a third area.

Here, the divided individual regions of individual objects in the real image 2300 are the same as the divided individual regions of individual objects in the thermal image 2500 .

26 is a diagram illustrating an example of executing a fire alarm by comparing a maximum temperature for each object with a safety temperature stored in a DB according to an embodiment of the present invention.

The highest temperature of the first object 2310 is 65 degrees, which becomes the third region 30 in FIG. 25 .

The safe temperature range of the first object 2310 stored in the DB is -10 degrees to 80 degrees.

Therefore, since the highest temperature of the first object belongs to the safe temperature range, the controller 230 does not execute a fire alarm.

The highest temperature of the second object 2320 is 98 degrees, which becomes the first region 40 in FIG. 25 .

The controller 230 may analyze the real image 2300 and detect the flame 5 of the first area 40 of the second object 2320 .

The controller 230 checks the maximum temperature of the first region 40 of the second object 2320 through the thermal image 2500, analyzes the real image 2300, and determines that the flame 5 is detected. It is confirmed that the area is the first area 40 of the second object 2320 .

Next, the controller 230 determines a fire by comparing the maximum temperature range of the second object 2320 with the safe temperature range.

The safe temperature range of the second object 2320 stored in the DB is -10 degrees to 80 degrees.

Therefore, since the maximum temperature of 98 degrees of the second object 2320 exceeds the safe temperature range, the controller 230 executes a fire alarm.

Also, the controller 230 may detect that a fire has occurred in the first area 40 of the second object 2320 . Therefore, there is an advantage in that the location of the fire can be found more accurately.

When determining a fire, the controller 230 may set different weights between the real image and the thermal image.

Specifically, the weight of the real image may be set lower than that of the thermal image.

For example, the weight of a real image may be set to 1 and the weight of a thermal image may be set to 3.

This is because, when determining whether or not there is a fire, it is more accurate to determine based on whether the maximum temperature of an object falls within a safe temperature range than simply determining based on flame and smoke images.

According to the present invention, it is possible to find the highest temperature by detecting a region where a flame has occurred from a visual image, and detecting a temperature for each region from a thermal image, and determine whether a fire has occurred and a fire by referring to a safe temperature for each object stored in a database. Since the occurrence location can be found more accurately, user convenience can be improved.

Various embodiments of the present disclosure are intended to explain representative aspects of the present disclosure, rather than listing all possible combinations, and matters described in various embodiments may be applied independently or in combination of two or more.

In addition, various embodiments of the present disclosure may be implemented by hardware, firmware, software, or a combination thereof. For hardware implementation, one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented by a processor (general processor), controller, microcontroller, microprocessor, or the like. For example, it is obvious that it can be implemented in the form of a program stored in a non-transitory computer readable medium that can be used at the end or edge, or in the form of a program stored in a non-transitory computer readable medium that can be used at the edge or in the cloud. do. In addition, it may be implemented with a combination of various hardware and software.

The scope of the present disclosure is software or machine-executable instructions (eg, operating systems, applications, firmware, programs, etc.) that cause operations according to methods of various embodiments to be executed on a device or computer, and such software or It includes a non-transitory computer-readable medium in which instructions and the like are stored and executable on a device or computer.

Since the present disclosure described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present disclosure to those skilled in the art, the scope of the present disclosure is limited to the foregoing. It is not limited by one embodiment and accompanying drawings.

100: mobile robot
110: real image camera unit
120: thermal imaging camera
130: communication department
140: control unit, 150: memory

Claims (10)

In the control method of a fire monitoring system using artificial intelligence,
Receiving a real image and a thermal image from a mobile robot;
detecting at least one of smoke and flame by analyzing the real image;
extracting a detection area corresponding to at least one of the smoke and the flame from a real image;
individually identifying at least one object included in the sensing area;
extracting a maximum temperature of the sensing area by analyzing a first area corresponding to the sensing area in the thermal image; and
Including determining a fire by comparing the individual allowable temperature of the identified object with the maximum temperature,
Control method of fire monitoring system.
According to claim 1,
The mobile robot includes a real image camera and a thermal image camera,
The mobile robot
Capturing a real image with the real image camera;
Capturing the thermal image with the thermal imaging camera,
Control method of fire monitoring system.
According to claim 2,
The capture cycle of the real image is different from the capture cycle of the thermal image,
The mobile robot includes a mobile robot moving along a rail,
Control method of fire monitoring system.
According to claim 1,
Further comprising determining as a fire when the maximum temperature is higher than the individual allowable temperature of the object.
Control method of fire monitoring system.
According to claim 1,
Further comprising determining that it is not a fire when the maximum temperature is lower than the individual allowable temperature of the object.
Control method of fire monitoring system.
According to claim 1,
The step of detecting at least one of smoke and flame by analyzing the real image
Including the step of detecting at least one of smoke and flame using a CNN in the real image,
Control method of fire monitoring system.
According to claim 6,
The step of detecting at least one of smoke and flame using a CNN in the real image
storing the real image as a real image in chronological order;
converting the stored real image into a black and white image;
image processing the converted black and white image;
extracting outlines from the image-processed black-and-white image;
Performing learning using the extracted image contour as an input to the CNN; and
Further comprising detecting at least one of smoke and flame based on the learning result.
Control method of fire monitoring system.
According to claim 1,
The step of individually identifying at least one object included in the sensing area,
Further comprising individually identifying the at least one object using YOLOv2,
Control method of fire monitoring system.
According to claim 8,
The step of individually identifying the at least one object using YOLOv2,
extracting the real picture screen as individual images;
Dividing the extracted individual images into grids of a predetermined size;
estimating the number of bounding boxes specified in a predefined form around the center of the divided grid;
determining whether the bounding box includes the at least one object by using reliability calculation; and
Further comprising identifying the at least one object according to a determination result.
Control method of fire monitoring system.
According to claim 1,
The step of determining a fire by comparing the allowable temperature for each identified object with the maximum temperature
Further comprising determining a fire by comparing the allowable temperature for each object with the maximum temperature with reference to a database including permissible temperature information for each object,
Control method of fire monitoring system.

Images