KR102465105B1 - Apparatus and method for detecting disaster based on artificial intelligence model using thermal image complex data - Google Patents

Abstract

The present disclosure relates to a method for an electronic device to detect a disaster based on thermal composite data and an electronic device for performing the same. According to one embodiment, the method for an electronic device to detect a disaster may comprise: a step of obtaining a thermal image by monitoring a region to be monitored; a step of obtaining an optical image by monitoring the region to be monitored; a step of obtaining a primary fire identification result for the region to be monitored based on whether the position of a thermal fire identification region representing a temperature within a preset threshold range in the thermal image and the position of an optical fire identification region determined according to flame or smoke detection conditions in the optical image match; a step of obtaining a secondary fire identification result regarding whether a fire has occurred in the region to be monitored from an artificial intelligence model by inputting the optical image into the artificial intelligence model; and a step of determining whether a fire has occurred in the region to be monitored based on at least one of the obtained primary fire identification result and secondary fire identification result. According to an embodiment, a disaster can be effectively detected by using the optical image and the thermal image in combination.

Description

A method for detecting a disaster based on an AI model using thermal image composite data and an electronic device for performing the same

The present disclosure relates to a method for detecting a disaster and an electronic device for performing the same. More particularly, it relates to a method of detecting a disaster related to at least one of a fire or an earthquake based on an artificial intelligence model using thermal image composite data, and an electronic device for performing the same.

With the gradual development of artificial intelligence technology, it is required to develop a technology to detect disasters using artificial intelligence technology. In particular, various disasters have recently occurred due to environmental issues such as climate change, and attempts are being made to apply artificial intelligence technology to various disaster monitoring technologies to reduce damage caused by these disasters.

In particular, in modern society, as the population density increases, there is a problem that the damage caused by a disaster also increases when a disaster occurs. It is necessary to prevent personal injury.

Various technologies have been tried to apply artificial intelligence technology to disaster detection, but there is a limit to the prediction accuracy, and the development of artificial intelligence model design technology optimized for various types of complex data acquired for the monitoring target space is required. . In addition, there is a demand for technology development for early detection and efficient suppression of fires in the space to be monitored by using artificial intelligence models and image processing algorithms in combination.

Korean Patent Publication No. 2019-0142843

According to an embodiment, a method for detecting a disaster and an electronic device for performing the same may be provided.

In more detail, a method for detecting a disaster by applying a flame or smoke analysis algorithm and an artificial intelligence model together to composite data including at least one of thermal image or optical image data, and an electronic device for performing the same may be provided. .

As a technical means for achieving the above-described technical problem, according to an embodiment, a method for detecting a disaster by an electronic device includes: acquiring a thermal image by monitoring a monitoring target area; acquiring an optical image by monitoring the area to be monitored; Based on whether the position of the thermal image fire identification area indicating the temperature in the preset critical range in the thermal image and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the monitoring obtaining a primary fire identification result for the target area; obtaining a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model by inputting the optical image into an artificial intelligence model; and determining whether a fire has occurred in the monitoring target area based on at least one of the obtained primary fire identification result and secondary fire identification result; may include.

According to another embodiment for solving the above technical problem, there is provided an electronic device for detecting a disaster, comprising: a thermal imaging camera; optical camera; network interface; a memory storing one or more instructions; and a plurality of processors executing the one or more instructions. Including, wherein the plurality of processors by executing the one or more instructions, obtain a thermal image by monitoring the area to be monitored, obtain an optical image by monitoring the area to be monitored, and a preset in the thermal image Based on whether the position of the thermal image fire identification area indicating the temperature in the critical range and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the primary fire for the area to be monitored Obtaining an identification result, and inputting the optical image into an artificial intelligence model, to obtain a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model, and the obtained primary fire An electronic device that determines whether a fire occurs in the monitoring target area based on at least one of an identification result or a secondary fire identification result may be provided.

According to another embodiment for solving the above-described technical problem, there is provided a method for detecting a disaster by an electronic device, the method comprising: acquiring a thermal image by monitoring an area to be monitored; acquiring an optical image by monitoring the area to be monitored; Based on whether the position of the thermal image fire identification area indicating the temperature in the preset critical range in the thermal image and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the monitoring obtaining a primary fire identification result for the target area; obtaining a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model by inputting the optical image into an artificial intelligence model; and determining whether a fire has occurred in the monitoring target area based on at least one of the obtained primary fire identification result and secondary fire identification result; A computer-readable recording medium in which a program for performing the method is stored, including a computer-readable recording medium may be provided.

According to an embodiment, a disaster may be effectively detected by using an optical image and a thermal image in combination.

According to an exemplary embodiment, optical image data and thermal image data may be effectively analyzed using a plurality of processors that respectively perform an artificial intelligence model and an image processing algorithm.

1 is a diagram schematically illustrating a process in which an electronic device detects a disaster according to an exemplary embodiment.
FIG. 2 is a diagram for describing an operation of an electronic device monitoring a predetermined monitoring target area according to an exemplary embodiment.
3 is a flowchart of a method for an electronic device to detect a disaster according to an embodiment.
4 is a flowchart of a method for detecting a disaster by an electronic device according to another exemplary embodiment.
5 is a diagram for describing in detail a process in which an electronic device determines whether an earthquake occurs, according to an exemplary embodiment.
6 is a diagram for describing an internal structure of an electronic device according to an exemplary embodiment.
7 is a diagram for describing in detail a process in which an electronic device acquires a first fire identification result, according to an exemplary embodiment.
FIG. 8 is a diagram for describing a process in which an electronic device identifies a location of an optical fire identification area according to an exemplary embodiment.
9 is a diagram for describing a process in which an electronic device determines a fire candidate area according to an exemplary embodiment.
10 is a diagram for describing a process in which an electronic device determines at least one fire index, according to an exemplary embodiment.
11 is a diagram for describing a process in which an electronic device determines a zero cross count index, according to an exemplary embodiment.
12 is a diagram for describing a process in which an electronic device determines a high-pass filter area index, according to an exemplary embodiment.
13 is a diagram for describing a process in which an electronic device determines a periodicity index, according to an embodiment.
14 is a diagram for describing a process in which an electronic device determines a motion region index, according to an embodiment.
15 is a diagram for describing a process in which an electronic device determines a moving object index, according to an embodiment.
16 is a flowchart illustrating a process in which an electronic device determines whether to identify a primary fire based on a result of detecting a flame, according to an embodiment.
17 is a block diagram of an electronic device according to an embodiment.
18 is a block diagram of an electronic device according to another embodiment.
19 is a block diagram of a server according to an embodiment.
20 is a diagram for explaining a process of detecting a disaster by interworking between an electronic device and a server according to an embodiment.
21 is a diagram for explaining a process of performing a method for detecting a disaster by interworking between an electronic device and a server according to another embodiment.
22 is a diagram for describing a process of generating, by an electronic device, learning data for learning an artificial intelligence model, according to an embodiment.
23 is a diagram for describing a process in which an electronic device generates an alarm event using an artificial intelligence model, according to an exemplary embodiment.
24 is a diagram for describing in detail a method for an electronic device to detect smoke according to a smoke detection condition, according to an exemplary embodiment.
25 is a diagram illustrating a smoke detection result detected by an electronic device according to an exemplary embodiment.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail.

The terms used in the present disclosure have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

In the entire specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software, or a combination of hardware and software. .

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those of ordinary skill in the art to which the present disclosure pertains can easily implement them. However, the present disclosure may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present disclosure in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

1 is a diagram schematically illustrating a process in which an electronic device detects a disaster according to an exemplary embodiment.

According to an embodiment, the electronic device 1000 may acquire at least one of the optical image 102 , the thermal image 104 , and other disaster information 105 . For example, the electronic device 1000 may acquire at least one of a plurality of optical images or thermal images having a frame interval of a preset interval. According to an embodiment, the other disaster information 105 may include seismic wave information obtained through at least one sensor (eg, a 3-axis acceleration sensor). The electronic device 1000 according to the present disclosure may be used in an intelligent fire detection system capable of early detection and effective suppression of a fire based on complex data appearing in an optical image and a thermal image.

For example, the electronic device 1000 may analyze at least one of the optical image 102 and the thermal image 104 and determine whether a fire occurs based on the image analysis result. Also, the electronic device 1000 may obtain the seismic wave information 104 and determine whether an earthquake occurs based on the analysis result of the seismic wave information. When at least one disaster of a fire or an earthquake occurs, the electronic device 1000 may output information about the disaster determined to have occurred as the disaster notification information 144 .

According to an embodiment, the electronic device 1000 may output the disaster notification information 144 by interworking with the server 2000 . According to an embodiment, the electronic device 1000 may acquire at least one optical image, at least one thermal image image, or seismic wave information through the server 2000 . According to an embodiment, the server 2000 is communicatively connected to the electronic device 1000 through a network and may include other computing devices capable of transmitting and receiving data. According to an embodiment, the network includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and It is a data communication network in a comprehensive sense that includes a combination of these and enables each network constituent entity shown in FIG. 1 to communicate smoothly with each other, and may include a wired Internet, a wireless Internet, and a mobile wireless communication network.

According to an embodiment, the electronic device 1000 may be implemented in various forms. According to an embodiment, the electronic device 1000 is implemented in a fire detection system inside a wind turbine nacelle, so that artificial intelligence PHM (Prognostics and Health Management) based on complex sensor data information, such as smoke, temperature, image, thermal image, and ultraviolet light, is implemented. ) technology can be a computing device used to detect fires early and extinguish them.

For example, the electronic device 1000 described herein is a digital camera, a mobile terminal, a smart phone, a CCTV, a camera device that is located inside a wind turbine nacelle internal fire detection system to monitor a monitoring target area. , a laptop computer, may be a tablet PC, but is not limited thereto. According to an embodiment, for convenience, the electronic device 1000 according to the present disclosure may be a computing device including a processor for controlling a CCTV device or a CCT device.

According to an embodiment, the electronic device 1000 includes an optical camera 110 , a thermal imaging camera 111 , at least one sensor 112 , a network interface 114 , a processor 120 , and a memory 122 . can do. However, the present invention is not limited to the above-described configuration, and the electronic device 1000 may include fewer components as shown in FIGS. 17 to 18 to be described later, or may include more components in addition to the above-described configuration. .

According to an embodiment, the processor 120 may control the overall operation of the electronic device 1000 by executing one or more instructions stored in the memory 122 . According to an embodiment, the optical camera 110 may acquire a plurality of optical images having a predetermined frame interval by photographing a predetermined monitoring target area under the control of the processor 120 . According to an embodiment, the thermal imaging camera 111 may obtain thermal image images in which temperature data for the target area is imaged by sensing infrared rays reflected and received by the monitoring target area by monitoring a predetermined monitoring target area. have.

According to an embodiment, the at least one sensor 112 may include at least one of an optical image sensor, a thermal image sensor, a three-axis acceleration sensor, and a geomagnetic sensor. At least one sensor 112 may acquire an image sensor value, acquire triaxial acceleration information, acquire geomagnetic information, and transmit the acquired sensor values to the processor 120 under the control of the processor 120 . have.

According to an embodiment, the network interface 114 may transmit sensor values acquired by the at least one sensor 112 to another electronic device connected to the electronic device. According to another embodiment, the network interface 114 may transmit information on whether a fire occurs, an earthquake, or a disaster determined by the electronic device to an external integrated control system. According to another embodiment, the network interface 114 may receive an image or seismic wave information from the server 2000 . Also, the network interface 114 may transmit disaster notification information analyzed by the electronic device to another electronic device connected to the electronic device. Also, the network interface 114 may further receive seismic wave information, earthquake location information (eg, epicenter information, PS city information), and the like, from another electronic device connected to the electronic device.

According to an embodiment, the disaster notification information 144 output by the electronic device 1000 may include at least one of fire occurrence information and earthquake occurrence information. According to an embodiment, the fire occurrence information includes at least one of a fire occurrence type (eg, flame or smoke type), fire occurrence location information, a fire occurrence scale, emergency relief information according to a fire occurrence, and real-time news information about a fire occurrence. can do. Also, according to an embodiment, the earthquake occurrence information may include at least one of an earthquake occurrence location, an earthquake magnitude, an earthquake intensity, emergency relief information according to an earthquake occurrence, and real-time news information on an earthquake occurrence.

FIG. 2 is a diagram for describing an operation of an electronic device monitoring a predetermined monitoring target area according to an exemplary embodiment.

According to an embodiment, the electronic device 1000 may be a CCTV device for monitoring a monitoring target area. According to an embodiment, the electronic device 1000 may be a CCTV device installed inside a wind turbine nacelle. However, it is not limited to the above example, and the electronic device 1000 may be a CCTV device installed on a building or a predetermined outer wall 220 . The electronic device 1000 may be attached to the outer wall 220 in a building or a predetermined space to monitor the monitoring target area 230 . The electronic device 1000 may acquire an image including a plurality of frame images at a preset interval by monitoring the monitoring target area. Also, the electronic device 1000 may sense seismic wave information according to the occurrence of an earthquake using at least one sensor at an installed position.

According to an embodiment, even when an earthquake does not occur, the electronic device 1000 may sense the vibration of a building or an exterior wall according to a preset period. According to an embodiment, the electronic device 1000 determines a resonance frequency of a building or an exterior wall in which the electronic device 1000 is installed by sensing vibration of a building or an exterior wall according to a preset period, and provides information on the determined resonance frequency. It may be transmitted in advance to another electronic device connected to the electronic device. According to an embodiment, the electronic device 1000 may store the resonant frequency of the building in which the electronic device 1000 is installed in advance, and then output notification information more strongly when seismic wave information corresponding to the resonant frequency is sensed. .

3 is a flowchart of a method for an electronic device to detect a disaster according to an embodiment.

In S310 , the electronic device 1000 may acquire a thermal image by monitoring the monitoring target area. According to an embodiment, the electronic device 1000 may acquire a plurality of thermal image images having a preset frame interval from the thermal image camera. The thermal images may include a plurality of pixels representing a temperature value by sensing an infrared wavelength reflected from the object to be monitored. In S320 , the electronic device 1000 may acquire an optical image by monitoring the monitoring target area. According to an embodiment, the electronic device 1000 may acquire a plurality of optical images having a preset frame interval from the optical camera.

In S330, the electronic device 1000 determines whether the position of the thermal image fire identification region indicating a temperature in a preset critical range in the thermal image matches the position of the optical fire identification region determined according to the flame or smoke detection condition in the optical image. Based on whether or not it is possible to obtain a primary fire identification result for the area to be monitored. For example, the electronic device 1000 determines whether there is an image region representing a temperature in a preset critical range in the thermal image based on pixel values in the thermal image, and When the area is equal to or greater than the critical area, the position of the image region representing the preset critical range temperature in the thermal image may be identified as the position of the thermal image fire identification region.

Also, for example, the electronic device 1000 determines at least one fire index based on whether a flame or smoke detection condition to be described later for the optical image is satisfied, and an optical fire identification area based on the determined at least one fire index. location can be identified. In the electronic device 1000, the position of the thermal image fire identification area is not determined from the thermal image, the position of the optical fire identification region is not identified from the optical image, or the position of the identified thermal image fire identification region and the identified optical fire If the location of the identification area does not match, it may be determined that a fire has not occurred. The electronic device 1000 according to the present disclosure may primarily determine a fire identification result by using a thermal image and an optical image in combination.

In S340 , the electronic device 1000 may obtain a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model by inputting the optical image into the artificial intelligence model. For example, the electronic device 1000 obtains a primary fire identification result from an optical image and a thermal image using a first processor in the electronic device, and uses a separate second processor for performing an artificial intelligence algorithm. By applying the artificial intelligence model to the optical image, secondary fire identification results can be obtained from the artificial intelligence model.

According to another embodiment, the electronic device 1000 pre-processes the acquired optical image in a format optimized for application of the artificial intelligence model, and inputs the pre-processed optical image to the artificial intelligence model, so that the area to be monitored from the artificial intelligence model A probability value regarding whether or not a fire has occurred may be obtained as a result of secondary fire identification.

According to an embodiment, the artificial intelligence model used by the electronic device 1000 may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the above examples does not

According to an embodiment, the artificial intelligence model used by the electronic device 1000 tracks pixels for movement of smoke (eg, an object) in an optical image, identifies a motion vector for the corresponding pixels, and provides a low-frequency signal to a background in the optical image. By applying a pass filter, it may be an artificial intelligence model pre-trained to identify smoke in an optical image according to its correlation with the identified smoke.

In S350 , the electronic device 1000 may determine whether a fire occurs in the monitoring target area based on at least one of the primary fire identification result and the secondary fire identification result. The electronic device 1000 according to the present disclosure may quickly and accurately determine whether or not a fire has occurred by using not only complex data combining an optical image and a thermal image, but also using a pre-learned artificial intelligence model.

4 is a flowchart of a method for detecting a disaster by an electronic device according to another exemplary embodiment.

Since S410 to S450 may correspond to S310 to S350 described above in FIG. 3 , a detailed description thereof will be omitted. In S460 , the electronic device 1000 may acquire seismic wave information by sensing a seismic wave at a location where the electronic device 1000 is installed using at least one sensor. According to an embodiment, the electronic device 1000 may acquire 3-axis acceleration information and geomagnetic information by using at least one of a 3-axis acceleration sensor and a geomagnetic sensor.

In S470 , the electronic device 1000 may determine at least one earthquake index based on seismic wave information, and determine whether an earthquake occurs based on the determined earthquake index. In S480 , the electronic device 1000 may output notification information according to whether a fire or an earthquake has occurred. The electronic device 1000 according to the present disclosure may detect a complex disaster by not only detecting whether a fire has occurred in a monitoring target area but also identifying an earthquake.

5 is a diagram for describing in detail a process in which an electronic device determines whether an earthquake occurs, according to an exemplary embodiment.

In S510 , the electronic device 1000 may identify a geomagnetic direction at a location where the electronic device is installed based on geomagnetic information. According to an embodiment, the geomagnetic information may include information about the east, west, north, and south directions. In S520 , the electronic device 1000 may identify a 3-axis acceleration direction based on 3-axis acceleration information. According to an embodiment, the electronic device 1000 may include a 3-axis acceleration sensor, and may identify a 3-axis acceleration direction based on 3-axis acceleration information obtained from the 3-axis acceleration sensor. According to an embodiment, the 3-axis acceleration direction may be determined based on an acceleration value for each axis in the 3-axis acceleration information. According to an embodiment, the three-axis acceleration direction may be in a state that does not match the geomagnetic direction.

According to an embodiment, although not shown in FIG. 5 , according to an embodiment, the electronic device 1000 may correct the acceleration information of the 3-axis acceleration installed in the electronic device. According to an embodiment, the 3-axis acceleration sensor in the electronic device may not be installed parallel to the ground, but may be installed to be inclined from the reference axis. The electronic device 1000 may further perform a process of obtaining triaxial acceleration information and correcting an acceleration value for each axis in the obtained triaxial acceleration information.

In S530, the electronic device 1000 may match the three-axis acceleration direction to the identified geomagnetic direction. For example, the three-axis acceleration direction and the geomagnetic direction may not match. The electronic device 1000 may identify which direction the 3-axis acceleration direction corresponds to based on the actual geomagnetic direction by matching the 3-axis acceleration direction with the geomagnetic direction.

In S540 , the electronic device 1000 may determine the maximum ground acceleration by monitoring the magnitude of the 3-axis acceleration based on the 3-axis acceleration direction matched to the geomagnetic direction. According to an embodiment, the maximum ground acceleration may be a maximum acceleration value measured from a pure seismic wave. According to an embodiment, based on the maximum ground acceleration, the electronic device 1000 may determine the currently measured intensity of the earthquake. In S450 , the electronic device 1000 may determine whether an earthquake occurs based on the three-axis acceleration direction matched to the geomagnetic direction and the determined maximum ground acceleration.

Hereinafter, a process in which the electronic device 1000 processes the 3-axis acceleration information will be described in detail. According to an embodiment, the 3-axis acceleration sensor in the electronic device 1000 may be disposed to be inclined from a horizontal plane or a vertical plane of the electronic device. For example, the Y axis of the acceleration sensor and the Z axis of the acceleration sensor may be inclined by a predetermined angle from the Y reference axis and the Z reference axis parallel to the horizontal and vertical, respectively, with respect to an axis horizontal to the ground.

와 같은 식을 이용하여 현재 가속도 센서의 z축 출력 값을 결정할 수 있다. 전자 장치(1000)는 가속도 센서의 y축 출력 값 및 z축 출력 값을

와 같은 수식에 적용함으로써 x축이 지면과 이루는 각도를 측정할 수 있다. 또한, 전자 장치(1000)는 가속도 센서 x축 출력 값 및 z축 출력 값을

와 같은 수식에 적용함으로써, y축에 대한 각도를 결정할 수 있다. 일 실시 예에 의하면 z축에 대한 각도를 결정되지 않을 수 있다. 일 실시 예에 의하면, 재난감지 카메라 내 3축 가속도 센서의 z축은 실제 z축 방향과 일치하도록 마련될 수 있다. 전자 장치(1000)는 가속도 센서의 각 축에 대한 각도 값을 상술한 바와 같이 결정한 다음, 결정된 가속도 센서의 각축에 대한 각도 값을 수평인 각도로 치환하여 보정된 3축 가속도 센서의 출력 값을 획득할 수 있다.The electronic device 1000 may obtain an x-axis output value and a y-axis output value from the 3-axis acceleration sensor. The electronic device 1000 is

The z-axis output value of the current accelerometer can be determined by using the same equation. The electronic device 1000 receives the y-axis output value and the z-axis output value of the acceleration sensor.

By applying the same formula as , the angle between the x-axis and the ground can be measured. Also, the electronic device 1000 receives an x-axis output value and a z-axis output value of the acceleration sensor.

By applying an equation such as , the angle with respect to the y-axis can be determined. According to an embodiment, the angle with respect to the z-axis may not be determined. According to an embodiment, the z-axis of the 3-axis acceleration sensor in the disaster detection camera may be provided to coincide with the actual z-axis direction. The electronic device 1000 determines an angular value for each axis of the acceleration sensor as described above, and then replaces the determined angular value for each axis of the acceleration sensor with a horizontal angle to obtain a corrected output value of the 3-axis acceleration sensor can do.

Hereinafter, a process in which the electronic device 1000 determines the progress based on the maximum ground acceleration will be described in detail. For example, the electronic device 1000 may determine the maximum ground acceleration by using the output value of the 3-axis acceleration sensor. According to an embodiment, the electronic device 1000 may measure the maximum ground acceleration based on a horizontal direction on the ground. The electronic device 1000 may determine the progress according to the determined maximum ground acceleration based on a table in which pre-stored maximum ground acceleration and progress information are matched. When the determined seismic intensity is equal to or greater than a preset threshold intensity, the electronic device 1000 may determine that an earthquake has occurred based on the three-axis acceleration direction matched to the geomagnetic direction. Also, according to an embodiment, the electronic device 1000 may image the seismic wave in real time by using the 3-axis acceleration information. The electronic device 1000 may measure the maximum ground acceleration by directly measuring the amplitude of the imaged seismic wave in real time.

Hereinafter, a process for the electronic device 1000 to identify an earthquake occurrence location will be described in detail. According to an embodiment, the electronic device 1000 may measure PS time, which is a difference between the arrival times of the P wave and the S wave, by using the 3-axis acceleration sensor. When the PS time is measured, the electronic device 1000 may determine a distance from the location where the electronic device is currently installed to the epicenter by using the measured PS time. The electronic device 1000 may determine a first epicenter with a distance to the epicenter as a radius. Also, the electronic device 1000 may obtain, from two or more other electronic devices connected to the electronic device, location information of the other electronic device and epicenter information regarding a distance from the other electronic device to the epicenter determined by the other electronic device. The electronic device 1000 may determine the second epicenter and the third epicenter based on epicenter information obtained from another electronic device.

According to an embodiment, the electronic device 1000 may identify an intersection point of a common chord of each epicenter having three epicenter distances as a radius as an actual epicenter. Also, the electronic device 1000 may determine a 1/2 value of the length of the shortest common chord passing through the epicenter as the depth from the epicenter to the epicenter. The electronic device 1000 may determine the location of the epicenter and depth information from the epicenter to the epicenter as the epicenter information. The electronic device 1000 may output epicenter information, epicenter information, and information on the intensity of an earthquake determined based on the maximum ground acceleration as notification information.

6 is a diagram for describing an internal structure of an electronic device according to an exemplary embodiment.

Referring to a figure 610 illustrated in FIG. 6 , components in the electronic device 1000 and relationships among the components are illustrated. According to an embodiment, the electronic device 1000 includes an embedded module 610 , a GPU module 620 , a 3-axis acceleration sensor 654 , an IP module 652 , a CMOS image sensor 630 , and a thermal image sensor ( 640) may be included. The embedded module 610 may output a primary fire identification result by processing thermal images and optical images obtained from a thermal image sensor and an optical image sensor (eg, a CMOS image sensor). Also, the GPU module 620 is a processor that applies an artificial intelligence model to optical images, and may be a separate processor for performing an artificial intelligence algorithm. The GPU module 620 obtains the secondary fire identification result from the artificial intelligence model by applying the artificial intelligence model to the optical image, and transfers the obtained secondary fire identification result to the embedded module 610, or separately in the electronic device. It can also be passed to the integrated processor of Since the functions of the image sensor, the thermal image sensor, the IP module, and the 3-axis acceleration sensor can be recognized by those skilled in the art from the name, a detailed description thereof will be omitted.

7 is a diagram for describing in detail a process in which an electronic device acquires a first fire identification result, according to an exemplary embodiment.

In S710 , the electronic device 1000 may identify the location of the thermal image fire identification region with respect to some image regions satisfying a temperature of a preset threshold range based on pixel values of the thermal image. For example, the electronic device 1000 introduces a sprinkler standard so that when heat between 72 degrees and 100 degrees is detected from the thermal image, the location of the thermal image fire identification area identified as a fire from the thermal image. can be identified.

In more detail, the electronic device 1000 may identify pixels representing a temperature in the preset threshold range based on pixel values in the thermal image, and determine a plurality of clusters including the identified pixels. The electronic device 1000 may identify a cluster having an area greater than or equal to a predetermined critical area among the determined clusters, and determine a cluster having the largest area among the identified clusters as the thermal image fire identification area.

The electronic device 1000 may identify a location on a thermal image image or a physical space indicated by the thermal image of the thermal image fire identification area having the largest area among clusters satisfying a predetermined critical range identified in the thermal image. can According to an embodiment, the location of the thermal image image of the thermal image fire identification region is the outer boundary according to coordinate information of pixels constituting the thermal image fire identification region and coordinate information of pixels located at the outermost edge of the thermal image fire identification region. It may include at least one of information. However, according to another embodiment, when a temperature having a predetermined critical range is not detected in the thermal image, the electronic device 1000 may not be able to identify the location of the thermal image fire identification area.

In S720 , the electronic device 1000 may identify the location of the optical fire identification area based on at least one fire index used to determine the flame or smoke detection condition in the optical image. According to an embodiment, the electronic device 1000 determines a fire candidate area from the optical image, determines various types of fire indices for a flame or smoke detection condition based on pixel values in the fire candidate area, and determines the determined fire When the indices satisfy the index condition, the fire candidate area may be identified as a fire suspicious area. The electronic device 1000 may identify the location of the fire candidate area as the location of the optical fire identification area when the number of times the fire candidate area is identified as the fire suspicious area is equal to or greater than a preset threshold. A method for the electronic device 1000 to determine the position of the optical fire identification area will be described in detail with reference to FIG. 8 to be described later.

In S730 , the electronic device 1000 may obtain the primary fire identification result based on whether the position of the identified thermal image fire identification region matches the position of the optical fire identification region. For example, when the location of the thermal image fire identification area determined based on the thermal image matches the location of the fire identification area determined from the optical image, the electronic device 1000 determines that a fire has occurred in the monitoring target area. It is possible to determine and output the primary fire identification result that a fire has occurred in the area to be monitored.

According to an embodiment, the electronic device 1000 determines whether the location of the thermal image fire identification area and the optical fire identification area match, whether the location of the actual fire identification area and the optical fire identification area match on the image. However, according to another embodiment, it may be determined according to whether the location in the physical space indicated by the location of the actual fire identification area and the location of the optical fire identification area match.

However, according to another embodiment, in the electronic device 1000, the position of the thermal image fire identification region is not determined from the thermal image, the position of the optical fire identification region is not determined from the optical image, or the position of the thermal image fire identification region is not determined from the optical image. Even if the location and the location of the optical fire identification area are determined, if the location of the thermal image fire identification area and the location of the optical fire identification area do not match, it is determined that a fire has not occurred, and the primary fire identification that no fire has occurred You can also print the results.

FIG. 8 is a diagram for describing a process in which an electronic device identifies a location of an optical fire identification area according to an exemplary embodiment.

In S810 , the electronic device 1000 may convert RGB data of pixels in the obtained optical image into HSL data. According to an embodiment, the HSL data may include values of hue (HUE), saturation (SATURATION), and brightness (LIGHTNESS). In S820 , the electronic device 1000 may determine a fire candidate area in the optical image based on the HSL data. According to an embodiment, the electronic device 1000 may more accurately determine the fire candidate area by using threshold values of different levels in determining the fire candidate area in the image. A method for the electronic device 1000 to determine an accurate fire candidate area in an optical image using thresholds of different levels will be described in detail with reference to FIG. 9 to be described later.

In S830, the electronic device 1000 may determine at least one fire index related to a flame or smoke detection condition for the determined fire candidate area. For example, the electronic device 1000 may determine at least one fire index for the fire candidate area. According to an embodiment, the fire index may be a parameter for an image or an image used by the electronic device 1000 to determine whether a fire has occurred. In S840 , the electronic device 1000 may identify the number of times that a fire candidate area is identified as a fire suspicious area based on at least one fire index. For example, when at least one fire index satisfies the fire index conditions for each, the electronic device 1000 may identify a corresponding fire candidate area as a fire suspicious area and increase the number of fire suspicious counts by 1. .

In S850, when the number of times identified is equal to or greater than a preset threshold, the electronic device 1000 may determine the location of the fire candidate area as the location of the optical fire identification area. According to an embodiment, the location of the fire candidate area may include coordinate information of pixels constituting the fire candidate area in the optical image, when the number of times the fire candidate area determined in the optical image is identified as a fire suspicious area is greater than or equal to a threshold. At least one of outer boundary information according to coordinate information of pixels located at the outermost edge may be included. According to another embodiment, the electronic device 1000 may identify a location in an actual physical space indicated by a fire candidate area in which the number of times of being identified as a fire suspicious area is equal to or greater than a threshold, as the location of the optical fire identification area.

9 is a diagram for describing a process in which an electronic device determines a fire candidate area according to an exemplary embodiment.

In S920 , the electronic device 1000 may identify color, saturation, and brightness values in the HSL data, and determine a first candidate area including pixels having all of the identified color, saturation, and brightness values greater than or equal to a predetermined threshold.

In S940 , the electronic device 1000 may identify pixels having at least one of the hue, the saturation, and the brightness in the HSL data equal to or greater than the predetermined threshold. According to an embodiment, the number of pixels identified in S940 may be greater than the number of pixels included in the first candidate area. In S960 , the electronic device 1000 selects pixels in which at least one of the hue, saturation, and brightness is equal to or greater than the predetermined threshold when all of the color, saturation, and brightness values of pixels surrounding the pixels identified in S940 are equal to or greater than the predetermined threshold. and a second candidate area including pixels included in the first candidate area. In S980 , the electronic device 1000 may determine the second candidate area as the fire candidate area.

That is, the electronic device 1000 according to the present disclosure determines a first candidate region from the image by using the first threshold criterion (eg, all of the color, saturation, and brightness of each pixel in the image are equal to or greater than the threshold), and then After determining the second candidate region using a second threshold criterion lower than the threshold criterion (eg, at least one of the color, saturation, or brightness of each pixel in the image is greater than or equal to the threshold), the second candidate region may be determined as a fire candidate region. have. According to an embodiment, the second candidate area may include pixels not included in the first candidate area.

Preferably, a pixel not included in the first candidate area may be included as a candidate area in the second candidate area. The electronic device 1000 according to the present disclosure may more accurately determine a fire candidate area by maintaining a standard of brightness among hue, saturation, and brightness, and setting a threshold standard by relaxing the standard of color.

10 is a diagram for describing a process in which an electronic device determines at least one fire index, according to an exemplary embodiment.

In S1010 , the electronic device 1000 may determine a zero cross count index by performing high-pass filtering on a fire candidate area in a plurality of optical images having three adjacent frame numbers among the plurality of optical images. According to an embodiment, when the zero cross count index is equal to or greater than a predetermined threshold, the electronic device 1000 may initialize the counted number of times of identification of the suspected fire area.

According to an embodiment, in S1020, the electronic device 1000 may determine the high-pass filter area index with respect to the amount of change for each frame image of the number of pixels for which the high-pass filter variable is not 0. According to an embodiment, the high-pass filter area index may correspond to the area of a non-zero area after the high-pass filter is applied.

According to an embodiment, the amount of change in the area of a region having a non-zero value belonging to a fire candidate region may not be large with respect to the result image after filtering is performed. When the high-pass filter area index is equal to or greater than a predetermined threshold, the electronic device 1000 may initialize the number of fire suspicious counts.

In S1030 , the electronic device 1000 may determine a periodicity index regarding the periodicity of the fire candidate area in the plurality of frame images. According to an embodiment, in the case of a rapidly repeated image, it may be erroneously detected as a fire suspicious area. Accordingly, when the amount of change in the periodicity index determined to reduce the false detection rate is less than or equal to a predetermined threshold, the electronic device 1000 may initialize the fire suspicious count.

In S1040 , the electronic device 1000 may determine a motion area index with respect to a degree of change in the number of pixels having a large pixel value variation among pixels in the plurality of frame images. For example, in the case of an actual fire area, the variation of RGB data on the time axis may be significantly measured. The electronic device 1000 may extract a pixel having a large variation in pixel value in a plurality of frame images as a moving pixel, and may determine a motion region index using an average and standard deviation of the number of moving pixels per frame.

In S1050 , the electronic device 1000 may determine a moving object index with respect to positions of pixels in which the number of times that a disparity value of a specific pixel appears greater than or equal to a predetermined threshold in the plurality of frame images is a preset number or greater. For example, the electronic device 1000 may monitor a change in the number of moving pixels in each frame based on the frame, but a fire occurs by monitoring a change in the pixel value based on a pixel at a specific position in each frame. It can be more precisely determined whether or not

Also, although not shown in FIG. 10 , the electronic device 1000 may determine a smoke index regarding whether a plurality of optical images satisfy a smoke detection condition. For example, the electronic device 1000 may determine a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images. The electronic device 1000 may identify the motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images.

The electronic device 1000 may cluster the identified motion regions and extract pixels representing a gray color value from among pixels in the clustered motion regions. For example, when the difference between the R value and the G value, the difference between the R value and the B value and the difference between the R value and the B value and the difference between the G value and the B value, among RGB data of the pixel, is smaller than a preset threshold difference, the corresponding A pixel can be extracted as a pixel representing a gray color value.

The electronic device 1000 may identify the motion vector by tracking image regions including pixels representing the gray color value in a plurality of optical images having adjacent frame numbers. When the motion vectors are identified, the electronic device 1000 may obtain an output value of the low-pass filter by applying the low-pass filter to the background image among the plurality of optical images. The electronic device 1000 may determine the smoke index regarding the correlation between the output value of the low-pass filter and the output value of the low-pass filter with respect to the background image when a preset smoke is generated. When the smoke index is less than a predetermined threshold, the electronic device 1000 may initialize the number of times that the fire candidate area is identified as the fire suspicious area. However, the electronic device 1000 determines that the above-described zero cross count index, high-pass filter area index, periodicity index, moving area index, moving object index, and smoke index are each index condition (each index that prevents the fire suspicious area count from being initialized). If a specific condition) is satisfied, the number of fire suspicious counts for the fire candidate area may be increased by 1.

However, according to another embodiment, when even one of the above-described fire indices does not satisfy the index condition, the electronic device 1000 may initialize the number of fire suspicion counts. Since the electronic device 1000 identifies that a fire has occurred only when the number of suspected fire counts is equal to or greater than a preset threshold count, the electronic device 1000 may determine whether a fire has occurred with high accuracy. The electronic device 1000 may determine the location of the fire candidate area as the location of the optical fire identification area when the number of suspicious fire counts is greater than the threshold as the number of times that the fire candidate area is identified as the fire suspicious area increases.

11 is a diagram for describing a process in which an electronic device determines a zero cross count index, according to an exemplary embodiment.

A process in which the electronic device 1000 determines the zero cross count index will be described in detail with reference to an example of an instruction shown in FIG. 1202 of FIG. 11 . According to one embodiment, one of the characteristics of the fire is flicker (flicker). For example, when high-pass filtering is temporally performed on a flickering pixel, frames that repeatedly cross a value of 0 may be obtained. The electronic device 1000 may determine the number of frames crossing 0 in a predetermined frame period unit as the zero cross count index, and increase the number of suspected fire counts when the determined zero cross count index is greater than a predetermined threshold. . However, according to another embodiment, when the determined zero cross count index is less than a predetermined threshold, the electronic device 1000 may initialize the number of fire suspicious counts.

According to an embodiment, the electronic device 1000 may determine the zero cross count index by performing high-pass filtering on the fire candidate area in three adjacent frames among the plurality of frame images.

In more detail, the electronic device 1000 configures pixels for each frame based on weights of −0.25, 0.5, and −0.25 determined for each of consecutive N-1 frames, N frames, and N+1 frames among the plurality of frame images. By weighted summing the values, high-pass filtering may be performed, and a flicker variable according to a result of performing high-pass filtering may be determined for each pixel. Referring to FIG. 11 , the flicker variable (Flick_var) is multiplied by -0.25 by the pixel value (pre_pre) in the N-1 frame at the corresponding position, by 0.5 by the pixel value (pre) in the N frame, and within the N+1 frame. It can be determined by multiplying the pixel value (cur) of the corresponding position by -0.25 and then adding them all together. The electronic device 1000 may perform high-pass filtering by applying a predetermined weight to each pixel in three adjacent frames at a corresponding position and performing weight summing. The electronic device 1000 may determine the flicker variable for each pixel in the frame.

The electronic device 1000 may determine the high-pass filter variable for each pixel by comparing the flicker variable determined for each pixel with a predetermined threshold. The electronic device 1000 may increase the zero cross count index based on whether the determined high-pass filter variable for each pixel satisfies a predetermined condition.

In more detail, the electronic device 1000 identifies even one of the pixels in the three adjacent frames as a pixel in the fire candidate area, and when the flicker variable of the corresponding pixel is less than ??threshold(5), the high value of the corresponding pixel is The pass filtering variable (HPFn) is set to -127, and if the flicker variable is greater than the threshold(5), the high pass filtering variable (HPFn) for the pixel is set to 128, and the flicker variable is set to ??threshold(5) and threshold If it is located between (5), the high-pass filtering variable (HPFn) can be set to 0. Also, the electronic device 1000 may set the high-pass filtering variable HPFn to 0 when pixels in three adjacent frames are not all identified as fire candidate areas.

Also, when the high-pass filtering variable HPFn is not 0, the electronic device 1000 may increase HPF_count, which is a variable related to the number of pixels in which the high-pass filtering variable HPFn for each frame is not 0. Also, in the electronic device 1000, if the high-pass filtering variable (HPFn) of a pixel at a specific position in the current frame is 128, and the high-pass filtering variable (HPFn-1) of the pixel at the corresponding position in the previous frame is -127, The high-pass filtering variable (HPFn) of the pixel at a specific location in the current frame is -127, the high-pass filtering variable (HPFn-1) of the pixel at the corresponding location in the previous frame is 128, or the high pass of the pixel at a specific location in the current frame When the filtering variable HPFn is 0 and the high-pass filtering variable HPFn-1 of the pixel at the corresponding position in the previous frame is 0, the zero cross count index may be increased.

According to the above-described process, when the zero cross count index is greater than a predetermined threshold, the electronic device 1000 may identify the fire candidate area as the fire suspicious area and increase the number of the fire suspicious count by 1. However, when the zero cross count index is less than a predetermined threshold, the electronic device 1000 may initialize the fire suspicious count.

12 is a diagram for describing a process in which an electronic device determines a high-pass filter area index, according to an exemplary embodiment.

According to an embodiment, the electronic device 1000 may determine the high-pass filter variable HPFn for each pixel according to the process described above with reference to FIG. 11 . The electronic device 1000 may store the number of pixels in which the high-pass filter variable is not 0 for each frame in the HPF_Count variable. Referring to the figure 1204 of FIG. 12 , the electronic device 1000 may monitor a change in the number of pixels in which the high-pass filter variable HPFn is not 0 for each frame. According to an embodiment, the electronic device 1000 identifies the number of pixels in which the high-pass filter variable HPFn is not 0 for each frame during frames 0 to 30, and the identified high-pass filter variable HPFn is 0. It is possible to determine the minimum, standard deviation, and mean of the number of non-pixels.

According to an embodiment, the electronic device 1000 may not determine the high-pass filter area index when the minimum value with the smallest number of pixels for which the high-pass filter variable for each frame is not 0 is less than a predetermined threshold. However, when the minimum value with the smallest number of pixels for which the high-pass filter parameter for each frame is not 0 is greater than a predetermined threshold, the electronic device 1000 sets the standard deviation of the number of pixels for which the high-pass filter parameter for each frame is non-zero. By dividing by the mean, the high-pass filter area index can be determined. When the high-pass filter area index is less than a predetermined threshold, the electronic device 1000 may increase the number of suspected fire counts. However, when the high-pass filter area index is greater than a predetermined threshold, the electronic device 1000 may initialize the fire suspicious count.

13 is a diagram for describing a process in which an electronic device determines a periodicity index, according to an embodiment.

Referring to the figure 1206 of FIG. 13 , the electronic device 1000 may determine one frame image among a plurality of frame images as a reference frame image. According to an embodiment, the electronic device 1000 may determine the reference frame image as the backup frame. The electronic device 1000 may perform a period check inter-frame difference between the reference frame image and a plurality of frame images (eg, 1 to 30 frames or 1 to 59 frames) including the reference frame.

For example, the electronic device 1000 may determine the difference variable (diff) for each pixel in the plurality of frame images by performing different mask operations on the reference frame image and one frame among the plurality of frame images. have. In more detail, it is assumed that the periodicity index is determined based on 30 frames and the reference frame is set as the first frame.

The electronic device 1000 may perform a period check inter-frame difference between frame 1, which is the reference frame, and frame 1 in a plurality of frame images (including the reference frame as frame 1). Also, the electronic device 1000 performs period check inter-frame difference with respect to frame 1 and frame 2, which are reference frames, and then repeats the above-described process until frame 1 and frame 30 period check inter-frame. Frame difference can be performed.

The electronic device 1000 performs period check inter-frame difference on any two frames including a reference frame among 30 frames, and applies different mask operations to the two frames. For example, the electronic device 1000 may display a 2 by 2 mask having -1 as an element and a 2 by 2 mask having 1 as an element in two frames including a reference frame within 30 frames. By applying each, period check inter-frame difference can be performed.

The electronic device 1000 determines result 1 by applying a 2 by 2 mask having -1 as an element to the reference frame, and sets 1 as an element to one frame other than the reference frame among 30 frames. Result 2 can be determined by applying a 2 by 2 mask with (element). The electronic device 1000 may determine the diff value by adding 128 to the average value of result 1 and result 2 .

Also, according to an embodiment, the electronic device 1000 may quantize the difference variable by comparing the difference variable (diff) determined for each pixel with a predetermined threshold value. For example, when the difference variable value is less than 128-threshold(5), the electronic device 1000 sets the corresponding difference variable to 0, and when the difference variable value is greater than 128+threshold(5), the corresponding difference variable is set to 255, and in other cases, the difference variable is set to 128, so that the difference variable can be quantized to a predetermined quantum value.

The electronic device 1000 may determine the periodicity index based on the standard deviation and average of the number of pixels per frame image in which the quantized difference variable value does not match the predetermined quantum value. For example, the electronic device 1000 counts the number of pixels whose quantized difference variable value (diff) is not 128 for each frame, and the number of quantized difference variable (diff) values other than 128 counted for each frame Standard deviation and mean can be obtained.

The electronic device 1000 may determine the periodicity index by dividing the standard deviation of the number of pixels whose quantized difference variable value is not 128 by the average for each frame. When the periodicity index is less than a predetermined threshold, the electronic device 1000 may identify the fire candidate area as the fire suspicious area and increase the number of fire suspicious counts by 1. However, when the periodicity index is greater than a predetermined threshold, the electronic device 1000 may initialize the number of suspicious fire counts.

14 is a diagram for describing a process in which an electronic device determines a motion region index, according to an embodiment.

Referring to the figure 1208 of FIG. 14 , the electronic device 1000 determines the number of moving pixels per frame, and determines the average and standard deviation of the number of moving pixels per frame, and the standard of the number of moving pixels per frame. The process of determining the motion range index by dividing the deviation by the mean is shown. When the movement area index is less than a predetermined threshold, the electronic device 1000 identifies the fire candidate area as a fire suspicious area and increases the number of fire suspicious counts by 1. You can initialize the count number.

In more detail, the electronic device 1000 may determine an average of R, G, and B values of pixel values in the frame image as a pixel value (Pixel_value). The electronic device 1000 may determine a difference between a pixel value and a pixel value at a corresponding position in a reference frame as a pixel value variance.

The electronic device 1000 may identify pixels having a pixel value variation for each frame equal to or greater than a predetermined threshold as moving pixels. The electronic device 1000 may determine the motion area index by dividing the standard deviation of the number of motion pixels identified in each of the plurality of frame images by the average value.

Meanwhile, when the variation of the pixel value for each frame is equal to or greater than a predetermined threshold, the electronic device 1000 may increase the moving object index for the corresponding pixel by 1. After increasing the moving object index by 1, the electronic device 1000 may update a pixel value (Background) of a corresponding position in a reference frame.

15 is a diagram for describing a process in which an electronic device determines a moving object index, according to an embodiment.

Referring to the figure 1212 of FIG. 15 , a process in which the electronic device determines the moving object index for each pixel is illustrated. For example, the electronic device 1000 may identify the number of times that a variance value of specific pixels at a corresponding position in the plurality of frame images is identified as being equal to or greater than a predetermined threshold. For example, it is assumed that the electronic device 1000 determines the moving object index in units of 30 frames.

For example, the electronic device 1000 may track the disparity value of the pixel at the (1,1) position for 1 to 30 frames. For example, the electronic device 1000 may determine the number of times the disparity value of the pixel at the (1,1) position is identified as being greater than or equal to the threshold value during 1 to 30 frames. The electronic device 1000 determines the moving object (1,1) to be high when the number of times the disparity value of the pixel at the position (1,1) is identified as being greater than or equal to the threshold for 1 to 30 frames is greater than or equal to a predetermined threshold. can However, the electronic device 1000 may determine the moving object(1,1) to be low when the number of times the disparity value of the pixel at the (1,1) position is identified as greater than or equal to the threshold during 1 to 30 frames is less than a predetermined threshold. have.

According to the above-described method, the electronic device 1000 may determine the number of times a disparity value per pixel is equal to or greater than a threshold for all pixels in a frame, and then determine a moving object index for pixels per frame. According to an embodiment, the electronic device 1000 may further include, in the moving object index, position information of pixels whose disparity value for each pixel at a corresponding position is equal to or greater than a threshold for 30 frames during 30 frames.

According to an embodiment, when the moving object index is determined, the electronic device 1000 sets the moving object index in the last frame used to determine the moving object index (eg, the last frame is the 30th frame when it is determined in units of 30 frames). Positions of pixels determined to be high may be identified. The electronic device 1000 may determine the RGB data average value of pixels whose moving object index is determined to be high. The electronic device 1000 may increase the number of suspected fire counts when average values of RGB data of pixels for which the moving object index is determined to be high are greater than a predetermined threshold. However, when at least one of the RGB data average values of pixels whose moving object index is determined to be high is not greater than a predetermined threshold, the electronic device 1000 may initialize the fire suspicious count.

Also, according to an embodiment, the electronic device 1000 sets a minimum distance (eg, a coordinate value) between positions of pixels having a high moving object index and positions of pixels having a saturation value of each pixel of a frame equal to or greater than a predetermined threshold. difference) and its average value can be determined. When the minimum distance between the positions of pixels having a moving object index of high and the positions of pixels having a saturation value of each pixel of the frame equal to or greater than a predetermined threshold is less than a predetermined threshold, the electronic device 1000 calculates the number of suspected fire counts. can be initialized. However, when the minimum distance between the positions of pixels having a moving object index of high and the positions of pixels having a saturation value of each pixel of the frame equal to or greater than a predetermined threshold is greater than a predetermined threshold, the electronic device 1000 may suspect a fire. The number of counts can be increased.

Also, according to an embodiment, the electronic device 1000 sets the minimum distance between positions of pixels having a high moving object index in the current frame and pixels having a high moving object index in a frame before a predetermined period (eg, 30 frames). The sum of the distances and their average can be determined. The electronic device 1000 fires when the minimum distance between the positions of pixels having a high moving object index in the current frame and the positions of pixels having a high moving object index in a frame before a predetermined period (eg, 30 frames) is less than a predetermined threshold. You can reset the number of suspicious counts. However, in the electronic device 1000, the minimum distance between the positions of pixels having a high moving object index in the current frame and the positions of pixels having a high moving object index in a frame before a predetermined period (eg, 30 frames) is greater than a predetermined threshold. If it is large, the number of suspicious fire counts may be increased.

16 is a flowchart illustrating a process in which an electronic device determines whether to identify a primary fire based on a result of detecting a flame, according to an embodiment.

The mark 1701 may indicate an image input standby state. In S1702, the electronic device 1000 converts RGB data in the image into HSL data. In S1706 , the electronic device 1000 extracts a fire candidate area (eg, a fire color area) from the image. In S1708 , the electronic device 1000 may determine a periodicity index from the fire candidate area.

In S1710 , the electronic device 1000 determines a movement area of the fire candidate area. In S1712 , the electronic device 1000 determines the width of the movement area in the fire candidate area for each frame, and determines the degree of change in the area of the movement area. The electronic device 1000 may determine the motion area index through S1710 and S1712. Also, according to an embodiment, the electronic device 1000 may further determine the moving object index for each pixel.

In S1714 , the electronic device 1000 determines the zero cross count index by counting the number of pixels crossing 0 for each pixel after performing the temporal wavelet in the fire candidate area. In S1716, the electronic device 1000 determines the high-pass filter area index by checking the area of a non-zero area by performing temporal wavelet transformation in the fire candidate area for each frame.

In S1718, the electronic device 1000 identifies whether the number of frame numbers has reached 30. For example, when the number of frame numbers does not reach 30, the electronic device 1000 may enter the fire suspicious count initialization state 1720 . However, when the number of frame numbers is 30, the electronic device 1000 identifies whether the number of times that a disparity value for each pixel is identified to be greater than or equal to a predetermined threshold during 30 frames is equal to or greater than a predetermined threshold. In S1724 , when the number of times the disparity value for each pixel is identified to be equal to or greater than the predetermined threshold during 30 frames is equal to or greater than the predetermined threshold, the electronic device 1000 determines the moving object index of the corresponding pixel as HIGH. However, in S1726 , when the number of times that the disparity value for each pixel is identified as greater than or equal to a predetermined threshold during 30 frames is less than the predetermined threshold, the electronic device 1000 determines the moving object index of the corresponding pixel as LOW.

In S1728 , when the moving object index is LOW, the electronic device 1000 enters S1736 to initialize the suspected fire count and enters the suspected fire count initialization state 1720 . However, in S1730 , when the moving object index is identified as HIGH, the electronic device 1000 may obtain an average of RGB data of pixels whose moving object index is identified as HIGH. In S1732, when the average value of each RGB data is greater than a predetermined threshold value, the electronic device 1000 enters the connection step 1734, but when at least one of the average values of the RGB data is not greater than the predetermined threshold value, The fire suspicious count may be initialized by entering step S1736.

In S1704 , the electronic device 1000 may determine a minimum distance between positions of pixels having a moving object index determined to be HIGH and positions of pixels having a saturation value of each pixel equal to or greater than a predetermined threshold. In S1742, when the minimum distance determined in S1704 is smaller than a predetermined threshold, the electronic device 1000 initializes the fire suspicious count. However, when the minimum distance is greater than the predetermined threshold, the electronic device 1000 may enter operation S1748.

In S1748, when the zero cross count index is greater than a predetermined threshold, the electronic device 1000 enters step S1750, but when the zero cross count index is less than a predetermined threshold, enters step S1736 to initialize the fire suspicious count. have.

In S1750 , the electronic device 1000 may determine the high pass filter area index by performing the process described above with reference to FIG. 12 . In S1752, when the determined high-pass filter area index is greater than a predetermined threshold, the electronic device 1000 enters step S1736 to initialize the number of suspected fire counts. However, when the determined high-pass filter area index is less than a predetermined threshold, the electronic device 1000 may enter step S1754.

In S1754, the electronic device 1000 may determine the periodicity index by checking the periodicity for each frame. In S1756 , when the periodicity index is smaller than a predetermined threshold, the electronic device 1000 may enter step S1736 to initialize the number of suspected fire counts. However, when the periodicity index is greater than a predetermined threshold, the electronic device 1000 may enter step S1758.

In S1758 , the electronic device 1000 determines the motion region index by calculating the area average and area change amount of the motion regions for each frame. In S1760 , when the motion region index is greater than a predetermined threshold, the electronic device 1000 may enter S1764 to increase the motion region variation count. In S1750 , when the movement area index is smaller than a predetermined threshold, the electronic device 1000 may enter S1766 to increase the number of suspected fire counts.

In S1768, when the movement area variation count is greater than a predetermined threshold, the electronic device 1000 enters step S1736 to initialize the fire suspicious count. However, in S1768, when the movement area variation count is less than a predetermined threshold or in S1772, when the increased number of suspicious fire counts is not greater than 6, the electronic device 1000 may enter the suspected fire count initialization state 1720. . However, in S1772, when the number of suspected fire counts is greater than 6, the electronic device 1000 may enter S1774 and output fire alarm information. After outputting the fire alarm information, the electronic device 1000 may enter the fire suspicious count initialization state 1720 again.

Also, although not shown in FIG. 16 , the electronic device 1000 may determine a smoke index regarding whether a plurality of optical images satisfy a smoke detection condition. For example, the electronic device 1000 may identify the motion vector by tracking image regions including pixels representing the gray color value in a plurality of optical images having adjacent frame numbers. When the motion vectors are identified, the electronic device 1000 may obtain an output value of the low-pass filter by applying the low-pass filter to the background image among the plurality of optical images. The electronic device 1000 may determine the smoke index regarding the correlation between the output value of the low-pass filter and the output value of the low-pass filter with respect to the background image when a preset smoke is generated.

When the smoke index is smaller than a predetermined threshold, the electronic device 1000 may initialize the number of times that the fire candidate area is identified as the fire suspicious area by entering step S1736. However, the electronic device 1000 determines that the above-described zero cross count index, high-pass filter area index, periodicity index, moving area index, moving object index, and smoke index are each index condition (each index that prevents the fire suspicious area count from being initialized). specific condition), by entering S1768, the number of fire suspicion counts for the fire candidate area may be increased by one.

That is, in the electronic device 1000 according to the present disclosure, as at least one index related to fire, the determined zero cross count index, the high-pass filter area index, the periodicity index, the moving area index, the moving object index, or the smoke index are selected. It is identified whether the index condition for at least one is satisfied, and when even one of the at least one fire indices does not satisfy the index condition, the number of times that the fire candidate area is identified as the fire suspicious area (eg, the number of fire suspicious counts) can be initialized. However, when the predetermined index conditions for the at least one fire indices are all satisfied, the electronic device 1000 increases the number of times a fire candidate area is identified as a fire suspicious area, and the increased number of times is equal to or greater than a preset threshold. (eg 6), the location of the fire candidate area may be determined as the location of the optical fire identification area.

Also, according to an embodiment, the electronic device 1000 may identify the position of the optical fire identification area according to a predetermined period. For example, the electronic device 1000 determines the at least one fire index for each of a plurality of frame images or a frame according to a predetermined frame period among the plurality of frame images, and sets a preset fire index based on the fire index. When the fire candidate area is counted as the fire suspicious area more than the number of times, the fire candidate area may be determined as the location of the optical fire identification area.

17 is a block diagram of an electronic device according to an embodiment.

18 is a block diagram of an electronic device according to another embodiment.

18 , the electronic device 1000 according to an embodiment may include at least one processor 1300 and a memory 1700 . However, not all illustrated components are essential components. The electronic device 1000 may be implemented by more components than the illustrated components, or the electronic device 1000 may be implemented by fewer than the illustrated components.

For example, as shown in FIG. 18 , the electronic device 1000 includes a user input interface 1100 , an output unit 1200 , a sensing unit 1400 , and a network interface in addition to the processor 1300 and the memory 1700 . 1500 , an A/V input unit 1600 , and a memory 1700 may be further included.

The user input interface 1100 means a means for a user to input data for controlling the electronic device 1000 . For example, the user input interface 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, red There may be a mechanical tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

The user input interface 1100 may obtain other user inputs for controlling the electronic device 1000 . Also, the user input interface 1100 may receive a user input of touching a predetermined audio control panel.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . have.

The display unit 1210 includes a screen for displaying and outputting information processed by the electronic device 1000 . Also, the screen may display an image. For example, the display unit 1210 may display at least one of disaster notification information, earthquake notification information, fire notification information, and a news screen.

The sound output unit 1220 outputs audio data received from the network interface 1500 or stored in the memory 1700 . Also, the sound output unit 1220 may output a predetermined warning sound along with the output of disaster notification information. Also, the sound output unit 1220 outputs a sound signal related to a function (eg, a call signal reception sound, a message reception sound, and a notification sound) performed by the electronic device 1000 .

The processor 1300 typically controls the overall operation of the electronic device 1000 . For example, the processor 1300 executes programs stored in the memory 1700 , so that the user input interface 1100 , the output unit 1200 , the sensing unit 1400 , the network interface 1500 , and the A/V input unit (1600) and the like can be controlled in general. In addition, the processor 1300 may execute the programs stored in the memory 1700 to perform the functions of the electronic device 1000 illustrated in FIGS. 1 to 16 . According to an embodiment, the electronic device 1000 may include a first processor to apply a flame or smoke detection algorithm to the optical image data and the thermal image data, and a second processor to apply an artificial intelligence algorithm to the optical image. have.

According to an embodiment, the processor 1300 obtains a thermal image by monitoring the area to be monitored by the at least one processor executing the one or more instructions, and acquires an optical image by monitoring the area to be monitored, , based on whether the position of the thermal image fire identification region indicating a temperature in a preset critical range in the thermal image and the position of the optical fire identification region determined according to the flame or smoke detection condition in the optical image match, the Obtaining the primary fire identification result for the monitoring target area, and inputting the optical image into the artificial intelligence model, to obtain the secondary fire identification result on whether a fire has occurred in the monitoring target area from the artificial intelligence model and, based on at least one of the obtained primary fire identification result and secondary fire identification result, it is possible to determine whether a fire occurs in the monitoring target area.

According to an embodiment, the processor 1300 may include a plurality of processors, and may perform a smoke or flame detection algorithm and an artificial intelligence algorithm using different processors.

According to an embodiment, the processor 1300 may acquire a plurality of thermal images and optical images having a preset frame interval through a thermal imaging camera and an optical camera device that photograph the monitoring target area.

According to an embodiment, the processor 1300 may obtain 3-axis acceleration information from a 3-axis acceleration sensor among the at least one sensor, and acquire geomagnetic information from a geomagnetic sensor among the at least one sensor.

According to an embodiment, the processor 1300 identifies a geomagnetic direction at a location where the electronic device is installed based on the geomagnetic information, identifies a 3-axis acceleration direction based on the 3-axis acceleration information, and identifies the 3-axis acceleration direction. matches the identified geomagnetic direction, and determines the maximum ground acceleration by monitoring the triaxial acceleration magnitude based on the triaxial acceleration direction matched to the geomagnetic direction, and the triaxial acceleration direction matched to the geomagnetic direction and whether an earthquake occurs based on the determined maximum ground acceleration may be determined.

According to an embodiment, the processor 1300 determines the intensity based on the maximum ground acceleration, and when the determined intensity is greater than or equal to a preset threshold intensity, based on the three-axis acceleration direction matched to the geomagnetic direction, the earthquake occurred it can be decided that According to another embodiment, the processor 1300 according to the present disclosure increases the seismic index by 1 when it is identified as the determined maximum ground acceleration greater than or equal to a predetermined threshold acceleration in the 3-axis acceleration direction matched to the geomagnetic direction, If an increased seismic index is identified above a threshold count, it can be determined that an earthquake has occurred. According to an embodiment, the processor 1300 determines the intensity based on the maximum ground acceleration, and when the determined intensity is greater than or equal to a preset threshold intensity, based on the three-axis acceleration direction matched to the geomagnetic direction, the earthquake occurred it can be decided that

According to an embodiment, when it is determined that the earthquake has occurred, the processor 1300 determines a PS time based on the triaxial acceleration information, determines an epicenter based on the determined PS time, and communicates with the electronic device. Acquire epicenter information from two or more other connected electronic devices, determine epicenter information where an earthquake occurs based on the determined epicenter and the epicenter information obtained from the other electronic devices, and determine the determined epicenter information and information on the determined seismic intensity may be output as the notification information.

The sensing unit 1400 may detect a state of the electronic device 1000 or a state around the electronic device 1000 , and transmit the sensed information to the processor 1300 . The sensing unit 1400 generates some of the specification information of the electronic apparatus 1000, the state information of the electronic apparatus 1000, the surrounding environment information of the electronic apparatus 1000, the user's state information, and the user's device use history information. can be used

The sensing unit 1400 includes a magnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460, barometric pressure sensor 1470, proximity sensor 1480, image sensor, thermal image sensor, 3-axis acceleration sensor or RGB sensor (illuminance sensor) 1490 may include at least one However, the present invention is not limited thereto. Since a function of each sensor can be intuitively inferred from a person skilled in the art from the name, a detailed description thereof will be omitted.

The network interface 1500 may include one or more components that allow the electronic device 1000 to communicate with another device (not shown) and the server 2000 . The other device (not shown) may be a computing device such as the electronic device 1000 or a sensing device, but is not limited thereto. For example, the network interface 1500 may include a short-range communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit 1510, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, and the like, but is not limited thereto.

The mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment, the electronic device 1000 may not include the broadcast receiver 1530 . Also, the network interface 1500 may acquire a target object image or a reference object image from an external device including a camera apparatus or a server. According to an embodiment, the network interface 1500 may transmit disaster alarm information determined by the electronic device to a server or an external device.

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like. The camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a photographing mode. The image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone 1620 receives an external sound signal and processes it as electrical voice data. For example, the microphone 1620 may receive an acoustic signal from an external device or a user. The microphone 1620 may receive a user's voice input. The microphone 1620 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal.

The memory 1700 may store a program for processing and controlling the processor 1300 , and may also store data input to or output from the electronic device 1000 . Also, in order for the electronic device 1000 to output disaster alarm information, the memory 1700 may store a series of methods for determining whether a fire or an earthquake has occurred in the form of instructions.

The memory 1700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , and the like. .

The UI module 1710 may provide a specialized UI, GUI, or the like that interworks with the electronic device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 1300 . The touch screen module 1720 according to some embodiments may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

The notification module 1730 may generate a signal for notifying the occurrence of an event in the electronic device 1000 . Examples of events generated in the electronic device 1000 include call signal reception, message reception, key signal input, schedule notification, and the like. The notification module 1730 may output a notification signal in the form of a video signal through the display unit 1210 , may output a notification signal in the form of an audio signal through the sound output unit 1220 , and the vibration motor 1230 . It is also possible to output a notification signal in the form of a vibration signal through

19 is a block diagram of a server according to an embodiment.

According to an embodiment, the server 2000 may include a processor 2300 , a network interface 2500 , and a database 2700 . According to an embodiment, the configuration of the processor 2300 , the network interface 2500 , and the database 2700 in the server 2000 is the configuration of the processor 1300 , the network interface 1500 and the memory 1700 in the electronic device 1000 . Each configuration may correspond to each other.

According to an embodiment, the server 2000 may transmit images acquired from the CCTV device to the electronic device 1000 or transmit seismic wave information. In addition, the server 2000 provides disaster information determined by the electronic device 1000 (eg, whether a fire occurs, whether an earthquake occurs, fire alarm information, earthquake alarm information, a fire occurrence scale, a fire occurrence location, an earthquake occurrence scale, an earthquake occurrence location, etc.) information) and may generate visual or auditory disaster content based on the acquired disaster information. The server 2000 may transmit the generated disaster content back to the electronic device 1000 , or may transmit it to another electronic device or a server device connected to the electronic device.

20 is a diagram for explaining a process of detecting a disaster by interworking between an electronic device and a server according to an embodiment.

The electronic device 1000 and the server 2000 according to the present disclosure may perform at least a part of a method of detecting a disaster by interworking with each other based on an artificial intelligence model. However, according to another embodiment, the artificial intelligence model stored in the server 2000 or instructions for the artificial intelligence model are stored in the electronic device, so that the electronic device 1000 alone uses the artificial intelligence model as described above in this specification. Of course, it is also possible to carry out a method to detect a disaster. Also, according to an embodiment, the electronic device 1000 may include a plurality of processors in the electronic device 1000 and effectively detect a disaster by using a separate processor for performing an artificial intelligence algorithm.

For example, when a region satisfying a temperature of a preset threshold range in the thermal image is identified using the first processor, the electronic device 1000 may determine the location of the thermal image fire identification region for the region. . In addition, the electronic device 1000 may determine whether an optical fire identification region in the optical image is derived by applying the flame detection algorithm shown in FIG. 16 or a smoke detection algorithm to be described later in the optical image using the first processor. Also, the electronic device 1000 may determine whether smoke or flame is detected from the optical image by applying the artificial intelligence model to the optical image using the second processor.

According to an embodiment, the artificial intelligence model used by the electronic device 1000 may include a plurality of neural network layers. Each of the plurality of neural network layers has a plurality of weight values, and a neural network operation is performed through an operation between an operation result of a previous layer and a plurality of weights. The plurality of weights of the plurality of neural network layers may be optimized by the learning result of the artificial intelligence model. For example, a plurality of weights may be updated so that a loss value or a cost value obtained from the artificial intelligence model during the learning process is reduced or minimized.

According to an embodiment, the artificial intelligence model used by the electronic device 1000 may include a deep neural network (DNN), for example, a convolutional neural network (CNN), a deep neural network (DNN), Recurrent Neural Network (RNN), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, etc., but are not limited to the above examples does not Also, according to an embodiment, the artificial intelligence model used by the electronic device 1000 may include at least one of YOLO and DARKNET.

The operation of the electronic device using the above-described first and second processors may be performed by the electronic device alone, or some processor functions may also be performed by the server 2000 . For example, as shown in FIG. 20 , the electronic device 1000 may detect a disaster quickly and effectively by interworking with the server 2000 and performing some of the methods of detecting a disaster using an artificial intelligence model. have.

For example, in S2002 , the electronic device 1000 may acquire an image of the monitoring target area by photographing the monitoring target area image. The electronic device 1000 may acquire a plurality of thermal image images and optical images with a predetermined frame interval. In S2004 , the electronic device 1000 may transmit a plurality of thermal image and optical images of the obtained monitoring target area to the server 2000 .

According to another embodiment, the electronic device 1000 may transmit RTSP Video Data to the server 2000 . In S2106, the server 2000 may pre-process the image received from the electronic device. For example, the server 2000 may convert an image or an image obtained from the electronic device 1000 into a format optimized for the artificial intelligence model by preprocessing the image. More specifically, the server 2000 may pre-process an image or an image obtained from the electronic device 1000 to convert it into a format optimized for each artificial intelligence model learned for each image type.

In S2008, the server 2000 checks whether the optical fire identification area is determined by applying a flame or smoke detection algorithm to the optical image obtained from the electronic device 1000, and the thermal image of a preset critical range temperature from the thermal image image. After confirming whether the image fire identification area is determined, it may be checked whether the positions of the optical fire identification area and the thermal image fire identification area match. Also, the server 2000 may obtain information on whether or not a fire has occurred from the artificial intelligence model by applying the artificial intelligence model to the optical image.

The server 2000 may determine the primary fire identification result based on whether the position of the optical fire identification region matches the position of the thermal image fire identification region, and may determine the secondary fire identification result from the artificial intelligence model. The server 2000 may finally determine whether a fire has occurred in the monitoring target area based on at least one of the primary fire identification result and the secondary fire identification result. In S2012 , when it is identified that a fire has occurred in the monitoring target area, the server 2000 may transmit notification information to the electronic device 1000 . According to an embodiment, the server 2000 may further transmit information on the cause of the fire and the type of fire to the electronic device 1000 in addition to the notification information.

In S2014 , the electronic device 1000 may store notification information received from the server 2000 . According to another example, the electronic device 1000 may further store information on the cause of the fire in addition to the notification information. In S2016, the electronic device 1000 may output stored notification information.

21 is a diagram for explaining a process of performing a method for detecting a disaster by interworking between an electronic device and a server according to another embodiment.

Referring to FIG. 21 , the electronic device 1000 may correspond to a FireWatch Camera. The electronic device 1000 may acquire an input video 2210 of a monitoring target area using a camera. In S2102 , the electronic device 1000 may determine first result information regarding whether a fire has occurred by performing the method of determining whether a fire has occurred with reference to FIGS. 1 to 18 . In S2104 , the electronic device 1000 may video-stream image information obtained by photographing the monitoring target area. In S2106 , the electronic device 1000 may obtain from the server 2000 external input event information regarding whether a fire has occurred, which the server has determined using the artificial intelligence model, as the second result information. In S2108, the electronic device 1000 may generate an alert based on the first result information and the second result information. For example, when the first result information indicates that the number of times that a fire candidate area is identified as a fire suspicious area is equal to or greater than a threshold, and the second result information is identified as equal to or greater than a predetermined threshold probability value, an alarm is issued may cause

The electronic device 1000 according to the present disclosure may transmit input video data 2210 obtained by photographing a monitoring target area to the server 2000 in the form of RTSP video data 2220, and the server 2000 may transmit the input video data 2210 to the electronic device ( 1000), an alarm event determined by the server 2000 may be generated by inputting the RTSP video data acquired in the artificial intelligence model.

For example, in S1210 , the server 2000 may generate an image or an image having a format suitable for the type of the artificial intelligence model by frame-processing the RTSP video data 2220 obtained by the electronic device 1000 . In S2112, the server 2000 may input the preprocessed image or image to the trained deep learning neural network model. In S2114, the server 2000 may infer whether a fire has occurred in an image or an image based on the output value of the deep learning neural network model. In S2116, the server 2000 generates alarm event data when the result value inferred based on the output value of the neural network model indicates a situation in which a fire has occurred or indicates a fire occurrence probability greater than or equal to a predetermined threshold probability, and the generated alarm Event data 2230 may be transmitted to the electronic device 1000 .

22 is a diagram for describing a process of generating, by an electronic device, learning data for learning an artificial intelligence model, according to an embodiment.

The electronic device 1000 according to the present disclosure may learn an artificial intelligence model that outputs a fire occurrence probability value from an optical image, a thermal image, or an image. Also, according to an embodiment, the electronic device 1000 may train an artificial intelligence model for each type of image or image. Also, the electronic device 1000 may generate training data for learning the artificial intelligence model for each predetermined type.

For example, the electronic device 1000 may acquire the training video data 2302 for learning the artificial intelligence model by photographing the monitoring target area. According to another embodiment, the electronic device 1000 may acquire the training video data 2302 for learning the AI model from the server 2000 or a pre-stored database. The electronic device 1000 may generate the training data 2340 for each predetermined image type by preprocessing the training video data 2302 .

For example, the electronic device 1000 pre-processes the training video data 2302 to obtain a high-pass filter image 2312 , a color (RGB) image 2314 , a black-and-white image 2316 , an edge image 2318 , and a motion. At least one of the image 2320 and the brightness control image 2322 may be generated. According to an embodiment, the motion image 2320 may be an image or an image in which vectors and motion amounts in the image are expressed in grayscale. According to an embodiment, the brightness adjustment image 2322 may be an image whose image brightness is adjusted. The electronic device 1000 may generate combination learning data by combining each type of training data generated by preprocessing the training video data 2302 or training data generated for each type, and may generate training data or combination learning generated for each type. An artificial intelligence model can be trained based on the data. The electronic device 1000 according to the present disclosure may train the artificial intelligence model by modifying and updating weights of nodes and layers in the artificial intelligence model based on the learning data.

According to an embodiment, the electronic device 1000 may train an artificial intelligence model by using video data selected by a person's judgment among the training video data 2302 . According to another embodiment, the electronic device 1000 determines whether a fire has occurred based on the algorithm for determining whether a fire has occurred with reference to FIGS. 1 to 18 , and as a result, an image or image identified as having occurred even though a fire has not occurred. can be removed from the training video data. The electronic device 1000 removes an image or an image for which a fire occurrence is erroneously determined in the fire generation algorithm described in FIGS. 1 to 18 from the training data and trains the artificial intelligence model based on the remaining training data, thereby making the artificial intelligence model. can further improve the accuracy of

23 is a diagram for describing a process in which an electronic device generates an alarm event using an artificial intelligence model, according to an exemplary embodiment.

A process of generating an alarm event by acquiring a real-time image 2402 by the electronic device 1000 by capturing a region to be monitored according to an embodiment, pre-processing the real-time image, and then inputting the pre-processed image generated into the neural network model. decide to do In S2340 , the electronic device 1000 may generate a pre-processed image generated by pre-processing the real-time image 2402 . For example, the electronic device 1000 pre-processes the real-time image 2402 to obtain a high-pass filter image 2412 , an RGB image 2414 , a black-and-white image 2416 , an edge image 2418 , a motion image 2428 , or A brightness control image 2422 may be generated. In S2342 , the electronic device 1000 may monitor the training result by inputting the preprocessed image to the deep learning neural network model 2430 .

In S2344 , the electronic device 1000 may determine whether an average accuracy of a result value obtained by accumulating an inference result of an artificial intelligence model (eg, a neural network model) for a preset time is equal to or greater than a predetermined threshold value. According to an embodiment, the electronic device 1000 may determine whether the average accuracy is equal to or greater than a threshold by accumulating the inference results of the artificial intelligence model for 1 to 10 seconds. In S2346 , the electronic device 1000 may generate an alarm event when it is identified that the accumulated average accuracy is equal to or greater than the threshold value.

24 is a diagram for describing in detail a method for an electronic device to detect smoke according to a smoke detection condition, according to an exemplary embodiment.

In S2402, the electronic device 1000 may sequence the optical images. For example, the electronic device 1000 may determine a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images. In S2404 , the electronic device 1000 may identify motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images.

In S2406 , the electronic device 1000 may cluster the identified motion regions. In S2408 , the electronic device 1000 may extract pixels representing a gray color value from among pixels in the identified motion regions. In S2410 , the electronic device 1000 may identify motion vectors by tracking image regions including pixels representing the gray color value in a plurality of optical images having adjacent frame numbers.

In S24142 , when the motion vectors are identified, the electronic device 1000 may obtain an output value of the low-pass filter by applying the low-pass filter to a background image among a plurality of optical images. The electronic device 1000 may determine the smoke index regarding the correlation between the output value of the low-pass filter and the output value of the low-pass filter with respect to the background image when a preset smoke is generated.

In S2414 , the electronic device 1000 may generate a result of performing the above-described process as training data, and train an artificial intelligence model based on the generated training data. In S2416 , the electronic device 1000 may verify whether the result output from the learned artificial intelligence model is the actual smoke. In S2418 , when it is verified that the result output from the artificial intelligence model is the actual smoke, the electronic device 1000 may generate alarm information and output the generated alarm information. The electronic device 1000 according to the present disclosure may perform the smoke detection algorithm described in FIG. 24 using an artificial intelligence model, but may determine whether to detect smoke directly from an image through actual image processing using an embedded module.

25 is a diagram illustrating a smoke detection result detected by an electronic device according to an exemplary embodiment.

Referring to figure 2510 of FIG. 25 , a result of detecting smoke in the optical image by the electronic device 1000 is shown. On the screen of the electronic device, the electronic device 1000 may partially cluster moving pixel regions and extract pixels representing gray color values from among the clustering regions. The electronic device 1000 may determine the image area 2512 including pixels representing gray color values as the smoke detection area.

A method for detecting a disaster by an electronic device according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software.

Also, there may be provided a computer program device including a recording medium in which a program for causing an electronic device to perform a method for detecting a disaster is stored.

Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

Some embodiments may also be implemented in the form of a recording medium containing instructions executable by a computer, such as program modules executed by a computer. Computer-readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer-readable media may include both computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media typically includes computer readable instructions, data structures, other data in modulated data signals, such as program modules, or other transport mechanisms, and includes any information delivery media. Also, some embodiments may be implemented as a computer program or computer program product comprising instructions executable by a computer, such as a computer program executed by a computer.

Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto. belong to the scope of the right.

Claims (17)

A method for an electronic device to detect a disaster,
acquiring a thermal image by monitoring the area to be monitored;
acquiring an optical image by monitoring the area to be monitored;
Based on whether the position of the thermal image fire identification area indicating the temperature in the preset critical range in the thermal image and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the monitoring obtaining a primary fire identification result for the target area;
obtaining a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model by inputting the optical image into an artificial intelligence model; and
determining whether a fire has occurred in the monitoring target area based on the obtained primary fire identification result and secondary fire identification result; including,
The step of obtaining the first fire identification result is
determining at least one fire index relating to the flame or the smoke detection condition, which is determined for a fire candidate area in the optical image; and
identifying the location of the fire candidate area as the location of the optical fire identification area when the number of times the fire candidate area is identified as the fire suspicious area is equal to or greater than a preset threshold based on the at least one fire index; including,
Acquiring the optical image comprises:
acquiring a plurality of optical images having a preset frame interval; includes,
Determining the at least one fire index comprises:
determining a smoke index as to whether the plurality of optical images satisfy the smoke detection condition; including,
The step of determining the smoke index,
determining a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
identifying motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
clustering the identified motion regions;
extracting pixels representing a gray color value from among the pixels in the clustered motion regions;
identifying motion vectors by tracking image regions comprising pixels representing the gray color value in a plurality of optical images having the contiguous frame number;
when the motion vectors are identified, obtaining an output value of the low-pass filter by applying a low-pass filter to a background image among the plurality of optical images; and
determining a smoke index with respect to the correlation between the obtained output value of the low-pass filter and the low-pass filter output value with respect to a background image when a preset smoke is generated; A method comprising The method of claim 1, wherein the method
acquiring seismic wave information by sensing seismic waves at a location where the electronic device is installed using at least one sensor;
determining at least one seismic index based on the seismic wave information;
determining whether an earthquake occurs based on the determined at least one earthquake index; and
outputting notification information according to whether the fire or the earthquake has occurred; A method further comprising: According to claim 1, wherein the step of obtaining the first fire identification result
identifying a location of a thermal image fire identification region with respect to some image regions satisfying a temperature of the preset threshold range based on a pixel value of the thermal image; and
obtaining the primary fire identification result based on whether the position of the identified thermal image fire identification area matches the position of the optical fire identification area; A method comprising The method of claim 3, wherein obtaining the primary fire identification result comprises:
obtaining the primary fire identification result regarding a result that a fire has occurred in the monitoring target area when it is identified that the position of the thermal image fire identification area matches the position of the optical fire identification area; and
If the location of the thermal image fire identification area or the optical fire identification area is not identified, or it is identified that the location of the thermal image fire identification area and the location of the optical fire identification area do not match, the monitoring target area obtaining the primary fire identification result with respect to the result that no fire has occurred; A method comprising 4. The method of claim 3, wherein identifying the location of the optical fire identification area comprises:
converting RGB data of pixels in the optical image into HSL data;
determining a fire candidate area in the optical image based on the HSL data;
determining at least one fire index related to the flame or the smoke detection condition for the determined fire candidate area;
identifying the number of times the fire candidate area is identified as a fire suspicious area based on the determined at least one fire index;
identifying the location of the fire candidate area as the location of the optical fire identification area when the number of times identified is equal to or greater than a preset threshold; A method comprising The method of claim 5, wherein the determining of the fire candidate area comprises:
determining a first candidate region including pixels having all of Hue, Saturation, and Lightness values in the HSL data equal to or greater than a predetermined threshold;
identifying pixels in the HSL data in which at least one of the hue, the saturation, and the brightness is equal to or greater than the predetermined threshold;
When all of the color, saturation, and brightness values of pixels around the identified pixels are equal to or greater than the predetermined threshold value, at least one of the color, the saturation, or the brightness is greater than or equal to the predetermined threshold value, and in the first candidate region determining a second candidate area including the included pixels; and
determining the second candidate area as the fire candidate area; A method comprising 6. The method of claim 5,
Acquiring the thermal image and acquiring the optical image
acquiring a plurality of thermal image images having a preset frame interval through each of a thermal imaging camera and an optical camera device for photographing the monitoring target area, and acquiring a plurality of optical images having the preset frame interval; A method comprising 8. The method of claim 7, wherein determining the at least one fire index comprises:
determining a zero cross count index by performing high-pass filtering on a fire candidate area in a plurality of optical images having three adjacent frame numbers among the plurality of optical images;
determining a high-pass filter variable for each pixel of the fire candidate area by comparing the zero cross count index and a predetermined threshold;
determining a high-pass filter area index with respect to a change amount for each of the optical images in the number of pixels in which the high-pass filter parameter is not zero; A method comprising 9. The method of claim 8, wherein determining the at least one fire index comprises:
determining a periodicity index with respect to the periodicity of the fire candidate area in the plurality of optical images;
determining a motion area index with respect to a degree of change in the number of pixels having a large variation in a pixel value among pixels in the plurality of optical images; and
determining, in the plurality of optical images, a moving object index with respect to positions of pixels in which a disparity value of a specific pixel appears more than a predetermined threshold value; A method further comprising: 9. The method of claim 8, wherein determining the at least one fire index comprises:
determining a smoke index as to whether the plurality of optical images satisfy the smoke detection condition; A method further comprising: The method of claim 10, wherein determining the smoke index comprises:
determining a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
identifying motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
clustering the identified motion regions;
extracting pixels representing a gray color value from among the pixels in the clustered motion regions;
identifying motion vectors by tracking image regions comprising pixels representing the gray color value in a plurality of optical images having the contiguous frame number;
when the motion vectors are identified, obtaining an output value of the low-pass filter by applying a low-pass filter to a background image among the plurality of optical images; and
determining a smoke index with respect to the correlation between the obtained output value of the low-pass filter and the low-pass filter output value with respect to a background image when a preset smoke is generated; A method comprising The method of claim 11 , wherein the extracting of pixels representing the gray color value comprises:
Among the RGB data of pixels in the optical image, when the difference between the R value and the G value, the difference between the R value and the B value, and the difference between the G value and the B value is smaller than a preset threshold difference, the corresponding pixel is converted to a gray color value extracting to the pixels representing; A method comprising The method of claim 8, wherein determining the zero cross count index comprises:
Pixel values for each optical image are calculated based on weights of -0.25, 0.5, and -0.25 determined for each of the optical images having consecutive N-1 frame, N frame, and N+1 frame numbers among the plurality of optical images. performing high-pass filtering by weight summing, and determining a flicker variable for each pixel according to a result of performing high-pass filtering;
determining a high-pass filter variable value for each pixel by comparing the determined flicker variable with a predetermined threshold; and
determining the zero-cross count index based on the high-pass filter variable value for each of the plurality of optical images; A method comprising The method of claim 5, wherein obtaining the primary fire identification result comprises:
determining whether the location of the thermal fire identification area and the location of the optical fire identification area match on an image; and
determining whether the location of the thermal image fire identification area and the location on the physical space respectively indicated by the location of the optical fire identification area match; A method further comprising: According to claim 1, wherein the step of obtaining the secondary fire identification results
pre-processing the obtained optical image;
inputting the preprocessed optical image into the artificial intelligence model; and
obtaining the secondary fire identification result including a probability value regarding whether a fire has occurred in the monitoring target area from the artificial intelligence model; A method comprising In an electronic device for detecting a disaster,
Thermal imaging camera;
optical camera;
network interface;
a memory storing one or more instructions; and
a plurality of processors executing the one or more instructions; including,
The plurality of processors by executing the one or more instructions,
To acquire a thermal image by monitoring the area to be monitored,
Obtaining an optical image by monitoring the area to be monitored,
Based on whether the position of the thermal image fire identification area indicating the temperature in the preset critical range in the thermal image and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the monitoring Obtain the primary fire identification result for the target area,
By inputting the optical image into the artificial intelligence model, a secondary fire identification result regarding whether a fire has occurred in the area to be monitored is obtained from the artificial intelligence model,
Determining whether or not a fire occurs in the monitoring target area based on the obtained primary fire identification result and secondary fire identification result,
The plurality of processors
determine at least one fire index relating to the flame or the smoke detection condition, which is determined for a fire candidate area in the optical image;
Based on the at least one fire index, when the number of times that the fire candidate area is identified as a fire suspicious area is equal to or greater than a preset threshold, the location of the fire candidate area is identified as the location of the optical fire identification area,
The plurality of processors
Obtaining a plurality of optical images having a preset frame interval,
determine a smoke index with respect to whether the plurality of optical images satisfy the smoke detection condition;
The plurality of processors
determine a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
identifying motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
clustering the identified motion regions;
extracting pixels representing a gray color value from among the pixels in the clustered motion regions;
identify motion vectors by tracking image regions comprising pixels representing the gray color value in a plurality of optical images having the adjacent frame number;
When the motion vectors are identified, an output value of the low-pass filter is obtained by applying a low-pass filter to a background image among the plurality of optical images;
and determining a smoke index regarding a correlation between the obtained output value of the low-pass filter and a low-pass filter output value with respect to a background image when a preset smoke is generated. A method for an electronic device to detect a disaster,
acquiring a thermal image by monitoring the area to be monitored;
acquiring an optical image by monitoring the area to be monitored;
Based on whether the position of the thermal image fire identification area indicating the temperature in the preset critical range in the thermal image and the position of the optical fire identification area determined according to the flame or smoke detection condition in the optical image match, the monitoring obtaining a primary fire identification result for the target area;
obtaining a secondary fire identification result regarding whether a fire has occurred in the area to be monitored from the artificial intelligence model by inputting the optical image into an artificial intelligence model; and
determining whether a fire has occurred in the monitoring target area based on the obtained primary fire identification result and secondary fire identification result; including,
The step of obtaining the first fire identification result is
determining at least one fire index relating to the flame or the smoke detection condition, which is determined for a fire candidate area in the optical image; and
identifying the location of the fire candidate area as the location of the optical fire identification area when the number of times the fire candidate area is identified as the fire suspicious area is equal to or greater than a preset threshold based on the at least one fire index; including,
Acquiring the optical image comprises:
acquiring a plurality of optical images having a preset frame interval; includes,
Determining the at least one fire index comprises:
determining a smoke index as to whether the plurality of optical images satisfy the smoke detection condition; including,
The step of determining the smoke index,
determining a foreground image and a background image for the plurality of optical images by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
identifying motion regions by comparing pixel values between a plurality of optical images having adjacent frame numbers among the plurality of optical images;
clustering the identified motion regions;
extracting pixels representing a gray color value from among the pixels in the clustered motion regions;
identifying motion vectors by tracking image regions comprising pixels representing the gray color value in a plurality of optical images having the contiguous frame number;
when the motion vectors are identified, obtaining an output value of the low-pass filter by applying a low-pass filter to a background image among the plurality of optical images; and
determining a smoke index with respect to the correlation between the obtained output value of the low-pass filter and the low-pass filter output value with respect to a background image when a preset smoke is generated; A computer-readable recording medium in which a program for performing the method is stored, comprising a.

Images