KR20230102532A - Fire detection system and method using time series data prediction thchnique and image classification - Google Patents

Abstract

In accordance with the present invention, a fire determination system employing a time-series data prediction technique and image classification, includes: an image data collection part collecting an image photographed by a camera; an image feature calculation part calculating fire-related feature information included in the collected image by applying an image classification technique; a sensor value collection part collecting measurement values of each sensor measuring each parameter on which a fire determination is based; a sensor value feature calculation part calculating fire-related feature information from measurement values of the sensors collected by a time series data prediction technique; and a confidence determination part calculating integrated confidence by applying a weighted value to the feature information calculated by the image feature calculation part, and the feature information calculated by the sensor value feature calculation part, and, when the calculated integrated confidence exceeds a predetermined reference value, determining the situation as a fire. Therefore, the present invention is capable of capturing various signs related to a fire from a combination of various sensors.

Description

Fire determination system and method using time series data prediction technique and image classification

The present invention relates to a system and method for determining a fire by applying a time-series data prediction technique and image classification capable of predicting a change in a sensor value and determining a fire based on various information.

A fire detector used for normal fire management detects a fire and sends a fire alarm directly or indirectly to a manager or fire control center. At this time, according to the fire signal, the sprinkler, which is an indoor and outdoor fire extinguishing facility, is automatically controlled to extinguish the fire.

Although these general fire detectors have multiple receivers installed inside and outside the building where firefighting facilities are installed, mutual communication is impossible between these fire detection periods, so they cannot quickly deliver fire information to building users. In particular, the first time and occurrence of fire There is a problem that the place is not delivered.

Fire departments in each region operate a 119 command room and control center, and when reporting a fire or recognizing a fire, they can recognize the location of the fire and surrounding information using GPS, GIS, etc., but a system that can detect the current fire situation in real time this isn't right

In order to improve the above-mentioned inefficiency of fire recognition, a real image taken by a real image camera and a thermal image taken through a thermal camera are mixed to detect a change in temperature even if there is no flame, temperature or smoke, and a fire occurs. A fire detection system predicting the possibility has been proposed. (KR10-1544019)

1 shows the configuration of a fire detection system according to the prior art.

In the fire detection system, a thermal image and a real image are synthesized and used, and even in a situation where a flame does not actually occur, temperature change can be detected to check latent heat, and the possibility of fire and fire can be detected in advance using this. components (① sensor unit, ② photographing unit, ③ synthesis unit) are provided.

However, since the prior art captures fire and signs of fire by relying on two types of data, a real image and a thermal image, there is a limit to information that cannot be obtained from a corresponding sensor.

On the contrary, even if the types of sensors are increased in the prior art, there is a limit to observing the amount of data change and applying the fire discrimination rule to individual sensors according to the level of change.

Republic of Korea Registration No. 10-1544019

An object of the present invention is to provide a fire determination system and method using a time-series data prediction technique and image classification capable of variably increasing the types of sensors in capturing fire signs.

An object of the present invention is to provide a fire discrimination system and method using a time-series data prediction method and image classification using a time-series data prediction method and image classification and performing fire discrimination based on a neural network.

An object of the present invention is to provide a fire detection system and method using a time-series data prediction technique and image classification that provide convenience without having to directly determine a rule for changes in sensors or measured values.

A fire determination system to which a time-series data prediction technique and image classification are applied according to an aspect of the present invention includes an image data collection unit for collecting camera-photographed images; an image feature calculation unit that calculates fire-related feature information included in the collected images by applying an image classification technique; a sensor value collection unit that collects measurement values of each sensor that measures each parameter that is a basis for determining a fire; a sensor value feature calculation unit that calculates fire-related feature information from measurement values of sensors collected using a time-series data prediction technique; Depending on the environment of the area to be monitored, the feature information calculated by the image feature calculation unit and the feature information calculated by the sensor value feature calculation unit are weighted to calculate integrated confidence, and if the calculated integrated confidence exceeds a predetermined reference value, It may include an integrated confidence determination unit that determines fire.

Here, the sensor value feature calculation unit includes: an LSTM neural network learning unit that collects measurement values of each sensor in time series and performs LSTM neural network learning; and a NN-dimensional reduction processing unit for inferring a class of measurement values of the sensor by using a result of performing the LSTM neural network learning.

Here, the LSTM neural network learning unit predicts the sensor value at time t+1 using the learned LSTM model, secures the sensor value at time t (current) and the sensor value at time t+1 (future), and obtains the current time Data and prediction data can be expressed as a single vector.

Here, the NN-dimensional reduction processing unit receives the vector output by the LSTM neural network learning unit, applies NN neural network learning as neural network learning for classifying sensor data, and classifies into a fire class or a non-fire class.

Here, the sensor value collection unit may configure a unit-sized feature vector for inputs of each sensor updated at each unit time.

Here, the image feature calculation unit may include a CNN neural network that performs neural network learning and processing for classifying image data; and an image fire determination unit for deriving a confidence value for determining a fire from the classified image.

Here, according to the version of the fire determination algorithm and learning model based on the sensor value and the image or the time point at which the unit learning of the prescribed amount is performed, the integrated confidence determination unit may assign a higher weight to the latest one.

A method for determining a fire using a time-series data prediction technique and image classification according to another aspect of the present invention includes the steps of collecting measurement values of sensors measuring each parameter that is a basis for determining a fire; Calculating fire-related characteristic information from measurement values of sensors collected using a time-series data prediction technique; Collecting camera-captured images of the fire surveillance area; Calculating fire-related feature information included in the collected images by applying an image classification technique; calculating an integrated confidence by assigning a weight according to the environment of a region to be monitored to the feature information calculated from the image and the feature information calculated from the measured value of the sensor; and determining that it is a fire when the calculated integrated confidence exceeds a predetermined reference value.

Here, calculating the fire-related feature information from the measurement values of the sensors includes predicting sensor data at time t+1 using the learned LSTM; constructing a value vector consisting of sensor data at time t and predicted data at time t+1; extracting a feature vector by passing through the learned NN model; performing dimensional reduction with a 2-dimensional or 3-dimensional feature vector; Calculating distances between central data and feature vectors of each of the fire class and the non-fire class, and assigning the distance to a nearby class; and calculating a confidence value in the case of a fire class.

Here, calculating the fire-related feature information included in the image may include calculating a feature vector using a CNN; Determining a fire class from CNN; and calculating a confidence value according to the activation function.

Here, in the step of collecting the measurement values of the sensors, a feature vector form of unit size may be configured for the input of each sensor updated at each unit time.

When the fire discrimination system and/or method using the time-series data prediction technique and image classification according to the spirit of the present invention having the above configuration is implemented, even if the type of sensor is variably increased in capturing fire signs, the amount of change in the sensor or measurement value is not affected. It has the advantage of not having to directly determine the rules for

The fire determination system and/or method of the present invention has the advantage of being able to predict the amount of change in sensor values and utilize various types of sensor values, thereby enabling fire determination based on various information. Specifically, similar to the prior art, the part of utilizing a camera image and sensor data is the same, but various signs related to fire can be captured from a combination of various sensors.

The fire determination system and/or method of the present invention not only obtains various information from various types of sensors simultaneously, but also includes information on the amount of change in sensor values through prediction of sensor values using an LSTM neural network to determine fire There are advantages to using

The fire determination system and/or method of the present invention has the advantage of being able to reflect the degree of confidence in each of determination by image and determination by sensor by setting different weight values when calculating the final confidence.

The fire discrimination system and / or method of the present invention has the advantage of being able to directly learn the relationship between a fire and each sensor and the complex relationship between a fire and the amount of change of each sensor at the same time by using deep learning rather than a rule base.

1 is a configuration diagram showing a fire detection system according to the prior art.
Figure 2 is a block diagram showing an embodiment of a fire detection system according to the spirit of the present invention.
Figure 3 is a detailed block diagram showing detailed configurations of the application technique of the fire determination system of Figure 2;
4 is a data structure diagram illustrating input sensor value data collected by the sensor value collection unit of FIG. 2;
5 is a diagram showing the learning structure of LSTM output data generated by the LSTM neural network training unit of FIG. 3;
6 is a detailed configuration diagram of a Neural Network (NN)-dimensional reduction processing unit of FIG. 3;
7 is a graph illustrating a process of deriving a vector distance from a class center point of a probability distribution.
8a and 8b are flow charts illustrating an embodiment of a fire determination method according to the spirit of the present invention.

In describing the present invention, terms such as first and second may be used to describe various components, but the components may not be limited by the terms. Terms are only for the purpose of distinguishing one element from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being connected or connected to another component, it may be directly connected or connected to the other component, but it may be understood that another component may exist in the middle. .

Terms used in this specification are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions may include plural expressions unless the context clearly dictates otherwise.

In this specification, the terms include or include are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features or numbers, It can be understood that the presence or addition of steps, operations, components, parts, or combinations thereof is not precluded.

In addition, shapes and sizes of elements in the drawings may be exaggerated for clearer description.

2 is a block diagram showing an embodiment of a fire detection system according to the spirit of the present invention.

The illustrated fire determination system includes an image data collection unit 140 for collecting images captured by cameras such as CCTV; An image feature calculator 400 that calculates fire-related feature information included in collected images by applying an image classification technique; a sensor value collection unit 120 that collects measurement values of each sensor that measures each parameter that is a basis for determining a fire; a sensor value feature calculation unit 200 that calculates fire-related feature information from measurement values of sensors collected by a time-series data prediction technique; Depending on the environment of the area to be monitored, an integrated confidence is calculated by assigning a weight to the feature information calculated by the image feature calculation unit and the feature information calculated by the sensor value feature calculation unit 200, and the calculated integrated confidence is It may include an integrated confidence (probability value) calculation and a fire determination unit 500 that determines a fire when a predetermined reference value is exceeded.

FIG. 3 is a detailed block diagram showing detailed configurations of an application technique of the fire detection system of FIG. 2 .

The sensor value feature calculation unit 200 of FIG. 2 includes an LSTM neural network learning unit 200-1 that performs LSTM neural network learning by collecting measurement values of each sensor in a time series in FIG. 3; and

It may include a NN-dimensional reduction processing unit 200-2 for inferring a class of measurement values of the sensor by using a result of performing the LSTM neural network learning.

FIG. 4 is a data structure diagram illustrating input sensor value data collected by the sensor value collection unit 120 of FIG. 2 .

The sensor value collection unit 120 is a part (temperature, CO, VOC, IR, etc.) that receives inputs from N sensors for the purpose of measuring values of different objects. For example, the input of each sensor is updated at the same unit time, and a 1XN size feature vector can be configured at every time step.

It is advantageous for the sensor value collection unit 120 to have a sufficient buffering size so that the time before point a (t-a) to the current point in time (t) is implemented as an input of the LSTM neural network to have time series characteristics.

For example, the size of the final input data is in the form of axN, and more specifically, when timestep a = 20 and the number of sensors (N) = 5, it can be a 20x5 array.

FIG. 5 illustrates LSTM output data generated by the LSTM neural network learning unit 200-1 of FIG. 3 .

5 shows the LSTM neural network learning structure, and it can be seen that LSTM neural network learning is performed using sensor data of a previous time point having time series characteristics.

As an example of the sensor value prediction process using the learned LSTM, the sensor value at time t+1 is predicted using the learned LSTM model, and the sensor value at time t (current) and the sensor value at time t+1 (future) It can be secured, and the current time data and predicted data can be expressed as a vector and used as an input for the next process.

LSTM is a neural network that is advantageous for extracting features with correlation between data, such as time series or strings. In the present invention, by applying the LSTM neural network learning method, the fire detection sensors selected from among sensors of various types of various parameters according to the field can be selected from the normal input range and abnormality during the learning period due to the nature of time series learning, even without inputting the respective specification information. When a (disaster) occurs, the standard deviation can be appropriately reflected.

FIG. 6 shows a detailed configuration of a Neural Network (NN)-dimensional reduction processing unit 200-2 of FIG. 3, and FIG. 7 shows a method of calculating a center distance of a class.

7 is a graph illustrating a process of deriving a vector distance from a class center point of a probability distribution. 7 shows the results of using the Mahalanobis distance calculation method considering the variance of data.

The NN-dimensional reduction processing unit 200-2 of FIG. 6 receives the vector output from the LSTM neural network learning unit 200-1 and applies NN neural network learning as neural network learning for classification of sensor data. That is, NN learning is performed as a NN learning model (fire class, non-fire class) to classify classes using the learning data that has passed through the sensor value collection unit 120 and the LSTM neural network learning unit 200-1. 260). In addition, center data is determined for each of the fire class and the non-fire class for the learning data to form a center vector of the class distribution as shown in FIG. 7 .

The process of inferring a class from sensor values is as follows.

The input sensor values pass through the sensor value collection unit 120 and the LSTM neural network learning unit 200-1, and pass through the learned NN model 260 to obtain feature vectors.

Next, the dimensionality reducer 270 plots the data on the coordinate plane by using a method such as TSNE/UMAP, which enables dimensionality reduction and visualization at the same time.

Next, the distance probability determination unit 280 determines the vector distance between the input data and the previously obtained center data according to the Mahalanobis distance measurement method considering the data distribution. In addition, a probability value according to the distance is determined through a probability distribution to derive a confidence value (confidence value = 0 in the case of non-fire).

Next, operations of the image feature calculating unit 400 and the integrated confidence calculating unit 500 of FIG. 3 will be described.

The image feature calculator 400 includes a Convolutional Neural Network (CNN) neural network 420 that performs neural network learning and processing for classifying image data; And it may include an image fire determination unit 450 for determining a fire from the image.

The CNN neural network 420 performs CNN learning to classify a fire image and an image without fire using an image input from a camera, and the image fire determination unit 450 performs CNN training in the image data collection unit 140. Using the collected images as input, it determines whether there is a fire using the learned neural network, and derives a confidence value.

The integrated confidence calculation unit 500 calculates a final total confidence by giving weights to the confidences output from the NN-dimensional reduction processing unit 200-2 and the image fire detection unit 450, respectively. . At this time, the confidence calculated from the path from the sensor value collection unit 120 utilizing sensor data to the NN-dimensional reduction processing unit 200-2 and the image feature calculation unit 400 calculate The weight can be determined by reflecting the degree of trust according to the environmental conditions for the established confidence.

For example, when the area to be monitored and installation locations of the sensors/cameras are indoors where environmental parameters, such as temperature/humidity, fluctuate less, a greater weight may be assigned to the confidence (characteristic) based on the sensor values.

Conversely, in the case of an outdoor area where a subject area to be monitored and installation locations of sensors/cameras have a large fluctuation range of temperature and humidity, a greater weight may be assigned to the confidence (characteristic) based on the image.

For example, according to the version of the fire determination algorithm/learning model based on sensor values and images or the time point at which unit learning of a prescribed quantity is performed, the newer one may be given a higher weight.

8a and 8b are flowcharts illustrating an embodiment of a fire determination method according to the spirit of the present invention. Steps S330 and after in FIGS. 8A and 8B may be integrated into one flow.

The illustrated fire determination method includes the steps of collecting measurement values of sensors for measuring each parameter that is a basis for determining a fire (S110); Calculating fire-related characteristic information from measurement values of sensors collected by a time-series data prediction technique (S120 to S180); Collecting camera images of the fire monitoring area (S210); Calculating fire-related feature information included in the collected images by applying an image classification technique (S250 to S280); calculating an integrated confidence by assigning weights according to the environment of the area to be monitored to the feature information calculated from the image and the feature information calculated from the measured value of the sensor (S330); and determining a fire if the calculated integrated confidence exceeds a predetermined reference value (S340, S350).

In the step of collecting the measured values of the sensors (S110), a feature vector form of unit size may be configured for the input of each sensor updated at each unit time.

For example, in the step of collecting the measured values of the sensors (S110), sensor data from time t-a to time t may be received from various sensors installed for the site to be monitored. Accordingly, in the step of collecting the measured values of the sensors (S110), a feature vector form of unit size may be configured for the input of each sensor updated at each unit time.

Calculating fire-related feature information from the measured values of the sensors shown (S120 to S180) includes predicting sensor data at time t+1 using the learned LSTM (S120); constructing a value vector consisting of sensor data at time t and predicted data at time t+1 (S130); Extracting a feature vector by passing through a learned Neural Network (NN) model (S140); performing dimensional reduction with a 2-dimensional or 3-dimensional feature vector (S150); Calculating the distance between the central data and the feature vector of each of the fire class and the non-fire class and assigning the distance to a nearby class, that is, determining the class according to the distance to the center (S160 and S170); and calculating a confidence value if it is a fire class and designating it as Confidence = 0 (S180, S185) if it is not.

Calculating the fire-related feature information included in the illustrated image (S250 to S280) includes calculating a feature vector using a CNN (S250); Determining a fire class from CNN (S270); Calculating the confidence value according to the activation function, and otherwise designating the confidence value as 0 (S280, S285) may be included.

In the step of calculating the integrated confidence (S330), the combined confidence is calculated as a final fire determination factor by applying a weight to each output result of the steps S180 and S280.

In the step of determining fire (S340, S350), if the final confidence value is greater than the threshold value, it is finally determined as fire (S350), otherwise it can be determined as no fire (S355).

Those skilled in the art to which the present invention pertains should understand that the embodiments described above are illustrative in all respects and not limiting, since the present invention can be embodied in other specific forms without changing the technical spirit or essential characteristics thereof. only do The scope of the present invention is indicated by the following claims rather than the detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

120: sensor value collection unit
140: image data collection unit
200: sensor value characteristic calculator
200-1: LSTM neural network learning part
200-2: NN-dimensional reduction processing unit
400: image feature calculator
500: integrated confidence calculation and fire determination unit

Claims (11)

an image data collection unit that collects images captured by the camera;
an image feature calculation unit that calculates fire-related feature information included in the collected images by applying an image classification technique;
a sensor value collection unit that collects measurement values of each sensor that measures each parameter that is a basis for determining a fire;
a sensor value feature calculation unit that calculates fire-related feature information from measurement values of sensors collected using a time-series data prediction technique;
Depending on the environment of the area to be monitored, the feature information calculated by the image feature calculation unit and the feature information calculated by the sensor value feature calculation unit are weighted to calculate integrated confidence, and if the calculated integrated confidence exceeds a predetermined reference value, Integrated confidence determination unit that determines fire
A fire discrimination system using time-series data prediction techniques and image classification.
According to claim 1,
The sensor value feature calculation unit,
an LSTM neural network learning unit that collects measurement values of each sensor in time series and performs LSTM neural network learning; and
NN-dimensional reduction processing unit for inferring the class of the measured values of the sensor by using the result of performing the LSTM neural network learning
A fire detection system comprising a.
According to claim 2,
The LSTM neural network learning unit,
Using the learned LSTM model, the sensor value at time t+1 is predicted, the sensor value at time t (current) and the sensor value at time t+1 (future) are secured, and the data at the current time and the predicted data are combined into one vector A fire identification system expressed as
According to claim 3,
The NN-dimensional reduction processing unit,
A fire discrimination system that receives the vector output by the LSTM neural network learning unit and classifies it into a fire class or a non-fire class by applying NN neural network learning as neural network learning for classifying sensor data.
According to claim 1,
The sensor value collection unit,
A fire detection system that constitutes a feature vector of unit size for each sensor's input updated at each unit time.
According to claim 1,
The image feature calculator,
a CNN neural network that performs neural network-type learning and processing for classifying image data; and
An image fire determination unit that derives a confidence value for determining a fire from a classified image
A fire detection system comprising a.
According to claim 1,
The integrated confidence determination unit,
Depending on the version of the fire determination algorithm and learning model based on sensor values and images or the time point at which the prescribed amount of unit learning has been performed, a fire determination system that assigns a higher weight to the latest one.
Collecting measurement values of sensors that measure each parameter that is a basis for determining a fire;
Calculating fire-related characteristic information from measurement values of sensors collected using a time-series data prediction technique;
Collecting camera-captured images of the fire surveillance area;
Calculating fire-related feature information included in the collected images by applying an image classification technique;
calculating an integrated confidence by assigning a weight according to the environment of a region to be monitored to the feature information calculated from the image and the feature information calculated from the measured value of the sensor; and
Determining a fire when the calculated integrated confidence exceeds a predetermined reference value
A fire discrimination method using time series data prediction techniques and image classification, including .
According to claim 8,
Calculating fire-related characteristic information from the measured values of the sensors,
Predicting sensor data at time t+1 using the learned LSTM;
constructing a value vector consisting of sensor data at time t and predicted data at time t+1;
extracting a feature vector by passing through the learned NN model;
performing dimensional reduction with a 2-dimensional or 3-dimensional feature vector;
Calculating distances between central data and feature vectors of each of the fire class and the non-fire class, and assigning the distance to a nearby class; and
Steps to calculate confidence value in case of fire class
Fire determination method comprising a.
According to claim 8,
Calculating the fire-related feature information included in the image,
Calculating a feature vector using CNN;
Determining a fire class from CNN; and
Calculating the confidence value according to the activation function
Fire determination method comprising a.
According to claim 8,
In the step of collecting the measured values of the sensors,
A fire discrimination method that constructs a feature vector form of unit size for the input of each sensor updated at each unit time.

Images