KR102540208B1 - Fire detection method using deep learning - Google Patents

Abstract

The present invention relates to a fire detection method for detecting a fire from an input image based on a fire detection neural network, wherein the fire detection neural network includes a backbone neural network, a neck neural network, and a head neural network, The backbone neural network includes a first convolution module to which the image is input, N backbone sub-neural networks serially connected to process outputs of the convolution modules, and an SPPF connected to an output terminal of the N-th backbone sub-neural network. module, and a second convolution module connected to an output terminal of the SPPF module, wherein the backbone sub-neural network includes a GhostNet convolution module and a C3Ghost module serially connected to the GhostNet convolution module. The neck neural network passes the output of the convolution module through the CBS module, upsamples it, concatenates it with the output of the N-1 th backbone sub-neural network, and then passes it through the first C3_F module. is composed of

Description

Fire detection method using deep learning

The present invention relates to a fire detection method, and more particularly, to a fire detection method capable of rapidly and accurately detecting fire by using a deep learning-based fire detection neural network from an image acquired by an image acquisition unit.

Fire is one of the disasters that cause human and material damage, and it may be caused by various causes. In particular, fire can minimize human and material damage depending on the success of the initial suppression. Various early warning systems including heat detection systems are being spread, and recently, research on fire detection systems using deep learning has been actively conducted.

Currently installed on the market, CCTV is a method of storing images in a storage device called a digital video recorder (DVR) by collecting individual or multiple CCTVs. In this method, when an event such as fire occurs, the DVR is connected to the server and the situation is monitored. The reason for this is that it is difficult to manage network traffic when a large amount of data is managed by the server.

In this respect, artificial intelligence software for fire detection should be installed in the DVR, not the server, and should play a role in delivering this information to the server only when a fire is detected. However, there is a disadvantage that the currently installed and used DVR does not have a GPU.

Therefore, it is important how quickly the neural network can recognize in an environment operated only by the CPU, and it is necessary to develop a lightweight, fast, and accurate deep learning architecture that can detect fires so that existing CCTVs can be used.

Recently, various methods of analyzing images and determining whether there is a fire using deep learning have been studied. Although the accuracy of fire detection is considerably improved, there is a problem in that when a picture of a fire is projected onto a CCTV, it is recognized as an actual fire.

The problem to be solved by the present invention is, first, to recognize a fire through reasoning using a deep learning-based artificial neural network in an image captured through CCTV, but the CCTV is a fake fire (eg, shooting fire or smoke) It is to provide a fire detection method having a high degree of accuracy so as not to mistake it for a fire even when a picture) is taken.

Second, it is to provide a fire detection method using an artificial neural network capable of light and rapid reasoning that can detect fire sufficiently quickly and accurately using only a CPU provided in CCTV.

Third, it is to provide a fire detection method using a new neural network improved by referring to GoshtNet in the YOLOv5 neural network.

The present invention relates to a fire detection method for detecting a fire from an input image based on a fire detection neural network, wherein the fire detection neural network includes a backbone neural network, a neck neural network, and a head neural network.

The backbone neural network includes a first convolution module to which the image is input, N backbone sub-neural networks serially connected to process outputs of the convolution modules, and an SPPF connected to an output terminal of the N-th backbone sub-neural network. module, and a second convolution module connected to an output terminal of the SPPF module.

The backbone sub-neural network includes a GhostNet convolution module and a C3Ghost module serially connected to the GhostNet convolution module.

The neck neural network passes the output of the convolution module through the CBS module, upsamples it, concatenates it with the output of the N-1 th backbone sub-neural network, and passes it through the first C3_F module. .

The first C3_F module comprises a CBS group consisting of three CBS modules connected in series, a CBS module connected in parallel with the CBS group, and a connet that connects the output of the CBS group and the output of the CBS module connected in parallel. It includes a Nate module and a CBS module connected to an output terminal of the Concatenate module.

The neck neural network includes a first concatenate module that combines the upsampled output of the first C3_F module after passing through the first CBS module with the output of the N-2 th backbone sub-neural network; A second C3_F module processing the output of the Ketnate module may be further included.

The head neural network may include a third convolution module to which an output of the second C3_F module is input.

The neck neural network includes a second CBS module connected in parallel to the third convolution module and receiving an output of the second C3_F module, and combining the output of the second CBS module with the output of the first CBS module. It may further include a second concatenate module and a third C3_F module processing an output of the second concatenate module.

The head neural network may further include a fourth convolution module receiving an output of the third C3_F module.

The neck neural network includes a third CBS module that receives the output of the third C3_F module, a third concatenate module that connects the output of the third CBS module and the output of the CBS module, and the third concatenate A fourth C3_F module processing an output of the Nate module may be further included.

The head neural network may further include a fifth convolution module receiving an output of the fourth C3_F module.

Alternatively, the present invention relates to a computer readable recording medium in which a fire detection neural network is recorded in the form of a program, wherein the fire detection neural network includes a backbone neural network, a neck neural network, and a head neural network. and the backbone neural network includes a first convolution module to which an image is input; N backbone sub-neural networks connected in series to process the output of the convolution module; An SPPF module connected to the output terminal of the N-th backbone sub-neural network; and a second convolution module connected to an output terminal of the SPPF module, wherein the backbone sub-neural network includes a convolution module of GhostNet; and a C3Ghost module serially connected to the convolution module of the ghostnet, wherein the neck neural network passes an output of the convolution module through a CBS module and then upsamples the N-1 th backbone sub-neural network. After concatenating with the output, it is configured to pass a first C3_F module, wherein the first C3_F module includes: a CBS group composed of three CBS modules connected in series; a CBS module connected in parallel with the CBS group; Concatenate module for connecting the output of the CBS group and the output of the CBS module connected in parallel; and a CBS module connected to an output terminal of the concatenate module.

First, the fire detection method of the present invention has a high level of accuracy that does not mistake a fake fire for a fire.

Second, by improving the existing YOLO neural network by referring to GhostNet, it is possible to increase accuracy and to make light and fast inference.

Third, compared to the existing YOLOv5, the accuracy is improved by about 1.02%, and there is an effect of showing a speed improvement of about 40 ms or more compared to YOLOv5 based on the inference time.

1 is a structural diagram of a fire detection neural network according to an embodiment of the present invention.
2 is a structural diagram of a fire detection neural network applied to a fire detection method according to an embodiment of the present invention. Hereinafter, reference is made to FIG. 2 .
Figure 3 shows the structure of the CBS module and the C3_F module.
4 shows the structure of the C3Ghost module.
5 shows a dataset for training a fire detection neural network.
6 shows the training result of the fire detection neural network.
7 shows the training result of the loss function.
8 shows the mAP_0.5 evaluation result, and FIG. 9 shows the mAP_0.5:0.05 evaluation result.
10 shows the result of evaluating the accuracy of the neural network by employing a precision-recall curve (PR curve).
11 shows the result of verifying whether the fire detection neural network can be recognized in a real environment.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to completely inform the person who has the scope of the invention, and the present invention is only defined by the scope of the claims. Like reference numbers designate like elements throughout the specification.

YOLO stands for You Only Look Once, and YOLO neural network is an object detection neural network with the advantage of being fast. YOLO has been released from yolov1 to yolov5 as of July 2020, and is used by many people, from non-majors to students as well as developers.

Since the YOLO neural network is a single neural network structure, the configuration is simple and fast, it learns surrounding information and processes the entire image, so the background error is low, and the detection accuracy is high even for new images not seen in the training stage. Compared to the detection model, the accuracy (mAP) is somewhat lower.

The present invention relates to a novel neural network (hereinafter, referred to as 'AeryNet') that improves somewhat insufficient accuracy while maintaining speed, which is an advantage, by improving the YOLO neural network. In the embodiment, AeryNet constructs a Backbone network to which the G3Ghost module is applied instead of the C3 module constituting the YOLOv5 neural network, and uses the newly developed C3_F module based on the CBS module to configure the Neck network. will be.

1 shows a fire detection system according to an embodiment of the present invention. Referring to FIG. 1, a fire detection system 1 according to an embodiment of the present invention includes an image acquisition unit 10 that acquires an image, and an object that identifies an object from an image acquired by the image acquisition unit 10. A recognition unit 20 may be included.

The object recognition unit 20 may include a data storage unit 22 in which Aery neural network software is recorded, and a processor 21 that drives the software.

The image acquisition unit 10 may include a CCTV that captures surrounding images in real time. The image (or image) acquired by the image acquisition unit 10 may be stored in the data storage unit 22, and the processor 21 drives Aery neural network software to fire from the image stored in the data storage unit 22. Identification or detection may be performed.

2 is a structural diagram of a fire detection neural network applied to a fire detection method according to an embodiment of the present invention. Hereinafter, reference is made to FIG. 2 .

A fire detection method according to an embodiment of the present invention detects a fire from an input image based on a fire detection neural network (Aery neural network). The input image may be a video (image) acquired by the image acquiring unit 10 .

The fire detection neural network may include a backbone neural network (100), a neck network (200), and a head neural network (300). The backbone neural network 100, the neck neural network 200, and/or the head neural network 300 may be improved based on the YOLO neural network. In the embodiment, YOLOv5 is improved, but it is not necessarily limited thereto.

The backbone neural network 100 can convert an input image into a feature map, and the head neural network 300 performs a location operation of the feature map to predict classes and bounding boxes. ) can be performed. Also, the neck neural network 200 is a part that connects the backbone neural network 100 and the head neural network 200, and can play a role in refining and reconstructing feature maps.

Meanwhile, the backbone neural network 100 may include a convolution module 110 , N backbone sub neural networks 120 , 130 , 140 , and 150 , an SPPF module 160 , and a convolution module 170 .

An image is input to the convolution module 110, and the image may be acquired by the image acquisition unit 10.

The N backbone sub-neural networks 120, 130, 140, and 150 are connected in series to each other so that the output of the previous backbone sub-neural network becomes the input of the backbone sub-neural network. Each of the backbone sub-neural networks 120, 130, 140, and 150 has a convolution module 121, 131, 141, and 151 of GhostNet and a C3Ghost module 122 connected in series with the convolution module of GhostNet. , 132, 142, 152).

A result output from at least one of the N backbone sub-neural networks 120 , 130 , 140 , and 150 may be input to a concatenate module ('Cancat', Concatenate module) constituting the neck neural network 200 . In the embodiment, the backbone neural network 100 includes four (N=4) backbone sub neural networks 120, 130, 140, and 150, and the output of the second backbone sub neural network 130 and the third backbone sub neural network ( 140) is input to the two concatenate modules 203 and 207 of the neck neural network 200, respectively.

On the other hand, the structure of the C3Ghost module (122, 132, 142, 152) is already well known as it is configured based on GhostNet's convolution (GhostConv) and bottleneck (GhostBottelneck), as shown in FIG. Even if there is no explanation, it is assumed that the contents disclosed in the references below are used.

Reference 1: Yang, W.; Ma, X.; Hu, W.; Tang, P. Lightweight Blueberry Fruit Recognition Based on Multi-Scale and Attention Fusion NCBAM. Agronomy 2022, 12, 2354. https://doi.org/10.3390/agronomy12102354

Reference 2: Citation: Chen, X.; Lian, Q.; Chen, X.; Shang, J. Surface Crack Detection Method for Coal Rock Based on Improved YOLOv5. Appl. Sci. 2022, 12, 9695. https://doi.org/10.3390/app12199695

The SPPF module 160 is an improved SPP (Spatial Pyramid Pooling) module of YOLOv5 (see References 1 and 2), and can be connected to the output terminal of the N-th backbone sub-neural network 150. That is, a result output from the C3Ghost module 152 of the Nth backbone sub-neural network 150 may be input to the SPPF module 160.

The convolution module 170 may be connected to the output terminal of the SPPF module 160.

The neck neural network 200 includes a CBS module 201 to which the output of the convolution module 170 is input, an upsample module 202 connected to the output terminal of the CBS module 201, and an upsample module 202 ) may include a concatenate module 203 connected to the output terminal.

As shown in FIG. 3, the CBS module 201 may have a structure in which a convolution module (conv), a batch normalization module (BatchNorm), and a SILU module (Sigmoid Linear Unit, SILU) are connected in series.

The CBS module 201 is well known, and even if there is no specific description, the contents disclosed in Reference 3 below will be used.

Reference 3: Qingqing Xu, Zhiyu Zhu, Huilin Ge, Zheqing Zhang, Xu Zang, "Effective Face Detector Based on YOLOv5 and Superresolution Reconstruction", Computational and Mathematical Methods in Medicine, 9 pages, 2021

The output of the convolution module 170 may pass through the CBS module 201, be up-sampled in the up-sample module 202, and then be input to the concatenate module 203. In addition, the output of the N-1th backbone submodule 140 may also be input to the concatenate module 203 and concatenated with the above-described upsampled result.

The concatenated result in the concatenate module 203 may be input to the C3_F module 204. As shown in FIG. 3, the C3_F module 204 may include a CBS group composed of three CBS modules connected in series and a CBS module connected in parallel with the CBS group. Further, a concatenate module connecting the output of the CBS group and the output of the CBS module connected in parallel, and a CBS module are further connected to the output terminal of the concatenate module.

Meanwhile, the neck neural network 300 includes the CBS module 205 receiving the output of the C3_F module 204, the upsampling module 206 upsampling the output of the CBS module 205, and the upsampling module 206. ) may further include a concatenate module 207 connected to the output terminal.

The output of the N−2 th backbone sub-neural network 130 is also input to the concatenate module 207, and is spliced with the output of the upsample module 206.

The C3_F module 208 may be connected to the output terminal of the concatenate module 207 . A result output from C3_F 208 may be input to the convolution module 301 of the head neural network 300 .

Meanwhile, the CBS module 209 may be connected to the output terminal of the C3_F module 208. Then, the output of the CBS module 209 is input again to the concatenate module 210, and at this time, the output of the aforementioned CBS module 205 is also input to the concatenate module 210 so that both can be joined. .

The combined result passes through the C3_F module 211 again and is then transferred to the convolution module 302 of the head neural network 300. At the same time, the CBS module 212 is further connected to the output of the C3_F module 211. , the combined result is also input to the CBS module 212.

A concatenate module 213 may be connected to an output terminal of the CBS module 212 . The output of the CBS module 201 and the output of the CBS module 212 are concatenated in the concatenate module 213, and the result is passed through the C3_F module 214 and then the convolution module 303 of the head neural network 300 ) is entered.

On the other hand, in order to prevent the fire detection neural network from mistaking an image obtained by the image acquisition unit 10 taking a picture of a fire (eg, a picture of smoke or flame) as an actual fire, the neural network is trained. need to do it A dataset was prepared for this, and the neural network was trained by labeling it as a separate class. Fire and smoke were extracted, and the fake fire class was designated to train the neural network.

In the example, the loss function for training the neural network was defined as Equation (1) below.

Equation (1)

Here, l _box represents the bounding box regression loss function, l _cls represents the clssification loss function, and l _obj represents the object's confidence loss function, respectively. The formulas for l _box , l _{cls ,} and l _obj are defined as Equations (2), (3), and (4) below, respectively.

Equation (2)

Equation (3)

Equation (4)

는 position loss coefficient를 의미하며,

는 category loss coefficient를 나타내고, x, y은 대상의 실재 좌표를 나타낸다. 그리고, w, h은 대상의 넓이와 높이를 나타낸다.here,

means position loss coefficient,

represents the category loss coefficient, and x and y represent the actual coordinates of the target. And, w and h represent the width and height of the object.

In YOLOv5, to solve the problem of overlapping bounding boxes, GIoU is used as a loss function to find the optimal frame. Equations (5) and (6) define the GIoU loss function used in the proposed neural network.

Equation (5)

Equation (6)

Here, C represents the smallest bounding box containing A and B. GIoU acquires the situations of A and B, and has scale invariance similar to IoU. GIoU can be viewed as the lower limit of IoU. GIoU focuses on the coverage area as well as other non-overlapping areas, allowing the mirror level of the crossover level between the two to be increased.

6 is a training result of a neural network. It can be seen that each loss (box, obj, class) converges to 0 during 300 epochs. The loss of the neural network proposed by Equation (1) is about 0.029, indicating that normal learning has progressed, and FIG. 7 shows the training result of the loss function.

On the other hand, there are various ways to express the experimental results in the neural network. In this embodiment, precision, recall, average precision (AP), and mean average precision (mAP) were used as evaluation techniques. The evaluation formula used is shown in Equation (7-10).

Equation (7)

Equation (8)

Equation (9)

Eq(10)

In the formula, TP represents true positive, FP represents false positive, and FN represents false negative. The results according to each formula are shown in [Table 1] below. The comparison neural network adopted Yolov5.

Parameter Yolov5 AeryNet mAP_0.5 0,956 0,960 (+0.4%) mAP_0.5:0.95 0,683 0,690 ( +1.02%) Precision 0,919 0,924 Recall 0,926 0,920

As shown in [Table 1], the neural network according to this embodiment shows performance improvement of 0.4% and 1.02%, respectively, in mAP_0.5 and mAP_0.5:0.95 compared to Yolov5. 8 shows the mAP_0.5 evaluation result, and FIG. 9 shows the mAP_0.5:0.05 evaluation result.

On the other hand, the neural network predicted a fire, and we need an indicator that indicates how many of them are fires, and what the result of the neural network predicting a fire in an actual fire is. In this embodiment, PR curve (precision-recall curve) was adopted and evaluated. The evaluation results are shown in FIG. 10 .

As shown in FIG. 10, it can be seen that the values for properly recognizing fire and smoke are 0.954 and 0.966, respectively, which are values that can be applied even in real environments. The result of verifying whether recognition is possible in a real environment is shown in FIG. 11 .

What can be confirmed in FIG. 11 is that the neural network of this embodiment accurately infers real fire and smoke, and when trying to recognize false fire such as a mobile phone screen and a picture in the image acquisition unit 10, in FIG. 11 (c) It can be confirmed that it is not recognized as such.

On the other hand, as shown in Table 2 below, the results of verification in the Google colab environment indicate that the neural network proposed in the paper infers at a speed of about 40 ms or more, which is the result of inferring 1000 images of 640 * 480 size.

Device Yolov5 AeryNet CPU (Google Colab) 194.2ms 152.9ms

Meanwhile, AeryNet and/or the fire detection method described above may be provided in the form of a program stored in a computer readable recording medium. The computer-readable recording medium includes a non-transitory computer recording medium (for example, various main storage devices or auxiliary storage devices) as well as a radio signal (transitory computer recording medium) (for example, , data signals that can be provided through the network) are also included.

Claims (5)

A fire detection method for detecting a fire from an input image based on a fire detection neural network,
The fire detection neural network includes a backbone neural network 100, a neck neural network 200, and a head neural network 300,
The backbone neural network 100,
a first convolution module 110 to which the image is input;
N backbone sub-neural networks 120, 130, 140, 150 connected in series to process the output of the first convolution module 110;
an SPPF module 160 connected to an output terminal of the N-th backbone sub-neural network 150; and
A second convolution module 170 connected to the output terminal of the SPPF module 160,
The backbone sub-neural networks 120, 130, 140, and 150,
Convolution modules (121, 131, 141, 151) of GhostNet (GhostNet); and
Includes C3Ghost modules (122, 132, 142, 152) connected in series with the convolution modules (121, 131, 141, 151) of the GhostNet,
The neck neural network 200,
The output of the second convolution module 170 is passed through the CBS module 201, upsampled, and concatenated with the output of the N-1 th backbone sub-neural network 140, and then the first C3_F module (204),
The first C3_F module 204,
CBS group consisting of three CBS modules connected in series;
a CBS module connected in parallel with the CBS group;
a concatenate module that connects the output of the CBS group and the output of the CBS module connected in parallel; and
A fire detection method comprising a CBS module connected to the output terminal of the concatenate module. According to claim 1,
The neck neural network 200,
A first concatenate module (207) that connects the output of the first C3_F module (204) through the first CBS module (205) and then upsamples the result with the output of the N-2-th backbone sub-neural network (130). ); and
A fire detection method further comprising a second C3_F module (208) processing an output of the first concatenate module (207). According to claim 2,
The head neural network 300,
A third convolution module 301 to which the output of the second C3_F module 208 is input,
The neck neural network 200,
a second CBS module 209 connected in parallel with the third convolution module 301 and receiving an output of the second C3_F module 208;
a second concatenate module (210) that couples the output of the second CBS module (209) with the output of the first CBS module (205); and
Further comprising a third C3_F module 211 processing the output of the second concatenate module 210,
The head neural network 300,
A fire detection method further comprising a fourth convolution module (302) receiving an output of the third C3_F module (211). According to claim 3,
The neck neural network 200,
a third CBS module 212 receiving the output of the third C3_F module 211;
a third concatenate module 213 combining an output of the third CBS module 212 and an output of the CBS module 201; and
A fourth C3_F module 214 processing the output of the third concatenate module 213 is further included,
The head neural network 300,
A fire detection method further comprising a fifth convolution module (303) receiving an output of the fourth C3_F module (214). In the computer readable recording medium in which the fire detection neural network is recorded in the form of a program,
The fire detection neural network,
It includes a Backbone neural network 100, a Neck neural network 200 and a Head neural network 300,
The backbone neural network 100,
A first convolution module 110 to which an image is input;
N backbone sub-neural networks 120, 130, 140, 150 connected in series to process the output of the convolution module 110;
an SPPF module 160 connected to an output terminal of the N-th backbone sub-neural network 150; and
A second convolution module 170 connected to the output terminal of the SPPF module 160,
The backbone sub-neural networks 120, 130, 140, and 150,
Convolution modules (121, 131, 141, 151) of GhostNet (GhostNet); and
Includes C3Ghost modules (122, 132, 142, 152) connected in series with the convolution modules (121, 131, 141, 151) of the GhostNet,
The neck neural network 200,
The output of the convolution module 170 is passed through the CBS module 201, upsampled, and concatenated with the output of the N-1 th backbone sub-neural network 140, and then the first C3_F module 204 ) is configured to pass,
The first C3_F module 204,
CBS group consisting of three CBS modules connected in series;
a CBS module connected in parallel with the CBS group;
Concatenate module for connecting the output of the CBS group and the output of the CBS module connected in parallel; and
A computer readable recording medium comprising a CBS module connected to an output terminal of the concatenate module.

Images