CN116310927A - Multi-source data analysis fire monitoring and identifying method and system based on deep learning - Google Patents

Multi-source data analysis fire monitoring and identifying method and system based on deep learning Download PDF

Info

Publication number
CN116310927A
CN116310927A CN202211099842.8A CN202211099842A CN116310927A CN 116310927 A CN116310927 A CN 116310927A CN 202211099842 A CN202211099842 A CN 202211099842A CN 116310927 A CN116310927 A CN 116310927A
Authority
CN
China
Prior art keywords
data
video
fire
visible light
network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202211099842.8A
Other languages
Chinese (zh)
Inventor
李晓贤
王奔
王刚
杜梦岩
谷陈丰
张歆萌
贾阳
马扬
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Xi'an Zhonghe Nuclear Instrument Co ltd
Original Assignee
Xi'an Zhonghe Nuclear Instrument Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Xi'an Zhonghe Nuclear Instrument Co ltd filed Critical Xi'an Zhonghe Nuclear Instrument Co ltd
Priority to CN202211099842.8A priority Critical patent/CN116310927A/en
Publication of CN116310927A publication Critical patent/CN116310927A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/40—Scenes; Scene-specific elements in video content
    • G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B17/00—Fire alarms; Alarms responsive to explosion
    • G08B17/10—Actuation by presence of smoke or gases, e.g. automatic alarm devices for analysing flowing fluid materials by the use of optical means
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
    • Y02A40/00—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production
    • Y02A40/10—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture
    • Y02A40/28—Adaptation technologies in agriculture, forestry, livestock or agroalimentary production in agriculture specially adapted for farming

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Molecular Biology (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Business, Economics & Management (AREA)
  • Emergency Management (AREA)
  • Fire-Detection Mechanisms (AREA)

Abstract

The invention relates to a multi-source data analysis fire monitoring and identifying method and system based on deep learning, wherein the method comprises the following steps: s1: collecting fire sample data under different working conditions to obtain a sample set; step S2: pre-processing sample setsProcessing; obtaining a preprocessed sample set, comprising: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, wherein the two-dimensional array X comprises a visible light video characteristic diagram sequence set V and an infrared video characteristic diagram sequence set I; step S3: obtaining a partial characteristic diagram C by passing X through a characteristic extraction network a a V and I are respectively processed by a video processing network and a feature extraction network to respectively obtain a visible light video part feature map C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I And (3) distributing different weights for the fire disaster prediction system through the attention management mechanism module, and fusing the weights to obtain a feature map C, and carrying out the fire disaster prediction based on the feature map C. The method provided by the invention can better fuse different types of data and alarm more quickly.

Description

Multi-source data analysis fire monitoring and identifying method and system based on deep learning
Technical Field
The invention relates to the fields of fire detection, image type fire detectors and novel fire detectors, in particular to a multi-source data analysis fire monitoring and identifying method and system based on deep learning.
Background
The multi-mode fire detector based on deep learning can find fire disaster in early stage of fire disaster occurrence, comprehensively judge whether fire disaster occurs or not according to various different forms of data, especially find fire disaster which is not easy to be reflected by other single data, be favorable for timely finding fire disaster, provide more abundant time for relevant safety management personnel to react to current situation, and especially have obvious advantages in indoor large space application occasions with high requirements on fire disaster fire protection.
At present, common fire detectors, such as smoke-sensing fire detectors, temperature-sensing fire detectors and other fire detectors with single channels, have respective applicable scenes, for example, smoke-sensing fire detectors can only sense smoke, and can not alarm for fires without smoke or with low smoke concentration; the temperature-sensing type fire detector can alarm only when the ambient temperature of the detector rises to a certain degree, so that the temperature-sensing type fire detector is not applicable to detecting smoldering fires; after the image type fire detector encounters a shielding object to shield a fire source and smoke, the fire cannot be detected; the multi-mode fire detection technology can judge whether the fire is on the spot or not through other data, so that the multi-mode fire detection technology is more reliable, and the alarm accuracy is improved. At present, in some places with precise instruments and precious devices, such as laboratory, machine room, museums, hospitals, nuclear power stations and the like, which have high fireproof requirements, once the fire is generated, huge economic loss and serious consequences are caused. However, the single type of fire detection method may not fully guarantee the safety of the fire-fighting site due to its limitations.
The patent application discloses an image type fire detector (CN 214279117U) of the Internet of things, which uses a multi-path image type fire detector for fire disaster identification, wherein the multi-path image type fire detector comprises a common image type fire detector of the Internet of things, an explosion-proof image type fire detector of the Internet of things, an infrared thermal imaging fire detector of the explosion-proof Internet of things, an image type beam smoke detector of the Internet of things, a dome camera and a gun type camera, but the data source used by the detector is mainly an image, and other physical quantities generated by the fire disaster are not comprehensively considered. The patent application discloses a smoke recognition method and a recognition system (CN 112257523A) of an image type fire detector, which are used for solving the technical problems of higher algorithm complexity and lower positioning accuracy in the existing smoke recognition method, and converting video images into image sequences by acquiring video images of a monitoring site; setting a reference point of an RGB color space; dividing the image sequence in the RGB color space into a motion foreground region to obtain a suspected target region; the invention also discloses a method for identifying fire smoke by means of a video image. The patent application discloses an ultraviolet, infrared and vision-based comprehensive flame detection method (CN 112069975A), which is characterized in that an ultraviolet detector and an infrared detector are used for respectively detecting the current environment, and trigger signals of the ultraviolet detector and the infrared detector are connected to a counter through an AND gate; in addition, the current image is acquired through the camera, the image is analyzed, whether the current environment has fire or not is judged, and if the fire passes through the area of the flame, the brightness of the surrounding environment is used for judging the current fire grade. The invention completely relies on the threshold value to judge fire, and although the setting is simpler, how to set the proper threshold value in a complex scene is difficult.
Therefore, how to realize multi-mode fire detection is a urgent problem to be solved.
Disclosure of Invention
In order to solve the technical problems, the invention provides a multi-source data analysis fire monitoring and identifying method and system based on deep learning.
The technical scheme of the invention is as follows: a multi-source data analysis fire monitoring and identifying method based on deep learning comprises the following steps:
step S1: collecting fire sample data under different working conditions, and labeling each sample with a flame or a flameless label; obtaining a sample set, wherein the sample set comprises five data: smoke data, temperature data, flame radiation data, visible light video data, and infrared video data;
step S2: preprocessing the sample set; obtaining a pretreated sample set; wherein the preprocessed sample set comprises: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, and constructing a visible light video feature map sequence set V and an infrared video feature map sequence set I from visible light video data;
step S3: inputting the preprocessed sample set into a fire monitoring and identifying network for feature extraction: the two-dimensional array X is processed through a feature extraction network a to obtain a partial feature map C a The visible light video characteristic image sequence set V and the infrared video characteristic image sequence set I are respectively processed by video processing networks 1 and 2, and a characteristic extraction network b1 and a characteristic extraction network b2 to respectively obtain a visible light video part characteristic image C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I The attention mechanism module distributes different weights for the attention mechanism module and fuses the different weights to obtain a complete characteristic diagram C; and predicting the characteristic map C, and outputting a predicted value of 1 or 0, wherein 1 represents the occurrence of fire and 0 represents the absence of fire.
Compared with the prior art, the invention has the following advantages:
the invention discloses a multi-source data analysis fire monitoring and identifying method based on deep learning, which is used for detecting fire based on multi-source data, can more comprehensively acquire physical information when the fire happens, and can monitor flame and smoke signals simultaneously compared with the existing fire detector, so that the defects of a single detection means can be avoided, and the reliability of fire monitoring is improved; and the deep learning method is adopted to analyze multi-source data, so that more sufficient fire characteristics can be extracted and fire targets and other interferences can be distinguished more effectively compared with a common threshold method.
The invention collects multi-mode fire monitoring data containing five information of temperature, smoke concentration, flame radiation, visible light video data and infrared video data, and constructs a fire data set for synchronously monitoring the five data; in the fire monitoring and identifying network structure, N one-dimensional data can be converted into R rows by adopting the design that the data are converted into image signals after being piled up, and the two-dimensional data of N/R elements in each row are used for extracting information; the video processing network for processing the current video frame and the adjacent video frame by adopting different convolution kernels can convert three-dimensional data into two-dimensional data to extract space-time information contained in the video; adopting an improved residual error network structure, wherein the improved residual error network structure comprises a network model fusing a shallow layer and two pieces of content with an improved bottleneck structure; the attention mechanism is adopted to carry out weight distribution on different channel information, so that different types of data can be better fused, the alarm can be carried out more quickly, more time is striven for later fire protection early warning, and the fire loss is reduced.
The invention discloses a multi-source data analysis fire monitoring and identifying system based on deep learning, which improves the practicability of the system and is beneficial to the later equipment transformation through the design of independent coupling of multiple interfaces and functional modules. In the data processing part, the information analysis is carried out by using a deep learning method, so that the fire detection and recognition are realized, and the intellectualization of the fire-fighting equipment is promoted.
Drawings
FIG. 1 is a flow chart of a multi-source data analysis fire monitoring and recognition method based on deep learning in an embodiment of the invention;
fig. 2 is a schematic structural diagram of a feature extraction network a according to an embodiment of the present invention;
FIG. 3A is a schematic diagram of a convolution block a according to an embodiment of the present disclosure;
FIG. 3B is a schematic diagram of a convolution block B according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of a video processing network according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a feature extraction network b1/b2 according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a residual network b1/b2 according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of a residual bottleneck structure of an improved residual network in an embodiment of the present invention;
FIG. 8 is a schematic diagram of a fire monitoring and recognition network according to an embodiment of the present invention;
fig. 9 is a block diagram of a multi-source data analysis fire monitoring and recognition system based on deep learning according to an embodiment of the present invention.
Detailed Description
The invention provides a multi-source data analysis fire monitoring and identifying method based on deep learning, which is characterized in that fire monitoring data are acquired through various channels, different types of data are fused better, and an alarm is given more quickly.
The present invention will be further described in detail below with reference to the accompanying drawings by way of specific embodiments in order to make the objects, technical solutions and advantages of the present invention more apparent.
Example 1
As shown in fig. 1, the method for monitoring and identifying fire disaster based on multi-source data analysis of deep learning provided by the embodiment of the invention comprises the following steps:
step S1: collecting fire sample data under different working conditions, and labeling each sample with a flame or a flameless label; obtaining a sample set, wherein the sample set comprises five data: smoke data, temperature data, flame radiation data, visible light video data, and infrared video data;
step S2: preprocessing a sample set; obtaining a pretreated sample set; wherein the preprocessed sample set comprises: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, and constructing a visible light video feature map sequence set V and an infrared video feature map sequence set I from visible light video data;
step S3: inputting the preprocessed sample set into a fire monitoring and identifying network for feature extraction: the two-dimensional array X is processed through a feature extraction network a to obtain a partial feature map C a The visible light video characteristic image sequence set V and the infrared video characteristic image sequence set I are respectively processed by video processing networks 1 and 2, and a characteristic extraction network b1 and a characteristic extraction network b2 to respectively obtain a visible light video part characteristic image C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I The attention mechanism module distributes different weights for the attention mechanism module and fuses the different weights to obtain a complete characteristic diagram C; the feature map C is predicted, and a predicted value of 1 or 0 is output, 1 indicating the occurrence of a fire, and 0 indicating the absence of a fire.
In one embodiment, step S1 described above: collecting fire sample data under different working conditions, and labeling each sample with a flame or a flameless label; obtaining a sample set, wherein the sample set comprises five data: smoke data, temperature data, flame radiation data, visible light video data, and infrared video data, including in particular:
collecting fire sample data { s, t, f, v, in }; wherein s is smoke data, t is temperature data, f is flame radiation data, v is visible light video data, and in is infrared video data.
Fire flame and smoke data under the interference-free condition and flame and smoke data under the interference conditions of different ventilation, illumination, combustion objects and pedestrians are collected and marked. One of the interference items is whether a visible light video is shielded or not, when the visible light video data is completely black, namely a shielding object appears, but flame smoke exists at the moment, the video sample is also marked with flame, the flameless data is marked with 0, the flameless data is marked with 1, and sample data { s, t, f, v, in }; wherein s is smoke data, t is temperature data, f is flame radiation data, v is visible light video data, and in is infrared video data.
In one embodiment, step S2 above: preprocessing a sample set to obtain a preprocessed sample set; wherein the preprocessed sample set comprises: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, constructing a visible light video feature map sequence set V from visible light video data and an infrared video feature map sequence set I constructed from infrared video data, and specifically comprising:
step S21: for smoke Data s, temperature Data t and flame radiation Data f in one dimension in a sample set, constructing a two-dimensional array Data according to sampling time raw_data As shown in formula (1):
Figure BDA0003839866580000051
wherein S is rn For raw smoke data at time n, T rn For raw temperature data at time n, F rn Raw flame emission data for time n;
setting the maximum value corresponding to S, T and F as S max, T max, F max Carrying out normalization processing according to the formula (2) to obtain a normalized two-dimensional array shown in the formula (3), and obtaining a two-dimensional array X:
Figure BDA0003839866580000052
Figure BDA0003839866580000053
wherein S is sn For normalized smoke data at time n, T sn For normalized temperature data at time n, F sn Normalized flame emission data for time n;
in the step, normalization processing is carried out on the data format, and one-dimensional data in sample data is obtained: the smoke data S, the temperature data t and the flame radiation data f are all coded by decimal numbers, the values of the smoke data S, the temperature data t and the flame radiation data f respectively represent the smoke concentration, the temperature and the intensity of electromagnetic radiation, and a reasonable upper limit value is determined according to priori knowledge (experimental data) and is recorded as S max ,T max ,F max The one-dimensional data is converted into a two-dimensional array mode by using the formula (1), and the two-dimensional array is normalized by using the formula (2) to obtain a standardized two-dimensional array X by using the formula (3). Through this step, N one-dimensional data can be converted into R rows of two-dimensional data information of N/R elements per row.
Step S22: after the format, the frame rate and the video frame size of the acquired visible light video data V and infrared video data in are standardized, a visible light video feature map sequence set V and an infrared video feature map sequence set I are constructed, wherein each sample is a video frame sampling sequence with the length of N, and each video frame sampling sequence adjacent to the ith video frame in front of and behind is selected
Figure BDA0003839866580000054
Respectively constructing visible light video characteristic diagram sequences V by video frames i ＝{...n i-2 ,n i-1, n i ,n i+1 ,n i+2 .. sequence of infrared video signatures I } i ＝{...m i-2 ,m i-1, m i ,m i+1 ,m i+2 .., where i is the video frame number.
For the visible video data v and the infrared video data in the sample data, a sequence of video frame samples of length N is constructed, e.g. n=7 is set, i.e. 7 frames are taken per sample, i.e. for sample 1, for video frame N3The corresponding sequence of video feature maps: v (V) 3 = { n0, n1, n2, n3, n4, n5, n6}, and similarly, for sample 2, for video frame n4, the corresponding video feature map sequence is obtained: v (V) 4 = { n1, n2, n3, n4, n5, n6, n7}, and so on, a visible light video feature map sequence set V and an infrared video feature map sequence set I are constructed.
In one embodiment, the step S3: inputting the preprocessed sample set into a fire monitoring and identifying network for feature extraction: the two-dimensional array X is processed through a feature extraction network a to obtain a partial feature map C a The visible light video characteristic image sequence set V and the infrared video characteristic image sequence set I are respectively processed by video processing networks 1 and 2, and a characteristic extraction network b1 and a characteristic extraction network b2 to respectively obtain a visible light video part characteristic image C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I The attention mechanism module distributes different weights for the attention mechanism module and fuses the different weights to obtain a complete characteristic diagram C; predicting the feature map C, and outputting a predicted value of 1 or 0, wherein 1 indicates that a fire occurs, and 0 indicates that no fire occurs, and specifically comprises:
step S31: extracting deep features from the two-dimensional array X through a feature extraction network a based on a residual error network to obtain a partial feature map C a As shown in formulas (4) to (5):
f a (x)＝r(B(x)) (4)
C a ＝d(D a (f a (x)))·(conv a 1×1) (5)
wherein f a (x) Extracting a convolution unit of the network a for the feature, and regularizing Batch Normalization in batches B (x); r (x) is a ReL U function; d (D) a (x) D (x) deconvolution function conv for features extracted via residual network a a 1×1 is a convolution kernel of 1×1 in the feature extraction network a;
as shown in fig. 2, a schematic structural diagram of a feature extraction network a is shown, in which the convolution unit includes: convolution operation, batch regularization BN and correction unit ReLU function;
step S32: the visible light video frame n at the moment i i And infrared video frames , m i Features are extracted by convolution blocks a in video processing networks 1 and 2, respectively, as shown in formulas (6) to (7):
V self ＝n·conv b1a (6)
I self ＝m·conv b2a (7)
wherein n is the current video frame in the visible light V video sequence, m is the current video frame in the infrared video I video sequence, conv b1a For convolving blocks a, conv in the video processing network 1 b2a Is a convolution block a in the video processing network 2;
for visible video frame n at instant i i And infrared video frames , m i Is the video frame at the current moment, namely, is selected as the target; and the function of the front frame and the rear frame is to provide the required characteristic information for the current frame, and the current frame is a reference frame. Therefore, IN the processing of the target frame, a convolution block a with a small number of convolution kernels and a small convolution kernel is selected, wherein the convolution block a comprises a convolution kernel of 3×3 and a convolution kernel of 1×1, a Instance Normalization (IN) normalization operation and a ReLU activation function are added after each convolution kernel, the 3×3 convolution kernel is utilized to promote the fusion of the video frame through the convolution operation, the 1×1 convolution kernel is utilized to introduce a nonlinear relation for the operation of the coiler, and meanwhile, the channel number of the feature map is flexibly generated.
As shown in fig. 3A, a schematic structure of the convolution block a is shown; the structures of convolution blocks a in the video processing networks 1 and 2 are the same;
step S33: the visible light video frame n adjacent to the moment i k And infrared video frame m k Features are extracted by convolution blocks b in the video processing networks 1 and 2, respectively, as shown in formulas (8) to (11):
Figure BDA0003839866580000071
Figure BDA0003839866580000072
Figure BDA0003839866580000073
Figure BDA0003839866580000074
wherein n is k Video frames numbered k, m in visible video sequence V k K video frames numbered in the infrared video sequence I; conv b1b For convolving blocks b, conv in the video processing network 1 b2b Is a convolution block b in the video processing network 2;
and n i And , m i front-back adjacent visible light video frame n k And infrared video frame m k As a reference frame of the current frame, a convolution block b having a large number and a large convolution kernel is used, wherein the convolution block b comprises four convolution kernels of 9×9,7×7,3×3 and 1×1, each convolution kernel is added with IN and ReLU, firstly, two large convolution kernels of 9×9 and 7×7 are used for adding the receptive field, then 3×3 convolution kernels are used for extracting details, and the convolution kernels of 1×1 introduce a linear relation and perform dimension reduction processing, so that the same characteristic information as the reference frame is better extracted. And finally, fusing the processed new video frames to obtain video data to be processed.
As shown in fig. 3B, a schematic structure of the convolution block B is shown; the structure of the convolution block b in the video processing networks 1 and 2 is the same;
step S34: fusing the results of steps S32 and S33 as shown in formulas (12) to (13):
V data ＝V adjacent 1 +V self +V adjacent 2 (12)
I data ＝I adjacent 1 +I self +I adjacent 2 (13)
fig. 4 is a schematic diagram showing the structure of the video processing networks 1 and 2; wherein, video frame 2 is the current frame, video 1, 3 are reference frames;
step S35: fusing the characteristic diagram V data And I data After the characteristics are respectively extracted through a characteristic extraction network b1 based on a residual network and a characteristic extraction network b2 based on the residual network, a partial characteristic diagram C is generated through a convolution kernel of a deconvolution function d (x) and 1 multiplied by 1 v And C I As shown in formulas (14) to (15):
C v ＝d(D v (x))·(conv b1 1×1) (14)
C I ＝d(D I (x))·(conv b2 1×1) (15)
wherein x is the input feature map V data And I data ；D v (x) D is a feature extracted via a residual network I (x) Is a feature extracted via a residual network; conv b1 1×1 is a convolution kernel of 1×1 in the feature extraction network b 1; conv b2 1×1 is a convolution kernel of 1×1 in the feature extraction network b 2;
fusing the characteristic diagram V data And I data The characteristic map is obtained by respectively passing through a characteristic extraction network b1 based on a residual network and a characteristic extraction network b2 based on the residual network, firstly passing through a convolution unit, wherein the convolution unit comprises 9 multiplied by 9 convolution kernels, IN and ReLU are subjected to fusion operation, then entering the residual network b1/b2 to extract depth characteristics, adding the extracted depth residual with initial information, and then utilizing deconvolution operation and the convolution kernels of 1 multiplied by 1.
Fig. 5 is a schematic structural diagram of a feature extraction network b1 based on a residual network and a feature extraction network b2 based on the residual network;
FIG. 6 is a schematic diagram of the structure of the residual network b1/b2, where an embodiment of the present invention modifies the original network structure on the deep residual network (ResNet-101). Adding skip units in the outputs of residual units res2 and res3 to obtain a fusion characteristic diagram 1, and adding skip units in the outputs of residual units res4 and res5 to obtain a fusion characteristic diagram 2, wherein in order to unify the dimensions of the outputs of the residual units, a convolution layer is added at the tail end of each residual unit. After a transposed layer is added behind the fusion feature map 2, fusion is carried out with the fusion feature map 1, and after that, output is carried out, and an average pooling layer and a full connection layer of the original network are deleted.
FIG. 7 is a block diagram of an improved bottleneck structure of a depth residual network (ResNet-101), compared with an original structure, an average pooling layer with the size of 3×3 and the step length of 2 is introduced on a main trunk of the bottleneck structure, complete information transmission is ensured on the premise that the feature images are the same in size, and then a convolution layer with the size of 1×1 and the step length of 1 and a BN layer are connected; and replacing the 1X 1 convolution dimensionality reduction of the first layer with the 3X 3 convolution dimensionality reduction and the stride with 1 on a residual branch, and extracting most important features while reducing the dimensionality.
Step S36: part of characteristic diagram C a And partial feature map C v And C I The integrated feature map C is shown as a formula (16), and different weights are distributed to different private features by using an attention mechanism module, as shown as a formula (17):
C＝C a +C v +C I (16)
F ex (c,W)＝g(r(c,W))＝g(W 2 r(W 1 c)) (17)
wherein C represents a complete feature map, C represents feature map data, W 1 And W is 2 Parameters respectively representing two full link layers, r (x) represents a ReLU function, and g (x) represents a Sigmoid function;
the attention mechanism module in the embodiment of the invention comprises a branch for dynamically generating weights, the input feature images are globally pooled to obtain one-dimensional vectors with the dimension identical to the dimension of the input feature images, then vector information is compressed by using a full connection layer 1 to perform node number compression operation, the vector information is expanded by a full connection layer 2 to recover the original node number, and finally scalar weights in the weight vectors and the input feature images are weighted to obtain different strengthening processing results of each feature channel.
As shown in fig. 8, a complete fire monitoring and identification network structure is shown.
Step S37: the feature map C is predicted, and a predicted value of 1 or 0 is output, 1 indicating the occurrence of a fire, and 0 indicating the absence of a fire.
The invention collects multi-mode fire monitoring data containing five information of temperature, smoke concentration, flame radiation, visible light video data and infrared video data, and constructs a fire data set for synchronously monitoring the five data; in the fire monitoring and identifying network structure, N one-dimensional data can be converted into R rows by adopting the design that the data are converted into image signals after being piled up, and the two-dimensional data of N/R elements in each row are used for extracting information; the video processing network for processing the current video frame and the adjacent video frame by adopting different convolution kernels can convert three-dimensional data into two-dimensional data to extract space-time information contained in the video; adopting an improved residual error network structure, wherein the improved residual error network structure comprises a network model fusing a shallow layer and two parts of contents of an improved bottleneck structure; the attention mechanism is adopted to carry out weight distribution on different channel information, so that different types of data can be better fused, the alarm can be carried out more quickly, more time is striven for later fire protection early warning, and the fire loss is reduced.
Example two
As shown in fig. 9, the embodiment of the invention provides a multi-source data analysis fire monitoring and recognition system based on deep learning, which comprises the following modules:
a multi-modal data acquisition unit 1 for acquiring data comprising: multiplex data of an electric smoke sensor, a temperature-sensing fire detector, a flame radiation detector, a visible light video camera and an infrared video camera;
the multi-mode data acquisition unit comprises a plurality of sensors and is used for acquiring data of a fire disaster monitoring site, the sampling frequency can be set according to actual requirements, and the acquired data is sent to the signal input unit through an interface;
the signal input unit 2 comprises a plurality of serial interfaces and a plurality of video signal interfaces and is used for receiving the multipath data acquired by the multimode data acquisition unit;
the data processing unit 3, based on the method in claim 1, processes the multipath data input by the signal input interface unit in real time, performs real-time data preprocessing, and performs feature extraction by using the trained fire monitoring and recognition network to obtain a flame detection result;
the module comprises a processor, can process multiple paths of input data in real time, comprises one-dimensional sensor signals and two-dimensional video image signals, can perform real-time signal preprocessing and signal analysis based on a deep learning algorithm, and has multiple paths of serial data output;
a signal output unit 4 for outputting a flame detection result;
the output result is 1 or 0,1 indicates that a fire disaster occurs, and a signal is transmitted to an alarm unit for alarm; if 0 indicates that no fire occurs, continuing fire monitoring;
the alarm unit 5 comprises an LED lamp and a buzzer and is used for giving a fire alarm;
if the signal output unit outputs a signal of 1, the fire disaster is indicated, the alarm model is driven, and sound and flickering alarm of the LED lamp are generated.
Power module 6: the system is used for providing power for the multi-mode data acquisition unit, the data processing unit and the alarm unit.
The invention discloses a multi-source data analysis fire monitoring and identifying system based on deep learning, which improves the practicability of the system and is beneficial to the later equipment transformation through the design of independent coupling of multiple interfaces and functional modules. In the data processing part, the information analysis is carried out by using a deep learning method, so that the fire detection and recognition are realized, and the intellectualization of the fire-fighting equipment is promoted.
The above examples are provided for the purpose of describing the present invention only and are not intended to limit the scope of the present invention. The scope of the invention is defined by the appended claims. Various equivalents and modifications that do not depart from the spirit and principles of the invention are intended to be included within the scope of the invention.

Claims (5)

1. The multi-source data analysis fire monitoring and identifying method based on deep learning is characterized by comprising the following steps:
step S1: collecting fire sample data under different working conditions, and labeling each sample with a flame or a flameless label; obtaining a sample set, wherein the sample set comprises five data: smoke data, temperature data, flame radiation data, visible light video data, and infrared video data;
step S2: preprocessing the sample set; obtaining a pretreated sample set; wherein the preprocessed sample set comprises: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, and constructing a visible light video feature map sequence set V and an infrared video feature map sequence set I from visible light video data;
step S3: inputting the preprocessed sample set into a fire monitoring and identifying network for feature extraction: the two-dimensional array X is processed through a feature extraction network a to obtain a partial feature map C a The visible light video characteristic image sequence set V and the infrared video characteristic image sequence set I are respectively processed by video processing networks 1 and 2, and a characteristic extraction network b1 and a characteristic extraction network b2 to respectively obtain a visible light video part characteristic image C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I The attention mechanism module distributes different weights for the attention mechanism module and fuses the different weights to obtain a complete characteristic diagram C; and predicting the characteristic map C, and outputting a predicted value of 1 or 0, wherein 1 represents the occurrence of fire and 0 represents the absence of fire.
2. The deep learning-based multi-source data analysis fire monitoring and recognition method according to claim 1, wherein the step S1: collecting fire sample data under different working conditions, and labeling each sample with a flame or a flameless label; obtaining a sample set, wherein the sample set comprises five data: smoke data, temperature data, flame radiation data, visible light video data, and infrared video data, including in particular:
collecting fire sample data { s, t, f, v, in }; wherein s is smoke data, t is temperature data, f is flame radiation data, v is visible light video data, and in is infrared video data.
3. The deep learning-based multi-source data analysis fire monitoring and recognition method according to claim 2, wherein the step S2: preprocessing the sample set to obtain a preprocessed sample set; wherein the preprocessed sample set comprises: constructing a two-dimensional array X from smoke data, temperature data and flame radiation data, constructing a visible light video feature map sequence set V from visible light video data and an infrared video feature map sequence set I constructed from infrared video data, and specifically comprising:
step S21: for the smoke Data s, the temperature Data t and the flame radiation Data f in one dimension in the sample set, constructing a two-dimensional array Data according to the sampling time raw_data As shown in formula (1):
Figure FDA0003839866570000021
wherein S is rn For raw smoke data at time n, T rn For raw temperature data at time n, F rn Raw flame emission data for time n;
setting the maximum value corresponding to S, T and F as S max ,T max ,F max Carrying out normalization processing according to the formula (2) to obtain a normalized two-dimensional array X shown in the formula (3):
Figure FDA0003839866570000022
Figure FDA0003839866570000023
wherein S is sn For normalized smoke data at time n, T sn For normalized temperature data at time n, F sn Normalized flame emission data for time n;
step S22: the format, the frame rate and the video frame size of the acquired visible light video data v and the acquired infrared video data in are standardized, and then the visible light video data v and the infrared video data in are constructedEstablishing a video characteristic diagram sequence, wherein each sample is a video frame sampling sequence with the length of N, and selecting each adjacent to the ith video frame in front of and behind
Figure FDA0003839866570000024
Respectively constructing visible light video characteristic diagram sequences V by video frames i ＝{...n i-2 ,n i-1 ,n i ,n i+1 ,n i+2 .. sequence of infrared video signatures I } i ＝{...m i-2 ,m i-1 ,m i ,m i+1 ,m i+2 .., where i is the video frame number.
4. The deep learning-based multi-source data analysis fire monitoring and recognition method according to claim 3, wherein the step S3: inputting the preprocessed sample set into a fire monitoring and identifying network for feature extraction: the two-dimensional array X is processed through a feature extraction network a to obtain a partial feature map C a The visible light video characteristic image sequence set V and the infrared video characteristic image sequence set I are respectively processed by video processing networks 1 and 2, and a characteristic extraction network b1 and a characteristic extraction network b2 to respectively obtain a visible light video part characteristic image C v And an infrared video partial feature map C I The method comprises the steps of carrying out a first treatment on the surface of the C is C a And C v 、C I The attention mechanism module distributes different weights for the attention mechanism module and fuses the different weights to obtain a complete characteristic diagram C; predicting the feature map C, and outputting a predicted value of 1 or 0, wherein 1 indicates that a fire occurs, and 0 indicates that no fire occurs, and specifically includes:
step S31: extracting deep features from the two-dimensional array X through a feature extraction network a based on a residual error network to obtain a partial feature map C a As shown in formulas (4) to (5):
f a (x)＝r(B(x)) (4)
C a ＝d(D a (f a (x)))·(conv a 1×1) (5)
wherein f a (x) Convolution unit of network a for extracting said featuresB (x) batch regularization Batch Normalization; r (x) is a ReL U function; d (D) a (x) D (x) deconvolution function conv for features extracted via residual network a a 1×1 is a convolution kernel of 1×1 in the feature extraction network a;
step S32: the visible light video frame n at the moment i i And infrared video frames , m i Features are extracted by convolution blocks a in video processing networks 1 and 2, respectively, as shown in formulas (6) to (7):
V self ＝n·conv b1a (6)
I self ＝m·conv b2a (7)
wherein n is the current video frame in the visible light V video sequence, m is the current video frame in the infrared video I video sequence, conv b1a For convolving blocks a, conv in the video processing network 1 b2a Is a convolution block a in the video processing network 2;
step S33: the visible light video frame n adjacent to the moment i k And infrared video frame m k Features are extracted by convolution blocks b in the video processing networks 1 and 2, respectively, as shown in formulas (8) to (11):
Figure FDA0003839866570000031
Figure FDA0003839866570000032
Figure FDA0003839866570000033
Figure FDA0003839866570000034
wherein n is k Visible light video sequence V i Video frame numbered k, m k For infrared video sequence I i The middle number is k video frames; conv b1b For convolving blocks b, conv in the video processing network 1 b2b Is a convolution block b in the video processing network 2;
step S34: fusing the results of steps S32 and S33 as shown in formulas (12) to (13):
V data ＝V adjacent 1 +V self +V adjacent 2 (12)
I data ＝I adjacent 1 +I self +I adjacent 2 (13)
step S35: fusing the characteristic diagram V data And I data After the characteristics are respectively extracted through a characteristic extraction network b1 based on a residual network and a characteristic extraction network b2 based on the residual network, a partial characteristic diagram C is generated through a convolution kernel of a deconvolution function d (x) and 1 multiplied by 1 v And C I As shown in formulas (14) to (15):
C v ＝d(D v (x))·(conv b1 1×1) (14)
C I ＝d(D I (x))·(conv b2 1×1) (15)
wherein x is the input feature map V data And I data ；D v (x) D is a feature extracted via a residual network I (x) Is a feature extracted via a residual network; conv b1 1×1 is a convolution kernel of 1×1 in the feature extraction network b 1; conv b2 1×1 is a convolution kernel of 1×1 in the feature extraction network b 2;
step S36: part of characteristic diagram C a And partial feature map C v And C I The integrated feature map C is shown as a formula (16), and different weights are distributed to different private features by using an attention mechanism module, as shown as a formula (17):
C＝C a +C v +C I (16)
F ex (c,W)＝g(r(c,W))＝g(W 2 r(W 1 c)) (17)
wherein C represents a complete feature map, C represents feature map data, W 1 And W is 2 Parameters respectively representing two full link layers, r (x) represents a ReLU function, and g (x) represents a Sigmoid function;
step S37: and predicting the characteristic map C, and outputting a predicted value of 1 or 0, wherein 1 represents the occurrence of fire and 0 represents the absence of fire.
5. A deep learning-based multi-source data analysis fire monitoring and recognition system, comprising the following modules:
the multimode data acquisition unit is used for acquiring the data comprising: multiplex data of an electric smoke sensor, a temperature-sensing fire detector, a flame radiation detector, a visible light video camera and an infrared video camera;
the signal input unit comprises a plurality of serial interfaces and a plurality of video signal interfaces and is used for receiving the multipath data acquired by the multimode data acquisition unit;
the data processing unit is used for processing the multipath data input by the signal input interface unit in real time based on the method in claim 1, preprocessing the real-time data, and extracting features by using a trained fire monitoring and identifying network to obtain a flame detection result;
the signal output unit is used for outputting a flame detection result;
the alarm unit comprises an LED lamp and a buzzer and is used for giving a fire alarm;
and a power supply module: the multi-mode data acquisition unit, the data processing unit and the alarm unit are used for providing power supply for the multi-mode data acquisition unit, the data processing unit and the alarm unit.
CN202211099842.8A 2022-09-09 2022-09-09 Multi-source data analysis fire monitoring and identifying method and system based on deep learning Pending CN116310927A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202211099842.8A CN116310927A (en) 2022-09-09 2022-09-09 Multi-source data analysis fire monitoring and identifying method and system based on deep learning

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202211099842.8A CN116310927A (en) 2022-09-09 2022-09-09 Multi-source data analysis fire monitoring and identifying method and system based on deep learning

Publications (1)

Publication Number Publication Date
CN116310927A true CN116310927A (en) 2023-06-23

Family

ID=86778481

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202211099842.8A Pending CN116310927A (en) 2022-09-09 2022-09-09 Multi-source data analysis fire monitoring and identifying method and system based on deep learning

Country Status (1)

Country Link
CN (1) CN116310927A (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117010532A (en) * 2023-10-07 2023-11-07 电子科技大学 Comprehensive pipe gallery fire trend prediction method based on multi-mode deep learning

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117010532A (en) * 2023-10-07 2023-11-07 电子科技大学 Comprehensive pipe gallery fire trend prediction method based on multi-mode deep learning
CN117010532B (en) * 2023-10-07 2024-02-02 电子科技大学 Comprehensive pipe gallery fire trend prediction method based on multi-mode deep learning

Similar Documents

Publication Publication Date Title
Toulouse et al. Computer vision for wildfire research: An evolving image dataset for processing and analysis
KR101237089B1 (en) Forest smoke detection method using random forest classifier method
CN111739250B (en) Fire detection method and system combining image processing technology and infrared sensor
Zhan et al. A high-precision forest fire smoke detection approach based on ARGNet
CN116310927A (en) Multi-source data analysis fire monitoring and identifying method and system based on deep learning
CN115294528A (en) Pedestrian safety monitoring method and device
Zheng et al. A forest fire smoke detection model combining convolutional neural network and vision transformer
Jaradat et al. A victims detection approach for burning building sites using convolutional neural networks
Zhao et al. FSDF: A high-performance fire detection framework
CN103020588B (en) Based on the flame detecting method of video image analysis
Agrafiotis et al. Real time earthquake's survivor detection using a miniaturized LWIR camera
CN112488213A (en) Fire picture classification method based on multi-scale feature learning network
Kinaneva et al. An artificial intelligence approach to real-time automatic smoke detection by unmanned aerial vehicles and forest observation systems
CN103020587B (en) Based on video image analysis flame regarding figure analysis method
Dilli et al. Early Thermal Forest Fire Detection using UAV and Saliency map
CN116546287A (en) Multi-linkage wild animal online monitoring method and system
Hanamaraddi A literature study on image processing for forest fire detection
CN114170548A (en) Oil field on-site micro-target detection method and system based on deep learning
Ginkal et al. Forest Fire Detection using AI.
CN110991243A (en) Straw combustion identification method based on combination of color channel HSV and convolutional neural network
CN117409191B (en) Fire inspection early warning method based on unmanned aerial vehicle and improved YOLOv8 target detection algorithm
Baldota et al. Multimodal Wildland Fire Smoke Detection
Qiao et al. FireFormer: an efficient Transformer to identify forest fire from surveillance cameras
Vu et al. Investigation of Deep Learning Method for Fire Detection from Videos
Ko et al. Fire and Smoke Image Recognition

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination