CN116977909B - Deep learning fire intensity recognition method and system based on multi-modal data - Google Patents

Deep learning fire intensity recognition method and system based on multi-modal data Download PDF

Info

Publication number
CN116977909B
CN116977909B CN202311226491.7A CN202311226491A CN116977909B CN 116977909 B CN116977909 B CN 116977909B CN 202311226491 A CN202311226491 A CN 202311226491A CN 116977909 B CN116977909 B CN 116977909B
Authority
CN
China
Prior art keywords
data
image
fire
deep learning
images
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202311226491.7A
Other languages
Chinese (zh)
Other versions
CN116977909A (en
Inventor
施朦
李汉博
章志超
张文科
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
South Central Minzu University
Original Assignee
South Central University for Nationalities
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by South Central University for Nationalities filed Critical South Central University for Nationalities
Priority to CN202311226491.7A priority Critical patent/CN116977909B/en
Publication of CN116977909A publication Critical patent/CN116977909A/en
Application granted granted Critical
Publication of CN116977909B publication Critical patent/CN116977909B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/72—Data preparation, e.g. statistical preprocessing of image or video features
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
    • G06V10/809—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data
    • G06V10/811—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of classification results, e.g. where the classifiers operate on the same input data the classifiers operating on different input data, e.g. multi-modal recognition
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • General Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Molecular Biology (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Image Analysis (AREA)

Abstract

The invention belongs to the technical field of fire intensity recognition, and particularly discloses a deep learning fire intensity recognition method and system based on multi-mode data, wherein the method comprises the following steps: collecting fire scene data; preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model; extracting multi-mode data characteristics; reorganizing the multi-mode information features by adopting a graph structure; and inputting the information characteristics into a deep learning model, and obtaining a predicted value of the heat release rate through forward propagation calculation of the deep learning model to finish fire intensity identification. The invention utilizes the deep learning technology to extract valuable information from the fire scene and related data so as to judge the fire intensity more accurately and more quickly and improve the accuracy of fire cause analysis, thereby providing more powerful support for fire prevention and control and prevention.

Description

Deep learning fire intensity recognition method and system based on multi-modal data
Technical Field
The invention relates to the technical field of fire intensity recognition, in particular to a deep learning fire intensity recognition method and system based on multi-mode data.
Background
Fire investigation is an emerging edge intersection subject, and has wide coverage, and mainly researches fire cause investigation, fire loss verification, fire responsibility identification and treatment, fire investigation inquiry and interrogation technology, fire trace material evidence technology identification and the like. The current investigation of the cause of fire and the verification of fire loss are mainly based on manual investigation and depend on the working experience of investigation staff. In order to solve the problem, the invention recognizes the trace of the fire scene by a deep learning method, reversely predicts the heat release rate in the fire process and provides more accurate quantification standard for investigators.
The deep learning model can be simply understood as a neural network model with a large number of hidden layers, can reveal a nonlinear relation between an input value and an output value, is a powerful tool in the field of data recognition, and is widely applied to tasks such as image recognition and voice recognition. The deep learning model may be divided into three layers, an input layer, an output layer, and a hidden layer. The input layer will receive sample data, which may be images, audio, text, etc. The output layer may output target data required for the problem. The hidden layer is arranged between the input layer and the output layer, contains a large number of parameters, and is continuously optimized by learning the target data, so that the effect of converting the input data into accurate target data is finally achieved. The more the number of hidden layers, the more the parameters, and thus the higher the complexity of the model, the more complex learning tasks can be accomplished.
Modalities refer to forms in which information exists or sources of information, and data composed of two or more modalities is referred to as multi-modal data. Multimodal data, which is common in the field of artificial intelligence, is images, speech, text. In a fire scene, multi-mode fire data information can be formed due to different data acquisition modes.
The heat release rate is generally used for representing the combustion intensity of fire, and the heat release rate per unit area is commonly used in practical application and is expressed in kW/m 2 . In performing fire hazard analysis, it is necessary to determine how much the heat release rate of a fire is, and how the heat release rate changes. The heat release rate can generally be calculated from the mass burning rate of the combustible, but the burning in a fire is generally incompleteFull combustion, therefore the calculation accuracy of the method is not high. Therefore, a method for efficiently and accurately calculating the heat release rate of a fire is needed.
After a fire occurs, fire extinguishing and investigation and evidence collection should be performed simultaneously. However, the current focus of fire work is how to extinguish the fire and ensure the safety of personnel. Only in the case of severe fire, serious damage to life and property occurs, but investigation of the cause and responsibility of the fire is emphasized. With the continuous development of society and technology, various new electronic products are continuously appeared, and new processes and new materials are applied, so that the cause of fire is increasingly complex, the concealment is stronger, and the requirements on detection equipment are also higher. Therefore, the problems of low efficiency and high cost occur.
Currently, in the fire field, applications of intelligent technologies are frequent, such as fire early warning systems, fire location recognition technologies, intelligent fire extinguishing devices, and the like. The advanced intelligent technologies play an important role before and during the occurrence of the fire, effectively inhibit the spread of the fire and reduce the damage of the fire. However, in the fire investigation phase after the end of the fire, there is little application of intelligent technology, especially in judging the intensity of the fire, and the traditional investigation method is still mainly relied on. This is not only inefficient, but also often makes it difficult to accurately determine the intensity of the fire due to the influence of human factors, thereby affecting the accuracy of analysis of the cause of the fire and the effectiveness of the investigation results. Therefore, developing and applying intelligent technologies, such as machine learning, deep learning, etc., to improve accuracy and efficiency of fire intensity judgment is a problem that needs to be solved in the current fire investigation field.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a multi-mode data-based deep learning fire intensity recognition method and system, which utilize a deep learning technology to extract valuable information from a fire scene and related data so as to judge the fire intensity more accurately and more rapidly. In addition, the fire intensity judging method based on deep learning can provide more analysis and judging basis for fire investigators, and improve the accuracy of fire cause analysis, thereby providing more powerful support for fire prevention and control and prevention and solving the problems in the background technology.
In order to achieve the above purpose, the present invention provides the following technical solutions: a deep learning fire intensity recognition method based on multi-mode data comprises the following steps:
s1, acquiring fire scene data;
s2, preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model;
s3, extracting multi-mode data characteristics;
s4, reorganizing the multi-mode information features by adopting a graph structure;
s5, inputting the information features into a deep learning model, and obtaining a predicted value of the heat release rate through forward propagation calculation of the deep learning model to finish fire intensity recognition.
Preferably, in step S1, fire scene data including, but not limited to, smoke trace information, glass trace information, metal discoloration trace information, and construction information is collected using cameras, sensors, and environmental monitoring equipment; including but not limited to images, video recordings, and building geometry parameters.
Preferably, in step S2, the sensor data preprocessing includes:
s211, data cleaning: dividing the data area by adopting a clustering algorithm, further carrying out standard deviation statistics on the divided subareas, calculating the standard deviation ratio of the data points to the mean value of the data points, and considering the data exceeding a certain threshold value as an abnormal value;
for abnormal values, the following two processing methods are adopted: 1) The sensor is used for acquiring data secondarily, and the abnormal value is corrected; 2) Correcting the data by adopting an interpolation or data smoothing method;
s212, data standardization: the scale of the data is normalized and the data range is adjusted to 0, 1.
Preferably, in step S2, the image data preprocessing includes:
s221, normalization: firstly, converting an image into a gray level image, and red of each pixel point in the imageGreen->Blue->The value of (2) is converted into a luminance value +.>Luminance value +.>Normalized to [0,1]]Or [ -1,1]The numerical ranges among different images are consistent;
s222, cutting: removing unnecessary parts of the image, including but not limited to removing noise at edges of the image;
s223, scaling: the size of the image is adjusted, and the images with different resolutions are scaled to the same size, so that model processing is facilitated.
The video recording preprocessing comprises the following steps:
s231, video segmentation: dividing a continuous piece of video into a plurality of meaningful sub-pieces, which typically contain a particular scene, event, or action;
s232, frame sampling: using the Brenner gradient function to judge the image quality of each frame in the sub-segment, and extracting the frame with the highest value of the Brenner gradient function so as to reduce the calculated amount of data processing;
s233, transferring the extracted frames into an image data preprocessing operation.
Preferably, in step S3, extracting the multi-modal data feature specifically includes:
s31, extracting sensor data characteristics: extracting regional features of data using sliding windows to obtain multipleData characterization of regions；
S32, extracting image features, specifically comprising:
s321, performing convolution operation on a convolution layer after the image is input, wherein the formula is expressed as follows:
；
wherein,representing a convolved output profile, < >>Representing an input image +.>Representing convolution kernel +.>And->Representing the size of the convolution kernel;
s322, carrying out pooling operation on the feature map obtained after convolution in a pooling layer to reduce redundancy of data, wherein the formula is as follows:
；
wherein,representing a pooled output profile, < >>Representing a convolved output profile, < >>Representing pooling method, < >>Andrepresenting the size of the pooling window;
s323, outputting a characteristic diagram obtained after the last pooling layerFlattening it into a one-dimensional image feature input vector +.>。
Preferably, in step S4, the reorganizing the multi-modal information feature by using the graph structure specifically includes:
s41, reorganizing a sensor data characteristic diagram structure: defining a sensor data profile structure asWherein the set of sensor data features is the set of vertices in the graph structure +.>There is a relationship between vertices of adjacent regions，/>Is the set of relations in the graph structure +.>Any two adjacent vertexes are set asThe data characteristic relation between them +.>The computational expression is:
；
combining the sensor data features according to the calculated data feature relation, and calculating the combined sensor data feature valueThe formula is expressed as follows:
；
s42, image feature map structure reorganization: set up the collectionAny two images of (a) are +.>Calculating the feature relation Euclidean distance between images>，
；
Wherein,and->Representing the size of the image; will->A threshold value of correlation degree with a preset image +.>Comparing when->Is greater than->When in use, will->Record collection->Otherwise, will be->Discarding;
after the relation of any two images in the set is calculated, the graph structure of the input image is obtained；
Combining the image features according to the calculated image feature relation, and setting any two imagesCharacteristic values of +.>The degree of association of the two images is +.>The combined image feature value is calculated according to the following formula，
；
S43, characterizing the sensor dataImage feature input vector +.>And other quantized feature vectors->Combining to obtain the final productIs input vector of the full connection layer->，/>。
Preferably, in step S5, an activation function ReLU is added after the convolutional layer, the pooling layer, and the hidden layer of the deep learning model to increase the nonlinear recognition capability of the model, where the activation function is specifically expressed as follows:
；
in the fully connected layer of the deep learning model, vectors are inputThe final predicted result is obtained by the following calculation:
；
wherein,representing the output vector, i.e., the prediction result of the deep learning model on the fire intensity, +.>The weight matrix is represented by a matrix of weights,representing the bias term.
On the other hand, in order to achieve the above purpose, the present invention further provides the following technical solutions: a multi-modal data based deep learning fire intensity recognition system, the system comprising:
the data acquisition module acquires fire scene data;
the data preprocessing module is used for preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model;
the feature extraction module is used for extracting multi-mode data features;
the diagram structure reorganization module reorganizes the multi-mode information characteristics by adopting a diagram structure;
and the fire intensity recognition module inputs the information characteristics into the deep learning model, and obtains a predicted value of the heat release rate through forward propagation calculation of the deep learning model to complete fire intensity recognition.
The beneficial effects of the invention are as follows: the method is a more comprehensive and more effective artificial intelligent driving method for researching the relation between the fire trace and the heat release rate, and takes the data quality, the characteristic engineering and the model selection into consideration, so that more accurate fire intensity prediction can be provided. The specific technical effects are as follows:
1) And 3, improving the prediction precision: the deep learning model has strong feature extraction and data fitting capability, and can automatically learn differentiated features from complex fire scene data. This helps to improve the accuracy of fire intensity prediction and reduce the false positive rate.
2) The manual intervention is reduced: compared with the traditional fire intensity judging method, the deep learning model can automatically perform feature extraction and model training, and the necessity of manual intervention is reduced. This will reduce the time and labor costs in the fire investigation process.
3) Versatility and adaptability: the deep learning model can automatically learn data characteristics, so that the method has good universality and adaptability. This means that the model can be suitable for fire cases of different scenes and types, thereby providing a more comprehensive and reliable reference basis for fire investigation.
4) Promote the development of fire investigation technology: the deep learning model provides a new research direction and technical means for fire intensity identification, and is helpful for promoting technical innovation and progress in the field of fire investigation.
Drawings
FIG. 1 is a schematic flow chart of the steps of the method of the present invention;
FIG. 2 is a schematic diagram of a common fire trace image;
FIG. 3 is a schematic diagram of a deep learning model execution process;
FIG. 4 is a schematic diagram of a system module structure according to the present invention;
in the figure, 110 is a data acquisition module; 120-a data preprocessing module; 130-a feature extraction module; 140-diagram structure reorganization module; 150-fire intensity recognition module.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The intensity of the fire, i.e., the rate of heat release, is difficult to identify in fire surveys. Current fire investigation techniques are mainly focused on the detection of combustion residues, and the fire intensity is determined by analyzing the form and composition of the residues. This process is cumbersome and time consuming, and therefore, in many small fire surveys, this procedure is not performed. This results in a large amount of imperfect fire case data, which affects the accuracy of analysis of the cause of the fire and the validity of the survey results.
In order to solve this problem, we propose a fast and effective method for judging fire intensity, please refer to fig. 1-3, and the present invention provides a technical scheme: a deep learning fire intensity recognition method based on multi-mode data comprises the following steps:
s1, acquiring fire scene data.
Determining a data acquisition source, such as a camera, a sensor, an environmental monitoring device, etc., and acquiring fire scene data including, but not limited to, smoke trace information, glass trace information, metal discoloration trace information and building information by using the camera, the sensor, the environmental monitoring device, etc., as shown in fig. 2. Including but not limited to images, video recordings, and building geometry parameters.
The image record should take multiple angles of the fire scene including overall scene, trace details, residue details. The image needs to be kept clear, and the next image preprocessing step can be performed after the image quality is judged.
The image quality may be judged using a Brenner gradient function defined as follows:
；
wherein,represents the Brenner gradient value,>representing an imagefCorresponding pixel point is at coordinatesGray values at that point.
S2, preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model.
Sensor data preprocessing, in which information such as combustion residue concentration, soot deposit concentration and the like can be detected by a sensor in a fire scene, wherein the sensor data preprocessing comprises:
s211, data cleaning: check whether the data has outliers, missing values, noise, etc. Based on the actual condition that the concentration value of the residues does not change greatly in a certain area, firstly dividing a data area by adopting a K-mean clustering algorithm, further carrying out standard deviation statistics on the divided subareas, calculating the standard deviation ratio of data points to the mean value of the data points, and considering the data exceeding a certain threshold value as an abnormal value;
for abnormal values, the following two processing methods are adopted: 1) The sensor is used for acquiring data secondarily, and the abnormal value is corrected; 2) Correcting the data by adopting an interpolation or data smoothing method;
s212, data standardization: data normalization is a linear transformation process that is generally adapted to scale data, where data is normalized in multi-modal data to provide comparability between different types of data, and the range of data is scaled to 0,1 by the following equation.
；
Representing characteristic value raw data +_>Representing the minimum value of the characteristic values, +.>Represents the maximum value in the characteristic values,/->Representing the normalized feature data.
The image data preprocessing includes:
s221, normalization: firstly, converting an image into a gray level image, and red of each pixel point in the imageGreen->Blue->The value of (2) is converted into a luminance value +.>Luminance value +.>NormalizationTo [0,1]]Or [ -1,1]And the numerical ranges among different images are consistent.
The gray map conversion process adopts a Gamma correction method:
；
、/>、/>respectively representing the brightness values of the pixels red, green and blue.
S222, cutting: removing unnecessary parts of the image, including but not limited to removing noise at edges of the image;
s223, scaling: the size of the image is adjusted, and the images with different resolutions are scaled to the same size, so that model processing is facilitated. The present embodiment scales the image to 256×256 pixels.
The video recording preprocessing comprises the following steps:
s231, video segmentation: a continuous piece of video is segmented into meaningful sub-segments, which typically contain specific scenes, events, or actions.
In this embodiment, the video is divided by frame difference calculation, and it is assumed that any two adjacent frames in the video are respectivelyAndcalculating Euclidean distance of corresponding pixels of two frames>，/>And->Representing the size of the image, when +.>When the number is more than 50, two frames are considered to belong to different scenes, and +.>The frames divide the video and the divided video sub-segments are stored to wait for further processing.
S232, frame sampling: using the Brenner gradient function to judge the image quality of each frame in the sub-segment, and extracting the frame with the highest value of the Brenner gradient function so as to reduce the calculated amount of data processing;
s233, transferring the extracted frames into an image data preprocessing operation.
S3, extracting multi-mode data features.
Further, extracting the multi-modal data features specifically includes:
s31, extracting sensor data characteristics:
sliding windows are used to capture local variations of the data, and statistical features within each window, such as mean, variance, etc., are calculated.
The traditional method usually uses a sliding window to extract time sequence data information, but the data on site after fire disaster often has no time sequence but regional, so the invention uses the sliding window to extract the regional characteristics of the data to obtain the data characteristics of a plurality of regions；
S32, extracting image features, specifically comprising:
s321, performing convolution operation on a convolution layer after the image is input, wherein the formula is expressed as follows:
；
wherein,representing a convolved output profile, < >>Representing an input image +.>Representing convolution kernel +.>And->Indicating the size of the convolution kernel.
Assuming that a convolution kernel of size 7 x 7 is used for the operation, the formula is written as:
；
wherein,representing a convolved output profile, < >>Representing an input image +.>Representing the convolution kernel.
S322, carrying out pooling operation on the feature map obtained after convolution in a pooling layer to reduce redundancy of data, wherein the formula is as follows:
；
assuming a pooling window of size 3 x 3 is employed, the formula is written as:
wherein,representing a pooled output profile, < >>Representing a convolved output profile, < >>Representing pooling method, < >>Andrepresenting the size of the pooling window; except that the first layer is a convolution layer and the last layer is a pooling layer, the convolution layer and the pooling layer can be repeatedly overlapped between the convolution layers.
S323, outputting a characteristic diagram obtained after the last pooling layerFlattening it into a one-dimensional image feature input vector +.>。
Since a plurality of region features are extracted from the sensor data and a plurality of image features are obtained in the case where a plurality of images are input, a plurality of region features or image features are required to be reorganized to enhance the correlation between the plurality of region features or image features and reduce the calculation pressure. The present invention reorganizes features using a graph structure, which is generally defined as，/>Representing the diagram structure->Set of middle vertices>Representing the diagram structure->Is a set of relationships in the set.
S4, reorganizing the multi-mode information features by adopting a graph structure.
Further, the reorganizing the multi-mode information features by using the graph structure specifically includes:
s41, reorganizing a sensor data characteristic diagram structure: defining a sensor data profile structure asWherein the set of sensor data features is the set of vertices in the graph structure +.>There is a relationship between vertices of adjacent regions，/>Is the set of relations in the graph structure +.>Any two adjacent vertexes are set asThe data characteristic relation between them +.>The computational expression is:
；
combining the sensor data features according to the calculated data feature relation, and calculating the combined sensor data feature valueFormula tableThe method comprises the following steps:
；
s42, image feature map structure reorganization: set up the collectionAny two images +.>Calculating the feature relation Euclidean distance between images>，
；
Wherein,and->Representing the size of the image; will->A threshold value of correlation degree with a preset image +.>Comparing when->Is greater than->When in use, will->Record collection->Otherwise, will be->Discarding;
after the relation of any two images in the set is calculated, the graph structure of the input image is obtained；
Combining the image features according to the calculated image feature relation, and setting any two imagesCharacteristic values of +.>The degree of association of the two images is +.>The combined image feature value is calculated according to the following formula，
；
S43, characterizing the sensor dataImage feature input vector +.>And other quantized feature vectors->Combining to obtain the final full connection layer input vector +.>，/>。/>May be, for example, building size information, weather environment information vectors, etc.
S5, inputting the information features into a deep learning model, and obtaining a predicted value of the heat release rate through forward propagation calculation of the deep learning model to finish fire intensity recognition, wherein the predicted value is shown in FIG. 3.
The activation function can increase the nonlinear recognition capability of the model, prevent the problems of gradient explosion and gradient disappearance of the model in the training process, and can be selectively added after each layer before the output layer of the fully connected layer.
Further, in step S5, an activation function ReLU is added after the convolutional layer, the pooling layer, and the hidden layer of the deep learning model to increase the nonlinear recognition capability of the model, where the activation function is specifically expressed as follows:
；
in the fully connected layer of the deep learning model, vectors are inputThe final predicted result is obtained by the following calculation:
；
wherein,representing the output vector, i.e., the prediction result of the deep learning model on the fire intensity, +.>The weight matrix is represented by a matrix of weights,representing the bias term.
The invention utilizes the deep learning technology to extract valuable information from the fire scene and related data so as to judge the fire intensity more accurately and more quickly. In addition, the fire intensity judging method based on deep learning can provide more analysis and judging basis for fire investigators, and improve the accuracy of fire cause analysis, thereby providing more powerful support for fire prevention and control and prevention.
Based on the same inventive concept as the above method embodiment, the present application embodiment further provides a deep learning fire intensity recognition system based on multi-modal data, which can implement the functions provided by the above method embodiment, as shown in fig. 4, and the system includes:
a data acquisition module 110 for acquiring fire scene data;
the data preprocessing module 120 is used for preprocessing the collected fire scene data, including preprocessing sensor data, video recording and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model;
the feature extraction module 130 extracts multi-modal data features;
the graph structure reorganizing module 140 reorganizes the multi-modal information features by adopting a graph structure;
the fire intensity recognition module 150 inputs the information features into a deep learning model, and obtains a predicted value of the heat release rate through forward propagation calculation of the deep learning model, thereby completing fire intensity recognition.
Although the present invention has been described with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described, or equivalents may be substituted for elements thereof, and any modifications, equivalents, improvements and changes may be made without departing from the spirit and principles of the present invention.

Claims (6)

1. The deep learning fire intensity recognition method based on the multi-mode data is characterized by comprising the following steps of:
s1, acquiring fire scene data;
s2, preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model;
s3, extracting multi-mode data characteristics;
s4, reorganizing the multi-mode data features by adopting a graph structure;
s5, inputting the multi-mode data characteristics into a deep learning model, and obtaining a predicted value of a heat release rate through forward propagation calculation of the deep learning model to finish fire intensity identification;
in step S2, the image data preprocessing includes:
s221, normalization: firstly converting an image into a gray level image, converting the values of red R, green G and blue B of each pixel point in the image into a brightness value Y, and normalizing the brightness value Y to be between 0 and 1 or-1 and 1 so as to enable the numerical ranges of different images to be consistent;
s222, cutting: removing unnecessary parts of the image, including but not limited to removing noise at edges of the image;
s223, scaling: the size of the image is adjusted, and the images with different resolutions are scaled to the same size, so that model processing is facilitated;
the video recording preprocessing comprises the following steps:
s231, video segmentation: dividing a continuous piece of video into a plurality of meaningful sub-pieces, which typically contain a particular scene, event, or action;
s232, frame sampling: using the Brenner gradient function to judge the image quality of each frame in the sub-segment, and extracting the frame with the highest value of the Brenner gradient function so as to reduce the calculated amount of data processing;
s233, transferring the extracted frames into an image data preprocessing operation;
in step S4, the reorganizing the multi-modal information feature by using the graph structure specifically includes:
s41, reorganizing a sensor data characteristic diagram structure: defining the sensor data feature map structure as G s (V s ,E s )，Wherein the set of sensor data features is set V of vertexes in the graph structure s The set of relations E, E existing between the vertexes of the adjacent areas is the set E of relations in the graph structure s Let v be any two adjacent vertexes si ＝X si And v sj ＝X sj The data characteristic relation e between them sij The computational expression is:
combining the sensor data features according to the calculated data feature relation, and calculating a combined sensor data feature value X t The formula is expressed as follows:
s42, image feature map structure reorganization: let any two images in set V be V pi And v pj Calculating the Euclidean distance d of the characteristic relation between the images,
wherein A and Z represent the size of the image; will beComparing with a preset image association threshold r, when +.>When r is greater, will be->Record set E, otherwise +_>Discarding;
after the relation of any two images in the set is calculated, obtaining a graph structure G of the input image p (V p ,E p )；
Combining the image features according to the calculated image feature relation, and setting any two images v pi And v pj The characteristic values of (2) are X 'respectively' pi And X' pj The association degree of the two images is e pij The combined image feature value X is calculated according to the following formula p ，
S43, characterizing the sensor data X t Image feature input vector X p And other quantized feature vectors X q Merging to obtain the final full connection layer input vector X, x=x t ∪X p ∪X q 。
2. The multi-modal data-based deep learning fire intensity recognition method of claim 1, wherein: in step S1, acquiring fire scene data including, but not limited to, smoke trace information, glass trace information, metal discoloration trace information and building information by using cameras, sensors and environment monitoring equipment; including but not limited to images, video recordings, and building geometry parameters.
3. The multi-modal data-based deep learning fire intensity recognition method of claim 1, wherein: in step S2, the sensor data preprocessing includes:
s211, data cleaning: dividing the data area by adopting a clustering algorithm, further carrying out standard deviation statistics on the divided subareas, calculating the standard deviation ratio of the data points to the mean value of the data points, and considering the data exceeding a certain threshold value as an abnormal value;
for abnormal values, the following two processing methods are adopted: 1) The sensor is used for acquiring data secondarily, and the abnormal value is corrected; 2) Correcting the data by adopting an interpolation or data smoothing method;
s212, data standardization: the scale of the data is normalized and the data range is adjusted to 0, 1.
4. The multi-modal data-based deep learning fire intensity recognition method of claim 1, wherein: in step S3, extracting the multi-modal data features specifically includes:
s31, extracting sensor data characteristics: extracting region features of data using a sliding window to obtain data features X of multiple regions s ＝{x 1 ,…,x n }；
S32, extracting image features, specifically comprising:
s321, performing convolution operation on a convolution layer after the image is input, wherein the formula is expressed as follows:
wherein O represents a convolution output feature map, I represents an input image, K represents a convolution kernel, and H and W represent the sizes of the convolution kernels;
s322, carrying out pooling operation on the feature map obtained after convolution in a pooling layer to reduce redundancy of data, wherein the formula is as follows:
wherein P represents a pooling output feature map, O represents a convolution output feature map, P represents a pooling method, and M and N represent the sizes of pooling windows;
s323, flattening the output feature map P obtained after the last pooling layer to form a one-dimensional image feature input vector X' p 。
5. The multi-modal data-based deep learning fire intensity recognition method of claim 1, wherein: in step S5, an activation function ReLU is added after the convolutional layer, the pooling layer, and the hidden layer of the deep learning model to increase the nonlinear recognition capability of the model, where the activation function is specifically expressed as follows:
f(x)＝max(0,x)
in the fully connected layer of the deep learning model, the input vector X is calculated to obtain a final prediction result through the following formula:
C＝XT+b
wherein, C represents the prediction result of the output vector, namely the deep learning model, on the fire intensity, T represents the weight matrix, and b represents the bias term.
6. A multi-mode data-based deep learning fire intensity recognition system is characterized in that: the system comprises the following steps:
the data acquisition module (110) is used for acquiring fire scene data;
the data preprocessing module (120) is used for preprocessing the collected fire scene data, including preprocessing sensor data, video recordings and images, and converting the fire scene data in various data forms into a format which can be input into a deep learning model;
the image data preprocessing includes:
s221, normalization: firstly converting an image into a gray level image, converting the values of red R, green G and blue B of each pixel point in the image into a brightness value Y, and normalizing the brightness value Y to be between 0 and 1 or-1 and 1 so as to enable the numerical ranges of different images to be consistent;
s222, cutting: removing unnecessary parts of the image, including but not limited to removing noise at edges of the image;
s223, scaling: the size of the image is adjusted, and the images with different resolutions are scaled to the same size, so that model processing is facilitated;
the video recording preprocessing comprises the following steps:
s231, video segmentation: dividing a continuous piece of video into a plurality of meaningful sub-pieces, which typically contain a particular scene, event, or action;
s232, frame sampling: using the Brenner gradient function to judge the image quality of each frame in the sub-segment, and extracting the frame with the highest value of the Brenner gradient function so as to reduce the calculated amount of data processing;
s233, transferring the extracted frames into an image data preprocessing operation;
a feature extraction module (130) for extracting multi-modal data features;
the diagram structure reorganization module (140) reorganizes the multi-mode data characteristics by adopting a diagram structure; the method specifically comprises the following steps:
s41, reorganizing a sensor data characteristic diagram structure: defining the sensor data feature map structure as G s (V s ,E s ) Wherein the set of sensor data features is set V of vertices in the graph structure s The set of relations E, E existing between the vertexes of the adjacent areas is the set E of relations in the graph structure s Let v be any two adjacent vertexes si ＝X si And v sj ＝X sj The data characteristic relation e between them sij The computational expression is:
combining the sensor data features according to the calculated data feature relation, and calculating a combined sensor data feature value X t The formula is expressed as follows:
s42, image feature map structure reorganization: let any two images in set V be V pi And v pj Calculating the Euclidean distance d of the characteristic relation between the images,
wherein A and Z represent the size of the image; will beComparing with a preset image association threshold r, when +.>When r is greater, will be->Record set E, otherwise +_>Discarding;
after the relation of any two images in the set is calculated, obtaining a graph structure G of the input image p (V p ,E p )；
Combining the image features according to the calculated image feature relation, and setting any two images v pi And v pj The characteristic values of (2) are X 'respectively' pi And X' pj The association degree of the two images is e pij The combined image feature value X is calculated according to the following formula p ，
S43, characterizing the sensor data X t Image feature input vector X p And other quantized feature vectors X q Merging to obtain the final full connection layer input vector X, x=x t ∪X p ∪X q ；
And the fire intensity recognition module (150) inputs the multi-mode data characteristics into a deep learning model, and obtains a predicted value of the heat release rate through forward propagation calculation of the deep learning model to complete fire intensity recognition.
CN202311226491.7A 2023-09-22 2023-09-22 Deep learning fire intensity recognition method and system based on multi-modal data Active CN116977909B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311226491.7A CN116977909B (en) 2023-09-22 2023-09-22 Deep learning fire intensity recognition method and system based on multi-modal data

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311226491.7A CN116977909B (en) 2023-09-22 2023-09-22 Deep learning fire intensity recognition method and system based on multi-modal data

Publications (2)

Publication Number Publication Date
CN116977909A CN116977909A (en) 2023-10-31
CN116977909B true CN116977909B (en) 2023-12-19

Family

ID=88473314

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311226491.7A Active CN116977909B (en) 2023-09-22 2023-09-22 Deep learning fire intensity recognition method and system based on multi-modal data

Country Status (1)

Country Link
CN (1) CN116977909B (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117789185B (en) * 2024-02-28 2024-05-10 浙江驿公里智能科技有限公司 Automobile oil hole gesture recognition system and method based on deep learning
CN118172882A (en) * 2024-03-21 2024-06-11 中国矿业大学 Tunnel fire monitoring system based on deep learning

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014205231A1 (en) * 2013-06-19 2014-12-24 The Regents Of The University Of Michigan Deep learning framework for generic object detection
CN111625994A (en) * 2020-05-25 2020-09-04 齐鲁工业大学 Multi-source information fusion fire prediction method based on dynamic integrated neural network
CN113642475A (en) * 2021-08-17 2021-11-12 中国气象局上海台风研究所(上海市气象科学研究所) Atlantic hurricane intensity estimation method based on convolutional neural network model
CN114821157A (en) * 2022-04-01 2022-07-29 山东大学 Multi-modal image classification method based on hybrid model network

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7786877B2 (en) * 2008-06-20 2010-08-31 Billy Hou Multi-wavelength video image fire detecting system
US20210319324A1 (en) * 2021-06-25 2021-10-14 Intel Corporation Technology for memory-efficient and parameter-efficient graph neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014205231A1 (en) * 2013-06-19 2014-12-24 The Regents Of The University Of Michigan Deep learning framework for generic object detection
CN111625994A (en) * 2020-05-25 2020-09-04 齐鲁工业大学 Multi-source information fusion fire prediction method based on dynamic integrated neural network
CN113642475A (en) * 2021-08-17 2021-11-12 中国气象局上海台风研究所(上海市气象科学研究所) Atlantic hurricane intensity estimation method based on convolutional neural network model
CN114821157A (en) * 2022-04-01 2022-07-29 山东大学 Multi-modal image classification method based on hybrid model network

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Predicting real-time fire heat release rate by flame images and deep learning;Wang Zilong等;Proceedings of the Combustion Institute;第39卷;2-4 *
基于深度学习和最大相关最小冗余的火焰图像检测方法;李梓瑞;王慧琴;胡燕;卢英;;激光与光电子学进展(第10期);160-170 *

Also Published As

Publication number Publication date
CN116977909A (en) 2023-10-31

Similar Documents

Publication Publication Date Title
CN116977909B (en) Deep learning fire intensity recognition method and system based on multi-modal data
CN110232380B (en) Fire night scene restoration method based on Mask R-CNN neural network
CN110084165B (en) Intelligent identification and early warning method for abnormal events in open scene of power field based on edge calculation
CN112200043B (en) Intelligent danger source identification system and method for outdoor construction site
CN111754498B (en) Conveyor belt carrier roller detection method based on YOLOv3
WO2019101220A1 (en) Deep learning network and average drift-based automatic vessel tracking method and system
CN112183313B (en) SlowFast-based power operation field action identification method
CN106991668B (en) Evaluation method for pictures shot by skynet camera
CN114241548A (en) Small target detection algorithm based on improved YOLOv5
CN103517042A (en) Nursing home old man dangerous act monitoring method
CN101179713A (en) Method of detecting single moving target under complex background
CN110096945B (en) Indoor monitoring video key frame real-time extraction method based on machine learning
CN115995056A (en) Automatic bridge disease identification method based on deep learning
CN116259002A (en) Human body dangerous behavior analysis method based on video
CN112488213A (en) Fire picture classification method based on multi-scale feature learning network
CN114926778A (en) Safety helmet and personnel identity recognition system under production environment
CN117576632B (en) Multi-mode AI large model-based power grid monitoring fire early warning system and method
CN115410114A (en) Urban rail flood prevention early warning method and system based on multiple characteristics
CN111783751A (en) Rifle ball linkage and BIM-based breeding house piglet abnormity early warning method
CN117690086A (en) Flood prevention gate identification and control method and system based on 5G and AI technology
CN117745945A (en) AI-based boiler furnace water wall tube wall image acquisition and processing system
CN116798117A (en) Video understanding-based method for identifying abnormal actions under mine
CN111091586A (en) Rapid smoke dynamic shielding area detection and positioning method and application thereof
CN111222477A (en) Vision-based method and device for detecting two hands leaving steering wheel
CN115049600A (en) Intelligent identification system and method for small sample pipeline defects

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant