CN116468974B - Smoke detection method, device and storage medium based on image generation - Google Patents
Smoke detection method, device and storage medium based on image generation Download PDFInfo
- Publication number
- CN116468974B CN116468974B CN202310702062.6A CN202310702062A CN116468974B CN 116468974 B CN116468974 B CN 116468974B CN 202310702062 A CN202310702062 A CN 202310702062A CN 116468974 B CN116468974 B CN 116468974B
- Authority
- CN
- China
- Prior art keywords
- smoke
- image
- training
- data set
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000000779 smoke Substances 0.000 title claims abstract description 289
- 238000001514 detection method Methods 0.000 title claims abstract description 79
- 238000012549 training Methods 0.000 claims abstract description 58
- 238000000034 method Methods 0.000 claims abstract description 29
- 238000012545 processing Methods 0.000 claims abstract description 15
- 238000002372 labelling Methods 0.000 claims abstract description 13
- 230000006870 function Effects 0.000 claims description 44
- 238000013528 artificial neural network Methods 0.000 claims description 17
- 230000008485 antagonism Effects 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 230000004927 fusion Effects 0.000 claims description 7
- 238000005070 sampling Methods 0.000 claims description 5
- 239000003086 colorant Substances 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 2
- 238000012544 monitoring process Methods 0.000 description 10
- 230000002265 prevention Effects 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000006872 improvement Effects 0.000 description 4
- 239000000284 extract Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000006467 substitution reaction Methods 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000003491 array Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000012080 ambient air Substances 0.000 description 1
- 230000008033 biological extinction Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 230000007797 corrosion Effects 0.000 description 1
- 238000005260 corrosion Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 239000003595 mist Substances 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000001308 synthesis method Methods 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000001931 thermography Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T11/00—2D [Two Dimensional] image generation
- G06T11/001—Texturing; Colouring; Generation of texture or colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/52—Scale-space analysis, e.g. wavelet analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B17/00—Fire alarms; Alarms responsive to explosion
- G08B17/12—Actuation by presence of radiation or particles, e.g. of infrared radiation or of ions
- G08B17/125—Actuation by presence of radiation or particles, e.g. of infrared radiation or of ions by using a video camera to detect fire or smoke
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The application discloses a smoke detection method, device and storage medium based on image generation, and belongs to the field of image data processing. The method comprises the following steps: acquiring a smoke data set and a smokeless data set; training an image generation network that generates images based on the image mask from the smoke dataset; processing the smokeless data set by adopting a trained image generation network to generate images containing smog with different concentrations and corresponding image masks; acquiring a training set according to the smoke data set and the generated image, and training a smoke detection model according to the training set; and acquiring an image to be detected, inputting the image to be detected into a trained smoke detection model, and detecting whether fire smoke exists in the image to be detected. According to the application, the image generation network is used for generating vivid smoke images with rich scenes and different concentrations, so that the training data volume of the smoke detection model can be greatly expanded on the premise of avoiding complicated manual labeling, and the detection precision and effect of the smoke detection model are improved.
Description
Technical Field
The present application relates to the field of image data processing, and in particular, to a smoke detection method and apparatus based on image generation, and a storage medium.
Background
The development of a fire can be divided into five phases, initial, development, violent, descent and extinction. In the initial stage, the combustion range is not large, the smoldering phenomenon is easy to occur, namely, no open flame exists, only smoke is generated, and the smoke can be continuously diffused along with the development of fire. Early detection of smoke from a fire is particularly important if smoke from a fire can be detected in time and appropriate fire rescue measures can be taken in time to minimize the impact and damage of the fire.
The traditional fire monitoring technology is based on sensors, such as an ionic smoke sensor, a photoelectric smoke sensor, a thermal imaging sensor and the like, and mostly senses smoke particle density and temperature change in a fire scene and triggers an alarm device, but the detection is interfered by the influence of conditions such as smoke concentration and ambient air flow change to different degrees, and obvious defects exist in the aspects of detection range and detection speed. With the development of video monitoring technology, a smoke detection method based on traditional image processing is developed, and the detection method can effectively improve the detection speed by detecting the characteristics of smoke such as color, texture and the like, but generally needs to integrate various image frame information, consumes a large amount of computing resources in image processing, and still improves the feedback speed.
With the advent of Convolutional Neural Networks (CNNs) in recent years, computer vision computing has progressed rapidly, and more target detection algorithms have been developed and have achieved satisfactory results. At present, detection algorithms are mainly divided into two types: single-stage detector and double-stage detector. The most representative neural network for a dual-stage detector is the Faster regional convolutional neural network (Faster-RCNN), which first extracts features, selects candidate regions using a regional proposal network, and then classifies the candidate regions by deep convolutional blocks. The dual-stage detector can greatly improve the detection accuracy through the regional proposal network, but the detection time is prolonged, and the actual requirement is difficult to meet. The single-stage detector combines the feature processing in the double stages into one stage to finish, can simultaneously propose candidate areas and carry out positioning classification of targets, and classical algorithms such as single-shot multi-box object detection (SSD) series and YOLO (You Only Look Once) series can greatly improve the detection speed, and meanwhile, the detection accuracy is kept high.
Training an efficient convolutional neural network often requires sufficient training data. However, fire scenes are relatively few, and fires tend to result in loss of monitoring data, making smoke scene data collection extremely difficult, and image manual annotation is expensive, so training a robust smoke detector with limited data remains a significant challenge. At present, although some smoke image synthesis methods exist, the synthesis efficiency is low, the quality of the synthesized image is poor, the performance of a smoke detector is difficult to be improved by truly utilizing synthesized data, and the interference of other objects on smoke detection is reduced.
Disclosure of Invention
In order to solve at least one of the technical problems existing in the prior art to a certain extent, the application aims to provide a smoke detection method, a device and a storage medium based on image generation.
The technical scheme adopted by the application is as follows:
a smoke detection method based on image generation, comprising the steps of:
acquiring a smoke data set and a smokeless data set; the smoke data set is a real image data set containing different scenes and different smoke concentrations, and the image data in the smoke data set comprises a smoke image and an image mask corresponding to the smoke image; the smokeless data set is a smokeless real image data set containing different scenes;
training an image generation network that generates images based on the image mask from the smoke dataset;
processing the smokeless data set by adopting a trained image generation network to generate images containing smog with different concentrations and corresponding image masks;
acquiring a training set according to the smoke data set and the generated image, and training a smoke detection model according to the training set;
and acquiring an image to be detected, inputting the image to be detected into the trained smoke detection model, and detecting whether fire smoke exists in the image to be detected or not and the position of the fire smoke in the image.
Further, the acquiring a smoke dataset comprises:
labeling the smoke images in the smoke data set, wherein the labeling form comprises an image mask and a smoke concentration value;
the size of the image mask is the same as that of the smoke image, and the image mask is provided with two areas with different colors and is used for distinguishing a foreground from a background and respectively corresponding to smoke and scene contents in the smoke image;
each smoke image corresponds to a smoke concentration value, and the smoke concentration value is the ratio of the occupied area of smoke in the smoke image to the total area of the smoke image.
Further, the image generation network comprises two branches:
the first branch is used for generating a corresponding image mask according to the input smoke concentration value; in the training process, constraint is carried out through a smoke area loss function and generation of an countermeasure network, so that the branch can generate a corresponding image mask according to an input smoke concentration value;
the second branch is used for generating a corresponding smoke image according to the learned image mask; in the training process, the whole neural network can generate a corresponding smoke scene image according to the input smoke concentration value by restricting the image mask restriction function and generating the antagonism network restriction function.
Further, the expression of the smoke area loss function is:
wherein ,for inputting a specified smoke concentration +.>Generating branches for image masks->An image mask generated for the branch based on the specified smoke concentration,>a function of smoke concentration is calculated from the image mask.
Further, the expression of the image mask constraint function is:
wherein For a true smoke image->For smoke image->Corresponding image mask, < >>For discriminating whether the generated smoke image is true or false under the condition of image mask +.> and />The decision device is respectively used for generating an image or a counterdamage function constraint designed for a real image aiming at the input image, and hopefully can correctly judge the true or false of the input image under the condition that the corresponding image mask is taken as prior information;
the expression for generating the antagonism network constraint function is:
wherein ,for the discriminator in discriminating the true or false of the generated smoke image, < +.> and />The discriminators are expected to be able to discriminate the input image directly, against loss function constraints that are either generated or true for the input image, respectivelyTrue and false; />For the preset smoke concentration value->Next, a generator that generates a smoke image based on the image mask.
Further, the expression of the training target of the image generation network is:
further, the acquiring a training set according to the smoke data set and the generated image, training a smoke detection model according to the training set, includes:
acquiring a rectangular frame corresponding to each smoke area according to the image mask, wherein the rectangular frame is provided with four corner coordinates and comprises the smoke areas corresponding to the image mask;
and forming a training set by all the smoke images and the corresponding smoke labeling rectangular frames, and training the smoke detection model according to the training set so that the smoke images can judge the existence of smoke in the images and the positions of the smoke in the images.
Further, the smoke detection model comprises an improved network obtained by improving a yolov5 network, and the working flow is as follows:
and (3) sending the multi-scale feature map extracted by the backbone network in the yolov5 network into a fusion network for fusion after up-sampling of different scales, and calculating a smoke concentration value in a smoke image by using the fused features.
The application adopts another technical scheme that:
a smoke detection apparatus based on image generation, comprising:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method as described above.
The application adopts another technical scheme that:
a computer readable storage medium, in which a processor executable program is stored, which when executed by a processor is adapted to carry out the method as described above.
The beneficial effects of the application are as follows: according to the application, the image generation network is used for generating vivid smoke images with rich scenes and different concentrations, so that the training data volume of the smoke detection model can be greatly expanded on the premise of avoiding complicated manual labeling, and the detection precision and effect of the smoke detection model are improved. In addition, the smoke detection model is beneficial to improving the smoke detection precision under a complex background, reducing misjudgment on interference objects such as cloud, fog and the like, and accurately and rapidly judging and positioning the smoke in the image.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following description is made with reference to the accompanying drawings of the embodiments of the present application or the related technical solutions in the prior art, and it should be understood that the drawings in the following description are only for convenience and clarity of describing some embodiments in the technical solutions of the present application, and other drawings may be obtained according to these drawings without the need of inventive labor for those skilled in the art.
FIG. 1 is a flow chart of steps of a method for image-based smoke detection in an embodiment of the application;
FIG. 2 is a block diagram of an image generation network in an embodiment of the application;
fig. 3 is a block diagram of a yolov 5-based improved smoke detection model in an embodiment of the application.
Detailed Description
Embodiments of the present application are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the application. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.
In the description of the present application, it should be understood that references to orientation descriptions such as upper, lower, front, rear, left, right, etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of description of the present application and to simplify the description, and do not indicate or imply that the apparatus or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and thus should not be construed as limiting the present application.
In the description of the present application, a number means one or more, a number means two or more, and greater than, less than, exceeding, etc. are understood to not include the present number, and above, below, within, etc. are understood to include the present number. The description of the first and second is for the purpose of distinguishing between technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or implicitly indicating the precedence of the technical features indicated.
Furthermore, in the description of the present application, unless otherwise indicated, "a plurality" means two or more. "and/or", describes an association relationship of an association object, and indicates that there may be three relationships, for example, a and/or B, and may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
In the description of the present application, unless explicitly defined otherwise, terms such as arrangement, installation, connection, etc. should be construed broadly and the specific meaning of the terms in the present application can be reasonably determined by a person skilled in the art in combination with the specific contents of the technical scheme.
As shown in fig. 1, the embodiment provides a smoke detection method based on image generation, which uses a smoke image training image generation network, combines a smoke image to obtain a smoke image data set containing accurate labels, trains a smoke detector modified based on yolov5 by using the data set, improves the performance of smoke detection and the capability of fire prevention, and can be widely applied to forest fire prevention, factory fire prevention, warehouse fire prevention, residential fire prevention or market fire prevention. The method specifically comprises the following steps:
s1, acquiring a smoke data set and a smokeless data set.
The smoke data set is a real image data set containing different scenes and different smoke concentrations, the image data in the smoke data set comprises a smoke image and an image mask corresponding to the smoke image, the image mask is divided into a foreground and a background, the foreground is smoke, and the background is a scene in the image. The smokeless data set is a smokeless real image data set that contains different scenes.
As an alternative embodiment, two different data sets are acquired, one data set is a smoke data set, the smoke data set is in the form of an image and a label, the image is an image with different scenes and different smoke concentrations, the label is in the form of an image mask and a smoke concentration value, the image mask is the same as the image in size, and two areas with different colors are used for distinguishing a foreground from a background and respectively correspond to smoke and scene contents in the image. In addition, each image corresponds to a smoke concentration, the concentration is the ratio of the occupied area of all smoke in the image to the area of the image, the images in the smoke data set come from monitoring camera pictures of different fire smoke scenes, and therefore the data size can be smaller. The other data set is a smokeless data set, only images without smoke exist in the data set, and the images come from daily monitoring cameras without special restrictions, so that more data can be acquired.
S2, training an image generation network for generating images based on the image masks according to the smoke data set.
Training a neural network (namely an image generation network) for generating images based on image masks by using a smoke data set, wherein in the generation process, firstly, the concentration is designated to randomly generate the image masks with different sizes, and through a smoke area loss function, an image mask constraint function and a learning process for generating a network constraint countermeasure neural network, the neural network can generate images containing smoke with different concentrations and corresponding image masks after the learning is completed; more smoke images of different scenes may be synthesized using the smoke-free dataset and the generated smoke images and the image mask.
As an alternative embodiment, the image generation network is split into two branches; the first branch is the part that generates the corresponding image mask based on the input smoke concentration, with the smoke area loss function constraint and the constraint of the antagonism network generated so that this branch can generate the appropriate image mask based on the given smoke concentration. The second branch is a neural network which generates a corresponding smoke image according to an image mask learned by the neural network, and the whole neural network can generate a vivid smoke scene image according to a given smoke concentration through an image mask constraint function and an antagonism network constraint generation.
And S3, processing the smokeless data set by adopting the trained image generation network to generate images containing smoke with different concentrations and corresponding image masks.
And combining the smoke in the smoke image and scene content in the smokeless image by using an image mask of the smoke image based on the image generation network to obtain the smoke images of different scenes.
S4, acquiring a training set according to the smoke data set and the generated image, and training a smoke detection model according to the training set.
And acquiring a rectangular frame corresponding to each smoke area according to the corresponding image mask, wherein the rectangular frame is provided with four corner coordinates and can contain the corresponding smoke area in the image mask. And forming a smoke detection training set by all the smoke images and the corresponding smoke labeling rectangular frames, and training a smoke detection model to enable the smoke detection model to judge the existence of smoke in the images and the positions of the smoke in the images.
As an alternative embodiment, the smoke detection model includes an improved network that is an improvement to the yolov5 network, the improvement comprising: and (3) sending the multi-scale feature map extracted from the yolov5 backbone network into a fusion network after up-sampling of different scales, calculating the smoke concentration of the image by using the fused features, and restricting a detection model according to the real smoke concentration.
The yolov5 network is improved, a backbone network of the yolov5 generally extracts multi-scale feature images with different resolutions for classification detection, three multi-scale features with different resolutions are subjected to up-sampling with different scales and then are fused through a fusion network, the fused features are used for predicting the smoke concentration of a model through a prediction network and comparing with the true smoke concentration value of a smoke image, so that the backbone network of the yolov5 is restrained from paying more attention to a smoke area, and misjudgment on other white or mist object areas is reduced.
As an alternative embodiment, the smoke detection model is specifically trained by:
collecting and generating a sample image containing smoke, marking the sample image and enhancing data to obtain a training data set;
preprocessing the data in the training data set to obtain a preprocessed data set;
building a virtual environment required by a training model on a server;
training the improved network in the virtual environment based on the preprocessed data set to obtain the trained detection model.
S5, acquiring an image to be detected, inputting the image to be detected into a trained smoke detection model, and detecting whether fire smoke exists in the image to be detected and the position of the fire smoke in the image.
The above method is explained in detail below with reference to the drawings and specific examples.
The present embodiment provides a smoke detection method based on image generation, which first requires collecting two data sets. One is a smoke data set containing a smoke image and corresponding labels, wherein the labels contain an image mask corresponding to the smoke image and a smoke concentration index, the image mask is divided into a foreground part and a background part, the foreground part is smoke, the background part is a scene in the image, and the other is a smokeless image of a non-labeled fire scene. The images of both data sets are taken by the monitoring cameras of different scenes. When the smoke image is acquired, the image is intercepted according to a preset interval frame, so that the excessive same images are prevented from being acquired, and the cost is increased. After the smoke image is obtained from the monitoring video, the image is required to be subjected to image processing operations such as filtering, expansion, corrosion and the like, parameter interference outside smoke is removed, an image mask of the image is extracted, the smoke concentration corresponding to the image is calculated, and a smoke data set is constructed. The smokeless data set is extracted from the monitoring video of the scene where the fire is likely to occur. The images in the smoke data set come from the monitoring camera pictures of different fire smoke scenes, so that the data size is small; only images without smoke in the smoke-free data set come from daily monitoring camera pictures, and no other special restrictions exist, so that more image data can be acquired. Unless otherwise stated, at all neural network training, the dataset was as per 9: the ratio of 1 is randomly divided into a training set and a test set.
After two different data sets are collected, a neural network for generating images based on image masks needs to be trained using the smoke data sets, as shown in fig. 2, which is split into two branches; the first branch is the part that generates the corresponding image mask according to the input smoke concentration, and is constrained by a smoke area loss function, so that the branch can generate the proper image mask according to the given smoke concentration, and the smoke area loss function is as follows:
wherein For inputting a specified smoke concentration +.>Generating branches for image masks->An image mask generated for the branch based on the specified smoke concentration,>a function of smoke concentration is calculated from the image mask. By this loss function, it can be constrained that the generated image mask needs to conform to the specified smoke concentration.
The second branch is a neural network which generates a corresponding smoke image according to an image mask learned by the neural network, and the whole neural network can generate a vivid smoke scene image according to a given smoke concentration through an image mask constraint function and an antagonism constraint function. The neural network is an image generation network based on the space self-adaptive regularization network SPADE after transformation, and the generation of the antagonism network constraint function is as follows:
wherein For the discriminator in discriminating the true or false of the generated smoke image, < +.>For a true smoke image->For a given smoke concentration->A generator that generates a smoke image based on the image mask. /> and />The corresponding input image is a contrast loss function that generates an image or is a true image, respectively. By means of the loss function, the image mask generated by the first branch in the neural network can be constrained to conform to the smoke distribution condition of a common fire smoke scene, and the generated smoke image is as lifelike as possible. The next is the image mask constraint function as follows:
wherein For a true smoke image->For smoke image->Corresponding image mask, < >>For discriminating whether the generated smoke image is true or false under the condition of image mask +.> and />The penalty function constraints are designed for whether the input image is a generated image or a real image, respectively, hisThe identifier can accurately identify the true or false of the input image under the condition that the corresponding image mask is taken as prior information, can restrict the generated image to be as true as possible under the given image mask condition, and accords with the image mask.
The total training targets are:
the image generation network after training is completed can generate smoke images with different smoke concentrations, but because the training data volume is small and the scene is single, a large number of smoke-free images are utilized to be combined through the image mask, the smoke images provide smoke prospects, the smoke-free images provide backgrounds, so that more smoke images are acquired, and the concentration distribution in the smoke images is as uniform as possible.
All acquired smoke images need to be processed to train a smoke detection model. The processing is mainly divided into labeling processing, image processing and formal processing. The labeling process is to obtain a rectangular frame corresponding to each smoke region according to the image mask, wherein the rectangular frame has four corner coordinates and can contain the corresponding smoke region in the image mask. And forming a smoke detection training set by all the smoke images and the corresponding smoke labeling rectangular frames, and training a smoke detection model so as to judge the existence of smoke in the images and the positions of the smoke in the images. The image processing further expands the data set for general object detection data enhancement operations using color transformation, scale transformation, random cropping, random scaling, etc., resulting in final training data. The formal treatment is to process the smoke data set into a VOC format, thereby facilitating training of the smoke detection model.
The smoke detection model is a network based on yolov5 improvement, and a backbone network of yolov5 usually extracts characteristic diagrams with different multi-scale resolutions and then inputs the characteristic diagrams into a subsequent classification network for classification detection. As shown in fig. 3, the smoke detection model fuses three multi-scale features with different resolutions extracted by the backbone network after up-sampling with different scales through the fusion network, predicts the smoke concentration of the image through the prediction network, and compares the smoke concentration with the true smoke concentration value of the smoke image, so that the backbone network of yolov5 is constrained to pay more attention to the smoke region and the smoke concentration, and misjudgment on other white or vaporous object regions is reduced.
After training, the smoke detection model can determine the presence of fire smoke in the input image and its location in the image.
In summary, compared with the prior art, the method of the embodiment only has the following advantages and beneficial effects:
(1) The embodiment combines various neural networks, can generate smoke image data with various scenes and vividness, contains accurate labeling information of corresponding images, and can greatly expand a data set for smoke detection training.
(2) The embodiment provides a smoke detection model based on yolov5 improvement, which can fully utilize concentration information in a smoke image and reduce the misjudgment rate of a detection network on smoke.
The embodiment also provides a smoke detection device based on image generation, including:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method as shown in fig. 1.
The smoke detection device based on image generation can execute the smoke detection method based on image generation provided by the embodiment of the method, can execute any combination implementation steps of the embodiment of the method, and has the corresponding functions and beneficial effects of the method.
Embodiments of the present application also disclose a computer program product or computer program comprising computer instructions stored in a computer readable storage medium. The computer instructions may be read from a computer-readable storage medium by a processor of a computer device, and executed by the processor, to cause the computer device to perform the method shown in fig. 1.
The embodiment also provides a storage medium which stores instructions or programs for executing the image generation-based smoke detection method provided by the embodiment of the method, and when the instructions or programs are run, the instructions or programs can execute the steps in any combination of the embodiment of the method, and the method has the corresponding functions and beneficial effects.
In some alternative embodiments, the functions/acts noted in the block diagrams may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Furthermore, the embodiments presented and described in the flowcharts of the present application are provided by way of example in order to provide a more thorough understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and in which sub-operations described as part of a larger operation are performed independently.
Furthermore, while the application is described in the context of functional modules, it should be appreciated that, unless otherwise indicated, one or more of the described functions and/or features may be integrated in a single physical device and/or software module or one or more functions and/or features may be implemented in separate physical devices or software modules. It will also be appreciated that a detailed discussion of the actual implementation of each module is not necessary to an understanding of the present application. Rather, the actual implementation of the various functional modules in the apparatus disclosed herein will be apparent to those skilled in the art from consideration of their attributes, functions and internal relationships. Accordingly, one of ordinary skill in the art can implement the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and are not intended to be limiting upon the scope of the application, which is to be defined in the appended claims and their full scope of equivalents.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable medium may even be paper or other suitable medium on which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.
It is to be understood that portions of the present application may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.
In the foregoing description of the present specification, reference has been made to the terms "one embodiment/example", "another embodiment/example", "certain embodiments/examples", and the like, means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the spirit and principles of the application, the scope of which is defined by the claims and their equivalents.
While the preferred embodiment of the present application has been described in detail, the present application is not limited to the above embodiments, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present application, and these equivalent modifications and substitutions are intended to be included in the scope of the present application as defined in the appended claims.
Claims (7)
1. A smoke detection method based on image generation, comprising the steps of:
acquiring a smoke data set and a smokeless data set; the smoke data set is a real image data set containing different scenes and different smoke concentrations, and the image data in the smoke data set comprises a smoke image and an image mask corresponding to the smoke image; the smokeless data set is a smokeless real image data set containing different scenes;
training an image generation network that generates images based on the image mask from the smoke dataset;
processing the smokeless data set by adopting a trained image generation network to generate images containing smog with different concentrations and corresponding image masks;
acquiring a training set according to the smoke data set and the generated image, and training a smoke detection model according to the training set;
acquiring an image to be detected, inputting the image to be detected into the trained smoke detection model, and detecting whether fire smoke exists in the image to be detected;
the image generation network comprises two branches:
the first branch is used for generating a corresponding image mask according to the input smoke concentration value; in the training process, constraint is carried out through a smoke area loss function and generation of an countermeasure network, so that the branch can generate a corresponding image mask according to an input smoke concentration value;
the second branch is used for generating a corresponding smoke image according to the learned image mask; in the training process, the whole neural network can generate a corresponding smoke scene image according to the input smoke concentration value by restricting the image mask restriction function and generating the antagonism network restriction function;
the expression of the smoke area loss function is as follows:
wherein ,for inputting a specified smoke concentration +.>Generating branches for image masks->An image mask generated for the branch based on the specified smoke concentration,>calculating a function of smoke concentration from the image mask;
the expression of the image mask constraint function is:
wherein For a true smoke image->For smoke image->Corresponding image mask, < >>For discriminating whether the generated smoke image is true or false under the condition of image mask +.> and />The penalty function constraints are designed for whether the input image is a generated image or a real image, respectively; />For a given smoke concentration->A generator that generates a smoke image based on the image mask;
the expression for generating the antagonism network constraint function is:
wherein ,for the discriminator in discriminating the true or false of the generated smoke image, < +.> and />The corresponding input image is an contrast loss function constraint that generates an image or is a true image, respectively.
2. The image generation-based smoke detection method of claim 1, wherein said acquiring a smoke dataset comprises:
labeling the smoke images in the smoke data set, wherein the labeling form comprises an image mask and a smoke concentration value;
the size of the image mask is the same as that of the smoke image, and the image mask is provided with two areas with different colors and is used for distinguishing a foreground from a background and respectively corresponding to smoke and scene contents in the smoke image;
each smoke image corresponds to a smoke concentration value, and the smoke concentration value is the ratio of the occupied area of smoke in the smoke image to the total area of the smoke image.
3. The image generation-based smoke detection method according to claim 1, wherein the expression of the training target of the image generation network is:
wherein ,as a function of smoke area loss.
4. The image-based generated smoke detection method of claim 1, wherein the acquiring a training set from the smoke data set and the generated image, training the smoke detection model from the training set, comprises:
acquiring a rectangular frame corresponding to each smoke area according to the image mask, wherein the rectangular frame is provided with four corner coordinates and comprises the smoke areas corresponding to the image mask;
and forming a training set by all the smoke images and the corresponding smoke labeling rectangular frames, and training the smoke detection model according to the training set so that the smoke images can judge the existence of smoke in the images and the positions of the smoke in the images.
5. An image generation based smoke detection method according to any one of claims 1-4 wherein said smoke detection model comprises an improved network from the yolov5 network, the workflow being as follows:
and (3) sending the multi-scale feature map extracted by the backbone network in the yolov5 network into a fusion network for fusion after up-sampling of different scales, and calculating a smoke concentration value in a smoke image by using the fused features.
6. An image generation-based smoke detection apparatus, comprising:
at least one processor;
at least one memory for storing at least one program;
the at least one program, when executed by the at least one processor, causes the at least one processor to implement the method of any one of claims 1-5.
7. A computer readable storage medium, in which a processor executable program is stored, characterized in that the processor executable program is for performing the method according to any of claims 1-5 when being executed by a processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310702062.6A CN116468974B (en) | 2023-06-14 | 2023-06-14 | Smoke detection method, device and storage medium based on image generation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310702062.6A CN116468974B (en) | 2023-06-14 | 2023-06-14 | Smoke detection method, device and storage medium based on image generation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116468974A CN116468974A (en) | 2023-07-21 |
| CN116468974B true CN116468974B (en) | 2023-10-13 |
Family
ID=87177410
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310702062.6A Active CN116468974B (en) | 2023-06-14 | 2023-06-14 | Smoke detection method, device and storage medium based on image generation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116468974B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110807743A (en) * | 2019-10-24 | 2020-02-18 | 华中科技大学 | Image defogging method based on convolutional neural network |
| WO2020181570A1 (en) * | 2019-03-08 | 2020-09-17 | 上海达显智能科技有限公司 | Intelligent smoke removal device and control method thereof |
| CN112150379A (en) * | 2020-09-22 | 2020-12-29 | 武汉工程大学 | Image defogging method and device for enhancing generation of countermeasure network based on perception discrimination |
| CN112903550A (en) * | 2021-01-28 | 2021-06-04 | 宁波方太厨具有限公司 | Oil smoke concentration data set generation method, device, system, equipment and medium |
| CN112949453A (en) * | 2021-02-26 | 2021-06-11 | 南京恩博科技有限公司 | Training method of smoke and fire detection model, smoke and fire detection method and smoke and fire detection equipment |
| CN113128422A (en) * | 2021-04-23 | 2021-07-16 | 重庆市海普软件产业有限公司 | Image smoke and fire detection method and system of deep neural network |
| CN114332739A (en) * | 2021-09-28 | 2022-04-12 | 同济人工智能研究院(苏州)有限公司 | Smoke detection method based on moving target detection and deep learning technology |
| CN114638767A (en) * | 2022-05-18 | 2022-06-17 | 山东师范大学 | Laparoscope image smoke removal method based on generation of countermeasure network |
| CN115187831A (en) * | 2022-06-24 | 2022-10-14 | 重庆紫光华山智安科技有限公司 | Model training and smoke detection method and device, electronic equipment and storage medium |
| CN115358942A (en) * | 2022-08-09 | 2022-11-18 | 山西大学 | Image defogging method combining course learning and teacher-student learning |
| CN115424171A (en) * | 2022-08-24 | 2022-12-02 | 深圳海翼智新科技有限公司 | Flame and smoke detection method, device and storage medium |
| CN115439847A (en) * | 2022-09-02 | 2022-12-06 | 成都视海芯图微电子有限公司 | Foggy day image defogging method and device based on generation countermeasure network |
| CN115830318A (en) * | 2022-11-25 | 2023-03-21 | 河南工业大学 | Method and device for segmenting overlapped tobacco shred images based on improved Mask-RCNN (Richardson network) |
-
2023
- 2023-06-14 CN CN202310702062.6A patent/CN116468974B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020181570A1 (en) * | 2019-03-08 | 2020-09-17 | 上海达显智能科技有限公司 | Intelligent smoke removal device and control method thereof |
| CN110807743A (en) * | 2019-10-24 | 2020-02-18 | 华中科技大学 | Image defogging method based on convolutional neural network |
| CN112150379A (en) * | 2020-09-22 | 2020-12-29 | 武汉工程大学 | Image defogging method and device for enhancing generation of countermeasure network based on perception discrimination |
| CN112903550A (en) * | 2021-01-28 | 2021-06-04 | 宁波方太厨具有限公司 | Oil smoke concentration data set generation method, device, system, equipment and medium |
| CN112949453A (en) * | 2021-02-26 | 2021-06-11 | 南京恩博科技有限公司 | Training method of smoke and fire detection model, smoke and fire detection method and smoke and fire detection equipment |
| CN113128422A (en) * | 2021-04-23 | 2021-07-16 | 重庆市海普软件产业有限公司 | Image smoke and fire detection method and system of deep neural network |
| CN114332739A (en) * | 2021-09-28 | 2022-04-12 | 同济人工智能研究院(苏州)有限公司 | Smoke detection method based on moving target detection and deep learning technology |
| CN114638767A (en) * | 2022-05-18 | 2022-06-17 | 山东师范大学 | Laparoscope image smoke removal method based on generation of countermeasure network |
| CN115187831A (en) * | 2022-06-24 | 2022-10-14 | 重庆紫光华山智安科技有限公司 | Model training and smoke detection method and device, electronic equipment and storage medium |
| CN115358942A (en) * | 2022-08-09 | 2022-11-18 | 山西大学 | Image defogging method combining course learning and teacher-student learning |
| CN115424171A (en) * | 2022-08-24 | 2022-12-02 | 深圳海翼智新科技有限公司 | Flame and smoke detection method, device and storage medium |
| CN115439847A (en) * | 2022-09-02 | 2022-12-06 | 成都视海芯图微电子有限公司 | Foggy day image defogging method and device based on generation countermeasure network |
| CN115830318A (en) * | 2022-11-25 | 2023-03-21 | 河南工业大学 | Method and device for segmenting overlapped tobacco shred images based on improved Mask-RCNN (Richardson network) |
Non-Patent Citations (4)
| Title |
|---|
| Deep convolutional generative adversarial network and convolutional neural network for smoke detection;hang yin 等;《WILEY》;全文 * |
| 一种双网融合的分阶段烟雾检测算法;杜立召;徐岩;张为;;西安电子科技大学学报(04);全文 * |
| 基于Faster R-CNN的颜色导向火焰检测;黄杰;巢夏晨语;董翔宇;高云;朱俊;杨波;张飞;尚伟伟;;计算机应用(05);全文 * |
| 基于LBP和小波纹理特征的室内室外场景分类算法;许勇 等;《中国图象图形学报》;全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116468974A (en) | 2023-07-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111860160B (en) | Method for detecting wearing of mask indoors | |
| CN109977191B (en) | Problem map detection method, device, electronic equipment and medium | |
| CN111881730A (en) | Wearing detection method for on-site safety helmet of thermal power plant | |
| CN112633231B (en) | Fire disaster identification method and device | |
| CN109389185B (en) | Video smoke identification method using three-dimensional convolutional neural network | |
| CN111680632A (en) | Smoke and fire detection method and system based on deep learning convolutional neural network | |
| CN109033950A (en) | Vehicle based on multiple features fusion cascade deep model, which is disobeyed, stops detection method | |
| CN103092930B (en) | Method of generation of video abstract and device of generation of video abstract | |
| CN113591968A (en) | Infrared weak and small target detection method based on asymmetric attention feature fusion | |
| CN116543241B (en) | Detection method and device for leakage gas cloud, storage medium and electronic equipment | |
| CN108229524A (en) | A kind of chimney and condensing tower detection method based on remote sensing images | |
| CN113343779B (en) | Environment abnormality detection method, device, computer equipment and storage medium | |
| CN111145222A (en) | Fire detection method combining smoke movement trend and textural features | |
| CN113378675A (en) | Face recognition method for simultaneous detection and feature extraction | |
| CN115100497A (en) | Robot-based method, device, equipment and medium for routing inspection of abnormal objects in channel | |
| CN110991245A (en) | Real-time smoke detection method based on deep learning and optical flow method | |
| CN114422720A (en) | Video concentration method, system, device and storage medium | |
| CN111723656B (en) | Smog detection method and device based on YOLO v3 and self-optimization | |
| Chen et al. | A novel smoke detection algorithm based on improved mixed Gaussian and YOLOv5 for textile workshop environments | |
| CN116468974B (en) | Smoke detection method, device and storage medium based on image generation | |
| CN115205793B (en) | Electric power machine room smoke detection method and device based on deep learning secondary confirmation | |
| CN115131826B (en) | Article detection and identification method, and network model training method and device | |
| CN103886607B (en) | A kind of detection for disturbance target and suppressing method | |
| CN111191575B (en) | Naked flame detection method and system based on flame jumping modeling | |
| CN111402185A (en) | Image detection method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |