CN115131622B - Night open fire detection method based on video time sequence correlation - Google Patents
Night open fire detection method based on video time sequence correlation Download PDFInfo
- Publication number
- CN115131622B CN115131622B CN202210559156.8A CN202210559156A CN115131622B CN 115131622 B CN115131622 B CN 115131622B CN 202210559156 A CN202210559156 A CN 202210559156A CN 115131622 B CN115131622 B CN 115131622B
- Authority
- CN
- China
- Prior art keywords
- open fire
- detection
- frame
- open
- fire detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 104
- 238000012549 training Methods 0.000 claims abstract description 10
- 238000000034 method Methods 0.000 claims description 16
- 238000000605 extraction Methods 0.000 claims description 13
- 238000005070 sampling Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 4
- 238000013507 mapping Methods 0.000 claims description 4
- 238000004364 calculation method Methods 0.000 claims description 2
- 238000013459 approach Methods 0.000 claims 1
- 238000013135 deep learning Methods 0.000 abstract description 10
- 238000004458 analytical method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000012544 monitoring process Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000001351 cycling effect Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013528 artificial neural network Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000004660 morphological change Effects 0.000 description 1
- 239000000779 smoke Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
- G06T7/62—Analysis of geometric attributes of area, perimeter, diameter or volume
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- General Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Quality & Reliability (AREA)
- Geometry (AREA)
- Fire-Detection Mechanisms (AREA)
Abstract
The invention discloses a night open fire detection method based on video time sequence correlation, and belongs to the technical field of target detection. The invention constructs an open flame image data set for training; constructing an open fire detection network; and constructing a time sequence judging module. The invention designs the open fire detection method by fully utilizing the advantages of deep learning in the target detection field, is not limited by scenes, distances and open fire types, designs the time sequence judging module according to the video sequence, combines the difference of open fire and lamplight, analyzes the state of the detected open fire, and effectively reduces the false detection of night lamplight on the basis of not reducing the real open fire detection precision.
Description
Technical Field
The invention relates to the field of intelligent security, in particular to a night open fire detection method based on video time sequence correlation.
Background
Currently, in the field of open fire detection, there are three main methods, i.e., manual field monitoring, detection using a temperature sensor and a smoke sensor, and detection using an image processing technique. The first method needs related personnel to detect the scene which is easy to generate fire through camera pictures or field monitoring, and has high requirement on human resources. The second method is limited by the detection range of the sensor, and has poor detection effect on the middle-long distance scene. The third image processing technique is divided into a conventional feature extraction method and a deep learning object detection method. The traditional feature extraction method is used for judging by extracting the color and the outline of the open fire, has poor generalization, is influenced by light, and has high false detection rate for outdoor scenes at night. The deep learning target detection method utilizes the neural network to learn a large amount of open fire image data, extracts the open fire characteristics of different distances, scenes and categories, can well finish the open fire detection task, but still has false detection on night lamplight.
Analysis shows that similar characteristics exist on the colors and the forms of night lamplight and open fire, the method for generating the mapping relation by learning the image characteristics is deep learning, open fire detection is performed based on a single image, and from the working principle, false detection is unavoidable because the difference of the time sequence correlation of the open fire video and the night lamplight video is ignored. A large number of test data also show that when the warm-light exists in the image, the existing method can easily identify the warm-light as open fire, so that the bottleneck of the open fire detection technology is how to reduce false detection of the night light. The method is designed to have a time sequence correlation detection method by taking the difference of dynamic characteristics of open fire and lamplight in a video into consideration and utilizes the static characteristics in the image, the morphological change of a target to be detected in the video is analyzed on the basis of the existing deep learning detection, the detection result is further restrained by the change trend, and the purpose of high-robustness night open fire detection is achieved. The problem to be solved is divided into two points. The first point is to construct a deep learning open fire detection model with excellent performance, so that the open fire detection precision is fundamentally improved; and a judging module of time sequence correlation is added on the basis of detection at the second point, so that false detection caused by light interference is reduced. The invention utilizes the existing deep learning image processing technology to accurately detect open fire, then combines each frame in the monitoring video to construct a video sequence, carries out time sequence evaluation on the state of the detected open fire, and gives the detection result after calibration constraint.
Disclosure of Invention
The invention mainly solves the technical problems that: aiming at the problems that night light interference exists and false detection is easy to occur in night open fire detection, the night open fire detection method based on video time sequence correlation is provided. Comprises the following steps:
step 1: an open flame image dataset for training is constructed.
The quality of the data set in the deep learning is critical to the detection result, so that an original open flame image data set is constructed through various ways of open source image data, crawlers and shooting; the original open fire image data set comprises open fire images of various scenes and different distances in daytime and at night. And then adopting data labeling software to frame open fire in the original image data set to obtain a square frame containing open fire center coordinates and length and width, wherein the square frame is used for training an open fire detection network.
Step 2: and constructing an open fire detection network.
The open fire detection network mainly comprises a feature extraction network, a convolution module and an up-sampling module; the feature extraction network consists of a convolution layer, compresses the video sequence transmitted to the open fire detection network in the length and width directions, expands the channel number, distributes objects with the same features to corresponding channels, and obtains a feature map containing open fire information after processing. And (3) transmitting the feature map containing the open fire information to a convolution module and an up-sampling module, wherein the input of a second convolution module and the input of a third convolution module are added with the branch output in a feature extraction network on the channels of the feature map, then concentrating the channels of the feature map into 5 channels through the convolution module, wherein the first 4 channels represent the center x, y coordinates and width and height of the open fire, and the 5 th channel represents the confidence of the open fire. The up-sampling module amplifies the width and height of the feature map to 2 times of the original width and height, the two up-sampling modules are combined with the output of the two convolution modules and the output of the first convolution module, three outputs with three sizes can be obtained, the size relation of the three outputs is 4:2:1, the feature map with the largest size ratio is responsible for small target open fire, and the feature map with the smallest size ratio is responsible for large target open fire, so that the device has good detection effect on both the large target open fire and the small target open fire. And selecting one of the three outputs with the highest confidence coefficient as a final detection result, and mapping the detection result onto the original video sequence to obtain a sequence containing an open fire detection frame.
Step 3: and constructing a time sequence judging module.
The sequence containing the open fire detection frame obtained in the step 2 may contain a real open fire detection frame or a detection frame for light false detection. The video sequence containing open fire detection frames is sent to a time sequence judging module, detection frame information of a first frame in the video sequence is recorded and stored, then information corresponding to the same detection frame is searched on a second frame, and an IOU (input output unit) namely an intersection ratio is calculated with the detection frames of the first frame, the calculation flow is that the intersection area of the two detection frames is divided by the intersection area of the two frames, and the larger the IOU value is, the higher the coincidence between the two detection frames is. And then, carrying out the same processing between the third frame and the second frame of the video sequence, and cycling to process all the video sequences, thus obtaining the value of the IOU between all open fire detection frames between the frames, and constructing a line graph between the value of the IOU and the sequence number in a combined way. Compared with open fire, the night light is stable, the state change can not occur in the video sequence, the line graph of the night light tends to be in a gentle straight line state, the state of the open fire is continuously changed, and the line graph can show a change trend. Therefore, according to the state of the broken line, the object in the detection frame with the broken line gently deflected to the straight line is judged to be non-open fire, and the object in the detection frame with the broken line oscillating change is judged to be open fire.
Compared with the prior art, the invention has the following advantages:
1. the invention adopts the deep learning image processing technique method, is more accurate in open flame detection than the traditional image processing by utilizing color and contour detection, is not limited by scene, distance and open flame type, and can detect and process the shot picture in real time.
2. The timing sequence judging module is designed on the prior basis of deep learning open fire detection, and the motion state analysis is carried out on the detected open fire by combining the difference between the open fire and the lamplight, so that the false detection of the lamplight at night is effectively reduced.
Drawings
Fig. 1 is a schematic flow chart of a night open fire detection method based on video time sequence correlation in the invention.
Fig. 2 is a frame of an open flame detection method in the present invention.
Detailed Description
The invention mainly realizes the detection of open fire at night, and a specific method adopted by the invention is described in detail below with reference to the accompanying drawings.
Specifically, the overall flow of the night open fire detection method based on the video timing correlation is as shown in appendix fig. 1, and includes the following steps. S1, constructing an open flame image data set for training. S2, constructing an open fire detection network. S3, constructing a time sequence judging module.
(1) For S1, a fracture image dataset for network training is constructed.
The original open fire image data set is obtained through public resources, and the open fire image data set comprises open fires of various scenes and distances in the daytime and at night, so that the diversity of open fire detection is improved. And the data are randomly rotated and supplemented with light, so that the generalization of the data is improved. And then, marking open fire in the image by adopting a data marking method, and detecting the training of the network.
(2) And for S2, constructing an open fire detection network.
Open flame detection networks are shown in the upper part of the appendix of fig. 2. The designed network can be divided into a feature extraction network, a convolution module and an up-sampling module. The feature extraction network mainly comprises a convolution layer and is responsible for carrying out feature extraction on a video sequence transmitted to the network to obtain a feature map containing open flame information. And then, the feature map is conveyed to a convolution module and an up-sampling module after the feature extraction network, wherein the convolution module is in feature fusion with two branches in the feature extraction network and is responsible for fusing information of the feature map into 5 channels, the first 4 channels represent the center x, y coordinates and width and height of the open fire, and the 5 th channel represents the confidence of the open fire. The up-sampling layer is responsible for amplifying the width and height of the feature map to 2 times of the original width and height to obtain three outputs with multiple scales, the feature map with large size is responsible for a small target, and the feature map with small size is responsible for a large target, so that the up-sampling layer has a good detection effect on open fire of the large target and open fire of the small target. And selecting the one with the highest confidence as the final detection result for the three output results, and mapping the detection result onto the original video sequence to obtain a sequence containing an open fire detection frame.
(3) And for S3, constructing a time sequence judging module.
The structure diagram of the time sequence judging module is shown in the lower half part of the annex 2. Considering that the detection result of the open fire detection network may have false detection of night light, sending the detection result to a time sequence judging module, firstly recording and storing the detection frame information of the first frame in the video sequence, then searching the information corresponding to the same detection frame on the second frame, and calculating an IOU (cross-over ratio) with the detection frame of the first frame, wherein the larger the value is, the higher the overlap ratio between the two frames is. 1 represents complete coincidence and 0 represents no coincidence. And then, carrying out the same processing between the third frame and the second frame of the video sequence, and cycling to process all the video sequences, thus obtaining the value of the IOU between all open fire detection frames between every two frames, and constructing a line graph between the value and the sequence number in a combined way. Compared with open fire, the night light is stable, the state change can not occur in the video sequence, the line graph of the night light tends to be in a gentle straight line state, the state of the open fire is continuously changed, and the line graph can show a change trend. The analysis based on the state of the fold line detects whether the object in the box is an open fire.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting. Those skilled in the art will appreciate that: the above embodiments are not intended to limit the present invention in any way, and all similar technical solutions obtained by equivalent substitution or equivalent transformation are included in the protection scope of the present invention.
Claims (1)
1. A night open fire detection method based on video time sequence correlation is characterized in that: the method comprises the steps of,
step 1: constructing an open flame image dataset for training;
constructing an original open flame image data set by means of the open source image data, the crawlers and shooting various approaches; the original open fire image data set comprises open fire images of various scenes and different distances in daytime and at night; then, adopting data labeling software to frame open fire in the original image data set to obtain a square frame containing open fire center coordinates and length and width, wherein the square frame is used for training an open fire detection network; the open fire detection network training is to calculate the loss of the network training result and the marked square frame and optimize the network so that the result is continuously approximate to the marked value;
step 2: constructing an open fire detection network;
the open fire detection network consists of a feature extraction network, a convolution module and an up-sampling module; the feature extraction network consists of a convolution layer, compresses a video sequence transmitted to the open fire detection network in the length and width directions, expands the number of channels, distributes objects with the same features to corresponding channels, and obtains a feature map containing open fire information after processing;
the characteristic diagram containing open fire information is conveyed to a convolution module and an up-sampling module, wherein the input of a second convolution module and the input of a third convolution module are added with the branch output in a characteristic extraction network on characteristic diagram channels, then the channels of the characteristic diagram are concentrated into 5 channels through the convolution module, the first 4 channels represent the center x, y coordinates and width and height of the open fire, and the 5 th channel represents the confidence of the open fire; the up-sampling module amplifies the width and height of the feature map to 2 times of the original width and height, the two up-sampling modules are combined with the output of the two convolution modules and the output of the first convolution module to obtain three outputs with three sizes, the size relation of the three outputs is 4:2:1, the feature map with the largest size ratio is responsible for small target open fire, and the feature map with the smallest size ratio is responsible for large target open fire; selecting one of the three outputs with the highest confidence coefficient as a final detection result, and mapping the detection result onto the original video sequence to obtain a sequence containing an open fire detection frame;
step 3: constructing a time sequence judging module;
the sequence containing the open fire detection frame obtained in the step 2 may contain a real open fire detection frame or a detection frame for light false detection; the video sequence containing open fire detection frames is sent to a time sequence judging module, detection frame information of a first frame in the video sequence is recorded and stored, then information corresponding to the same detection frame is searched on a second frame, and an IOU (input output unit) namely an intersection ratio is calculated with the detection frames of the first frame, the calculation flow is that the area of intersection of the two detection frames is divided by the area of intersection of the two frames, and the larger the value of the IOU is, the higher the coincidence degree between the two detection frames is; then, the same processing is carried out between the third frame and the second frame of the video sequence, and the processing is circulated to the completion of all video sequences, so that the value of the IOU between all open fire detection frames between the frames is obtained, and a line graph is constructed between the value of the IOU and the sequence number in a combined way; and according to the state of the broken line, judging the object in the detection frame with the broken line gently deflected to the straight line as non-open fire, and judging the object in the detection frame with the broken line oscillating change as open fire.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210559156.8A CN115131622B (en) | 2022-05-22 | 2022-05-22 | Night open fire detection method based on video time sequence correlation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210559156.8A CN115131622B (en) | 2022-05-22 | 2022-05-22 | Night open fire detection method based on video time sequence correlation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115131622A CN115131622A (en) | 2022-09-30 |
| CN115131622B true CN115131622B (en) | 2024-03-29 |
Family
ID=83376947
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210559156.8A Active CN115131622B (en) | 2022-05-22 | 2022-05-22 | Night open fire detection method based on video time sequence correlation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115131622B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN112861635A (en) * | 2021-01-11 | 2021-05-28 | 西北工业大学 | Fire and smoke real-time detection method based on deep learning |
-
2022
- 2022-05-22 CN CN202210559156.8A patent/CN115131622B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2020181685A1 (en) * | 2019-03-12 | 2020-09-17 | 南京邮电大学 | Vehicle-mounted video target detection method based on deep learning |
| CN112861635A (en) * | 2021-01-11 | 2021-05-28 | 西北工业大学 | Fire and smoke real-time detection method based on deep learning |
Non-Patent Citations (2)
| Title |
|---|
| 基于改进YOLOv3的火灾检测与识别;任嘉锋;熊卫华;吴之昊;姜明;;计算机系统应用;20191215(12);全文 * |
| 背照CCD驱动电路技术研究;龚志鹏;于正阳;苏健;申才立;蔡帅;毛岩;;电子测量技术;20180408(07);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115131622A (en) | 2022-09-30 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209810B (en) | Boundary frame segmentation supervision deep neural network architecture for accurately detecting pedestrians in real time through visible light and infrared images | |
| CN113052835B (en) | Medicine box detection method and system based on three-dimensional point cloud and image data fusion | |
| CN107610175A (en) | The monocular vision SLAM algorithms optimized based on semi-direct method and sliding window | |
| CN106897720A (en) | A kind of firework detecting method and device based on video analysis | |
| CN103886107B (en) | Robot localization and map structuring system based on ceiling image information | |
| CN106384106A (en) | Anti-fraud face recognition system based on 3D scanning | |
| CN112686928B (en) | Moving target visual tracking method based on multi-source information fusion | |
| WO2022067668A1 (en) | Fire detection method and system based on video image target detection, and terminal and storage medium | |
| US11361534B2 (en) | Method for glass detection in real scenes | |
| CN112257526B (en) | Action recognition method based on feature interactive learning and terminal equipment | |
| CN111274921A (en) | Method for recognizing human body behaviors by utilizing attitude mask | |
| CN112257554B (en) | Forest fire recognition method, system, program and storage medium based on multiple spectra | |
| CN109887029A (en) | A kind of monocular vision mileage measurement method based on color of image feature | |
| CN115170792B (en) | Infrared image processing method, device and equipment and storage medium | |
| CN112561973A (en) | Method and device for training image registration model and electronic equipment | |
| Zhang et al. | An algorithm for automatic identification of multiple developmental stages of rice spikes based on improved Faster R-CNN | |
| CN109741307A (en) | Veiling glare detection method, veiling glare detection device and the veiling glare detection system of camera module | |
| Cao et al. | EFFNet: Enhanced feature foreground network for video smoke source prediction and detection | |
| CN111461036A (en) | Real-time pedestrian detection method using background modeling enhanced data | |
| CN116416576A (en) | Smoke/flame double-light visual detection method based on V3-YOLOX | |
| CN113298177B (en) | Night image coloring method, device, medium and equipment | |
| CN109684982B (en) | Flame detection method based on video analysis and combined with miscible target elimination | |
| CN113901931A (en) | Knowledge distillation model-based behavior recognition method for infrared and visible light videos | |
| CN114399734A (en) | Forest fire early warning method based on visual information | |
| CN114022820A (en) | Intelligent beacon light quality detection method based on machine vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |