CN110516609B

CN110516609B - Fire disaster video detection and early warning method based on image multi-feature fusion

Info

Publication number: CN110516609B
Application number: CN201910802918.0A
Authority: CN
Inventors: 陈美娟; 何爱龙; 管铭锋
Original assignee: Nanjing University of Posts and Telecommunications
Current assignee: Nanjing University of Posts and Telecommunications
Priority date: 2019-08-28
Filing date: 2019-08-28
Publication date: 2022-04-22
Anticipated expiration: 2039-08-28
Also published as: CN110516609A

Abstract

The invention discloses a fire video detection and early warning method based on image multi-feature fusion, which comprises the steps of firstly preprocessing after acquiring an image sequence of a video; then extracting a foreground region and then obtaining a detected candidate region; secondly, extracting static characteristics and dynamic characteristics from the candidate region, judging whether the flame is contained or not as the input of an SVM classifier when the flame is detected, and obtaining whether the smoke is contained or not after logical combination selection and calculation are carried out on the judgment result of the characteristics when the smoke is detected; and finally, if the fire disaster detector detects that the fire disaster detector contains flame or smoke, judging the fire disaster according to the growth trend, and alarming the fire disaster on a monitoring site when the fire disaster is judged to be formed, otherwise, only warning the fire disaster in the background. The invention can be combined with the existing monitoring system, is applied to places such as markets, warehouses and the like, reduces the fire detection early warning cost, has good generalization capability and applicability of the detection method, can provide reliable fire detection and early warning functions, and has practical value.

Description

Fire disaster video detection and early warning method based on image multi-feature fusion

Technical Field

The invention relates to the field of image processing and recognition in computer vision, in particular to a fire disaster video detection and early warning method based on image multi-feature fusion.

Background

The fire hazard is one of the main threats threatening the safety of human life and property, and along with the development of the society, the harm of the fire hazard to human society and the natural environment is more and more serious. Therefore, it is an important research subject to accurately and rapidly detect a fire. The traditional fire detection equipment at present mainly uses a temperature-sensing and smoke-sensing detector, but the detector mainly using a sensor needs to reach a detection threshold value to alarm, the delay is serious, and the detection effect can be achieved only by laying the detector in a large scale in space due to a small detection range, so that the cost is very high. In addition, in high buildings, forests, tunnels and the like, such devices often fail to accurately detect fires in time due to dilution of space and airflow.

With the development of a video monitoring system, monitoring cameras are distributed throughout streets and alleys, and the fire detection task can be completed on the basis of the existing video monitoring system by combining the image processing and image recognition technologies, so that the cost can be reduced, the anti-interference capability is improved, and the system is well suitable for complex environments with large space and much airflow; there are many image-based fire detection methods, but these methods have the following problems: on the premise of ensuring reliable detection, the delay needs to be further improved; the detection of flame is mainly used, and the detection capability of the flame detecting device for smoldering fire is insufficient; the lack of a mechanism for distinguishing whether the detection result constitutes a fire or not and for classifying and alarming the fire causes panic to public alarms under normal fire use or controllable conditions.

Disclosure of Invention

The purpose of the invention is as follows: in order to overcome the defects in the prior art, the invention provides a fire video detection and early warning method based on image multi-feature fusion. The invention can combine the existing video monitoring system to simultaneously realize the detection of flame and smoke, and firstly, the median filtering is utilized to carry out denoising pretreatment on the image; a new background model is adopted to better extract a motion area; the generalization capability and the detection efficiency of the detection method are improved by adopting a support vector machine classifier and a logic arithmetic unit capable of adjusting the detection sensitivity; and finally, a judgment mechanism for judging whether the fire is possible to form is introduced to process the fire in a grading way, so that the reliability of fire early warning is improved, and the harm caused by the fire is reduced to the maximum extent.

The technical scheme is as follows: in order to achieve the purpose, the invention adopts the technical scheme that:

a fire video detection and early warning method based on image multi-feature fusion is characterized in that after an image sequence of a video is obtained, preprocessing is firstly carried out to reduce noise and improve image quality; then extracting a foreground region by adopting a moving region extraction method to eliminate most interference, selecting the foreground region by using a color model, and performing morphological closing operation to obtain a detected candidate region; secondly, extracting static characteristics and dynamic characteristics from the candidate region, judging whether the flame is contained or not as the input of an SVM classifier when the flame is detected, and obtaining whether the smoke is contained or not after logical combination selection and calculation are carried out on the judgment result of the characteristics when the smoke is detected; and finally, if the fire disaster detector detects that the fire disaster detector contains flame or smoke, judging the fire disaster according to the growth trend, and alarming the fire disaster on a monitoring site when the fire disaster is judged to be formed, otherwise, only warning the fire disaster in the background. The invention can be combined with the existing monitoring system, applied to places such as shopping malls, storage and the like, reduces the fire detection early warning cost, has good generalization capability and applicability of the detection method, can provide reliable fire detection and early warning functions, has practical value, and specifically comprises the following steps:

s1, carrying out denoising pretreatment on the obtained video image sequence by using a median filtering method;

s2, establishing a background model for the preprocessed image by using an improved method based on a gradient motion historical map to extract a motion region;

s3, extracting the pixels of the motion area by using the color models of flame and smoke respectively to obtain a color suspected area;

s4, performing morphological closing operation on the color suspected area to obtain a smooth and continuous suspected area;

s5, extracting dynamic and static characteristics of the suspected area;

s6, inputting the extracted flame characteristic values as SVM classifier to judge whether the flame is contained, simultaneously, respectively calculating the extracted smoke characteristics and inputting the calculation results into a logic arithmetic unit to judge whether the smoke is contained;

s7 inputs a continuous sequence of images containing flames or smoke into a fire discriminator to determine whether or not a fire is established, and different warning processes are performed.

Further, the step S1 of using median filtering on the acquired video image sequence specifically includes:

and traversing and calculating the gray value of each pixel point of the initial image, and calculating the median of the gray values in the L multiplied by L neighborhood of each pixel point to replace the original gray value of the pixel, so that the noise in the image, particularly the salt and pepper noise, can be effectively removed, and the image quality is improved.

Further, the step S2 of extracting the motion region by using a method based on a gradient motion history map specifically includes the following steps:

s21 first calculates a binary gradient map BGI (x, y) by the following formula:

wherein, BGI_x(x, y) and BGI_y(x, y) are images after binarization processing is carried out on gradient images in the directions of an x axis and a y axis respectively, and the gradient images are calculated from gray level images;

s22 calculates a gradient motion history map GMHI (x, y), which records motion history information of each pixel, as follows:

t is the current time, T is a preset time interval, the time is usually represented by a frame, and only the motion history information in the latest T frame is recorded;

s23 calculates the effective motion map EMI (x, y) by the following formula:

the method includes the steps that MAX (x, y) and MIN (x, y) are maximum GMHI values and minimum GMHI values in an m multiplied by m field of a pixel point (x, y) in a gradient motion historical map respectively, MIN and MAX are preset constants, when a difference value of the MAX (x, y) and the MIN (x, y) is larger than MAX, the pixel point (x, y) is represented as a static pixel or a pixel point on a background edge, and conversely, when the difference value is smaller than MIN, the pixel point (x, y) is represented as a static pixel or a pixel on an inner edge of the background, so that a motion area is recorded in an effective motion map.

Further, in step S3, establishing a color model for the flame and smoke and extracting a suspected color area from the moving area, the method specifically includes:

s31 early stage flame temperature is low, so color is mainly distributed in red, orange, and yellow regions, and red is a main component, and the color features of the flame are described based on RGB color space according to the features, in order to prevent interference of luminance information on color information, and the luminance component in the flame image is large, HSV and YCbCr color spaces are used to further define the luminance and color features, and the flame color model rule is based on three color spaces of RGB, HSV, and YCbCr, and the rule phase is matched to obtain a color suspected region of flame, and the specific rule includes:

rule 1: r (x, y)>R_T

Rule 2: r (x, y) ≧ G (x, y) > B (x, y)

Rule 3: s (x, y) ≥ 255-R (x, y)) S_T/R_T

Rule 4: h₁<H(x,y)<H₂,S₁<S(x,y)<S₂,V₁<V(x,y)<V₂

Rule 5: y (x, Y) is more than or equal to Cr (x, Y) is more than or equal to Cb (x, Y)

Rule 6: y (x, Y)>Y_mean,Cb(x,y)<Cb_mean,Cr(x,y)>Cr_mean

Rule 7: cr (x, y) -Cb (x, y) ≥ lambda

In the formula, R (x, y), G (x, y) and B (x, y) respectively represent red, green and blue components of the pixel point (x, y) in RGB space, and R (x, y) represents red, green and blue components in RGB space_TH (x, y), S (x, y) and V (x, y) are three components of hue, saturation and brightness of the pixel point (x, y) in HSV color space respectively, and S is a threshold value of a red component_TIs a red component of R_TCorresponding to the value of saturation in HSV space. H₁、H₂、S₁、S₂、V₁、V₂Is a threshold value preset for hue, saturation and brightness, Y (x, Y), Cb (x, Y) and Cr (x, Y) are respectively the brightness and red components of pixel point (x, Y) in YCbCr spaceBlue component, lambda is the non-negative threshold value of the difference value of the two components Cr (x, Y) and Cb (x, Y), and Y is_mean、Cb_mean、Cr_meanRespectively taking the average values of all pixels in the current image on three components of Y, Cb and Cr;

the smoke of S32 is the main sign of early fire and smoldering fire, and is often represented as gray, so its R, G, B three component values are approximately equal, and its brightness value also varies within the corresponding interval due to the difference of the combustion products and the separation into light gray and dark gray. The adopted smog color model rule is based on RGB and HSI spaces, and the rule is subjected to phase comparison to obtain a color suspected area, wherein the rule specifically comprises the following steps:

rule 1: delta is less than or equal to | R (x, y) -G (x, y) |₁

Rule 2: delta is less than or equal to | R (x, y) -B (x, y) |₂

Rule 3: delta is less than or equal to | G (x, y) -B (x, y) |₃

Rule 4: l is₁≤I(x,y)≤L₂||D₁≤I(x,y)≤D₂

Wherein, delta₁、δ₂、δ₃Is a threshold value of the absolute difference between R, G, B three components, L₁、L₂、D₁、D₂The upper and lower limits of the variation of the luminance I (x, y) in light gray and dark gray, respectively, are constant.

Further, in step S4, a morphological operation of expanding and then corroding the suspected color area is performed, and the small holes and the small seams are filled and leveled, so as to obtain a smooth and continuous suspected flame area and a smooth and continuous suspected smoke area.

Further, in the step S5, the extracting of the dynamic and static characteristics of the suspected areas of the flame and the smoke respectively includes:

s51 extracts dynamic and static characteristic parameters of the flame from the suspected flame region, specifically including edge complexity, randomness, similarity, disorder change rate, and jitter frequency of the suspected flame region, and respectively expressed as E, Δ a, θ, ω, and p, and the specific calculation method is as follows:

static characteristics of A

Edge complexity

The edge of the flame between the continuous frames is different and varies greatly, the complexity of the edge is different from that of other interference objects, and the edge can be used as a characteristic of other objects in a partition and is defined as follows:

e is the average edge complexity of k continuous flame regions of the current frame, the larger E represents the higher edge complexity, P_jThe perimeter of the jth continuous suspected flame area in the frame can be represented by the number of pixels of the edge, A_jThe area of the region can be represented by the number of pixels in the region;

b dynamic characteristics

1 degree of randomness

The flame changes from one frame to another frame with a certain degree of randomness, so the difference between two continuous frames can be counted to represent the characteristic, and the calculation formula is as follows:

ΔA＝|A_t-A_t-1|

A_tand A_t-1Respectively representing the pixel quantity of candidate areas of a t frame and a t-1 frame, wherein Delta A is the absolute pixel quantity difference of the candidate areas of two continuous frames;

2 degree of similarity

The shape of the flame looks irregular but shows a certain similarity in a short time interval, and adjacent frames change within a certain range, which is different from other moving light sources or interferents with similar colors, and the description formula is as follows:

wherein, F_tAnd F_t-1The areas of suspected flames in the t-th and t-1-th frames, respectively, F_t∩F_t-1To calculate the area of the overlapping area of the suspected flame areas in two adjacent frames, F_t∪F_t-1After the areas of suspected flames in two adjacent frames are calculated and mergedArea of the region (d);

3 rate of change of disorder degree

In the process of flame combustion, the flame shape can be changed all the time due to the influence of factors such as combustion substances, airflow and the like, and a disorder degree change rate formula for describing the change is as follows:

FD_t(x,y)＝F_t(x,y)-F_t-1(x,y)

FD_t+1(x,y)＝F_t+1(x,y)-F_t(x,y)

wherein ω represents the disorder degree change rate, F_t(x, y) denotes the t-th frame pseudo-flame area, FD_t(x, y) represents the degree of disorder of the region, V_t(i) Denotes FD_t(x, y) a value of an ith pixel, N representing an amount of pixels in the suspected flame region;

4 jitter frequency

The jitter frequency is a characteristic which has high reliability, strong anti-interference capability and easy description in dynamic characteristics. The main idea of the jitter frequency detection is to count the frame difference result of two adjacent frames in a certain period, and the jitter frequency is calculated in the following manner:

where T is the period of the statistics, D_i(t) is the value of the ith pixel in the suspected flame area in the tth frame, N is the pixel amount in the area, p_iThe accumulated difference of pixel values in any pixel point T frame in the region is shown, and p is the jitter frequency;

s52, after obtaining the suspected smoke area, extracting and analyzing the dynamic and static characteristic parameters of the smoke, wherein the parameters specifically comprise the area growth value, the circularity and the displacement of the smoke area, and the specific calculation methods of the characteristics are as follows:

1 area growth analysis

The smoke will increase after the occurrence of the smoke, so the pixel quantity of the candidate area will show an increasing characteristic, and the calculation formula is:

where T is the period of the statistic, S_tAnd S_t-1The area of the smoke candidate area of the t frame and the t-1 frame is obtained, and g is the average growth rate;

2 degree of circularity analysis

The circularity describes the complexity of the shape of the object, when the object is circular, the circularity is the lowest and has a value of 1, and when the shape of the object is more complex, the circularity is the larger, the circularity calculation formula is:

wherein L is_tIs the perimeter of the smoke candidate region in the t-th frame, S_tThe area of the smoke candidate area in the t frame is shown;

3 degree of displacement analysis

After the smoke appears, the shape, the area and the shape of the smoke are changed constantly, but the central position and the main body area of the smoke are kept in a certain area, so that the change of the central position is not enough to be considered singly, and the dynamic change of the area needs to be considered, so that the expression is more reasonable by using the area overlapping degree, namely the area overlapping degree of the same suspected target area between adjacent frames is studied, and the mathematical expression is as follows:

s (t, j) and S (t-1, j) represent the area of the jth candidate region in the t-th and t-1 frames, respectively, S_mIndicates that the two frames are includedThe area of the minimum bounding rectangle of the jth candidate region;

finally, the obtained characteristic parameters are respectively compared with a predefined threshold value to obtain an output result whether the output result accords with the smoke characteristics, and the output result is true or false;

further, the step S6 is to use the extracted flame feature values as SVM classifier inputs to determine whether flame is contained, and to separately calculate the extracted smoke features and input the calculation results to the logic arithmetic unit to determine whether smoke is contained, and the specific steps include:

s61, the SVM classifier has a faster detection speed than other classifiers and a neural network, a better detection effect can be obtained on a small sample training set, and the generalization capability is excellent, so that the SVM classifier is adopted to classify flame images, parameters in the SVM are optimized firstly, training samples are collected to train an SVM model, the characteristic value of flame is used as the input of the SVM, whether flame exists in a video or not is judged, wherein the input characteristic vector is as follows: x is the number of_i＝[E,△A,θ,ω,p](ii) a The output result of the SVM is as follows:

wherein, the weight value w_e＝α_ey_e，K(x,x_e) Is a kernel function, α_eAs lagrange multipliers, constraints

α_eMore than or equal to 0, e is 1,2, n; the kernel function is a radial basis kernel function, and the formula is as follows:

K(x_e,x_f)＝exp(-γ||x_e-x_f||²)

wherein, | | x-x_e||²Is the squared euclidean distance between the two eigenvectors, gamma being a parameter of the kernel function;

s62, true and false results obtained by comparing the smoke growth value, the circularity and the displacement with a threshold value are input into a logic arithmetic unit, so that whether smoke exists in the image or not is judged, the logic arithmetic unit can flexibly select and combine logical arithmetic combinations such as AND, OR, NOT, XOR and the like to calculate the three results, and according to different detection scenes, different arithmetic modes are adopted to adjust the sensitivity of the detection algorithm and increase the generalization capability of the algorithm, for example:

for highly sensitive areas such as petrochemical storage and the like, an OR gate can be used, and smoke is judged to be contained when a suspected smoke area meets a condition;

for monitoring a general production and living area, at least any two conditions meet the requirement of judging that the image contains smoke;

in places such as workshops containing fire work and smoke dust, the output results of the three judgment conditions can be subjected to logical operation by using an AND gate, and the smoke is judged to be contained when the three conditions are simultaneously met.

Further, the step S7 inputs the image sequence containing flame or smoke into the fire discriminator to determine whether a fire is formed, and performs different processes, including:

s71 obtaining two continuous frames to judge the area containing flame or smoke and calculating the area M of each corresponding area_aAnd M_a-1，M_aAnd M_a-1The areas of corresponding regions in the current frame and the previous frame, respectively;

s72, calculating the area growth rate beta of the flame or smoke between two continuous frames, wherein the calculation formula is as follows:

s73 if the area growth rate beta is greater than zero and greater than the threshold value G_TIf the initial value is 0, the variable Count is increased by 1, and the calculation formula is:

s74 when the growth rate of the continuous h frames is larger than the threshold value G_TWhen the fire is detected, i.e. when the Count is equal to h, it is determined that the monitored site contains the fire, and the fire is detectedCarrying out fire alarm on a monitoring site, otherwise, only carrying out fire alarm prompt on a monitoring background, wherein the threshold value G of the growth rate of flame and smoke_TThe specific values of (a) are different.

The invention has the following beneficial effects:

the invention provides a fire video detection and early warning method based on image multi-feature fusion, which is characterized in that a fire image is processed, and a plurality of features are extracted and analyzed on the basis of a suspected fire area, so that flame and smoke shown in a fire are detected at the same time, the method is suitable for detecting both open fire conditions and smoldering fire conditions of smoke, and has better application range and reliability;

2, denoising an original image through median filtering, improving the image quality, and combining a foreground extraction method based on gradient motion historical image improvement to achieve the purposes of improving the motion region extraction effect, removing most useless pixels and improving the calculation speed;

3, the method further detects the motion area by combining a multi-color space and establishing a color model rule, fills small holes and gaps by adopting morphological closing operation, smoothes and highlights a target area to obtain a complete suspected flame or smoke area, eliminates the motion area which does not accord with a color model, further reduces the detection range, obtains the suspected flame or smoke area as a candidate area, and ensures the extraction and judgment of later dynamic and static characteristics, thereby reducing the false alarm rate;

4, by researching the characteristic engineering, the invention adopts a plurality of fire characteristics which are not easy to be interfered, and optimizes the mathematical calculation process of the characteristics, so that the characteristics are more concise and effective, the calculation time is shortened, and the anti-interference capability of the detection method is improved;

the suspected area is further judged through the SVM classifier, so that the method has excellent generalization capability, can obtain better detection accuracy and shorter detection time than other algorithms on a small sample training set, and is more suitable for a video monitoring system;

the logic arithmetic unit provided by the invention can change the sensitivity of the detection method according to different scenes by calculating the judgment results of multiple characteristics of smoke through the combination of different logic gates, thereby achieving better early warning effect in different scenes;

7 the fire judgment module provided by the invention provides a fire grading early warning mechanism, when the fire condition is judged to form a fire trend, the fire grading early warning mechanism alarms a monitoring field and informs personnel to evacuate orderly, otherwise, the fire condition early warning prompt is only carried out on management personnel at a background, and processing measures are taken to prevent the production and entertainment personnel in a monitoring area from unnecessarily hurry, even trample and other accidents.

Drawings

FIG. 1 is an overall flowchart of a fire video detection and early warning method based on image multi-feature fusion according to an embodiment of the present invention;

FIG. 2 is a motion region extraction image of an embodiment of the present invention;

FIG. 3 is a flow chart of the calculation of the fire determination module according to the embodiment of the present invention;

fig. 4 is a diagram illustrating an exemplary video sample detection result according to an embodiment of the present invention.

Detailed Description

For a better understanding of the present invention, the present invention will be described in detail with reference to the accompanying drawings and the following embodiments, and it is to be understood that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments may be obtained by those skilled in the art without any inventive work based on the embodiments of the present invention.

Example 1

As shown in fig. 1, the present embodiment provides a fire video detecting and early warning method based on image multi-feature fusion, which includes the following steps:

s3, extracting the pixels of the motion area by using the color models of flame and smoke respectively to obtain a suspected color area;

s5, extracting dynamic and static characteristics of the suspected area;

s6, inputting the extracted flame characteristic value as an SVM classifier to judge whether the flame is contained, simultaneously calculating the extracted smoke characteristic and inputting the result to a logic arithmetic unit to judge whether the smoke is contained;

s7 inputs the continuous image sequence containing flame or smoke into a fire discriminator to determine whether a fire is formed, and different warning processes are performed.

The embodiment combines the existing video monitoring system, can realize the detection of flame and smoke simultaneously in the monitoring area, thereby realizing the function of fire early warning. The image quality is improved by denoising preprocessing, then a background model is adopted to extract a motion area, most useless pixels are eliminated, the calculation speed is greatly improved, the adopted characteristics are not easily interfered by noise through characteristic engineering research, the mathematical description of the characteristics is convenient and direct, the detection speed and accuracy are increased, the generalization capability and the detection efficiency of the detection method are increased through a support vector machine classifier and a logic arithmetic unit capable of adjusting the detection sensitivity, and finally the method for judging the fire is introduced to process the fire in a grading mode to increase the reliability of fire alarm, so that the harm caused by the fire is reduced to the maximum extent.

Example 2

The embodiment is further optimized based on embodiment 1, and specifically includes:

the step S1 of using median filtering on the obtained video image sequence specifically includes:

and traversing and calculating the gray value of each pixel point of the initial image, and calculating the median of the gray values in the 3 x 3 neighborhood of each pixel point to replace the original gray value of the pixel, so that the noise in the image, particularly the salt and pepper noise, can be effectively removed, and the image quality is improved.

In step S2, a motion region is extracted by using a method based on a gradient motion history map, where the motion region extraction result is shown in fig. 2, white is a motion region, and black is a background, and the method specifically includes the following steps:

s21 first calculates a binary gradient map BGI (x, y) by the following formula:

wherein, BGI_x(x, y) and BGI_y(x, y) are images obtained after binarization processing is carried out on gradient maps in the directions of an x axis and a y axis respectively, and the gradient maps are calculated from gray-scale maps after median filtering;

t is the current time, T is a preset time interval, generally, a frame is used for representing the time, and is set to be 10, and only the motion history information in the latest 10 frames is recorded;

s23 calculates the effective motion map EMI (x, y) by the following formula:

wherein, MAX (x, y) and MIN (x, y) are the maximum and minimum GMHI values in the 5 × 5 field of the pixel (x, y) in the gradient motion history map, MIN and MAX are preset constants and are respectively set to 0 and 10, when the difference between MAX (x, y) and MIN (x, y) is greater than MAX, it indicates that the pixel (x, y) is a stationary pixel or a pixel on the edge of the background, otherwise, when the difference is less than MIN, it indicates that the pixel (x, y) is a stationary pixel or a pixel on the inner edge of the background, therefore, the effective motion map records the motion region.

In the step S3, a color model is established for the flame and smoke, and a suspected color area is extracted from the moving area, and the specific steps include:

rule 1: r (x, y)>R_T

Rule 2: r (x, y) ≧ G (x, y) > B (x, y)

Rule 3: s (x, y) ≥ 255-R (x, y)) S_T/R_T

Rule 4: h₁<H(x,y)<H₂,S₁<S(x,y)<S₂,V₁<V(x,y)<V₂

Rule 6: y (x, Y)>Y_mean,Cb(x,y)<Cb_mean,Cr(x,y)>Cr_mean

Rule 7: cr (x, y) -Cb (x, y) ≥ lambda

In the formula, R (x, y), G (x, y) and B (x, y) respectively represent red, green and blue components of the pixel point (x, y) in RGB space, and R (x, y) represents red, green and blue components in RGB space_TThe threshold value for the red component is 115 in this embodiment, H (x, y), S (x, y), and V (x, y) are three components of hue, saturation, and brightness of the pixel (x, y) in HSV color space, respectively, and S is a value of the threshold value for the red component_TIs a red component of R_TThe value corresponding to the saturation in the HSV space is set to 55 in this embodiment. H₁、H₂、S₁、S₂、V₁、V₂The threshold values are preset for hue, saturation and brightness, and are respectively 0.02, 0.30, 0.20, 1.00, 0.98 and 1.00 in this embodiment, and Y (x, Y), Cb (x, Y) and Cr (x, Y) are respectively the brightness, red component, blue,And the blue component, lambda is a non-negative threshold value of the difference value of the two components Cr (x, y) and Cb (x, y), and is set to be 40. Y is_mean、Cb_mean、Cr_meanRespectively obtaining the average values of all pixels in the current image on three components of Y, Cb and Cr;

rule 1: delta is less than or equal to | R (x, y) -G (x, y) |₁

Rule 2: delta is less than or equal to | R (x, y) -B (x, y) |₂

Rule 3: delta is less than or equal to | G (x, y) -B (x, y) |₃

Rule 4: l is₁≤I(x,y)≤L₂||D₁≤I(x,y)≤D₂

Wherein, delta₁、δ₂、δ₃The threshold value for the absolute difference between the three components R, G, B is set as 16, 48, 34, L in this embodiment₁、L₂、D₁、D₂The upper and lower limit constants of the luminance I (x, y) change in light gray and dark gray, respectively, and are set to 150, 220, 67, and 130 in this embodiment.

In the step S4, a morphological operation of expanding and then corroding is performed on the suspected color area, and the small holes and the small seams are filled and leveled, so as to obtain a smooth and continuous suspected flame area and a smooth and continuous suspected smoke area.

In the step S5, the extraction of the dynamic and static characteristics of the suspected areas of the flame and the smoke is performed respectively, and the specific steps include:

s51 extracts dynamic and static characteristic parameters of the flame from the suspected flame region, specifically including edge complexity, randomness, similarity, disorder change rate, and jitter frequency of the flame region, and respectively expressed as E, Δ a, θ, ω, and p, and the specific calculation method is as follows:

static characteristics of A

Edge complexity

b dynamic characteristics

1 degree of randomness

ΔA＝|A_t-A_t-1|

2 degree of similarity

wherein, F_tAnd F_t-1The areas of suspected flames in the t-th and t-1-th frames, respectively, F_t∩F_t-1To calculate the area of the overlapping area of the suspected flame areas in two adjacent frames, F_t∪F_t-1Calculating the area of the area after the areas of the suspected flames in the two adjacent frames are subjected to union set;

3 rate of change of disorder degree

FD_t(x,y)＝F_t(x,y)-F_t-1(x,y)

FD_t+1(x,y)＝F_t+1(x,y)-F_t(x,y)

wherein ω represents the disorder degree change rate, F_t(x, y) denotes the area of suspected flame in the t-th frame, FD_t(x, y) represents the degree of disorder in the region t, V_t(i) Denotes FD_t(x, y) a value of an ith pixel, N representing an amount of pixels in the suspected flame region;

4 jitter frequency

where T is the statistical period 10, D_i(t) is the value of the ith pixel in the suspected flame area in the tth frame, N is the pixel amount in the area, p_iThe accumulated difference of pixel values in any pixel point T frame in the region is shown, and p is the jitter frequency;

1 area growth analysis

where T is the statistical period 10, S_tAnd S_t-1The areas of the smoke candidate regions of the t-th frame and the t-1 th frame are calculated by pixel quantity, g is an average growth rate, the lower limit of a variation range is set to be 0.005, and the upper limit is set to be 0.1;

2 degree of circularity analysis

Circularity describes the complexity of the shape of an object, and when the object is circular, its circularity is the lowest, and its value is 1. The more complex the shape of the object, the more its circularity is. The circularity calculation formula is:

wherein L is_tThe perimeter of the smoke candidate area in the t frame is expressed by the area edge pixel quantity, S_tThe area of the smoke candidate area in the t-th frame is represented by the pixel quantity of the candidate area, the variation range of c is set to be 3-40, and the output result is true in the range;

3 degree of displacement analysis

s (t, j) and S (t-1, j) represent the area of the jth candidate region in the t-th and t-1 frames, respectively, S_mRepresenting the area of the smallest bounding rectangle containing the jth candidate region in the two frames, where the area is replaced in the calculation by the amount of pixels in the candidate region, when d>When 0.5 hour, the smoke is judged to be smoke, and the output is true;

and finally, comparing the acquired characteristic parameters with a predefined threshold value respectively to obtain an output result of whether the smoke characteristics are met, wherein the output result comprises true and false.

In step S6, the extracted flame feature parameters are input as SVM classifiers to determine whether flames are contained, and in addition, the extracted smoke features are analyzed respectively and the analysis results are input to a logic operator to determine whether smoke is contained, which includes the following specific steps:

the S61SVM classifier has a faster detection speed than other classifiers and a neural network, can obtain a better detection effect on a small sample training set, and has excellent generalization capability, so that the SVM classifier is adopted to classify flame images, firstly, training samples are collected to train an SVM model, the characteristic parameters of flame are used as the input of an SVM, whether flame exists in a video is judged, and the input characteristic vector is as follows: x is the number of_i＝[E,△A,θ,ω,p](ii) a The output result of the SVM is as follows:

wherein, the weight value w_e＝α_ey_e，K(x_i,x_j) Is a kernel function, α_eAs lagrange multipliers, constraints

K(x_e,x_f)＝exp(-γ||x_e-x_f||²)

wherein, | | x_e-x_f||²The squared Euclidean distance between two feature vectors is adopted, gamma is a parameter of a kernel function, and when the parameter is 0.008, a better classification result is obtained;

As shown in fig. 3, the step S7 inputs the image sequence containing flame or smoke into the fire discriminator to determine whether a fire is formed, and performs different processes, including:

s71 obtaining two continuous frames to judge the area containing flame or smoke and calculating the area M of each corresponding area_aAnd M_a-1，M_aAnd M_a-1The areas of corresponding regions in the current frame and the previous frame, respectively, here expressed in terms of the amount of pixels in the region;

s74 when the growth rate of five continuous frames is greater than the threshold value G_TIf the number is 5, judging that the monitoring site contains fire, carrying out fire alarm on the monitoring site, otherwise, only carrying out fire alarm prompt on the monitoring background, wherein the growth rate threshold G for flame and smoke is_TIs different, the threshold value G of the flame_TSet to 0.18 and smoke set to 0.01.

In this embodiment, the training set is derived from a fire video library of the university of Bilkent, the test set is collected in the Internet, the test result is shown in FIG. 4, where a is a flame video, b is a smoke video, and e is a non-fire video with color interference, the fire area is identified by the algorithm and its position is marked by a rectangular frame, but the non-fire video is not detected.

The above description is only a preferred embodiment of the present invention, and not intended to limit the present invention, the scope of the present invention is defined by the appended claims, and all structural changes that can be made by using the contents of the description and the drawings of the present invention are intended to be embraced therein.

Claims

1. A fire video detection and early warning method based on image multi-feature fusion is characterized by comprising the following steps:

step S1, acquiring a video image, and performing denoising pretreatment on the acquired video image sequence;

step S2, establishing a background model for the image after the denoising pretreatment by using an improved method based on a gradient motion historical map to extract a motion region;

in step S21, first, a binary gradient map BGI (x, y) is calculated as:

step S22, calculating a gradient motion history map GMHI (x, y), and recording motion history information of each pixel, wherein the calculation formula is as follows:

wherein t represents the current frame;

step S23, calculating an effective motion map EMI (x, y), the calculation formula is as follows:

wherein, MAX (x, y) and MIN (x, y) are the maximum and minimum GMHI values in the m × m field of the pixel (x, y) in the gradient motion history map, MIN and MAX are preset constants, when the difference between MAX (x, y) and MIN (x, y) is greater than MAX, it indicates that the pixel (x, y) is a stationary pixel or a pixel on the edge of the background, and when the difference is less than MIN, it indicates that the pixel (x, y) is a stationary pixel or a pixel on the inner edge of the background;

step S3, extracting the pixels of the motion area by using the color models of flame and smoke respectively to obtain a suspected color area;

step S4, performing morphological closing operation on the color suspected area to obtain a smooth and continuous suspected area;

step S5, extracting dynamic and static characteristics of the suspected area;

the method comprises the following steps of respectively extracting dynamic and static characteristics of suspected areas of flame and smoke, and specifically comprises the following steps:

step S51 is to extract dynamic and static characteristic parameters of the flame from the suspected flame region, specifically including edge complexity, randomness, similarity, disorder change rate, and jitter frequency of the flame region, and respectively expressed as E, Δ a, θ, ω, and p, and the specific calculation method is as follows:

static characteristics of A

The edge complexity is defined as follows:

wherein E is the average edge complexity of k continuous flame regions of the current frame, the larger E represents the higher edge complexity, and P_jIs the perimeter of the jth continuous suspected flame region of the frame, A_jIs the area of the region;

b dynamic characteristics

The random degree formula is as follows:

ΔA＝|A_t-A_t-1|

wherein A is_tAnd A_t-1Respectively representing the pixel quantity of candidate areas of a t frame and a t-1 frame, wherein Delta A is the absolute pixel quantity difference of the candidate areas of two continuous frames;

the similarity θ is formulated as:

wherein, F_t(x, y) and F_t-1(x, y) indicates the suspected flame area of the t-th frame and t-1 frame, respectively, F_t(x,y)∩F_t-1(x, y) calculating the overlapping area of the suspected flame areas in two adjacent frames, F_t(x,y)∪F_t-1(x, y) calculating the area of the areas of the suspected flame areas in the two adjacent frames after the suspected flame areas are subjected to union set;

the disorder degree change rate formula is as follows:

FD_t(x,y)＝F_t(x,y)-F_t-1(x,y)

FD_t+1(x,y)＝F_t+1(x,y)-F_t(x,y)

wherein ω represents the disorder degree change rate, F_t(x, y) denotes a pseudo-flame area at the t-th frame, FD_t(x, y) represents the degree of disorder in the region t, V_t(i) Denotes FD_t(x, y) a pixel value of an ith pixel, N representing an amount of pixels in the region;

the jitter frequency mode is as follows:

step S52, after the suspected smoke area is obtained, the dynamic and static characteristic parameters of the smoke are extracted and analyzed;

1, area growth analysis

Where T is the period of the statistic, S_tAnd S_t-1The areas of the smoke candidate areas of the t frame and the t-1 frame are obtained, g is the average growth rate, and if the g is within a predefined range, the output of the area growth analysis is true;

2, analysis of circularity

Wherein L is_tThe perimeter of the smoke candidate area in the t frame is expressed by the area edge pixel quantity, S_tC is the circularity of the suspected smoke area, and when the circularity is within a predefined range, the feature analysis output result is true;

3, degree of Displacement analysis

Wherein S (t, j) and S (t-1, j) respectively represent the area of the jth candidate region in the t-th and t-1 frames, and S_mThe area of the minimum circumscribed rectangle containing the jth candidate region in the two frames is represented, the area is replaced by the pixel quantity in the candidate region in calculation, and when d is larger than a predefined threshold value, the smoke is judged to be smoke, and the output is true;

finally, the obtained characteristic parameters are respectively compared with a predefined threshold value to obtain an output result whether the smoke characteristics are met, and the output result comprises true and false;

step S6, the extracted flame characteristic value is used as SVM classifier input to judge whether flame is contained, simultaneously, the extracted smoke characteristic is calculated and the result is input to a logic arithmetic unit to judge whether smoke is contained;

step S7, inputting the continuous image sequence containing flame or smoke into a fire discriminator to judge whether a fire is formed, thereby respectively making different early warning treatments;

step S71, acquiring the area M of the flame or smoke area in two continuous frames_aAnd M_a-1，M_aAnd M_a-1The areas of corresponding regions in the current frame and the previous frame, respectively;

step S72, calculating the area growth rate beta of the flame or the smoke between two continuous frames, wherein the calculation formula is as follows:

step S73, if the area increase rate beta is larger than zero and larger than the threshold value G_TIf the initial value is 0, the variable Count is increased by 1, and the calculation formula is:

step S74, when the growth rate of the continuous h frames is larger than the threshold value G_TIf the fire alarm is not carried out, only a monitoring background is used for carrying out fire alarm prompting, wherein the threshold value G of the growth rate of flame and smoke is used for judging that the monitoring site contains the fire, otherwise, the fire alarm is carried out on the monitoring site, and the fire alarm prompting is carried out only on the monitoring background_TThe specific values of (a) are different.

2. The fire video detection and early warning method based on image multi-feature fusion as claimed in claim 1, wherein: in step S1, a median filtering method is used to perform denoising preprocessing on the obtained video image sequence.

3. The fire video detection and early warning method based on image multi-feature fusion as claimed in claim 2, characterized in that: in step S3, a color model is established for the flame and smoke, and a suspected color area is extracted from the moving area, and the specific steps include:

step S31, designing a flame color model rule based on RGB, HSV and YCbCr color spaces, and performing phase comparison on the rule to obtain a color suspected region of the flame, wherein the specific rule comprises the following steps:

rule 1: r (x, y)>R_T

Rule 2: r (x, y) ≧ G (x, y) > B (x, y)

Rule 3: s (x, y) ≥ 255-R (x, y)). S_T/R_T

Rule 4: h₁<H(x,y)<H₂,S₁<S(x,y)<S₂,V₁<V(x,y)<V₂

Rule 6: y (x, Y)>Y_mean,Cb(x,y)<Cb_mean,Cr(x,y)>Cr_mean

Rule 7: cr (x, y) -Cb (x, y) ≥ lambda

In the formula, R (x, y), G (x, y) and B (x, y) respectively represent red, green and blue components of the pixel point (x, y) in RGB space, and R (x, y) represents red, green and blue components in RGB space_TH (x, y), S (x, y) and V (x, y) are three components of hue, saturation and brightness of the pixel point (x, y) in HSV color space respectively, and S is a threshold value of a red component_TRed component equal to R_TCorresponding to the value of saturation in HSV space, H₁、H₂、S₁、S₂、V₁、V₂The color value is a threshold value preset for hue, saturation and brightness, Y (x, Y), Cb (x, Y) and Cr (x, Y) are respectively the brightness, red component and blue component of a pixel point (x, Y) in a YCbCr space, lambda is a non-negative threshold value of the difference value of the two components Cr (x, Y) and Cb (x, Y), and Y is a non-negative threshold value of the difference value of the two components Cr (x, Y) and Cb (x, Y)_mean、Cb_mean、Cr_meanRespectively taking the average values of all pixels in the current image on three components of Y, Cb and Cr;

step S32, designing a smoke color model rule based on RGB and HSI space, and performing phase-joining on the rule to obtain a color suspected area, wherein the rule specifically comprises the following steps:

rule 1: delta is less than or equal to | R (x, y) -G (x, y) |₁

Rule 2: delta is less than or equal to | R (x, y) -B (x, y) |₂

Rule 3: delta is less than or equal to | G (x, y) -B (x, y) |₃

Rule 4: l is₁≤I(x,y)≤L₂||D₁≤I(x,y)≤D₂

4. The fire video detection and early warning method based on image multi-feature fusion as claimed in claim 3, characterized in that: in the step S4, a morphological operation of expanding and then corroding is performed on the suspected color area, and the small holes and the small seams are filled and leveled, so as to obtain a smooth and continuous suspected flame area and a smooth and continuous suspected smoke area.

5. The fire video detection and early warning method based on image multi-feature fusion as claimed in claim 4, wherein: step S6 is to input the extracted flame features as SVM classifier to determine whether there is flame, and to analyze the extracted smoke features and input the analysis result to the logic operator to determine whether there is smoke, and the specific steps include:

step S61, firstly, determining the optimal parameters in the SVM, acquiring training samples to train the SVM model, then taking the characteristics of flame extraction as the input of the SVM, and judging whether flame exists in the video;

step S62, extracting each characteristic of the suspected area of the smoke, comparing the calculated characteristic with a threshold value, inputting the obtained true and false results into a logic arithmetic unit so as to judge whether the image contains the smoke, wherein the logic arithmetic unit calculates the judgment results of the three characteristics of the smoke by adopting a mode of combining AND, OR, NOT and XOR logic arithmetic, and adjusts the sensitivity of the detection algorithm by adopting different logic arithmetic combination modes according to different detection scenes so as to increase the generalization capability of the algorithm;

and step S63, when the SVM judges that the flame is contained or the output of the logic arithmetic unit is true, the monitored area is judged to contain the fire.