NL2030149B1

NL2030149B1 - A method and system for closed-loop automatic control for whole-process intelligent monitoring of fire activity operations

Info

Publication number: NL2030149B1
Application number: NL2030149A
Authority: NL
Inventors: Yang Jun; Gao Hao; Liu Deyang; Gao Xun; Feng Guohai
Original assignee: Nanjing Inst Tech; Jiangsu D C S Automation Eng Co Ltd
Priority date: 2021-09-15
Filing date: 2021-12-16
Publication date: 2023-03-29
Also published as: CN113792658B; CN113792658A; NL2030149A

Abstract

The invention discloses a method and system for closed-loop automatic control and management for whole-process intelligent monitoring of fire activity operations, 5 establishing a safety loss function for fire-activated operations and achieving a measure of the safety of fire-activated operations, mainly involving the three elements of fire-activated operations as defined by the fire-activated operation ticket: ® fire-activated operation area, ® fire-activated operation type, © fire-activated operation spatial distance from the hazard source. On this basis, a set of machine 10 vision intelligent detection and recognition technology is used to detect and classify the fire operation area, the fire operation behavior and the neighboring hazard sources. The detection and classification results are further fed back to the safety loss function of the fire operation, and according to the regularized closed-loop control rules of the fire operation, the command of fire operation termination or 15 continuation is issued to realize the closed-loop automatic control of the fire operation.

Description

1 001874P-NL

A method and system for closed-loop automatic control for whole-process intelligent monitoring of fire activity operations

Technical field

The present invention relates to a method and system of closed-loop automatic control for whole-process intelligent monitoring of fire activity operations, which belongs to the technical field of control of fire-activated operation.

Background Technology

The materials, intermediate parts and final products in the production process of chemical enterprises are flammable and explosive, toxic and harmful, and the safety requirements for production processes and equipment are extremely demanding.

With the increasing scale of chemical industry production, production equipment has been operating continuously for a long time in high temperature, high pressure and strong corrosive environment, which inevitably causes equipment aging and structural defects. The urgent need for all kinds of regular maintenance of chemical industry equipment, inevitably produce fire operations. In the chemical production site of the fire work is a high-risk operation, the implementation of strict control is the key to prevent poisoning, fire, explosion and other serious accidents.

At present, the control of fire activity is mainly carried out through the issuance of the fire activity chapter for preventive operation review, and the management and control of fire activity by fire supervisors. This prevention strategy requires a lot of manpower and resources, and has a severe test on the operation and monitoring personnel's business level, in addition to personnel negligence and fatigue can also cause fire activity safety hazards.

With the development of artificial intelligence and machine vision technology, automated intelligent imaging monitoring has become an effective technical means to replace all kinds of manual observation work, has seen the target detection, classification and behavior recognition of the field of application, with high monitoring efficiency, timeliness, accuracy and other technical advantages, and has the ability to work for a long time without interruption. However, in the field of fire fighting

2 001874P-NL operation control, artificial intelligence technology and machine vision technology face application fault, there is no technical means for fire fighting operation automation control. The main problems are: (i) lack of regularized description of fire activity safety measurement, (ii) lack of regularized description of fire activity specification, and (iii) lack of area detection, behavior recognition and target detection technology for fire acticity.

Content of the Invention

Objective of the invention: To address the problems and shortcomings of the prior art, the invention provides a method and system for whole-process intelligent monitoring of fire activity operations, which identifies the location, behavior and hazard source target of fire activity through artificial intelligence technology, and measures the safety of fire activity through the safety loss function of fire activity at time t to achieve closed-loop automatic control of fire activity.

Technical solution: A closed-loop automatic control method for the whole process of intelligent monitoring of fire fighting operations, combining fire fighting operation tickets, machine vision technology and deep learning technology based on convolutional neural network establishes a safety loss function for fire fighting operations at time t. Machine vision technology is used to detect the area where the fire is generated, a deep learning model for behavior recognition is used to determine the type of fire fighting operation behavior at time t, and a deep learning model for target detection is used to detect and classify hazard sources. The model detects and classifies the hazard sources, and finally calculates the safety measure of the fire operation at time t. According to the established closed-loop control rules for fire operation based on the fire operation safety loss function, the command to terminate or continue the fire operation is sent, so as to complete the closed-loop automatic control of the fire operation.

The said safety loss function of the fire fighting operation at time t is used to measure the safety of the fire fighting operation; the safety loss function of the fire fighting operation at time t is:

3 001874P-NL

T 4 2

E(1)=] a, (I) - 2) +a, (ht) -n) + Sv x, ©) dt nt i=l where I(t) is the location of the fire operation at time t, » is the location of the fire operation area marked by the fire operation ticket, h(t) is the type of fire operation at time t, 77 is the type of fire operation marked by the fire operation ticket, wi is the weight of the hazard source, i is the level of the hazard source, si(t) is the distance of the fire operation area within the monitoring range from the i-level hazard source at time t, and 7,,7, are the starting and ending moments of the fire operation marked by the fire operation ticket, «,,«, are the loss weights. The safety of the fire operation increases with the decreasing value of the fire operation safety loss function.

The said i is the level of hazard sources, divided into 4 levels of hazard sources, the weight of the hazard sources are: w,=1 w,=0.7, w,=0.6, w,=0.5 .

Using optical features-based seed point selection and random wandering method of moving fires to detect the location of the occurrence of moving fires at time t:

Optical features 1: The color channel variance features of RGB color images of the fire-activated area within the monitoring range are calculated as:

Co 2

I(x)+1 (x)+1 (x

Vix) = I (x)= OHO HL) J+) 3 . 2

L(x) +1 (x)+1 (x

V(x — I, (x) _ ( ) o( ) »( ) & & 3 2

CLF (x)+1,(x))

Vlad = J (2) — trem remetiooni 3 where V(x), Vg(x), and V(x) are the variances of the red, green, and blue channels in the optical information at the pixel point, I(x) is the red channel light intensity at point x, Ig(x) is the green channel light intensity at point x, and p(x) is the blue channel light intensity at point x.

4 001874P-NL

Optical feature 2: The global contrast feature of the red channel is calculated as:

C (x)= > 17,C0-1,(y)

VL (vel, where C(x) is the global contrast characteristic of the red channel, I(x) is the red channel light intensity at point x, I:{y) is the red channel light intensity at any point in the environment, and I; is the red channel component of the ambient optical information.

Principles of fire-activated seed point selection: 3(x)= (1 (x)= ¥, (3) + (7, (3) =, (x) +C, (2)

The first k maximum I(x) values are taken and their corresponding points form a seed point set &.

According to the seed point set £, a random walk model is used to obtain the complete fire area, establish the minimum outside rectangle of the fire area, and mark the center of the rectangle as the fire location I(t).

Combined with the fire location I(t), the behavior recognition deep learning model is used to determine the type of fire operation behavior at time t.

The imaging area with radius ¢ at the center of the fire location I(t) is designed as the area s(t) of the fire behavior, where 5=0.5N and N is the image length.

By deploying video sensors in the hazardous chemical plant area, capturing and storing images of different scenes, different plant areas, different lighting conditions and different types of fire operations, manually labeling the types of fire operations in the images, determining the label of each image, constructing a typical fire operation image database by combining the images and the corresponding image labels, inputting the behavior recognition deep learning model for training, and using the

001874P-NL trained behavior recognition deep learning model to realize the classification of the types of fire operations in the fire operation behavior occurrence area at time t.

Through the four types of hazardous sources on-site image acquisition and online 5 public image crawling, the four types of hazardous sources images in different scenes, different plants and different lighting conditions are captured and stored, the types of the four types of hazardous sources in the images are manually labeled, the label of each image is determined, the image database of the four types of hazardous sources is constructed by combining the images and the corresponding image labels, and the target detection and recognition deep learning model is input for training. The trained deep learning model for target detection and recognition is used to detect and classify the major hazards within the imaging field of view at time t, classify the regions of four different classes of hazards, and mark the center of the smallest outer rectangle of the region of four different classes of hazards as the location of the hazards dit) i=1,2,3,4.

If there is a class i hazardous source in the imaging field of view: s, (=r) —d, (1) where || || is the Euclidean distance metric function for two locations.

If there is no class i hazard source in the imaging field of view, the default value: s,(1)=10°".

According to the closed-loop control rules of the fire operation based on the safety loss function of the fire operation, the fire operation is controlled at time t:

When E{1}>e, immediately terminate the fire operation at t+1 moments.

Otherwise, the fire activity continue to occur, calculate E(t+1).

So far, the closed-loop control of the fire activity at time t is completed.

6 001874P-NL

A system for realizing the above-mentioned closed-loop automatic control method for whole-process intelligent monitoring of fire activity operations, including a fire activity safety loss function calculation module, a fire area detection module, a fire activity closed-loop control module, a module for determining the type of fire activity behavior and a module for marking the location of the hazard source.

The said fire activity safety loss function calculation module combines fire activity ticket, machine vision technology and deep learning technology based on convolutional neural network to establish a fire activity safety loss function at time t.

Said moving fire area detection module uses machine vision technology to detect the area where the moving fire is generated.

The said judging the kind of kinetic fire operation behavior module uses behavior recognition deep learning model to determine the kind of kinetic fire operation behavior at t moments.

The location marking module of the said hazard source uses the target detection deep learning model to detect and classify the hazard source, and finally calculates the safety metric of the fire operation at the moment t.

Said fire activity closed-loop control module according to the established fire activity safety loss function based on the fire activity closed-loop control rules, send fire activity to terminate or continue the command, so as to complete the fire activity closed-loop automatic control.

A computer device, the computer device includes a memory, a processor and a computer program stored in the memory and runable on the processor, the processor executes the above computer program to achieve the full process intelligent monitoring of the fire activity closed-loop automatic control method as described above.

A computer-readable storage medium, the computer-readable storage medium stores a computer program executing the method of closed-loop automatic control of

7 001874P-NL the whole process intelligent monitoring of the fire moving operation as described above.

Beneficial effect: Compared with the prior art, the present invention provides a closed-loop automatic control method and system for the whole process of intelligent monitoring of fire moving operations, which can automatically measure the safety of fire moving operations and automatically monitor and control the whole process of fire moving operations, realizing intelligent monitoring and automatic control of the fire moving operations process.

Description of the attached drawings

Figure 1 is a flow chart of the method of the embodiment of the present invention.

Specific implementation

The present invention is further elucidated hereinafter in conjunction with specific embodiments, it should be understood that these embodiments are used only to illustrate the present invention and not to limit the scope of the present invention, and after reading the present invention, various equivalent forms of modification of the present invention by a person skilled in the art fall within the scope defined by the claims appended to the present application.

As shown in FIG. 1, a closed-loop automatic control method for the whole process intelligent monitoring of fire-activation operations, the core of the method is the safety loss function of fire-activation operations at time t for measuring the safety of fire-activation operations:

E(1)=[" a (10) -) +a, (h(n) —n)’ ih W, XS, © | dt 1 i=1

Where, I(t) is the location of the fire operation at time t, > is the location of the fire operation area marked by the fire operation ticket, h(t) is the type of fire operation at time t, 77 is the type of fire operation marked by the fire operation ticket, wi is the weight of the hazard source, i is the level of the hazard source, which is divided into 4 levels of hazard sources w,=1, w,=0.7, w,=0.6, w,=0.5 , si(t) is the distance of the fire operation area within the monitoring range at time t The distance of the ith level

8 001874P-NL of hazardous sources, 7,,7, for the fire activity ticket marked by the start and end of the fire activity moment, «,,«, for the loss weight. The safety of fire operation increases with decreasing value of fire operation safety loss function.

The specific fire activity control process is as follows:

Step 1: Detect the location of the fire at time t. Using the optical feature-based seed point selection and random walk method:

Optical features 1: channel variance features are calculated as;

Fim | 1-20 2) 10 = Iz co LL] oo LOL HLO 10 = where V(x), V(x), and Vr(x) are the variance of the red, green, and blue channels in the optical information at the point, I:{x) is the red channel light intensity at the point x, lg(x) is the green channel light intensity at the point x, and lp(x) is the blue channel light intensity at the point x.

Optical feature 2: The global contrast feature of the red channel is calculated as

C.x)= > LoL) vi. (vel, where C:(x) is the global contrast feature of the red channel, I(x) is the red channel light intensity at the point, ly) is the red channel light intensity at any point in the environment, and I; is the red channel component of the environmental optical information;

Seed point selection principle: 3(x)=(7, (x)= (1) + (7, (x)= V(x) + C. (x)

The first k maximum I(x) values are taken and their corresponding points form the seed point set &.

The complete moving fire region is obtained using the random walk model according to the seed point set £. The target region extraction based on the random walk

9 001874P-NL method can be found in the literature [Ramadan H, Tairi H. Pattern mining-based video saliency detection: application to moving object segmentation. Computers &

Electrical Engineering, 2018, 70: 567-579.], which will not be repeated here.

The minimum outside rectangle of this moving fire region is established, and the center point of this rectangle is marked as the moving fire occurrence location I(t).

Step 2: Combining the fire location I(t), the behavior recognition deep learning model is used to determine the type of fire operation at time t.

The imaging area with radius o at the center of the fire location I(t) is designed as the area s(t) of the fire operation behavior, where 5=0.5N, N is the image length.

By deploying video sensors in the hazardous chemical plant area, capturing and storing video images of different scenes, different plant areas, different lighting conditions and different types of fire fighting operations, manually annotating the video images, constructing a typical fire fighting operation image database, inputting the behavior recognition deep learning model for training, and using the trained behavior recognition deep learning model to realize the classification of the types of fire fighting operations in the fire fighting operation behavior occurrence area at time t The trained behavior recognition deep learning model is used to classify the types of fire fighting operation behavior in the region at time t. The model training and classification process of behavior recognition based on deep learning model

References [Wang Zhongmin, Cao Hongjiang, Fan Lin. A deep learning method for human behavior recognition based on convolutional neural network. Computer

Science, 2016, 43(s2). 56-58.], which is not repeated here.

Step 3: Through four types of hazardous sources on-site image acquisition and online public image crawling, capture and store four types of hazardous sources images under different scenes, different plants, and different lighting conditions, manually annotate the images, construct an image database of the four types of hazardous sources, , input the target detection and recognition deep learning model for training, and use the trained target detection and recognition deep learning model to realize the detection of dynamic fire at moment t The trained target detection and

10 001874P-NL recognition deep learning model is used to detect and classify the major hazards in the imaging field of view where the fire behavior occurs at moment t. The regions of four different classes of hazards are classified, and the center of the smallest outer rectangle of the four different classes of hazards is marked as the location of the hazards i=1,2,3,4. The model training and classification process based on deep learning target detection and classification is referenced in [Lu, K., Chen, J., Little, J.

J., & He, H. (2018). Lightweight convolutional neural networks for player detection and classification. Computer Vision and Image Understanding, 172, 77-87.], which is not repeated here.

If there is a class i hazard source in the imaging field of view : s,(0=|l(0)—d, (1)

Where, || || is the Euclidean distance metric function for two locations, dit) i=1,2,3,4 is the center of the smallest outer rectangle of four different classes of hazardous source areas for the location of the hazardous source.

If there is no class i hazard source in the imaging field of view, the default value: 5,(£)=10".

Step 4: According to the closed-loop control rule of fire activity based on the safety loss function of fire activity, the fire activity control is carried out at time t.

When E(t)>s, immediately terminate the operation of the fire at the moment t + 1.

Otherwise, the fire activity continue to occur, and E(t+1) is calculated.

So far, the closed-loop control of the fire activity at time t is completed.

The system for realizing the above-mentioned closed-loop automatic control method of fire activity with intelligent monitoring of the whole process includes a fire activity safety loss function calculation module, a fire area detection module, a fire activity closed-loop control module, a module for determining the type of fire activity behavior and a module for marking the location of the hazard source.

11 001874P-NL

The fire operation safety loss function calculation module combines fire operation tickets, machine vision technology and deep learning technology based on convolutional neural network to establish the fire operation safety loss function at time t. The fire operation safety loss function at time t is as described above.

The fire area detection module uses machine vision technology to detect the area where the fire is generated. The specific implementation process is as described above in step one.

Judgment of the kind of fire fighting operation module using behavior recognition deep learning model to determine the kind of fire fighting operation at time t. The specific implementation process is as described in step 2 above.

The location marking module of the hazard source uses the deep learning model of target detection to detect and classify the hazard source, and finally calculates the safety metric of the fire operation at time t. The specific implementation process is described in step 3 above.

Firework closed-loop control module according to the established firework safety loss function based on the firework closed-loop control rules, send firework termination or continuation of instructions, so as to complete the firework closed-loop automatic control. The specific implementation process is as described in step 4 above.

Obviously, it should be understood by those skilled in the art that the above- mentioned steps of the method of closed-loop automatic control of fire fighting operations with intelligent monitoring of the whole process or the modules of the closed-loop automatic control system of fire fighting operations with intelligent monitoring of the whole process can be implemented with common computing devices, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, optionally, they can be implemented in program code executable by the computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases the steps shown or described can be executed in a different order than herein, or they can be implemented separately as individual integrated circuit modules, or

12 001874P-NL multiple modules or steps of them can be implemented as a single integrated circuit module.

In this way, embodiments of the present invention are not limited to any particular combination of hardware and software.

Claims

13 001874P-NL Patent Claims

1. A closed-loop automatic control method for whole-process intelligent control of fire activity operations, characterized by combining firefighting operation tickets, machine vision technology and convolutional neural network deep learning technology to establish a safety loss function for firefighting operations at time t. The machine vision technology is used to detect the area where the fire originates, a deep learning behavioral recognition model is used to determine the type of behavior of the firefighting operation at time t, a deep learning target detection model is used to The model detects and classifies the hazard sources, and finally calculates the safety measure of the fire operation at time t. According to the established closed-loop control rules for the fire operation based on the fire operation safety loss function, the command to terminate or continue the fire operation is sent, which completes the closed-loop automatic control of the fire operation.

2. The method of automatic closed-loop control for intelligent monitoring of the whole process of fire operations according to theorem 1, characterized in that said safety loss function of fire-fighting operations at time t is used to measure the safety of fire-fighting operations; the safety loss function of firefighting operations at time tis: 4 -2 E(n)=[" a, (In - yY +a, (A) -n) [3 Xs, ©) di ! i=l where I(t) is the place where the fire action takes place at time t, » the location of the area where the fire action takes place as indicated on the fire action ticket, h(t) the type of behavior of the fire action at time t, 77 the type of behavior of the fire action as indicated on the fire action ticket, w‚ the weight of the hazard source, i the level of the hazard source, si(t) the distance from the area where the fire action takes place within the control range to the level i hazard source at time t, 7,,7, the start and end times of the fire action as indicated on the fire action ticket, «,,2, the loss weights; the safety of the fire action

14 001874P-EN increases with decreasing value of the fire action safety loss function.

3. The method of automatic closed-loop control for intelligent monitoring of the whole process of fire activities according to theorem 2, characterized in that said i is the level of the hazard sources, which are divided into 4 levels of hazard sources, and the weights of the hazard sources are: w=, w,=0.7, w;=0.6, w‚70.5.

4. The method of automatic closed-loop control for intelligent monitoring of the whole process of fire activity according to theorem 1, characterized in that a method of dynamic fire source selection and random drift based on optical characteristics is used to determine the place of occurrence of a dynamic detect fire at time t: Optical characteristic 1: The channel variance characteristic is calculated as: à à L(O)+L (0) +1, (0) 3 ‚ ‚ 2 LG) +1 (x) +1, (x V (x)= (x)— tian] * * 3 2 à | L(x)+L(x)+1,(x) V(x) = [an- WHOLE 3 Where V(x), Vg(x), V(X) are the variance of the red, green and blue channel in the optical information at the point, I{x) is the intensity of the light from the red channel at point x, Ig{x) is the intensity of the light from the green channel at point x and h(x) is the intensity of the light from the blue channel at point x; Optical Attribute 2: The global contrast characteristic of the red channel is calculated as:

C.(x)= > D= VI (el, Where C{x) is the global contrast characteristic of the red channel, I(x) is the light intensity of the red channel at point x, I:(y) is the light intensity of the red channel

15 001874P-NL at any point in the vicinity, and |; the red channel component of the environmental optical information; Principles of fire source selection: I(x)=(1 (x) 21, (1) + (F(x) =, (0) + C(x) Take the first k maximum I(x) values, and the points corresponding to it form a starting point set & Based on the set of starting points & a random walk model is used to obtain the entire moving fire area, a minimum outer rectangle of the moving fire area is determined, and the center of the rectangle is marked as the position I(t) of the moving firing area.

5. The method of automatic closed-loop control for intelligent monitoring of the entire process of fire operations according to theorem 1, characterized by the fact that, in combination with the position of the fire displacement operation I(t), a deep learning behavioral recognition model is used to determine the type of behavior of the fire move operation at time t; The image area of radius o in the center of the fire location I(t) is designed as the area s(t) of the fire operation; A database of typical firefighting images is constructed and fed into the deep learning behavioral recognition model for training. The trained deep learning behavior recognition model is used to classify the types of firefighting behavior in the area where the firefighting behavior occurs at time t.

6. The method of automatic closed-loop control for intelligent monitoring of the entire process of fire activities according to theorem 1, characterized by building an image database of four kinds of dangerous sources, introducing a deep learning target detection and recognition model for training, and the trained deep learning target detection and recognition model is used to detect and classify the significant hazardous sources in the field of view where the firefighting behavior occurs at time t. The area of four different classes of hazardous sources is classified, and the center of the smallest outer rectangle of the area of four different classes of

16 001874P-EN hazardous sources is marked as the position of the hazardous source dit) i=1,2,3,4 If a class i hazard source exists within the imaging field: D= —d where || | the metric Euclidean distance function is for both locations: If there is no risk source of type i in the imaging field, si(t) is set to the default value.

7. The method of automatic closed-loop control for intelligent monitoring of the entire process of fire activities according to theorem 2, provided that the fireworks control at time t is performed according to closed-loop control rules for fireworks based on the safety loss function for fireworks: When E (r)>¢, the firing operation at time t+1 is terminated immediately. Otherwise, ignition continues and E(t+1) is calculated. This completes the closed-loop control of the fireworks at time t.

8. A closed-loop automatic control system for the whole process of intelligent fire activity monitoring, characterized in that it consists of a firing safety loss function calculation module, a firing area detection module, a closed-loop control module of firing, a module for determining the type of firing behavior and a module for marking the location of the danger source; the above fire operation safety loss calculation module combines fire operation maps, machine technology and deep learning technology based on convolutional neural networks to construct a fire operation safety loss function at time t; a moving fire area detection module that uses automatic vision techniques to detect the moving fire area; the moving fire action type determination module uses a deep learning behavioral recognition model to determine the moving fire action type at time t;

17 001874P-EN the hazard source location marking module uses a deep learning target detection model to detect and classify the hazard source, and finally calculates the security measure of the firing operation at time t; and the firework closed loop control module sends a command to terminate or continue the firework operation according to the determined firework company closed loop control rules, based on the safety loss function of the firework company, thereby completing the automatic closed loop control of the firework operation.

9. A computing device, characterized in that the computing device comprises a memory, a processor and a computer program stored in memory and executable on the processor, the processor executing said computer program to implement the method of closed-loop automatic control for intelligent monitoring of the whole process of fire activities is asserted in any of requirements 1-7.

10. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program that implements the method of closed-loop automatic control for intelligent monitoring of the entire process of burning activities in any of the statements 1-/.