CN117975376B - Mine operation safety detection method based on depth grading fusion residual error network - Google Patents
Mine operation safety detection method based on depth grading fusion residual error network Download PDFInfo
- Publication number
- CN117975376B CN117975376B CN202410391684.6A CN202410391684A CN117975376B CN 117975376 B CN117975376 B CN 117975376B CN 202410391684 A CN202410391684 A CN 202410391684A CN 117975376 B CN117975376 B CN 117975376B
- Authority
- CN
- China
- Prior art keywords
- convolution
- layer
- fusion
- output
- size
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 101
- 238000001514 detection method Methods 0.000 title claims abstract description 18
- 238000012549 training Methods 0.000 claims abstract description 46
- 238000000034 method Methods 0.000 claims abstract description 20
- 238000012544 monitoring process Methods 0.000 claims abstract description 14
- 238000005065 mining Methods 0.000 claims abstract description 6
- 238000012423 maintenance Methods 0.000 claims abstract description 5
- 230000008569 process Effects 0.000 claims abstract description 5
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 239000010410 layer Substances 0.000 claims description 182
- 230000006399 behavior Effects 0.000 claims description 27
- 238000012360 testing method Methods 0.000 claims description 19
- 238000010606 normalization Methods 0.000 claims description 18
- 238000011176 pooling Methods 0.000 claims description 15
- 230000004913 activation Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000001914 filtration Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 claims description 6
- 238000010586 diagram Methods 0.000 claims description 4
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 230000008859 change Effects 0.000 claims description 3
- 238000005520 cutting process Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 239000002356 single layer Substances 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 239000013598 vector Substances 0.000 claims description 3
- 230000008901 benefit Effects 0.000 abstract description 3
- 238000004519 manufacturing process Methods 0.000 abstract description 3
- 238000013461 design Methods 0.000 abstract description 2
- 238000005457 optimization Methods 0.000 abstract description 2
- 238000002372 labelling Methods 0.000 abstract 1
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 4
- 230000006872 improvement Effects 0.000 description 3
- 238000007689 inspection Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000428 dust Substances 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000391 smoking effect Effects 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a mine operation safety detection method based on a depth grading fusion residual error network; s1, mining operation data acquisition; s2, preprocessing mine operation data; s3, constructing a depth grading fusion residual error network, S4, inputting divided labeling data for training and storing a model in a depth grading fusion residual error network training process and parameter design; s5, model identification early warning unsafe behavior, and loading model identification new monitoring data; s6, model optimization updating and equipment maintenance. According to the method, the problem of difficulty in extracting the characteristics of the mine tunnel image is solved by introducing the grading fusion residual error network, the multi-level trend characteristics can be extracted, and the information of each characteristic layer is more comprehensively represented, so that the comprehensive performance of the network is improved; has remarkable help to improve the safety, production efficiency and economic benefit of mine operation.
Description
Technical Field
The invention belongs to the technical field of mining safety, and particularly relates to a mine operation safety detection method based on a depth grading fusion residual error network.
Background
Mine operation is an industry with high risk and high risk, and safety production is related to personal safety and enterprise interests. Mine safety has been a major concern worldwide over the last decades. Although advances in technology and stringent safety regulations have significantly reduced the rate of casualties for miners, there are still many potential hazards for mining operations. Miners face various potential safety risks during operation. For the operation safety of mine workers, the traditional monitoring mode mainly depends on manual inspection or monitoring cameras. However, such methods all have certain limitations; the manual inspection has the problems of low inspection efficiency, poor real-time performance, easy fatigue and the like; and the traditional monitoring cameras can only provide passive monitoring, can only record accident occurrence, and cannot predict or prevent accidents. Based on this, it is urgent to develop a method capable of effectively detecting unsafe behavior in the operation of mine workers in real time and improving the safety of the mine operation.
Disclosure of Invention
The embodiment of the invention aims to provide a mine operation safety detection method based on a depth grading fusion residual error network, so as to realize real-time effective automatic detection and early warning of unsafe behaviors of mine operation, improve the recognition capability of complex scenes and behaviors, and improve the safety level of mine operation.
In order to solve the technical problems, the technical scheme adopted by the invention is that the mine operation safety detection method based on the depth grading fusion residual error network is as follows; the method comprises the following steps:
S1, mining operation data acquisition; installing cameras at different key positions in the mine well to acquire real-time video data of the operation scene of the mine workers;
s2, preprocessing the acquired real-time video data of the mine operation, and dividing a data set;
S3, constructing a depth grading fusion residual error network;
S4, training the network model established in the S3 by utilizing the data set divided in the S2;
s5, inputting the real-time video data acquired newly into a network model obtained through training in S4 to identify and early warn unsafe behaviors;
And S6, optimizing and updating the network model and maintaining equipment.
Further, in the step S1, the image acquisition function is started for different cameras with cross coverage areas.
Further, the step S2 specifically includes the following steps:
S21, decomposing different behavior video data of workers acquired by the camera in the S1 into video frame data, and extracting images from continuous video frames of the same behavior at a speed of 5 frames per second;
S22, cutting the extracted behavior frame image to a uniform size 224 multiplied by 224;
s23, performing Gaussian filtering processing on the clipped image:
Wherein/> Is the coordinates of the pixel,/>Representing the distance of the coordinate to the center of the convolution kernel,/>Is a cut-off frequency, and takes the value of 50/>Representing the Gaussian filtering of the image;
S24, carrying out data enhancement on the filtered image;
s25, merging the data enhanced image and the original image into a sample library, and carrying out normalization operation on all the images:
Wherein/> Is normalized pixel value,/>And/>Maximum and minimum, respectively,/>, of image pixelsPixel values representing an image;
s26, normalizing the image according to 4:1 is randomly divided into a training database and a test database.
Further, in S2, the data enhancement operation includes one or more of translation, flipping, rotation, occlusion, scaling, and transformation.
Further, the step S3 specifically includes the following steps:
S31, constructing a first residual convolution block: the convolution block consists of a single-layer convolution layer and a pooling layer, wherein the convolution kernel is 7×7 in size, the step length is 2, the number is 64, and the size of the output characteristic diagram is 112×112; the window of the pooling layer is 3×3, the step length is 2, and the size of the output feature map is 56×56;
S32, constructing a second residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 64; the convolution kernel of the second convolution layer has the size of 3×3, the step length of 2, the number of 64, and the size of the output feature map is halved; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 256; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 64; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 64; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 256; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S33, constructing a third residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 128; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 128, and an output feature pattern size of 28×28; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 512; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S34, constructing a fourth residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 256; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 256, and an output feature pattern size of 14×14; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 1024; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S35, constructing a fifth residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 512; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 512, and an output feature map size of 7×7; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 2048; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 512; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 2048; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S36, constructing a classification prediction layer: the classification prediction layer comprises a global pooling layer, a full connection layer and a softmax layer; the global pooling layer sets pooling windows according to the size of the feature map, outputs vectors with the same number as the feature map channels, the number of nodes of the full-connection layer is taken as the number of categories of the training sample library, and the softmax layer carries out exponential normalization on the output of the full-connection layer, wherein the calculation formula is as follows:
Wherein/> For/>Exponential function of/>For the number of nodes,/>Input firstPersonal node,/>An output representing a full connection layer;
S37, serially connecting 1 first residual convolution block, 2 second residual convolution blocks, 4 third residual convolution blocks, 23 fourth residual convolution blocks, 3 fifth residual convolution blocks and a classification prediction layer to form a depth grading fusion residual network of 101 layers; and for the second convolution layer convolution kernel step sizes of the non-first second residual convolution block, the third residual convolution block, the fourth residual convolution block and the fifth residual convolution block, the corresponding step sizes of the second-stage fusion branch convolution layer convolution kernels are 1, and the node number of the output layer is 9.
Further, the step S4 specifically includes the following steps:
S41, after the sample libraries divided in the S26 are disordered, taking 64 pictures as a batch, and sequentially inputting depth grading fusion residual network training established in the S3, wherein the number and the size of training samples and test samples in the same batch are the same;
S42, after the image is input into a depth grading fusion residual error network, obtaining prediction output through forward propagation, and calculating cross entropy loss CEL by using the label and the prediction output; for the calculated CEL, updating the weights of the convolutional layers of the network back with an Adam optimizer at a learning rate of 0.001; the cross entropy loss CEL is calculated as follows:
Wherein/> For the number of samples of a batch, here take 64,/>For category number, 9 is taken here,/>For tag value,/>Is a predicted value of the network; /(I)Represents the/>Batch sample,/>Represents the/>A category;
S43, after updating of all weight parameters of the depth grading fusion residual error network is completed, predicting and outputting the input test set data of the same batch, and comparing the test set data with corresponding label data to calculate accuracy and loss index to assist in judging whether the training ending condition is mature;
s44, continuously inputting training data and test data of the next batch into the depth grading fusion residual error network, recording all behavior images in the sample library as one cycle after training, and ending training until the change of the reduction amplitude of the loss index tends to be stable;
s45, storing the trained depth grading fusion residual error network model for subsequent identification.
Further, the step S5 specifically includes the following steps:
S51, loading the depth grading fusion residual error network model trained in the S4;
S52, if S1 and S2 are adopted, mine operation data are collected again and preprocessed, and are input into a depth grading fusion residual error network model, so that a prediction result is obtained;
And S53, when the predicted unsafe behavior probability exceeds a preset threshold value, early warning information is sent.
Further, the process of dividing the data set is not performed in S52.
Further, the step S6 specifically includes the following steps:
S61, when the newly acquired data quantity reaches the original data level, optimizing and updating the depth grading fusion residual error network model, and updating sample training data, training and adjusting model parameters according to the new data in the steps S21-S26;
and S62, maintaining and managing the system every 6 months, including the maintenance of the monitoring camera, the management of data storage and backup, and the test of the early warning system.
The beneficial effects of the invention are as follows: according to the invention, the real-time video data is processed through the depth grading fusion residual error network (DHFRN), so that unsafe behaviors of mine workers in the operation process can be effectively identified, and real-time monitoring and early warning of mine operation safety are realized. Aiming at complex conditions and terrains of mine operation sites; the invention collects a large number of samples of behaviors in complex scenes through multiple angles for training; and the model is continuously optimized and updated after each training, so that the model has strong advantages in recognition of complex scenes and behaviors. The mine operation safety detection method can obviously improve the mine operation safety level; further improving the production efficiency and economic benefit of mine enterprises.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions in the prior art, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a flow chart of a mining operation safety detection method based on a depth grading fusion residual network;
FIG. 2 is a schematic diagram of a hierarchical fusion residual module structure of the present invention;
FIG. 3 is a schematic diagram of the overall network architecture of the present invention;
FIG. 4 is a graph of training results training round number versus accuracy for the method of the present invention;
FIG. 5 is a training results training round number-CEL loss graph for the method of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The invention provides a mine operation safety detection method based on a depth grading fusion residual error network; as in fig. 1, the method comprises the following steps:
S1, acquiring mine operation data, installing a high-definition monitoring camera at a key position of a mine, and covering the whole surface and monitoring the behaviors of workers;
s11, acquiring real-time video data of operation scenes of mine workers by installing cameras at different key positions underground the mine. And when a worker enters a monitoring scene, starting a video acquisition image. This step is the key to obtain the raw data, and needs to ensure that the position, angle, etc. of the camera can comprehensively and clearly capture the operation behaviors of mine workers.
S12, simultaneously starting an image acquisition function for different cameras with cross coverage areas. In the embodiment, 9 kinds of illegal behaviors of workers are collected in a mine tunnel, namely double operation is not carried out according to requirements, safety belts are not fastened according to requirements, a dust mask is not worn according to requirements, heads or bodies are pulled out of a vehicle and enter a dangerous area during operation, an air door is in an open state for a long time, illegal smoking is carried out, a self-rescuer is not carried according to regulations, and a safety helmet is not worn according to regulations.
S2, preprocessing the mine operation data acquired in the step S1
S21, decomposing different behavior video data of workers acquired by the monitoring video camera into video frame data, and extracting images from continuous video frames of the same behavior at a speed of 5 frames per second.
S22, cutting the behavior frame image extracted from the video to 224×224 with uniform size.
S23, carrying out Gaussian filtering processing on the cut image, wherein the calculation formula is as follows,
Wherein/>Is the coordinates of the pixel,/>Representing the distance of the coordinate to the center of the convolution kernel,/>Is a cut-off frequency, and takes the value of 50/>Representing a gaussian filtering of the image.
And S24, carrying out data enhancement on the filtered image. The data enhancement operations include translation, flipping, rotation, occlusion, scaling, and transformation, among others, with the different images randomly selecting one or more of the data enhancement operations to perform.
S25, combining the image subjected to data enhancement and the original image into a sample library, and carrying out normalization operation on all the images, wherein the calculation formula is as follows:
Wherein/> Is normalized pixel value,/>And/>Maximum and minimum, respectively,/>, of image pixelsRepresenting pixel values of the image.
S26, the normalized image sample library is processed according to the following steps of 4:1 is randomly divided into a training database and a test database.
S3, constructing a depth grading fusion residual network, namely connecting the first residual convolution block, the second residual convolution block, the third residual convolution block and the classification prediction layer in series according to different repetition numbers to form the depth grading fusion residual network, and determining parameters of the branch only on the branch by referring to the feature map size of the main output of the residual block, so that the results of the main and the branch can be added and fused.
S31, constructing a first residual convolution block. The convolution block is composed of a single-layer convolution layer and a pooling layer, wherein the convolution kernel is 7×7 in size, the step size is 2, the number is 64, and the output feature map size is 112×112. The window of the pooling layer is 3×3, the step length is 2, and the size of the output feature map is 56×56; in each residual block, the fusion enhancement of the characteristic information is carried out on the convolution result of each layer through the information of a plurality of branches, so that forgetting and losing of important characteristic information are effectively reduced. And the branch convolution kernel is only 1 multiplied by 1, so that the demand on the calculation amount is smaller compared with a dense network, and the ideal balance is achieved on the improvement of the model performance and the increase of parameters.
S32, constructing a second residual convolution block, wherein the convolution block is composed of a main path formed by 3 convolution layers and a branch path with hierarchical fusion. Wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 64; the convolution kernel of the second convolution layer has the size of 3×3, the step length of 2, the number of 64, and the size of the output feature map is halved; the convolution kernel of the third convolution layer has a size of 1 x 1 and a number of 256. For branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 64; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 64; the convolution kernel of the third-stage fusion branch convolution layer is 1×1 in size and 256 in number. The output of each level of branch convolution is integrated with the output of the normalization layer in the convolution layer of the corresponding main circuit, and then is activated and output by the activation layer.
S33, constructing a third residual convolution block. The convolution block is composed of a main path formed by 3 convolution layers and a branch path which is merged in a grading way. Wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 128; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 128, and an output feature pattern size of 28×28; the convolution kernel of the third convolution layer is 1 x1 in size and 512 in number. For branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1×1 in size and 512 in number. The output of each level of branch convolution is integrated with the output of the normalization layer in the convolution layer of the corresponding main circuit, and then is activated and output by the activation layer.
S34, constructing a fourth residual convolution block. The convolution block is composed of a main path formed by 3 convolution layers and a branch path which is merged in a grading way. Wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 256; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 256, and an output feature pattern size of 14×14; the convolution kernel of the third convolution layer is 1 x1 in size and 1024 in number. For branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1×1 in size and 512 in number. The output of each level of branch convolution is integrated with the output of the normalization layer in the convolution layer of the corresponding main circuit, and then is activated and output by the activation layer.
S35, constructing a fifth residual convolution block. The convolution block is composed of a main path formed by 3 convolution layers and a branch path which is merged in a grading way. Wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 512; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 512, and an output feature map size of 7×7; the convolution kernel of the third convolution layer is 1×1 in size and 2048 in number. For branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 512; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 2048. The output of each level of branch convolution is integrated with the output of the normalization layer in the convolution layer of the corresponding main circuit, and then is activated and output by the activation layer.
S36, constructing a classification prediction layer. The class prediction layer includes a global pooling layer, a fully connected layer, and a softmax layer. The global pooling layer sets pooling windows according to the size of the feature map, outputs vectors with the same number as the feature map channels, the number of nodes of the full-connection layer is taken as the number of categories of the training sample library, and the softmax layer carries out exponential normalization on the output of the full-connection layer, wherein the calculation formula is as follows:
Wherein/> For/>Exponential function of/>For the number of nodes,/>Input firstPersonal node,/>Representing the output of the fully connected layer.
S37, serially connecting 1 first residual convolution block, 2 second residual convolution blocks, 4 third residual convolution blocks, 23 fourth residual convolution blocks, 3 fifth residual convolution blocks and a classification prediction layer to form a depth grading fusion residual network of 101 layers. And for the second convolution layer convolution kernel step sizes of the non-first second residual convolution block, the third residual convolution block, the fourth residual convolution block and the fifth residual convolution block, the corresponding step sizes of the second-stage fusion branch convolution layer convolution kernels are 1, and the node number of the output layer is 9.
As the number of layers of the network increases, the dimension of the feature map in the network increases, a large amount of secondary information is lost, and the network performance is also blocked at the bottleneck, so that the improvement is difficult. The invention conveys the original characteristics to the convolution layers of different layers through a plurality of simple branches, and enhances the expression of different characteristics. This also allows the network to extract more efficient features and additional detail features, making the network break through bottleneck performance. The invention solves the problem that different scenes have different requirements on network parameters and performances, and for scenes with complex parameters and high requirements on high performances, the accuracy can be improved by repeating more numbers, and the repeated number can be reduced for light scenes. In addition, no link exists between different residual blocks, so that the residual blocks can be replaced more conveniently to meet more complex scene requirements.
S4, inputting divided annotation data for training and storing a model in a depth grading fusion residual error network training process and parameter design;
s41, after the sample libraries (training libraries and test libraries) divided in the S26 are disordered, the sample libraries are sequentially input into a depth grading fusion residual network for training according to 64 pictures as a batch, and the number and the size of training samples and test samples in the same batch are the same.
S42, after the batch of images are input into a depth grading fusion residual error network, obtaining prediction output through forward propagation, and calculating cross entropy loss CEL by using labels and the prediction output; the cross entropy loss CEL is calculated as follows:
Wherein/> For the number of samples of a batch, here take 64,/>Is determined according to hardware computational force conditions,/>For category number, 9 is taken here,/>For tag value,/>Is a predicted value for the network. For the calculated CEL, the weight of the convolution layer of the network is reversely updated by using the Adam optimizer at the learning rate of 0.001, and the embodiment of the invention can ensure that the model reaches the optimal optimizing point when the learning rate is 0.001 through a large number of experiments, and the model can not reach the optimal optimizing point due to overlarge, and can slowly converge to be in local optimum instead of global optimum due to overlarge.
And S43, after the updating of all weight parameters of the depth grading fusion residual error network is completed, predicting and outputting the input test set data of the same batch, and comparing the test set data with the corresponding label data to calculate the accuracy and loss index (as shown in figures 4-5), so as to assist in judging whether the training ending condition is mature.
S44, continuously inputting training data and test data of the next batch into the depth grading fusion residual error network, recording all behavior images in the sample library as one cycle when training is completed, and continuously repeating the cycle until the change of the reduction amplitude of the loss index tends to be stable (as shown in fig. 5), wherein the training cycle number is 150.
S45, storing the trained depth grading fusion residual error network model for subsequent identification.
S5, model identification early warning unsafe behavior, and loading model identification new monitoring data;
s51, loading the depth grading fusion residual error network model trained in the step S4.
S52, after newly collecting camera data to be identified, performing operations such as step 1 and step 2 (training data and tested data are not divided any more), inputting the camera data into a loaded depth grading fusion residual error network model, and obtaining a prediction result.
And S53, triggering the early warning system when the predicted unsafe behavior probability exceeds a preset threshold value, and sending early warning information to a control center and a site responsible person.
S6, model optimization updating and equipment maintenance.
And S61, when the newly acquired data quantity reaches the original data level, optimizing and updating the DHFRN model, updating sample training data, training and adjusting model parameters according to the new data in the steps S21-S26.
And S62, maintaining and managing the system every 6 months, including maintenance of the monitoring camera, management of data storage and backup, testing of the early warning system and the like.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for system embodiments, since they are substantially similar to method embodiments, the description is relatively simple, with reference to the description of method embodiments in part.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (8)
1. A mine operation safety detection method based on depth grading fusion residual error network; the method is characterized by comprising the following steps of:
S1, mining operation data acquisition; installing cameras at different key positions in the mine well to acquire real-time video data of the operation scene of the mine workers;
s2, preprocessing the acquired real-time video data of the mine operation, and dividing a data set;
S3, constructing a depth grading fusion residual error network;
S4, training the network model established in the S3 by utilizing the data set divided in the S2;
s5, inputting the real-time video data acquired newly into a network model obtained through training in S4 to identify and early warn unsafe behaviors;
S6, optimizing, updating and maintaining equipment of the network model;
The step S3 specifically comprises the following steps:
S31, constructing a first residual convolution block: the convolution block consists of a single-layer convolution layer and a pooling layer, wherein the convolution kernel is 7×7 in size, the step length is 2, the number is 64, and the size of the output characteristic diagram is 112×112; the window of the pooling layer is 3×3, the step length is 2, and the size of the output feature map is 56×56;
S32, constructing a second residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 64; the convolution kernel of the second convolution layer has the size of 3×3, the step length of 2, the number of 64, and the size of the output feature map is halved; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 256; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 64; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 64; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 256; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S33, constructing a third residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 128; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 128, and an output feature pattern size of 28×28; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 512; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S34, constructing a fourth residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 256; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 256, and an output feature pattern size of 14×14; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 1024; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 128; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 128; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S35, constructing a fifth residual convolution block: the convolution block consists of a main path formed by 3 convolution layers and a branch path with hierarchical fusion; wherein for the main path, the convolution kernel of the first convolution layer has a size of 1×1 and a number of 512; the convolution kernel of the second convolution layer has a size of 3×3, a step size of 2, a number of 512, and an output feature map size of 7×7; the convolution kernel of the third convolution layer has a size of 1×1 and a number of 2048; for branches, the convolution kernel of the first-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 512; the convolution kernel of the convolution layer of the second-stage fusion branch is 3 multiplied by 3, the step length is 2, and the number is 512; the convolution kernel of the third-stage fusion branch convolution layer is 1 multiplied by 1, and the number of the convolution kernels is 2048; after the output of each level of branch convolution is fused with the output of the normalization layer in the convolution layer of the corresponding main circuit, the output is activated and output by the activation layer;
S36, constructing a classification prediction layer: the classification prediction layer comprises a global pooling layer, a full connection layer and a softmax layer; the global pooling layer sets pooling windows according to the size of the feature map, outputs vectors with the same number as the feature map channels, the number of nodes of the full-connection layer is taken as the number of categories of the training sample library, and the softmax layer carries out exponential normalization on the output of the full-connection layer, wherein the calculation formula is as follows:
；
Wherein, For/>Exponential function of/>For the number of nodes,/>Input of/>Personal node,/>An output representing a full connection layer;
S37, serially connecting 1 first residual convolution block, 2 second residual convolution blocks, 4 third residual convolution blocks, 23 fourth residual convolution blocks, 3 fifth residual convolution blocks and a classification prediction layer to form a depth grading fusion residual network of 101 layers; and for the second convolution layer convolution kernel step sizes of the non-first second residual convolution block, the third residual convolution block, the fourth residual convolution block and the fifth residual convolution block, the corresponding step sizes of the second-stage fusion branch convolution layer convolution kernels are 1, and the node number of the output layer is 9.
2. The mine operation safety detection method based on the depth grading fusion residual network according to claim 1; the method is characterized in that in the step S1, the image acquisition function is started for different cameras with cross coverage areas.
3. The mine operation safety detection method based on the depth grading fusion residual network according to claim 1; the method is characterized in that the step S2 specifically comprises the following steps:
S21, decomposing different behavior video data of workers acquired by the camera in the S1 into video frame data, and extracting images from continuous video frames of the same behavior at a speed of 5 frames per second;
S22, cutting the extracted behavior frame image to a uniform size 224 multiplied by 224;
s23, performing Gaussian filtering processing on the clipped image:
；
Wherein, Is the coordinates of the pixel,/>Representing the distance of the coordinate to the center of the convolution kernel,/>Is a cut-off frequency, and takes the value of 50/>Representing the Gaussian filtering of the image;
S24, carrying out data enhancement on the filtered image;
s25, merging the data enhanced image and the original image into a sample library, and carrying out normalization operation on all the images:
；
Wherein, Is normalized pixel value,/>And/>Maximum and minimum, respectively,/>, of image pixelsPixel values representing an image;
s26, normalizing the image according to 4:1 is randomly divided into a training database and a test database.
4. The mine operation safety detection method based on the depth grading fusion residual network according to claim 2; wherein in S2, the data enhancement operation includes one or more of translation, flipping, rotation, occlusion, scaling, and transformation.
5. A mine operation safety detection method based on a depth grading fusion residual network according to claim 1 or 3; the method is characterized in that the step S4 specifically comprises the following steps:
S41, after the sample libraries divided in the S26 are disordered, taking 64 pictures as a batch, and sequentially inputting depth grading fusion residual network training established in the S3, wherein the number and the size of training samples and test samples in the same batch are the same;
S42, after the image is input into a depth grading fusion residual error network, obtaining prediction output through forward propagation, and calculating cross entropy loss CEL by using the label and the prediction output; for the calculated CEL, updating the weights of the convolutional layers of the network back with an Adam optimizer at a learning rate of 0.001; the cross entropy loss CEL is calculated as follows:
；
Wherein, For the number of samples of a batch, here take 64,/>For category number, 9 is taken here,/>For tag value,/>Is a predicted value of the network; /(I)Represents the/>Batch sample,/>Represents the/>A category;
S43, after updating of all weight parameters of the depth grading fusion residual error network is completed, predicting and outputting the input test set data of the same batch, and comparing the test set data with corresponding label data to calculate accuracy and loss index to assist in judging whether the training ending condition is mature;
s44, continuously inputting training data and test data of the next batch into the depth grading fusion residual error network, recording all behavior images in the sample library as one cycle after training, and ending training until the change of the reduction amplitude of the loss index tends to be stable;
S45, storing the depth grading fusion residual error network model trained in the S44 for subsequent identification.
6. The mine operation safety detection method based on the depth grading fusion residual network according to claim 1; the method is characterized in that the step S5 specifically comprises the following steps:
S51, loading the depth grading fusion residual error network model trained in the S4;
S52, if S1 and S2 are adopted, mine operation data are collected again and preprocessed, and are input into a depth grading fusion residual error network model, so that a prediction result is obtained;
And S53, when the predicted unsafe behavior probability exceeds a preset threshold value, early warning information is sent.
7. The mine operation safety detection method based on the depth grading fusion residual network according to claim 6; wherein the process of dividing the data set is not performed in S52.
8. A mine operation safety detection method based on a depth grading fusion residual network according to claim 1 or 3; the method is characterized in that the step S6 specifically comprises the following steps:
S61, when the newly acquired data quantity reaches the original data level, optimizing and updating the depth grading fusion residual error network model, updating training data, training and adjusting model parameters according to the new data in the steps S21-S26;
and S62, maintaining and managing the system every 6 months, including the maintenance of the monitoring camera, the management of data storage and backup, and the test of the early warning system.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410391684.6A CN117975376B (en) | 2024-04-02 | 2024-04-02 | Mine operation safety detection method based on depth grading fusion residual error network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410391684.6A CN117975376B (en) | 2024-04-02 | 2024-04-02 | Mine operation safety detection method based on depth grading fusion residual error network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117975376A CN117975376A (en) | 2024-05-03 |
| CN117975376B true CN117975376B (en) | 2024-06-07 |
Family
ID=90850029
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410391684.6A Active CN117975376B (en) | 2024-04-02 | 2024-04-02 | Mine operation safety detection method based on depth grading fusion residual error network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117975376B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112464844A (en) * | 2020-12-07 | 2021-03-09 | 天津科技大学 | Human behavior and action recognition method based on deep learning and moving target detection |
| WO2021069053A1 (en) * | 2019-10-07 | 2021-04-15 | Huawei Technologies Co., Ltd. | Crowd behavior anomaly detection based on video analysis |
| CN112949572A (en) * | 2021-03-26 | 2021-06-11 | 重庆邮电大学 | Slim-YOLOv 3-based mask wearing condition detection method |
| CN113221817A (en) * | 2021-05-27 | 2021-08-06 | 江苏奥易克斯汽车电子科技股份有限公司 | Abnormal behavior detection method, device and equipment |
| WO2021232969A1 (en) * | 2020-05-20 | 2021-11-25 | 百果园技术(新加坡)有限公司 | Action recognition method and apparatus, and device and storage medium |
| CN114170618A (en) * | 2021-09-28 | 2022-03-11 | 广州新华学院 | Video human behavior recognition algorithm based on double-flow space-time decomposition |
| CN115346264A (en) * | 2022-02-28 | 2022-11-15 | 湖南大学 | Human behavior recognition algorithm based on multi-stream graph convolution residual error network |
| CN115601654A (en) * | 2022-10-09 | 2023-01-13 | 四川煤矿安全监察局安全技术中心（Cn） | Mine tailing pond disaster early warning method based on air-space-ground integration |
| CN115761888A (en) * | 2022-11-22 | 2023-03-07 | 桂林电子科技大学 | Tower crane operator abnormal behavior detection method based on NL-C3D model |
| CN117593689A (en) * | 2023-10-20 | 2024-02-23 | 中南大学 | Mine personnel dangerous behavior identification method and system based on computer vision |
| CN117612249A (en) * | 2023-11-13 | 2024-02-27 | 陕西彬长小庄矿业有限公司 | Underground miner dangerous behavior identification method and device based on improved OpenPose algorithm |
-
2024
- 2024-04-02 CN CN202410391684.6A patent/CN117975376B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021069053A1 (en) * | 2019-10-07 | 2021-04-15 | Huawei Technologies Co., Ltd. | Crowd behavior anomaly detection based on video analysis |
| WO2021232969A1 (en) * | 2020-05-20 | 2021-11-25 | 百果园技术(新加坡)有限公司 | Action recognition method and apparatus, and device and storage medium |
| CN112464844A (en) * | 2020-12-07 | 2021-03-09 | 天津科技大学 | Human behavior and action recognition method based on deep learning and moving target detection |
| CN112949572A (en) * | 2021-03-26 | 2021-06-11 | 重庆邮电大学 | Slim-YOLOv 3-based mask wearing condition detection method |
| CN113221817A (en) * | 2021-05-27 | 2021-08-06 | 江苏奥易克斯汽车电子科技股份有限公司 | Abnormal behavior detection method, device and equipment |
| CN114170618A (en) * | 2021-09-28 | 2022-03-11 | 广州新华学院 | Video human behavior recognition algorithm based on double-flow space-time decomposition |
| CN115346264A (en) * | 2022-02-28 | 2022-11-15 | 湖南大学 | Human behavior recognition algorithm based on multi-stream graph convolution residual error network |
| CN115601654A (en) * | 2022-10-09 | 2023-01-13 | 四川煤矿安全监察局安全技术中心（Cn） | Mine tailing pond disaster early warning method based on air-space-ground integration |
| CN115761888A (en) * | 2022-11-22 | 2023-03-07 | 桂林电子科技大学 | Tower crane operator abnormal behavior detection method based on NL-C3D model |
| CN117593689A (en) * | 2023-10-20 | 2024-02-23 | 中南大学 | Mine personnel dangerous behavior identification method and system based on computer vision |
| CN117612249A (en) * | 2023-11-13 | 2024-02-27 | 陕西彬长小庄矿业有限公司 | Underground miner dangerous behavior identification method and device based on improved OpenPose algorithm |
Non-Patent Citations (1)
| Title |
|---|
| 基于改进双流算法的矿工不安全行为识别方法研究;陈天;中国优秀硕士学位论文全文数据库 工程科技Ⅰ辑;20220315(第3期);B021-542 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117975376A (en) | 2024-05-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109829429B (en) | Security sensitive article detection method based on YOLOv3 under monitoring scene | |
| CN109961019B (en) | Space-time behavior detection method | |
| CN111160125B (en) | Railway foreign matter intrusion detection method based on railway monitoring | |
| CN110232380B (en) | Fire night scene restoration method based on Mask R-CNN neural network | |
| CN110084165B (en) | Intelligent identification and early warning method for abnormal events in open scene of power field based on edge calculation | |
| CN111222478A (en) | Construction site safety protection detection method and system | |
| CN113128412B (en) | Fire trend prediction method based on deep learning and fire monitoring video | |
| CN110910360B (en) | Positioning method of power grid image and training method of image positioning model | |
| CN112766091B (en) | Video unsafe behavior recognition system and method based on human skeleton key points | |
| CN115641357A (en) | Intelligent storage personnel tracking algorithm and video monitoring system based on machine learning | |
| CN116456075A (en) | Automatic inspection system for monitoring video quality | |
| CN116052082A (en) | Power distribution station room anomaly detection method and device based on deep learning algorithm | |
| CN114373162B (en) | Dangerous area personnel intrusion detection method and system for transformer substation video monitoring | |
| CN115906337A (en) | Engineering structure sectional type main aftershock toughness analysis method and system | |
| CN116977292A (en) | Method for detecting cold joint of solar cell | |
| CN115546742A (en) | Rail foreign matter identification method and system based on monocular thermal infrared camera | |
| Lu et al. | Using cased based reasoning for automated safety risk management in construction industry | |
| CN117612249A (en) | Underground miner dangerous behavior identification method and device based on improved OpenPose algorithm | |
| CN117975376B (en) | Mine operation safety detection method based on depth grading fusion residual error network | |
| CN113223200A (en) | Road stealing and escaping intelligent prevention and control method, storage medium and system based on index dimension | |
| CN117237418A (en) | Moving object detection method and system based on deep learning | |
| CN112686107A (en) | Tunnel invading object detection method and device | |
| CN114898181A (en) | Hidden danger violation identification method and device for explosion-related video | |
| CN113963233A (en) | Target detection method and system based on double-stage convolutional neural network | |
| CN114267014A (en) | Subway construction behavior identification method and device, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |