CN115601709A

CN115601709A - Coal mine employee violation statistical system, method and device and storage medium

Info

Publication number: CN115601709A
Application number: CN202211387755.2A
Authority: CN
Inventors: 赵峰; 董云龙
Original assignee: Beijing Wanli Software Development Co ltd
Current assignee: Beijing Wanli Software Development Co ltd
Priority date: 2022-11-07
Filing date: 2022-11-07
Publication date: 2023-01-13
Anticipated expiration: 2042-11-07
Also published as: CN115601709B

Abstract

The application discloses a coal mine employee violation statistical system, method and device and a storage medium. Wherein, the system includes: the image acquisition equipment is arranged on the site of the coal mine production equipment; and the computing equipment is in communication connection with the image acquisition equipment. And the computing device is configured to perform the following: acquiring a field image sequence of corresponding coal mine production equipment acquired by image acquisition equipment; detecting a target image frame containing a worker operating a corresponding coal mine production device in the image sequence; judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and counting the illegal operation of the staff according to the judgment result. Therefore, the technical problem that the existing monitoring technology in the prior art can not effectively count the illegal operation condition of the production equipment, and can not effectively monitor the illegal operation staff in coal mine production is solved.

Description

Coal mine employee violation statistical system, method and device and storage medium

Technical Field

The application relates to the technical field of coal mine safety, in particular to a coal mine employee violation statistical system, a method, a device and a storage medium.

Background

In recent years, coal mine production safety has received increasing attention. More and more monitoring techniques are applied in coal mine production to monitor the safety production of coal mine workers.

The patent publication No. CN114123832A discloses a method, a device, electronic equipment and a storage medium for detecting the crossing risk of the underground pedestrian, wherein the method comprises the following steps: acquiring an underground pedestrian monitoring video image in real time; performing frame cutting processing on the video image to obtain an image to be processed, wherein the image to be processed comprises: a plurality of pedestrian images; identifying the multiple pedestrian images based on a human body posture estimation algorithm to generate human body posture information of each pedestrian; tracking the multiple pedestrian images based on a multi-target pedestrian tracking algorithm to generate a motion track of each pedestrian; and performing coordinate comparison operation according to the human body posture information, the motion trail and the preset dangerous area of each pedestrian, and determining whether the pedestrian has the boundary crossing risk according to the coordinate comparison operation result. By the method and the device, the underground pedestrian boundary crossing detection can be effectively carried out, real-time accurate early warning can be provided for the boundary crossing detection of the pedestrian danger area in the fully mechanized mining face, and the working safety of coal mine workers can be better ensured.

In addition, the published invention patent (publication number CN105956549 a) discloses an automatic inspection system for safety equipment and behavior ability before worker operation, which comprises an image acquisition module, an image preprocessing module, a database module, a processing module, an information output module, a display and a sound which are connected in sequence; the image acquisition module is used for acquiring color image information and skeleton node information of the body of the worker to be detected and transmitting the information to the image preprocessing module; the image preprocessing module establishes a human body three-dimensional model and a human body skeleton node model according to the received information and segments the human body three-dimensional model; the database module comprises a worker information library, a work task library and a safety equipment model library; the processing module is used for comparing the received information after the image preprocessing module is divided with the data in the database module, and transmitting the processing result to the display and the sound through the information transmission module. The invention can realize a simple, effective and quick automatic safety inspection system before worker operation, and is applied to the fields of building construction safety, coal mine production and the like.

In recent years, as coal mining machines, coal caving machines, hydraulic supports and other coal mine production equipment are applied to coal mine production, the safety of coal mine production is continuously guaranteed. In this case, it is preferable that the air conditioner, the potential risk in the coal mine production process is caused by the illegal operation of coal mine workers on coal mine production equipment to a great extent. Wherein the behavior of the violation operation comprises: workers without operation authority operate production equipment in an illegal way, for example, workers without operation authority are authorized to operate a coal mining machine in an illegal way, so that production risk is caused; and workers with operation authority do not operate the production equipment according to the specified flow, for example, operators of the coal mining machine do not operate the coal mining machine according to the requirements of safe production regulations, so that production risks are caused.

However, the existing monitoring technology monitors the safety condition of coal mine production mainly by determining the track, the posture or the position of a worker at a production site, so that the monitoring of illegal operation of production equipment is lacked. Therefore, the existing monitoring technology cannot effectively count the illegal operation condition of the production equipment, and thus, the illegal operation staff in coal mine production cannot be effectively monitored.

Aiming at the technical problem that the existing monitoring technology in the prior art can not effectively count the illegal operation condition of production equipment, so that the illegal operation staff in coal mine production can not be effectively monitored, an effective solution is not provided at present.

Disclosure of Invention

The embodiment of the disclosure provides a coal mine employee violation statistical system, a method, a device and a storage medium, which at least solve the technical problems that the existing monitoring technology in the prior art can not effectively count the violation operation condition of production equipment, and can not effectively monitor the violation operation employees in coal mine production.

According to an aspect of the disclosed embodiment, a coal mine employee violation statistics system is provided, which includes: the image acquisition equipment is arranged on the site of the coal mine production equipment; and the computing equipment is in communication connection with the image acquisition equipment. And the computing device is configured to perform the following: acquiring a field image sequence of corresponding coal mine production equipment acquired by image acquisition equipment; detecting a target image frame containing a worker operating a corresponding coal mine production device in the image sequence; judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and counting the illegal operation of the staff according to the judgment result.

According to another aspect of the disclosed embodiment, there is also provided a coal mine employee violation statistical method, including: acquiring a field image sequence of coal mine production equipment; detecting a target image frame containing a worker operating a coal mine production facility in an image sequence; judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and counting the illegal operation of the staff according to the judgment result.

According to another aspect of the embodiments of the present disclosure, there is also provided a storage medium including a stored program, wherein the method described above is performed by a processor when the program is executed.

According to another aspect of the disclosed embodiment, there is further provided a coal mine employee violation statistics apparatus, including: the image sequence acquisition module is used for acquiring the on-site image sequence of the coal mine production equipment; the target frame detection module is used for detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; the illegal operation judging module is used for judging whether the staff carries out illegal operation on the coal mine production equipment or not according to the target image frame; and the illegal operation counting module is used for counting the illegal operation of the staff according to the judgment result.

According to another aspect of the disclosed embodiment, there is further provided a coal mine employee violation statistics device, including: a processor; and a memory coupled to the processor for providing instructions to the processor for processing the following processing steps: acquiring a field image sequence of coal mine production equipment; detecting a target image frame containing a worker operating a coal mine production facility in an image sequence; judging whether a worker carries out illegal operation on the coal mine production equipment or not according to the target image frame; and counting the illegal operation of the staff according to the judgment result.

Therefore, the technical scheme of the application utilizes the image acquisition equipment to acquire the on-site image sequence of each coal mine production equipment. And detecting from the sequence of images a target image frame containing a worker operating the coal mine production facility. And then judging whether the staff carries out illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the staff according to the judgment result. Therefore, according to the technical scheme, the operation condition of the workers on the coal mine production equipment can be effectively counted, and the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safe production condition of coal mine workers. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation condition of the production equipment, and cannot effectively monitor the illegal operation staff in coal mine production is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the disclosure and together with the description serve to explain the disclosure and not to limit the disclosure. In the drawings:

fig. 1 is a schematic diagram for implementing a coal mine employee violation statistics system according to embodiment 1 of the present disclosure;

fig. 2 is a schematic block architecture diagram of a coal mine employee violation statistics system according to embodiment 1 of the present disclosure;

FIG. 3 is a schematic operational flow diagram of a computing device of a coal mine employee violation statistics system according to embodiment 1 of the present disclosure;

FIG. 4 is a schematic diagram of a first behavior identification module in a computing device according to embodiment 1 of the present disclosure;

FIG. 5 is a diagram illustrating exemplary input and output information for an employee behavior recognition model in a first behavior recognition module;

6A-6I are schematic diagrams illustrating different categories of image frames in an image sequence at a shearer site;

FIG. 7A is a diagram illustrating the reordering module reordering semantic vectors corresponding to behavior classes of a target image frame and semantic vectors corresponding to device status information;

FIG. 7B is a diagram illustrating another example of the reordering module reordering semantic vectors corresponding to behavior classes of a target image frame and semantic vectors corresponding to respective device state information;

FIG. 8A shows a schematic representation of the generation of a second encoded sequence from a first encoded sequence using a sequence processing model;

FIG. 8B shows a schematic diagram of another example of generating a second encoded sequence from a first encoded sequence using a sequence processing model;

FIG. 9A shows a schematic diagram of a sequence processing model formed by an encoder and a decoder;

FIG. 9B shows a schematic diagram of a sequence2sequence model as a sequence processing model;

fig. 10 is a schematic diagram of a coal mine employee violation statistics apparatus according to embodiment 2 of the present disclosure; and

fig. 11 is a schematic diagram of a coal mine employee violation statistics apparatus according to embodiment 3 of the present disclosure.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure. It is to be understood that the described embodiments are merely exemplary of some, and not all, of the present disclosure. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present disclosure without making creative efforts shall fall within the protection scope of the present disclosure.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the above-described drawings are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the disclosure described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

Fig. 1 is a schematic diagram of a coal mine employee violation statistics system according to the embodiment. Referring to fig. 1, the system includes: image acquisition equipment 111 to 113, wherein the image acquisition equipment 111 to 113 is arranged on the site of coal mine production equipment 121 to 123; and the computing equipment 200, wherein the computing equipment 200 is in communication connection with the image acquisition equipment 111-113.

Specifically, referring to fig. 1, the system includes a first image capturing device 111 disposed on site of the shearer 121, a second image capturing device 112 disposed on site of the coal cutter 122, and a third image capturing device 113 disposed on site of the emulsion pump 123.

The first image acquisition device 111 is thus used to acquire a sequence of images of the site of the shearer 121; the second image acquisition device 112 is used for acquiring an image sequence of the site of the coal discharger 122; the third image capturing device 113 is used to capture a sequence of images of the emulsion pump 123 in situ.

Although fig. 1 exemplarily shows a coal mining machine 121, a coal caving machine 122 and an emulsion pump 123 as examples of the coal mining equipment, it should be clear to those skilled in the art that the technical solution of the present disclosure may also include an image acquisition device arranged on the site of other coal mining equipment (such as a coal crusher, etc.) for acquiring an image sequence of the site of the corresponding coal mining equipment. And will not be described in detail herein.

Further, referring to fig. 1, the system further includes a computing device 200 communicatively connected to the image capturing devices 111 to 113. Specifically, the computing device 200 may be in communication connection with the image acquisition devices 111 to 113 through a network, so that the computing device 200 may acquire, through the network, image sequences acquired by the image acquisition devices 111 to 113. Where the computing device 200 may be, for example, a remotely located server. In addition, as shown with reference to FIG. 1, the system further includes a database 300 communicatively connected to the computing device 200 for storing information related to personnel of the coal mine.

Where fig. 3 illustrates a flow diagram of the operation of computing device 200, with reference to fig. 3, computing device 200 is configured to perform the following operations:

s302: acquiring a field image sequence of corresponding coal mine production equipment acquired by image acquisition equipment;

s304: detecting a target image frame containing a worker operating a corresponding coal mine production device in the image sequence;

s306: judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and

s308: and counting the illegal operation of the staff according to the judgment result.

Fig. 2 further shows a schematic block architecture diagram of the coal mine employee violation statistics system shown in fig. 1. Referring to fig. 2, the architecture of the computing device 200 sequentially includes an interface layer 210, a data buffer layer 220, a behavior recognition layer 230, and a violation statistics layer 240 from top to bottom.

Wherein the interface layer 210 includes an image receiving module 211 and a device information receiving module 212. The image receiving module 211 receives image sequences acquired by the image acquisition devices 111 to 113 in real time, and transmits the acquired image sequences to the data buffer layer 220. The equipment information receiving module 212 receives the equipment state information of the coal mine production equipment 121 to 123 in real time and transmits the equipment state information to the data buffer layer 220.

The data buffer layer 220 is used for buffering data received by the interface layer 210 in real time, and includes buffering image sequences received by the image receiving module 211 in real time and device status information received by the device information receiving module 212 in real time.

The behavior recognition layer 230 is configured to recognize behaviors of workers on the production equipment 121 to 123 according to the image sequence temporarily stored in the data buffer layer 220, and determine worker behavior information corresponding to the behaviors of the workers. The behavior recognition layer 230 includes a first behavior recognition module 231, a reordering module 232, and a second behavior recognition module 232. The first behavior recognizing module 231, the reordering module 232, and the second behavior recognizing module 233 will be described in detail below.

The violation statistics layer 240 is used for determining staff who perform violation operations on the coal mine production equipment and counting violation conditions of the staff. Violation statistics layer 240 includes an identity verification module 241, a violation determination module 242, and a violation statistics module 243. The identity verification module 241, violation determination module 242, and violation statistics module 243 are described in detail below.

Therefore, after the image acquisition devices 111 to 113 acquire the on-site image sequences of the coal mine production devices, the image sequences are transmitted to the computing device 200. For example, the image capture device 111 transmits a first image sequence of the shearer 121 site to the computing device 200, the image capture device 112 transmits a second image sequence of the coal discharger 122 site to the computing device 200, and the image capture device 113 transmits a third image sequence of the emulsion pump 123 site to the computing device 200.

Then, the computing device 200 receives the image sequences (i.e., the first to third image sequences) acquired by the image acquisition devices 111 to 113 through the image receiving module 211 of the interface layer 210, and temporarily stores the image sequences in the data buffer layer 220 (S302).

Then, a first behavior recognition module 231 in the behavior recognition layer 230 of the computing device 200 acquires an image sequence from the data buffer layer 220, and detects a target image frame containing workers operating the corresponding coal mine production equipment 111 to 113 from the image sequence. For example, the first behavior identification module 231 detects target image frames containing workers operating the shearer 121 from the first sequence of images; the first behavior recognition module 231 detects a target image frame containing a worker operating the coal player 122 from the second image sequence; or the first behavior recognition module 231 detects a target image frame of a worker operating the coal player 123 from the third image sequence (S304), and transmits the target image frame to the violation statistics layer 240.

And the violation statistic layer 240 of the computing equipment interacts with the behavior recognition layer 230, and judges whether the staff in the target image frame carries out violation operation on the coal mine production equipment or not according to the target image frame. For example, the violation statistical layer 240 determines whether the worker performs the violation operation on the coal mining machine 121 according to the target image frames in the first image sequence, the violation statistical layer 240 determines whether the worker performs the violation operation on the coal discharging machine 122 according to the target image frames in the second image sequence, or the violation statistical layer 240 determines whether the worker performs the violation operation on the coal discharging machine 123 according to the target image frames in the third image sequence (S306).

Then, the violation statistics layer 240 counts the violation operations of the worker by the violation statistics module 243 (S308). The violation operation statistics comprises the statistics of the violation operation conditions of all the workers in a certain period or the overall violation operation conditions of all the workers in a certain period. Therefore, subsequent departments can evaluate the working condition of each worker according to the statistical information of the illegal operation, or evaluate the condition of the safety production of the whole coal mine.

As described in the background art, in recent years, as coal mining machines, coal caving machines, hydraulic supports, and other coal mining production equipment are applied to coal mining production, safety of coal mining production is continuously guaranteed. In this case, the risk potential in the coal mine production process is largely due to the illegal operation of coal mine production equipment by coal mine workers. Wherein the behavior of the violation comprises: workers without operation authority operate production equipment in an illegal way, for example, workers without operation authority are authorized to operate a coal mining machine in an illegal way, so that production risk is caused; and workers with operation authority do not operate the production equipment according to the specified flow, for example, operators of the coal mining machine do not operate the coal mining machine according to the requirements of safe production regulations, so that production risks are caused.

In view of this, the technical solution of the present application utilizes an image acquisition device to acquire an image sequence of each coal mine production device site. And detecting from the sequence of images a target image frame containing a worker operating the coal mine production facility. And then judging whether the staff carries out illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the staff according to the judgment result. Therefore, according to the technical scheme, the operation condition of the workers on the coal mine production equipment can be effectively counted, and the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluating the safety production condition of coal mine workers. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation condition of the production equipment, and cannot effectively monitor the illegal operation staff in coal mine production is solved.

Optionally, the operation of detecting a target image frame containing a worker operating a corresponding coal mine production facility in the image sequence includes: inputting the image sequence into an employee behavior recognition model, wherein the employee behavior recognition model is a neural network-based image classification model, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment; determining employee behavior categories corresponding to each image frame in the image sequence through an employee behavior recognition model; and detecting a target image frame from the image sequence according to the employee behavior category.

In particular, fig. 4 shows a schematic diagram of the first behavior recognizing module 231. As shown in fig. 4, the first behavior recognition module 231 is provided with a plurality of employee behavior recognition models 2311a to 2311c (i.e., employee behavior recognition models) and a target frame transmission sub-module 2312, for example. The behavior recognition layer 230 detects that the image sequences acquired by the image acquisition equipment 111 to 113 contain target image frames of workers operating the coal mine production equipment 121 to 123 by using the first behavior recognition module 231 a.

Specifically, the first behavior recognition module 231 inputs image sequences collected by the image collection devices 111 to 113 into employee behavior recognition models 2311a to 2311c. For example, the first behavior recognition module 231 inputs the first sequence of images of the site of the shearer 121 acquired by the image acquisition device 111 to the employee behavior recognition model 2311a; inputting a second sequence of images of the site of the coal player 122 acquired by the image acquisition device 112 into the employee behavior recognition model 2312a; and inputting the on-site third image sequence of the emulsion pump 123 acquired by the image acquisition device 113 to the employee behavior recognition model 2313a. Thereby detecting a target image frame containing a worker operating the shearer 121 in the first image sequence using the worker behavior recognition model 2311a; detecting a target image frame containing a worker operating the coal discharger 122 in the second image sequence by using the worker behavior recognition model 2311 b; the employee behavior recognition model 2311c is used to detect a target image frame in the third image sequence that contains the employee who is operating the emulsion pump 123.

The employee behavior recognition models 2311a to 2311c can be image classification models based on a neural network, for example. Preferably, the employee behavior recognition models 2311a-2311c may be image classification models based on the resnet50 neural network.

Wherein the categories of the image classification model correspond to different categories of behavior associated with the coal mine production facility. The employee behavior recognition model 2311a will be described as an example.

Referring to fig. 5, the first image sequence of the site of the shearer 121 transmitted by the first image acquisition device 111 includes image framesF ₁ 、F ₂ 、F ₃ 、.....、F _n . The first behavior recognition module 231 images the respective image frames of the first image sequenceF ₁ ~F _n The employee behavior recognition model 2311a is input, and the employee behavior recognition model 2311a is generated and associated with the image frameF ₁ ~F _n Corresponding category vectorC ₁ ~C _n . Wherein the category vectorC ₁ And the image frameF ₁ Corresponding, class vectorC ₂ And the image frameF ₂ Corresponding, analogizing, category vectorC _n And the image frameF _n And (7) corresponding.

Wherein the image framesF _i （i= 1~n) class vectorC _i （i= 1~n) may be, for example, a vector comprising m elements, where each element corresponds to a probability value for a behavior class, where the behavior class corresponding to the element with the highest probability value is determined to correspond to the image frameF _i A corresponding behavior category.

Further, FIGS. 6A-6I illustrate examples of image frames corresponding to different behavior classes.

In particular, fig. 6A shows a schematic view of one image frame of the first image sequence, shown with reference to fig. 6A, in which the staff member 401 is located in a non-operational zone with respect to the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: there is no operation.

Fig. 6B shows a schematic view of yet another image frame in the first image sequence, which is shown with reference to fig. 6B, in which the staff member 401 is manipulating the control panel of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and controlling a coal cutter control panel.

A schematic view of yet another image frame of the first image sequence is shown in fig. 6C and 6D, and with reference to fig. 6C and 6D, in which the worker 401 is servicing the drums and picks of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and (5) overhauling the roller of the coal mining machine.

A schematic view of yet another image frame of the first image sequence is shown in fig. 6E and 6F, and with reference to fig. 6E and 6F, in which the staff member 401 is servicing the water outlet line of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and (5) overhauling the water outlet pipeline of the coal mining machine.

A schematic illustration of yet another image frame of the first image sequence is shown in fig. 6G-6I, and with reference to fig. 6G-6I, in which the staff member 401 is making rounds around the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and (5) patrolling the periphery of the coal mining machine.

The above figures only exemplarily show image frames of several different behavior categories of the staff with respect to the shearer 121, and certainly, the technical solution of the present application may also define more behavior categories with respect to the shearer 121, which is not described herein again.

Thus, table 1 below exemplarily shows the category vectors output by the employee behavior recognition model 2311aC _i Each element of (1)c _i,j （j=1~m) corresponding to the behavior category:

TABLE 1

Thus, when the image frame isF _i

Corresponding category vectorC

_i 1 st element of the respective elementsc _i,1 When the probability of (2) is highest, then the image frameF _i The corresponding behavior class is no operation, so that the first behavior recognition module 231 determines the image frameF _i The staff 401 in (a) has no operation on the shearer. When the image frame isF _i Corresponding category vectorC _i Of the respective elements of (2)c _i,2 When the probability of (2) is highest, then the image frameF _i The corresponding action category is to operate the coal cutter control panel so that the first action recognition module 231 determines the image frameF _i The staff 401 is handling the coal cutter control panel and so on.

And wherein the behavior category 1 shown in table 1 may be considered that the operator in the image frame is not operating the shearer 121. The behavior types 2 to 5 in table 1 can be regarded as the operation of the coal mining machine 121 by the operator in the image frame. In this manner, the first behavior recognition module 231 may thus detect a target image frame of the worker's operation of the shearer 121 from the first image sequence and transmit the target image frame to the behavior recognition layer of the computing device 200 via the target frame transmission sub-module 2312.

The above description has been made only for the first image series corresponding to the shearer 121 and the employee behavior recognition model 2311 a. However, it should be understood by those skilled in the art that the second image sequence and the employee behavior recognition model 2311b on the coal caving machine site, or the third image sequence and the employee behavior recognition model 2311c on the emulsion pump site may be implemented by using an image classification model with the same architecture as that of the employee behavior recognition model 2311a, as long as different image sample sets are used for training to obtain different weight parameters.

Optionally, determining whether the staff performs an illegal operation on the coal mine production equipment according to the target image frame includes: determining identity information of a worker according to the target image frame; and judging whether the worker performs illegal operation on the coal mine production equipment or not according to the identity information of the worker.

Referring to fig. 2 and 4, violation statistics layer 240 of computing device 200 includes an authentication module 241 and a violation determination module 242. After the first behavior recognition module 231 of the behavior recognition layer 230 transmits the target image frame to the violation statistics layer 240 through the target frame transmission sub-module 2312. The identity verification module 241 of the violation statistics layer 240 receives the target image frame and determines identity information of the worker in the target image frame.

Specifically, the identity verification module 241 is preset with a face detection recognition model, so as to perform face detection and face feature extraction on the staff in the target image frame, and perform retrieval in the database 300 based on the extracted face features, so as to determine the identity of the staff in the target image frame. The face detection and recognition model may adopt a common target detection model, such as a Faster RCNN network, which is not described herein again.

For example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame including the worker operating the shearer 121 through the worker behavior recognition model 2311 a. The identity verification module 241 may thus determine identity information of workers operating the shearer 121 based on the target image frames. The violation determination module 242 then makes a determination based on the identity information of the worker. If the identity information of the worker indicates that the worker has the authority to operate the coal mining machine 121, the violation determination module 242 does not determine that the operation of the worker is a violation operation; if the identity information of the staff member indicates that the staff member does not have the authority to operate the shearer 121, the determination module 242 determines that the operation of the staff member is an illegal operation.

For another example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame including the worker operating the coal player 122 through the worker behavior recognition model 2311 b. The identity verification module 241 may thus determine identity information of the personnel operating the coal player 122 from the target image frames. The violation determination module 242 then makes a determination based on the identity information of the worker. If the identity information of the worker indicates that the worker has the authority to operate the coal discharger 122, the violation determination module 242 does not determine that the operation of the worker is a violation operation; if the identity information of the worker indicates that the worker does not have the authority to operate the coal discharger 122, the determination module 242 determines that the operation of the worker is an illegal operation.

For another example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame including the staff who operates the emulsion pump 123 through the staff behavior recognition model 2311c. The identity module 241 can thus determine identity information of the personnel operating the emulsion pump 123 based on the target image frames. The violation determination module 242 then makes a determination based on the identity information of the worker. If the identity information of the worker indicates that the worker has the authority to operate the emulsion pump 123, the violation determination module 242 does not determine that the operation of the worker is a violation operation; if the identity information of the worker indicates that the worker does not have the authority to operate the emulsion pump 123, the determination module 242 determines that the operation of the worker is an illegal operation.

Therefore, according to the technical scheme, the illegal operation of workers without operation authority on the coal mine production equipment can be accurately monitored through the image acquisition equipment and the computing equipment which are arranged on the site of the coal mine production equipment. Therefore, the production risk condition of illegal operation of the non-authorized staff can be effectively monitored.

Optionally, under the condition that it is determined according to the identity information of the staff that the staff does not perform an illegal operation on the coal mine production equipment and the staff behavior category corresponding to the target image frame is a control panel for controlling the coal mine production equipment, determining whether the staff performs the illegal operation on the coal mine production equipment according to the target image frame further includes: determining the control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information related to the target image frame, wherein the equipment state information indicates the equipment state of the coal mine production equipment corresponding to the target image frame; and determining whether the operation of controlling a control panel of the coal mine production equipment by a worker is illegal according to the determined control operation.

Specifically, the present embodiment will be described below focusing on one of the points of the present application that greatly contributes to the prior art.

Fig. 6B illustrates, by way of example of the shearer 121, image frames of a control panel (i.e., a control panel of a coal mine production facility) on which a worker operates the shearer 121. A control panel of the coal mine production facility includes a plurality of switches or buttons. Taking the coal mining machine 121 as an example, the control panel comprises: the device comprises a scraper conveyor unlocking button, a scraper conveyor switch button, a coal mining machine switch button, an isolating switch handle, a motor switch button, a left roller clutch handle, a left roller adjusting button, a right roller clutch handle, a right roller adjusting button, a left roller switch button, a right roller switch button, a traction switch button, a spraying button and other controls.

However, when the image acquisition device 111 is disposed at a distance from the control panel of the shearer 121, the employee behavior recognition model 2311a of the first behavior recognition module 231 of the computing device 200 can only determine, through the image frames shown in fig. 6B, that the current operation of the worker 401 is to operate the control panel of the shearer 121. However, since the actions of the worker 401 pressing different buttons or operating different switch handles on the control panel are relatively small, the worker behavior recognition model 2311a is difficult to recognize the specific control operation of the worker on the control panel of the shearer 121 through the image frames shown in fig. 6B.

On the other hand, for safe production of the coal mine, there is a corresponding operating regulation for the operation of the control panel of the shearer 121. For example, for the shearer 121, according to the operating rules, simply starting the shearer 121 requires the staff to perform the following procedures in order:

1) Releasing the locking of the scraper conveyor;

2) Starting the scraper conveyor;

3) Starting the coal mining machine;

4) Closing the isolating switch;

5) Starting the motor;

6) Closing the left drum clutch;

7) Adjusting the position of the left roller;

8) Closing the right drum clutch;

9) Adjusting the position of the right roller;

10 Starting the left roller;

11 Start the right drum;

12 Start traction;

13 ) turn on the spray device.

Accordingly, the correct operational flow for the worker is: pressing a scraper conveyor unlock button, pressing a scraper conveyor shift knob, pressing a shearer shift knob, closing an isolation switch handle, pressing a motor shift knob, closing a left drum clutch handle, pressing a left drum adjustment button, closing a right drum clutch handle, pressing a right drum adjustment button, pressing a left drum shift knob, pressing a right drum shift knob, pressing a tow shift knob, and pressing a spray button.

As can be seen, even if the worker 401 having the operation authority of the shearer 121 operates the control panel of the shearer 121, if the worker 401 does not operate the controls on the control panel according to the prescribed regulations, there is still an illegal operation on the shearer 121.

Since the employee behavior recognition model 2311a of the first behavior recognition module 231 of the computing device 200 is only able to determine, from the image frames shown in fig. 6B, that the current operation of the worker 401 is operating the control panel of the shearer 121, and is not able to identify the specific operation of the worker at the control panel of the shearer 121. It is difficult to determine whether the worker 401 operates the coal cutter control panel according to the prescribed safe operating rules by the worker behavior recognition model 2311a of the first behavior recognition model 231. Therefore, it is difficult to accurately evaluate the work violation of the staff of the shearer 121.

In addition, similar problems also exist in other coal mine safety production equipment, and the description is omitted here.

In view of this, referring to fig. 2, the interface layer 210 of the computing device 230 is further provided with a device information receiving module 212, configured to receive device status information of the coal mine production devices 121 to 123 in real time. Thus, the behavior recognition layer 230 of the computing device 230 can determine the specific operation of the control panel of the coal mine production device by the staff in the target image frame according to the target image frame and the device state information of the coal mine production device. For example, the behavior recognition layer 230 may determine the specific operation of the staff at the control panel of the shearer 121 based on the target image frames extracted from the first image sequence and the equipment state information of the shearer 121; determining the specific operation of the staff at the control panel of the coal discharger 122 according to the target image frame extracted from the second image sequence and the equipment state information of the coal discharger 122; and determining the specific operation of the staff at the control panel of the emulsion pump 123 according to the target image frame extracted from the third image sequence and the device state information of the emulsion pump 123. Specifically, the following description will be given taking the shearer 121 as an example.

Referring to fig. 2 and table 1, for the target image frame of which the behavior category determined by the employee behavior recognition model 2311a is the behavior category 2, the violation statistics layer 240 returns the target image frame to the behavior recognition layer 230 after determining that the worker is a worker with the operation authority of the shearer 121 through the authentication module 241 and the violation determination module 242.

The behavior recognition layer 230 thus acquires, from the data buffer layer 220, a plurality of pieces of equipment state information of the shearer 121 within a predetermined period of time corresponding to the time stamp information of the target image frame, according to the time stamp information of the target image frame. Wherein the equipment state information is used for indicating the equipment state of the shearer 121, the equipment state including: complete shutdown, scraper conveyor lockout, scraper conveyor unlock, shearer shutdown, shearer operation, disconnector open, disconnector closed, left drum clutch open, left drum clutch closed, left drum rocker swing, left drum rocker stop, right drum clutch open, right drum clutch closed, right drum rocker swing, right drum rocker stop, left drum operation, right drum stop, right drum operation, traction stop, traction start, spray stop, and spray start, among others. In addition, the device states also include different combinations of the above states, which are not described herein again.

For example, the behavior recognition layer 230 acquires, from the timestamp information of the target image frame, the device state information transmitted by the shearer 121 at each of the predetermined times before and after the time corresponding to the timestamp information, as the device state information associated with the target image frame.

The behavior recognition layer 230 then determines the specific control operation of the staff in the target image frame on the control panel of the shearer 121 according to the target image frame and the device state information of the shearer 121, and transmits the control operation information of the determined control operation to the violation determination module 242 of the violation statistics layer 240. Therefore, the violation determining module 242 determines whether the control operation of the worker on the control panel meets the requirement of the safety operation rule according to the control operation information, and further determines whether the control operation is a violation operation.

For other coal mine production equipment (such as the coal discharger 122 and the emulsion pump 123), whether the staff with the operation authority performs the illegal operation on the control panel of the coal mine production equipment can be determined through the operation similar to the operation. And will not be described in detail herein.

Therefore, according to the technical scheme, the employee behavior category of the control panel of the coal mine production equipment, which is identified by the image, is combined with the state information of the coal mine production equipment, so that the operation of an operator with an operation authority on the control panel of the coal mine production equipment can be identified more accurately, and the illegal operation of the operator on the control panel of the coal mine production equipment can be detected. Therefore, the condition of illegal operation of coal mine workers can be monitored more effectively. And, with respect to specific method steps for determining a specific control operation of the worker on the control panel of the shearer 121 in the target image frame according to the target image frame and the device state information of the shearer 121, detailed description will be made below.

Optionally, determining, according to the employee behavior category and the device state information associated with the target image frame, a control operation of the employee in the target image frame on the control panel, including: determining the employee behavior category and semantic vectors corresponding to the equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first coding sequence related to the equipment state information and the employee behavior category; generating a second coding sequence corresponding to the first coding sequence by utilizing a preset sequence processing model based on an encoder and a decoder; extracting a target semantic vector corresponding to the employee behavior category from the second coding sequence; and determining the control operation of the staff in the target image frame on the control panel according to the extracted target semantic vector.

Specifically, referring to fig. 2, the behavior recognition layer 230 is deployed with a reordering module 232 and a second behavior recognition module 233.

The reordering module 232 obtains, according to the timestamp information of the target image frame, device status information transmitted by the coal mine production device corresponding to the target image frame at each time before and after the time corresponding to the timestamp information as device status information associated with the target image frame. For example, still taking the shearer 121 as an example, the reordering module 232 acquires, according to the timestamp information of the target image frame of the shearer 121 (the behavior category corresponding to the target image frame is category 2 in table 1, i.e., the control panel of the shearer), the device status information transmitted by the shearer 121 at each of the predetermined times before and after the time corresponding to the timestamp information, as the device status information associated with the target image frame.

Then, referring to FIG. 7A, the reordering module 232 generates semantic vectors corresponding to the respective device state information and behavior categories of the target image frameqe ₀ ~qe _u And to semantic vectors in chronological orderqe ₀ ~qe _u And sequencing to form a first coding sequence.

The inventor notices that each state information of the coal mine production equipment presents semantic related information in time sequence because each link of the coal mine production equipment in the operation process needs to be carried out according to a specified operation rule. Still taking the shearer 121 as an example, the shearer 121 is started, paused, and stopped according to prescribed safety regulations. Thus, during the operation of the control panel of the shearer 121 by the staff, the respective equipment state information of the shearer 121 exhibits a time-series correlation in time sequence, and the time-series correlation is similar to the semantic correlation of the respective words of the natural language. In addition, the individual equipment status information also exhibits a chronological correlation with the operator's operation of the control panel of the shearer 121 (although the specific operation is unknown).

Therefore, the inventor regards the state information of each device and the operation of the control panel of the shearer 121 by the staff as different words by using the natural language processing method for encoding, obtains a semantic vector corresponding to the operation behavior of the staff on the control panel (since the specific operation of the staff on the control panel cannot be determined by the first behavior recognition module 231, the semantic vector is unique), and obtains a semantic vector corresponding to different device state information.

Specifically, for example, referring to table 2 below, a word library may be constructed in a hot unique code manner, and the word library includes codes corresponding to the operation behaviors of the control panel by the staff and codes corresponding to the state information of each device.

TABLE 2

Then, using the sequence segments of the device status information in the actual safe production of the shearer 121 as samples, using the existing word2vec model (e.g., cbow model), semantic vectors corresponding to the different definitions described in table 2 are generated. As shown in table 3 below:

TABLE 3

Thus, referring to FIG. 7A, the reordering module 232 may determine semantic vectors for the respective device status information and semantic vectors corresponding to the status "operator applies action to control panel" according to Table 3. Thereby obtaining a first coding sequenceqe ₀ ~qe _u 。

In addition, as shown in fig. 7B, the reordering module 232 may also arrange a plurality of target image frames, which are acquired from the violation determination module 242 and whose employee behavior category is behavior category 2 (which controls the coal mining machine control panel), in a time sequence. And for the time period spanned by the plurality of target image frames, acquiring equipment state information associated with the shearer 121. For example, the predetermined time before the time period, the predetermined time after that, and the equipment state information of the shearer 1 acquired during the time period are acquired. The reordering module 232 orders the semantic vectors of the acquired device status information and the semantic vectors corresponding to the target image frames according to time to generate a first codeSequence ofqe ₀ ~qe _u . Considering that the frequency of the control panel of the coal mining machine 121 operated by the staff is much lower than the sampling frequency of the coal mining machine 121 on the equipment state, the reordering module 232 can appropriately reduce the sampling frame rate of the target image frames in the reordering process. Thus, the first code sequence shown in fig. 7B reflects not only the timing relationship between the device status information and the status "the worker applies the operation to the control panel", but also the timing relationship between the different time statuses "the worker applies the operation to the control panel".

Then, as shown in fig. 8A and 8B, after the reordering module 232 constructs the first coding sequence, the second behavior recognizing module 233 of the behavior recognizing layer 230 generates a corresponding second coding sequence from the first coding sequence using a preset sequence processing modelqs ₀ ~qs _v . Wherein the second coded sequence is also composed of a plurality of semantic vectorsqs ₀ ~qs _v And (4) forming. Wherein each semantic vector of the second coding sequence corresponds to a control operation of the control panel of the shearer 121 by the worker.

Wherein Table 4 below shows semantic vectors corresponding to control operations by a worker

TABLE 4

The semantic vector shown in table 4 may be generated by referring to a method similar to the semantic vector shown in table 3, and is not described herein again. In addition, the frequency of the staff operating the control panel is far lower than the sampling frequency of the shearer 121 for acquiring the equipment state information. Table 4 is therefore filled with a last semantic vector corresponding to "no operation".

Referring also to fig. 9A, the sequence processing model is, for example, an encoder and decoder based processing model. Referring to FIG. 9A, the encoder of the sequence processing model is based on the first encoded sequence of the inputqe ₀ ~qe _u Generating intermediate semantic featuresC ₁ ~C _v . Wherein the intermediate semantic featuresC ₁ ~C _v Respectively associated with the second code sequenceqs ₀ ~qs _v And (7) corresponding. Wherein the encoder may be an encoder known in the art and will not be described herein.

The decoder may then generate a second encoded sequence according to the following equationqs ₀ ~qs _v 。

Wherein the functionfAs a function of the decoder, it may use a decoder function known in the art and will not be described herein.

Therefore, according to the technical scheme of the disclosure, a sequence processing model frequently used in natural language processing can be utilized, and a second coding sequence for representing the control operation of a worker on a control scheme can be generated according to the representation equipment state information and the first coding sequence of the state of applying the operation to the control panel by the worker. In this manner, the computing device may thus accurately identify control operations of the worker at the control panel of the shearer 121 via the second behavior identification module 233. The second behavior identification module 233 then transmits the identified control operation to the violation determination module 242 of the violation statistics layer 240. The violation determination module 242 may thus determine whether the control operation of the worker is a violation operation based on the prescribed safe operation rule.

Although the shearer 121 is described above as an example, reference may be made to the above scenario for operation of the control panel of the remaining coal production equipment (e.g., the coal discharger 122 and the emulsion pump 123).

Therefore, the technical scheme of the application utilizes the sequence processing model based on natural language processing, and can predict and identify the control operation of the staff on the control panel based on the equipment state information of the coal mine production equipment, so that the control operation of the staff on the control panel can be accurately identified.

Referring also to fig. 9B, as an example of the sequence processing model, the sequence processing model may be a sequence2sequence model based on LSTM.

Optionally, the operation of counting the illegal operation of the worker according to the determination result includes: and evaluating the working state of the worker according to the number of illegal operations of the worker in a preset period.

Specifically, referring to fig. 1 and 2, the computing device 200 may count the number of illegal operations of each employee of the coal mine at predetermined intervals (e.g., monthly, quarterly, or yearly) based on the determination of the illegal operations of the employee. Thus, the working status of the staff member every month, every quarter, or every year can be evaluated. For example, if the number of illegal operations of a certain worker in a certain month exceeds a predetermined threshold, the worker is in a poor working state, and so on. Therefore, according to the technical scheme, the actual working state of the worker can be evaluated by counting the illegal operation condition of the worker on the coal mine production equipment. Thereby being beneficial to the coal mine production equipment which is operated by the monitoring staff of the coal mine unit in a compliance way.

Further, according to a second aspect of the present embodiment, a coal mine employee violation statistics method is provided, which is implemented by the computing device 200 shown in fig. 1. The method comprises the following steps:

s102: acquiring a field image sequence of coal mine production equipment;

s104: detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence;

s106: judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and

s108: and counting the illegal operation of the staff according to the judgment result.

Optionally, the operation of detecting a target image frame containing a worker operating a corresponding coal mine production facility in the image sequence includes: inputting the image sequence into a first employee behavior recognition model, wherein the first employee behavior recognition model is a neural network-based image classification model, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment; determining employee behavior categories corresponding to each image frame in the image sequence through a first employee behavior recognition model; and detecting a target image frame from the image sequence according to the employee behavior category.

Optionally, determining whether the staff performs an illegal operation on the coal mine production equipment according to the target image frame includes: determining identity information of a worker according to the target image frame; and judging whether the worker conducts illegal operation on the coal mine production equipment or not according to the identity information of the worker.

Optionally, under the condition that it is determined according to the identity information of the staff that the staff does not perform an illegal operation on the coal mine production equipment and the staff behavior category corresponding to the image frame is a control panel for controlling the coal mine production equipment, it is determined according to the target image frame whether the staff performs the illegal operation on the coal mine production equipment, further comprising: acquiring a plurality of pieces of equipment state information related to a target image frame according to the timestamp of the target image frame; determining the control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the equipment state information indicates the equipment state of the coal mine production equipment corresponding to the target image frame; and determining whether the operation of a control panel of the coal mine production equipment operated by a worker is illegal according to the determined control operation.

Optionally, determining, according to the employee behavior category and the device state information associated with the target image frame, a control operation of the employee in the target image frame on the control panel, including: determining the employee behavior category and the semantic vector corresponding to the equipment state information, and arranging the determined semantic vectors according to the time sequence to form a first coding sequence related to the equipment state information and the employee behavior category; generating a second coding sequence corresponding to the first coding sequence by utilizing a preset sequence processing model based on an encoder and a decoder, wherein semantic vectors contained in the second coding sequence are respectively related to control operation aiming at a control panel; extracting a target semantic vector corresponding to the employee behavior category from the second coding sequence; and determining the control operation of the staff in the target image frame on the control panel according to the extracted target semantic vector.

Further, according to a third aspect of the present embodiment, there is provided a storage medium. The storage medium includes a stored program, wherein the method of any of the above is performed by a processor when the program is run.

Therefore, the technical scheme of the embodiment utilizes the image acquisition equipment to acquire the on-site image sequence of each coal mine production equipment. And detecting from the sequence of images a target image frame containing a worker operating the coal mine production facility. And then judging whether the staff carries out illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the staff according to the judgment result. Therefore, according to the technical scheme, the operation condition of the workers on the coal mine production equipment can be effectively counted, and the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safe production condition of coal mine workers. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation condition of the production equipment, and cannot effectively monitor the illegal operation staff in coal mine production is solved.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but the former is a better implementation mode in many cases. Based on such understanding, the technical solutions of the present invention or portions thereof contributing to the prior art may be embodied in the form of a software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal device (which may be a mobile phone, a computer, a server, or a network device) to execute the method according to the embodiments of the present invention.

Example 2

Fig. 10 shows a coal mine employee violation statistics apparatus 1000 according to the present embodiment, the apparatus 1000 corresponding to the method according to the second aspect of embodiment 1. Referring to fig. 10, the apparatus 1000 includes: the image sequence acquisition module is used for acquiring the on-site image sequence of the coal mine production equipment; the target frame detection module is used for detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; the illegal operation judging module is used for judging whether the staff carries out illegal operation on the coal mine production equipment or not according to the target image frame; and the illegal operation counting module is used for counting the illegal operation of the staff according to the judgment result.

Optionally, the target frame detection module includes: the input submodule is used for inputting the image sequence into a first staff behavior recognition model, wherein the first staff behavior recognition model is an image classification model based on a neural network, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment; the behavior recognition submodule is used for determining staff behavior categories corresponding to all image frames in the image sequence through the first staff behavior recognition model; and the detection sub-module is used for detecting a target image frame from the image sequence according to the employee behavior category.

Optionally, the violation operation determination module includes: the identity recognition sub-module is used for determining identity information of the staff according to the target image frame; and the illegal operation judgment submodule is used for judging whether the staff carries out illegal operation on the coal mine production equipment or not according to the identity information of the staff.

Optionally, the violation operation determination sub-module further includes: the device information acquisition unit is used for acquiring a plurality of device state information related to the target image frame according to the time stamp of the target image frame; the control operation identification unit is used for determining the control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the equipment state information indicates the equipment state of the coal mine production equipment corresponding to the target image frame; and the illegal operation judging unit is used for determining whether the operation of the control panel of the coal mine production equipment controlled by the staff is illegal operation according to the determined control operation.

Optionally, the control operation recognition unit includes: the first coding sequence generation subunit is used for determining the employee behavior category and the semantic vector corresponding to the equipment state information, and arranging the determined semantic vectors according to the time sequence to form a first coding sequence related to the equipment state information and the employee behavior category; the second coding sequence generation subunit is used for generating a second coding sequence corresponding to the first coding sequence by utilizing a preset sequence processing model based on an encoder and a decoder, wherein semantic vectors contained in the second coding sequence are respectively related to control operation aiming at the control panel; the target semantic vector extraction subunit extracts a target semantic vector corresponding to the employee behavior category from the second coding sequence; and the violation operation judgment subunit is used for determining the control operation of the staff in the target image frame on the control panel according to the extracted target semantic vector.

Optionally, the violation operation statistics module includes: and the illegal operation counting submodule is used for evaluating the working state of the staff according to the number of illegal operations of the staff in a preset period.

Therefore, the technical scheme of the embodiment obtains the image sequence of each coal mine production equipment site. And detecting a target image frame from the sequence of images containing a worker operating the coal mine production facility. And then judging whether the staff carries out illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the staff according to the judgment result. Therefore, according to the technical scheme, the operation condition of the workers on the coal mine production equipment can be effectively counted, and the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluating the safety production condition of coal mine workers. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation condition of the production equipment, and cannot effectively monitor the illegal operation staff in coal mine production is solved.

Example 3

Fig. 11 shows a coal mine employee violation statistics apparatus 1100 according to the present embodiment, where the apparatus 1100 corresponds to the method according to the second aspect of embodiment 1. Referring to fig. 11, the apparatus 1100 includes: a processor 1110; and a memory 1120, coupled to the processor 1110, for providing instructions to the processor 1110 to process the following processing steps: acquiring a field image sequence of coal mine production equipment; detecting a target image frame containing a worker operating a coal mine production facility in an image sequence; judging whether a worker conducts illegal operation on coal mine production equipment or not according to the target image frame; and counting the illegal operation of the staff according to the judgment result.

Optionally, under the condition that it is determined according to the identity information of the staff that the staff does not perform an illegal operation on the coal mine production equipment and the staff behavior category corresponding to the image frame is a control panel for controlling the coal mine production equipment, it is determined according to the target image frame whether the staff performs the illegal operation on the coal mine production equipment, further comprising: acquiring a plurality of pieces of equipment state information related to the target image frame according to the time stamp of the target image frame; determining the control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the equipment state information indicates the equipment state of the coal mine production equipment corresponding to the target image frame; and determining whether the operation of controlling a control panel of the coal mine production equipment by a worker is illegal according to the determined control operation.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and various media capable of storing program codes.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A coal mine employee violation statistics system, comprising:

the system comprises image acquisition equipment (111 to 113), wherein the image acquisition equipment (111 to 113) is arranged on the site of coal mine production equipment (121 to 123); and

a computing device (200), the computing device (200) being communicatively connected with the image acquisition devices (111 to 113) and configured to:

acquiring on-site image sequences of corresponding coal mine production equipment, which are acquired by the image acquisition equipment (111 to 113);

detecting target image frames containing workers operating corresponding coal mine production equipment (111-113) in the image sequence;

judging whether the worker carries out illegal operation on the coal mine production equipment (111-113) or not according to the target image frame; and

and counting the illegal operations of the workers according to the judgment result.

2. The system of claim 1, wherein detecting in the sequence of images a target image frame containing a worker operating a respective coal mine production facility (111-113) comprises:

inputting the image sequence into an employee behavior recognition model, wherein the employee behavior recognition model is a neural network-based image classification model, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment;

determining employee behavior categories corresponding to each image frame in the image sequence through the employee behavior recognition model; and

and detecting the target image frame from the image sequence according to the employee behavior category.

3. The system according to claim 2, wherein the step of determining whether the worker performs illegal operation on the coal mine production equipment (111 to 113) according to the target image frame comprises the following steps:

determining identity information of the staff according to the target image frame; and

and judging whether the worker carries out illegal operation on the coal mine production equipment (111-113) or not according to the identity information of the worker.

4. The system according to claim 3, wherein when it is determined that the staff does not perform illegal operation on the coal mine production equipment according to the identity information of the staff, and the staff behavior category corresponding to the target image frame is a control panel for controlling the coal mine production equipment, it is determined whether the staff performs illegal operation on the coal mine production equipment (111-113) according to the target image frame, further comprising:

acquiring a plurality of pieces of equipment state information associated with the target image frame according to the timestamp of the target image frame;

determining the control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the equipment state information indicates the equipment state of the coal mine production equipment corresponding to the target image frame; and

and determining whether the operation of the staff for operating the control panel of the coal mine production equipment is illegal operation according to the determined control operation.

5. The system of claim 4, wherein determining control operations at the control panel with the staff in the target image frame according to the staff behavior category and the device status information comprises:

determining the employee behavior category and a semantic vector corresponding to the equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first coding sequence related to the equipment state information and the employee behavior category;

generating a second coding sequence corresponding to the first coding sequence by utilizing a preset sequence processing model based on an encoder and a decoder, wherein semantic vectors contained in the second coding sequence are respectively related to control operation aiming at the control panel;

extracting a target semantic vector from the second coding sequence, wherein the target semantic vector corresponds to a control operation of a worker on the control panel in the target image frame; and

and determining the control operation of the staff in the target image frame on the control panel according to the extracted target semantic vector.

6. The system according to claim 1, wherein the operation of counting the illegal operation of the staff according to the determination result comprises: and evaluating the working state of the worker according to the number of illegal operations of the worker in a preset period.

7. A coal mine employee violation statistical method is characterized by comprising the following steps:

acquiring a field image sequence of coal mine production equipment;

detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence;

judging whether the staff carries out illegal operation on the coal mine production equipment or not according to the target image frame; and

8. A storage medium comprising a stored program, wherein the method of claim 7 is performed by a processor when the program is run.

9. The utility model provides a colliery staff violation statistics device which characterized in that includes:

the image sequence acquisition module is used for acquiring the on-site image sequence of the coal mine production equipment;

the target frame detection module is used for detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence;

the illegal operation judging module is used for judging whether the staff carries out illegal operation on the coal mine production equipment or not according to the target image frame; and

and the violation operation counting module is used for counting the violation operations of the workers according to the judgment result.

10. The utility model provides a colliery staff violation statistics device which characterized in that includes:

a processor; and

a memory coupled to the processor for providing instructions to the processor for processing the following processing steps:

acquiring a field image sequence of coal mine production equipment;

judging whether the worker carries out illegal operation on the coal mine production equipment or not according to the target image frame; and