CN115601709B

CN115601709B - Colliery staff violation statistics system, method, device and storage medium

Info

Publication number: CN115601709B
Application number: CN202211387755.2A
Authority: CN
Inventors: 赵峰; 董云龙
Original assignee: Beijing Wanli Software Development Co ltd
Current assignee: Beijing Wanli Software Development Co ltd
Priority date: 2022-11-07
Filing date: 2022-11-07
Publication date: 2023-10-27
Anticipated expiration: 2042-11-07
Also published as: CN115601709A

Abstract

The application discloses a coal mine employee violation statistics system, method and device and a storage medium. Wherein, the system includes: the image acquisition equipment is arranged on the site of the coal mine production equipment; and the computing device is in communication connection with the image acquisition device. And the computing device is configured to perform the following: acquiring an image sequence of a scene of corresponding coal mine production equipment acquired by image acquisition equipment; detecting a target image frame containing a worker operating the corresponding coal mine production equipment in the image sequence; judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and counting the illegal operations of the staff according to the judgment result. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment, and cannot effectively monitor illegal operation staff in coal mine production is solved.

Description

Colliery staff violation statistics system, method, device and storage medium

Technical Field

The application relates to the technical field of coal mine safety, in particular to a coal mine employee violation statistics system, method and device and a storage medium.

Background

In recent years, coal mine production safety has received increasing attention. More and more monitoring technology is applied to coal mine production to monitor the safety production of coal mine workers.

The disclosed invention patent (publication number CN114123832 a) discloses a method, a device, an electronic device and a storage medium for detecting risk of crossing a boundary of a pedestrian in a well, wherein the method comprises the following steps: acquiring underground pedestrian monitoring video images in real time; performing frame cutting processing on the video image to obtain an image to be processed, wherein the image to be processed comprises: a plurality of pedestrian images; the pedestrian images are identified based on a human body posture estimation algorithm, so that human body posture information of each pedestrian is generated; tracking the plurality of pedestrian images based on a multi-target pedestrian tracking algorithm to generate a motion trail of each pedestrian; and carrying out coordinate comparison operation according to the human body posture information, the motion trail and the preset dangerous area of each pedestrian, and determining whether the pedestrians are out of range according to the coordinate comparison operation result. By the method, underground pedestrian crossing detection can be effectively carried out, real-time accurate early warning can be provided for pedestrian dangerous area crossing detection in the fully-mechanized mining face, and working safety of coal mine workers can be well ensured.

In addition, the disclosed invention patent (publication number CN105956549 a) discloses an automatic checking system for safety equipment and behavior capability before worker operation, comprising an image acquisition module, an image preprocessing module, a database module, a processing module, an information output module, a display and a sound which are connected in sequence; the image acquisition module is used for acquiring color image information and skeleton node information of the body of a worker to be inspected and transmitting the information to the image preprocessing module; the image preprocessing module establishes a human body three-dimensional model and a human body skeleton node model according to the received information, and segments the human body three-dimensional model; the database module comprises a worker information base, a work task base and a safety equipment model base; the processing module is used for comparing the received information after the segmentation of the image preprocessing module with the data in the database module, and transmitting the processing result to the display and the sound equipment through the information transmission module. The invention can realize a simple, effective and quick automatic safety inspection system before the worker works, and is applied to the fields of building construction safety, coal mine production and the like.

In recent years, as coal mine production equipment such as coal mining machines, coal caving machines and hydraulic supports is applied to coal mine production, the safety of coal mine production is continuously ensured. In this case, the risk potential in the coal mine production process is largely derived from the illegal operation of coal mine production equipment by coal mine staff. Wherein the act of violating operations includes: a worker without operation authority illegally operates production equipment, for example, the worker without operation authority illegally operates a coal mining machine, so that production risks are caused; and operating personnel with operation authority do not operate production equipment according to a specified flow, for example, coal mining machine operators do not operate the coal mining machine according to the requirements of a safety production rule, so that production risks are caused.

However, existing monitoring techniques monitor the safety of coal mine production mainly by determining the trajectory, posture or position of workers at the production site, and thus lack monitoring of illegal operation of production equipment. Therefore, the existing monitoring technology cannot effectively count the illegal operation conditions of production equipment, and therefore cannot effectively monitor illegal operation staff in coal mine production.

Aiming at the technical problems that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment and cannot effectively monitor illegal operation staff in coal mine production, no effective solution is proposed at present.

Disclosure of Invention

The embodiment of the disclosure provides a coal mine employee violation statistics system, method, device and storage medium, which at least solve the technical problem that the existing monitoring technology in the prior art cannot effectively count the condition of the violation operation of production equipment, so that the violation operation staff in the coal mine production cannot be effectively monitored.

According to one aspect of the disclosed embodiments, there is provided a coal mine employee violation statistics system, comprising: the image acquisition equipment is arranged on the site of the coal mine production equipment; and the computing device is in communication connection with the image acquisition device. And the computing device is configured to perform the following: acquiring an image sequence of a scene of corresponding coal mine production equipment acquired by image acquisition equipment; detecting a target image frame containing a worker operating the corresponding coal mine production equipment in the image sequence; judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and counting the illegal operations of the staff according to the judgment result.

According to another aspect of the embodiment of the present disclosure, there is also provided a coal mine employee violation statistics method, including: acquiring an image sequence of a site of coal mine production equipment; detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and counting the illegal operations of the staff according to the judgment result.

According to another aspect of the embodiments of the present disclosure, there is also provided a storage medium including a stored program, wherein the method described above is performed by a processor when the program is run.

According to another aspect of the embodiment of the present disclosure, there is also provided a colliery staff violation statistics device, including: the image sequence acquisition module is used for acquiring an image sequence of a site of coal mine production equipment; a target frame detection module for detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; the illegal operation judging module is used for judging whether a worker performs illegal operation on the coal mine production equipment according to the target image frame; and the illegal operation statistics module is used for counting the illegal operation of the staff according to the judgment result.

According to another aspect of the embodiment of the present disclosure, there is also provided a colliery staff violation statistics device, including: a processor; and a memory, coupled to the processor, for providing instructions to the processor for processing the steps of: acquiring an image sequence of a site of coal mine production equipment; detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; counting illegal operations of staff according to the judgment result

Therefore, the technical scheme of the application utilizes the image acquisition equipment to acquire the image sequence of each coal mine production equipment site. And detecting a target image frame from the image sequence containing a worker operating the coal mine production equipment. And then judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the worker according to a judgment result. Therefore, according to the technical scheme of the application, the operation condition of workers on the coal mine production equipment can be effectively counted, and thus, the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safety production condition of coal mine staff. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment, and cannot effectively monitor illegal operation staff in coal mine production is solved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the disclosure, illustrate and explain the present disclosure, and together with the description serve to explain the present disclosure. In the drawings:

FIG. 1 is a schematic diagram of a system for implementing coal mine employee violation statistics in accordance with embodiment 1 of the present disclosure;

FIG. 2 is a schematic block diagram of a colliery employee violation statistics system according to embodiment 1 of the disclosure;

FIG. 3 is a schematic flow diagram of the operation of a computing device of a coal mine employee violation statistics system according to embodiment 1 of the present disclosure;

FIG. 4 is a schematic diagram of a first behavior recognition module in a computing device in accordance with embodiment 1 of the present disclosure;

FIG. 5 is a schematic diagram schematically illustrating input information and output information of an employee behavior recognition model in a first behavior recognition module;

6A-6I are schematic diagrams illustrating different classes of image frames in an image sequence of a shearer scene;

FIG. 7A is a schematic diagram showing a reordering module reordering semantic vectors corresponding to behavior categories of a target image frame and semantic vectors corresponding to respective device state information;

FIG. 7B is a schematic diagram illustrating another example of a reordering module reordering semantic vectors corresponding to behavior categories of a target image frame and semantic vectors corresponding to respective device state information;

FIG. 8A shows a schematic diagram of generating a second coding sequence from a first coding sequence using a sequence processing model;

FIG. 8B illustrates a schematic diagram of another example of generating a second coding sequence from a first coding sequence using a sequence processing model;

FIG. 9A shows a schematic diagram of a sequence processing model of an encoder and decoder;

FIG. 9B shows a schematic diagram of a sequence2sequence model as a sequence processing model;

FIG. 10 is a schematic diagram of a colliery employee violation statistics device according to embodiment 2 of the disclosure; and

fig. 11 is a schematic diagram of a colliery employee violation statistics device according to embodiment 3 of the disclosure.

Detailed Description

In order to better understand the technical solutions of the present disclosure, the following description will clearly and completely describe the technical solutions of the embodiments of the present disclosure with reference to the drawings in the embodiments of the present disclosure. It will be apparent that the described embodiments are merely embodiments of a portion, but not all, of the present disclosure. All other embodiments, which can be made by one of ordinary skill in the art without inventive effort, based on the embodiments in this disclosure, shall fall within the scope of the present disclosure.

It should be noted that the terms "first," "second," and the like in the description and claims of the present disclosure and in the foregoing figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the disclosure described herein may be capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example 1

Fig. 1 is a schematic diagram of a coal mine employee violation statistics system according to the present embodiment. Referring to fig. 1, the system includes: image acquisition devices 111-113, wherein the image acquisition devices 111-113 are arranged on the site of coal mine production devices 121-123; and the computing device 200, wherein the computing device 200 is in communication connection with the image acquisition devices 111-113.

Specifically, referring to fig. 1, the system includes a first image capturing device 111 disposed at the site of a shearer 121, a second image capturing device 112 disposed at the site of a coal caving machine 122, and a third image capturing device 113 disposed at the site of an emulsion pump 123.

The first image acquisition device 111 is thus used to acquire a sequence of images of the scene of the shearer 121; the second image acquisition device 112 is used for acquiring an image sequence of the site of the coal caving machine 122; the third image acquisition device 113 is used to acquire a sequence of images of the scene of the emulsion pump 123.

Although coal cutter 121, coal caving machine 122, and emulsion pump 123 are illustratively shown in fig. 1 as examples of coal mine production equipment, it should be apparent to those skilled in the art that the disclosed aspects may also include image acquisition devices disposed at the site of other coal mine production equipment (e.g., coal pulverizer, etc.) for acquiring image sequences of the site of the corresponding coal mine production equipment. And will not be described in detail here.

Further, referring to fig. 1, the system further includes a computing device 200 communicatively coupled to the image capturing devices 111-113. Specifically, the computing device 200 may be communicatively connected to each of the image capturing devices 111 to 113 through a network, so that the computing device 200 may obtain, through the network, an image sequence captured by each of the image capturing devices 111 to 113. Wherein the computing device 200 may be a remotely located server, for example. In addition, and with reference to FIG. 1, the system also includes a database 300 communicatively coupled to the computing device 200 for storing information related to staff members of the coal mine.

Wherein fig. 3 illustrates a flowchart of the operation of computing device 200, and with reference to fig. 3, computing device 200 is configured to:

s302: acquiring an image sequence of a scene of corresponding coal mine production equipment acquired by image acquisition equipment;

s304: detecting a target image frame containing a worker operating the corresponding coal mine production equipment in the image sequence;

s306: judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and

s308: and counting the illegal operation of the staff according to the judgment result.

Wherein fig. 2 further illustrates a schematic block diagram of the coal mine employee violation statistics system illustrated in fig. 1. Referring to FIG. 2, the architecture of the computing device 200 includes, in order from top to bottom, an interface layer 210, a data buffer layer 220, a behavior recognition layer 230, and a violation statistics layer 240.

Wherein the interface layer 210 includes an image receiving module 211 and a device information receiving module 212. The image receiving module 211 receives the image sequences acquired by the image acquisition devices 111 to 113 in real time, and transmits the acquired image sequences to the data buffer layer 220. The equipment information receiving module 212 receives the equipment status information of the coal mine production equipment 121 to 123 in real time, and transmits the equipment status information to the data buffer layer 220.

The data buffer layer 220 is configured to buffer data received in real time by the interface layer 210, and includes an image sequence received in real time by the buffered image receiving module 211 and device status information received in real time by the device information receiving module 212.

The behavior recognition layer 230 is configured to recognize behaviors of staff on the site of the production devices 121-123 according to the image sequence temporarily stored in the data buffer layer 220, and determine staff behavior information corresponding to the behaviors of the staff. The behavior recognition layer 230 includes a first behavior recognition module 231, a reordering module 232, and a second behavior recognition module 232. Regarding the first behavior recognition module 231, the reordering module 232, and the second behavior recognition module 233, detailed description will be made below.

The violation statistics layer 240 is used for determining staff members performing violation operations on coal mine production facilities and counting violation conditions of the staff members. The violation statistics layer 240 includes an identity verification module 241, a violation determination module 242, and a violation statistics module 243. The authentication module 241, the violation determination module 242, and the violation statistics module 243 will be described in detail below.

Thus, after the image collection apparatuses 111 to 113 collect the image sequence of the site of the coal mine production apparatus, the image sequence is transmitted to the computing apparatus 200. For example, image acquisition device 111 transmits a first image sequence of the job site of coal cutter 121 to computing device 200, image acquisition device 112 transmits a second image sequence of the job site of coal cutter 122 to computing device 200, and image acquisition device 113 transmits a third image sequence of the job site of emulsion pump 123 to computing device 200.

Then, the computing device 200 receives the image sequences (i.e. the first image sequence to the third image sequence) acquired by the image acquisition devices 111 to 113 through the image receiving module 211 of the interface layer 210, and temporarily stores the image sequences to the data buffer layer 220 (S302).

The first behavior recognition module 231 in the behavior recognition layer 230 of the computing device 200 then obtains a sequence of images from the data buffer layer 220 and detects a target image frame from the sequence of images that contains staff operating the respective coal mine production devices 111-113. For example, the first behavior recognition module 231 detects a target image frame containing a worker operating the shearer 121 from the first image sequence; the first behavior recognition module 231 detects a target image frame containing a worker operating the coal caving machine 122 from the second image sequence; or the first behavior recognition module 231 detects a target image frame of a worker operating the coal caving machine 123 from the third image sequence (S304), and transmits the target image frame to the violation statistics layer 240.

The violation statistics layer 240 of the computing device interacts with the behavior recognition layer 230 to determine from the target image frames whether a worker in the target image frame is performing a violation operation on the coal mine production device. For example, the violation statistics layer 240 determines whether the worker performs a violation operation on the coal mining machine 121 based on the target image frame in the first image sequence, the violation statistics layer 240 determines whether the worker performs a violation operation on the coal caving machine 122 based on the target image frame in the second image sequence, or the violation statistics layer 240 determines whether the worker performs a violation operation on the coal caving machine 123 based on the target image frame in the third image sequence (S306).

Then, the violation statistics layer 240 performs statistics on the violation operations of the staff members through the violation statistics module 243 (S308). Wherein the illegal operation statistics comprise statistics of illegal operation conditions of each staff in a certain period or the whole illegal operation conditions of all staff in a certain period. Therefore, the follow-up department can evaluate the working condition of each staff according to the statistical information of the illegal operation or evaluate the condition of the whole coal mine safety production.

As described in the background art, in recent years, as coal mine production equipment such as a coal cutter, a coal caving machine, and a hydraulic support is applied to coal mine production, safety of coal mine production is continuously ensured. In this case, the risk potential in the coal mine production process is largely derived from the illegal operation of coal mine production equipment by coal mine staff. Wherein the act of violating operations includes: a worker without operation authority illegally operates production equipment, for example, the worker without operation authority illegally operates a coal mining machine, so that production risks are caused; and operating personnel with operation authority do not operate production equipment according to a specified flow, for example, coal mining machine operators do not operate the coal mining machine according to the requirements of a safety production rule, so that production risks are caused.

In view of the above, the technical scheme of the application utilizes the image acquisition equipment to acquire the image sequence of each coal mine production equipment site. And detecting a target image frame from the image sequence containing a worker operating the coal mine production equipment. And then judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the worker according to a judgment result. Therefore, according to the technical scheme of the application, the operation condition of workers on the coal mine production equipment can be effectively counted, and thus, the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safety production condition of coal mine staff. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment, and cannot effectively monitor illegal operation staff in coal mine production is solved.

Optionally, detecting in the image sequence an operation of a target image frame containing a worker operating the respective coal mine production apparatus, comprising: inputting the image sequence into an employee behavior recognition model, wherein the employee behavior recognition model is a neural network-based image classification model, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment; determining employee behavior categories corresponding to each image frame in the image sequence through an employee behavior recognition model; and detecting the target image frames from the image sequence according to the employee behavior category.

Specifically, fig. 4 shows a schematic diagram of the first behavior recognition module 231. Referring to fig. 4, the first behavior recognition module 231 is provided with a plurality of employee behavior recognition models 2311 a-2311 c (i.e., employee behavior recognition models) and a target frame transmission sub-module 2312, for example. The behavior recognition layer 230 detects, by using the first behavior recognition module 231a, that the image sequence acquired by the image acquisition devices 111 to 113 contains a target image frame of a worker operating the coal mine production devices 121 to 123.

Specifically, the first behavior recognition module 231 inputs the image sequences acquired by the image acquisition devices 111 to 113 to the employee behavior recognition models 2311a to 2311c. For example, the first behavior recognition module 231 inputs a first image sequence of the scene of the shearer 121 acquired by the image acquisition device 111 into the employee behavior recognition model 2311a; inputting the second image sequence of the site of the coal caving machine 122 acquired by the image acquisition device 112 into the employee behavior recognition model 2311b; and inputting the third image series of the site of the emulsion pump 123 acquired by the image acquisition device 113 to the employee behavior recognition model 2311c. Thereby detecting a target image frame in the first image sequence containing a worker operating the shearer 121 using the staff behavior recognition model 2311a; detecting a target image frame in the second image sequence containing a worker operating the coal caving machine 122 using the employee behavior identification model 2311b; the employee behavior recognition model 2311c is utilized to detect a target image frame in the third image sequence that contains a worker operating the emulsion pump 123.

The employee behavior recognition models 2311a through 2311c may be, for example, image classification models based on neural networks. Preferably, the employee behavior recognition models 2311 a-2311 c may be image classification models based on a resnet50 neural network.

Wherein the categories of the image classification model correspond to different behavioral categories associated with the coal mine production facility. The employee behavior recognition model 2311a is described below as an example.

Referring to fig. 5, a first image sequence of the scene of the shearer 121 transmitted by the first image capturing device 111 includes image framesF ₁ 、F ₂ 、F ₃ 、.....、F _n . The first behavior recognition module 231 recognizes each image frame of the first image sequenceF ₁ ~F _n The employee behavior recognition model 2311a is input such that the employee behavior recognition model 2311a generates and image framesF ₁ ~F _n Corresponding class vectorC ₁ ~C _n . Wherein the category vectorC ₁ And image framesF ₁ Corresponding, class vectorC ₂ And image framesF ₂ Correspondingly, and so on, category vectorsC _n And image framesF _n Corresponding to the above.

Wherein the image framesF _i （iClass vector of =1 to n)C _i （i=1 to n) may be, for example, a vector containing m elements, wherein each element corresponds to a probability value of one behavior class, wherein the behavior class corresponding to the element with the highest probability value is determined as being associated with the image frame F _i A corresponding behavior class.

Further, fig. 6A to 6I show examples of image frames corresponding to different behavior categories.

Specifically, fig. 6A shows a schematic view of an image frame in the first image sequence, and referring to fig. 6A, in which the worker 401 is located in a non-operating region with respect to the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: no operation is performed.

Fig. 6B shows a schematic view of yet another image frame in the first image sequence, with reference to fig. 6B, in which the operator 401 is manipulating the control panel of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and controlling a control panel of the coal mining machine.

A schematic representation of yet another image frame in the first image sequence is shown in fig. 6C and 6D, with reference to fig. 6C and 6D, in which the operator 401 is servicing the drums and picks of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and overhauling the drum of the coal mining machine.

A schematic representation of yet another image frame in the first image sequence is shown in fig. 6E and 6F, with reference to fig. 6E and 6F, in which the operator 401 is servicing the water outlet line of the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and overhauling a water outlet pipeline of the coal mining machine.

A schematic diagram of yet another image frame in the first image sequence is shown in fig. 6G-6I, with reference to fig. 6G-6I, in which the worker 401 is making a tour around the shearer 121. The definition of the behavior class corresponding to the image frame is therefore: and (5) patrolling around the coal mining machine.

The above figures only show the image frames of the staff member for several different behavior categories of the shearer 121 by way of example, but of course the technical solution of the present application may also define more behavior categories for the shearer 121, which are not described here in detail.

Thus, table 1 below exemplarily shows the class vectors output by employee behavior recognition model 2311aC _i Are each of the elements of (1)c _i,j （j=1~m) definition of the behavior class corresponding to:

TABLE 1

Thus, when an image frameF _i Corresponding class vectorC _i 1 st element of the respective elements of (2)c _i,1 When the probability of (1) is highest, then the image frameF _i The corresponding behavior category is no operation, so that the first behavior recognition module 231 determines an image frameF _i The worker 401 in (a) does not operate the shearer. When the image framesF _i Corresponding class vectorC _i The 2 nd element of the respective elements of (2)c _i,2 When the probability of (1) is highest, then the image frameF _i The corresponding behavior category is to control the shearer control panel so that the first behavior recognition module 231 determines the image frame F _i The operator 401 pair is manipulating the shearer control panel, and so on.

And wherein the behavior category 1 shown in table 1 may be regarded as the operator in the image frame is not operating the shearer 121. The behavior categories 2 to 5 in table 1 can be regarded as the operator operating the coal mining machine 121 in the image frame. Thus, in this manner, the first behavior recognition module 231 may detect a target image frame of the operator's operation of the shearer 121 from the first image sequence and transmit the target image frame to the behavior recognition layer of the computing device 200 through the target frame transmission sub-module 2312.

The above description is only given for the first image sequence corresponding to the shearer 121 and the employee behavior recognition model 2311 a. It will be appreciated by those skilled in the art that the second image sequence and employee behavior recognition model 2311b for the coal caving machine site, or the third image sequence and employee behavior recognition model 2311c for the emulsion pump site, may be implemented using an image classification model of the same architecture as the employee behavior recognition model 2311a, provided that different sets of image samples are used for training to obtain different weight parameters.

Optionally, determining whether the staff is performing the illegal operation on the coal mine production equipment according to the target image frame comprises: determining the identity information of the staff according to the target image frame; and judging whether the staff performs illegal operation on the coal mine production equipment according to the identity information of the staff.

Referring to fig. 2 and 4, the violation statistics layer 240 of the computing device 200 includes an identity verification module 241 and a violation determination module 242. After the first behavior recognition module 231 of the behavior recognition layer 230 transmits the target image frame to the violation statistics layer 240 through the target frame transmission sub-module 2312. The identity verification module 241 of the violation statistics layer 240 receives the target image frame and determines the identity information of the staff in the target image frame.

Specifically, the identity verification module 241 is provided with a face detection recognition model in advance, so that face detection and face feature extraction can be performed on the staff in the target image frame, and retrieval is performed in the database 300 based on the extracted face features, so that the identity of the staff in the target image frame is determined. The face detection and recognition model may be a commonly used target detection model, such as a fast RCNN network, which is not described herein.

For example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame containing the worker operating the shearer 121 through the worker behavior recognition model 2311 a. So that the identity verification module 241 can determine the identity information of the worker operating the shearer 121 from the target image frames. The violation determination module 242 then determines based on the identity information of the worker. If the identity information of the worker indicates that the worker has the right to operate the shearer 121, the violation determination module 242 does not determine that the worker's operation is a violation operation; if the identity information of the worker indicates that the worker does not have the right to operate the shearer 121, the determination module 242 determines that the worker's operation is an illegal operation.

For another example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame containing the worker operating the coal caving machine 122 through the worker behavior recognition model 2311 b. So that the identity verification module 241 can determine the identity information of the worker operating the coal caving machine 122 from the target image frames. The violation determination module 242 then determines based on the identity information of the worker. If the identity information of the worker indicates that the worker has the right to operate the coal caving machine 122, the violation determination module 242 does not determine that the worker's operation is a violation operation; if the identity information of the worker indicates that the worker does not have the authority to operate the coal caving machine 122, the determination module 242 determines that the worker's operation is an offending operation.

For another example, the first behavior recognition module 231 transmits the target image frame to the violation statistics layer 240 after detecting the target image frame containing the worker operating the emulsion pump 123 through the worker behavior recognition model 2311 c. So that the identity verification module 241 can determine the identity information of the worker operating the emulsion pump 123 from the target image frames. The violation determination module 242 then determines based on the identity information of the worker. If the identity information of the worker indicates that the worker has authority to operate the emulsion pump 123, the violation determination module 242 does not determine that the worker's operation is a violation operation; if the identity information of the worker indicates that the worker does not have the authority to operate the emulsion pump 123, the determination module 242 determines that the worker's operation is an illegal operation.

Therefore, the technical scheme of the disclosure can accurately monitor the illegal operation of the coal mine production equipment by the staff without operation authority through the image acquisition equipment and the computing equipment arranged on the coal mine production equipment site. Therefore, the production risk condition of illegal operation of unauthorized staff can be effectively monitored.

Optionally, in the case that it is determined that the worker does not perform the illegal operation on the coal mine production equipment according to the identity information of the worker, and the employee behavior category corresponding to the target image frame is a control panel for controlling the coal mine production equipment, it is determined whether the worker performs the illegal operation on the coal mine production equipment according to the target image frame, further including: determining control operation of staff in the target image frame on a control panel according to staff behavior category and equipment state information associated with the target image frame, wherein the equipment state information indicates equipment state of coal mine production equipment corresponding to the target image frame; and determining whether the operation of the control panel of the coal mine production equipment operated by the staff is illegal operation or not according to the determined control operation.

In particular, the present embodiment will hereinafter focus on one point of the application that the present application has a great contribution to the art.

Fig. 6B illustrates an image frame of a control panel of the coal cutter 121 (i.e., a control panel of a coal mine production facility) operated by a worker, taking the coal cutter 121 as an example. But the control panel of the coal mine production facility includes a plurality of switches or buttons. Taking the coal mining machine 121 as an example, the control panel includes: the device comprises a scraper conveyor unlocking button, a scraper conveyor switch button, a coal mining machine switch button, a disconnecting switch handle, a motor switch button, a left roller clutch handle, a left roller adjusting button, a right roller clutch handle, a right roller adjusting button, a left roller switch button, a right roller switch button, a traction switch button, a spraying button and a plurality of other controls.

However, when the image capturing apparatus 111 is set to be a distance from the control panel of the shearer 121, the employee behavior recognition model 2311a of the first behavior recognition module 231 of the computing apparatus 200 can determine that the current operation of the worker 401 is to operate the control panel of the shearer 121 only by the image frame shown in fig. 6B. However, since the actions of the worker 401 pressing different buttons or operating different switch levers on the control panel are relatively small, it is difficult for the worker behavior recognition model 2311a to recognize the specific control operation of the worker on the control panel of the shearer 121 by the image frame shown in fig. 6B.

On the other hand, for safe production of coal mine, there is a corresponding operation procedure for the operation of the control panel of the shearer 121. For example, for the shearer 121, merely starting the shearer 121 requires the operator to sequentially perform the following procedures in accordance with the operating protocol:

1) Releasing the locking of the scraper conveyor;

2) Starting the scraper conveyor;

3) Starting the coal mining machine;

4) Closing the isolating switch;

5) Starting the motor;

6) Closing the left drum clutch;

7) Adjusting the position of the left roller;

8) Closing the right drum clutch;

9) Adjusting the position of the right roller;

10 A) starting the left roller;

11 A) starting the right roller;

12 A) starting traction;

13 Opening the spraying device.

Accordingly, the correct operational flow for the staff is accordingly: pressing a scraper conveyor unlocking button, pressing a scraper conveyor switch button, pressing a coal mining machine switch button, closing a disconnecting switch handle, pressing a motor switch button, closing a left roller clutch handle, pressing a left roller adjusting button, closing a right roller clutch handle, pressing a right roller adjusting button, pressing a left roller switch button, pressing a right roller switch button, pressing a traction switch button and pressing a spraying button.

It follows that even if a worker 401 having the operation authority of the shearer 121 operates the control panel of the shearer 121, there is still an illegal operation of the shearer 121 if the worker 401 does not operate the controls on the control panel according to the prescribed procedure.

Since the employee behavior recognition model 2311a of the first behavior recognition module 231 of the computing device 200 is only able to determine that the current operation of the worker 401 is to operate the control panel of the shearer 121 through the image frame shown in fig. 6B, the specific operation of the worker at the control panel of the shearer 121 cannot be recognized. It is therefore difficult to determine whether the worker 401 operates the shearer control panel in accordance with the prescribed safety operating procedures by the employee behavior recognition model 2311a of the first behavior recognition model 231. Thus, it is difficult to accurately evaluate the working violation of the operator of the shearer 121.

In addition, similar problems exist for other coal mine safety production equipment, and the description is omitted here.

In view of this, referring to fig. 2, the interface layer 210 of the computing device 230 is further provided with a device information receiving module 212, configured to receive device status information of each of the coal mine production devices 121 to 123 in real time. Thus, the behavior recognition layer 230 of the computing device 230 can determine specific operations of the control panel of the coal mine production equipment by the staff in the target image frame based on the target image frame and the equipment status information of the coal mine production equipment. For example, the behavior recognition layer 230 may determine a specific operation of a worker at a control panel of the shearer 121 based on the target image frames extracted from the first image sequence and the equipment state information of the shearer 121; determining a specific operation of a worker on a control panel of the coal discharger 122 according to the target image frame extracted from the second image sequence and the equipment state information of the coal discharger 122; and determining a specific operation of the worker on the control panel of the emulsion pump 123 based on the target image frame extracted from the third image sequence and the equipment state information of the emulsion pump 123. Specifically, the following description will be given by taking the shearer 121 as an example.

Referring to fig. 2 and table 1, for the target image frame of which the behavior class determined by the employee behavior recognition model 2311a is behavior class 2, the violation statistics layer 240 returns the target image frame to the behavior recognition layer 230 after determining that the worker is a worker having the operation authority of the shearer 121 through the identity verification module 241 and the violation determination module 242.

The behavior recognition layer 230 thus acquires a plurality of pieces of equipment state information of the shearer 121 for a predetermined period of time corresponding to the timestamp information of the target image frame from the data buffer layer 220 according to the timestamp information of the target image frame. Wherein the equipment status information is used to indicate equipment status of the shearer 121, the equipment status including: complete shutdown, scraper conveyor locking, scraper conveyor unlocking, coal mining machine shutdown, coal mining machine operation, disconnecting switch opening, disconnecting switch closing, left roller clutch opening, left roller clutch closing, left roller rocker arm swinging, left roller rocker arm stopping, right roller clutch opening, right roller clutch closing, right roller rocker arm swinging, right roller rocker arm stopping, left roller operation, right roller stopping, right roller operation, traction stopping, traction starting, spraying stopping, spraying starting and the like. In addition, the device status also includes different combinations of the above states, which are not described herein.

For example, the behavior recognition layer 230 acquires, from the time stamp information of the target image frame, the equipment state information transmitted by the shearer 121 at each of the times before and after the time corresponding to the time stamp information as the equipment state information associated with the target image frame.

The behavior recognition layer 230 then determines a specific control operation of the worker in the target image frame on the control panel of the shearer 121 according to the target image frame and the equipment state information of the shearer 121, and transmits control operation information of the determined control operation to the violation determination module 242 of the violation statistics layer 240. The violation determination module 242 determines whether the control operation of the worker on the control panel meets the requirement of the safety operation procedure according to the control operation information, and further determines whether the control operation is a violation operation.

For other coal mine production equipment (such as the coal caving machine 122 and the emulsion pump 123), whether the staff member with the operation authority performs the illegal operation on the control panel of the coal mine production equipment can also be judged through the operation similar to the operation. And will not be described in detail herein.

Therefore, the technical scheme combines the staff behavior category of the control panel of the coal mine production equipment with the state information of the coal mine production equipment, so that the operation of the staff with the operation authority on the control panel of the coal mine production equipment can be more accurately identified, and the illegal operation of the staff on the control panel of the coal mine production equipment is detected. Therefore, the illegal operation condition of coal mine staff can be monitored more effectively. And specific method steps for determining specific control operations of the control panel of the shearer 121 by the staff member in the target image frame based on the target image frame and the equipment state information of the shearer 121 will be described in detail below.

Optionally, determining a control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information associated with the target image frame includes: determining semantic vectors corresponding to employee behavior types and equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first coding sequence related to the equipment state information and the employee behavior types; generating a second coding sequence corresponding to the first coding sequence by using a preset sequence processing model based on an encoder and a decoder; extracting a target semantic vector corresponding to the employee behavior category from the second code sequence; and determining the control operation of the staff in the control panel in the target image frame according to the extracted target semantic vector.

Specifically, referring to FIG. 2, behavior recognition layer 230 is deployed with a reordering module 232 and a second behavior recognition module 233.

Wherein the reordering module 232 acquires, as the equipment status information associated with the target image frame, equipment status information transmitted by the coal mine production equipment corresponding to the target image frame at each time within a predetermined time before and after the time corresponding to the time stamp information according to the time stamp information of the target image frame. For example, still taking the shearer 121 as an example, the reordering module 232 obtains, as the equipment status information associated with the target image frame of the shearer 121, equipment status information transmitted by the shearer 121 at each of a predetermined time before and after a time corresponding to timestamp information, according to the timestamp information of the target image frame (the behavior category corresponding to the target image frame is category 2 in table 1, i.e., the shearer control panel is manipulated).

Then, referring to FIG. 7A, the reordering module 232 generates semantic vectors corresponding to the respective device status information and behavior categories of the target image framesqe ₀ ~qe _u And time-sequentially aligning semantic vectorsqe ₀ ~qe _u Ordering is performed to form a first coding sequence.

The inventor notices that each state information of the coal mine production equipment presents semantic related information in time sequence as each link of the coal mine production equipment in the operation process needs to be carried out according to a specified operation rule. Still taking the shearer 121 as an example, the start, pause, and stop of the shearer 121 all need to be in accordance with prescribed safety regulations. Thus, during the operation of the control panel of the shearer 121 by the worker, the respective equipment state information of the shearer 121 exhibits a timing dependency in time series, and the timing dependency is similar to the semantic dependency of the respective words of natural language. In addition, the individual equipment status information also exhibits a time-series correlation with the operator's operation of the control panel of the shearer 121 (although the specific operation is not known).

The inventor refers to the natural language processing method, and regards each piece of equipment state information and the control of the control panel of the coal mining machine 121 by the staff as different words, so as to encode, obtain a semantic vector corresponding to the operation behavior of the staff on the control panel (the semantic vector is unique because the specific operation of the staff on the control panel cannot be determined according to the first behavior recognition module 231), and obtain a semantic vector corresponding to the different equipment state information.

Specifically, for example, referring to table 2 below, a word stock may be constructed in a thermally unique manner, which includes codes corresponding to the operation actions of the staff on the control panel and codes corresponding to the respective device state information.

TABLE 2

Then, using the sequence pieces of equipment status information in the actual safe production of the shearer 121 as samples, semantic vectors corresponding to the different definitions described in table 2 are generated using the existing word2vec model (e.g., cbow model). As shown in table 3 below:

TABLE 3 Table 3

Thus, referring to FIG. 7A, the reordering module 232 may determine semantic vectors for various device state information and determine semantic vectors corresponding to the state "staff apply operation to control panel" according to Table 3. Thereby obtaining a first code sequenceqe ₀ ~qe _u 。

Further, referring to fig. 7B, the reordering module 232 may also arrange a plurality of target image frames of which the employee behavior category is behavior category 2 (manipulating the shearer control panel) acquired from the violation determination module 242 in time series. And obtains equipment status information associated with the shearer 121 for a time period spanned by the plurality of target image frames. For example, a predetermined time before the time period, a predetermined time after the time period, and the equipment state information of the shearer 1 acquired during the time period are acquired. And the reordering module 232 sorts the semantic vectors of the acquired device state information and the semantic vectors corresponding to the target image frames according to time to generate a first code sequence qe ₀ ~qe _u . Considering that the frequency at which the operator operates the control panel of the shearer 121 is much lower than the sampling frequency of the shearer 121 for the equipment status, the reordering module 232 may properly reduce the sampling frame rate for the target image frames during the reordering process. Thus, the first code sequence shown in fig. 7B reflects not only the timing correlation between the device state information and the state "worker applies an operation to the control panel", but also the timing correlation between the different time states "worker applies an operation to the control panel".

Then, referring to fig. 8A and 8B, after the reordering module 232 constructs the first encoded sequence, the second behavior recognition module 233 of the behavior recognition layer 230 uses a preset sequence processing model according to the first code sequenceA code sequence for generating a corresponding second code sequenceqs ₀ ~qs _v . Wherein the second code sequence is also composed of a plurality of semantic vectorsqs ₀ ~qs _v The composition is formed. Wherein each semantic vector of the second code sequence corresponds to a control operation of the control panel of the shearer 121 by a worker.

Wherein the following Table 4 shows semantic vectors corresponding to control operations of a worker

TABLE 4 Table 4

The semantic vectors shown in table 4 may be generated with reference to a method similar to the semantic vectors shown in table 3, and will not be described again here. It is also contemplated that the frequency at which the operator operates the control panel is much lower than the sampling frequency at which the shearer 121 collects equipment status information. Thus, table 4 finally sets a semantic vector corresponding to "no operation" to be filled.

Referring also to fig. 9A, the sequence processing model is, for example, an encoder-and decoder-based processing model. Referring to FIG. 9A, the encoder of the sequence processing model is based on the input first encoded sequenceqe ₀ ~qe _u Generating intermediate semantic featuresC ₁ ~C _v . Wherein the intermediate semantic featuresC ₁ ~C _v Respectively with the second coding sequenceqs ₀ ~qs _v Corresponding to the above. Wherein the encoder may use an encoder known in the art, and will not be described in detail herein.

The decoder may then generate a second encoded sequence according to the following formulaqs ₀ ~qs _v 。

；

Wherein the function isfAs a function of the decoder, it is possible to use decoder functions known in the art, which are not hereAnd are further described.

According to the technical scheme, a sequence processing model frequently used in natural language processing can be utilized, and a second coding sequence representing the control operation of a worker on a control surface case can be generated according to the state information of the representation equipment and the first coding sequence representing the state that the worker applies the operation to the control panel. In this way, the computing device can accurately recognize the control operation of the worker on the control panel of the shearer 121 through the second behavior recognition module 233. The second behavior recognition module 233 then communicates the recognized control operation to the violation determination module 242 of the violation statistics layer 240. The violation determination module 242 may thereby determine whether the control operation of the staff member is a violation operation based on the prescribed safety operation procedure.

Although the description has been made above taking the coal cutter 121 as an example, the above scheme may be referred to for the operation of the control panels of the remaining coal mine production apparatuses (e.g., the coal caving machine 122 and the emulsion pump 123).

Therefore, the technical scheme of the application utilizes a sequence processing model based on natural language processing, and can predict and identify the control operation of the staff on the control panel based on the equipment state information of the coal mine production equipment, thereby accurately identifying the control operation of the staff on the control panel.

Referring also to FIG. 9B, as an example of a sequence processing model, the sequence processing model may be an LSTM based sequence2sequence model.

Optionally, the operation of counting the illegal operations of the staff according to the determination result includes: and evaluating the working state of the staff according to the number of illegal operations of the staff in a preset period.

In particular, referring to fig. 1 and 2, the computing device 200 may count the number of violations of individual employees of the coal mine every predetermined period (e.g., each month, quarter, or year) based on determining the violations of the staff. Thus, the working status of the staff at each month, each quarter or each year can be evaluated. For example, if the number of times a certain worker operates illicitly within a certain month exceeds a predetermined threshold, this is an indication that the worker's work status is not good, and so on. According to the technical scheme of the application, the actual working state of the staff can be estimated by counting the illegal operation condition of the staff on the coal mine production equipment. Thereby being beneficial to the compliance operation of coal mine production equipment by monitoring staff in coal mine units.

Further in accordance with a second aspect of the present embodiment, a coal mine employee violation statistics method is provided, implemented by a computing device 200 shown in fig. 1. The method comprises the following steps:

s102: acquiring an image sequence of a site of coal mine production equipment;

s104: detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence;

s106: judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and

s108: and counting the illegal operation of the staff according to the judgment result.

Optionally, detecting in the image sequence an operation of a target image frame containing a worker operating the respective coal mine production apparatus, comprising: inputting the image sequence into a first artificial action recognition model, wherein the first artificial action recognition model is an image classification model based on a neural network, and the classification of the image classification model corresponds to different action classes related to corresponding coal mine production equipment; determining staff behavior categories corresponding to each image frame in the image sequence through a first staff behavior recognition model; and detecting the target image frames from the image sequence according to the employee behavior category.

Optionally, in the case that it is determined that the worker does not perform the illegal operation on the coal mine production equipment according to the identity information of the worker, and the control panel for controlling the coal mine production equipment is selected according to the employee behavior category corresponding to the image frame, it is determined whether the worker performs the illegal operation on the coal mine production equipment according to the target image frame, and the method further includes: acquiring a plurality of pieces of equipment state information associated with the target image frame according to the time stamp of the target image frame; determining control operation of staff in the target image frame on a control panel according to staff behavior category and equipment state information, wherein the equipment state information indicates equipment state of coal mine production equipment corresponding to the target image frame; and determining whether the operation of the control panel of the coal mine production equipment operated by the staff is illegal operation or not according to the determined control operation.

Optionally, determining a control operation of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information associated with the target image frame includes: determining semantic vectors corresponding to the employee behavior categories and the equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first coding sequence related to the equipment state information and the employee behavior categories; generating a second coding sequence corresponding to the first coding sequence by using a preset sequence processing model based on an encoder and a decoder, wherein semantic vectors contained in the second coding sequence are respectively related to control operations aiming at a control panel; extracting a target semantic vector corresponding to the employee behavior category from the second code sequence; and determining the control operation of the staff in the control panel in the target image frame according to the extracted target semantic vector.

Further, according to a third aspect of the present embodiment, there is provided a storage medium. The storage medium includes a stored program, wherein the method of any one of the above is performed by a processor when the program is run.

The technical scheme of the embodiment utilizes the image acquisition equipment to acquire the image sequence of each coal mine production equipment site. And detecting a target image frame from the image sequence containing a worker operating the coal mine production equipment. And then judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the worker according to a judgment result. Therefore, according to the technical scheme of the application, the operation condition of workers on the coal mine production equipment can be effectively counted, and thus, the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safety production condition of coal mine staff. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment, and cannot effectively monitor illegal operation staff in coal mine production is solved.

It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.

From the description of the above embodiments, it will be clear to a person skilled in the art that the method according to the above embodiments may be implemented by means of software plus the necessary general hardware platform, but of course also by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the method according to the embodiments of the present invention.

Example 2

Fig. 10 shows a coal mine employee violation statistics device 1000 according to the present embodiment, the device 1000 corresponding to the method according to the second aspect of embodiment 1. Referring to fig. 10, the apparatus 1000 includes: an image sequence acquisition module 1010 for acquiring an image sequence of a site of a coal mine production facility; a target frame detection module 1020 for detecting a target image frame containing a worker operating the coal mine production facility in the image sequence; the illegal operation determination module 1030 is configured to determine whether a worker performs illegal operation on the coal mine production equipment according to the target image frame; and the illegal operation statistics module 1040 is configured to perform statistics on illegal operations of the staff according to the determination result.

Optionally, the target frame detection module includes: an input sub-module for inputting the image sequence into a first artificial action recognition model, wherein the first artificial action recognition model is an image classification model based on a neural network, and the classification of the image classification model corresponds to different action classes related to corresponding coal mine production equipment; the behavior recognition sub-module is used for determining staff behavior categories corresponding to each image frame in the image sequence through the first staff behavior recognition model; and a detection sub-module for detecting a target image frame from the image sequence according to the employee behavior category.

Optionally, the illegal operation determination module includes: the identity recognition sub-module is used for determining the identity information of the staff according to the target image frame; and the illegal operation judging submodule is used for judging whether the worker performs illegal operation on the coal mine production equipment according to the identity information of the worker.

Optionally, the violation operation judgment submodule further includes: an apparatus information acquisition unit configured to acquire a plurality of apparatus state information associated with a target image frame based on a time stamp of the target image frame; a control operation identification unit for determining control operation of staff in the target image frame on the control panel according to staff behavior category and equipment state information, wherein the equipment state information indicates equipment state of coal mine production equipment corresponding to the target image frame; and the illegal operation judging unit is used for determining whether the operation of controlling the control panel of the coal mine production equipment by the staff is illegal operation or not according to the determined control operation.

Optionally, the control operation identifying unit includes: the first code sequence generation subunit is used for determining semantic vectors corresponding to the employee behavior types and the equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first code sequence related to the equipment state information and the employee behavior types; a second coding sequence generating subunit, configured to generate a second coding sequence corresponding to the first coding sequence by using a preset sequence processing model based on an encoder and a decoder, where semantic vectors included in the second coding sequence are related to control operations for a control panel, respectively; a target semantic vector extraction subunit extracting a target semantic vector corresponding to the employee behavior category from the second code sequence; and the illegal operation judging subunit is used for determining the control operation of the staff in the control panel in the target image frame according to the extracted target semantic vector.

Optionally, the violation operation statistics module includes: and the illegal operation statistics sub-module is used for evaluating the working state of the staff according to the number of illegal operations of the staff in a preset period.

The technical scheme of the embodiment obtains the image sequence of each coal mine production equipment site. And detecting a target image frame from the image sequence containing a worker operating the coal mine production equipment. And then judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame, and counting the illegal operation of the worker according to a judgment result. Therefore, according to the technical scheme of the application, the operation condition of workers on the coal mine production equipment can be effectively counted, and thus, the illegal operation of the workers in the coal mine production can be effectively monitored. And further effectively evaluate the safety production condition of coal mine staff. Therefore, the technical problem that the existing monitoring technology in the prior art cannot effectively count the illegal operation conditions of production equipment, and cannot effectively monitor illegal operation staff in coal mine production is solved.

Example 3

Fig. 11 shows a coal mine employee violation statistics device 1100 according to the present embodiment, the device 1100 corresponding to the method according to the second aspect of embodiment 1. Referring to fig. 11, the apparatus 1100 includes: a processor 1110; and a memory 1120, coupled to the processor 1110, for providing instructions to the processor 1110 for processing the following processing steps: acquiring an image sequence of a site of coal mine production equipment; detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence; judging whether a worker performs illegal operation on coal mine production equipment according to the target image frame; and counting the illegal operations of the staff according to the judgment result.

The foregoing embodiment numbers of the present application are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.

In the foregoing embodiments of the present application, the descriptions of the embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed technology may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of the units, such as the division of the units, is merely a logical function division, and may be implemented in another manner, for example, multiple units or components may be combined or may be integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.

The foregoing is merely a preferred embodiment of the present invention and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.

Claims

1. A coal mine employee violation statistics system, comprising:

the image acquisition equipment is arranged on the site of the coal mine production equipment; and

a computing device communicatively connected to the image acquisition device and configured to:

acquiring an image sequence of the scene of the corresponding coal mine production equipment acquired by the image acquisition equipment;

detecting in the image sequence a target image frame containing a worker operating the respective coal mine production facility;

judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame; and

counting the illegal operations of the staff according to the judgment result;

detecting in the image sequence an operation comprising a target image frame of a worker operating the respective coal mine production apparatus, comprising:

Inputting the image sequence into an employee behavior recognition model, wherein the employee behavior recognition model is an image classification model based on a neural network, and the categories of the image classification model correspond to different behavior categories related to corresponding coal mine production equipment;

determining staff behavior categories corresponding to each image frame in the image sequence through the staff behavior recognition model; and

detecting the target image frame from the image sequence according to the employee behavior category;

determining whether the worker performs illegal operation on the coal mine production equipment according to the target image frame comprises the following steps:

determining the identity information of the staff according to the target image frame; and

judging whether the staff performs illegal operation on the coal mine production equipment according to the identity information of the staff;

under the condition that the staff does not perform illegal operation on the coal mine production equipment according to the identity information of the staff, and the staff behavior category corresponding to the target image frame is a control panel for controlling the coal mine production equipment, whether the staff performs illegal operation on the coal mine production equipment or not is judged according to the target image frame, and the method further comprises the following steps:

Acquiring a plurality of pieces of equipment state information associated with the target image frame according to the time stamp of the target image frame;

determining control operation of staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the equipment state information indicates equipment state of coal mine production equipment corresponding to the target image frame; and

determining whether the operation of the control panel of the coal mine production equipment operated by the staff is illegal operation or not according to the determined control operation;

determining control operations of the staff in the target image frame on the control panel according to the staff behavior category and the equipment state information, wherein the control operations comprise the following steps:

determining semantic vectors corresponding to the employee behavior categories and the equipment state information, and arranging the determined semantic vectors according to a time sequence to form a first coding sequence related to the equipment state information and the employee behavior categories;

generating a second coding sequence corresponding to the first coding sequence by using a preset sequence processing model based on an encoder and a decoder, wherein semantic vectors contained in the second coding sequence are respectively related to control operation aiming at the control panel;

Extracting a target semantic vector from the second coding sequence, wherein the target semantic vector corresponds to a control operation of a staff member in the target image frame on the control panel; and

and determining the control operation of the staff in the target image frame on the control panel according to the extracted target semantic vector.

2. The system of claim 1, wherein the act of counting the staff's offending operations based on the determination comprises: and evaluating the working state of the staff according to the number of illegal operations of the staff in a preset period.

3. A coal mine employee violation statistical method, comprising:

acquiring an image sequence of a site of coal mine production equipment;

detecting in the image sequence a target image frame containing a worker operating the coal mine production apparatus;

counting the illegal operations of the staff according to the judgment result;

4. A storage medium comprising a stored program, wherein the method of claim 3 is performed by a processor when the program is run.

5. A colliery staff violation statistics device, characterized in that includes:

the image sequence acquisition module is used for acquiring an image sequence of a site of coal mine production equipment;

a target frame detection module for detecting a target image frame containing a worker operating the coal mine production equipment in the image sequence;

the illegal operation judging module is used for judging whether the worker performs illegal operation on the coal mine production equipment according to the target image frame; and

the illegal operation statistics module is used for counting the illegal operation of the staff according to the judgment result;

6. A colliery staff violation statistics device, characterized in that includes:

a processor; and

a memory, coupled to the processor, for providing instructions to the processor to process the following processing steps:

acquiring an image sequence of a site of coal mine production equipment;

counting the illegal operations of the staff according to the judgment result;