CN116092199B - Employee working state identification method and identification system - Google Patents
Employee working state identification method and identification system Download PDFInfo
- Publication number
- CN116092199B CN116092199B CN202310377644.1A CN202310377644A CN116092199B CN 116092199 B CN116092199 B CN 116092199B CN 202310377644 A CN202310377644 A CN 202310377644A CN 116092199 B CN116092199 B CN 116092199B
- Authority
- CN
- China
- Prior art keywords
- model
- staff
- employee
- image
- identification
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 31
- 230000006399 behavior Effects 0.000 claims abstract description 72
- 238000012549 training Methods 0.000 claims abstract description 26
- 230000006835 compression Effects 0.000 claims abstract description 11
- 238000007906 compression Methods 0.000 claims abstract description 11
- 238000013138 pruning Methods 0.000 claims abstract description 10
- 238000001514 detection method Methods 0.000 claims description 62
- 239000013598 vector Substances 0.000 claims description 30
- 230000009471 action Effects 0.000 claims description 12
- 238000012360 testing method Methods 0.000 claims description 12
- 238000012545 processing Methods 0.000 claims description 8
- 238000006243 chemical reaction Methods 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000012937 correction Methods 0.000 claims description 6
- 230000036544 posture Effects 0.000 claims description 5
- 230000008569 process Effects 0.000 claims description 5
- 230000001133 acceleration Effects 0.000 claims description 4
- 238000002372 labelling Methods 0.000 claims description 4
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 claims description 3
- 230000006837 decompression Effects 0.000 claims description 3
- 230000009466 transformation Effects 0.000 claims description 3
- 238000013519 translation Methods 0.000 claims description 3
- 238000007726 management method Methods 0.000 description 5
- 238000004364 calculation method Methods 0.000 description 3
- 210000000544 articulatio talocruralis Anatomy 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 210000002310 elbow joint Anatomy 0.000 description 2
- 210000004394 hip joint Anatomy 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 210000000629 knee joint Anatomy 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 210000000323 shoulder joint Anatomy 0.000 description 2
- 210000003857 wrist joint Anatomy 0.000 description 2
- 239000008186 active pharmaceutical agent Substances 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000004556 brain Anatomy 0.000 description 1
- 210000004027 cell Anatomy 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 238000013139 quantization Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/70—Labelling scene content, e.g. deriving syntactic or semantic representations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Human Computer Interaction (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Computational Linguistics (AREA)
- Social Psychology (AREA)
- Psychiatry (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a staff working state identification method and an employee working state identification system, which belong to the technical field of machine identification and comprise the steps of acquiring an image of a region to be detected; establishing an employee identification model and a behavior identification model, and performing model training; performing model pruning and quantitative compression on the trained employee identification model and behavior identification model; performing image recognition by using an employee recognition model, and performing identity recognition on employees in the image by using the employee recognition model; after the identification of the staff is completed, the behaviors of the staff in the image are identified, and whether the staff has illegal behaviors is determined; and uploading employee violation related data information to a cloud server by utilizing an edge computer and alarming. The invention can rapidly and continuously judge the states of the staff, effectively avoid subjectivity of manual judgment and make the staff check more fair and fair.
Description
Technical Field
The invention relates to the technical field of machine identification, in particular to an employee working state identification method and an employee working state identification system.
Background
In some special workplaces, staff is required to be kept attentive at all times and stay on duty for a long time, and the staff cannot keep an ideal working state for a long time, so that working efficiency is seriously affected.
In daily practice, the inventor finds that the prior technical scheme has the following problems:
at present, manual supervision is adopted for compliance inspection of staff during work, for example, by checking and judging the illegal behaviors of service staff at a workplace through a manager, mainly the manager performs manual judgment, the manager makes good evaluation standards in advance, and then whether each staff has the illegal behaviors at the workplace is subjectively judged through daily observation, blind visit or assault inspection. However, the method of performing the identification on site through the manager has defects and shortcomings such as strong subjectivity, strong sampling property and weak timeliness in manual identification judgment, and meanwhile, the manager has high labor cost for spot check, meanwhile, the method has no persistence due to the spot check mode, and the situations of collusion and falsification can exist, so that the effect of manual identification is limited.
In view of the foregoing, it is necessary to provide a new solution to the above-mentioned problems.
Disclosure of Invention
In order to solve the technical problems, the employee working state identification method and the employee working state identification system can rapidly and continuously judge the states of the employees, meanwhile, subjectivity of manual judgment is effectively avoided, and staff assessment is fairer and more fair.
A staff work state identification method comprises the following steps:
acquiring an image of a region to be detected by using an image acquisition module;
establishing an employee identification model and a behavior identification model, and performing model training;
performing model pruning and quantitative compression on the trained employee identification model and behavior identification model;
carrying out image recognition by using an employee recognition model, confirming whether an employee is contained in the image, and carrying out identity recognition on the employee in the image by using the employee recognition model when the employee is contained in the image;
after the identification of the staff is completed, the behavior of the staff in the image is identified by using a behavior identification model, and the behavior of the staff in the image is judged according to a judgment standard to determine whether the staff has illegal behaviors or not;
and uploading employee violation related data information to a cloud server by utilizing an edge computer and alarming.
Preferably, in performing image recognition by using an employee recognition model, determining whether an employee is included in the image, and in recognizing an employee identity in the image when the employee is included in the image, the recognizing the employee identity in the image includes:
performing face detection by using the deployed yolov7 target detection model;
calibrating the detected face image;
and sending the calibrated face image into a FaceNet target detection model to carry out employee identification.
Preferably, the performing face detection by using the deployed yolov7 target detection model includes:
after the yolov7 target detection model detects a human face, the confidence coefficient of the human face is obtained through network output after processing, and if the confidence coefficient is larger than a set threshold value, the human face is determined to appear in a picture; and storing the frame image, and cutting a face image according to the coordinates of the face in the image, which is output by the yolov7 target detection model.
Preferably, the calibrating the detected face image includes:
the intercepted face image is stretched into the size of 96 pixels by 96 pixels;
obtaining an affine transformation matrix according to 5 reference feature points of the face recognized in the face detection result and 5 reference feature point coordinates of the face standard;
carrying out rotation and translation correction on the face image through the matrix to obtain a corrected face;
the 5 reference feature points of the human face comprise left and right mouth corners, centers of eyes and a nose.
Preferably, an employee identification model and a behavior identification model are established, and in model training, the model training includes:
collecting a plurality of employee images in a video stream as a data set;
labeling the employee images;
dividing the marked data set into a training data set and a test data set;
training the model by using a training data set, testing the trained model by using a testing data set, and improving the number of training rounds and adjusting parameters during training until each index of the model on the testing set meets the actual working requirement.
Preferably, after model pruning and quantitative compression are performed on the trained employee identification model and the behavior identification model, the method further comprises:
converting the network model based on the PyTorch framework into an ONNX intermediate framework;
converting the network model in ONNX format into a network model format which can be identified by the edge computing chip by using a network model conversion tool on the edge computing chip;
and deploying the converted model on an edge computing chip.
Preferably, after the staff identification is completed, the staff behavior in the image is identified by using a staff behavior identification model, the staff behavior in the image is judged according to a judgment standard, and whether the staff has illegal behaviors or not is determined, wherein the staff behavior identification in the image comprises mobile phone playing behavior identification, sleeping behavior identification, staff on duty judgment and staff alarming behavior identification.
According to another aspect of the present application, there is also provided an employee working status recognition system, including: the device comprises a high-definition camera, an edge computing chip, a matched network port and a power line; the edge computing chip is integrated in the high-definition camera shell; the high-definition camera is connected with the edge computing chip through an MIPI interface; and the power line supplies power for the high-definition camera and the edge computing chip.
Preferably, the edge computing chip is integrated with a hardware decoding module, a pixel format conversion module, an LDC module, a fixed angle rotation module, an arbitrary angle rotation module, a fisheye correction module, a compression decompression module, a mosaic filling module and an image processing hardware acceleration module.
Compared with the prior art, the application has the following beneficial effects:
1. the invention can automatically identify the nonstandard behaviors of staff in work, overcomes the defects of the existing staff identification by a manager on site, improves the accuracy, normalization and accuracy of staff working state identification, and increases the fairness of staff assessment.
2. The method and the device rely on the characteristic of strong real-time performance of edge calculation, can rapidly and efficiently judge the states of staff, and are high in recognition efficiency.
3. According to the cloud side management method, the cloud side combination advantage is utilized, unified management of the working states of the staff can be achieved through the cloud side server, a manager can obtain detailed working state information of the staff including illegal behaviors and illegal time through the cloud side, and management is convenient.
4. According to the staff working state identification system, the edge computing chip is positioned in the camera shell, the integration degree is high, the power consumption is low, and low-cost large-scale rapid deployment can be realized.
Drawings
Some specific embodiments of the invention will be described in detail hereinafter by way of example and not by way of limitation with reference to the accompanying drawings. The same reference numbers will be used throughout the drawings to refer to the same or like parts or portions. It will be appreciated by those skilled in the art that the drawings are not necessarily drawn to scale. In the accompanying drawings:
FIG. 1 is a schematic overall flow chart of the present invention;
FIG. 2 is a schematic diagram of an employee work status identification system according to the present invention.
Wherein the above figures include the following reference numerals:
1. high definition digtal camera, 2, edge calculation chip, 3, supporting net gape, 4, power cord.
Detailed Description
For the purposes, technical solutions and advantages of the present application, the technical solutions of the present application will be clearly and completely described below with reference to specific embodiments of the present application and corresponding drawings. It will be apparent that the described embodiments are only some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art without undue burden from the present disclosure, are within the scope of the present disclosure.
As shown in fig. 1, a staff work state identification method includes the following steps:
step S1, an image acquisition module is utilized to acquire an image of a region to be detected.
The image acquisition module adopts a high-definition camera.
And S2, building an employee identification model and a behavior identification model, and performing model training.
The training and deployment flows of the employee identification model and the behavior identification model are the same.
The staff identification model adopts a yolov7 target detection model and a Facenet target detection model to carry out staff identification, and targets need to be selected for target picture frames in images to finish labeling before model training. The Yolov7 target detection model uses a re-parameterized convolution, the role of re-parameterization: the network is accelerated under the condition of ensuring the performance of the model, and the convolution +BN layer and different convolutions are mainly fused and combined into a convolution module.
The behavior recognition model is in the form of human body key point detection, and in the embodiment, a G-RMI model is selected.
In the early stage, 20000 employee images in the video stream are collected as a data set, and then the employee images are marked.
Dividing the marked data set into a training data set and a test data set, training the model by using the training data set, simultaneously testing the trained model by using the test data set, and if the technical indexes of the model do not reach the standard, increasing the number of training rounds and adjusting the parameters during training until each index of the trained model on the test set meets the actual working requirements.
And marking 17 characteristic points of the human body selected by the image center according to the marking rule, wherein the characteristic points comprise nasal tip, left eye, right eye, left ear, right ear, left shoulder joint, right shoulder joint, left elbow joint, right elbow joint, left wrist joint, right wrist joint, left hip joint, right hip joint, left knee joint, right knee joint, left ankle joint and right ankle joint. Wherein, the left and right are the left and right of the index annotating person.
Wherein, the labeling rule is: distinguishing left and right, distinguishing visible and shielding points based on the marked person, setting the visible points to be yellow, the self shielding to be red and the shielding to be blue. In addition, the visible point needs to be marked at the center of the sphere of each joint, the shielding point needs to be approximately in a brain-supplementing position, and the shielding point is cut off from the picture, so that the brain needs to be outside the picture.
In addition, in order to expand the number of data sets, enhancement processing is required on the data sets, including image blending using mixup, and gray processing is used on the images to simulate recognition under different scenes of light.
And S3, performing model pruning and quantitative compression on the trained employee identification model and the trained behavior identification model.
For model pruning, firstly, we need to measure the weight ratio of neurons in the original model, remove partial neurons with the lowest weight, fine tune the network by adopting feature extraction and linear classification, then train the model again to obtain the weight after pruning, compare with the accuracy of the original model, and finish the pruning from this round. In the aspect of model design, the idea of light weight is adopted, for example, a light weight convolution mode such as depth separable convolution, grouping convolution and the like is adopted, the calculated amount of a convolution process is reduced, in addition, global pooling is adopted to replace a full-connection layer, and the channel dimension reduction of the features is realized by using 1 x 1 convolution, so that the calculated amount of the model is greatly reduced. Regarding the aspect of model quantization, a half-precision mode of converting 32-bit floating point numbers into 16-bit floating point numbers is adopted, so that the calculation speed is improved, and meanwhile, the size of the model is reduced.
By compression, the volume of the model is reduced by more than 10 times, so that the employee identification model and the behavior identification model can be rapidly operated on low-calculation-force edge computing equipment.
And S4, carrying out image recognition by using an employee recognition model, confirming whether the image contains employees, and carrying out identity recognition on the employees in the image by using the employee recognition model when the image contains the employee time. The method specifically comprises the following steps:
and S41, carrying out face detection by using the deployed yolov7 target detection model. After the yolov7 target detection model detects a human face, the confidence coefficient of the human face can be obtained through network output after processing, if the confidence coefficient is larger than a set threshold value, the human face appears in a picture is determined, the frame image is stored, and the human face image is cut according to the coordinates of the human face in the image, which is output by the yolov7 target detection model.
And step S42, calibrating the detected face image. The method specifically comprises the following steps: and stretching the intercepted face image into 96 pixels by 96 pixels, solving an affine transformation matrix according to 5 reference feature points of the face recognized in the face detection result and 5 reference feature point coordinates of the face standard, and carrying out rotary translation correction on the face image through the matrix to obtain the corrected face. The 5 reference feature points of the human face comprise left and right mouth corners, centers of eyes and a nose.
And step S43, sending the calibrated face image into a FaceNet target detection model for staff identification.
Specifically, the FaceNet object detection model can directly map the face image to euclidean space, and the spatial distance represents the similarity of the face image. The tasks of face recognition, verification, clustering and the like can be easily completed only by generating the mapping space training. The output of the FaceNet object detection model is the feature ID of the input face. An administrator can upload the front image of the staff through logging in the system management interface, and the FaceNet target detection model automatically generates the uploaded staff image into the corresponding feature ID and stores the corresponding feature ID in the staff database, so that the FaceNet target detection model obtains the identity of the staff in the picture by matching the detected face feature ID with the feature ID in the staff database.
And S5, after the identification of the staff is completed, identifying the behaviors of the staff in the image by using a behavior identification model, judging the behaviors of the staff in the image according to a judgment standard, and determining whether the staff has illegal behaviors or not.
Identifying the employee's behavior in the image includes cell phone playing behavior identification, sleeping behavior identification, employee on duty judgment, and employee alarm behavior identification.
In the mobile phone playing behavior recognition, the method is realized by combining two models, namely a human body key point detection network G-RMI and a target detection network yolov 7.
Firstly, actions of staff playing mobile phones are required to be collected, a plurality of people are collected through a high-definition camera before model deployment, images of various angles are obtained through the plurality of people using the mobile phones, then human body key points in the images are identified through a human body key point detection network G-RMI model, meanwhile, the mobile phones are identified through a target detection network yolov7 target detection model, position information of the mobile phones in the images is identified through the yolov7 target detection model, and feature vectors are formed by combining the position information with output results of the human body key point detection network and are stored in a database.
After the mobile phone playing action feature vector is collected, the mobile phone playing action can be identified in real time, the identification flow is similar to the collection flow, the output of the human body key point detection network is combined with the output of the target detection network, so that the real-time action feature of the staff is generated, the staff feature is matched with all mobile phone playing feature vectors in the database by the system, and if the matching degree of the system is greater than a set threshold value, the staff can be judged to use the mobile phone.
In sleeping behavior recognition, the method is mainly realized by a human body key point detection network G-RMI model.
The feature vector of the sleeping behavior is required to be collected firstly, and the collection process is as follows: firstly, a camera collects images of a tester sleeping in various angles and various postures, then the images are sent into a human body key point detection network to generate corresponding feature vectors, and finally the feature vectors are stored in a database.
After the action collection is finished, the key points of the staff can be detected in real time, the human body key point detection network detects the human body key points of the staff in the picture in real time and generates corresponding feature vectors, and if the matching degree of the feature vectors and the feature vectors in the database exceeds a threshold value, the staff is judged to have sleeping behaviors.
In the on-duty judgment of staff, the staff is realized mainly by means of a yolov7 target detection model.
The judgment firstly needs to upload the working position of each employee by an administrator, the administrator logs in the cloud management interface, the real-time picture of each device can be seen in the interface, at the moment, the administrator can select the working position of each employee in the picture by clicking the picture position function, and meanwhile, the administrator needs to input the name of the person corresponding to the position. The system stores the coordinates of the center point of the drawn frame and the position coordinates of the edge points selected by the frame, and binds the coordinate information with the personnel ID in the personnel identification library.
Then the algorithm can identify the personnel on duty, the algorithm obtains the real-time picture in the camera and detects the personnel id and the personnel position coordinate in the picture through the personnel identification function module, and then the algorithm judges whether the personnel is on duty or not through calculating the coincidence degree of the personnel position coordinate and the corresponding station position coordinate of the personnel.
In staff alarm behavior recognition, the staff alarm behavior recognition is realized mainly by means of a human body key point detection network G-RMI model.
The similar collection flow of mobile phone playing behavior recognition firstly needs to collect feature vectors of staff alarm behaviors, and the collection process is as follows: firstly, a plurality of images of a tester which are alarmed in various angles and various postures are collected by a camera, then the images are sent into a human body key point detection network to generate corresponding feature vectors, and finally the feature vectors are stored in a database.
Specifically, after the action collection is finished, the key points of the staff can be detected in real time, the human body key points of the staff in the picture are detected in real time by the human body key point detection network, corresponding feature vectors are generated, and if the matching degree of the feature vectors and the feature vectors in the database exceeds a threshold value, the staff is judged to have alarm behaviors.
And S6, uploading relevant data information of employee violation behaviors to a cloud server by utilizing an edge computer and alarming.
In addition, after model pruning and quantitative compression are performed on the trained employee identification model and behavior identification model in step S3, the method further includes:
and converting the network model based on the PyTorch framework into an ONNX intermediate framework, converting the network model in the ONNX format into a TensorRT format model by using a network model conversion tool on the edge computing chip, and deploying the converted model on the edge computing chip. TensorRT supports different data formats, is an acceleration package made by Injeida for a home platform, can compress, optimize and run-time deploy a network, and has no overhead of a framework. The TensorRT improves the delay, throughput and efficiency of the network by optimizing selection of combinations layers and kernel, and performing normalization and conversion to an optimal matrix math method according to specified accuracy. The computing chip TX2 used in the present embodiment is an AI computing platform developed by the inflight corporation, and is based on the GPU of the NVIDIA Pascal architecture: fully support all modern graphics APIs, unify shaders, and support GPU computing.
As shown in fig. 2, an employee work status recognition system includes: the high-definition camera 1, the edge computing chip 2, the matched network port 3 and the power line 4, and the edge computing chip 2 is integrated inside the shell of the high-definition camera 1. The high-definition camera 1 is connected with the edge computing chip 2 through an MIPI interface. The power line 4 supplies power to the high-definition camera 1 and the edge computing chip 2.
Preferably, the edge computing chip 2 is integrated with a hardware decoding module, a pixel format conversion module, an LDC module, a fixed angle rotation module, an arbitrary angle rotation module, a fisheye correction module, a compression decompression module, a mosaic filling module and an image processing hardware acceleration module.
Spatially relative terms, such as "above … …," "above … …," "upper surface at … …," "above," and the like, may be used herein for ease of description to describe one device or feature's spatial location relative to another device or feature as illustrated in the figures. It will be understood that the spatially relative terms are intended to encompass different orientations in use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, elements described as "above" or "over" other devices or structures would then be oriented "below" or "beneath" the other devices or structures. Thus, the exemplary term "above … …" may include both orientations of "above … …" and "below … …". The device may also be positioned in other different ways (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments in accordance with the present application. As used herein, the singular is also intended to include the plural unless the context clearly indicates otherwise, and furthermore, it is to be understood that the terms "comprises" and/or "comprising" when used in this specification are taken to specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof.
It should be noted that the terms "first," "second," and the like in the description and claims of the present application and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that embodiments of the present application described herein may be implemented in sequences other than those illustrated or described herein.
The above description is only of the preferred embodiments of the present invention and is not intended to limit the present invention, but various modifications and variations can be made to the present invention by those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. The employee work status identification method is characterized by comprising the following steps:
acquiring an image of a region to be detected by using an image acquisition module;
establishing an employee identification model and a behavior identification model, and performing model training;
performing model pruning and quantitative compression on the trained employee identification model and behavior identification model;
carrying out image recognition by using an employee recognition model, confirming whether an employee is contained in the image, and carrying out identity recognition on the employee in the image by using the employee recognition model when the employee is contained in the image;
after the identification of the staff is completed, the behavior of the staff in the image is identified by using a behavior identification model, and the behavior of the staff in the image is judged according to a judgment standard to determine whether the staff has illegal behaviors or not;
uploading employee violation related data information to a cloud server by utilizing an edge computer and alarming;
after the identification of the staff is completed, the staff behavior in the image is identified by using a staff behavior identification model, the staff behavior in the image is judged according to a judgment standard, and whether the staff has illegal behaviors or not is determined, wherein the identification of the staff behavior in the image comprises mobile phone playing behavior identification, sleeping behavior identification, staff on duty judgment and staff alarm behavior identification;
in the mobile phone playing behavior recognition, the method is realized by combining two models of a human body key point detection network G-RMI and a target detection network yolov 7; firstly, acquiring actions of an employee to play mobile phones, acquiring images of a plurality of people in various postures and at various angles by using a plurality of mobile phones through a high-definition camera before model deployment, then identifying human key points in the images by using a human key point detection network G-RMI model, identifying the mobile phones by using a target detection network yolov7 target detection model, identifying the position information of the mobile phones in the images by using the yolov7 target detection model, and combining the position information with an output result of the human key point detection network to form a feature vector to be stored in a database;
after the mobile phone playing action feature vector is collected, the mobile phone playing action is identified in real time, the identification flow is similar to the collection flow, the real-time action feature of staff is generated by combining the output of the human body key point detection network with the output of the target detection network, the staff feature is matched with all mobile phone playing feature vectors in a database by a system, and if the system matching degree is greater than a set threshold, the staff can be judged to use the mobile phone;
in sleeping behavior recognition, the method is realized mainly by means of a human body key point detection network G-RMI model; firstly, feature vectors of sleeping behaviors need to be collected, and the collection process is as follows: firstly, collecting images of a tester sleeping in various angles and various postures by a camera, then sending the images into a human body key point detection network to generate corresponding feature vectors, and finally storing the feature vectors in a database;
after the action collection is finished, the key points of the staff can be detected in real time, the human body key points of the staff in the picture are detected in real time by a human body key point detection network, corresponding feature vectors are generated, and if the matching degree of the feature vectors and the feature vectors in the database exceeds a threshold value, the staff is judged to have sleeping behaviors;
in the on-duty judgment of staff, the staff is realized mainly by means of a yolov7 target detection model; the judgment firstly needs to upload the working position of each employee by an administrator, the administrator logs in a cloud management interface, a real-time picture of each device can be seen in the interface, at the moment, the administrator frames the working position of each employee in the picture by clicking a picture position function, and meanwhile, the administrator needs to input the name of the person corresponding to the position; the system stores the coordinates of the center point of the drawn frame and the position coordinates of each edge point selected by the frame, and binds the coordinate information with the personnel ID in the personnel identification library; then the algorithm can carry out on-duty recognition of the personnel, the algorithm obtains a real-time picture in the camera and detects personnel id and personnel position coordinates in the picture through the personnel recognition functional module, and then the algorithm judges whether the personnel is in an off-duty state or not through calculating the coincidence degree of the personnel position coordinates and the corresponding station position coordinates of the personnel;
in staff alarm behavior recognition, the staff alarm behavior recognition is realized mainly by means of a human body key point detection network G-RMI model; firstly, feature vectors of staff alarm behaviors need to be collected, and the collection process is as follows: firstly, collecting images of a plurality of testers in various angles and various postures by a camera, then sending the images into a human body key point detection network to generate corresponding feature vectors, and finally storing the feature vectors in a database; specifically, after the action collection is finished, the key points of the staff can be detected in real time, the human body key points of the staff in the picture are detected in real time by the human body key point detection network, corresponding feature vectors are generated, and if the matching degree of the feature vectors and the feature vectors in the database exceeds a threshold value, the staff is judged to have alarm behaviors.
2. An employee work status identification method as claimed in claim 1, wherein in performing image identification using the employee identification model, determining whether the image includes an employee, including a member man-hour, identifying an employee identity in the image, said identifying the employee identity in the image includes:
performing face detection by using the deployed yolov7 target detection model;
calibrating the detected face image;
and sending the calibrated face image into a FaceNet target detection model to carry out employee identification.
3. An employee performance status identification method as in claim 2 wherein said face detection by a deployed yolov7 target detection model comprises:
after the yolov7 target detection model detects a human face, the confidence coefficient of the human face is obtained through network output after processing, and if the confidence coefficient is larger than a set threshold value, the human face is determined to appear in a picture; and storing the frame image, and cutting a face image according to the coordinates of the face in the image, which is output by the yolov7 target detection model.
4. A method of identifying an employee status as defined in claim 2 wherein calibrating the detected face image comprises:
the intercepted face image is stretched into the size of 96 pixels by 96 pixels;
obtaining an affine transformation matrix according to 5 reference feature points of the face recognized in the face detection result and 5 reference feature point coordinates of the face standard;
carrying out rotation and translation correction on the face image through the matrix to obtain a corrected face;
the 5 reference feature points of the human face comprise left and right mouth corners, centers of eyes and a nose.
5. An employee performance recognition method as defined in claim 1, wherein an employee recognition model and a behavior recognition model are established and model training is performed, the model training comprising:
collecting a plurality of employee images in a video stream as a data set;
labeling the employee images;
dividing the marked data set into a training data set and a test data set;
training the model by using a training data set, testing the trained model by using a testing data set, and improving the number of training rounds and adjusting parameters during training until each index of the model on the testing set meets the actual working requirement.
6. An employee performance status identification method as in claim 1, further comprising, after model pruning and quantitative compression of the trained employee identification model and behavior identification model:
converting the network model based on the PyTorch framework into an ONNX intermediate framework;
converting the network model in ONNX format into a network model format which can be identified by the edge computing chip by using a network model conversion tool on the edge computing chip;
and deploying the converted model on an edge computing chip.
7. An employee work status identification system, adapted to a method of identifying an employee as claimed in any one of claims 1 to 6, comprising: the device comprises a high-definition camera, an edge computing chip, a matched network port and a power line; the edge computing chip is integrated in the high-definition camera shell; the high-definition camera is connected with the edge computing chip through an MIPI interface; the power line supplies power for the high-definition camera and the edge computing chip;
the edge computing chip is integrated with a hardware decoding module, a pixel format conversion module, an LDC module, a fixed angle rotation module, an arbitrary angle rotation module, a fisheye correction module, a compression decompression module, a mosaic filling module and an image processing hardware acceleration module.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310377644.1A CN116092199B (en) | 2023-04-11 | 2023-04-11 | Employee working state identification method and identification system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310377644.1A CN116092199B (en) | 2023-04-11 | 2023-04-11 | Employee working state identification method and identification system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116092199A CN116092199A (en) | 2023-05-09 |
| CN116092199B true CN116092199B (en) | 2023-07-14 |
Family
ID=86212380
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310377644.1A Active CN116092199B (en) | 2023-04-11 | 2023-04-11 | Employee working state identification method and identification system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116092199B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758295B (en) * | 2023-08-15 | 2024-06-04 | 摩尔线程智能科技(北京)有限责任公司 | Key point detection method and device, electronic equipment and storage medium |
| CN117787815B (en) * | 2024-02-27 | 2024-05-07 | 山东杰出人才发展集团有限公司 | Human resource outsourcing service system and method based on big data |
| CN118015663B (en) * | 2024-04-09 | 2024-07-02 | 浙江深象智能科技有限公司 | Staff identification method, device and equipment |
| CN118278822A (en) * | 2024-05-29 | 2024-07-02 | 湖南众东信息科技有限公司 | Working data acquisition method and system based on image data analysis and electronic equipment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614939A (en) * | 2018-12-13 | 2019-04-12 | 四川长虹电器股份有限公司 | " playing mobile phone " behavioral value recognition methods based on human body attitude estimation |
| CN115588165A (en) * | 2022-10-26 | 2023-01-10 | 国网重庆市电力公司建设分公司 | Tunnel worker safety helmet detection and face recognition method |
Family Cites Families (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110569711B (en) * | 2019-07-19 | 2022-07-15 | 沈阳工业大学 | Human body action oriented recognition method |
| CN113139403A (en) * | 2020-01-17 | 2021-07-20 | 顺丰科技有限公司 | Violation behavior identification method and device, computer equipment and storage medium |
| CN112183438B (en) * | 2020-10-13 | 2022-11-04 | 深圳龙岗智能视听研究院 | Image identification method for illegal behaviors based on small sample learning neural network |
| CN113269142A (en) * | 2021-06-18 | 2021-08-17 | 中电科大数据研究院有限公司 | Method for identifying sleeping behaviors of person on duty in field of inspection |
| CN113762115B (en) * | 2021-08-27 | 2024-03-15 | 国网浙江省电力有限公司 | Distribution network operator behavior detection method based on key point detection |
| CN115861940B (en) * | 2023-02-24 | 2023-04-28 | 珠海金智维信息科技有限公司 | Work scene behavior evaluation method and system based on human body tracking and recognition technology |
-
2023
- 2023-04-11 CN CN202310377644.1A patent/CN116092199B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109614939A (en) * | 2018-12-13 | 2019-04-12 | 四川长虹电器股份有限公司 | " playing mobile phone " behavioral value recognition methods based on human body attitude estimation |
| CN115588165A (en) * | 2022-10-26 | 2023-01-10 | 国网重庆市电力公司建设分公司 | Tunnel worker safety helmet detection and face recognition method |
Non-Patent Citations (3)
| Title |
|---|
| Action Recognition in Video by Covariance Matching of Silhouette Tunnels;Kai Guo et al.;《2009 XXII Brazilian Symposium on Computer Graphics and Image Processing》;第299-306页 * |
| Matching mixtures of curves for human action recognition;Michalis Vrigkas et al.;《Computer Vision and Image Understanding》;第27-40页 * |
| 基于机器视觉的高危企业生产过程智能监控;候景严;《中国优秀硕士学位论文全文数据库 信息科技辑》;I138-1464 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116092199A (en) | 2023-05-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN116092199B (en) | Employee working state identification method and identification system | |
| CN110502965B (en) | Construction safety helmet wearing monitoring method based on computer vision human body posture estimation | |
| CN105719188B (en) | The anti-method cheated of settlement of insurance claim and server are realized based on plurality of pictures uniformity | |
| CN109819208A (en) | A kind of dense population security monitoring management method based on artificial intelligence dynamic monitoring | |
| CN106295567A (en) | The localization method of a kind of key point and terminal | |
| US20220180534A1 (en) | Pedestrian tracking method, computing device, pedestrian tracking system and storage medium | |
| CN109002783A (en) | Rescue the human testing in environment and gesture recognition method | |
| CN110738154A (en) | pedestrian falling detection method based on human body posture estimation | |
| CN112131936A (en) | Inspection robot image identification method and inspection robot | |
| CN105022999A (en) | Man code company real-time acquisition system | |
| CN112016497A (en) | Single-view Taijiquan action analysis and assessment system based on artificial intelligence | |
| CN112183472A (en) | Method for detecting whether test field personnel wear work clothes or not based on improved RetinaNet | |
| CN110929687A (en) | Multi-user behavior recognition system based on key point detection and working method | |
| CN110478862A (en) | A kind of exercise guide system and its guidance method | |
| CN106056069A (en) | Unmanned aerial vehicle image analysis-based forest land resource asset evaluation method and evaluation system | |
| CN115082825A (en) | Video-based real-time human body falling detection and alarm method and device | |
| CN110287862A (en) | Anti- detection method of taking on the sly based on deep learning | |
| CN109902550A (en) | The recognition methods of pedestrian's attribute and device | |
| CN114049325A (en) | Construction method and application of lightweight face mask wearing detection model | |
| CN114049683A (en) | Post-healing rehabilitation auxiliary detection system, method and medium based on three-dimensional human skeleton model | |
| CN109145752A (en) | For assessing the method, apparatus, equipment and medium of object detection and track algorithm | |
| CN115273150A (en) | Novel identification method and system for wearing safety helmet based on human body posture estimation | |
| CN115035381A (en) | Lightweight target detection network of SN-YOLOv5 and crop picking detection method | |
| CN111382629B (en) | Footprint identification and information mining method and system based on convolutional neural network | |
| CN114937293A (en) | Agricultural service management method and system based on GIS |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20231228 Address after: Room 201, 97 Building, No.16 Xingyuan Road, Zhifu District, Yantai City, Shandong Province, 264001 Patentee after: Yinghui Digital Services (Yantai) Co.,Ltd. Address before: Room 522, Plant 2, No. 32, the Pearl River Road, Yantai Economic and Technological Development Zone, Shandong Province, 264000 Patentee before: Shandong Yishi Intelligent Technology Co.,Ltd. |