CN117975547A - Operator behavior recognition method and system - Google Patents
Operator behavior recognition method and system Download PDFInfo
- Publication number
- CN117975547A CN117975547A CN202211300049.XA CN202211300049A CN117975547A CN 117975547 A CN117975547 A CN 117975547A CN 202211300049 A CN202211300049 A CN 202211300049A CN 117975547 A CN117975547 A CN 117975547A
- Authority
- CN
- China
- Prior art keywords
- behavior
- action
- module
- information
- standard
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 30
- 230000006399 behavior Effects 0.000 claims abstract description 331
- 230000009471 action Effects 0.000 claims abstract description 156
- 238000001514 detection method Methods 0.000 claims abstract description 92
- 238000004519 manufacturing process Methods 0.000 claims abstract description 30
- 238000004458 analytical method Methods 0.000 claims description 31
- 238000012552 review Methods 0.000 claims description 14
- 230000002159 abnormal effect Effects 0.000 claims description 12
- 230000004927 fusion Effects 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 6
- 206010000117 Abnormal behaviour Diseases 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000004422 calculation algorithm Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000012806 monitoring device Methods 0.000 description 3
- 230000005856 abnormality Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012797 qualification Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- General Factory Administration (AREA)
Abstract
The invention relates to the technical field of intelligent vehicle manufacturing, in particular to a method and a system for identifying the behaviors of operators, wherein the method comprises the following steps: collecting operation behavior data of an operator, and establishing an operation behavior standard table, a behavior detection model and a target detection model; acquiring operation behavior video information of an operator in real time, segmenting the operation behavior video information into small frame number behavior video information, detecting an action behavior type through the behavior detection model, detecting a start state frame number and an end state frame number of the action behavior through the target detection model, combining the action behavior type with the start state frame number and the end state frame number of the action behavior, thereby obtaining an accurate behavior video frame interval, and combining the accurate behavior video frame intervals into the accurate operation behavior video information in sequence; judging whether the action of the operator is standard or not, the invention solves the problem of large error of the granularity of the identification time in the existing behavior detection scheme.
Description
Technical Field
The invention relates to the technical field of intelligent vehicle manufacturing, in particular to a method and a system for identifying behaviors of operators.
Background
The manufacturing industry is a main pillar of national economy, and as a target of the manufacturing industry, the intelligent manufacturing development of the automobile industry is always in the front. With the rapid development of artificial intelligence technology, the method is widely applied in the automobile industry, and the method brings great technical improvement and transformation to the automobile industry and improves the industrial chain efficiency in the fields of unmanned robots, product quality inspection and the like of automatic driving automobiles and automobile production workshops. However, in an automobile production workshop, there is still no effective intelligent supervision for the production behavior and action standard of operators, which results in outflow of unqualified products and negative influence on enterprise brands.
Disclosure of Invention
The invention provides an operator behavior recognition method and system for solving the defects and shortcomings of the prior art.
In order to achieve the above purpose, the technical scheme adopted by the invention is an operator behavior recognition method, which comprises the following steps:
S1: collecting operation behavior data of an operator, and establishing an operation behavior standard table, a behavior detection model and a target detection model;
S2: acquiring operation behavior video information of an operator in real time, segmenting the operation behavior video information into small frame number behavior video information, detecting action behavior types of the small frame number behavior video information through the behavior detection model, detecting initial state frame numbers and end state frame numbers of action behaviors of the small frame number behavior video information through the target detection model, combining the action behavior types with the initial state frame numbers and the end state frame numbers of the action behaviors to obtain accurate initial state frame numbers and end state frame numbers of the action behaviors, obtaining accurate behavior video frame intervals, and sequentially combining the accurate behavior video frame intervals into accurate operation behavior video information;
S3: and comparing the video information of the accurate operation behavior with standard behaviors on an operation behavior standard table to judge whether the actions of operators are standard or not.
Preferably, the step of combining the action behavior type with the start state frame number and the end state frame number of the action behavior in S2 includes:
S21, setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0, 1);
S22, comparing the sizes of W1 and W2, and if the value of W1 and W2 is larger than the value of W2 and C2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1 and C1+W 2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
Preferably, the step S3 includes the steps of:
s31, determining the model of a work piece of the operation through the target detection model, and calling standard behavior information corresponding to the operation behavior standard table according to the model of the work piece;
And S32, comparing the behavior information of the accurate operation behavior video information with the standard behavior information to judge whether the action behavior of the operator is standard or not.
Preferably, the behavior information includes a starting and ending time point, an operation sequence and a duration time of the action behavior.
Preferably, the method further comprises S4, if the starting and ending time points of the actions are inconsistent with the standard behavior information, generating warning information and sending the warning information to a manager for review and production line control operation;
If the operation sequence of the action behaviors is inconsistent with the standard behavior information, generating warning information, sending the warning information to a manager for review and production line control operation, and reminding the operator;
if the duration time length of the action behavior is inconsistent with the standard behavior information, displaying and storing the action behavior as an abnormal event through operating a display screen.
The invention also provides an operator behavior recognition system for realizing the operator behavior recognition method, which comprises an acquisition module, an analysis and judgment module, a display module and a warning module;
The acquisition module is used for acquiring operation behavior video information of an operator and sending the operation behavior video information to the analysis and judgment module and the display module;
The analysis judging module is used for analyzing and judging whether the action behaviors in the operation behavior video information are standard or not and sending an analysis result to the display module;
The display module is used for displaying action behaviors and sending action behavior abnormal information to the warning module;
The warning module generates warning information according to the action behavior abnormal information to prompt management personnel and operators.
Preferably, the analysis and judgment module comprises a splitting module, a behavior detection module, a target detection module, a fusion module, a comparison and judgment module and an analysis module;
the splitting module is used for splitting the operation behavior video information into small-frame behavior video information;
the behavior detection module is used for detecting the action behavior type of the small-frame number behavior video information;
The target detection module is used for detecting the initial state frame number and the end state frame number of the action behaviors of the small frame number behavior video information and judging the workpiece model of the operation;
The fusion module is used for combining the action type with the initial state frame number and the end state frame number of the action to obtain the accurate initial state frame number and the end state frame number of the action, so as to obtain an accurate action video frame interval, and combining the accurate action video frame intervals into accurate operation action video information in sequence;
The comparison judging module is used for comparing the accurate operation behavior video information with standard behaviors on an operation behavior standard table to judge whether the action behaviors of operators are standard or not;
The analysis module is used for analyzing the nonstandard type of the action behaviors of the operators and sending the nonstandard type to the display module, the display module sends out warning indication to the warning module, and the warning module sends out warning to the operators and the management staff.
Preferably, the fusion module works as follows: setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0 and 1); comparing the sizes of W1 and W2, and if the value of W1 is larger than the value of W2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1+W2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
Preferably, the comparison judging module determines a workpiece model of the behavior video information operation through the target detecting module, and invokes standard behavior information corresponding to the operation behavior standard table according to the workpiece model, and compares the behavior information of the accurate operation behavior video information with the standard behavior information to judge whether the action of an operator is standard or not.
Preferably, the analysis module analyzes the nonstandard type of the action behaviors of the operator, wherein the nonstandard type of the action behaviors comprises nonstandard starting and ending time points of actions, nonstandard operation sequences of the action behaviors and nonstandard duration time lengths of the action behaviors;
The starting and ending time point of the action is not standard, namely the starting and ending time point of the action is inconsistent with the standard behavior information, and if the starting and ending time point of the action is not standard, the warning module generates warning information and sends the warning information to a manager for review and production line control operation;
The operation sequence of the action behavior is inconsistent with the standard behavior information, and if the operation sequence of the action behavior is inconsistent, the warning module generates warning information and sends the warning information to a manager for review and production line control operation, and reminds the operator;
and if the duration of the action behavior is not standard, the warning module displays and stores the action behavior as an abnormal event through an operation display screen.
The invention has the beneficial effects that:
The invention provides an operator behavior recognition method and system, which are used for solving the problems of large recognition time granularity error and low calculation speed in the existing behavior detection scheme.
Drawings
FIG. 1 is a schematic block diagram of an operator behavior recognition system according to the present invention;
FIG. 2 is a schematic diagram of an analysis module of an operator behavior recognition system according to the present invention.
Detailed Description
The invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic illustrations, illustrating the basic structure of the invention by way of illustration only, the direction of this embodiment being the direction of fig. 1.
A method of operator behavior identification, the method comprising the steps of:
S1: collecting operation behavior data of an operator, and establishing an operation behavior standard table, a behavior detection model and a target detection model;
S2: acquiring operation behavior video information of an operator in real time, segmenting the operation behavior video information into small frame number behavior video information, detecting action behavior types of the small frame number behavior video information through the behavior detection model, detecting initial state frame numbers and end state frame numbers of action behaviors of the small frame number behavior video information through the target detection model, combining the action behavior types with the initial state frame numbers and the end state frame numbers of the action behaviors to obtain accurate initial state frame numbers and end state frame numbers of the action behaviors, obtaining accurate behavior video frame intervals, and sequentially combining the accurate behavior video frame intervals into accurate operation behavior video information;
S3: and comparing the video information of the accurate operation behavior with standard behaviors on an operation behavior standard table to judge whether the actions of operators are standard or not.
In the early stage, a camera is used for collecting a working behavior video data set of an operator so as to complete the establishment of an operation behavior standard table, a behavior detection model and a target detection model. The operation standard table comprises specific workpiece types, starting and ending time points of standard actions, operation sequences of standard action behaviors and duration time lengths of the standard action behaviors, the behavior detection model uses resnet-50 as a 3D CNN model of a backstone, weak classification of a traditional 2D neural network on time sequence characteristics is solved, a model on Kinetics-400 big data is used as a pre-training model, and training period of the model is shortened while model accuracy is improved.
In the application period, acquiring operation behavior video information of an operator through a camera, dividing the operation behavior video information into two paths, transmitting one path of video to an application display interface, displaying the operation process of the operator in real time, and storing the operation process in a video storage server in a resolution reducing way so as to be convenient for traceability; and the other way is used for judging whether the action of the operator is standard or not.
After the current operation plan of the user is obtained based on the production management MSE system mark, judging whether the workpiece enters the process or not through the target detection algorithm, if so, taking a video segment between the workpiece arriving and leaving the target workbench as the operation behavior video information and starting the behavior recognition and target detection analysis process. When the workpiece enters the detection area, the target detection module 23 obtains a passing signal, and meanwhile, the passing signal is sent to the behavior detection module 22 in a websocket protocol mode, so that the detection flows of the two models are asynchronously performed. The target detection algorithm adopts yolov algorithm as detection algorithm, so that the speed and the precision of the target detection algorithm can reach better indexes.
Dividing the operation behavior video information into small frame number behavior video information, wherein the small frame number behavior video information is 16 frames, so that detection is carried out once every 16 frames, firstly, detecting actions under every 16 frames by a behavior detection module 22, and detecting action types in the time period; the specific frame information of the behavior action start-end state at every 16 frames is then detected by the object detection module 23. The action type is combined with the initial state frame number and the end state frame number of the action to obtain the accurate initial state frame number of the action, so that action time error information is reduced, meanwhile, according to a voting principle, the judgment result of action under every 16 frames is corrected, therefore, the action initial end time error can be improved, and meanwhile, the judgment accuracy of action type caused by external factors such as shielding interference is improved.
Preferably, the step of combining the action behavior type with the start state frame number and the end state frame number of the action behavior in S2 includes:
S21, setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0, 1);
S22, comparing the sizes of W1 and W2, and if the value of W1 and W2 is larger than the value of W2 and C2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1 and C1+W 2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
Preferably, the step S3 includes the steps of:
s31, determining the model of a work piece of the operation through the target detection model, and calling standard behavior information corresponding to the operation behavior standard table according to the model of the work piece;
And S32, comparing the behavior information of the accurate operation behavior video information with the standard behavior information to judge whether the action behavior of the operator is standard or not.
Preferably, the behavior information includes a starting and ending time point, an operation sequence and a duration time of the action behavior.
Preferably, the method further comprises S4, if the starting and ending time points of the actions are inconsistent with the standard behavior information, generating warning information and sending the warning information to a manager for review and production line control operation;
If the operation sequence of the action behaviors is inconsistent with the standard behavior information, generating warning information, sending the warning information to a manager for review and production line control operation, and reminding the operator;
if the duration time length of the action behavior is inconsistent with the standard behavior information, displaying and storing the action behavior as an abnormal event through operating a display screen.
Therefore, under the condition that workshop production is not affected, production behaviors of operators can be effectively monitored, irregular behaviors of the operators are prompted, business proficiency of the operators is improved, production efficiency and production reliability are improved, and product qualification rate is improved.
As shown in fig. 1, the invention also provides an operator behavior recognition system, which is used for implementing the above operator behavior recognition method, and comprises an acquisition module 1, an analysis and judgment module 2, a display module 3 and a warning module 4;
The acquisition module 1 is used for acquiring operation behavior video information of an operator and sending the operation behavior video information to the analysis and judgment module 2 and the display module 3;
the analysis judging module 2 is used for analyzing and judging whether the action behaviors in the operation behavior video information are standard or not and sending an analysis result to the display module 3;
The display module 3 is used for displaying action behaviors and sending action behavior abnormal information to the warning module 4;
the warning module 4 generates warning information according to the abnormal action behavior information to prompt the manager and the operator.
Preferably, as shown in fig. 2, the analysis and judgment module 2 includes a splitting module 21, a behavior detection module 22, a target detection module 23, a fusion module 24, an alignment and judgment module 25, and an analysis module 26;
the splitting module 21 is configured to split the operation behavior video information into small frame number behavior video information;
The behavior detection module 22 is used for detecting the action behavior type of the small frame number behavior video information;
the target detection module 23 is configured to detect a start state frame number and an end state frame number of an action behavior of the small frame number behavior video information, and determine a workpiece model of the operation;
The fusion module 24 is configured to combine the action type with a start state frame number and an end state frame number of the action to obtain a start state frame number and an end state frame number of the action, thereby obtaining an accurate action video frame interval, and sequentially combine the accurate action video frame intervals into accurate operation action video information;
The comparison and judgment module 25 is used for comparing the accurate operation behavior video information with standard behaviors on an operation behavior standard table to judge whether the action behaviors of operators are standard or not;
The analysis module 26 is configured to analyze an nonstandard type of action behavior of an operator, and send the nonstandard type to the display module 3, the display module 3 sends a warning indication to the warning module 4, and the warning module 4 sends a warning to the operator and a manager.
Preferably, the fusion module 24 operates as follows: setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0 and 1); comparing the sizes of W1 and W2, and if the value of W1 is larger than the value of W2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1+W2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
Preferably, the comparison and judgment module 25 determines a workpiece model of the behavior video information operation through the target detection module 23, and invokes standard behavior information corresponding to the operation behavior standard table according to the workpiece model, so as to compare the behavior information of the accurate operation behavior video information with the standard behavior information to judge whether the action of the operator is standard or not.
Preferably, the analysis module 26 analyzes the nonstandard type of the action behavior of the operator, including nonstandard starting and ending time points of the action, nonstandard operation sequence of the action behavior and nonstandard duration time of the action behavior;
the starting and ending time point of the action is not standard, namely the starting and ending time point of the action is inconsistent with the standard behavior information, and if the starting and ending time point of the action is not standard, the warning module 4 generates warning information and sends the warning information to a manager for review and production line control operation;
If the starting and ending time points of the actions are not standard, the analysis module 26 sends the nonstandard information of the starting and ending time points of the actions to the display module 3, the display module 3 sends a warning indication of the nonstandard starting and ending time points of the actions to the warning module 4, and the warning module 4 generates warning information corresponding to the nonstandard starting and ending time points of the actions and sends the warning information to a manager for review and production line control operation;
The operation sequence of the action behavior is inconsistent with the standard behavior information, and if the operation sequence of the action behavior is inconsistent, the warning module 4 generates warning information and sends the warning information to a manager for review and production line control operation, and reminds the operator;
If the operation sequence of the action is not standard, the analysis module 26 sends the operation sequence non-standard information of the action to the display module 3, the display module 3 sends a warning indication of the operation sequence non-standard of the action to the warning module 4, and the warning module 4 generates warning information corresponding to the operation sequence non-standard of the action and sends the warning information to a manager for review and production line control operation and reminds the operator;
the duration of the action is not standard, namely the duration of the action is inconsistent with the standard action information, and if the duration of the action is not standard, the warning module 4 displays and stores the action as an abnormal event through the operation display screen.
If the duration of the action is not standard, the analysis module 26 sends the duration of the action to the display module 3, the display module 3 sends a warning indication of the duration of the action to the warning module 4, and the warning module 4 displays and stores the warning as an abnormal event by operating a display screen.
The system also comprises a man-machine interaction module which is used for displaying information such as connection running state between the application server and the equipment, production line production load, station worker operation reminding and the like, and can be used for configuring running parameters of equipment such as cameras and threshold triggering conditions in a warning rule base.
One embodiment of the invention is as follows:
The system comprises a monitoring device, an edge server, an application server, a warning prompt device and a remote operation tablet. The monitoring equipment is the acquisition module 1 in the system, the edge server is the analysis and judgment module 2 in the system, the application server is the display module 3 in the system, the warning prompt device is the warning module 4 in the system, and the remote operation panel is a monitoring management tool for management personnel.
The monitoring device is connected with the application server through a wired or wireless network, the edge server is connected with the application server through a wired or wireless network, the application server is connected with the warning prompt device through a wired or wireless network, the remote operation panel accesses the application server client through a wireless network, and the monitoring device can be installed in each operation site area needing to be monitored.
According to the operation station to be detected, corresponding monitoring equipment is installed according to the relation between the workpiece and the human operation position, video of the monitoring equipment is transmitted to an application server, one path of the video is displayed in real time through an SDK, the other path of the video is analyzed into images, the images are stored under a fixed folder, the images are pushed to an edge server designated folder through sftp, the edge server analyzes whether the operation behavior types, the operation time lengths and the operation sequences of operators in the area accord with regulations or not through reading the analyzed video, the analysis results are written into a database in the application server, the application server displays and displays abnormal operation, if the abnormality exists, an instruction is sent to a warning device, the warning device carries out warning prompt corresponding to the abnormality, and a manager checks an abnormal prompt event through a remote operation panel and looks back at information in the abnormal video range, so that the next operation of the workshop line is determined.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
While preferred embodiments of the present invention have been described, additional variations and modifications in those embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. It is therefore intended that the following claims be interpreted as including the preferred embodiments and all such alterations and modifications as fall within the scope of the invention.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (10)
1. A method of operator behavior recognition, the method comprising the steps of:
S1: collecting operation behavior data of an operator, and establishing an operation behavior standard table, a behavior detection model and a target detection model;
S2: acquiring operation behavior video information of an operator in real time, segmenting the operation behavior video information into small frame number behavior video information, detecting action behavior types of the small frame number behavior video information through the behavior detection model, detecting initial state frame numbers and end state frame numbers of action behaviors of the small frame number behavior video information through the target detection model, combining the action behavior types with the initial state frame numbers and the end state frame numbers of the action behaviors to obtain accurate initial state frame numbers and end state frame numbers of the action behaviors, obtaining accurate behavior video frame intervals, and sequentially combining the accurate behavior video frame intervals into accurate operation behavior video information;
S3: and comparing the video information of the accurate operation behavior with standard behaviors on an operation behavior standard table to judge whether the actions of operators are standard or not.
2. The operator behavior recognition method according to claim 1, wherein the step of combining the action behavior type with the start state frame number and the end state frame number of the action behavior in S2 is:
S21, setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0, 1);
S22, comparing the sizes of W1 and W2, and if the value of W1 and W2 is larger than the value of W2 and C2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1 and C1+W 2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
3. The operator behavior recognition method according to claim 2, wherein the S3 includes the steps of:
s31, determining the model of a work piece of the operation through the target detection model, and calling standard behavior information corresponding to the operation behavior standard table according to the model of the work piece;
And S32, comparing the behavior information of the accurate operation behavior video information with the standard behavior information to judge whether the action behavior of the operator is standard or not.
4. The operator behavior recognition method of claim 3, wherein the behavior information includes a start-end time point, an operation sequence, and a duration of the action behavior.
5. The method for recognizing the behavior of an operator according to claim 4, further comprising S4, if the starting and ending time points of the actions are inconsistent with the standard behavior information, generating warning information and sending the warning information to a manager for review and production line control operation;
If the operation sequence of the action behaviors is inconsistent with the standard behavior information, generating warning information, sending the warning information to a manager for review and production line control operation, and reminding the operator;
if the duration time length of the action behavior is inconsistent with the standard behavior information, displaying and storing the action behavior as an abnormal event through operating a display screen.
6. An operator behavior recognition system, which is characterized by being used for realizing the operator behavior recognition method according to claims 1-5, and comprising an acquisition module (1), an analysis and judgment module (2), a display module (3) and a warning module (4);
the acquisition module (1) is used for acquiring operation behavior video information of an operator and sending the operation behavior video information to the analysis and judgment module (2) and the display module (3);
The analysis judging module (2) is used for analyzing and judging whether the action behaviors in the operation behavior video information are standard or not and sending an analysis result to the display module (3);
the display module (3) is used for displaying action behaviors and sending action behavior abnormal information to the warning module (4);
the warning module (4) generates warning information according to the abnormal action behavior information to prompt management personnel and operators.
7. The operator behavior recognition system of claim 6, wherein the analysis and judgment module (2) comprises a splitting module (21), a behavior detection module (22), a target detection module (23), a fusion module (24), an alignment and judgment module (25), and an analysis module (26);
the splitting module (21) is used for splitting the operation behavior video information into small-frame behavior video information;
the behavior detection module (22) is used for detecting the action behavior type of the small frame number behavior video information;
the target detection module (23) is used for detecting the initial state frame number and the end state frame number of the action behaviors of the small frame number behavior video information and judging the workpiece model of the operation;
The fusion module (24) is used for combining the action type with the initial state frame number and the end state frame number of the action to obtain the initial state frame number and the end state frame number of the accurate action, so as to obtain an accurate action video frame interval, and combining the accurate action video frame interval into accurate operation action video information in sequence;
The comparison judging module (25) is used for comparing the accurate operation behavior video information with standard behaviors on an operation behavior standard table to judge whether the action behaviors of operators are standard or not;
The analysis module (26) is used for analyzing the nonstandard type of the action behaviors of the operators and sending the nonstandard type to the display module (3), the display module (3) sends out a warning indication to the warning module (4), and the warning module (4) sends out a warning to the operators and the manager.
8. The operator behavior recognition system of claim 7, wherein the fusion module (24) operates as follows: setting the confidence coefficient of the small-frame behavior video information detected by the behavior detection model as conf1 and the weight as W1, setting the confidence coefficient of the small-frame behavior video information detected by the target detection model as conf2 and the weight as W2, and respectively taking values of the weights W1 and W2 according to the data scale and the test precision of the behavior detection model and the target detection model, wherein the value interval of the weights W1 and W2 is (0 and 1); comparing the sizes of W1 and W2, and if the value of W1 is larger than the value of W2, taking the small frame number behavior video information detected by the behavior detection model as an accurate behavior video frame interval, wherein the corresponding confidence is W1+W2; otherwise, taking the intersection of the video frame sequence of the small-frame behavior video information detected by the behavior detection model and the video frame sequence of the small-frame behavior video information detected by the target detection model as an accurate behavior video frame interval.
9. The system according to claim 8, wherein the comparison and judgment module (25) determines a workpiece model of the behavior video information job through the target detection module (23), and invokes standard behavior information corresponding to the operation behavior standard table according to the workpiece model, and compares the behavior information of the accurate job behavior video information with the standard behavior information to judge whether the action of the operator is standard.
10. The operator behavior recognition system of claim 9, wherein the analysis module (26) analyzes the nonstandard type of action behavior of the operator including a start-end time point nonstandard of action, an order of operation nonstandard of action behavior, a duration length nonstandard of action behavior;
The starting and ending time point of the action is not standard, namely the starting and ending time point of the action is inconsistent with the standard behavior information, and if the starting and ending time point of the action is not standard, the warning module (4) generates warning information and sends the warning information to a manager for review and production line control operation;
The operation sequence of the action behavior is inconsistent with the standard behavior information, and if the operation sequence of the action behavior is inconsistent, the warning module (4) generates warning information and sends the warning information to a manager for review and production line control operation, and reminds the operator;
the duration of the action is not standard, namely the duration of the action is inconsistent with the standard action information, and if the duration of the action is not standard, the warning module (4) displays and stores the action as an abnormal event through the operation display screen.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211300049.XA CN117975547A (en) | 2022-10-24 | 2022-10-24 | Operator behavior recognition method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211300049.XA CN117975547A (en) | 2022-10-24 | 2022-10-24 | Operator behavior recognition method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117975547A true CN117975547A (en) | 2024-05-03 |
Family
ID=90863793
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211300049.XA Pending CN117975547A (en) | 2022-10-24 | 2022-10-24 | Operator behavior recognition method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117975547A (en) |
-
2022
- 2022-10-24 CN CN202211300049.XA patent/CN117975547A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8757469B2 (en) | Binary classification of items of interest in a repeatable process | |
| CN110738135B (en) | Method and system for judging and guiding worker operation step standard visual recognition | |
| CN110889339B (en) | Head and shoulder detection-based dangerous area grading early warning method and system | |
| CN111069819A (en) | Welding quality prediction system and method based on artificial intelligence | |
| CN111507261B (en) | Visual target positioning-based process operation quality monitoring method | |
| CN110570146A (en) | Product sorting method and device and computer readable storage medium | |
| JP2012022602A (en) | Operation improvement analysis system | |
| CN110659201A (en) | Intelligent test analysis system for safety technology protection engineering | |
| CN113084388B (en) | Welding quality detection method, system, device and storage medium | |
| CN111126196A (en) | Equipment oil leakage detection method | |
| CN105809674A (en) | Machine vision based die protection apparatus and its functioning method | |
| CN112687022A (en) | Intelligent building inspection method and system based on video | |
| KR20210097491A (en) | Tool-related abnormal data detection system of CNC machines | |
| CN115577151A (en) | Vehicle fault real-time diagnosis method and device, electronic equipment and storage medium | |
| CN114200877B (en) | Monitoring method and device for electric equipment | |
| CN111091104A (en) | Target object protection detection method, device, equipment and storage medium | |
| CN117114420B (en) | Image recognition-based industrial and trade safety accident risk management and control system and method | |
| CN117372377B (en) | Broken line detection method and device for monocrystalline silicon ridge line and electronic equipment | |
| CN116703914B (en) | Welding defect detection method, equipment and medium based on generation type artificial intelligence | |
| CN117975547A (en) | Operator behavior recognition method and system | |
| CN112613476A (en) | Method for automatically detecting unsafe behaviors of workers based on machine vision | |
| CN114998357B (en) | Industrial detection method, system, terminal and medium based on multi-information analysis | |
| CN115564141A (en) | Welding spot quality monitoring method and device and medium thereof | |
| CN110134069B (en) | Self-diagnosis method and system of numerical control machine tool | |
| WO2024065189A1 (en) | Method, system, apparatus, electronic device, and storage medium for evaluating work task |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |