CN113326713A - Action recognition method, device, equipment and medium - Google Patents
Action recognition method, device, equipment and medium Download PDFInfo
- Publication number
- CN113326713A CN113326713A CN202010127230.XA CN202010127230A CN113326713A CN 113326713 A CN113326713 A CN 113326713A CN 202010127230 A CN202010127230 A CN 202010127230A CN 113326713 A CN113326713 A CN 113326713A
- Authority
- CN
- China
- Prior art keywords
- operation step
- gesture feature
- gesture
- standard
- determining
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/117—Biometrics derived from hands
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
The embodiment of the invention discloses a method, a device, equipment and a medium for identifying actions. The action recognition method comprises the following steps: receiving an operation video of an operator in the operation process, which is uploaded by the AR equipment; identifying at least one gesture feature in the operation video, and determining at least one operation step included in a work flow according to the gesture feature, wherein the operation step includes at least one gesture feature; and determining the working hours of each operation step, and determining whether the operation step is matched with a standard operation step according to the working hours of each operation step. According to the technical scheme of the embodiment of the invention, the operation flow is divided into a plurality of operation steps for analysis by identifying the gesture characteristics contained in the operation video, so that the efficiency of identifying and monitoring the operation action in the operation flow is improved.
Description
Technical Field
The embodiment of the invention relates to an industrial operation monitoring technology, in particular to a method, a device, equipment and a medium for recognizing actions.
Background
In industrial production, in order to ensure product quality and production efficiency, it is necessary to monitor the operation of an operator, record detailed operation time, and form a working time database so that a process engineer can monitor the operation normative and standardize the operation time.
The existing operation action monitoring method is that a process engineer directly observes the action of an operator on site, and the action is split according to a standard method of action analysis to form working hour records.
Disclosure of Invention
The embodiment of the invention provides an action recognition method, an action recognition device, equipment and a medium, which can be used for dividing a work flow into a plurality of operation steps for analysis by recognizing gesture characteristics contained in an operation video, thereby improving the efficiency of recognizing and monitoring the work action in the work flow.
In a first aspect, an embodiment of the present invention provides an action recognition method, where the method is applied to a background system, and the method includes:
receiving an operation video of an operator in the operation process, which is uploaded by the AR equipment;
identifying at least one gesture feature in the operation video, and determining at least one operation step included in a work flow according to the gesture feature, wherein the operation step includes at least one gesture feature;
and determining the working hours of each operation step, and determining whether the operation step is matched with a standard operation step according to the working hours of each operation step.
In a second aspect, an embodiment of the present invention provides an action recognition method, where the method includes:
the AR equipment acquires an operation video of an operator in the operation process through a camera and uploads the operation video to a background system;
the background system identifies at least one gesture feature in the operation video and determines at least one operation step included in the operation process according to the gesture feature, wherein the operation step includes at least one gesture feature;
and the background system determines the working hours of each operation step and determines whether the operation step is matched with a standard operation step according to the working hours of each operation step.
In a third aspect, an embodiment of the present invention further provides an action recognition apparatus, where the apparatus includes:
the operation video receiving module is used for receiving operation videos uploaded by the AR equipment during the operation process of an operator;
the operation step determination module is used for identifying at least one gesture feature in the operation video and determining at least one operation step included in the operation process according to the gesture feature, wherein the operation step includes at least one gesture feature;
and the operation step judgment module is used for determining the working hours of each operation step and determining whether the operation step is matched with the standard operation step according to the working hours of each operation step.
In a fourth aspect, an embodiment of the present invention further provides an apparatus, where the apparatus includes:
one or more processors;
a memory for storing one or more programs;
when the one or more programs are executed by the one or more processors, the one or more processors implement the action recognition method provided by any embodiment of the present invention.
In a fifth aspect, the embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the motion recognition method provided in any embodiment of the present invention.
According to the technical scheme of the embodiment of the invention, the background system firstly identifies at least one gesture feature in the operation video uploaded by the AR equipment, then determines at least one operation step contained in the operation flow according to the gesture feature, finally determines the working hour of each operation step, and determines whether the operation step is matched with a standard operation step or not according to the working hour of each operation step, so that the problems that in the prior art, an engineer needs to observe the operation flow of an operator on site and split and analyze the action, the time and the economic cost are high are solved, and the efficiency of identifying and monitoring the operation action in the operation flow is improved.
Drawings
Fig. 1 is a flowchart of a motion recognition method according to a first embodiment of the present invention;
FIG. 2a is a flowchart of a motion recognition method according to a second embodiment of the present invention;
FIG. 2b is a schematic diagram illustrating feature extraction of a single-handed gesture according to a second embodiment of the present invention;
FIG. 2c is a schematic diagram illustrating a two-hand position relationship extraction according to a second embodiment of the present invention;
FIG. 3 is a flowchart of a motion recognition method according to a third embodiment of the present invention;
FIG. 4 is a flowchart of a method for recognizing actions according to a fourth embodiment of the present invention;
fig. 5 is a schematic structural diagram of a motion recognition apparatus according to a fifth embodiment of the present invention;
fig. 6 is a schematic structural diagram of an apparatus according to a sixth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
Example one
Fig. 1 is a flowchart of an action recognition method in a first embodiment of the present invention, where the technical solution of this embodiment is applied to a background system, and is suitable for a situation where a workflow is split and analyzed by the background system, the method may be executed by an action recognition device, and the action recognition device may be implemented by software and/or hardware, and may be integrated in various general-purpose computer devices, and specifically includes the following steps:
and step 110, receiving an operation video uploaded by the AR equipment during the operation process of an operator.
The AR device used in this embodiment is a head-ring AR device, is configured with a camera, a microphone, and a speaker, is suitable for an application scene where images are worn and recorded for a long time, and may be connected to a backend system in various ways such as wireless, bluetooth, or Type C. Before the AR equipment is used, the access authority of the AR equipment is set in a background system, so that the AR equipment can conveniently send the collected image information to the background system and receive the instruction information processed by the background system.
In this embodiment, the background system first receives an operation video of an operator in an operation process uploaded by the AR device, specifically, in a process in which the operator wears the AR device to perform an operation, a camera of the AR device records a video of a first viewing angle of a hand motion of the operator to obtain the operation video, and then uploads the operation video to the background system, and the background system receives the operation video to further identify and process operation steps in the operation video. For example, the operator's workflow may be a product part assembly flow.
And 120, identifying at least one gesture feature in the operation video, and determining at least one operation step included in the operation process according to the gesture feature, wherein the operation step includes at least one gesture feature.
The gesture features refer to hand features capable of representing hand motions of the current operator and are used for determining operation steps executed by the previous operator. For example, the gesture features may be hand contour information of the operator and the center of hand rotation.
In this embodiment, the background system identifies the received operation video to obtain at least one gesture feature included in the operation video, and then divides the entire operation flow into at least one operation step according to the gesture feature, so as to identify and analyze the operation of the operator.
Illustratively, the background system identifies at least one gesture feature contained in the operation video, for example, identifies a left-right hand geometric feature and a two-hand positional relationship to jointly form the gesture feature, compares each gesture feature with a set gesture feature identification condition, determines an operation step to which each gesture feature belongs, and finally divides the operation flow into a plurality of operation steps, for example, the gesture feature identification condition for taking the material 1 includes a gesture feature that an operator stretches his hand to take the material 1 and a gesture feature that the material 1 is put on a workbench, and determines that at least one gesture feature belongs to the operation step for taking the material 1 after identifying at least one gesture feature corresponding to the gesture feature identification condition in the operation video.
Optionally, after identifying at least one gesture feature in the operation video and determining at least one operation step included in the workflow according to the gesture feature, the method further includes:
and simulating the virtual operation flow according to the at least one operation step, and displaying a simulation result to assist the monitoring personnel to analyze the operation steps.
In this optional embodiment, a specific operation after at least one operation step included in the operation flow is determined according to the gesture feature, the virtual operation flow is simulated according to the determined at least one operation step, the simulation result can be displayed on a display screen, and a background monitoring person can conveniently and intuitively see the whole operation process. Illustratively, when recognizing that an operator is performing product assembly operation, virtual assembly simulation can be performed according to a product CAD model and an operation flow of the operator, and finally a simulated assembly flow of the whole assembly operation is displayed on a screen, so that background monitoring personnel can visually observe and analyze the operation flow of the operator.
And step 130, determining the working hours of each operation step, and determining whether the operation step is matched with the standard operation step according to the working hours of each operation step.
In this embodiment, after the work flow is divided into a plurality of operation steps, the man-hours corresponding to each operation step are determined according to the time consumption of each operation step, and whether each operation step matches the standard operation step is determined according to the man-hours corresponding to each operation step. Specifically, the working hours corresponding to each operation step are determined according to the starting time and the ending time of each operation step, and then the working hours of the operation step are compared with the standard working hours corresponding to the operation step to determine whether the operation step is matched with the standard operation step.
For example, if the starting time of the operation step of obtaining the material 1 is 3 minutes and 5 seconds, and the ending time of the step is 3 minutes and 9.5 seconds, the man-hour of the operation step can be determined to be 4.5 seconds, and then the standard man-hour corresponding to the operation step of obtaining the material 1, for example, 4 seconds, obviously, both of them have a man-hour error of 0.5 seconds, then according to the preset judgment standard, it is judged whether the operation step matches the standard operation step, for example, when the man-hour error is greater than 1 second, it is determined that the task operation step does not match the standard operation step, and the operation step of obtaining the material 1 matches the standard operation step.
According to the technical scheme of the embodiment of the invention, the background system firstly identifies at least one gesture feature in the operation video uploaded by the AR equipment, then determines at least one operation step contained in the operation flow according to the gesture feature, finally determines the working hour of each operation step, and determines whether the operation step is matched with a standard operation step or not according to the working hour of each operation step, so that the problems that in the prior art, an engineer needs to observe the operation flow of an operator on site and split and analyze the action, the time and the economic cost are high are solved, and the efficiency of identifying and monitoring the operation action in the operation flow is improved.
Example two
Fig. 2a is a flowchart of an action recognition method in the second embodiment of the present invention, which is further detailed based on the above embodiments, and this embodiment provides specific steps of recognizing at least one gesture feature in the operation video, and determining at least one operation step included in a workflow according to the gesture feature. A motion recognition method provided by a second embodiment of the present invention is described below with reference to fig. 2a, and includes the following steps:
and step 210, receiving an operation video uploaded by the AR equipment during the operation process of an operator.
And step 220, identifying the geometric characteristics of a single hand corresponding to the left hand and the right hand of the operator in the operation video respectively, wherein the geometric characteristics of the single hand comprise a gesture outline and a hand motion center.
The single-hand geometric features are geometric features which characterize the current single-hand action of the operator and are used for determining the operation steps to which the current action of the operator belongs, wherein the single-hand geometric features are shown in fig. 2b and comprise gesture outlines and hand motion centers.
In this embodiment, a manner of recognizing gesture features of an operator is provided, specifically, by performing frame-by-frame recognition on an operation video, single-hand geometric features corresponding to left and right hands of the operator in the operation video are obtained, illustratively, a left-hand contour feature and a left-hand movement center of the operator, and a right-hand contour feature and a right-hand movement center of the operator are obtained in sequence, where the gesture contour features may include a palm and a fingertip of each finger.
And step 230, determining a two-hand position relation according to the geometric characteristics of the single hand, and forming the gesture characteristics of the operator by the geometric characteristics of the single hand and the two-hand position relation.
In this embodiment, after the one-hand geometric features corresponding to the left hand and the right hand respectively are obtained, the two-hand positional relationship is determined according to the one-hand geometric features, as shown in fig. 2c, the one-hand geometric features and the two-hand positional relationship together finally form the gesture features of the operator, and for example, the two-hand positional relationship may include distance and angle information of the two hands.
the gesture feature recognition condition is at least one preset standard gesture feature corresponding to each operation step.
In this embodiment, it is determined whether the recognized gesture feature satisfies a set gesture recognition condition, specifically, the gesture feature recognition condition is a preset standard gesture feature corresponding to each operation step, the recognized gesture feature recognition condition is matched with the standard gesture feature, if the matching is successful, it is determined that the current gesture feature belongs to the operation step corresponding to the gesture feature recognition condition, and finally, the operation flow is divided into at least one operation step according to the gesture feature.
Illustratively, the defining of the gesture feature recognition condition corresponding to the assembly part 2 includes an operation step of an operator grasping the gesture feature of the assembly part 2 with one hand and assembling the assembly part 2 with two hands, and determining that at least one gesture feature belongs to the assembly part 2 after at least one gesture feature matching the gesture feature recognition condition in the operation video is recognized. And finally, determining operation steps corresponding to all the gesture features identified in the operation video, and realizing that the whole operation flow is divided into at least one operation step.
And step 250, determining the working hours of each operation step, and determining whether the operation step is matched with the standard operation step according to the working hours of each operation step.
According to the technical scheme of the embodiment of the invention, the gesture characteristics of the operator are determined by identifying the geometric characteristics of one hand and the position relation of two hands of the operator, the operation steps to which the gesture characteristics belong are determined according to the gesture characteristic identification conditions, the operation flow is divided into at least one operation step, the working hours of each operation step are finally determined, and whether the operation steps are matched with the standard operation steps or not is determined according to the working hours of each operation step.
EXAMPLE III
Fig. 3 is a flowchart of an action recognition method in a third embodiment of the present invention, which is further detailed based on the above embodiments, and provides specific steps of determining the working hours of each operation step, and determining whether the operation step matches a standard operation step according to the working hours of each operation step. A motion recognition method provided by a third embodiment of the present invention is described below with reference to fig. 3, and includes the following steps:
and 310, receiving an operation video uploaded by the AR equipment during the operation process of the operator.
And step 320, identifying at least one gesture feature in the operation video, and determining at least one operation step included in the operation flow according to the gesture feature, wherein the operation step includes at least one gesture feature.
And 330, determining the working hours of the operation steps according to the appearance time of the starting gesture feature and the appearance time of the ending gesture feature corresponding to the operation steps, wherein the starting gesture feature is the first gesture feature corresponding to the operation step, and the ending gesture feature is the last gesture feature corresponding to the operation step.
In this embodiment, a manner of determining the working hours of the operation steps is provided, specifically, the starting gesture feature occurrence time and the ending gesture feature occurrence time corresponding to the operation steps are first obtained, and a difference between the ending gesture feature occurrence time and the starting gesture feature occurrence time is calculated, that is, the working hours of the current operation step, where the starting gesture feature is a first gesture feature corresponding to the operation step, and the ending gesture feature is a last gesture feature corresponding to the operation step.
Illustratively, after determining that the first three gesture features in the operation video belong to the operation step 1, acquiring the occurrence time of the first gesture feature and the occurrence time of the third gesture feature, and calculating the time difference as the working hours of the operation step 1, so as to analyze whether each operation step meets the specification or not according to the working hours of each operation step.
And 340, acquiring each standard working hour corresponding to each operation step.
In this embodiment, in order to determine whether each operation step meets the specification according to the working hours of each operation step, it is necessary to obtain the standard working hours corresponding to each operation step, where the standard working hours corresponding to each operation step are obtained by analyzing the standard work flow designed in advance by an engineer. For example, for assembly work of a certain product, before entering a factory for production, an engineer may refer to a set of standard operation flows in advance to provide guidance for production operation of an operator, analyze the standard operation flows, and obtain standard man-hours corresponding to each standard operation step in the standard operation flows, so as to determine whether the operation steps of the operator meet specifications in a generation process.
And 350, calculating the error of the working hours of each operation step according to the working hours of each operation step and each standard working hour.
In this embodiment, after the standard man-hours corresponding to the operation steps are obtained, the error between the man-hours of each operation step and the standard man-hours is calculated to determine whether each operation step meets the specification.
And step 360, when the man-hour error is larger than the set threshold, determining that the operation step is not matched with the standard operation step.
In this embodiment, a manner of determining whether each operation step meets the specification according to the labor-hour error is provided, specifically, when the labor-hour error is greater than a set threshold, it is determined that the operation step does not match the standard operation step. For example, if the threshold value of the error of the man-hour is set to 1 second, the standard man-hour for assembling the component 2 is set to 5 seconds, and the current operator works, the operation step corresponds to the man-hour of 6.5 seconds, the error of the man-hour can be 1.5 seconds, and obviously, if the error is larger than the set threshold value of the error of the man-hour, the current operation step is determined not to match the standard operation step.
Optionally, after determining that the operation step does not match the standard operation step, the method further includes:
dividing the operation step into at least one job action according to at least one gesture feature contained in the operation step, wherein each job action corresponds to one gesture feature;
acquiring the time for the operator to keep the gesture characteristics as the working hours of the operation action corresponding to the gesture characteristics;
calculating the working hour error of each operation according to the working hour of each operation and the standard working hour corresponding to each operation;
when the man-hour error is greater than a set threshold, determining that the work action does not match a standard work action.
In this optional embodiment, a specific analysis manner after determining that the operation step is not matched with the standard operation step is provided, in order to further analyze the specific time generated by the man-hour error, the operation step is first divided into at least one operation action according to at least one gesture feature included in each operation step, wherein each gesture feature corresponds to one operation step, further, the man-hour of the operation action corresponding to the gesture feature is determined according to the time for holding the gesture feature by the operator, finally, the man-hour of each operation action is compared with the standard man-hour corresponding to each operation action, and when the man-hour error is greater than a set threshold, it is determined that the current operation action is not matched with the standard operation action.
For example, when it is determined that the operation step of assembling the part 2 does not match the standard operation step, the operation step is further analyzed, the operation step is divided into two operation steps corresponding to the respective gesture features according to the two gesture features included in the operation step of assembling the part 2, the operation step is the operation of taking the part 2 and the operation step of assembling the part 2, the stay time of the operator in the two operation steps is obtained as the working hours corresponding to the two operation steps, and finally, the working hours of the two operation steps are compared with the corresponding standard working hours, so that when it is determined that the working hour error of the operation step of assembling the part 2 is greater than the set threshold, for example, 0.5 second, it is determined that the operation step does not match the standard operation step, which is the main factor of generating the working hour error.
Optionally, after determining that the job action does not match the standard job action, the method further includes:
when the working hour of the operation action is larger than the standard working hour corresponding to each operation, operation error information is sent to the AR equipment to indicate the AR equipment to initiate an operation non-standard voice prompt to the operator;
and when the working hours of the working actions are less than the standard working hours corresponding to all the works, displaying the working actions and initiating operation optimization prompts to remind monitoring personnel to analyze whether the working actions have optimization or not.
In this optional embodiment, a specific operation after a specific operation action is determined to be not matched with a standard operation action is provided, and first, whether the working hours of the operation action are greater than the corresponding standard working hours is determined, if yes, it is indicated that the operation action of the operator is not standard, and the reason may be that the operation attention is not concentrated or the time consumed by a certain operation action is too long, at this time, operation error information needs to be sent to the AR device to indicate the AR device to initiate a voice prompt that the operation is not standard to the operator; if not, the operation action of the operator is not matched with the standard operation action, which may be because a short operation is not in place in a certain operation action, or the operator has a better operation mode to improve the operation efficiency.
According to the technical scheme of the embodiment of the invention, the background system firstly identifies at least one gesture feature in the operation video uploaded by the AR equipment, then determines the working hours of each operation step according to at least one operation step contained in the operation flow according to the gesture feature, calculates the error between the working hours of each operation step and the standard working hours, and further identifies and analyzes the operation steps when the working hour error is greater than a set threshold value, so that the operation actions with larger errors are directly displayed, a monitoring person does not need to search the operation actions generating the working hour errors from all the operation flows, and the efficiency of identifying and monitoring the operation actions in the operation flows is improved.
Example four
Fig. 4 is a flowchart of an action recognition method in the fourth embodiment of the present invention, where the technical solution of this embodiment is applied to an action recognition system, and is suitable for a situation where an operation video is collected by an AR device and a work flow is split and analyzed by a background system, and the action recognition method provided in the fourth embodiment of the present invention is described below with reference to fig. 4, and includes the following steps:
and step 410, the AR equipment acquires an operation video of an operator in the operation process through the camera and uploads the operation video to the background system.
In this embodiment, in the process of wearing the AR device to perform an operation, an operator records a video of a hand action of the operator at a first viewing angle through a camera of the AR device, acquires an operation video of the operator, and uploads the operation video to a background system to further identify and process operation steps in the operation video.
Step 420, the background system identifies at least one gesture feature in the operation video, and determines at least one operation step included in the workflow according to the gesture feature, wherein the operation step includes at least one gesture feature.
In this embodiment, the background system identifies the received operation video to obtain at least one gesture feature included in the operation video, and then divides the entire operation flow into at least one operation step according to the gesture feature, so as to identify and analyze the operation of the operator.
Illustratively, the background system identifies 5 gesture features contained in the operation video, then matches each gesture feature with a preset standard gesture feature, determines that the gesture feature belongs to an operation step corresponding to the standard gesture feature when the matching is successful, and finally determines that the first 3 gesture features belong to an operation step 1 and the last 2 gesture features belong to an operation step 2, namely, the operation flow is divided into 2 operation steps.
And 430, determining the working hours of each operation step by the background system, and determining whether the operation step is matched with the standard operation step according to the working hours of each operation step.
In this embodiment, after the work flow is divided into a plurality of operation steps, the man-hours corresponding to each operation step are determined according to the duration of each operation step, and whether each operation step matches a standard operation step is determined according to the man-hours corresponding to each operation step. Illustratively, the occurrence time of the first gesture feature and the occurrence time of the last gesture feature included in the operation step are acquired, the working hour of the operation step is determined according to the two times, the working hour of the operation step is finally compared with the standard working hour corresponding to the operation step, the working hour error is calculated, and the operation step with the working hour being larger than the set threshold is finally determined to be not matched with the standard operation step, namely, the operation step may have an operation irregularity problem.
According to the technical scheme of the embodiment of the invention, the operation video of an operator in the operation process is collected through the AR device, the operation video is uploaded to the background system, at least one gesture feature in the operation video uploaded by the AR device is identified through the background system, at least one operation step included in the operation process is determined according to the gesture feature, and finally, after the working hours of each operation step are determined, whether the operation step is matched with a standard operation step is determined according to the working hours of each operation step, so that an engineer is not required to observe the operation process of the operator on site and perform action splitting and analysis, and the efficiency of identifying and monitoring the operation action in the operation process is improved.
EXAMPLE five
Fig. 5 is a schematic structural diagram of a motion recognition device according to a fifth embodiment of the present invention, where the motion recognition device includes: an operation video receiving module 510, an operation step determining module 520, and an operation step judging module 530.
An operation video receiving module 510, configured to receive an operation video uploaded by the AR device during a working process of an operator;
an operation step determination module 520, configured to identify at least one gesture feature in the operation video, and determine at least one operation step included in the workflow according to the gesture feature, where the operation step includes at least one gesture feature;
an operation step judging module 530, configured to determine the working hours of each operation step, and determine whether the operation step matches a standard operation step according to the working hours of each operation step.
According to the technical scheme of the embodiment of the invention, the background system firstly identifies at least one gesture feature in the operation video uploaded by the AR equipment, then determines at least one operation step contained in the operation flow according to the gesture feature, finally determines the working hour of each operation step, and determines whether the operation step is matched with a standard operation step or not according to the working hour of each operation step, so that the problems that in the prior art, an engineer needs to observe the operation flow of an operator on site and split and analyze the action, the time and the economic cost are high are solved, and the efficiency of identifying and monitoring the operation action in the operation flow is improved.
Optionally, the operation step determining module 520 includes:
the single-hand geometric feature recognition unit is used for recognizing single-hand geometric features corresponding to the left hand and the right hand of an operator in the operation video respectively, wherein the single-hand geometric features comprise gesture outlines and hand motion centers;
the gesture feature forming unit is used for determining a two-hand position relationship according to the one-hand geometric feature, and forming a gesture feature of the operator by the one-hand geometric feature and the two-hand position relationship;
the operation step determination unit is used for determining that the gesture features belong to operation steps corresponding to the gesture feature recognition conditions when the gesture features meet the set gesture feature recognition conditions, and dividing the operation flow into at least one operation step according to the gesture features;
the gesture feature recognition condition is at least one preset standard gesture feature corresponding to each operation step.
Optionally, the operation step determining module 530 includes:
the labor hour determining unit is used for determining the labor hour of each operation step according to the appearance time of the starting gesture feature and the appearance time of the ending gesture feature corresponding to each operation step, wherein the starting gesture feature is the first gesture feature corresponding to the operation step, and the ending gesture feature is the last gesture feature corresponding to the operation step;
a standard man-hour obtaining unit for obtaining each standard man-hour corresponding to each operation step;
a man-hour error calculation unit for calculating the man-hour error of each operation step according to the man-hour of each operation step and each standard man-hour;
and the operation step judgment module unit is used for determining that the operation step is not matched with the standard operation step when the man-hour error is larger than a set threshold value.
Optionally, the motion recognition apparatus further includes:
the operation action dividing module is used for dividing the operation steps into at least one operation action according to at least one gesture feature contained in the operation steps after the operation steps are determined not to be matched with the standard operation steps, wherein each operation action corresponds to one gesture feature;
the man-hour determining module is used for acquiring the time for the operator to keep the gesture characteristics as the man-hour of the operation action corresponding to the gesture characteristics;
a man-hour error calculation module for calculating the man-hour error of each operation according to the man-hour of each operation and the standard man-hour corresponding to each operation;
and the working action judging module is used for determining that the working action is not matched with the standard working action when the working hour error is larger than a set threshold value.
Optionally, the motion recognition apparatus further includes:
the operation error information sending module is used for sending operation error information to the AR equipment when the working hour of the operation action is larger than the standard working hour corresponding to each operation so as to indicate the AR equipment to initiate an operation non-standard voice prompt to the operator;
and the operation optimization prompting module is used for displaying the operation actions and initiating operation optimization prompts to remind monitoring personnel to analyze whether the operation actions have optimization or not when the working hours of the operation actions are less than the standard working hours corresponding to each operation.
Optionally, the motion recognition apparatus further includes:
and the simulation result display module is used for simulating the virtual operation flow according to the at least one operation step after identifying at least one gesture characteristic in the operation video and determining the at least one operation step included in the operation flow according to the gesture characteristic, and displaying a simulation result so as to assist the monitoring personnel to analyze the operation step.
The action recognition device provided by the embodiment of the invention can execute the action recognition method provided by any embodiment of the invention, and has corresponding functional modules and beneficial effects of the execution method.
EXAMPLE six
Fig. 6 is a schematic structural diagram of an AR device according to a sixth embodiment of the present invention, and as shown in fig. 6, the electronic device includes a processor 60 and a memory 61; the number of processors 60 in the device may be one or more, and one processor 60 is taken as an example in fig. 6; the processor 60 and the memory 61 in the device may be connected by a bus or other means, as exemplified by the bus connection in fig. 6.
The memory 61 is used as a computer-readable storage medium for storing software programs, computer-executable programs, and modules, such as program instructions/modules corresponding to a motion recognition method in the embodiment of the present invention (for example, the operation video receiving module 510, the operation step determining module 520, and the operation step determining module 530 in the motion recognition apparatus). The processor 60 executes various functional applications of the device and data processing by executing software programs, instructions, and modules stored in the memory 61, that is, implements the above-described motion recognition method.
The method comprises the following steps:
receiving an operation video of an operator in the operation process, which is uploaded by the AR equipment;
identifying at least one gesture feature in the operation video, and determining at least one operation step included in a work flow according to the gesture feature, wherein the operation step includes at least one gesture feature;
and determining the working hours of each operation step, and determining whether the operation step is matched with a standard operation step according to the working hours of each operation step.
The memory 61 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function; the storage data area may store data created according to the use of the terminal, and the like. Further, the memory 61 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid state storage device. In some examples, the memory 61 may further include memory located remotely from the processor 60, which may be connected to the device over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
EXAMPLE seven
An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, where the computer program is used for executing an action recognition method when executed by a computer processor, and the method includes:
receiving an operation video of an operator in the operation process, which is uploaded by the AR equipment;
identifying at least one gesture feature in the operation video, and determining at least one operation step included in a work flow according to the gesture feature, wherein the operation step includes at least one gesture feature;
and determining the working hours of each operation step, and determining whether the operation step is matched with a standard operation step according to the working hours of each operation step.
From the above description of the embodiments, it is obvious for those skilled in the art that the present invention can be implemented by software and necessary general hardware, and certainly, can also be implemented by hardware, but the former is a better embodiment in many cases. Based on such understanding, the technical solutions of the present invention may be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as a floppy disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, and includes several instructions for enabling a computer device (which may be a personal computer, a server, or a network device) to execute the methods according to the embodiments of the present invention.
It should be noted that, in the embodiment of the motion recognition device, the included units and modules are merely divided according to functional logic, but are not limited to the above division as long as the corresponding functions can be realized; in addition, specific names of the functional units are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present invention.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in greater detail by the above embodiments, the present invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims (10)
1. An action recognition method is applied to a background system, and comprises the following steps:
receiving an operation video of an operator in the operation process, which is uploaded by the AR equipment;
identifying at least one gesture feature in the operation video, and determining at least one operation step included in a work flow according to the gesture feature, wherein the operation step includes at least one gesture feature;
and determining the working hours of each operation step, and determining whether the operation step is matched with a standard operation step according to the working hours of each operation step.
2. The method according to claim 1, wherein identifying at least one gesture feature in the operation video and determining at least one operation step included in the workflow according to the gesture feature comprises:
identifying single-hand geometric features respectively corresponding to the left hand and the right hand of an operator in the operation video, wherein the single-hand geometric features comprise gesture outlines and hand motion centers;
determining a two-hand position relation according to the one-hand geometric characteristics, and forming gesture characteristics of an operator by the one-hand geometric characteristics and the two-hand position relation;
when the gesture features meet set gesture feature recognition conditions, determining that the gesture features belong to operation steps corresponding to the gesture feature recognition conditions, and dividing the operation flow into at least one operation step according to the gesture features;
the gesture feature recognition condition is at least one preset standard gesture feature corresponding to each operation step.
3. The method of claim 1, wherein determining the man-hours of each operational step and determining whether the operational step matches a standard operational step based on the man-hours of each operational step comprises:
determining the working hours of the operation steps according to the appearance time of the starting gesture feature and the appearance time of the ending gesture feature corresponding to the operation steps, wherein the starting gesture feature is the first gesture feature corresponding to the operation steps, and the ending gesture feature is the last gesture feature corresponding to the operation steps;
acquiring each standard working hour corresponding to each operation step;
calculating the man-hour error of each operation step according to the man-hour of each operation step and each standard man-hour;
when the man-hour error is greater than a set threshold, determining that the operation step does not match the standard operation step.
4. The method of claim 3, further comprising, after determining that the operational step does not match the standard operational step:
dividing the operation step into at least one job action according to at least one gesture feature contained in the operation step, wherein each job action corresponds to one gesture feature;
acquiring the time for the operator to keep the gesture characteristics as the working hours of the operation action corresponding to the gesture characteristics;
calculating the working hour error of each operation according to the working hour of each operation and the standard working hour corresponding to each operation;
when the man-hour error is greater than a set threshold, determining that the work action does not match a standard work action.
5. The method of claim 4, after determining that the job action does not match a standard job action, further comprising:
when the working hour of the operation action is larger than the standard working hour corresponding to each operation, operation error information is sent to the AR equipment to indicate the AR equipment to initiate an operation non-standard voice prompt to the operator;
and when the working hours of the working actions are less than the standard working hours corresponding to all the works, displaying the working actions and initiating operation optimization prompts to remind monitoring personnel to analyze whether the working actions have optimization or not.
6. The method according to claim 5, after identifying at least one gesture feature in the operation video and determining at least one operation step included in a workflow according to the gesture feature, further comprising:
and simulating the virtual operation flow according to the at least one operation step, and displaying a simulation result to assist the monitoring personnel to analyze the operation steps.
7. A motion recognition method, comprising:
the AR equipment acquires an operation video of an operator in the operation process through a camera and uploads the operation video to a background system;
the background system identifies at least one gesture feature in the operation video and determines at least one operation step included in the operation process according to the gesture feature, wherein the operation step includes at least one gesture feature;
and the background system determines the working hours of each operation step and determines whether the operation step is matched with a standard operation step according to the working hours of each operation step.
8. An action recognition device, comprising:
the operation video receiving module is used for receiving operation videos uploaded by the AR equipment during the operation process of an operator;
the operation step determination module is used for identifying at least one gesture feature in the operation video and determining at least one operation step included in the operation process according to the gesture feature, wherein the operation step includes at least one gesture feature;
and the operation step judgment module is used for determining the working hours of each operation step and determining whether the operation step is matched with the standard operation step according to the working hours of each operation step.
9. An apparatus, comprising:
one or more processors;
a storage device for storing one or more programs,
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010127230.XA CN113326713B (en) | 2020-02-28 | 2020-02-28 | Action recognition method, device, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010127230.XA CN113326713B (en) | 2020-02-28 | 2020-02-28 | Action recognition method, device, equipment and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113326713A true CN113326713A (en) | 2021-08-31 |
| CN113326713B CN113326713B (en) | 2023-06-09 |
Family
ID=77412531
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010127230.XA Active CN113326713B (en) | 2020-02-28 | 2020-02-28 | Action recognition method, device, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113326713B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661726A (en) * | 2022-12-26 | 2023-01-31 | 江苏中车数字科技有限公司 | Autonomous video acquisition and analysis method for rail train workpiece assembly |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205080500U (en) * | 2015-09-07 | 2016-03-09 | 哈尔滨市一舍科技有限公司 | Electronic equipment with 3D subassembly of making a video recording |
| CN107784481A (en) * | 2017-08-30 | 2018-03-09 | 平安科技（深圳）有限公司 | Task timeliness method for early warning and device |
| CN110111016A (en) * | 2019-05-14 | 2019-08-09 | 深圳供电局有限公司 | Precarious position monitoring method, device and the computer equipment of operating personnel |
| JP2020013341A (en) * | 2018-07-18 | 2020-01-23 | コニカミノルタ株式会社 | Work process management system, work process management method, and work process management program |
| CN110738135A (en) * | 2019-09-25 | 2020-01-31 | 艾普工华科技（武汉）有限公司 | worker work step specification visual identification judgment and guidance method and system |
-
2020
- 2020-02-28 CN CN202010127230.XA patent/CN113326713B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN205080500U (en) * | 2015-09-07 | 2016-03-09 | 哈尔滨市一舍科技有限公司 | Electronic equipment with 3D subassembly of making a video recording |
| CN107784481A (en) * | 2017-08-30 | 2018-03-09 | 平安科技（深圳）有限公司 | Task timeliness method for early warning and device |
| JP2020013341A (en) * | 2018-07-18 | 2020-01-23 | コニカミノルタ株式会社 | Work process management system, work process management method, and work process management program |
| CN110111016A (en) * | 2019-05-14 | 2019-08-09 | 深圳供电局有限公司 | Precarious position monitoring method, device and the computer equipment of operating personnel |
| CN110738135A (en) * | 2019-09-25 | 2020-01-31 | 艾普工华科技（武汉）有限公司 | worker work step specification visual identification judgment and guidance method and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115661726A (en) * | 2022-12-26 | 2023-01-31 | 江苏中车数字科技有限公司 | Autonomous video acquisition and analysis method for rail train workpiece assembly |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113326713B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110196580B (en) | Assembly guidance method, system, server and storage medium | |
| CN109740415B (en) | Vehicle attribute identification method and related product | |
| CN110738135A (en) | worker work step specification visual identification judgment and guidance method and system | |
| WO2021017286A1 (en) | Facial recognition method and apparatus, electronic device and non-volatile computer readable storage medium | |
| KR20200068075A (en) | Remote guidance apparatus and method capable of handling hyper-motion step based on augmented reality and machine learning | |
| CN112950400A (en) | Data processing platform | |
| CN113380088A (en) | Interactive simulation training support system | |
| CN108288025A (en) | A kind of car video monitoring method, device and equipment | |
| CN114495057A (en) | Data acquisition method, electronic device and storage medium | |
| US20180165622A1 (en) | Action analysis device, acton analysis method, and analysis program | |
| CN111857470A (en) | Unattended control method and device for production equipment and controller | |
| CN113326713B (en) | Action recognition method, device, equipment and medium | |
| CN115100563A (en) | Production process interaction and monitoring intelligent scene based on video analysis | |
| CN113888024A (en) | Operation monitoring method and device, electronic equipment and storage medium | |
| CN114513694A (en) | Scoring determination method and device, electronic equipment and storage medium | |
| WO2019127437A1 (en) | Map labeling method and apparatus, and cloud server, terminal and application program | |
| CN112860821A (en) | Human-vehicle trajectory analysis method and related product | |
| WO2022121886A1 (en) | Method and apparatus for identifying dress code for electric power operations | |
| JP2021163293A (en) | Work analyzer and work analysis program | |
| CN115510253A (en) | Transformer substation equipment AR auxiliary maintenance method and terminal based on three-dimensional knowledge graph | |
| CN112183284B (en) | Safety information verification and designated driving order receiving control method and device | |
| CN111881733B (en) | Method and system for judging and guiding worker operation step standard visual identification | |
| EP3432204B1 (en) | Telepresence framework for region of interest marking using headmount devices | |
| CN114694166A (en) | Equipment control method and system, storage medium and electronic device | |
| CN115576434B (en) | AR-based equipment operation guiding method, AR-based equipment operation guiding system and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 266000 No. 1, Minshan Road, Qingdao area, China (Shandong) pilot Free Trade Zone, Qingdao, Shandong Applicant after: CAOS industrial Intelligence Research Institute (Qingdao) Co.,Ltd. Applicant after: Qingdao blue whale Technology Co.,Ltd. Applicant after: Haier digital technology (Qingdao) Co.,Ltd. Applicant after: Karos IoT Technology Co.,Ltd. Address before: Room 257, management committee of Sino German ecological park, 2877 Tuanjie Road, Huangdao District, Qingdao City, Shandong Province, 266510 Applicant before: QINGDAO HAIER INDUSTRIAL INTELLIGENCE RESEARCH INSTITUTE Co.,Ltd. Applicant before: Qingdao blue whale Technology Co.,Ltd. Applicant before: Haier digital technology (Qingdao) Co.,Ltd. Applicant before: Haier CAOS IOT Ecological Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP01 | Change in the name or title of a patent holder |
Address after: 266000 No. 1, Minshan Road, Qingdao area, China (Shandong) pilot Free Trade Zone, Qingdao, Shandong Patentee after: CAOS industrial Intelligence Research Institute (Qingdao) Co.,Ltd. Patentee after: Qingdao blue whale Technology Co.,Ltd. Patentee after: Kaos Digital Technology (Qingdao) Co.,Ltd. Patentee after: Karos IoT Technology Co.,Ltd. Address before: 266000 No. 1, Minshan Road, Qingdao area, China (Shandong) pilot Free Trade Zone, Qingdao, Shandong Patentee before: CAOS industrial Intelligence Research Institute (Qingdao) Co.,Ltd. Patentee before: Qingdao blue whale Technology Co.,Ltd. Patentee before: Haier digital technology (Qingdao) Co.,Ltd. Patentee before: Karos IoT Technology Co.,Ltd. |
|
| CP01 | Change in the name or title of a patent holder | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20230810 Address after: 266000 No. 1, Minshan Road, Qingdao area, China (Shandong) pilot Free Trade Zone, Qingdao, Shandong Patentee after: CAOS industrial Intelligence Research Institute (Qingdao) Co.,Ltd. Patentee after: Foshan Shunde Haier Intelligent Electronic Co.,Ltd. Patentee after: Kaos Digital Technology (Qingdao) Co.,Ltd. Patentee after: Karos IoT Technology Co.,Ltd. Address before: 266000 No. 1, Minshan Road, Qingdao area, China (Shandong) pilot Free Trade Zone, Qingdao, Shandong Patentee before: CAOS industrial Intelligence Research Institute (Qingdao) Co.,Ltd. Patentee before: Qingdao blue whale Technology Co.,Ltd. Patentee before: Kaos Digital Technology (Qingdao) Co.,Ltd. Patentee before: Karos IoT Technology Co.,Ltd. |
|
| TR01 | Transfer of patent right |