CN112084898B - Assembly operation action recognition method based on static and dynamic separation - Google Patents
Assembly operation action recognition method based on static and dynamic separation Download PDFInfo
- Publication number
- CN112084898B CN112084898B CN202010863071.XA CN202010863071A CN112084898B CN 112084898 B CN112084898 B CN 112084898B CN 202010863071 A CN202010863071 A CN 202010863071A CN 112084898 B CN112084898 B CN 112084898B
- Authority
- CN
- China
- Prior art keywords
- gesture
- action
- finger
- value
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000009471 action Effects 0.000 title claims abstract description 139
- 238000000034 method Methods 0.000 title claims abstract description 57
- 230000003068 static effect Effects 0.000 title claims abstract description 39
- 238000000926 separation method Methods 0.000 title claims abstract description 35
- 238000012549 training Methods 0.000 claims abstract description 38
- 238000000605 extraction Methods 0.000 claims abstract description 14
- 210000003811 finger Anatomy 0.000 claims description 120
- 230000008859 change Effects 0.000 claims description 50
- 238000005452 bending Methods 0.000 claims description 36
- 238000004364 calculation method Methods 0.000 claims description 26
- 239000013598 vector Substances 0.000 claims description 24
- 238000006073 displacement reaction Methods 0.000 claims description 13
- 210000003813 thumb Anatomy 0.000 claims description 10
- 210000004932 little finger Anatomy 0.000 claims description 6
- 238000012360 testing method Methods 0.000 description 14
- 238000002474 experimental method Methods 0.000 description 10
- 238000010586 diagram Methods 0.000 description 9
- 238000011160 research Methods 0.000 description 7
- 230000009467 reduction Effects 0.000 description 5
- 238000004458 analytical method Methods 0.000 description 4
- 238000010276 construction Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 210000004247 hand Anatomy 0.000 description 3
- 238000007781 pre-processing Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000013139 quantization Methods 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- FGUUSXIOTUKUDN-IBGZPJMESA-N C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 Chemical compound C1(=CC=CC=C1)N1C2=C(NC([C@H](C1)NC=1OC(=NN=1)C1=CC=CC=C1)=O)C=CC=C2 FGUUSXIOTUKUDN-IBGZPJMESA-N 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000013075 data extraction Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000000284 resting effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000011425 standardization method Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
- 238000002759 z-score normalization Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
- G06F18/24147—Distances to closest patterns, e.g. nearest neighbour classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/29—Graphical models, e.g. Bayesian networks
- G06F18/295—Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Psychiatry (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses an assembly operation action recognition method based on static and dynamic separation, which specifically comprises the following steps: collecting action gestures, and dividing original data corresponding to the action gestures into training samples and recognition samples; extracting effective data segments from the collected original data of the action gestures; calculating a characteristic threshold according to the action gestures in the training sample, and dividing the action gestures in the recognition sample according to the characteristic threshold; and respectively inputting the characteristic values of the gesture unchanged and gesture changed in the divided recognition samples into a KNN recognition model and a GMM-HMM recognition model for training to respectively obtain the recognition models of the gesture unchanged and gesture changed. According to the gesture recognition method, the characteristic values of the gesture invariant type gesture and the gesture variant type gesture are analyzed, the extraction of the threshold is completed, the gesture types are effectively distinguished according to the threshold, the recognition models of the gesture are respectively obtained, and the recognition accuracy and the recognition speed are improved.
Description
Technical Field
The invention belongs to the technical field of gesture motion recognition methods, and relates to an assembly operation motion recognition method based on static and dynamic separation.
Background
The gesture recognition technology is used as a key technology for realizing an emerging intelligent man-machine interaction system and a virtual reality system, and research theory and method of the gesture recognition technology are widely focused at home and abroad.
Whether based on gesture recognition research of wearable equipment or based on gesture recognition of computer vision, the recognition process of the wearable equipment comprises five steps of gesture data acquisition, gesture feature recognition, gesture tracking, gesture classification and gesture instruction mapping. The research of methods such as gesture feature recognition, gesture tracking, gesture classification and the like is a key for solving the problem of limb or gesture recognition, and high attention of students is obtained. Algorithms such as hidden Markov, support vector machine, neural network and deep learning become a main means of gesture recognition, and after a part of algorithms are improved to a certain extent, the accuracy of recognition can reach more than 95%, so that a solid foundation is laid for further research of human-machine cooperation.
In general, the research results about gesture recognition are quite rich and comprehensive, but have a few defects. In the aspects of data preprocessing and feature extraction, most researches are to directly set features for training, learning and recognition, and the feature values are not analyzed, so that the recognition effects can be distinguished due to the difference of the feature values; in the aspect of gesture recognition methods, most researches adopt a single recognition scheme, and the validity of the recognition scheme is rarely verified.
Disclosure of Invention
The invention aims to provide an assembly operation action recognition method based on static and dynamic separation, which is used for analyzing characteristic values of two gestures of a gesture invariant type and a gesture variant type to finish extraction of a threshold value, effectively distinguishing gesture types according to the threshold value, respectively obtaining recognition models of the two gestures, and improving recognition accuracy and recognition speed.
The technical scheme adopted by the invention is that the assembly operation action recognition method based on static and dynamic separation is implemented according to the following steps:
step 1, dividing a complete operation action into n action gestures, collecting M groups of original data by each action gesture, and dividing the original data corresponding to the action gesture into a training sample and an identification sample;
step 2, extracting effective data segments from the original data of each action gesture acquired in the step 1;
step 3, calculating a characteristic threshold according to the action gestures in the training sample, and dividing the action gestures in the recognition sample into gesture-changing gestures and gesture-unchanged gestures according to the characteristic threshold;
step 4, inputting the characteristic value of the gesture with unchanged gesture in the recognition sample divided in the step 3 into a KNN recognition model for training to obtain a recognition model of the gesture with unchanged gesture;
and 5, inputting the characteristic values of the gesture in the gesture change type gesture in the recognition sample divided in the step 3 into a GMM-HMM recognition model for training to obtain a recognition model of the gesture change type gesture.
The present invention is also characterized in that,
the raw data of each action gesture in step 1 includes five finger tip coordinates of a human hand: f (F) ti (i=1, 2,3,4, 5), five heel joint coordinates of a human hand:finger length:Palm point coordinates: p, palm normal vector: h, pointing to the inner side of the palm, and pointing to the palm: f, the palm points to the direction of the finger, and the speed of the finger tips of five fingers: v (V) ti (i=1, 2,3,4, 5), palm rate: v (V) z 。
The extracting of the effective data segment in the step 2 specifically comprises the following steps:
setting a speed threshold V, respectively taking out an active segment interval with the speed value of palm center speed, thumb tip speed, index finger tip speed, middle finger tip speed, ring finger tip speed and little finger tip speed not smaller than the threshold V in continuous N frames, and taking the active segment interval in the original data as effective data to finish the extraction of the effective segment data, wherein M groups of the original data of the same action gesture correspondingly extract M groups of effective segment data.
The step 3 is specifically as follows: step 3.1, dividing each action gesture in the training sample into a gesture change type and a gesture invariant type according to the gesture of each action gesture;
step 3.2, calculating the upward pitching angle alpha of each finger plane in each frame of action gesture in each effective data segment corresponding to each action gesture according to the effective segment data extracted in the step 2 i Finger opening degree beta i Bending μ of finger i And as a feature value, i=1, 2,3,4,5, each motion gesture j-th frame gesture feature describes a 15-dimensional feature vector:
O j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 )；
step 3.2, obtaining a difference value by taking the difference between the maximum value and the minimum value corresponding to each feature value in the N frames of motion gestures in the same effective data segment corresponding to the same motion gesture in the training sample, taking the difference value as a gesture state change amount feature value, and marking the obtained gesture state change amount feature vector as:
C m ＝(α 1c ，...，α 5c ，β 1c ，...，β 5c ，μ 1c ，...，μ 5c )
wherein m=1, 2, M, C m Representing a gesture state change amount feature vector corresponding to the mth valid data segment corresponding to the same action gesture, alpha 1c ，...，α 5c Respectively representing the minimum value of the upward inclination of each finger plane in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture; beta 1c ，...，β 5c Respectively representing the extremely poor value of each finger opening degree in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture; mu (mu) 1c ，...，μ 5c Respectively representing the extremely-difference value of the bending degree of each finger in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture;
step 3.3, comparing each characteristic value in the M characteristic vectors of the gesture state change amounts corresponding to each action gesture to obtain a maximum value and a minimum value of a difference value of the upward inclination of each finger plane, a difference value of the bending of the finger and a difference value of the bending of the finger corresponding to each action gesture respectively;
step 3.4, assuming that the training sample divided in step 3.1 has a gesture-changing action gestures, b gesture-unchanged action gestures, comparing the maximum value of the difference value of the upward degree of each finger plane, the difference value of the bending degree of the finger and the difference value of the bending degree of the finger corresponding to the b gesture-invariant action gestures divided in the step 3.1, taking the maximum value of the maximum values of the b difference values, comparing the corresponding difference value of the upward inclination degree of each finger plane, the difference value of the bending degree of the finger and the minimum value of the difference value of the bending degree of the finger, which are divided in the step 3.1, taking the minimum value of the minimum values of the a difference values, taking the difference between the maximum value of the maximum values of the b difference values corresponding to each characteristic value and the minimum value of the minimum values of the a difference values, taking the characteristic that the difference value is smaller than 0 as a threshold value distinguishing characteristic, and taking the gesture difference value corresponding to the characteristic as a characteristic threshold value;
and 3.5, selecting a feature threshold corresponding to any feature, calculating a very poor value corresponding to the feature of the action gesture in the recognition sample, if the feature threshold is large, recognizing the action gesture corresponding to the sample as a gesture change type gesture, and if the feature threshold is small, recognizing the action gesture corresponding to the sample as a gesture non-change type gesture.
Alpha in step 3.2 1c ，...，α 5c The calculation method comprises the following steps: alpha ic ＝α maxc -α minc ；
Wherein i=1, 2,3,4,5, α maxc Representing maximum value of upward pitch of finger plane corresponding to N frames of motion gestures in the same effective data segment, alpha minc Representing the minimum value of the upward pitch of the finger plane corresponding to N frames of action gestures in the same effective data segment;
β 1c ，...，β 5c the calculation method comprises the following steps: beta ic ＝β maxc -β minc ；
Wherein beta is maxc Representing the maximum value, beta, of the finger opening corresponding to N frames of motion gestures in the same effective data segment minc Representing the minimum value of the finger opening corresponding to the N frames of action gestures in the same effective data segment;
μ 1c ，...，μ 5c the calculation method comprises the following steps: mu (mu) ic ＝μ maxc -μ minc ；
Wherein mu maxc Represents the maximum value, mu, of the bending degree of the finger corresponding to N frames of motion gestures in the same effective data segment minc And representing the minimum value of the finger bending corresponding to the N frames of action gestures in the same effective data segment.
The calculation method of the upward camber of the finger plane comprises the following steps:
the calculation method of the finger opening degree comprises the following steps:
when i=5, let i+1=1, then β 5 The angle value between the thumb and the little finger;
the calculation method of the finger curvature comprises the following steps:
L i the length of a finger refers to the total length of three segments of condyles from the fingertip to the root joint of the finger.
The step 4 is specifically as follows:
the KNN algorithm is adopted as a static gesture recognition algorithm, a KNN recognition model is established, and 15-dimensional feature vectors of the gesture with the unchanged gesture in the recognition sample divided in the step 3 are extracted, namely: the j-th frame gesture feature of the motion gesture describes a 15-dimensional feature vector:
O j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 )
and then the gesture recognition model is sent into a KNN model for training, and a gesture recognition model with unchanged gesture is obtained.
The step 5 is specifically as follows:
establishing a GMM-HMM recognition model, and extracting 19-dimensional feature vectors of gesture change type gestures in recognition samples divided in the step 3, namely, gesture features of a j frame of action gestures:
g j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 ，w hj ，w fj ，a j ，b j )
then the gesture is sent into a GMM-HMM recognition model for training to obtain a gesture recognition model with a gesture of a gesture changing type, wherein w hj Representing the palm direction h of the current frame j Direction h corresponding to the palm center of the previous frame j+1 Included angle w of fj Representing the palm direction f of the current frame j With the palm direction f of the previous frame j+1 Is included in the plane of the first part; a, a j 、b j Two characteristic values of a two-dimensional displacement characteristic.
w hj The calculation method of (1) is as follows:
w fj the calculation method of (1) is as follows:
the two-dimensional displacement characteristic determining method comprises the following steps:
removing Y-axis data in the palm coordinate P three-dimensional data, taking the palm coordinate of the previous frame of continuous data as an origin, and taking the palm coordinate P of the current frame as an origin xzj Projecting to an XOZ plane chain code disc to obtain P xzj The region number where the projection is located is used as the chain code of the current frameValue a j ；
Removing Z-axis data in the palm coordinate P three-dimensional data, taking the palm coordinate of the previous frame of continuous data as an origin, and taking the next frame of data P xyj Projecting to an XOY plane chain code disk to obtain P xyj The region number where the projection is located is used as the chain code value b of the current frame j To form two-dimensional displacement characteristics [ a ] j ,b j ]；
The specific calculation is as follows:
where j=1, 2, …, N.
The beneficial effects of the invention are as follows: according to the invention, the hand description features are divided into hand displacement features, rotation features and gesture features, the original data are divided into gesture invariant and gesture variant, the state of the current hand and the change of the hand state in a continuous sequence can be completely described, the motion data can be effectively segmented by adopting a speed threshold, and the extraction task of effective segment data is completed; by analyzing the characteristic values of the two gestures, the characteristic types and the threshold values which are suitable for distinguishing the two different types of gestures are obtained, the extraction of the state change quantity threshold values is completed, the correct distinguishing of test data is realized, the distinguishing of the gesture types is effectively carried out, the modeling is carried out on the data which are not deformed in state by adopting a gesture recognition method (KNN) facing a single frame, the modeling is carried out on the state change type data by adopting a modeling method of GMM-HMM, the construction of a static and dynamic separation recognition model is completed, and the recognition accuracy and recognition speed are improved.
Drawings
FIG. 1 is a flow chart of an overall method for identifying assembly operation actions based on static and dynamic separation in the invention;
FIG. 2 is a diagram depicting palm orientation in the method for identifying assembly work actions based on static and dynamic separation of the present invention;
FIG. 3 is a diagram of a chain code wheel in the method for identifying the assembly operation action based on static and dynamic separation;
FIG. 4 is a schematic diagram of the GMM-HMM algorithm recognition in the static and dynamic separation-based assembly operation action recognition method of the present invention;
FIG. 5 is a plot of x, y, z coordinates of data points of a human hand in a resting state in an example of an assembly operation action recognition method based on static and dynamic separation in accordance with the present invention;
FIG. 6 is a plot of x, y, z coordinates of hand data points processed by a window moving average method in an example of an assembly operation action recognition method based on static and dynamic separation in accordance with the present invention;
FIG. 7 is a principal component analysis chart of PCA dimension reduction data in an example of an assembly operation action recognition method based on static and dynamic separation;
FIG. 8 is a diagram of valid segment data extraction in an example of an assembly job action recognition method based on static and dynamic separation in accordance with the present invention;
FIG. 9 is a graph comparing the finger lift with the characteristic values of the fingertip angles of nine gestures in an example of the assembly operation action recognition method based on static and dynamic separation;
FIG. 10 is a graph comparing nine characteristic values of bending of a gesture finger in an example of an assembly operation action recognition method based on static and dynamic separation according to the present invention;
FIG. 11 is a graph of nine characteristic hand orientation changes in an example of an assembly job action recognition method based on static and dynamic separation according to the present invention;
FIG. 12 is a graph of nine direction angle quantization characteristic values in an example of the method for recognizing the assembly operation action based on static and dynamic separation according to the present invention;
FIG. 13 is a graph of nine gesture single feature range comparisons in an example of an assembly job action recognition method based on static and dynamic separation of the present invention;
FIG. 14 is a diagram of classification of test data in an example of an assembly job action recognition method based on static and dynamic separation in accordance with the present invention.
Detailed Description
The invention will be described in detail below with reference to the drawings and the detailed description.
The invention relates to an assembly operation action recognition method based on static and dynamic separation, which is implemented according to the following steps, wherein the flow of the assembly operation action recognition method is shown in figure 1:
step 1, dividing a complete operation action into n action gestures, collecting M groups of original data by each action gesture, and dividing the original data corresponding to the action gesture into a training sample and an identification sample; wherein, the raw data of each action gesture comprises five finger tip coordinates of a human hand: f (F) ti (i=1, 2,3,4, 5), five heel joint coordinates of a human hand: f (F) bi (i=1, 2,3,4, 5), finger length: l (L) i (i=1, 2,3,4, 5), palm point coordinates: p, palm normal vector: h, pointing to the inner side of the palm, and pointing to the palm: f, the palm points to the direction of the finger, as shown in fig. 2, the speed of the finger tips of the five fingers: v (V) ti (i=1, 2,3,4, 5), palm rate: v (V) z ；
Step 2, extracting effective data segments of the original data of each action gesture acquired in the step 1, specifically:
setting a speed threshold V, respectively taking out an active segment interval with the speed value of not less than the threshold V in N continuous frames, namely the palm center speed, the thumb tip speed, the index finger tip speed, the middle finger tip speed, the ring finger tip speed and the little finger tip speed, and taking the active segment interval in the original data as effective data to finish the extraction of the effective segment data, wherein M groups of the original data of the same action gesture correspondingly extract M groups of effective segment data;
step 3, calculating a characteristic threshold according to the action gestures in the training sample, and dividing the action gestures in the recognition sample into gesture-changing gestures and gesture-unchanged gestures according to the characteristic threshold;
the method comprises the following steps:
step 3.1, dividing each action gesture in the training sample into a gesture change type and a gesture invariant type according to the gesture of each action gesture;
step 3.2, calculating the upward pitching angle alpha of each finger plane in each frame of action gesture in each effective data segment corresponding to each action gesture according to the effective segment data extracted in the step 2 i Finger opening degree beta i AndBending mu of finger i And as a feature value, i=1, 2,3,4,5, each motion gesture j-th frame gesture feature describes a 15-dimensional feature vector:
O j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 )；
step 3.2, obtaining a difference value by taking the difference between the maximum value and the minimum value corresponding to each feature value in the N frames of motion gestures in the same effective data segment corresponding to the same motion gesture in the training sample, taking the difference value as a gesture state change amount feature value, and marking the obtained gesture state change amount feature vector as:
C m ＝(α 1c ，...，α 5c ，β 1c ，...，β 5c ，μ 1c ，...，μ 5c )
wherein m=1, 2, M, C m Representing a gesture state change amount feature vector corresponding to the mth valid data segment corresponding to the same action gesture, alpha 1c ，...，α 5c Respectively representing the minimum value of the upward inclination of each finger plane in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture; beta 1c ，...，β 5c Respectively representing the extremely poor value of each finger opening degree in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture; mu (mu) 1c ，...，μ 5c Respectively representing the extreme value of the bending degree of each finger in N frames of motion gestures in the mth effective data segment corresponding to the same motion gesture, wherein alpha 1c ，...，α 5c The calculation method comprises the following steps:
α ic ＝α maxc -α minc ；
wherein i=1, 2,3,4,5, α maxc Representing maximum value of upward pitch of finger plane corresponding to N frames of motion gestures in the same effective data segment, alpha minc Representing the minimum value of the upward pitch of the finger plane corresponding to N frames of action gestures in the same effective data segment;
β 1c ，...，β 5c the calculation method comprises the following steps: beta ic ＝β maxc -β minc ；
Wherein beta is maxc Representing the maximum value, beta, of the finger opening corresponding to N frames of motion gestures in the same effective data segment minc Representing the minimum value of the finger opening corresponding to the N frames of action gestures in the same effective data segment;
μ 1c ，...，μ 5c the calculation method comprises the following steps: mu (mu) ic ＝μ maxc -μ minc ；
Wherein mu maxc Represents the maximum value, mu, of the bending degree of the finger corresponding to N frames of motion gestures in the same effective data segment minc Representing the minimum value of the finger bending corresponding to the N frames of action gestures in the same effective data segment;
the calculation method of the upward camber of the finger plane comprises the following steps:
the calculation method of the finger opening degree comprises the following steps:
when i=5, let i+1=1, then β 5 The angle value between the thumb and the little finger;
the calculation method of the finger curvature comprises the following steps:
L i the length of the finger is the total length of three sections of condyles from the fingertip to the root joint of the finger;
step 3.3, comparing each characteristic value in the M characteristic vectors of the gesture state change amounts corresponding to each action gesture to obtain a maximum value and a minimum value of a difference value of the upward inclination of each finger plane, a difference value of the bending of the finger and a difference value of the bending of the finger corresponding to each action gesture respectively;
step 3.4, assuming that the training sample divided in step 3.1 has a gesture-changing action gestures, b gesture-unchanged action gestures, comparing the maximum value of the difference value of the upward degree of each finger plane, the difference value of the bending degree of the finger and the difference value of the bending degree of the finger corresponding to the b gesture-invariant action gestures divided in the step 3.1, taking the maximum value of the maximum values of the b difference values, comparing the corresponding difference value of the upward inclination degree of each finger plane, the difference value of the bending degree of the finger and the minimum value of the difference value of the bending degree of the finger, which are divided in the step 3.1, taking the minimum value of the minimum values of the a difference values, taking the difference between the maximum value of the maximum values of the b difference values corresponding to each characteristic value and the minimum value of the minimum values of the a difference values, taking the characteristic that the difference value is smaller than 0 as a threshold value distinguishing characteristic, and taking the gesture difference value corresponding to the characteristic as a characteristic threshold value;
and 3.5, selecting a feature threshold corresponding to any feature, calculating a very poor value corresponding to the feature of the action gesture in the recognition sample, if the feature threshold is large, recognizing the action gesture corresponding to the sample as a gesture change type gesture, and if the feature threshold is small, recognizing the action gesture corresponding to the sample as a gesture non-change type gesture.
Step 4, inputting the characteristic value of the gesture with unchanged gesture in the recognition sample divided in the step 3 into a KNN recognition model for training to obtain a recognition model of the gesture with unchanged gesture; the method comprises the following steps: the KNN algorithm is adopted as a static gesture recognition algorithm, a KNN recognition model is established, and 15-dimensional feature vectors of the gesture with the unchanged gesture in the recognition sample divided in the step 3 are extracted, namely: the j-th frame gesture feature of the motion gesture describes a 15-dimensional feature vector:
O j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 )
then the gesture recognition model is sent into a KNN model for training, and a gesture recognition model with unchanged gesture is obtained;
step 5, inputting the characteristic values of the gesture in the gesture change type gesture in the recognition sample divided in the step 3 into a GMM-HMM recognition model for training to obtain a gesture change type gesture recognition model, wherein a GMM-HMM algorithm recognition schematic diagram is shown in fig. 4, and specifically comprises the following steps:
establishing a GMM-HMM recognition model, and extracting 19-dimensional feature vectors of gesture change type gestures in recognition samples divided in the step 3, namely, gesture features of a j frame of action gestures:
g j ＝f j (α 1 ，...，α 5 ，β 1 ，...，β 5 ，μ 1 ，...，μ 5 ，w hj ，w fj ，a j ，b j )
then the gesture is sent into a GMM-HMM recognition model for training to obtain a gesture recognition model with a gesture of a gesture changing type, wherein w hj Representing the palm direction h of the current frame j Direction h corresponding to the palm center of the previous frame j+1 Included angle w of fj Representing the palm direction f of the current frame j With the palm direction f of the previous frame j+1 Is included in the plane of the first part; a, a j 、b j Two eigenvalues of a two-dimensional displacement feature, where w hj The calculation method of (1) is as follows:
w fj the calculation method of (1) is as follows:
the two-dimensional displacement characteristic determining method comprises the following steps:
removing Y-axis data in the palm coordinate P three-dimensional data, taking the palm coordinate of the previous frame of continuous data as an origin, and taking the palm coordinate P of the current frame as an origin xzj The projected XOZ plane chain code disc, as shown in fig. 3, is 16 sector discs equally divided on the disc, and P is obtained xzj The region number where the projection is located is used as the chain code value a of the current frame j ；
Removing Z-axis data in palm coordinate P three-dimensional data to continuously data the palm sitting position of the previous frameMarked as origin, the next frame of data P xyj Projecting to an XOY plane chain code disk to obtain P xyj The region number where the projection is located is used as the chain code value b of the current frame j To form two-dimensional displacement characteristics [ a ] j ，b j ]The method comprises the steps of carrying out a first treatment on the surface of the The specific calculation is as follows:
where j=1, 2, …, N.
Examples
In the assembly operation scene, nine gesture motion libraries are obtained by analyzing common bolt assembly cases and more complex part of ECU assembly cases of manufacturing enterprises, as shown in Table 1.
Table 1 hand-held action library
1. Action classification
For the summarized 9 hand-shaped actions, the hand-shaped actions are classified into gesture-changing type and gesture-unchanged type according to three description features of the hand displacement features, the rotation features and the gesture features, and specific classifications are shown in table 2.
TABLE 2 action classification
2. Data acquisition
And carrying out data acquisition according to the hand shape of the set operation action gesture library, acquiring gesture data of 4 subjects in total in experiments, wherein each person acquires 9 gestures defined in the text, each gesture comprises 50 samples, after the acquisition is completed, integrating all sample data and storing the sample data in a CSV file, 360 samples in 450 samples acquired by each person are used for training a model, and the rest 90 samples are used for testing the accuracy of the model. 26 parameters including the speed, position and time of the palm center and five fingers are collected through the Leap Motion. The names and meanings of the parameters at the time of experimental data collection are shown in Table 3.
Table 3 collection parameter annotation table
3. Data preprocessing
Firstly, extracting the characteristics of original data, then adopting a certain data dimension reduction method to solve the problem of multi-characteristic data redundancy, reducing the data magnitude, and finally using a data standardization method to improve the comparability of the data, wherein the main adopted methods are data regularity, PCA dimension reduction and Z-SCORE standardization.
(1) Denoising data: as shown in FIG. 5, a-c in FIG. 5 are respectively line diagrams of three-dimensional x, y and z coordinate values of the thumb root of continuous 1000 frames of data in a certain static state, and as shown in FIG. 5, the equipment acquisition values have the problems of larger fluctuation, discontinuous data acquisition values, too many data peak values and the like, and do not accord with the actual hand movement coordinate conditions. The original value is smoothed by using a K-approach method based on distance detection, the K-approach method filters the original data by adopting a window moving average mode, and the average value of continuous data frames is used for replacing the original value, so that the preprocessing of the data is completed. The K adjacent value parameter is taken as 20, the coordinate broken line processed by the window moving average method is shown in fig. 6, and the processed X, Y and Z coordinate data are closer to the actual condition and are more continuous.
(2) PCA dimension reduction: for the 15-dimensional pose invariant hand pose data of this example, the feature dimension is reduced from 15 dimensions to 5 dimensions by principal component analysis after dimension reduction as shown in fig. 7.
(3) Z-Score normalization: when the human hand expression model has a part of parameters as angle values, the acquired data are different in size due to the personalized difference of human hands when facing a new sample, and the recognition error caused by the personalized difference is reduced by a Z-Score method.
4. Analysis and acquisition of hand state thresholds
(1) Effective operation section segmentation and acquisition: the example sets the data segment with the continuous 20 frames larger than the 20000 speed threshold as the effective data segment in the operation sequence, and realizes the data segmentation of continuous operation. Taking screwing motion consisting of six motion gestures of hand extension, grabbing, moving, sleeving, rotating and releasing as an example, the example firstly obtains speed values of a palm center and a five finger fingertip, takes out 6 movable section intervals with speed values larger than 20000 in 20 continuous frames, and then takes the data of the section interval in the original data as effective data to finish the extraction of the effective section data. As shown in fig. 8, (a) in fig. 8 is the speed of five fingertips and palms of the raw data of the screwing operation, (b) is the length of each active segment interval after the extraction of the active segment, and (c) is the speed of the active segment after the extraction. As can be taken from fig. 8, the motion data can be effectively segmented by using the speed threshold, and the task of extracting the effective segment data is completed.
(2) Action threshold extraction based on morphological feature change: the example obtains a characteristic parameter image suitable for distinguishing gesture with unchanged gesture from gesture change type through experiments on the basis of the gesture feature, the hand displacement feature and the hand rotation feature, and compares and calculates characteristic values of two gesture state change amounts of a training sample to obtain a characteristic value most suitable for distinguishing threshold characteristic types. The 15-dimensional gesture features of the nine actions summarized herein are classified, extracted and plotted according to the kinds of feature values. The finger-to-finger angle for the nine actions is shown as f in FIG. 9 11 To f 15 The finger lift is shown as f in the figure 31 To f 35 The abscissa in the figure is a time frame, and the ordinate is an angle value. As shown in fig. 9, a-f are angle characteristic values of six gestures including hand extension, object movement, sleeving, pressing in, pulling out and prying, g-i are angle characteristic values of three gestures including grabbing, rotating and releasing, respectively, as shown in fig. 9, the first six gestures are gesture without deformation of the gesture defined in the example, and the last three are gesture with deformation of the gesture. In FIG. 9 aIn the step f, the single-feature angle change value is smaller, so that the angle information is not obviously changed when the motion gesture moves, and in the step g-i of fig. 9, the single-feature angle change amplitude is larger, so that the angle information is obviously changed in the movement process. The bending degree of the finger is shown in fig. 10, and compared with the angle information in fig. 9, the bending degree of the finger in the first six diagrams in fig. 10 and the bending degree change information in the last three diagrams are not large, and the single characteristic range is within the change range of 0.4. However, for the front-back comparison, the change amplitude of a certain bending characteristic value of grabbing, rotating and releasing hands can still be obtained, and the change amplitude of bending of six gestures of stretching hands, moving objects, sleeving in, pressing in, pulling out and prying is obviously larger. In addition to the gesture feature, the example compares the hand orientation change value feature of nine actions with the direction angle quantization value, where the hand orientation change value feature is shown in fig. 11, and the abscissa in the figure is a time frame, and the ordinate is an angle value. The quantized value characteristic of the direction angle is shown in fig. 12, in which the abscissa represents a time frame and the ordinate represents a chain code value. The hand orientation feature and direction angle quantization value feature difference between the gesture of the gesture variation type and the gesture of the gesture non-variation type are small relative to the 15-dimensional human hand state feature, and thus it is difficult to distinguish by means of the two feature gestures. Through the above analysis, 15 gesture features may be temporarily determined, suitable as feature threshold categories to distinguish between two data types. The example carries out the extremely-poor value operation processing on the collected 40 multiplied by 5 group training sample data characteristic values, and obtains the characteristic value of the gesture state change quantity by calculating the difference between the maximum value and the minimum value in the effective section of the data. At this time, the very poor comparison value of the gesture characteristics of the gesture invariant gesture and the gesture variant gesture in the multiple groups of data can be obtained, and the comparison result is shown in table 4.
Table 4 gesture characteristics extremely poor contrast table of gesture invariant type gesture and gesture variant type gesture
The features 1, 4, 7, 10 and 13 compared in the table are respectively upper elevation angle feature extreme values of the thumb, the index finger, the middle finger, the ring finger and the pinky, the features 2, 5, 8, 11 and 14 are respectively bending ratio feature extreme values of the thumb, the index finger, the middle finger, the ring finger and the pinky, and the features 3, 6, 9, 12 and 15 are respectively three-dimensional angle feature extreme values of the thumb and the index finger, the index finger and the middle finger, the middle finger and the ring finger, and the pinky. The maximum, average and minimum values in the table refer to the maximum, average and minimum values of the characteristic difference values of the gesture samples in the total 160 groups of training samples of 4×40. Therefore, the visual observation shows that the characteristics of different gestures are greatly different, and the gesture can be effectively characterized.
In order to better compare the above features, the hand gesture difference value of the maximum value of the extreme differences in the previous six gesture non-deformation gestures is taken as a boundary line, and the maximum value of the first six gesture features and the minimum value of the last three gesture features are selected for drawing, so that a single feature comparison result is shown in fig. 13.
For all data sets, the example adopts the difference between the maximum value of the gesture with unchanged gesture and the minimum value of the gesture with changed gesture, and takes the characteristic with the difference less than 0 as the threshold distinguishing characteristic. The differences are shown in table 5.
TABLE 5 gesture-invariant gesture and gesture-variant gesture list feature difference List
As can be seen from the above analysis, when the maximum value of the single feature of the gesture range of the gesture with unchanged gesture is taken to be different from the minimum value of the single feature of the gesture with unchanged gesture, the feature 1, the feature 4 and the feature 13 can be obtained to be the gesture range of the gesture with unchanged gesture greater than the gesture range of the gesture with unchanged gesture, so that the feature 1, the feature 4 and the feature 13 are taken as the data types for distinguishing the gesture with unchanged gesture from the gesture with unchanged gesture, and at the moment, the three feature thresholds are respectively the gesture range values of 38.15, 30.50 and 14.6.
Test set 40 sets of data were tested using feature 1, feature 4, and feature 13, with the test results shown in fig. 14. In the figure, nine kinds of motion data of extending, moving, telescoping, pushing in, pulling out, prying, grabbing, rotating, and releasing are spliced in order, and then a threshold test experiment is performed, fig. 14 (a) and 14 (b) are differential graphs for classifying data by means of a feature value 1 and a feature value 4, and fig. 14 (c) is a data classification graph for a feature value 13. From fig. a and b, it can be derived that feature 1 and feature 4 have good distinguishability for the grab and release gestures in the gesture variant from the gesture invariant gestures. From fig. 14 (c), it can be derived that the feature 13 has good distinction between the gesture-change type rotation gesture and the gesture-non-change type gesture.
From the above test results, it can be obtained that the threshold value adopted can achieve 100% correct discrimination of test data in 360 samples in total of 4×10×9 of the experimental test. Thus, by combining 3 feature thresholds, it is possible to effectively distinguish gesture types.
5. Recognition experiment and recognition result analysis
5.1 gesture experiment with unchanged gesture
In the case experiment, firstly, the 6 gesture data which are obtained in advance and have no deformation are processed, then, the training data are preprocessed, the characteristics are extracted, the dimension is reduced, and the single frame data at any position in the effective data section are taken as input data. For gesture recognition with no variation of gesture, the total number of recognition samples is 6×40×4 total 960, and the total of 240 pieces of data of 6×10×4 are used for testing the accuracy of the obtained model.
The example sends the processed data into a KNN model, Z-Score is selected for data processing in KNN model construction, parameter K=5 is used for recognition model construction, the result shows that the model recognition rate reaches 100%, the time consumption is 0.62ms, and the recognition accuracy and recognition speed are greatly improved.
5.2 continuous operation gesture recognition experiments
The experiment is carried out on 3 gesture data with changed gestures, a 3-channel GMM-HMM recognition model is established, and recognition rate is obtained on test sample data. Aiming at a training sample of the GMM-HMM model, the model is different from a KNN model facing gesture recognition with unchanged gesture, and for gesture change type operation gestures, 4 features of hand gesture angle features and hand motion chain code features are added in operation gesture features, so that the hand gesture state features are increased from 15 dimensions to 19 dimensions.
When the GMM-HMM model is trained, one action sample is one data source, so that the total of training data and test data is 3×40×4 and total 480 pieces of 3×10×4 are used for the posture-changing type 3-channel GMM-HMM model test. For the acquired data, a 3-channel GMM-HMM recognition model is established in the experiment, the experimental result is shown in table 6, the recognition rate is 83.3%, and the time consumption is 2.85s.
TABLE 6 Complex actions Using GMM-HMM recognition rate and recognition timetable
| Type of algorithm | Identifying objects | Data processing | Recognition rate | Run time |
| GMM-HMM | Three gesture with changeable gesture | NormalizationTreatment of | 83.3％ | 2.85s |
The static and dynamic separation recognition model is comprehensively analyzed, and the comprehensive recognition rate is as follows: p=p i ×(P j ×6+P k X 3)/9, wherein: p (P) i -threshold discrimination accuracy; p (P) j -a gesture invariant recognition rate; p (P) k -a gesture-change recognition rate. The comprehensive recognition speed is as follows: t= (T i ×6+T j X 3)/9, wherein: t (T) i -six poses are not deformed and are time-consuming to identify; t (T) j Three gesture change type recognition is time-consuming. The comprehensive recognition rate and the speed of the static and dynamic separation recognition model are obtained by calculating the recognition speed and the recognition rate and are shown in table 7.
TABLE 7 comprehensive recognition rate and speedometer for static and dynamic separation recognition model
As can be seen from the table, the comprehensive recognition rate of the gestures of the KNN-GMM-HMM static and dynamic separation recognition model for recognizing nine gestures in the experiment is 94.33%, the comprehensive recognition speed is only 950ms, and it can be seen that the recognition model based on static and dynamic separation is effective.
The method can be used in the assembly operation gesture recognition process with a time sequence, and has the following beneficial effects: 1) According to the invention, the hand description features are divided into hand displacement features, rotation features and gesture features, and the original data are divided into gesture invariant and gesture variant, so that the state of the current hand and the change of the hand state in a continuous sequence can be completely described; 2) The invention adopts the speed threshold to effectively divide the motion data, and completes the extraction task of the effective segment data; 3) According to the method, the characteristic values of the two gestures are analyzed, the characteristic types and the threshold values which are suitable for distinguishing the two different types of gestures are obtained, the extraction of the state change quantity threshold value is completed, the correct distinguishing of the test data is realized, and the types of the gestures are effectively distinguished; 4) When the gesture is identified, the single-frame-oriented gesture identification method (KNN) is adopted for modeling aiming at the data with unchanged state, and the GMM-HMM modeling method is adopted for modeling aiming at the data with changed state, so that the static and dynamic separation identification model is constructed, and the identification accuracy and the identification speed are improved.
Claims (7)
1. The assembling operation action recognition method based on static and dynamic separation is characterized by comprising the following steps of:
step 1, dividing a complete operation action into n action gestures, collecting M groups of original data by each action gesture, and dividing the original data corresponding to the action gesture into a training sample and an identification sample;
step 2, extracting effective data segments from the original data of each action gesture acquired in the step 1;
step 3, calculating a characteristic threshold according to the action gestures in the training sample, and dividing the action gestures in the recognition sample into gesture-changing gestures and gesture-unchanged gestures according to the characteristic threshold;
step 4, inputting the characteristic value of the gesture with unchanged gesture in the recognition sample divided in the step 3 into a KNN recognition model for training to obtain a recognition model of the gesture with unchanged gesture;
step 5, inputting the characteristic values of the gesture in the gesture change type gesture in the recognition sample divided in the step 3 into a GMM-HMM recognition model for training to obtain a recognition model of the gesture change type gesture;
the raw data of each action gesture in the step 1 includes five finger tip coordinates of a human hand: f (F) ti (i=1, 2,3,4, 5), five heel joint coordinates of a human hand: f (F) bi (i=1, 2,3,4, 5), finger length: l (L) i (i=1, 2,3,4, 5), palm point coordinates: p, palm normal vector: h, pointing to the inner side of the palm, and pointing to the palm: f, the palm points to the direction of the finger, and the speed of the finger tips of five fingers: v (V) ti (i=1, 2,3,4, 5), palm rate: v (V) z ；
The extracting of the effective data segment in the step 2 specifically includes:
setting a speed threshold V, respectively taking out an active segment interval with the speed value of not less than the threshold V in N continuous frames, namely the palm center speed, the thumb tip speed, the index finger tip speed, the middle finger tip speed, the ring finger tip speed and the little finger tip speed, and taking the active segment interval in the original data as effective data to finish the extraction of the effective segment data, wherein M groups of the original data of the same action gesture correspondingly extract M groups of effective segment data;
the step 3 specifically comprises the following steps:
step 3.1, dividing each action gesture in the training sample into a gesture change type and a gesture invariant type according to the gesture of each action gesture;
step 3.2, calculating the upward pitching angle alpha of each finger plane in each frame of action gesture in each effective data segment corresponding to each action gesture according to the effective segment data extracted in the step 2 i Finger opening degree beta i Bending μ of finger i And as a feature value, i=1, 2,3,4,5, each motion gesture j-th frame gesture feature describes a 15-dimensional feature vector:
O j ＝f j (α 1 ,…,α 5 ,β 1 ,…,β 5 ,μ 1 ,…,μ 5 )；
step 3.2, obtaining a difference value by taking the difference between the maximum value and the minimum value corresponding to each feature value in the N frames of motion gestures in the same effective data segment corresponding to the same motion gesture in the training sample, taking the difference value as a gesture state change amount feature value, and marking the obtained gesture state change amount feature vector as:
C m ＝(α 1c ,…,α 5c ，β 1c ,…，β 5c ,μ 1c ,…,μ 5c )
wherein m=1, 2, …, M, C m Representing a gesture state change amount feature vector corresponding to the mth valid data segment corresponding to the same action gesture, alpha 1c ,…,α 5c Respectively represent N frames of action gestures in the m-th effective data segment corresponding to the same action gestureA minimum value of the upward pitch of each finger plane; beta 1c ,…,β 5c Respectively representing the extremely poor value of each finger opening degree in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture; mu (mu) 1c ,…,μ 5c Respectively representing the extremely-difference value of the bending degree of each finger in the N frames of action gestures in the mth effective data segment corresponding to the same action gesture;
step 3.3, comparing each characteristic value in the M characteristic vectors of the gesture state change amounts corresponding to each action gesture to obtain a maximum value and a minimum value of a difference value of the upward inclination of each finger plane, a difference value of the bending of the finger and a difference value of the bending of the finger corresponding to each action gesture respectively;
step 3.4, assuming that the training sample divided in step 3.1 has a gesture-changing action gestures, b gesture-unchanged action gestures, comparing the maximum value of the difference value of the upward degree of each finger plane, the difference value of the bending degree of the finger and the difference value of the bending degree of the finger corresponding to the b gesture-invariant action gestures divided in the step 3.1, taking the maximum value of the maximum values of the b difference values, comparing the corresponding difference value of the upward inclination degree of each finger plane, the difference value of the bending degree of the finger and the minimum value of the difference value of the bending degree of the finger, which are divided in the step 3.1, taking the minimum value of the minimum values of the a difference values, taking the difference between the maximum value of the maximum values of the b difference values corresponding to each characteristic value and the minimum value of the minimum values of the a difference values, taking the characteristic that the difference value is smaller than 0 as a threshold value distinguishing characteristic, and taking the gesture difference value corresponding to the characteristic as a characteristic threshold value;
and 3.5, selecting a feature threshold corresponding to any feature, calculating a very poor value corresponding to the feature of the action gesture in the recognition sample, if the feature threshold is large, recognizing the action gesture corresponding to the sample as a gesture change type gesture, and if the feature threshold is small, recognizing the action gesture corresponding to the sample as a gesture non-change type gesture.
2. The static and dynamic separation-based assembly operation action recognition method according to claim 1, characterized in thatIn said step 3.2, α 1c ,…,α 5c The calculation method comprises the following steps:
α ic ＝α maxc -α minc ；
wherein i=1, 2,3,4,5, α maxc Representing maximum value of upward pitch of finger plane corresponding to N frames of motion gestures in the same effective data segment, alpha minc Representing the minimum value of the upward pitch of the finger plane corresponding to N frames of action gestures in the same effective data segment;
β 1c ,…,β 5c the calculation method comprises the following steps:
β ic ＝β maxc -β minc ；
wherein beta is maxc Representing the maximum value, beta, of the finger opening corresponding to N frames of motion gestures in the same effective data segment minc Representing the minimum value of the finger opening corresponding to the N frames of action gestures in the same effective data segment;
μ 1c ,…,μ 5c the calculation method comprises the following steps:
μ ic ＝μ maxc -μ minc ；
wherein mu maxc Represents the maximum value, mu, of the bending degree of the finger corresponding to N frames of motion gestures in the same effective data segment minc And representing the minimum value of the finger bending corresponding to the N frames of action gestures in the same effective data segment.
3. The method for identifying the assembly operation action based on static and dynamic separation according to claim 2, wherein the method for calculating the upward pitch of the finger plane is as follows:
the calculation method of the finger opening degree comprises the following steps:
when i=5Let i+1=1, then β 5 The angle value between the thumb and the little finger;
the calculation method of the finger curvature comprises the following steps:
L i the length of a finger refers to the total length of three segments of condyles from the fingertip to the root joint of the finger.
4. The static and dynamic separation-based assembly operation action recognition method according to claim 2, wherein the step 4 is specifically:
the KNN algorithm is adopted as a static gesture recognition algorithm, a KNN recognition model is established, and 15-dimensional feature vectors of the gesture with the unchanged gesture in the recognition sample divided in the step 3 are extracted, namely: the j-th frame gesture feature of the motion gesture describes a 15-dimensional feature vector:
O j ＝f j (α 1 ,…,α 5 ,β 1 ,…,β 5 ,μ 1 ,…,μ 5 )
and then the gesture recognition model is sent into a KNN model for training, and a gesture recognition model with unchanged gesture is obtained.
5. The static and dynamic separation-based assembly operation action recognition method according to claim 2, wherein the step 5 is specifically:
establishing a GMM-HMM recognition model, and extracting 19-dimensional feature vectors of gesture change type gestures in recognition samples divided in the step 3, namely, gesture features of a j frame of action gestures:
g j ＝f j (α 1 ,…,α 5 ,β 1 ,…,β 5 ,μ 1 ,…,μ 5 ,w j ,w fj ,a j ,b j )
then the gesture is sent into a GMM-HMM recognition model for training to obtain a gesture recognition model with a gesture of a gesture changing type, wherein w hj Representing the palm direction h of the current frame j Direction h corresponding to the palm center of the previous frame j+1 Included angle w of fj Representing the palm direction f of the current frame j With the palm direction f of the previous frame j+1 Is included in the plane of the first part; a, a j 、b j Two characteristic values of a two-dimensional displacement characteristic.
6. The static and dynamic separation-based assembly operation action recognition method according to claim 5, wherein the w hj The calculation method of (1) is as follows:
the w is fj The calculation method of (1) is as follows:
7. the static and dynamic separation-based assembly operation action recognition method according to claim 5, wherein the two-dimensional displacement characteristic determination method is as follows:
removing Y-axis data in the palm coordinate P three-dimensional data, taking the palm coordinate of the previous frame of continuous data as an origin, and taking the palm coordinate P of the current frame as an origin xzj Projecting to an XOZ plane chain code disc to obtain P xzj The region number where the projection is located is used as the chain code value a of the current frame j ；
Removing Z-axis data in the palm coordinate P three-dimensional data, taking the palm coordinate of the previous frame of continuous data as an origin, and taking the next frame of data P xyj Projecting to an XOY plane chain code disk to obtain P xyj The region number where the projection is located is used as the chain code value b of the current frame j To form two-dimensional displacement characteristics [ a ] j ,b j ]；
The specific calculation is as follows:
where j=1, 2, …, N.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010863071.XA CN112084898B (en) | 2020-08-25 | 2020-08-25 | Assembly operation action recognition method based on static and dynamic separation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010863071.XA CN112084898B (en) | 2020-08-25 | 2020-08-25 | Assembly operation action recognition method based on static and dynamic separation |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112084898A CN112084898A (en) | 2020-12-15 |
| CN112084898B true CN112084898B (en) | 2024-02-09 |
Family
ID=73728585
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010863071.XA Active CN112084898B (en) | 2020-08-25 | 2020-08-25 | Assembly operation action recognition method based on static and dynamic separation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112084898B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113111844B (en) * | 2021-04-28 | 2022-02-15 | 中德（珠海）人工智能研究院有限公司 | Operation posture evaluation method and device, local terminal and readable storage medium |
| CN113282167B (en) * | 2021-05-08 | 2023-06-27 | 青岛小鸟看看科技有限公司 | Interaction method and device of head-mounted display equipment and head-mounted display equipment |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109634415A (en) * | 2018-12-11 | 2019-04-16 | 哈尔滨拓博科技有限公司 | It is a kind of for controlling the gesture identification control method of analog quantity |
| CN109993073A (en) * | 2019-03-14 | 2019-07-09 | 北京工业大学 | A kind of complicated dynamic gesture identification method based on Leap Motion |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8620024B2 (en) * | 2010-09-17 | 2013-12-31 | Sony Corporation | System and method for dynamic gesture recognition using geometric classification |
| US10488939B2 (en) * | 2017-04-20 | 2019-11-26 | Microsoft Technology Licensing, Llc | Gesture recognition |
-
2020
- 2020-08-25 CN CN202010863071.XA patent/CN112084898B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109634415A (en) * | 2018-12-11 | 2019-04-16 | 哈尔滨拓博科技有限公司 | It is a kind of for controlling the gesture identification control method of analog quantity |
| CN109993073A (en) * | 2019-03-14 | 2019-07-09 | 北京工业大学 | A kind of complicated dynamic gesture identification method based on Leap Motion |
| CN110837792A (en) * | 2019-11-04 | 2020-02-25 | 东南大学 | Three-dimensional gesture recognition method and device |
Non-Patent Citations (2)
| Title |
|---|
| 杨文璐 ; 乔海丽 ; 谢宏 ; 夏斌 ; .基于Leap Motion和支持向量机的手势识别.传感器与微系统.2018,(第05期),全文. * |
| 陈国良 ; 葛凯凯 ; 李聪浩 ; .基于多特征HMM融合的复杂动态手势识别.华中科技大学学报(自然科学版).2018,(第12期),全文. * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112084898A (en) | 2020-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106682598B (en) | Multi-pose face feature point detection method based on cascade regression | |
| Ibraheem et al. | Survey on various gesture recognition technologies and techniques | |
| Bhuyan et al. | Fingertip detection for hand pose recognition | |
| Shin et al. | Skeleton-based dynamic hand gesture recognition using a part-based GRU-RNN for gesture-based interface | |
| Popli et al. | Classification and recognition of online hand-written alphabets using machine learning methods | |
| Chang et al. | Spatio-temporal hough forest for efficient detection–localisation–recognition of fingerwriting in egocentric camera | |
| CN112084898B (en) | Assembly operation action recognition method based on static and dynamic separation | |
| CN103971102A (en) | Static gesture recognition method based on finger contour and decision-making trees | |
| Hemayed et al. | Edge-based recognizer for Arabic sign language alphabet (ArS2V-Arabic sign to voice) | |
| Jambhale et al. | Gesture recognition using DTW & piecewise DTW | |
| Gupta et al. | FPGA based real time human hand gesture recognition system | |
| Mahmood et al. | A Comparative study of a new hand recognition model based on line of features and other techniques | |
| Nguyen et al. | Geometry-based static hand gesture recognition using support vector machine | |
| CN110717385A (en) | Dynamic gesture recognition method | |
| Ranawat et al. | Hand gesture recognition based virtual mouse events | |
| CN114792443A (en) | Intelligent device gesture recognition control method based on image recognition | |
| Mohammadi et al. | Air-writing recognition system for Persian numbers with a novel classifier | |
| CN105760828A (en) | Visual sense based static gesture identification method | |
| Sooai et al. | Comparison of recognition accuracy on dynamic hand gesture using feature selection | |
| Pradhan et al. | A hand gesture recognition using feature extraction | |
| Fahad et al. | Deep learning-based deaf & mute gesture translation system | |
| CN112101293A (en) | Facial expression recognition method, device, equipment and storage medium | |
| Shitole et al. | Dynamic hand gesture recognition using PCA, Pruning and ANN | |
| Ibraheem | Finger identification and gesture recognition using Gaussian classifier model | |
| Teja et al. | A ballistic stroke representation of online handwriting for recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |