CN117496303A - Gesture recognition model generation method and device - Google Patents
Gesture recognition model generation method and device Download PDFInfo
- Publication number
- CN117496303A CN117496303A CN202311579134.9A CN202311579134A CN117496303A CN 117496303 A CN117496303 A CN 117496303A CN 202311579134 A CN202311579134 A CN 202311579134A CN 117496303 A CN117496303 A CN 117496303A
- Authority
- CN
- China
- Prior art keywords
- recognition model
- gesture recognition
- information
- gesture
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 58
- 238000012549 training Methods 0.000 claims abstract description 160
- 238000012545 processing Methods 0.000 claims abstract description 22
- 230000003993 interaction Effects 0.000 claims abstract description 17
- 238000005452 bending Methods 0.000 claims description 88
- 230000006870 function Effects 0.000 claims description 72
- 230000001133 acceleration Effects 0.000 claims description 36
- 238000004364 calculation method Methods 0.000 claims description 20
- 239000011159 matrix material Substances 0.000 claims description 20
- 238000002372 labelling Methods 0.000 claims description 13
- 238000006243 chemical reaction Methods 0.000 claims description 10
- 238000013508 migration Methods 0.000 claims description 4
- 230000005012 migration Effects 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 19
- 238000005457 optimization Methods 0.000 abstract description 19
- 230000008569 process Effects 0.000 description 9
- 238000013528 artificial neural network Methods 0.000 description 5
- 238000004590 computer program Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 5
- 230000007246 mechanism Effects 0.000 description 4
- 238000010606 normalization Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 210000002478 hand joint Anatomy 0.000 description 2
- 230000001629 suppression Effects 0.000 description 2
- 206010011878 Deafness Diseases 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 210000003414 extremity Anatomy 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 210000003857 wrist joint Anatomy 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V10/7753—Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/107—Static hand or arm
- G06V40/113—Recognition of static hand signs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/20—Movements or behaviour, e.g. gesture recognition
- G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/07—Target detection
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Artificial Intelligence (AREA)
- Human Computer Interaction (AREA)
- Psychiatry (AREA)
- Social Psychology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a gesture recognition model generation method and device, wherein the method comprises the following steps: acquiring a first training set and a second training set; the first training set comprises a plurality of first gesture image samples; processing the initial gesture recognition model based on the first training set, and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit; determining a gesture recognition model based on the second training set and the standby gesture recognition model for the initial gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system. Therefore, the method and the device are beneficial to training the model by utilizing the multi-mode gesture information, reduce the data requirement required by model training, improve the efficiency and the accuracy of model generation, and further improve the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for generating a gesture recognition model.
Background
In modern society, human-computer interaction has gradually become an integral part of our work and life. With the continuous development of artificial intelligence, especially deep learning, man-machine interaction becomes more intelligent and efficient. In recent years, with the cooperation and research among students in different fields, various man-machine interaction modes are developed to facilitate our lives and works. As an important limb language, gestures are an indispensable part of our life, for example, traffic police on the road command traffic with gestures, deaf-mutes achieve communication with others with sign language, and the like. Gesture interaction is also receiving more and more attention from academia and industry, but has become one of the important interaction modes in the field of man-machine interaction.
The existing gesture recognition model is usually constructed by adopting a neural network, and as the color image and the information represented by the I MU data are different, the characteristic of different mode data is not fully considered in a way of directly using the neural network to extract the characteristics for fusion so as to achieve a better effect. In addition, it should be noted that the current methods are developed based on deep neural networks, and the performance of the methods depends on the scale of the labeled data, and labeling of hand gesture data, especially acquiring three-dimensional gesture coordinates, often requires a great deal of time and effort. Therefore, the gesture recognition model generation method and device are provided to realize training of the model by utilizing multi-mode gesture information, reduce data requirements required by model training, improve model generation efficiency and accuracy, and further improve gesture recognition model optimization capacity and gesture recognition model adaptation capacity to different use situations.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a gesture recognition model generation method and device which are beneficial to training a model by utilizing multi-mode gesture information, reduce the data requirement required by model training, improve the efficiency and accuracy of model generation, and further improve the optimization capacity of the gesture recognition model and the capacity of the gesture recognition model to adapt to different use scenes.
In order to solve the above technical problems, a first aspect of an embodiment of the present invention discloses a gesture recognition model generating method, which includes:
acquiring a first training set and a second training set; the first training set comprises a plurality of first gesture image samples;
processing the initial gesture recognition model based on the first training set, and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit;
processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
In a first aspect of the embodiment of the present invention, the processing the initial gesture recognition model based on the first training set, to determine a standby gesture recognition model includes:
randomly determining a target first gesture image sample from all the first gesture image samples in the first training set;
randomly generating a mask matrix block; the pixel value size of the mask matrix block is 255;
Adding the mask matrix block to the target first gesture image sample to obtain a mask first gesture image sample; the mask matrix block covers at least 75% of the image area of the target first gesture image sample;
inputting the mask first gesture image sample into an initial gesture recognition model for training to obtain a first training gesture recognition model and output image result information;
calculating the output image result information and the target first gesture image sample by using a first loss function to obtain a first loss function value;
wherein the first loss function is
Wherein L is 1 -a value of the first loss function; y is 12 Image data information corresponding to the target first gesture image sample;image data information corresponding to the output image result information;
judging whether the first loss function value is smaller than or equal to a first loss threshold value or not, and obtaining a first loss judgment result;
when the first loss judgment result is negative, replacing and updating the initial gesture recognition model by using the first training gesture recognition model, and triggering and executing the random determination of a target first gesture image sample from all the first gesture image samples in the first training set;
And when the first loss judgment result is yes, determining the first training gesture recognition model as a standby gesture recognition model.
As an optional implementation manner, in the first aspect of the embodiment of the present invention, the determining, based on the second training set and the standby gesture recognition model, the gesture recognition model for the initial gesture recognition model includes:
performing parameter migration and updating on the coding unit of the initial gesture recognition model by using the coding unit in the standby gesture recognition model to obtain a reconstructed gesture recognition model;
and performing semi-supervised training on the reconstructed gesture recognition model by using the second training set to obtain a gesture recognition model.
In a first aspect of the embodiment of the present invention, training the reconstructed gesture recognition model by using the second training set to obtain a gesture recognition model includes:
randomly determining a target second gesture image sample from all second gesture image samples in the second training set;
inputting the target second gesture image sample into a reconstructed gesture recognition model for training to obtain a second training gesture recognition model and image joint point information;
Performing loss function calculation on the image node information to obtain a second loss function value;
judging whether the second loss function value is smaller than or equal to a second loss threshold value or not, and obtaining a second loss judgment result;
when the second loss judgment result is negative, replacing and updating the reconstructed gesture recognition model by using the second training gesture recognition model, and triggering and executing the random determination of a target second gesture image sample from all second gesture image samples in the second training set;
and when the second loss judgment result is yes, determining that the second training gesture recognition model is a gesture recognition model.
In an optional implementation manner, in a first aspect of the embodiment of the present invention, the calculating a loss function of the image node information to obtain a second loss function value includes:
calculating an included angle of the image articulation point information by using a bending angle calculation model to obtain joint bending angle information;
wherein the bending angle calculation model is as follows
Wherein θ is the joint bending angle in the joint bending angle information; l (L) 0→1 Is the vector from joint 0 to joint 1; l (L) 1→2 Is a vector from the articulation point 1 to an articulation point 2; (x) 1 ，y 1 ，z 1 ) For said l 0→1 Coordinates of (c); (x) 2 ，y 2 ，z 2 ) For said l 1→2 Coordinates of (c);
judging whether the target second gesture image sample contains marked node coordinate information or not to obtain marked information judgment results;
when the labeling information judging result is negative, calculating the joint bending angle information and the sample joint bending angle information in the target second gesture image sample by using a second loss function to obtain a second loss function value;
wherein the second loss function is
Wherein L is 20 -a value of the second loss function; x is x 21 Sample joint bending angle information in the target second gesture image sample;information about the angle of bending of the joint;
when the judgment result of the labeling information is yes, calculating the joint bending angle information, the sample joint bending angle information in the target second gesture image sample, the image articulation point information and the labeling articulation point coordinate information by using a third loss function to obtain a second loss function value;
wherein the third loss function is
Wherein L is 21 -a value of the second loss function; z 21 Marking node coordinate information for the label;and node information for the image.
As an optional implementation manner, in the first aspect of the embodiment of the present invention, the sample joint bending angle information is obtained based on the following steps:
collecting acceleration information and angular velocity information;
calculating the acceleration signal information and the angular velocity information to obtain acceleration attitude angle information and angular velocity attitude angle information;
fusing the acceleration attitude angle information and the angular velocity attitude angle information to obtain target attitude angle information;
and calculating the bending angle of the target attitude angle information to obtain the sample joint bending angle information.
In an optional implementation manner, in a first aspect of the embodiment of the present invention, the calculating the bending angle of the target attitude angle information to obtain the sample joint bending angle information includes:
converting the target attitude angle information by using an angle conversion model to obtain angle quaternary vector information;
wherein the angle conversion model is as follows
In the formula, isFor the angle in the angle quadrilaterals informationA degree quaternary vector; the r, p and yaw are rotation angles around an X axis, a Y axis and a Z axis in the target attitude angle information respectively;
Normalizing the angle quaternary vector information to obtain angle quaternary vector information;
calculating the four-component vector information of the return angle by using a bending angle model to obtain the bending angle information of the sample joint;
wherein the bending angle model is
θ 0_1_2 ＝180°+2arccos(Q 0 ·Q 1 ×180°/π)-2arccos(Q 1 ·Q 2 ×180°/π)；
In θ 0_1_2 The joint bending angle of the joint point 1 relative to the joint point 0 to the joint point 2 in the sample joint bending angle information; q (Q) 0 、Q 1 And Q 2 The four-component vector is respectively a return angle four-component vector corresponding to the joint point 0, a return angle four-component vector corresponding to the joint point 1 and a return angle four-component vector corresponding to the joint point 2.
The second aspect of the embodiment of the invention discloses a gesture recognition model generating device, which comprises:
the acquisition module is used for acquiring the first training set and the second training set; the first training set comprises a plurality of first gesture image samples;
the first determining module is used for processing the initial gesture recognition model based on the first training set and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit;
the second determining module is used for processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
In a third aspect, the present invention discloses another gesture recognition model generating device, which includes:
a memory storing executable program code;
a processor coupled to the memory;
the processor invokes the executable program code stored in the memory to perform some or all of the steps in the gesture recognition model generation method disclosed in the first aspect of the embodiment of the present invention.
A fourth aspect of the present invention discloses a computer readable storage medium storing computer instructions for executing some or all of the steps in the gesture recognition model generation method disclosed in the first aspect of the embodiments of the present invention when the computer instructions are called.
Compared with the prior art, the embodiment of the invention has the following beneficial effects:
in the embodiment of the invention, a first training set and a second training set are acquired; the first training set comprises a plurality of first gesture image samples; processing the initial gesture recognition model based on the first training set, and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit; processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system. Therefore, the method and the device are beneficial to training the model by utilizing the multi-mode gesture information, reduce the data requirement required by model training, improve the efficiency and the accuracy of model generation, and further improve the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a gesture recognition model generation method disclosed in an embodiment of the present invention;
FIG. 2 is a schematic diagram of a gesture recognition model generating device according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of another gesture recognition model generating apparatus according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a hand gesture key point according to an embodiment of the present invention.
Detailed Description
In order to make the present invention better understood by those skilled in the art, the following description will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present invention, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
The terms first, second and the like in the description and in the claims and in the above-described figures are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order. Furthermore, the terms "comprise" and "have," as well as any variations thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, article, or device that comprises a list of steps or elements is not limited to the list of steps or elements but may, in the alternative, include other steps or elements not expressly listed or inherent to such process, method, article, or device.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the invention. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The invention discloses a gesture recognition model generation method and device, which are beneficial to realizing training of a model by utilizing multi-mode gesture information, reduce data requirements required by model training, improve the efficiency and accuracy of model generation, and further improve the optimization capacity of a gesture recognition model and the capacity of the gesture recognition model to adapt to different use scenes. The following will describe in detail.
Example 1
Referring to fig. 1, fig. 1 is a flowchart of a gesture recognition model generating method according to an embodiment of the present invention. The gesture recognition model generation method described in fig. 1 is applied to a gesture interaction system, such as a local server or a cloud server for gesture recognition model generation management, and the embodiment of the invention is not limited. As shown in fig. 1, the gesture recognition model generation method may include the following operations:
101. a first training set and a second training set are obtained.
In the embodiment of the invention, the first training set comprises a plurality of first gesture image samples.
102. And processing the initial gesture recognition model based on the first training set to determine a standby gesture recognition model.
In an embodiment of the present invention, the initial gesture recognition model includes an encoding unit and a decoding unit.
103. And processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine the gesture recognition model.
In the embodiment of the invention, the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
It should be noted that, most of the current gesture recognition methods are constructed based on single-mode image data, and stability and robustness of the gesture recognition methods in an occlusion scene cannot be guaranteed; although the existing multi-mode data can alleviate the problem, the method has a single characteristic extraction mode for the data, only uses a neural network to extract and fuse the characteristics, and does not consider the characteristics of the data to design, so that the data utilization rate and the performance of the method are limited. In the application of the invention, the full supervision training is performed on a small amount of labeled data; in order to utilize a large amount of unlabeled data, the hand joint bending angle obtained by the acceleration and angular velocity signals acquired by the inertial sensor is used as weak supervision information for further training, so that better performance is realized.
In this alternative embodiment, as an alternative implementation manner, the first gesture image sample is obtained based on the following steps:
acquiring a hand image;
extracting features of the hand image by using a feature extractor to obtain feature information;
carrying out target detection of different sizes on the characteristic information by using a prediction frame generator to obtain hand prediction frame information;
performing overlapping degree calculation and sequencing treatment on the hand prediction frame information to obtain a target hand prediction frame;
and cutting the image corresponding to the target hand prediction frame from the hand image to obtain a first gesture image sample.
It should be noted that the feature extractor is constructed based on CBL components, namely, a convolution layer (Conv) +a batch normalization layer (BN) +a nonlinear active layer (leak ReLU). Furthermore, the feature extractor based on the CBL component combines the shallow features and the deep features, and then the accurate detection of targets with different sizes can be realized through the prediction frame generator of the multi-head mechanism.
It should be noted that the prediction frame generator is constructed based on a multi-head attention mechanism.
The above-mentioned overlapping degree calculation and sorting process are performed on the hand prediction frame information to obtain the target hand prediction frame, that is, non-maximum suppression (Non-Maximum Suppression, NMS) is used to evaluate the outputs of different prediction heads and select the final prediction result. The NMS mechanism is to sort all the boundary frames predicted by the network according to the score, then select the first boundary frame as the target frame, calculate the overlapping degree of the target frame and all the rest predicted frames, if the overlapping degree is greater than a preset threshold, then consider the predicted frame and the target frame as the same object, delete, otherwise keep. The above operation is then repeated in the remaining prediction frames until all prediction frames have been screened.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In an alternative embodiment, the processing the initial gesture recognition model based on the first training set to determine the standby gesture recognition model includes:
randomly determining a target first gesture image sample from all first gesture image samples in the first training set;
randomly generating a mask matrix block; the pixel value size of the mask matrix block is 255;
adding the mask matrix block to the target first gesture image sample to obtain a mask first gesture image sample; the mask matrix block covers at least 75% of the image area of the target first gesture image sample;
inputting the mask first gesture image sample into an initial gesture recognition model for training to obtain a first training gesture recognition model and output image result information;
calculating the output image result information and the target first gesture image sample by using a first loss function to obtain a first loss function value;
Wherein the first loss function is
Wherein L is 1 Is a first loss function value; y is 12 Image data information corresponding to a first gesture image sample of the target;for outputting the corresponding graph of the image result informationImage data information;
judging whether the first loss function value is smaller than or equal to a first loss threshold value or not to obtain a first loss judgment result;
when the first loss judgment result is negative, replacing and updating the initial gesture recognition model by using the first training gesture recognition model, and triggering and executing to randomly determine a target first gesture image sample from all first gesture image samples in the first training set;
and when the first loss judgment result is yes, determining the first training gesture recognition model as a standby gesture recognition model.
It should be noted that, the first loss threshold may be set fixedly, or may be set according to actual needs, which is not limited by the embodiment of the present invention.
It should be noted that, when the first training set is used to train the initial gesture recognition model, a random mask is added to the target first gesture image sample, and then the target first gesture image sample is sent to the self-encoder for reconstruction, that is, the encoder is used to extract the features first, and then the decoder is used to reconstruct the masked part, so as to help to improve the robustness of the model in the occlusion scene.
Further, the self-encoder may be a variable self-encoder VAE, or may be a U-Net network model, which is not limited by the embodiment of the present invention. The decoder may be a model based on a deep neural network, or may be a model based on a self-attention mechanism, which is not limited by the embodiment of the present invention.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In another alternative embodiment, processing the initial gesture recognition model based on the second training set and the standby gesture recognition model, determining the gesture recognition model includes:
performing parameter migration and updating on the coding unit of the initial gesture recognition model by using the coding unit in the standby gesture recognition model to obtain a reconstructed gesture recognition model;
and performing semi-supervised training on the reconstructed gesture recognition model by using the second training set to obtain the gesture recognition model.
It should be noted that, the above method for transferring the model parameters of the coding unit in the standby gesture recognition model to the coding unit of the initial gesture recognition model can transfer the prior knowledge of the hand structure learned in the pre-training stage to the semi-supervised learning model in a parameter loading manner, so as to accelerate the model training process and improve the performance of the model.
It should be noted that, the second training set for performing semi-supervised training on the reconstructed gesture recognition model by using the second training set includes two kinds of supervision information: for a small amount of labeled data, the supervision information is respectively the hand joint point coordinates obtained from the image and the joint bending angle related information obtained from the inertial sensor data; whereas for large amounts of unlabeled data, the supervisory information is only information about the angle of joint bending from inertial sensor data. Based on the semi-supervised learning mode, a good training effect can be achieved by using a small amount of marked data.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In yet another alternative embodiment, training the reconstructed gesture recognition model using the second training set to obtain the gesture recognition model includes:
randomly determining a target second gesture image sample from all second gesture image samples in the second training set;
Inputting a target second gesture image sample into the reconstructed gesture recognition model for training to obtain a second training gesture recognition model and image joint point information;
performing loss function calculation on the image node information to obtain a second loss function value;
judging whether the second loss function value is smaller than or equal to a second loss threshold value or not, and obtaining a second loss judgment result;
when the second loss judgment result is negative, replacing and updating the reconstructed gesture recognition model by using the second training gesture recognition model, and triggering and executing to randomly determine a target second gesture image sample from all second gesture image samples in the second training set;
and when the second loss judgment result is yes, determining that the second training gesture recognition model is a gesture recognition model.
The image node information is a two-dimensional vector having a size (21, 3). 21 is the number of key points, namely one key point at the wrist joint, and four key points are arranged on each finger; 3 represents the three-dimensional coordinates of each key point.
Training the reconstructed gesture recognition model by using the second training set, and performing semi-supervised learning by using key point coordinates obtained by the images and joint bending angles obtained by the inertial sensor data.
It should be noted that, the second loss threshold may be fixed, or may be set according to actual needs, which is not limited in the embodiment of the present invention.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 4, performing a loss function calculation on the image node information to obtain a second loss function value, including:
calculating an included angle of the image articulation point information by using a bending angle calculation model to obtain joint bending angle information;
wherein the bending angle calculation model is as follows
Wherein θ is the joint bending angle in the joint bending angle information; l (L) 0→1 Is the vector from joint 0 to joint 1; l (L) 1→2 Is the vector from node 1 to node 2; (x) 1 ，y 1 ，z 1 ) Is l 0→1 Coordinates of (c); (x) 2 ，y 2 ，z 2 ) Is l 1→2 Coordinates of (c);
judging whether the target second gesture image sample contains marked node coordinate information or not to obtain marked information judgment results;
When the judgment result of the labeling information is negative, calculating the joint bending angle information and the sample joint bending angle information in the target second gesture image sample by using a second loss function to obtain a second loss function value;
wherein the second loss function is
Wherein L is 20 Is a second loss function value; x is x 21 Sample joint bending angle information in a target second gesture image sample;is joint bending angle information;
when the judgment result of the labeling information is yes, calculating joint bending angle information, sample joint bending angle information in a target second gesture image sample, image joint point information and labeling joint point coordinate information by using a third loss function to obtain a second loss function value;
wherein the third loss function is
Wherein L is 21 Is a second loss function value; z 21 Marking the coordinate information of the joint point;is image joint point information.
It should be noted that, the above-mentioned joint point 0, the above-mentioned joint point 1 and the above-mentioned joint point 2 may be key points, such as the key point 10, the key point 11 and the key point 12, in the order from bottom to top in fig. 4, which is not limited in the embodiment of the present invention.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In an alternative embodiment, the sample joint bending angle information is obtained based on the following steps:
collecting acceleration information and angular velocity information;
calculating acceleration signal information and angular velocity information to obtain acceleration attitude angle information and angular velocity attitude angle information;
fusing the acceleration attitude angle information and the angular velocity attitude angle information to obtain target attitude angle information;
and calculating the bending angle of the target attitude angle information to obtain sample joint bending angle information.
The acceleration information and the angular velocity information are acquired by an inertial sensor and a gyroscope.
In this optional embodiment, as an optional implementation manner, the calculating processing of the acceleration signal information and the angular velocity information to obtain acceleration attitude angle information and angular velocity attitude angle information includes:
calculating the acceleration signal information by using an angle conversion model to obtain acceleration attitude angle information;
wherein the angle conversion model is as follows
Wherein r is acc And p acc Acceleration attitude angles around an X axis and a Y axis are respectively; a, a x 、a y And a z Acceleration about the X, Y and Z axes, respectively;
and integrating the angular velocity information to obtain angular velocity attitude angle information.
The angular velocity attitude angle around the Z axis is obtained by directly measuring the angular velocity attitude angle with an inertial sensor.
In this optional embodiment, as an optional implementation manner, the fusing acceleration attitude angle information and angular velocity attitude angle information to obtain target attitude angle information includes:
constructing state quantity information and angular velocity prediction angle information based on the angular velocity attitude angle information;
wherein the State quantity information State is
Angular velocity prediction angle informationIs that
Wherein Q (k) is an input noise matrix at time k; angle (k) and Angle (k+1) are angular velocity attitude angles at times k and k+1Angular velocity attitude angle in the information; gyro bias (k) And Gyro bias (k+1) is the offset at time k and time k+1, respectively; gyro (k) is the control input at time k;
constructing acceleration prediction angle information based on the acceleration attitude angle information;
wherein the acceleration predicting Angle information Angle acc (k) Is that
In the formula, angle acc (k) The acceleration attitude angle in the acceleration attitude angle information at the time k; r (k) is an observation noise matrix at k moment;
and carrying out parameter adjustment and state prediction updating iteration on the state quantity information, the angular velocity prediction angle information and the acceleration prediction angle information to obtain target attitude angle information.
It should be noted that, the above-mentioned parameter adjustment and state prediction update iteration for the state quantity information, the angular velocity prediction angle information and the acceleration prediction angle information are implemented based on a kalman filter algorithm.
It should be noted that, the input noise matrix, the observation noise matrix and the bias may be set according to actual needs, or may be preset default values, which is not limited by the embodiment of the present invention. Further, the control inputs are directly obtained from the actual model, which is known in the art.
It should be noted that, the method of the attitude angle of the acceleration calculation is only suitable for the static process, and the accelerometer is easily affected by vibration, so that the acceleration data has high-frequency noise, and the attitude angle of the three-axis acceleration calculation is also quite noisy; the angular velocity data measured by the gyroscope has small noise, but the gyroscope has the zero drift problem, and errors are accumulated after the attitude angle is obtained through integration. The above-mentioned attitude angle information of acceleration and attitude angle information of angular velocity are fused together the attitude angle that is calculated based on above-mentioned two methods, can get an angle data that has not accumulated error, noise and is small.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
In another optional embodiment, the calculating the bending angle of the target attitude angle information to obtain sample joint bending angle information includes:
converting the target attitude angle information by using an angle conversion model to obtain angle quaternary vector information;
wherein the angle conversion model is as follows
/>
In the formula, isThe angle quaternary vector is an angle quaternary vector in the angle quaternary vector information; r, p, and yaw are rotation angles around the X axis, the Y axis, and the Z axis in the target attitude angle information respectively;
carrying out normalization processing on the angle quaternary vector information to obtain angle quaternary vector information;
calculating the four-component vector information of the return angle by using a bending angle model to obtain sample joint bending angle information;
wherein the bending angle model is
θ 0_1_2 ＝180°+2arccos(Q 0 ·Q 1 ×180°/π)-2arccos(Q 1 ·Q 2 ×180°/π)；
In θ 0_1_2 The joint bending angle of the joint point 1 relative to the joint point 0 to the joint point 2 in the sample joint bending angle information; q (Q) 0 、Q 1 And Q 2 Respectively a return angle quadripod corresponding to the joint point 0 and a return angle quadripod corresponding to the joint point 1The quantity corresponds to the return angle quaternary vector of the joint point 2.
It should be noted that, the above-mentioned angle quadripole may also be obtained by calculating angles of cosine and sine around the X-axis and around the Z-axis, which is not limited in the embodiment of the present invention.
It should be noted that the number of the substrates,is an element of an angular quaternion vector, which is a real number.
Therefore, the method for generating the gesture recognition model described by the embodiment of the invention is beneficial to realizing training of the model by utilizing multi-mode gesture information, reduces the data requirement for model training, improves the efficiency and accuracy of model generation, and further improves the optimization capacity of the gesture recognition model and the capacity of adapting to different use scenes of the gesture recognition model.
Example two
Referring to fig. 2, fig. 2 is a schematic structural diagram of a gesture recognition model generating device according to an embodiment of the present invention. The device described in fig. 2 can be applied to a gesture interaction system, such as a local server or a cloud server for gesture recognition model generation management, and the embodiment of the invention is not limited. As shown in fig. 2, the apparatus may include:
An acquisition module 201, configured to acquire a first training set and a second training set; the first training set comprises a plurality of first gesture image samples;
a first determining module 202, configured to process the initial gesture recognition model based on the first training set, and determine a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit;
a second determining module 203, configured to process the initial gesture recognition model based on the second training set and the standby gesture recognition model, and determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In another alternative embodiment, as shown in fig. 2, the first determining module 202 processes the initial gesture recognition model based on the first training set, and determines the standby gesture recognition model, including:
Randomly determining a target first gesture image sample from all first gesture image samples in the first training set;
randomly generating a mask matrix block; the pixel value size of the mask matrix block is 255;
adding the mask matrix block to the target first gesture image sample to obtain a mask first gesture image sample; the mask matrix block covers at least 75% of the image area of the target first gesture image sample;
inputting the mask first gesture image sample into an initial gesture recognition model for training to obtain a first training gesture recognition model and output image result information;
calculating the output image result information and the target first gesture image sample by using a first loss function to obtain a first loss function value;
wherein the first loss function is
Wherein L is 1 Is a first loss function value; y is 12 Image data information corresponding to a first gesture image sample of the target;image data information corresponding to the output image result information;
judging whether the first loss function value is smaller than or equal to a first loss threshold value or not to obtain a first loss judgment result;
when the first loss judgment result is negative, replacing and updating the initial gesture recognition model by using the first training gesture recognition model, and triggering and executing to randomly determine a target first gesture image sample from all first gesture image samples in the first training set;
And when the first loss judgment result is yes, determining the first training gesture recognition model as a standby gesture recognition model.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 2, the second determining module 203 processes the initial gesture recognition model based on the second training set and the standby gesture recognition model, and determines the gesture recognition model, including:
performing parameter migration and updating on the coding unit of the initial gesture recognition model by using the coding unit in the standby gesture recognition model to obtain a reconstructed gesture recognition model;
and performing semi-supervised training on the reconstructed gesture recognition model by using the second training set to obtain the gesture recognition model.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 2, the second determining module 203 trains the reconstructed gesture recognition model by using the second training set to obtain the gesture recognition model, including:
randomly determining a target second gesture image sample from all second gesture image samples in the second training set;
inputting a target second gesture image sample into the reconstructed gesture recognition model for training to obtain a second training gesture recognition model and image joint point information;
performing loss function calculation on the image node information to obtain a second loss function value;
judging whether the second loss function value is smaller than or equal to a second loss threshold value or not, and obtaining a second loss judgment result;
when the second loss judgment result is negative, replacing and updating the reconstructed gesture recognition model by using the second training gesture recognition model, and triggering and executing to randomly determine a target second gesture image sample from all second gesture image samples in the second training set;
and when the second loss judgment result is yes, determining that the second training gesture recognition model is a gesture recognition model.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 2, the second determining module 203 performs a loss function calculation on the image node information to obtain a second loss function value, including:
calculating an included angle of the image articulation point information by using a bending angle calculation model to obtain joint bending angle information;
wherein the bending angle calculation model is as follows
Wherein θ is the joint bending angle in the joint bending angle information; l (L) 0→1 Is the vector from joint 0 to joint 1; l (L) 1→2 Is the vector from node 1 to node 2; (x) 1 ，y 1 ，z 1 ) Is l 0→1 Coordinates of (c); (x) 2 ，y 2 ，z 2 ) Is l 1→2 Coordinates of (c);
judging whether the target second gesture image sample contains marked node coordinate information or not to obtain marked information judgment results;
when the judgment result of the labeling information is negative, calculating the joint bending angle information and the sample joint bending angle information in the target second gesture image sample by using a second loss function to obtain a second loss function value;
wherein the second loss function is
Wherein L is 20 Is a second loss function value; x is x 21 Sample joint bending angle information in a target second gesture image sample;is joint bending angle information;
when the judgment result of the labeling information is yes, calculating joint bending angle information, sample joint bending angle information in a target second gesture image sample, image joint point information and labeling joint point coordinate information by using a third loss function to obtain a second loss function value;
Wherein the third loss function is
Wherein L is 21 Is a second loss function value; z 21 Marking the coordinate information of the joint point;is image joint point information.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 2, the sample joint bending angle information is obtained based on the acquisition module 201 performing the following steps:
collecting acceleration information and angular velocity information;
calculating acceleration signal information and angular velocity information to obtain acceleration attitude angle information and angular velocity attitude angle information;
fusing the acceleration attitude angle information and the angular velocity attitude angle information to obtain target attitude angle information;
and calculating the bending angle of the target attitude angle information to obtain sample joint bending angle information.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
In yet another alternative embodiment, as shown in fig. 2, the obtaining module 201 performs bending angle calculation on the target attitude angle information to obtain sample joint bending angle information, including:
converting the target attitude angle information by using an angle conversion model to obtain angle quaternary vector information;
wherein the angle conversion model is as follows
In the formula, isThe angle quaternary vector is an angle quaternary vector in the angle quaternary vector information; r, p, and yaw are rotations about the X-axis, Y-axis, and Z-axis, respectively, in the target attitude angle informationAn angle;
carrying out normalization processing on the angle quaternary vector information to obtain angle quaternary vector information;
calculating the four-component vector information of the return angle by using a bending angle model to obtain sample joint bending angle information;
wherein the bending angle model is
θ 0_1_2 ＝180°+2arccos(Q 0 ·Q 1 ×180°/π)-2arccos(Q 1 ·Q 2 ×180°/π)；
In θ 0_1_2 The joint bending angle of the joint point 1 relative to the joint point 0 to the joint point 2 in the sample joint bending angle information; q (Q) 0 、Q 1 And Q 2 The four-component vector is respectively a return angle four-component vector corresponding to the joint point 0, a return angle four-component vector corresponding to the joint point 1 and a return angle four-component vector corresponding to the joint point 2.
Therefore, implementing the gesture recognition model generating device described in fig. 2 is beneficial to realizing training of the model by using multi-mode gesture information, reducing data requirements required by model training, improving efficiency and accuracy of model generation, and further improving optimization capacity of the gesture recognition model and capacity of adapting to different use situations of the gesture recognition model.
Example III
Referring to fig. 3, fig. 3 is a schematic structural diagram of another gesture recognition model generating apparatus according to an embodiment of the present invention. The device described in fig. 3 can be applied to a gesture interaction system, such as a local server or a cloud server for gesture recognition model generation management, and the embodiment of the invention is not limited. As shown in fig. 3, the apparatus may include:
a memory 301 storing executable program code;
a processor 302 coupled with the memory 301;
the processor 302 invokes executable program code stored in the memory 301 for performing the steps in the gesture recognition model generation method described in embodiment one.
Example IV
The embodiment of the invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program causes a computer to execute the steps in the gesture recognition model generation method described in the embodiment one.
Example five
The embodiment of the invention discloses a computer program product, which comprises a non-transitory computer readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the steps in the gesture recognition model generating method described in the embodiment.
The apparatus embodiments described above are merely illustrative, in which the modules illustrated as separate components may or may not be physically separate, and the components shown as modules may or may not be physical, i.e., may be located in one place, or may be distributed over multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
From the above detailed description of the embodiments, it will be apparent to those skilled in the art that the embodiments may be implemented by means of software plus necessary general hardware platforms, or of course by means of hardware. Based on such understanding, the foregoing technical solutions may be embodied essentially or in part in the form of a software product that may be stored in a computer-readable storage medium including Read-Only Memory (ROM), random-access Memory (Random Access Memory, RAM), programmable Read-Only Memory (Programmable Read-Only Memory, PROM), erasable programmable Read-Only Memory (Erasable Programmable Read Only Memory, EPROM), one-time programmable Read-Only Memory (OTPROM), electrically erasable programmable Read-Only Memory (EEPROM), compact disc Read-Only Memory (Compact Disc Read-Only Memory, CD-ROM) or other optical disc Memory, magnetic disc Memory, tape Memory, or any other medium that can be used for computer-readable carrying or storing data.
Finally, it should be noted that: the embodiment of the invention discloses a gesture recognition model generation method and device, which are disclosed in the preferred embodiment of the invention, and are only used for illustrating the technical scheme of the invention, but not limiting the technical scheme; although the invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art will understand that; the technical scheme recorded in the various embodiments can be modified or part of technical features in the technical scheme can be replaced equivalently; such modifications and substitutions do not depart from the spirit and scope of the corresponding technical solutions.
Claims (10)
1. A method for generating a gesture recognition model, the method comprising:
acquiring a first training set and a second training set; the first training set comprises a plurality of first gesture image samples;
processing the initial gesture recognition model based on the first training set, and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit;
processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
2. The method of claim 1, wherein the processing the initial gesture recognition model based on the first training set to determine the standby gesture recognition model comprises:
randomly determining a target first gesture image sample from all the first gesture image samples in the first training set;
randomly generating a mask matrix block; the pixel value size of the mask matrix block is 255;
adding the mask matrix block to the target first gesture image sample to obtain a mask first gesture image sample; the mask matrix block covers at least 75% of the image area of the target first gesture image sample;
inputting the mask first gesture image sample into an initial gesture recognition model for training to obtain a first training gesture recognition model and output image result information;
calculating the output image result information and the target first gesture image sample by using a first loss function to obtain a first loss function value;
wherein the first loss function is
Wherein L is 1 -a value of the first loss function; y is 12 Image data information corresponding to the target first gesture image sample; Image data information corresponding to the output image result information;
judging whether the first loss function value is smaller than or equal to a first loss threshold value or not, and obtaining a first loss judgment result;
when the first loss judgment result is negative, replacing and updating the initial gesture recognition model by using the first training gesture recognition model, and triggering and executing the random determination of a target first gesture image sample from all the first gesture image samples in the first training set;
and when the first loss judgment result is yes, determining the first training gesture recognition model as a standby gesture recognition model.
3. The method of generating a gesture recognition model according to claim 1, wherein the processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model comprises:
performing parameter migration and updating on the coding unit of the initial gesture recognition model by using the coding unit in the standby gesture recognition model to obtain a reconstructed gesture recognition model;
and performing semi-supervised training on the reconstructed gesture recognition model by using the second training set to obtain a gesture recognition model.
4. The method according to claim 2, wherein training the reconstructed gesture recognition model with the second training set to obtain a gesture recognition model comprises:
randomly determining a target second gesture image sample from all second gesture image samples in the second training set;
inputting the target second gesture image sample into a reconstructed gesture recognition model for training to obtain a second training gesture recognition model and image joint point information;
performing loss function calculation on the image node information to obtain a second loss function value;
judging whether the second loss function value is smaller than or equal to a second loss threshold value or not, and obtaining a second loss judgment result;
when the second loss judgment result is negative, replacing and updating the reconstructed gesture recognition model by using the second training gesture recognition model, and triggering and executing the random determination of a target second gesture image sample from all second gesture image samples in the second training set;
and when the second loss judgment result is yes, determining that the second training gesture recognition model is a gesture recognition model.
5. The method of claim 4, wherein the performing a loss function calculation on the image node information to obtain a second loss function value includes:
calculating an included angle of the image articulation point information by using a bending angle calculation model to obtain joint bending angle information;
wherein the bending angle calculation model is as follows
Wherein θ is the joint bending angle in the joint bending angle information; l (L) 0→1 Is the vector from joint 0 to joint 1; l (L) 1→2 Is a vector from the articulation point 1 to an articulation point 2; (x) 1 ，y 1 ，z 1 ) For said l 0→1 Coordinates of (c); (x) 2 ，y 2 ，z 2 ) For said l 1→2 Coordinates of (c);
judging whether the target second gesture image sample contains marked node coordinate information or not to obtain marked information judgment results;
when the labeling information judging result is negative, calculating the joint bending angle information and the sample joint bending angle information in the target second gesture image sample by using a second loss function to obtain a second loss function value;
wherein the second loss function is
Wherein L is 20 -a value of the second loss function; x is x 21 Sample joint bending angle information in the target second gesture image sample; Information about the angle of bending of the joint;
when the judgment result of the labeling information is yes, calculating the joint bending angle information, the sample joint bending angle information in the target second gesture image sample, the image articulation point information and the labeling articulation point coordinate information by using a third loss function to obtain a second loss function value;
wherein the third loss function is
Wherein L is 21 -a value of the second loss function; z 21 Marking node coordinate information for the label;and node information for the image.
6. The method of claim 5, wherein the sample joint bending angle information is obtained based on:
collecting acceleration information and angular velocity information;
calculating the acceleration signal information and the angular velocity information to obtain acceleration attitude angle information and angular velocity attitude angle information;
fusing the acceleration attitude angle information and the angular velocity attitude angle information to obtain target attitude angle information;
and calculating the bending angle of the target attitude angle information to obtain the sample joint bending angle information.
7. The method of generating a gesture recognition model according to claim 6, wherein the calculating the bending angle of the target attitude angle information to obtain the sample joint bending angle information includes:
converting the target attitude angle information by using an angle conversion model to obtain angle quaternary vector information;
wherein the angle conversion model is as follows
In the formula, isThe angle quaternary vector in the angle quaternary vector information is the angle quaternary vector; the r, p and yaw are rotation angles around an X axis, a Y axis and a Z axis in the target attitude angle information respectively;
normalizing the angle quaternary vector information to obtain angle quaternary vector information;
calculating the four-component vector information of the return angle by using a bending angle model to obtain the bending angle information of the sample joint;
wherein the bending angle model is theta 0_1_2 ＝180°+2arccos(Q 0 ·Q 1 ×180°/π)-2arccos(Q 1 ·Q 2 ×180°/π)；
In θ 0_1_2 A sample joint bending angle from the joint point 1 to the joint point 2 relative to the joint point 0 in the sample joint bending angle information; q (Q) 0 、Q 1 And Q 2 The four-component vector is respectively a return angle quaternary vector corresponding to the joint point 0, a return angle quaternary vector corresponding to the joint point 1 and a return angle quaternary vector corresponding to the joint point 2.
8. A gesture recognition model generation apparatus, the apparatus comprising:
the acquisition module is used for acquiring the first training set and the second training set; the first training set comprises a plurality of first gesture image samples;
the first determining module is used for processing the initial gesture recognition model based on the first training set and determining a standby gesture recognition model; the initial gesture recognition model comprises an encoding unit and a decoding unit;
the second determining module is used for processing the initial gesture recognition model based on the second training set and the standby gesture recognition model to determine a gesture recognition model; the gesture recognition model is used for detecting and recognizing gestures in the gesture interaction system.
9. A gesture recognition model generation apparatus, the apparatus comprising:
a memory storing executable program code;
a processor coupled to the memory;
the processor invokes the executable program code stored in the memory to perform the gesture recognition model generation method of any one of claims 1-7.
10. A computer readable storage medium storing computer instructions which, when invoked, are adapted to perform the gesture recognition model generation method of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311579134.9A CN117496303A (en) | 2023-11-23 | 2023-11-23 | Gesture recognition model generation method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311579134.9A CN117496303A (en) | 2023-11-23 | 2023-11-23 | Gesture recognition model generation method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117496303A true CN117496303A (en) | 2024-02-02 |
Family
ID=89670755
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311579134.9A Pending CN117496303A (en) | 2023-11-23 | 2023-11-23 | Gesture recognition model generation method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117496303A (en) |
-
2023
- 2023-11-23 CN CN202311579134.9A patent/CN117496303A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112734852A (en) | Robot mapping method and device and computing equipment | |
| CN109242003B (en) | Vehicle-mounted vision system self-motion determination method based on deep convolutional neural network | |
| CN111680550B (en) | Emotion information identification method and device, storage medium and computer equipment | |
| CN112861808B (en) | Dynamic gesture recognition method, device, computer equipment and readable storage medium | |
| CN116263622A (en) | Gesture recognition method, gesture recognition device, electronic equipment, gesture recognition medium and gesture recognition program product | |
| CN114241597A (en) | Posture recognition method and related equipment thereof | |
| CN114593735B (en) | Pose prediction method and device | |
| CN113887501A (en) | Behavior recognition method and device, storage medium and electronic equipment | |
| CN112363617A (en) | Method and device for acquiring human body action data | |
| CN116654022A (en) | Pedestrian track prediction method, system, equipment and medium based on multiple interactions | |
| US20210225008A1 (en) | Moving state analysis device, moving state analysis method, and program | |
| CN117496303A (en) | Gesture recognition model generation method and device | |
| CN116561649A (en) | Diver motion state identification method and system based on multi-source sensor data | |
| CN116734834A (en) | Positioning and mapping method and device applied to dynamic scene and intelligent equipment | |
| CN111797986A (en) | Data processing method, data processing device, storage medium and electronic equipment | |
| CN114937153A (en) | Neural network-based visual feature processing system and method under weak texture environment | |
| CN113706606A (en) | Method and device for determining position coordinates of spaced gestures | |
| KR102678174B1 (en) | Method of human activity recognition and classification using convolutional LSTM, apparatus and computer program for performing the method | |
| CN117115238B (en) | Pose determining method, electronic equipment and storage medium | |
| CN114419451B (en) | Method and device for identifying inside and outside of elevator, electronic equipment and storage medium | |
| CN117975201B (en) | Training data generation method, device, computer equipment and storage medium | |
| CN117173731B (en) | Model training method, image processing method and related device | |
| CN118015709B (en) | Action recognition method and device based on multi-modal graph neural network | |
| Zhang et al. | An efficient loop closure detection method based on spatially constrained feature matching | |
| CN117292435A (en) | Action recognition method and device and computer equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |