CN111984119A - Gesture recognition model establishing method, gesture recognition method and device and data glove - Google Patents
Gesture recognition model establishing method, gesture recognition method and device and data glove Download PDFInfo
- Publication number
- CN111984119A CN111984119A CN202010832352.9A CN202010832352A CN111984119A CN 111984119 A CN111984119 A CN 111984119A CN 202010832352 A CN202010832352 A CN 202010832352A CN 111984119 A CN111984119 A CN 111984119A
- Authority
- CN
- China
- Prior art keywords
- gesture recognition
- sensors
- recognition model
- node
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
Abstract
The invention provides a gesture recognition model establishing method, a gesture recognition device and a data glove, wherein the gesture recognition model establishing method comprises the following steps: acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the calibration action; determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule; and establishing a gesture recognition model based on a graph attention network according to the semantic relation among the sensors. According to the technical scheme, the gesture recognition model for gesture recognition is established according to the semantic relation among the sensors, so that the gesture recognition precision can be improved, and the recognition speed is high.
Description
Technical Field
The invention relates to the technical field of gesture recognition, in particular to a gesture recognition model establishing method, a gesture recognition method and device and a data glove.
Background
The data glove is a multi-mode interactive device, can be used for completing actions such as grabbing, moving and rotating in a virtual scene, and can also be used for controlling a robot. By recognizing the gestures of the data gloves, various human-computer interaction tasks can be completed.
Currently, a commonly used gesture recognition method based on data gloves is to acquire measurement data acquired by each sensor on the data gloves, compare the measurement data with template data of each gesture template established in advance, and determine a gesture corresponding to an action of the data gloves according to a comparison result. However, this method is low in gesture recognition accuracy and low in efficiency.
Disclosure of Invention
The invention solves the problem of how to improve the gesture recognition precision and recognition speed of the data-based gloves.
In order to solve the above problems, the present invention provides a gesture recognition method, device, storage medium and data glove.
In a first aspect, the present invention provides a method for establishing a gesture recognition model, including:
acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the action;
determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and establishing a gesture recognition model based on a graph attention network according to the semantic relation among the sensors.
Optionally, the semantic relationship includes a linkage relationship and a concurrency relationship, and the determining, based on a preset rule, the semantic relationship between the sensors according to the acquisition time of each sensor includes:
for any two sensors, if the two sensors acquire data successively, a linkage process occurs between the two sensors;
if the two sensors simultaneously acquire data, a concurrence process occurs between the two sensors;
when the data glove executes the action calibration times, determining a first probability of a linkage process between the two sensors and a second probability of a concurrent process between the two sensors;
comparing the first probability with a first preset threshold value, and comparing the second probability with a second preset threshold value;
when the first probability is larger than the first preset threshold, a linkage relation exists between the two sensors;
and when the second probability is greater than the second preset threshold value, a concurrency relation exists between the two sensors.
Optionally, the establishing a gesture recognition model based on a graph attention network according to the semantic relationship between the sensors includes:
establishing a sensor network by taking the sensors as nodes and the semantic relation among the sensors as the connection relation among the nodes;
and constructing the gesture recognition model based on the graph attention network according to the sensor network.
Optionally, the method further comprises the step of acquiring data acquired by each sensor when the data glove completes calibration action;
after the constructing the gesture recognition model based on the graph attention network according to the sensor network, further comprising: and training the gesture recognition model by adopting the data collected by each sensor.
Optionally, the training of the gesture recognition model using the data collected by each of the sensors includes a forward propagation step, where the forward propagation step includes:
taking the data collected by each sensor as the initial characteristic representation of the corresponding node;
for any one node in the sensor network, determining attention coefficients between the node and each first-order neighbor node of the node respectively;
normalizing all the attention coefficients corresponding to the nodes to obtain normalized attention coefficients;
updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node to obtain the updated feature representation of the node;
determining the updated feature representations of all the nodes, and constructing an updated feature matrix according to all the updated feature representations;
inputting the updated feature matrix into a full-connection network of the gesture recognition model, determining the probability that the motion of the data glove is respectively the gesture, and determining the gesture with the maximum probability as a predicted gesture.
Optionally, the determining attention coefficients between the node and respective first-order neighbor nodes of the node comprises:
setting the node as a node i, setting any one of the first-order neighbor nodes of the node i as a node j, and determining an attention coefficient between the node i and the node j by adopting a first formula according to the initial feature representation of the node i and the initial feature representation of the node j, wherein the first formula comprises:
wherein e isijRepresenting the attention coefficient, a representing a corresponding function from the attention neural network layer,
for the initial characteristic representation of the node i,
w is a weight matrix for the initial feature representation of the node j.
Optionally, the normalizing all the attention coefficients corresponding to the node, and obtaining the normalized attention coefficients includes:
normalizing all the attention coefficients corresponding to the node i by adopting a second formula, wherein the second formula comprises:
wherein alpha isijRepresenting the normalized attention coefficient,
a set of all the first-order neighbor nodes representing the node i.
Optionally, the updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each of the first-order neighbor nodes comprises:
updating the feature representation of the node i by adopting a third formula according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node, wherein the third formula comprises:
wherein the content of the first and second substances,
representing the updated feature representation, σ representing an activation function of the gesture recognition model, αijIs the normalized attention coefficient, W is a weight matrix,
is the initial feature representation of the node j.
Optionally, the training the gesture recognition model using the data collected by each of the sensors further includes:
a back propagation step, which comprises making cross entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss;
and circularly repeating the forward propagation step and the backward propagation step until the loss does not decrease any more, and obtaining a stable gesture recognition model.
In a second aspect, the present invention provides a gesture recognition method, including:
acquiring data collected by each sensor when the data glove finishes the current action;
inputting all data collected by each sensor into a gesture recognition model, and determining a gesture corresponding to the current action of the data glove;
wherein the gesture recognition model is established according to the gesture recognition model establishing method.
In a third aspect, the present invention provides a gesture recognition model building apparatus, including:
the acquisition module is used for acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the action;
the processing module is used for determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and the establishing module is used for establishing a gesture recognition model based on the graph attention network according to the semantic relation among the sensors.
In a fourth aspect, the present invention provides a gesture recognition model building apparatus, including a memory and a processor;
the memory for storing a computer program;
the processor is configured to implement the gesture recognition model building method as described above when executing the computer program.
In a fifth aspect, the present invention provides a gesture recognition apparatus, including:
the acquisition module is used for acquiring data acquired by each sensor when the data glove finishes the current action;
the recognition module is used for inputting all data collected by the sensors into a gesture recognition model and determining a gesture corresponding to the current action of the data glove;
wherein the gesture recognition model is established according to the gesture recognition model establishing method.
In a sixth aspect, the present invention provides a gesture recognition apparatus, comprising a memory and a processor;
the memory for storing a computer program;
the processor is configured to implement the gesture recognition method as described above when executing the computer program.
In a seventh aspect, the present invention provides a computer-readable storage medium, on which a computer program is stored, which, when executed by a processor, implements the gesture recognition model building method as described above.
In an eighth aspect, the present invention provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements a gesture recognition method as described above.
In a ninth aspect, the invention provides a data glove, which comprises a glove, a plurality of sensors and the gesture recognition device according to the sixth aspect, wherein the plurality of sensors are respectively electrically connected with the gesture recognition device, and are respectively arranged on the glove for detecting the motion data of each finger.
Optionally, the sensor comprises a plurality of fully flexible capacitive sensors and a plurality of piezoelectric thin film sensors;
the fully-flexible capacitive sensors are respectively arranged at the back of each finger and the back of the thumb of the glove and are respectively used for measuring motion data of the fingers in the flexing or extending process and motion data of the thumb in the transverse motion process;
the piezoelectric film sensors are respectively arranged at joints on the inner sides of the fingers of the glove and between two adjacent fingers and are respectively used for measuring motion data of joints of the fingers and motion data of abduction or adduction of the fingers.
The gesture recognition model establishing method, the gesture recognition method and device and the data glove have the advantages that: when a user wears the data glove and finishes actions through the data glove, acquiring the data acquisition time of each sensor of the data glove, determining the semantic relation among the sensors according to the data acquisition time of each sensor, wherein the semantic relation represents the relevance among the sensors, and the sensors are connected together to establish a gesture recognition model based on a graph attention network, so that the gesture recognition precision of the gesture recognition model can be improved. Moreover, the gesture recognition speed can be improved by automatically recognizing the gesture through the gesture recognition model based on the graph attention network.
Drawings
FIG. 1 is a schematic backside view of a data glove according to an embodiment of the present invention;
FIG. 2 is a schematic front view of a data glove according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the electrical connections of a data glove according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a capacitor acquisition circuit according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of a piezoelectric signal acquisition circuit according to an embodiment of the present invention;
FIG. 6 is a flowchart illustrating the process of executing the Pcap01 chip according to an embodiment of the invention;
FIG. 7 is a flowchart illustrating the execution of a program by the controller according to an embodiment of the present invention;
FIG. 8 is a flowchart illustrating a method for establishing a gesture recognition model according to an embodiment of the present invention;
FIG. 9 is a diagram illustrating a sensor network according to an embodiment of the present invention;
FIG. 10 is a schematic structural diagram of a gesture recognition model building apparatus according to an embodiment of the present invention;
FIG. 11 is a flowchart illustrating a gesture recognition method according to an embodiment of the present invention;
fig. 12 is a schematic structural diagram of a gesture recognition apparatus according to an embodiment of the present invention.
Description of reference numerals:
10-gloves, 20-full flexible capacitive sensors, 30-piezoelectric film sensors, and 40-bases.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. In the description of the present invention, "a plurality" means at least two, e.g., two, three, etc., unless specifically limited otherwise. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein.
As shown in fig. 1 and 2, an embodiment of the present invention provides a data glove, which includes a glove 10, a plurality of sensors and a gesture recognition device, wherein the plurality of sensors are respectively electrically connected to the gesture recognition device, and are respectively disposed on the glove for detecting motion data of each finger.
Preferably, the sensors include a plurality of fully flexible capacitive sensors 20 and a plurality of piezoelectric thin film sensors 30;
the fully flexible capacitive sensors 20 are respectively disposed at the back of each finger and the back of the thumb of the glove 10, and are respectively used for measuring motion data of each finger during flexion or extension and motion data of the thumb during transverse movement, wherein the motion data includes displacement data and the like.
The piezoelectric film sensors 30 are respectively disposed at joints inside each finger of the glove 10 and between two adjacent fingers, and are used for measuring motion data of each finger joint and motion data of abduction or adduction of the finger.
Specifically, when the piezoelectric thin film sensor 30 detects pressure or tension, the direction of the voltage signal is different, and thus the motion direction of the finger can be distinguished according to the direction of the voltage signal. And the piezoelectric film sensor 30 can be used to capture changes in the moment of grasping an object, such as: if the amplitude of the voltage signal output by the piezoelectric film sensor 30 increases, it indicates that the user is gripping an object through the data glove; if the amplitude of the voltage signal decreases, it indicates that the user gradually loosens the object. In addition, the degree of finger flexion may be determined from the stretch signal detected by the fully flexible capacitive sensor 20 in combination, so that under certain scenarios, attributes of the object being gripped may be inferred, such as: when the detected signal indicates that the fingers are bent to a greater degree, it can be inferred that the user is gripping an object with a smaller diameter such as a tool handle; if the detected signal indicates that the finger is bent to a small degree, it can be inferred that the user is gripping an object with a large diameter such as a can.
The piezoelectric film sensor 30 is disposed between two adjacent fingers, and can be used for measuring and judging the magnitude and direction of abduction or adduction of the fingers. The piezoelectric film sensor 30 detects a change in tension or pressure during the abduction or adduction of the finger, and accordingly, the voltage signal detected by the piezoelectric film sensor is a positive output or a negative output, and a change in the amplitude of the voltage signal can be detected. The sensor can be a piezoelectric film sensor with the specification that the output amplitude is less than or equal to 1V and the parameter is 15ms/67Hz, the sensor can be directly used without an amplifying Circuit, two I2C (Inter-Integrated Circuit) modules are reserved and can be used for expanding and fusing with other sensors or inertial measurement units, one SPI (serial peripheral interface) is used for an SD card, one UART (Universal Asynchronous Receiver/Transmitter) is used for time code synchronization, and a Zif (Zero Insertion Force) connector is adopted. Moreover, the piezoelectric film sensor 30 is a passive sensor, and does not need to be additionally powered by a battery.
Meanwhile, an inertia measurement unit for measuring three-axis attitude angles and accelerations of the hand may be provided on the glove, and displacement and hand motion of the hand in a three-dimensional space may be measured in combination with the piezoelectric film sensor 30 and the fully flexible capacitive sensor 20.
In the preferred embodiment, the fully flexible capacitive sensor 20 and the piezoelectric film sensor 30 are adopted, so that the resistance brought by the sensors is smaller, and the data glove is light, thin and easy to wear while the normal motion of the hand is not influenced. Also, combining the fully flexible capacitive sensor 20 and the piezoelectric film sensor 30 to capture hand gestures is more accurate than using a single bending or stretching sensor.
The piezoelectric film sensor 30 has a good dynamic response, a resonance frequency of about 1khz, and no signal output in a static state. On the contrary, the flexible capacitive sensor 20 has a good static response, but only the joint motion displacement can be converted according to the total elongation, the specific joint motion state cannot be accurately judged, and meanwhile, delay occurs in the data acquisition process of more than 70 Hz. The hand gesture can be accurately judged by fusing the data acquired by the two sensors.
Preferably, as shown in fig. 3, the data glove further includes a capacitance acquisition circuit, a piezoelectric signal acquisition circuit, a communication circuit and a controller, the fully flexible capacitance sensor 20 is electrically connected to an input end of the controller through the capacitance acquisition circuit, the piezoelectric film sensor 30 is electrically connected to an input end of the controller through the piezoelectric signal acquisition circuit, and an output end of the controller is electrically connected to the gesture recognition device through the communication circuit.
Electric capacity acquisition circuit, piezoelectricity signal acquisition circuit, communication circuit and controller etc. can integrate on a circuit board, and the circuit board passes through base 40 and installs as required in any position of conveniently wearing and moving, for example: the back of the wrist, back of the palm, on the lower arm and upper arm of the data glove, etc., the base 40 may be manufactured by 3D printing.
As shown in fig. 4, the capacitance collecting circuit includes a chip with a model of Pcap01 and a peripheral circuit, and is configured to Process a capacitance variation collected by the fully flexible capacitance sensor, a micro DSP (Digital Signal processing) chip is integrated in the Pcap01 chip, and can complete some Digital processing operations to convert the capacitance variation into a Digital Signal that can be processed by the controller, and there are many communication methods between the controller and the capacitor, and in this embodiment, the controller communicates with the capacitor in an SPI method.
In fig. 4, C1 is a reference capacitor, the fully flexible capacitive sensor 20 is connected between the PC1 and the PC7, C3, C4, C5, and C6 are necessary coupling capacitors, R5 is a necessary ground resistor, and R1, R2, R3, and R4 are resistors for SPI communication.
As shown in fig. 5, the piezoelectric signal acquisition circuit is composed of an input signal conditioning stage and an ADC stage. The current input signal collected and output by the piezoelectric film sensor 30 is converted into a voltage by a charge-voltage converter (a charge amplifier of the operational amplifier U2A and a capacitor C7), and amplified by a non-inverting amplifier (the operational amplifier U2D and resistors R15 and R16). The reference voltage (VREF ═ 2.5V) of the ADC is buffered and attenuated (op amps U2B and U2C and resistors R9 and R10), producing an offset HREF of 1.25V for conditioning the ac signal from the sensor into the input range of the ADC. Operational amplifiers U2A, U2B, U2C, and U2D are all four channel AD 8608. The output of the U2D op amp is 0.1V to 2.4V, matching the input range of the ADC (0V to 2.5V), while providing 100mV margin to maintain linearity.
The minimum rated output voltage of the AD8608 amplifier is 50mV (2.7V power supply) and 290mV (5V power supply), the load current is 10mA, and the temperature range is-40 ℃ to +125 ℃. Under the conditions of a 3.3V power supply, load current lower than 1mA and a narrower temperature range, the conservative estimation minimum output voltage is 45mV to 60 mV. The circuit design supports a single power supply.
The controller can adopt the singlechip that the model is STM32WB55, and the controller carries out categorised packing with received data to transmit to gesture recognition device through communication circuit. The STM32WB55 wireless microcontroller is a dual-core processor based on a Cortex-M4 core (application processor) and a Cortex-M0+ core (network processor) and supports BluetoothTM5.0 wireless standard. Meanwhile, the STM32WB55 is used as a control core of the whole system, and also controls the functions of the system such as cyclic measurement, data storage, battery charging and discharging management and the like.
Adopt the lithium cell to supply power for each components and parts, the operating voltage of Pcap01 chip and STM32WB55 singlechip is 3.3V, supplies power through external power source when external power source inserts, charges for the lithium cell simultaneously, when no external power source, through the lithium cell power supply.
After the program of the Pcap01 chip and the STM32WB55 singlechip is configured, the fully flexible capacitive sensor 20 is started to start acquisition, the acquired capacitance variation can be stored in the Pap01, the analog quantity is converted into a digital quantity which can be recognized by the singlechip, then the data is transmitted to the STM32WB55 singlechip through an SPI bus, a piezoelectric signal acquired by the piezoelectric film sensor is received, finally the STM32WB55 singlechip sends the data to the gesture recognition device for processing through a communication circuit, the communication circuit can be a bluetooth communication circuit, and the software configuration process of the controller is shown in fig. 7.
Before the gesture corresponding to the motion of the data glove is recognized, the off-line calibration is needed, that is, the size of the hand needs to be measured, and the size of the hand needs to be measured includes, but is not limited to, the palm axial length, the wrist axial length, the distance from the middle finger tip to the wrist (hand length), the distance from the finger gap between the middle finger and the ring finger or the finger gap between the middle finger and the index finger to the wrist, the length of each finger, and the like. And selecting the data glove with the corresponding size according to the measured data. The dimensions of the data glove are shown in table 1:
table 1 data glove dimensions
Wherein, the data of each row corresponds to a data glove of one specification, the unit of the data in table 1 is centimeter, for example: for a small size data glove, the circumference of the palm is 14cm, the circumference of the wrist is about 13cm, the length from the middle finger to the wrist is about 14-15cm, the distance from the wrist to the middle finger and the ring finger (or the index finger) is about 9cm, the length of the corresponding sleeve is about 11cm, the length of the middle finger is 6cm, the length of the index finger and the ring finger is 3.5-4.5cm, and the length of the little finger and the big finger is 2.5-3.5 cm.
After selecting a data glove with a proper size for a user according to an offline calibration result, the user wears the data glove to execute a plurality of calibration actions, obtains hand data collected by each sensor in the process, inputs the hand data into a gesture recognition model, realizes self-adaption of the user and completes online calibration.
As shown in fig. 8, a method for establishing a gesture recognition model according to an embodiment of the present invention includes:
step 110, acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the calibration action;
step 120, determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and step 130, establishing a gesture recognition model based on the graph attention network according to the semantic relation among the sensors.
In the embodiment, when a user wears the data glove and finishes actions through the data glove, the data acquisition time of each sensor of the data glove for acquiring data is obtained, the semantic relation among the sensors is determined according to the data acquisition time of each sensor, the semantic relation represents the relevance among the sensors, the sensors are connected together to establish a gesture recognition model based on a graph attention network, and the gesture recognition precision of the gesture recognition model can be improved. Moreover, the gesture recognition speed can be improved by automatically recognizing the gesture through the gesture recognition model based on the graph attention network.
Preferably, the semantic relationships include a linkage relationship and a concurrency relationship, and the determining the semantic relationship between the sensors according to the acquisition time of each sensor based on a preset rule includes:
for any two sensors, if the two sensors acquire data successively, a linkage process occurs between the two sensors.
Specifically, the two sensors are respectively a sensor a and a sensor B, and if the sensor a detects a data change and the sensor B detects a data change during the process of completing one calibration operation S, it is determined that a linkage process occurs between the sensors a and B.
If the two sensors simultaneously acquire data, a concurrence process occurs between the two sensors.
Specifically, if the sensor a and the sensor B detect data changes simultaneously during the process of completing one calibration operation S, it is determined that a concurrent process occurs between the sensors a and B.
When the data glove executes the calibration action for the calibration times, determining a first probability of a linkage process between the two sensors and a second probability of a concurrent process between the two sensors;
comparing the first probability with a first preset threshold value, and comparing the second probability with a second preset threshold value;
when the first probability is larger than the first preset threshold, a linkage relation exists between the two sensors;
and when the second probability is greater than the second preset threshold value, a concurrency relation exists between the two sensors.
Specifically, when the data glove repeatedly calibrates the number of times of the calibration action S, the number of times of the linkage process between the sensor A and the sensor B is determined, and when the confidence coefficient of the number of times of the calibration action S is larger than a first preset threshold value, the linkage relation between the sensor A and the sensor B is determined.
Expressed by a naive Bayes formula, the number of independent repeated experiments (namely the calibration number) is set as n, one linkage process is determined to be the existence of one linkage event, one concurrency process is determined to be the existence of one concurrency event, the probability of linkage relationship between the sensor A and the sensor B is p (B | A) ═ P (AB)/P (A), wherein P (AB) ═ the number of times/n of linkage events between the sensor A and the sensor B, and P (A) ═ the number of times/n of numerical change of the sensor A, and when the calculated p (B | A) is larger than a first preset threshold value mu, the sensor A and the sensor B are determined to have linkage relationship with respect to the standard action S.
And when the calculated p (B | A) is larger than a second preset threshold value, the sensor A and the sensor B are considered to have a concurrent relation with respect to the standard action S.
Hand motion is the result of the cooperative working of the various joints of the hand, and a gesture typically requires multiple joint linkage and/or concurrent motion.
Preferably, the establishing a gesture recognition model based on a graph attention network according to the semantic relationship between the sensors comprises:
and constructing a sensor network by taking the sensors as nodes and the semantic relation among the sensors as the connection relation among the nodes.
Specifically, as shown in fig. 9, taking the 6 fully flexible capacitive sensors and 18 piezoelectric film sensors of the data glove as an example, a sensor network is established according to semantic relationships between the sensors. A connecting line exists between the two nodes, which indicates that a concurrency relation or a linkage relation exists between the two corresponding sensors, and a connecting line of a single arrow indicates the linkage relation; the connection line of the double arrows represents the concurrency relationship, and the connection line between the two nodes does not exist, which represents that no semantic relationship exists between the two corresponding sensors.
The sensor network can intuitively reflect the relevance among the sensors when a user executes actions through the data glove, can sort and establish the relation among the sensors, and provides prior knowledge for establishing a nonlinear equation among data acquired by the sensors.
And constructing the gesture recognition model based on the graph attention network according to the sensor network.
Specifically, the gesture recognition problem is converted into a multi-posture classification problem based on multi-sensor data fusion, and the method is simple and efficient.
Preferably, the method further comprises the steps of acquiring data collected by each sensor when the data glove finishes the calibration action, and enabling the data collected by each sensor when the data glove finishes the calibration action to be first hand movement data;
after the constructing the gesture recognition model based on the graph attention network according to the sensor network, further comprising: training the gesture recognition model using the first hand motion data.
Specifically, when the data glove finishes the calibration action, first hand motion data and corresponding acquisition time acquired by each sensor are acquired, the first hand motion data and the corresponding acquisition time form time sequence data, and a user is assumed to be in a time sequence interval [0, t ]]The calibration action is completed by the data glove
A matrix formed by the first hand motion data acquired by the n sensors is a matrix
Wherein D isnThe vector represents the nth sensor in the time sequence interval 0, t]And the vectors are formed by the first hand motion data collected in the inner part according to the time sequence.
Preferably, the training of the gesture recognition model using the first hand motion data comprises a forward propagation step, the forward propagation step comprising:
taking the first hand motion data acquired by each sensor as an initial feature representation of the corresponding node;
for any one of the nodes in the sensor network, determining attention coefficients between the node and each first-order neighbor node of the node.
Specifically, the determining attention coefficients between the node and respective first-order neighbor nodes of the node includes:
let the node be a node i, any one of the first-order neighbor nodes of the node i be a node j, and determine an attention coefficient between the node i and the node j according to a first formula according to the initial feature representation of the node i and the initial feature representation of the node j, where the first formula includes:
wherein e isijRepresenting the attention coefficient, a representing a corresponding function from the attention neural network layer,
for the initial characteristic representation of the node i,
w is a weight matrix for the initial feature representation of the node j.
Attention coefficient eijIndicating how important the data characteristic of node j is to node i.
And normalizing all the attention coefficients corresponding to the nodes to obtain normalized attention coefficients.
Specifically, the normalizing all the attention coefficients corresponding to the node, and obtaining the normalized attention coefficients includes:
normalizing all the attention coefficients corresponding to the node i by adopting a second formula, wherein the second formula comprises:
wherein alpha isijRepresenting the normalized attention coefficient,
representing the set of all the first-order neighbor nodes of the node i, the softmax () function being a normalized exponential function.
And updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node to obtain the updated feature representation of the node.
Specifically, the updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each of the first-order neighbor nodes includes:
updating the feature representation of the node i by adopting a third formula according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node, wherein the third formula comprises:
wherein the content of the first and second substances,
represents the updated feature representation, σ represents a tanh activation function of the gesture recognition model, αijIs the normalized attention coefficient, W is a weight matrix,
is the initial feature representation of the node j.
Determining the updated feature representations of all the nodes, and constructing an updated feature matrix according to all the updated feature representations;
inputting the updated feature matrix into a full-connection network of the gesture recognition model, determining the probability that the motion of the data glove is respectively the gesture, and determining the gesture with the maximum probability as a predicted gesture.
Specifically, assume that the updated feature matrix is
The updated feature matrix
Inputting into a fully-connected network, passing through a softmax network layerThe motion of the glove with the output data is the probability of each gesture, and the gesture with the highest probability is selected as the time range [0, t]The internal and sensor time sequence state matrix is
Predicting gestures in situations
Preferably, the training the gesture recognition model using the first hand motion data further comprises:
and a back propagation step, which comprises the steps of making cross entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss.
In particular, to calibration action
And predicting gestures
And performing cross entropy loss, and taking the loss as a parameter for driving and optimizing the gesture recognition model.
And circularly repeating the forward propagation step and the backward propagation step until the loss does not decrease any more, and obtaining a stable gesture recognition model.
Specifically, through iterative optimization of the gesture recognition model for multiple times, when the loss no longer decreases, the gesture recognition model reaches a steady state.
As shown in fig. 10, an embodiment of the present invention provides a gesture recognition model building apparatus, including:
the acquisition module is used for acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the calibration action;
the processing module is used for determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and the establishing module is used for establishing a gesture recognition model based on the graph attention network according to the semantic relation among the sensors.
Another embodiment of the present invention provides a gesture recognition model building apparatus, including a memory and a processor; the memory for storing a computer program; the processor is configured to implement the gesture recognition model building method as described above when executing the computer program. The device can be a computer, a server and the like.
Another embodiment of the present invention provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the computer program implements the gesture recognition model building method as described above.
As shown in fig. 11, a gesture recognition method provided in the embodiment of the present invention includes:
step 210, acquiring data collected by each sensor when the data glove finishes the current action, and enabling the data collected by each sensor when the data glove finishes the current action to be second hand movement data;
step 220, inputting all the second hand motion data into a gesture recognition model, and determining a gesture corresponding to the current motion of the data glove;
wherein the gesture recognition model is established according to the gesture recognition model establishing method.
In this embodiment, when the user carries out the hand activity through data gloves, the sensor gathers the second hand motion data of hand activity process in real time, combines second hand motion data and the collection time that corresponds to form the chronogenesis data, inputs the stable gesture recognition model that the training obtained with the chronogenesis state matrix that the chronogenesis data are constituteed, through the forward propagation step, just can discern the gesture that the user expressed through data gloves, and the recognition accuracy is high, and recognition speed is fast.
As shown in fig. 12, a gesture recognition apparatus provided in an embodiment of the present invention includes:
the acquisition module is used for acquiring data acquired by each sensor when the data glove finishes the current action, and the data acquired by each sensor when the data glove finishes the current action is made to be second hand movement data;
the recognition module is used for inputting all the second hand motion data into a gesture recognition model and determining a gesture corresponding to the current motion of the data glove;
wherein the gesture recognition model is established according to the gesture recognition model establishing method.
Another embodiment of the present invention provides a gesture recognition apparatus including a memory and a processor; the memory for storing a computer program; the processor is configured to implement the gesture recognition method as described above when executing the computer program. The device can be a computer, a server and the like.
A further embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, where the computer program is executed by a processor to implement the gesture recognition model building method as described above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like. In this application, the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment of the present invention. In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
Although the present disclosure has been described above, the scope of the present disclosure is not limited thereto. Various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the spirit and scope of the present disclosure, and these changes and modifications are intended to be within the scope of the present disclosure.
Claims (18)
1. A gesture recognition model building method is characterized by comprising the following steps:
acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the action;
determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and establishing a gesture recognition model based on a graph attention network according to the semantic relation among the sensors.
2. The method for establishing a gesture recognition model according to claim 1, wherein the semantic relationships include a linkage relationship and a concurrency relationship, and the determining the semantic relationship between the sensors according to the acquisition time of each sensor based on a preset rule comprises:
for any two sensors, if the two sensors acquire data successively, a linkage process occurs between the two sensors;
if the two sensors simultaneously acquire data, a concurrence process occurs between the two sensors;
when the data glove executes the action calibration times, determining a first probability of a linkage process between the two sensors and a second probability of a concurrent process between the two sensors;
comparing the first probability with a first preset threshold value, and comparing the second probability with a second preset threshold value;
when the first probability is larger than the first preset threshold, a linkage relation exists between the two sensors;
and when the second probability is greater than the second preset threshold value, a concurrency relation exists between the two sensors.
3. The method according to claim 2, wherein the establishing a gesture recognition model based on a graph attention network according to the semantic relationship between the sensors comprises:
establishing a sensor network by taking the sensors as nodes and the semantic relation among the sensors as the connection relation among the nodes;
and constructing the gesture recognition model based on the graph attention network according to the sensor network.
4. The method for establishing the gesture recognition model according to claim 3, further comprising the step of acquiring data collected by each sensor when the data glove completes calibration action;
after the constructing the gesture recognition model based on the graph attention network according to the sensor network, further comprising: and training the gesture recognition model by adopting the data collected by each sensor.
5. The method according to claim 4, wherein the training of the gesture recognition model using the data collected by each sensor comprises a forward propagation step, and the forward propagation step comprises:
taking the data collected by each sensor as the initial characteristic representation of the corresponding node;
for any one node in the sensor network, determining attention coefficients between the node and each first-order neighbor node of the node respectively;
normalizing all the attention coefficients corresponding to the nodes to obtain normalized attention coefficients;
updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node to obtain the updated feature representation of the node;
determining the updated feature representations of all the nodes, and constructing an updated feature matrix according to all the updated feature representations;
inputting the updated feature matrix into a full-connection network of the gesture recognition model, determining the probability that the motion of the data glove is respectively the gesture, and determining the gesture with the maximum probability as a predicted gesture.
6. The method according to claim 5, wherein the determining attention coefficients between the respective nodes and respective first-order neighbor nodes of the node comprises:
setting the node as a node i, setting any one of the first-order neighbor nodes of the node i as a node j, and determining an attention coefficient between the node i and the node j by adopting a first formula according to the initial feature representation of the node i and the initial feature representation of the node j, wherein the first formula comprises:
wherein e isijRepresenting the attention coefficient, a representing a corresponding function from the attention neural network layer,
for the initial characteristic representation of the node i,
w is a weight matrix for the initial feature representation of the node j.
7. The method according to claim 6, wherein the normalizing all the attention coefficients corresponding to the nodes to obtain normalized attention coefficients comprises:
normalizing all the attention coefficients corresponding to the node i by adopting a second formula, wherein the second formula comprises:
wherein alpha isijRepresenting the normalized attention coefficient,
a set of all the first-order neighbor nodes representing the node i.
8. The method according to claim 7, wherein the updating the feature representation of the node according to the normalized attention coefficient and the initial feature representation of each of the first-order neighbor nodes comprises:
updating the feature representation of the node i by adopting a third formula according to the normalized attention coefficient and the initial feature representation of each first-order neighbor node, wherein the third formula comprises:
wherein the content of the first and second substances,
representing the updated feature representation, σ representing an activation function of the gesture recognition model, αijIs the normalized attention coefficient, W is a weight matrix,
is the initial feature representation of the node j.
9. The method for building a gesture recognition model according to any one of claims 5 to 8, wherein the training the gesture recognition model using the data collected by each sensor further comprises:
a back propagation step, which comprises making cross entropy loss according to the calibration action and the predicted gesture, and optimizing the gesture recognition model according to the loss;
and circularly repeating the forward propagation step and the backward propagation step until the loss does not decrease any more, and obtaining a stable gesture recognition model.
10. A gesture recognition method, comprising:
acquiring data collected by each sensor when the data glove finishes the current action;
inputting all data collected by each sensor into a gesture recognition model, and determining a gesture corresponding to the current action of the data glove;
wherein the gesture recognition model is built according to the gesture recognition model building method of any one of claims 1 to 9.
11. A gesture recognition model creation apparatus, comprising:
the acquisition module is used for acquiring the data acquisition time of each sensor of the data glove when the data glove finishes the action;
the processing module is used for determining semantic relations among the sensors according to the acquisition time of the sensors based on a preset rule;
and the establishing module is used for establishing a gesture recognition model based on the graph attention network according to the semantic relation among the sensors.
12. A gesture recognition model building device is characterized by comprising a memory and a processor;
the memory for storing a computer program;
the processor, when executing the computer program, is configured to implement the gesture recognition model building method according to any one of claims 1 to 9.
13. A gesture recognition apparatus, comprising:
the acquisition module is used for acquiring data acquired by each sensor when the data glove finishes the current action;
the recognition module is used for inputting all data collected by the sensors into a gesture recognition model and determining a gesture corresponding to the current action of the data glove;
wherein the gesture recognition model is built according to the gesture recognition model building method of any one of claims 1 to 9.
14. A gesture recognition apparatus comprising a memory and a processor;
the memory for storing a computer program;
the processor, when executing the computer program, for implementing the gesture recognition method of claim 10.
15. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out a gesture recognition model building method according to any one of claims 1 to 9.
16. A computer-readable storage medium, characterized in that the storage medium has stored thereon a computer program which, when being executed by a processor, carries out the gesture recognition method as claimed in claim 10.
17. A data glove comprising a glove, a plurality of sensors and a gesture recognition device according to claim 14, wherein the plurality of sensors are electrically connected to the gesture recognition device respectively, and the plurality of sensors are disposed on the glove respectively for detecting motion data of each finger.
18. The data glove of claim 17, wherein the sensors comprise a plurality of fully flexible capacitive sensors and a plurality of piezoelectric thin film sensors;
the fully-flexible capacitive sensors are respectively arranged at the back of each finger and the back of the thumb of the glove and are respectively used for measuring motion data of the fingers in the flexing or extending process and motion data of the thumb in the transverse motion process;
the piezoelectric film sensors are respectively arranged at joints on the inner sides of the fingers of the glove and between two adjacent fingers and are respectively used for measuring motion data of joints of the fingers and motion data of abduction or adduction of the fingers.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010832352.9A CN111984119B (en) | 2020-08-18 | 2020-08-18 | Gesture recognition model establishing method, gesture recognition method and device and data glove |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010832352.9A CN111984119B (en) | 2020-08-18 | 2020-08-18 | Gesture recognition model establishing method, gesture recognition method and device and data glove |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111984119A true CN111984119A (en) | 2020-11-24 |
| CN111984119B CN111984119B (en) | 2023-03-31 |
Family
ID=73434013
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010832352.9A Active CN111984119B (en) | 2020-08-18 | 2020-08-18 | Gesture recognition model establishing method, gesture recognition method and device and data glove |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111984119B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112971773A (en) * | 2021-03-12 | 2021-06-18 | 哈尔滨工业大学 | Hand motion mode recognition system based on palm bending information |
| CN113408349A (en) * | 2021-05-17 | 2021-09-17 | 浙江大华技术股份有限公司 | Training method of motion evaluation model, motion evaluation method and related equipment |
| CN115294658A (en) * | 2022-08-24 | 2022-11-04 | 哈尔滨工业大学 | Personalized gesture recognition system and gesture recognition method for multiple application scenes |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008207262A (en) * | 2007-02-23 | 2008-09-11 | Matsushita Electric Works Ltd | Manipulator system |
| CN105930785A (en) * | 2016-04-15 | 2016-09-07 | 丁盛 | Intelligent concealed-type interaction system |
| CN107678550A (en) * | 2017-10-17 | 2018-02-09 | 哈尔滨理工大学 | A kind of sign language gesture recognition system based on data glove |
| CN109446944A (en) * | 2018-10-15 | 2019-03-08 | 西安文理学院 | Vision semanteme-structuring analytic method of sign language |
| CN109976526A (en) * | 2019-03-27 | 2019-07-05 | 广东技术师范大学 | A kind of sign Language Recognition Method based on surface myoelectric sensor and nine axle sensors |
| CN110765967A (en) * | 2019-10-30 | 2020-02-07 | 腾讯科技（深圳）有限公司 | Action recognition method based on artificial intelligence and related device |
| CN111476161A (en) * | 2020-04-07 | 2020-07-31 | 金陵科技学院 | Somatosensory dynamic gesture recognition method fusing image and physiological signal dual channels |
| WO2021123772A1 (en) * | 2019-12-20 | 2021-06-24 | Brightsign Technology Limited | Method and system for gesture recognition |
-
2020
- 2020-08-18 CN CN202010832352.9A patent/CN111984119B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008207262A (en) * | 2007-02-23 | 2008-09-11 | Matsushita Electric Works Ltd | Manipulator system |
| CN105930785A (en) * | 2016-04-15 | 2016-09-07 | 丁盛 | Intelligent concealed-type interaction system |
| CN107678550A (en) * | 2017-10-17 | 2018-02-09 | 哈尔滨理工大学 | A kind of sign language gesture recognition system based on data glove |
| CN109446944A (en) * | 2018-10-15 | 2019-03-08 | 西安文理学院 | Vision semanteme-structuring analytic method of sign language |
| CN109976526A (en) * | 2019-03-27 | 2019-07-05 | 广东技术师范大学 | A kind of sign Language Recognition Method based on surface myoelectric sensor and nine axle sensors |
| CN110765967A (en) * | 2019-10-30 | 2020-02-07 | 腾讯科技（深圳）有限公司 | Action recognition method based on artificial intelligence and related device |
| WO2021123772A1 (en) * | 2019-12-20 | 2021-06-24 | Brightsign Technology Limited | Method and system for gesture recognition |
| CN111476161A (en) * | 2020-04-07 | 2020-07-31 | 金陵科技学院 | Somatosensory dynamic gesture recognition method fusing image and physiological signal dual channels |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112971773A (en) * | 2021-03-12 | 2021-06-18 | 哈尔滨工业大学 | Hand motion mode recognition system based on palm bending information |
| CN113408349A (en) * | 2021-05-17 | 2021-09-17 | 浙江大华技术股份有限公司 | Training method of motion evaluation model, motion evaluation method and related equipment |
| CN113408349B (en) * | 2021-05-17 | 2023-04-18 | 浙江大华技术股份有限公司 | Training method of motion evaluation model, motion evaluation method and related equipment |
| CN115294658A (en) * | 2022-08-24 | 2022-11-04 | 哈尔滨工业大学 | Personalized gesture recognition system and gesture recognition method for multiple application scenes |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111984119B (en) | 2023-03-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111984119B (en) | Gesture recognition model establishing method, gesture recognition method and device and data glove | |
| Huang et al. | Machine learning-based multi-modal information perception for soft robotic hands | |
| Schurmann et al. | A modular high-speed tactile sensor for human manipulation research | |
| CN102686366A (en) | Controller and control method for master-slave robot, master-slave robot, control program, and integrated electronic circuit | |
| CN210402266U (en) | Sign language translation system and sign language translation gloves | |
| Adnan et al. | Measurement of the flexible bending force of the index and middle fingers for virtual interaction | |
| CN105919599B (en) | Finger movement detection identifying system and method based on Magnetic Sensor | |
| CN110827987A (en) | Myoelectricity continuous prediction method and system for wrist joint torque in multi-grabbing mode | |
| CN105446485A (en) | System and method for capturing hand movement function based on data glove and position tracker | |
| CN112428308A (en) | Robot touch action recognition system and recognition method | |
| Van Volkinburg et al. | Development of a wearable controller for gesture-recognition-based applications using polyvinylidene fluoride | |
| Pan et al. | State-of-the-art in data gloves: a review of hardware, algorithms, and applications | |
| CN108523281B (en) | Glove peripheral, method, device and system for virtual reality system | |
| CN109542220B (en) | Sign language gloves with calibration and learning functions, system and implementation method | |
| McCaw et al. | Sensory glove for dynamic hand proprioception and tactile sensing | |
| Wang et al. | SmartHand: Towards embedded smart hands for prosthetic and robotic applications | |
| CN116909390A (en) | Multi-mode data acquisition system based on glove | |
| Wagner et al. | Smart glove with fully integrated textile sensors and wireless sensor frontend for the tactile internet | |
| Fang et al. | A petal-array capacitive tactile sensor with micro-pin for robotic fingertip sensing | |
| Ha et al. | Use of deep learning for position estimation and control of soft glove | |
| Jayaraman et al. | Nonspecific-user hand gesture recognition by using MEMS accelerometer | |
| Lin et al. | Multimodal Surface Sensing based on Hybrid Flexible Triboelectric and Piezoresistive Sensor | |
| CN220313362U (en) | Under-driven bionic smart hand | |
| Morozova et al. | Robotic glove prototype: Development and simulation | |
| CN104656883A (en) | Gesture acquisition system based on multiple acceleration sensors and ZigBee network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |