US12422935B2 - Gesture information processing method and apparatus, electronic device, and storage medium - Google Patents

Gesture information processing method and apparatus, electronic device, and storage medium

Info

Publication number
US12422935B2
US12422935B2 US18/227,857 US202318227857A US12422935B2 US 12422935 B2 US12422935 B2 US 12422935B2 US 202318227857 A US202318227857 A US 202318227857A US 12422935 B2 US12422935 B2 US 12422935B2
Authority
US
United States
Prior art keywords
neural network
network model
electromyography
gesture information
gesture
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US18/227,857
Other versions
US20230376120A1 (en
Inventor
Xiaolin HONG
Qingqing Zheng
Xinmin Wang
Kai Ma
Yefeng Zheng
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tencent Technology Shenzhen Co Ltd
Original Assignee
Tencent Technology Shenzhen Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tencent Technology Shenzhen Co Ltd filed Critical Tencent Technology Shenzhen Co Ltd
Priority to US18/227,857 priority Critical patent/US12422935B2/en
Assigned to TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED reassignment TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MA, KAI, HONG, Xiaolin, WANG, XINMIN, ZHENG, Qingqing, ZHENG, YEFENG
Publication of US20230376120A1 publication Critical patent/US20230376120A1/en
Application granted granted Critical
Publication of US12422935B2 publication Critical patent/US12422935B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/014—Hand-worn input/output arrangements, e.g. data gloves
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/70—Denoising; Smoothing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/02—Preprocessing
    • G06F2218/04—Denoising
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/15—Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
    • Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
    • Y02P90/30—Computing systems specially adapted for manufacturing

Definitions

  • This application relates to information processing technologies, and in particular, to a gesture information processing method and apparatus, an electronic device, and a storage medium.
  • Electromyography signals are superposition of motor unit action potentials (MUAPs) in numerous muscle fibers in time and space. According to different recording manners, electromyography signals recorded by an electromyography may be classified into surface electromyography signals and needle electromyography signals, all of which include anatomical and physiological properties of muscles.
  • the surface Electromyography (sEMG) signals are a combined effect of superficial muscle electromyography signals and nerve stem electrical activities on skin surfaces, which can reflect neuromuscular activities to some extent. Compared with the needle electromyography signals, the surface electromyography signals have advantages such as non-invasiveness and simple operation in measurement.
  • the surface electromyography signals have important application values in clinical medicine, rehabilitation medicine, human-computer interfaces and the like.
  • Conventional electromyography-based gesture recognition technologies generally use gesture recognition of visual sensors (such as ordinary cameras or depth cameras) and are based on wearable sensors (such as data gloves).
  • wearable sensors such as data gloves.
  • the wearable sensors cannot timely and accurately recognize users' gestures.
  • embodiments of this application provide a gesture information processing method and apparatus, an electronic device, and a storage medium, so as to accurately and timely recognize gesture information in different usage environments and have stronger robustness for automatically extracted electromyography signals, thereby improving accuracy of classification of surface electromyography signals and reducing interference of noise with gesture information recognition.
  • the embodiments of this application provide a gesture information processing method performed by an electronic device, the method including:
  • the embodiments of this application further provide a gesture information processing apparatus, including:
  • inventions of this application further provide an electronic device, comprising:
  • the embodiments of this application further provide a non-transitory computer-readable storage medium, storing executable instructions, the gesture information processing method being performed when the executable instructions are executed by a processor.
  • FIG. 1 is a schematic diagram of an application environment of a gesture recognition model according to an embodiment of this application
  • FIG. 2 is a schematic structural diagram of composition of an electronic device according to an embodiment of this application.
  • FIG. 3 is a schematic diagram of a process of generating gesture information in a conventional solution
  • FIG. 4 is a schematic diagram of a process of generating gesture information in the conventional solution
  • FIG. 5 is an exemplary schematic flowchart of a gesture information processing method according to an embodiment of this application.
  • FIG. 6 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application.
  • FIG. 7 is an exemplary schematic flowchart of a method of using the gesture recognition model according to an embodiment of this application.
  • FIG. 8 is an exemplary schematic structural diagram of a first neural network model according to an embodiment of this application.
  • FIG. 9 is an exemplary schematic structural diagram of the first neural network model according to an embodiment of this application.
  • FIG. 10 is a schematic diagram of an application environment of the gesture recognition model according to an embodiment of this application.
  • FIG. 11 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application.
  • FIG. 12 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application.
  • FIG. 13 is an exemplary schematic diagram of sample acquisition of the gesture recognition model according to an embodiment of this application.
  • FIG. 14 is an exemplary schematic structural diagram of the gesture recognition model according to an embodiment of this application.
  • FIG. 15 is an exemplary schematic flowchart of a process of using the gesture recognition model according to an embodiment of this application.
  • An “artificial neural network”, Neural Network (NN) for short, is a mathematical model or computational model that imitates structures and functions of biological neural networks in fields of machine learning and cognitive science, for estimating or approximating functions.
  • model parameter is a quantity to use a generic variable to establish a relationship between functions and variables.
  • the model parameter is generally a real matrix.
  • An “sEMG” refers to a muscle biograph obtained by measuring and recording bioelectricity of surface muscles through a special electromyography or polygraph.
  • a “residual network (ResNet)” is a deep feedforward neural network that introduces skip connections to achieve identity mapping.
  • a “contrastive loss” may learn a mapping relation, which enables points of a same category but far away in a high-dimensional space to become closer after being mapped to a low-dimensional space by the contrastive loss, and points of different categories but close to each other to become further in the low-dimensional space after being mapped.
  • points of a same kind may produce a clustering effect, and mean of different kinds may be separated. It is similar to fisher dimension reduction, but fisher dimension reduction does not have an effect of out-of-sample extension and cannot affect a new sample.
  • Model training means performing multi-classification learning on image data sets.
  • the model may be constructed by deep learning frameworks such as TensorFlow and torch, and a multi-classification model is formed using a multi-layer combination of neural network layers such as a convolutional neural network (CNN).
  • Input to the model is a three-channel or original-channel matrix formed after an image is read by a tool such as openCV.
  • Output from the model is multi-classification probability.
  • a web page category is finally outputted through an algorithm such as softmax.
  • the model approximates to a correct trend through an objective function such as cross entropy.
  • a “client” is a carrier to achieve a specific function in a terminal.
  • a mobile client APP
  • APP mobile client
  • Soft max is a normalized exponential function that is a promotion of a logic function.
  • the function can “compress” a K-dimension vector including any real number into another K-dimension real vector, so that each element ranges within (0, 1), and a sum of all elements is 1.
  • FIG. 1 is a schematic diagram of an application environment of a gesture recognition model according to an embodiment of this application.
  • a terminal including a terminal 10 - 1 and a terminal 10 - 2
  • a client for gesture recognition which may acquire electromyography samples of different users.
  • a human performs different gestures
  • a muscle group may produce different electromyography signals under the coordination of a nervous system.
  • An sEMG records bioelectrical signals on skin surfaces of human muscles and effectively captures information of muscle activities. Intentions of human gestures may be decoded through gesture analysis on the surface electromyograph.
  • the terminal is connected to a server 200 by using a network 300 .
  • the network 300 may be a wide area network, a local area network, or a combination of the wide area network and the local area network, and achieves data transmission by using a radio link.
  • the terminal 10 - 1 and the terminal 10 - 2 may also function as powered skeletons or somatosensory game machines with a gesture recognition model run therein. Surface electromyography signals are transmitted, decoded and controlled by embedding the gesture recognition model into different hardware systems or software systems.
  • 1) medical systems combined with powered exoskeletons may be configured for active rehabilitation of arm motor functions in patients with hemiplegia and spinal cord injuries.
  • Monitoring systems combined with physical training may evaluate muscle fatigue, balance and use efficiency in real time to improve training efficiency and reduce training injuries.
  • Gesture interaction systems combined with games enable humans to control activities of objects in the virtual world through gesture movement.
  • Gesture information obtained by the terminal (including the terminal 10 - 1 and the terminal 10 - 2 ) from a corresponding server 200 via a network 300 is of different types.
  • the terminal including the terminal 10 - 1 and the terminal 10 - 2
  • different types of gesture information stored in the server 200 may be written in software code environments of different programming languages, and code objects may be different types of code entities.
  • a code object may be a function.
  • a code object may be a class, which may be a piece of object code in IOS OC language.
  • a code object may be a class or a function to perform electromyography signals from different terminals. Sources of gesture information to be processed by the gesture recognition model are not distinguished from one another in this application.
  • the server 200 needs to determine and recognize electromyography signals generated during gesture triggering when the server 200 sends different types of gesture information to the terminal (including the terminal 10 - 1 and the terminal 10 - 2 ) via the network 300 .
  • the server 200 is configured to obtain different electromyography signals of a same target object and denoise the different electromyography signals; recognize the denoised different electromyography signals based on the gesture information label through the gesture recognition model, and determine probabilities of gesture information represented by the different electromyography signals; and weight the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
  • a process of training the gesture recognition model includes: obtaining a first training sample set, where the first training sample set includes at least one group of electromyography signal samples and the corresponding gesture information label; denoising the first training sample set, so as to form a corresponding second training sample set; determining initial parameters of the first neural network model in the gesture recognition model; processing the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determining update parameters corresponding to different neural networks of the gesture recognition model; and iteratively updating parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model, so as to recognize different gesture information through the gesture recognition model.
  • FIG. 2 is a schematic structural diagram of composition of an electronic device according to an embodiment of this application. It may be understood that, FIG. 2 shows only an exemplary structure rather than a complete structure of the electronic device. The structure shown in FIG. 2 may be partially or entirely implemented based on requirements.
  • the electronic device provided in this embodiment of this application includes: at least one processor 201 , a memory 202 , a user interface 203 , and at least one network interface 204 .
  • the components in the electronic device are coupled by using a bus system 205 .
  • the bus system 205 is configured to implement connection and communication between the components.
  • the bus system 205 further includes a power bus, a control bus, and a state signal bus.
  • all types of buses are labeled as the bus system 205 in FIG. 2 .
  • the user interface 203 may include a display, a keyboard, a mouse, a track ball, a click wheel, a key, a button, a touch panel, a touchscreen, or the like.
  • the memory 202 may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory.
  • the memory 202 in the embodiment of this application can store data to support operation of the terminal (for example, 10 - 1 ).
  • An example of the data includes any computer program to be operated on the terminal (for example, 10 - 1 ), for example, an operating system and an application program.
  • the operating system includes various system programs, such as framework layers, kernel library layers, and driver layers used for implementing various basic services and processing hardware-based tasks.
  • the application program may include various application programs.
  • a gesture information recognition apparatus provided in the embodiments of this application may be implemented in the form of a combination of software and hardware.
  • the gesture information recognition apparatus provided in the embodiments of this application may be a processor in the form of a hardware decoding processor, and is programmed to execute the gesture information recognition method provided in the embodiments of this application.
  • the processor in the form of a hardware decoding processor may use one or more ASICs, digital signal processor (DSP), programmable logic devices (PLDs), complex programmable logic device (CPLD), FPGAs, or other electronic elements.
  • the gesture information recognition provided in the embodiments of this application may be directly embodied as a combination of software modules executed by the processor 201 .
  • the software modules may be located in a storage medium, and the storage medium is located in the memory 202 .
  • the processor 201 reads executable instructions included in the software modules in the memory 202 and uses necessary hardware (for example, including the processor 201 and other components connected to the bus 205 ) in combination, to complete the gesture information recognition method provided in the embodiments of this application.
  • the processor 201 may be an integrated circuit chip, and has a signal processing capability, for example, a general purpose processor, a digital signal processor (DSP), or another programmable logical device, a discrete gate or a transistor logical device, or a discrete hardware component.
  • the general purpose processor may be a microprocessor, any conventional processor, or the like.
  • the gesture information recognition apparatus provided in the embodiments of this application may be directly executed by using the processor 201 in the form of a hardware decoding processor, for example, one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, or other electronic elements, to execute the gesture information recognition method provided in the embodiments of this application.
  • a hardware decoding processor for example, one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, or other electronic elements, to execute the gesture information recognition method provided in the embodiments of this application.
  • the memory 202 in this embodiment of this application is configured to store various types of data to support operation of the electronic device.
  • An example of the data includes: any executable instruction configured to be operated on the electronic device, such as an executable instruction, and a program that implements the gesture information recognition method of the embodiments of this application may be included in the executable instruction.
  • the gesture information recognition apparatus may be implemented in the form of software.
  • FIG. 2 shows a gesture information recognition apparatus 2020 stored in the memory 202 , which may be software in the form of a program, a plug-in, or the like, and include a series of modules.
  • An example of the program stored in the memory 202 may include the gesture information recognition apparatus 2020 .
  • the gesture information recognition 2020 includes the following software modules: a first information transmission module 2081 and an information processing module 2082 .
  • the software modules in the gesture information recognition apparatus 2020 are read into the RAM by the processor 201 and executed, the gesture information recognition method according to the embodiment of this application is performed. Functions of the software modules in the gesture information recognition apparatus 2020 are introduced below.
  • the information processing module 2082 is configured to determine an electromyography signal collection target object in a gesture information usage environment.
  • the first information transmission module 2081 is configured to acquire an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label.
  • the information processing module 2082 is configured to divide the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoise the different electromyography signals of the target object.
  • the information processing module 2082 is configured to recognize the denoised different electromyography signals based on the gesture information label, and determine probabilities of gesture information represented by the different electromyography signals.
  • the information processing module 2082 is configured to weight the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
  • FIG. 3 is a schematic diagram of a process of generating gesture information in a conventional solution.
  • PSD power spectral density
  • FIG. 4 is a schematic diagram of a process of generating gesture information in the conventional solution.
  • a classification algorithm based on FFT+PCA+CNN requires pre-extraction of FFT features and reduction of data dimensions by principal component analysis, which not only has a plurality of uncertain hyperparameters, but also results in inconsistent objectives of feature extraction and classification model optimization, and end-to-end signal decoding cannot be realized.
  • the relevant algorithms do not take continuity before and after gesture prediction into account, which may also lead to poor stability of real-time gesture prediction, affect recognition of continuous user gestures, and affect user experience.
  • FIG. 5 is an exemplary schematic flowchart of a gesture information processing method according to an embodiment of this application. It may be understood that, steps shown in FIG. 5 may be performed by various electronic devices running the gesture information processing apparatus, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 5 .
  • Step 501 A gesture information processing apparatus obtains a first training sample set.
  • the first training sample set includes at least one group of electromyography signal samples and the corresponding gesture information label.
  • Step 502 The gesture information processing apparatus denoises the first training sample set, so as to form a corresponding second training sample set.
  • the denoising the first training sample set, so as to form a corresponding second training sample set may be performed in following manners:
  • the denoising the first training sample set, so as to form a corresponding second training sample set may be performed in following manners:
  • the gesture information processing method further includes:
  • Step 503 The gesture information processing apparatus determines initial parameters of the first neural network model in the gesture recognition model.
  • the determining initial parameters of the first neural network model in the gesture recognition model may be performed in following manners:
  • the gesture information processing method further includes:
  • the first neural network model in the gesture recognition model may be a trained neural network, and the first neural network model in the gesture recognition model may be a large-scale neural network.
  • a quantity of network parameters of the first neural network model in the gesture recognition model is greater than a value, which is not limited in the embodiment of this application.
  • the first neural network model in the gesture recognition model may be a residual network (ResNet), a CNN, a deep neural network (DNN), a recurrent neural network (RNN) or the like.
  • the type of the first neural network model in the gesture recognition model is not limited in the embodiment of this application.
  • the first neural network model in the gesture recognition model may be a neural network suitable for different computer vision tasks, such as a target recognition task, a target classification task, a target detection task or an attitude estimation task.
  • the first neural network model in the gesture recognition model may also be a neural network suitable for different application scenarios, such as a security monitoring scenario, a gesture unlocking scenario or an intelligent driving scenario.
  • An application scope of the first neural network model in the gesture recognition model is not limited in the embodiment of this application.
  • a network structure of the first neural network model in the gesture recognition model may be designed according to computer vision tasks or adopt at least part of a related network structure, such as a DNN or a visual geometrygroup network (VGGNet).
  • VVGNet visual geometrygroup network
  • the network structure of the first neural network model in the gesture recognition model is not limited in the embodiment of this application.
  • a contrastive loss is applied to reduce the loss of the differences between the signals of the same kind and improve robustness of the algorithm.
  • a second neural network model and a third neural network model may also be introduced to determine a classification loss function for evaluating gesture information classification and a contrastive loss of separability between inner classes of corresponding individuals respectively.
  • Step 504 The gesture information processing apparatus processes the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determines update parameters corresponding to different neural networks of the gesture recognition model.
  • Step 505 The gesture information processing apparatus iteratively updates parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model.
  • FIG. 6 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application. It may be understood that, steps shown in FIG. 6 may be performed by various electronic devices running the gesture information processing apparatus, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 6 .
  • Step 601 Keep the initial parameters of the gesture recognition model unchanged, and determine output results of the gesture recognition model.
  • Step 602 Substitute, in response to the output results of the gesture recognition model, different training samples in the second training sample set into the fusion loss matching the gesture recognition model.
  • Step 603 Determine parameters corresponding to different networks in the gesture recognition model when the fusion loss meets the convergence condition as update parameters of the gesture recognition model.
  • Step 604 Iteratively update parameters of the different neural networks of the gesture recognition model respectively based on the second training sample set and according to the update parameters corresponding to the different neural networks of the gesture recognition model, until losses corresponding to the different neural networks of the gesture recognition model meets the corresponding convergence condition.
  • FIG. 7 is an exemplary schematic flowchart of a method of using the gesture recognition model according to an embodiment of this application. It may be understood that, steps shown in FIG. 7 may be performed by various electronic devices running the gesture recognition model, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 7 .
  • Step 701 A server determines an electromyography signal collection target object in a gesture information usage environment of a gesture recognition model.
  • Step 702 The server acquires an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label.
  • Step 703 The server divides the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoises the different electromyography signals of the target object.
  • Step 704 The server recognizes the denoised different electromyography signals based on the gesture information label, and determines probabilities of gesture information represented by the different electromyography signals.
  • Step 705 The server weights the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
  • the different electromyography signals of the target object may be obtained in following manners:
  • the denoising the different electromyography signals may be performed in following manners:
  • features of corresponding electromyography signals are determined when a usage environment of the gesture recognition model is a powered skeleton.
  • the output sizes of the different networks in the first neural network model of the gesture recognition model and the quantity of the corresponding convolution kernels are determined according to features of electromyography signals.
  • the quantities of the temporal convolutional layer network, the spatial convolutional layer network, the fully connected layer network and the residual module network of the first neural network model are determined according to features of the powered skeleton, so as to match initialization parameters of the gesture recognition model with the features of the powered skeleton.
  • the training time of the model may be shortened, and the trained model may be provided for users to directly call through an applet.
  • FIG. 8 is an exemplary schematic structural diagram of a first neural network model according to an embodiment of this application.
  • the first neural network model is a full convolutional residual neural network, which successively includes four convolutional layers: convolutional layers 1 to 4 . 17 serial residual blocks and an addition layer are included between the convolutional layer 1 and the convolutional layer 2 , and output of the residual block 17 and output of the convolutional layer 1 serve as input to the addition layer.
  • Preliminary feature extraction is performed on an input image by using one convolutional layer 1 , to obtain preliminarily extracted features.
  • Feature extraction is continuously performed on the preliminarily extracted features by using the series residual blocks 1 to 17 and features and texture are supplemented to output of a previous residual block.
  • the output of the residual block 17 is supplemented to the preliminarily extracted features through the addition layer.
  • Resolution of an image outputted from the addition layer is amplified to super-resolution of a specified multiple by using series convolutional layers 2 and 3 —pixel realignment layer—convolutional layer 4 , to obtain a super-resolution image.
  • FIG. 9 is an exemplary schematic structural diagram of the first neural network model according to an embodiment of this application.
  • a network structure of a residual block i is shown in FIG. 9 , including a convolutional layer i 1 , a convolutional layer i 2 and an addition layer i 1 connected in series. 0 ⁇ i ⁇ 1.
  • the addition layer i 1 adds input of residual block i 1 to a corresponding element outputted from the convolutional layer i 2 , which is inputted to the following fully connected layer and processed by the fully connected layer and a normalized exponential function layer connected to the fully connected layer, to determine a probability that the electromyography signal belongs to the corresponding gesture information.
  • FIG. 10 is a schematic diagram of an application environment of the gesture recognition model according to an embodiment of this application.
  • a terminal including a powered exoskeleton 100 - 1 and a terminal 100 - 2
  • a client for gesture recognition which may acquire electromyography samples of different users.
  • a human performs different gestures
  • a muscle group may produce different electromyography signals under the coordination of a nervous system.
  • An sEMG records bioelectrical signals on skin surfaces of human muscles and effectively captures information of muscle activities. Intentions of human gestures may be decoded through gesture analysis on the surface electromyograph.
  • the terminal is connected to a server 200 via a network 300 .
  • the network 300 may be a wide area network, a local area network, or a combination thereof, and transmit data by using wireless links.
  • the terminal 10 - 1 and the terminal 10 - 2 may also function as powered skeletons or somatosensory game machines with a gesture recognition model run therein. Surface electromyography signals are transmitted, decoded and controlled by embedding the gesture recognition model into different hardware systems or software systems.
  • 1) medical systems combined with powered exoskeletons may be configured for active rehabilitation of arm motor functions in patients with hemiplegia and spinal cord injuries.
  • Monitoring systems combined with user posture monitoring may evaluate, in real time, fatigue, balance and use efficiency of muscles to which the powered exoskeleton is attached, improving training efficiency and reducing injuries of a target user.
  • FIG. 11 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application, specifically including the following steps:
  • Step 1101 Obtain a first training sample set, where the first training sample set includes at least one group of electromyography samples.
  • Step 1102 Denoise the first training sample set, so as to form a corresponding second training sample set.
  • FIG. 12 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application.
  • Original multi-channel surface electromyography signals are preprocessed to obtain training samples, the training samples are then inputted into a deep residual network to extract features which mainly include time domain features and space domain features related to electromyography gestures, probabilities of different gestures are obtained and classified, and finally, network parameters are continuously optimized by iterative training.
  • Cortical electromyography data of a target user may be obtained by a corresponding acquisition device.
  • the data set includes electromyography data acquired by 10 subjects (e.g., 5 males and 5 females) in two time periods.
  • the electromyography data in the first time period is used as a training sample data set, and the electromyography data in the second time period is used as a test sample data set.
  • a data acquisition process is as follows:
  • an acquisition apparatus e.g., an MYO electromyography bracelet
  • An experimenter informs the subject of the data acquisition process and specifications.
  • surface electromyography signals may be divided by a sliding window as a single sample to be sent to the network.
  • a time window is 200 ms, and a window sliding stride is 40 ms. Since a signal sampling frequency is 200 Hz and 8 electrode channels are included, each signal sample has a dimension of 8 ⁇ 40.
  • Step 1103 Determine initial parameters of the first neural network model in the gesture recognition model.
  • the first neural network model may be a ResNet.
  • an inputted sEMG signal is processed by a spatial convolutional layer for the spatial properties and a temporal convolutional layer for the temporal properties.
  • Model parameters of the first neural network model are shown in Table 1.
  • the signal inputted to the network has a dimension of 8 ⁇ 40.
  • First two temporal-spatial convolutional layers Spatial Cony and Temporal Cony perform convolutional operations on the signal in a spatial (channel) dimension and a temporal dimension respectively, with strides of 1, to realize spatial and temporal filtering of the sEMG signal.
  • each residual module includes two residual networks.
  • Each residual network is formed by a convolutional layer with a 1*3 convolutional kernel and a batch normalization layer.
  • the features are further fused through the fully connected layer, and classification results of electromyography gestures are outputted by SoftMax.
  • the network structure of the solution may be realized by appropriately increasing a quantity of the temporal and spatial convolutional layers, increasing a number of the fully connected layer, increasing a quantity of the residual modules and modifying sizes of different convolutional kernels.
  • model parameters are initialized by Xavier, and loss weights of electromyography gesture classification and the contrastive loss are 0.8 and 0.2, respectively. Margin used in the contrastive loss is 1.
  • Step 1104 Process the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determine update parameters corresponding to respective neural networks of the gesture recognition model.
  • Step 1105 Iteratively update parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model, so as to recognize different gesture information through the gesture recognition model.
  • the model loss of the technical solution is mainly divided into two parts: a classification loss for evaluating gesture classification; and a contrastive loss for increasing separability between inner classes within an individual.
  • L emg is a loss function that constrains a feature distance of a sample pair.
  • Loss functions that measure feature distances between classes and within classes may also be generalized to other losses, for example, different feature distances (such as cosine distances), or different metric functions such as a maximum mean error, a hinge loss and a triplet loss.
  • all sEMG data and gesture labels thereof may be sent to the network for learning, and the model may be optimized through error back propagation until the loss function meets a corresponding convergence condition.
  • FIG. 15 is an exemplary schematic flowchart of a process of using the gesture recognition model according to an embodiment of this application.
  • electromyography data samples sent into the model for classification each time generally have a length ranging from 200 ms to 1 s.
  • an electromyography gesture decoding scheme of the related art only each classification result is outputted, but in an actual process, a same gesture is often predicted unstably, especially in an early stage of the recognition.
  • weighted average is performed on consecutive N model classification probabilities to obtain a final output result, so that predictive output for a same gesture is more stable during practical application.
  • end-to-end decoding classification may be performed on electromyography signals, and decoding results may be directly obtained by inputting original signals, which only needs to be driven by input data, without a need to manually design prior knowledge of feature extraction.
  • the electromyography signals are divided by the sliding window with coincidence, so that consecutive signal samples have certain signal repetition and feature coincidence, and post-processing makes prediction results have higher consistency through a voting mechanism.
  • the term “unit” or “module” refers to a computer program or part of the computer program that has a predefined function and works together with other related parts to achieve a predefined goal and may be all or partially implemented by using software, hardware (e.g., processing circuitry and/or memory configured to perform the predefined functions), or a combination thereof.
  • Each unit or module can be implemented using one or more processors (or processors and memory).
  • a processor or processors and memory
  • each module or unit can be part of an overall module that includes the functionalities of the module or unit.
  • an electromyography signal collection target object in a gesture information usage environment is determined.
  • An electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label are acquired.
  • the electromyography signal sample is divided through a sliding window having a fixed window value and a fixed stride, different electromyography signals of the target object are formed, and the different electromyography signals of the target object are denoised.
  • the denoised different electromyography signals are recognized based on the gesture information label, and probabilities of gesture information represented by the different electromyography signals are determined.
  • the probabilities of the gesture information represented by the different electromyography signals are weighted, so as to determine gesture information matching the target object.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Data Mining & Analysis (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • Biophysics (AREA)
  • Mathematical Physics (AREA)
  • Molecular Biology (AREA)
  • Human Computer Interaction (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Evolutionary Biology (AREA)
  • Medical Informatics (AREA)
  • Databases & Information Systems (AREA)
  • Dermatology (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Neurosurgery (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)
  • User Interface Of Digital Computer (AREA)
  • Position Input By Displaying (AREA)

Abstract

This application provides a gesture information processing method and apparatus, an electronic device, and a storage medium. The method includes: acquiring an electromyography signal sample generated by an electromyography signal collection target object in connection with performing multiple gestures; dividing the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object; and applying the different electromyography signals to a first neural network model to determine gesture information matching the multiple gestures performed by the target object.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application is a continuation application of U.S. patent application Ser. No. 17/580,545, entitled “GESTURE INFORMATION PROCESSING METHOD AND APPARATUS, ELECTRONIC DEVICE, AND STORAGE MEDIUM” filed on Jan. 20, 2022, which is a continuation application of PCT Patent Application No. PCT/CN2020/130567, entitled “GESTURE INFORMATION PROCESSING METHOD AND APPARATUS, ELECTRONIC DEVICE, AND STORAGE MEDIUM” filed on Nov. 20, 2020, which claims priority to Chinese Patent Application No. 202010033904.X, filed with the State Intellectual Property Office of the People's Republic of China on Jan. 13, 2020, and entitled “GESTURE INFORMATION PROCESSING METHOD AND APPARATUS, ELECTRONIC DEVICE, AND STORAGE MEDIUM”, all of which are incorporated herein by reference in their entirety.
FIELD OF THE TECHNOLOGY
This application relates to information processing technologies, and in particular, to a gesture information processing method and apparatus, an electronic device, and a storage medium.
BACKGROUND OF THE DISCLOSURE
Gestures are the most basic way of human communication, through which different actions may be realized. Electromyography signals are superposition of motor unit action potentials (MUAPs) in numerous muscle fibers in time and space. According to different recording manners, electromyography signals recorded by an electromyography may be classified into surface electromyography signals and needle electromyography signals, all of which include anatomical and physiological properties of muscles. The surface Electromyography (sEMG) signals are a combined effect of superficial muscle electromyography signals and nerve stem electrical activities on skin surfaces, which can reflect neuromuscular activities to some extent. Compared with the needle electromyography signals, the surface electromyography signals have advantages such as non-invasiveness and simple operation in measurement. Thus, the surface electromyography signals have important application values in clinical medicine, rehabilitation medicine, human-computer interfaces and the like. Conventional electromyography-based gesture recognition technologies generally use gesture recognition of visual sensors (such as ordinary cameras or depth cameras) and are based on wearable sensors (such as data gloves). However, due to the inconvenience and delay of recognition, the wearable sensors cannot timely and accurately recognize users' gestures.
SUMMARY
In view of the above, embodiments of this application provide a gesture information processing method and apparatus, an electronic device, and a storage medium, so as to accurately and timely recognize gesture information in different usage environments and have stronger robustness for automatically extracted electromyography signals, thereby improving accuracy of classification of surface electromyography signals and reducing interference of noise with gesture information recognition.
Technical solutions in the embodiments of this application are implemented as follows.
The embodiments of this application provide a gesture information processing method performed by an electronic device, the method including:
    • determining an electromyography signal collection target object in a gesture information usage environment of a gesture recognition model;
    • acquiring an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label;
    • dividing the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoising the different electromyography signals of the target object;
    • recognizing the denoised different electromyography signals based on the gesture information label, and determining probabilities of gesture information represented by the different electromyography signals; and
    • weighting the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
The embodiments of this application further provide a gesture information processing apparatus, including:
    • an information processing module configured to determine an electromyography signal collection target object in a gesture information usage environment of a gesture recognition model;
    • a first information transmission module configured to acquire an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label;
    • the information processing module being configured to divide the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoise the different electromyography signals of the target object;
    • the information processing module being configured to recognize the denoised different electromyography signals based on the gesture information label, and determine probabilities of gesture information represented by the different electromyography signals; and
    • the information processing module being configured to weight the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
The embodiments of this application further provide an electronic device, comprising:
    • a memory configured to store executable instructions; and
    • a processor configured to perform the gesture information processing method when running the executable instructions stored in the memory.
The embodiments of this application further provide a non-transitory computer-readable storage medium, storing executable instructions, the gesture information processing method being performed when the executable instructions are executed by a processor.
BRIEF DESCRIPTION OF THE DRAWINGS
To describe the technical solutions of the embodiments of this application or the related art more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments or the related art. The accompanying drawings in the following description show merely some embodiments of this application, and a person of ordinary skill in the art may still derive other drawings from these accompanying drawings without creative efforts.
FIG. 1 is a schematic diagram of an application environment of a gesture recognition model according to an embodiment of this application;
FIG. 2 is a schematic structural diagram of composition of an electronic device according to an embodiment of this application;
FIG. 3 is a schematic diagram of a process of generating gesture information in a conventional solution;
FIG. 4 is a schematic diagram of a process of generating gesture information in the conventional solution;
FIG. 5 is an exemplary schematic flowchart of a gesture information processing method according to an embodiment of this application;
FIG. 6 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application;
FIG. 7 is an exemplary schematic flowchart of a method of using the gesture recognition model according to an embodiment of this application;
FIG. 8 is an exemplary schematic structural diagram of a first neural network model according to an embodiment of this application;
FIG. 9 is an exemplary schematic structural diagram of the first neural network model according to an embodiment of this application;
FIG. 10 is a schematic diagram of an application environment of the gesture recognition model according to an embodiment of this application;
FIG. 11 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application;
FIG. 12 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application;
FIG. 13 is an exemplary schematic diagram of sample acquisition of the gesture recognition model according to an embodiment of this application;
FIG. 14 is an exemplary schematic structural diagram of the gesture recognition model according to an embodiment of this application; and
FIG. 15 is an exemplary schematic flowchart of a process of using the gesture recognition model according to an embodiment of this application.
 DESCRIPTION OF EMBODIMENTS
To make the objectives, technical solutions, and advantages of this application clearer, the following describes this application in further detail with reference to the accompanying drawings. The described embodiments are not to be considered as a limitation to this application. All other embodiments obtained by a person of ordinary skill in the art without creative efforts shall fall within the protection scope of this application.
In the following descriptions, related “some embodiments” describe a subset of all possible embodiments. However, it may be understood that the “some embodiments” may be the same subset or different subsets of all the possible embodiments, and may be combined with each other without conflict.
Before the embodiments of this application are further described in detail, a description is made on nouns and terms involved in the embodiments of this application, and the nouns and terms involved in the embodiments of this application are applicable to the following explanations.
1) An “artificial neural network”, Neural Network (NN) for short, is a mathematical model or computational model that imitates structures and functions of biological neural networks in fields of machine learning and cognitive science, for estimating or approximating functions.
2) A “model parameter” is a quantity to use a generic variable to establish a relationship between functions and variables. In the artificial neural network, the model parameter is generally a real matrix.
3) An “sEMG” refers to a muscle biograph obtained by measuring and recording bioelectricity of surface muscles through a special electromyography or polygraph.
4) A “residual network (ResNet)” is a deep feedforward neural network that introduces skip connections to achieve identity mapping.
5) A “contrastive loss” may learn a mapping relation, which enables points of a same category but far away in a high-dimensional space to become closer after being mapped to a low-dimensional space by the contrastive loss, and points of different categories but close to each other to become further in the low-dimensional space after being mapped. As a result, in low-dimensional space, points of a same kind may produce a clustering effect, and mean of different kinds may be separated. It is similar to fisher dimension reduction, but fisher dimension reduction does not have an effect of out-of-sample extension and cannot affect a new sample.
6) “Model training” means performing multi-classification learning on image data sets. The model may be constructed by deep learning frameworks such as TensorFlow and torch, and a multi-classification model is formed using a multi-layer combination of neural network layers such as a convolutional neural network (CNN). Input to the model is a three-channel or original-channel matrix formed after an image is read by a tool such as openCV. Output from the model is multi-classification probability. A web page category is finally outputted through an algorithm such as softmax. During training, the model approximates to a correct trend through an objective function such as cross entropy.
7) “In response to” is used for representing a condition or status on which one or more operations to be performed depend. When the condition or status is satisfied, the one or more operations may be performed immediately or after a set delay. Unless explicitly stated, there is no limitation on the order in which the plurality of operations are performed.
8) A “client” is a carrier to achieve a specific function in a terminal. For example, a mobile client (APP) is a carrier of a specific function in a mobile terminal, such as a program to perform a user gesture recognition function.
9) Soft max is a normalized exponential function that is a promotion of a logic function. The function can “compress” a K-dimension vector including any real number into another K-dimension real vector, so that each element ranges within (0, 1), and a sum of all elements is 1.
FIG. 1 is a schematic diagram of an application environment of a gesture recognition model according to an embodiment of this application. As shown in FIG. 1 , a terminal (including a terminal 10-1 and a terminal 10-2) is provided with a client for gesture recognition, which may acquire electromyography samples of different users. When a human performs different gestures, a muscle group may produce different electromyography signals under the coordination of a nervous system. An sEMG records bioelectrical signals on skin surfaces of human muscles and effectively captures information of muscle activities. Intentions of human gestures may be decoded through gesture analysis on the surface electromyograph. The terminal is connected to a server 200 by using a network 300. The network 300 may be a wide area network, a local area network, or a combination of the wide area network and the local area network, and achieves data transmission by using a radio link. Certainly, the terminal 10-1 and the terminal 10-2 may also function as powered skeletons or somatosensory game machines with a gesture recognition model run therein. Surface electromyography signals are transmitted, decoded and controlled by embedding the gesture recognition model into different hardware systems or software systems. For example, 1) medical systems combined with powered exoskeletons may be configured for active rehabilitation of arm motor functions in patients with hemiplegia and spinal cord injuries. 2) Monitoring systems combined with physical training may evaluate muscle fatigue, balance and use efficiency in real time to improve training efficiency and reduce training injuries. 3) Gesture interaction systems combined with games enable humans to control activities of objects in the virtual world through gesture movement.
Gesture information obtained by the terminal (including the terminal 10-1 and the terminal 10-2) from a corresponding server 200 via a network 300 is of different types. For example, the terminal (including the terminal 10-1 and the terminal 10-2) may either obtain any type of gesture information from the corresponding server 200 via the network 300 or obtain gesture information only matching a corresponding control instruction from the corresponding server 200 through the network 300 for execution. In some embodiments of this application, different types of gesture information stored in the server 200 may be written in software code environments of different programming languages, and code objects may be different types of code entities. For example, in C language software code, a code object may be a function. In JAVA language software code, a code object may be a class, which may be a piece of object code in IOS OC language. In C++ language software code, a code object may be a class or a function to perform electromyography signals from different terminals. Sources of gesture information to be processed by the gesture recognition model are not distinguished from one another in this application.
The server 200 needs to determine and recognize electromyography signals generated during gesture triggering when the server 200 sends different types of gesture information to the terminal (including the terminal 10-1 and the terminal 10-2) via the network 300. As an instance, the server 200 is configured to obtain different electromyography signals of a same target object and denoise the different electromyography signals; recognize the denoised different electromyography signals based on the gesture information label through the gesture recognition model, and determine probabilities of gesture information represented by the different electromyography signals; and weight the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
Certainly, before the gesture recognition model run in the server 200 is used, the corresponding gesture recognition model further needs to be trained, so as to deploy the trained gesture recognition model in a corresponding server. Specifically, a process of training the gesture recognition model includes: obtaining a first training sample set, where the first training sample set includes at least one group of electromyography signal samples and the corresponding gesture information label; denoising the first training sample set, so as to form a corresponding second training sample set; determining initial parameters of the first neural network model in the gesture recognition model; processing the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determining update parameters corresponding to different neural networks of the gesture recognition model; and iteratively updating parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model, so as to recognize different gesture information through the gesture recognition model.
A structure of an electronic device according to an embodiment of this application is described in detail below. The electronic device may be implemented in a variety of forms, for example, a dedicated terminal with a gesture information recognition function, or an electronic device with a gesture information recognition function, such as the server 200 in FIG. 1 . FIG. 2 is a schematic structural diagram of composition of an electronic device according to an embodiment of this application. It may be understood that, FIG. 2 shows only an exemplary structure rather than a complete structure of the electronic device. The structure shown in FIG. 2 may be partially or entirely implemented based on requirements.
The electronic device provided in this embodiment of this application includes: at least one processor 201, a memory 202, a user interface 203, and at least one network interface 204. The components in the electronic device are coupled by using a bus system 205. It may be understood that the bus system 205 is configured to implement connection and communication between the components. In addition to a data bus, the bus system 205 further includes a power bus, a control bus, and a state signal bus. However, for ease of clear description, all types of buses are labeled as the bus system 205 in FIG. 2 .
The user interface 203 may include a display, a keyboard, a mouse, a track ball, a click wheel, a key, a button, a touch panel, a touchscreen, or the like.
It may be understood that, the memory 202 may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The memory 202 in the embodiment of this application can store data to support operation of the terminal (for example, 10-1). An example of the data includes any computer program to be operated on the terminal (for example, 10-1), for example, an operating system and an application program. The operating system includes various system programs, such as framework layers, kernel library layers, and driver layers used for implementing various basic services and processing hardware-based tasks. The application program may include various application programs.
In some embodiments, a gesture information recognition apparatus provided in the embodiments of this application may be implemented in the form of a combination of software and hardware. In an example, the gesture information recognition apparatus provided in the embodiments of this application may be a processor in the form of a hardware decoding processor, and is programmed to execute the gesture information recognition method provided in the embodiments of this application. For example, the processor in the form of a hardware decoding processor may use one or more ASICs, digital signal processor (DSP), programmable logic devices (PLDs), complex programmable logic device (CPLD), FPGAs, or other electronic elements.
In an example in which the gesture information recognition apparatus provided in the embodiments of this application is implemented by a combination of software and hardware, the gesture information recognition provided in the embodiments of this application may be directly embodied as a combination of software modules executed by the processor 201. The software modules may be located in a storage medium, and the storage medium is located in the memory 202. The processor 201 reads executable instructions included in the software modules in the memory 202 and uses necessary hardware (for example, including the processor 201 and other components connected to the bus 205) in combination, to complete the gesture information recognition method provided in the embodiments of this application.
For example, the processor 201 may be an integrated circuit chip, and has a signal processing capability, for example, a general purpose processor, a digital signal processor (DSP), or another programmable logical device, a discrete gate or a transistor logical device, or a discrete hardware component. The general purpose processor may be a microprocessor, any conventional processor, or the like.
In an example in which the gesture information recognition apparatus provided in the embodiments of this application is implemented by hardware, the gesture information recognition apparatus provided in the embodiments of this application may be directly executed by using the processor 201 in the form of a hardware decoding processor, for example, one or more ASICs, DSPs, PLDs, CPLDs, FPGAs, or other electronic elements, to execute the gesture information recognition method provided in the embodiments of this application.
The memory 202 in this embodiment of this application is configured to store various types of data to support operation of the electronic device. An example of the data includes: any executable instruction configured to be operated on the electronic device, such as an executable instruction, and a program that implements the gesture information recognition method of the embodiments of this application may be included in the executable instruction.
In some other embodiments, the gesture information recognition apparatus according to the embodiment of this application may be implemented in the form of software. FIG. 2 shows a gesture information recognition apparatus 2020 stored in the memory 202, which may be software in the form of a program, a plug-in, or the like, and include a series of modules. An example of the program stored in the memory 202 may include the gesture information recognition apparatus 2020. The gesture information recognition 2020 includes the following software modules: a first information transmission module 2081 and an information processing module 2082. When the software modules in the gesture information recognition apparatus 2020 are read into the RAM by the processor 201 and executed, the gesture information recognition method according to the embodiment of this application is performed. Functions of the software modules in the gesture information recognition apparatus 2020 are introduced below.
The information processing module 2082 is configured to determine an electromyography signal collection target object in a gesture information usage environment.
The first information transmission module 2081 is configured to acquire an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label.
The information processing module 2082 is configured to divide the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoise the different electromyography signals of the target object.
The information processing module 2082 is configured to recognize the denoised different electromyography signals based on the gesture information label, and determine probabilities of gesture information represented by the different electromyography signals.
The information processing module 2082 is configured to weight the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
The gesture information processing method according to the embodiment of this application is described in combination with the gesture information recognition apparatus 2020 shown in FIG. 2 . Before the gesture information processing method according to the embodiment of this application is introduced, a process of generating, by a gesture recognition model, corresponding gesture information based on electromyography signals in this application is introduced first. FIG. 3 is a schematic diagram of a process of generating gesture information in a conventional solution. In the related art, based on sEMG classification of PSD+Shallow ConvNet, firstly, power spectral density (PSD) of original surface electromyography signals may be calculated to obtain a feature map; then, the feature map is used as network input, and temporal and spatial features are extracted from a PSD map by using a convolution module (a convolutional layer, a Batch Normalization layer, and a maximum pooling layer); finally, a corresponding classification probability is obtained through full connection and SoftMax. Alternatively, refer to FIG. 4 which is a schematic diagram of a process of generating gesture information in the conventional solution. Based on sEMG classification of fast Fourier transform (FFT)+principal component analysis (PCA)+CNN, first 25 principal components of a spectrogram of original signals are obtained by PCA based on the spectrogram, the spectrogram after dimension reduction is used as input to the CNN, and features therein are extracted using a convolutional layer. Finally, a corresponding classification probability is obtained through full connection and SoftMax. However, in the related art, an sEMG classification algorithm based on PSD+Shallow ConvNet requires manual extraction of feature information, so features that can be learned by the model are limited and not enough to fit complex sEMG signals. A classification algorithm based on FFT+PCA+CNN requires pre-extraction of FFT features and reduction of data dimensions by principal component analysis, which not only has a plurality of uncertain hyperparameters, but also results in inconsistent objectives of feature extraction and classification model optimization, and end-to-end signal decoding cannot be realized. At the same time, the relevant algorithms do not take continuity before and after gesture prediction into account, which may also lead to poor stability of real-time gesture prediction, affect recognition of continuous user gestures, and affect user experience.
In order to solve the above problem, referring to FIG. 5 , the gesture information processing method according to the embodiment of this application is described with reference to the electronic device shown in FIG. 2 . FIG. 5 is an exemplary schematic flowchart of a gesture information processing method according to an embodiment of this application. It may be understood that, steps shown in FIG. 5 may be performed by various electronic devices running the gesture information processing apparatus, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 5 .
Step 501: A gesture information processing apparatus obtains a first training sample set.
The first training sample set includes at least one group of electromyography signal samples and the corresponding gesture information label.
Step 502: The gesture information processing apparatus denoises the first training sample set, so as to form a corresponding second training sample set.
In some embodiments of this application, the denoising the first training sample set, so as to form a corresponding second training sample set may be performed in following manners:
    • determining a dynamic noise threshold matching a usage environment of the gesture recognition model; and denoising the first training sample set according to the dynamic noise threshold, so as to form a second training sample set matching the dynamic noise threshold. Due to different usage environments of the gesture recognition model, the dynamic noise threshold matching the gesture recognition model also varies. For example, in a usage environment of a robotic arm in somatosensory game, a dynamic noise threshold matching gesture information to be recognized and executed by the terminal and the usage environment of the gesture recognition model is required to be greater than a dynamic noise threshold of a precision manipulator.
In some embodiments of this application, the denoising the first training sample set, so as to form a corresponding second training sample set may be performed in following manners:
    • determining a fixed noise threshold corresponding to the gesture recognition model, and denoising the first training sample set according to the fixed noise threshold, so as to form a second training sample set matching the fixed noise threshold. When the gesture recognition model is solidified in a corresponding hardware mechanism, such as a prosthetic system or a posture monitoring system, and the usage environment is large-gesture recognition and triggering, since noise is relatively single, a training speed of the gesture recognition model can be effectively increased, the user's waiting time can be shortened, and the user experience can be improved through the fixed noise threshold corresponding to the fixed gesture recognition model.
In some embodiments of this application, the gesture information processing method further includes:
    • determining a corresponding training sample collection object according to the usage environment of the gesture recognition model; acquiring an electromyography signal sample matching the training sample collection object, and a corresponding gesture information label; and dividing the electromyography signal sample through the sliding window having the fixed window value and the fixed stride to form the first training sample set. The corresponding training sample collection object is determined, and the electromyography signal sample matching the training sample collection object, and the corresponding gesture information label are acquired, so that the gesture recognition model to be used by a target in a specific field can be trained, so as to improve accuracy of the trained gesture recognition model.
Step 503: The gesture information processing apparatus determines initial parameters of the first neural network model in the gesture recognition model.
In some embodiments of this application, the determining initial parameters of the first neural network model in the gesture recognition model may be performed in following manners:
    • determining the output sizes of the different networks in the first neural network model of the gesture recognition model and the quantity of the corresponding convolution kernels according to features of electromyography signals in the first training sample set; and adjusting the quantities of the temporal convolutional layer network, the spatial convolutional layer network, the fully connected layer network and the residual module network of the first neural network model according to an application environment of the gesture recognition model, so as to match initialization parameters of the gesture recognition model with the application environment.
In some embodiments of this application, the gesture information processing method further includes:
    • determining a classification loss and a contrastive loss corresponding to the first neural network model in the gesture recognition model; and determining a fusion loss matching the gesture recognition model according to the classification loss and the contrastive loss, where the fusion loss is configured to stop training the gesture recognition model when a corresponding convergence condition is reached.
In some embodiments of this application, the first neural network model in the gesture recognition model may be a trained neural network, and the first neural network model in the gesture recognition model may be a large-scale neural network. For example, a quantity of network parameters of the first neural network model in the gesture recognition model is greater than a value, which is not limited in the embodiment of this application. Optionally, the first neural network model in the gesture recognition model may be a residual network (ResNet), a CNN, a deep neural network (DNN), a recurrent neural network (RNN) or the like. The type of the first neural network model in the gesture recognition model is not limited in the embodiment of this application. The first neural network model in the gesture recognition model may be a neural network suitable for different computer vision tasks, such as a target recognition task, a target classification task, a target detection task or an attitude estimation task. The first neural network model in the gesture recognition model may also be a neural network suitable for different application scenarios, such as a security monitoring scenario, a gesture unlocking scenario or an intelligent driving scenario. An application scope of the first neural network model in the gesture recognition model is not limited in the embodiment of this application. Optionally, A network structure of the first neural network model in the gesture recognition model may be designed according to computer vision tasks or adopt at least part of a related network structure, such as a DNN or a visual geometrygroup network (VGGNet). The network structure of the first neural network model in the gesture recognition model is not limited in the embodiment of this application.
In some embodiments of this application, since, in the related art, only softmax is used as a loss function of the model, but differences between signals of a same kind within individuals are not taken into account, in this application, a contrastive loss is applied to reduce the loss of the differences between the signals of the same kind and improve robustness of the algorithm. A second neural network model and a third neural network model may also be introduced to determine a classification loss function for evaluating gesture information classification and a contrastive loss of separability between inner classes of corresponding individuals respectively.
Step 504: The gesture information processing apparatus processes the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determines update parameters corresponding to different neural networks of the gesture recognition model.
Step 505: The gesture information processing apparatus iteratively updates parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model.
Thus, different gesture information can be recognized through the gesture recognition model.
Still referring to FIG. 6 , the gesture information processing method according to the embodiment of this application is described with reference to the electronic device shown in FIG. 2 . FIG. 6 is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application. It may be understood that, steps shown in FIG. 6 may be performed by various electronic devices running the gesture information processing apparatus, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 6 .
Step 601: Keep the initial parameters of the gesture recognition model unchanged, and determine output results of the gesture recognition model.
Step 602: Substitute, in response to the output results of the gesture recognition model, different training samples in the second training sample set into the fusion loss matching the gesture recognition model.
Step 603: Determine parameters corresponding to different networks in the gesture recognition model when the fusion loss meets the convergence condition as update parameters of the gesture recognition model.
Step 604: Iteratively update parameters of the different neural networks of the gesture recognition model respectively based on the second training sample set and according to the update parameters corresponding to the different neural networks of the gesture recognition model, until losses corresponding to the different neural networks of the gesture recognition model meets the corresponding convergence condition.
Still referring to FIG. 7 , the gesture information processing method according to the embodiment of this application is described with reference to the electronic device shown in FIG. 2 . FIG. 7 is an exemplary schematic flowchart of a method of using the gesture recognition model according to an embodiment of this application. It may be understood that, steps shown in FIG. 7 may be performed by various electronic devices running the gesture recognition model, such as a dedicated terminal, an electronic device, or an electronic device cluster with a gesture information recognition function. The following describes the steps shown in FIG. 7 .
Step 701: A server determines an electromyography signal collection target object in a gesture information usage environment of a gesture recognition model.
Step 702: The server acquires an electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label.
Step 703: The server divides the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object, and denoises the different electromyography signals of the target object.
Step 704: The server recognizes the denoised different electromyography signals based on the gesture information label, and determines probabilities of gesture information represented by the different electromyography signals.
Step 705: The server weights the probabilities of the gesture information represented by the different electromyography signals, so as to determine gesture information matching the target object.
In some embodiments of this application, the different electromyography signals of the target object may be obtained in following manners:
    • determining a corresponding electromyography signal collection object according to the usage environment of the gesture recognition model; acquiring an electromyography signal sample matching the electromyography signal collection object, and a corresponding gesture information label; and dividing the electromyography signal sample through the sliding window having the fixed window value and the fixed stride, to form different electromyography signals of the target object.
In some embodiments of this application, the denoising the different electromyography signals may be performed in following manners:
    • determining a dynamic noise threshold matching a usage environment of the gesture recognition model; and denoising the different electromyography signals according to the dynamic noise threshold, so as to form a dynamic noise electromyography signal matching the dynamic noise threshold. Certainly, a usage environment of fixed noise may also be realized in a following manner: determining a fixed noise threshold corresponding to the gesture recognition model, and denoising the different electromyography signals according to the fixed noise threshold, so as to form a fixed noise electromyography signal matching the fixed noise threshold. Due to different usage environments of the gesture recognition model, the dynamic noise threshold matching the gesture recognition model also varies. For example, in a usage environment of a robotic arm in somatosensory game, a dynamic noise threshold matching gesture information to be recognized and executed by the terminal and the usage environment of the gesture recognition model is required to be greater than a dynamic noise threshold of a precision manipulator.
In some embodiments of this application, features of corresponding electromyography signals are determined when a usage environment of the gesture recognition model is a powered skeleton. The output sizes of the different networks in the first neural network model of the gesture recognition model and the quantity of the corresponding convolution kernels are determined according to features of electromyography signals. The quantities of the temporal convolutional layer network, the spatial convolutional layer network, the fully connected layer network and the residual module network of the first neural network model are determined according to features of the powered skeleton, so as to match initialization parameters of the gesture recognition model with the features of the powered skeleton. Thus, the training time of the model may be shortened, and the trained model may be provided for users to directly call through an applet.
Thus, while real-time performance of recognition is ensured, weighted average is performed on consecutive N model classification probabilities to obtain a final output result, so that predictive output for a same gesture is more stable during practical application, improving the robustness of the gesture recognition model.
Still refer to FIG. 8 which is an exemplary schematic structural diagram of a first neural network model according to an embodiment of this application. The first neural network model is a full convolutional residual neural network, which successively includes four convolutional layers: convolutional layers 1 to 4. 17 serial residual blocks and an addition layer are included between the convolutional layer 1 and the convolutional layer 2, and output of the residual block 17 and output of the convolutional layer 1 serve as input to the addition layer.
Preliminary feature extraction is performed on an input image by using one convolutional layer 1, to obtain preliminarily extracted features. Feature extraction is continuously performed on the preliminarily extracted features by using the series residual blocks 1 to 17 and features and texture are supplemented to output of a previous residual block. The output of the residual block 17 is supplemented to the preliminarily extracted features through the addition layer. Resolution of an image outputted from the addition layer is amplified to super-resolution of a specified multiple by using series convolutional layers 2 and 3—pixel realignment layer—convolutional layer 4, to obtain a super-resolution image.
FIG. 9 is an exemplary schematic structural diagram of the first neural network model according to an embodiment of this application. A network structure of a residual block i is shown in FIG. 9 , including a convolutional layer i1, a convolutional layer i2 and an addition layer i1 connected in series. 0<i<1. The addition layer i1 adds input of residual block i1 to a corresponding element outputted from the convolutional layer i2, which is inputted to the following fully connected layer and processed by the fully connected layer and a normalized exponential function layer connected to the fully connected layer, to determine a probability that the electromyography signal belongs to the corresponding gesture information.
The gesture recognition model according to the embodiment of this application is described below by taking a powered exoskeleton as an example. FIG. 10 is a schematic diagram of an application environment of the gesture recognition model according to an embodiment of this application. As shown in FIG. 10 , a terminal (including a powered exoskeleton 100-1 and a terminal 100-2) is provided with a client for gesture recognition, which may acquire electromyography samples of different users. When a human performs different gestures, a muscle group may produce different electromyography signals under the coordination of a nervous system. An sEMG records bioelectrical signals on skin surfaces of human muscles and effectively captures information of muscle activities. Intentions of human gestures may be decoded through gesture analysis on the surface electromyograph. The terminal is connected to a server 200 via a network 300. The network 300 may be a wide area network, a local area network, or a combination thereof, and transmit data by using wireless links. Certainly, the terminal 10-1 and the terminal 10-2 may also function as powered skeletons or somatosensory game machines with a gesture recognition model run therein. Surface electromyography signals are transmitted, decoded and controlled by embedding the gesture recognition model into different hardware systems or software systems. For example, 1) medical systems combined with powered exoskeletons may be configured for active rehabilitation of arm motor functions in patients with hemiplegia and spinal cord injuries. 2) Monitoring systems combined with user posture monitoring may evaluate, in real time, fatigue, balance and use efficiency of muscles to which the powered exoskeleton is attached, improving training efficiency and reducing injuries of a target user.
Refer to FIG. 11 which is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application, specifically including the following steps:
Step 1101: Obtain a first training sample set, where the first training sample set includes at least one group of electromyography samples.
Step 1102: Denoise the first training sample set, so as to form a corresponding second training sample set.
Refer to FIG. 12 which is an exemplary schematic flowchart of the gesture information processing method according to an embodiment of this application. Original multi-channel surface electromyography signals are preprocessed to obtain training samples, the training samples are then inputted into a deep residual network to extract features which mainly include time domain features and space domain features related to electromyography gestures, probabilities of different gestures are obtained and classified, and finally, network parameters are continuously optimized by iterative training.
Refer to FIG. 13 which is an exemplary schematic diagram of sample acquisition of the gesture recognition model according to an embodiment of this application. Cortical electromyography data of a target user may be obtained by a corresponding acquisition device. The data set includes electromyography data acquired by 10 subjects (e.g., 5 males and 5 females) in two time periods. The electromyography data in the first time period is used as a training sample data set, and the electromyography data in the second time period is used as a test sample data set.
In some embodiments of this application, a data acquisition process is as follows:
1) A subject sit in a chair and remains relaxed, wears an acquisition apparatus (e.g., an MYO electromyography bracelet), and lets arms hang down naturally. An experimenter informs the subject of the data acquisition process and specifications.
2) During data acquisition, the subject is required to perform eight different gestures as prompted, including “relax”, “clench”, “open hand”, “swing left”, “swing right”, “give a like”, “give a peace sign” and “index finger up” (as shown below). Each action lasts for 2 s, and a corresponding gesture label is recorded.
In some embodiments of this application, surface electromyography signals may be divided by a sliding window as a single sample to be sent to the network. A time window is 200 ms, and a window sliding stride is 40 ms. Since a signal sampling frequency is 200 Hz and 8 electrode channels are included, each signal sample has a dimension of 8×40.
Step 1103: Determine initial parameters of the first neural network model in the gesture recognition model.
Refer to FIG. 14 which is an exemplary schematic structural diagram of the gesture recognition model according to an embodiment of this application. The first neural network model may be a ResNet. In the first neural network model, according to temporal and spatial properties, an inputted sEMG signal is processed by a spatial convolutional layer for the spatial properties and a temporal convolutional layer for the temporal properties. Model parameters of the first neural network model are shown in Table 1. The signal inputted to the network has a dimension of 8×40. First two temporal-spatial convolutional layers Spatial Cony and Temporal Cony perform convolutional operations on the signal in a spatial (channel) dimension and a temporal dimension respectively, with strides of 1, to realize spatial and temporal filtering of the sEMG signal. After temporal-spatial convolution, one-dimensional time features are outputted and are subsequently feature-extracted and fused by 6 residual modules. Each residual module includes two residual networks. Each residual network is formed by a convolutional layer with a 1*3 convolutional kernel and a batch normalization layer. Finally, the features are further fused through the fully connected layer, and classification results of electromyography gestures are outputted by SoftMax.
In practical application, the network structure of the solution may be realized by appropriately increasing a quantity of the temporal and spatial convolutional layers, increasing a number of the fully connected layer, increasing a quantity of the residual modules and modifying sizes of different convolutional kernels.
Further, during initialization, the model parameters are initialized by Xavier, and loss weights of electromyography gesture classification and the contrastive loss are 0.8 and 0.2, respectively. Margin used in the contrastive loss is 1.
TABLE 1
Layer name Output size Net
Input 8 × 40 —
Spatial Conv 1 × 40 8 × 1, 128, stride 1 × 1
Temporal Conv 1 × 18 1 × 5, 256, stride 1 × 2
ResBlock1 1 × 18 1 × 3, 128, stride 1 × 1
ResBlock2 1 × 18 1 × 3, 64, stride 1 × 1
ResBlock3 1 × 9 1 × 3, 64, stride 1 × 2
ResBlock4 1 × 9 1 × 3, 64, stride 1 × 1
ResBlock5 1 × 9 1 × 3, 64, stride 1 × 1
ResBlock6 1 × 9 1 × 3, 64, stride 1 × 1
Flatten 576 —
Fully connected  5 5
Output  5 5
Step 1104: Process the second training sample set through the gesture recognition model based on the initial parameters of the first neural network model, and determine update parameters corresponding to respective neural networks of the gesture recognition model.
Step 1105: Iteratively update parameters of the first neural network model of the gesture recognition model through the second training sample set according to the update parameters corresponding to the different neural networks of the gesture recognition model, so as to recognize different gesture information through the gesture recognition model.
In the related art, only a softmax loss is used as a loss function of the model, but differences between signals of a same kind within individuals are not taken into account. Therefore, in this application, a contrastive loss is applied to reduce the loss of the differences between the signals of the same kind and improve robustness of the algorithm.
Based on the above, the model loss of the technical solution is mainly divided into two parts: a classification loss for evaluating gesture classification; and a contrastive loss for increasing separability between inner classes within an individual. A loss function of the whole network model is:
L=αL emg +βL cont
where Lemg is softmax loss=−Σi cti log (si).
Lemg is a loss function that constrains a feature distance of a sample pair. By minimizing the loss function, feature distribution between sample pairs of a same kind may be as close as possible, while feature distances of sample pairs of different kinds exceed a certain margin. According to the above rule, a mathematical expression thereof is:
L cont = 1 2 ⁢ N ⁢ ∑ n - 1 N yd 2 + ( 1 - y ) ⁢ max ⁡ ( margin - d , 0 ) 2
where d denotes a Euclidean distance between two samples, and N denotes a total number of sample pairs. Loss functions that measure feature distances between classes and within classes may also be generalized to other losses, for example, different feature distances (such as cosine distances), or different metric functions such as a maximum mean error, a hinge loss and a triplet loss.
During the training, all sEMG data and gesture labels thereof may be sent to the network for learning, and the model may be optimized through error back propagation until the loss function meets a corresponding convergence condition.
Refer to FIG. 15 which is an exemplary schematic flowchart of a process of using the gesture recognition model according to an embodiment of this application. In order to ensure real-time performance of electromyography recognition, electromyography data samples sent into the model for classification each time generally have a length ranging from 200 ms to 1 s. In an electromyography gesture decoding scheme of the related art, only each classification result is outputted, but in an actual process, a same gesture is often predicted unstably, especially in an early stage of the recognition. Thus, as shown in FIG. 15 , while real-time performance of recognition is ensured, weighted average is performed on consecutive N model classification probabilities to obtain a final output result, so that predictive output for a same gesture is more stable during practical application.
Compared with recognition of gesture information in the conventional art, through the technical solutions according to this application, end-to-end decoding classification may be performed on electromyography signals, and decoding results may be directly obtained by inputting original signals, which only needs to be driven by input data, without a need to manually design prior knowledge of feature extraction.
The electromyography signals are divided by the sliding window with coincidence, so that consecutive signal samples have certain signal repetition and feature coincidence, and post-processing makes prediction results have higher consistency through a voting mechanism.
In this application, the term “unit” or “module” refers to a computer program or part of the computer program that has a predefined function and works together with other related parts to achieve a predefined goal and may be all or partially implemented by using software, hardware (e.g., processing circuitry and/or memory configured to perform the predefined functions), or a combination thereof. Each unit or module can be implemented using one or more processors (or processors and memory). Likewise, a processor (or processors and memory) can be used to implement one or more modules or units. Moreover, each module or unit can be part of an overall module that includes the functionalities of the module or unit. The foregoing descriptions are merely exemplary embodiments of this application, but are not intended to limit this application. Any modification, equivalent replacement and improvement made within the spirit and principle of this application shall fall within the protection scope of this application.
Industrial Practicability
In the embodiments of this application, an electromyography signal collection target object in a gesture information usage environment is determined. An electromyography signal sample matching the electromyography signal collection target object, and a corresponding gesture information label are acquired. The electromyography signal sample is divided through a sliding window having a fixed window value and a fixed stride, different electromyography signals of the target object are formed, and the different electromyography signals of the target object are denoised. The denoised different electromyography signals are recognized based on the gesture information label, and probabilities of gesture information represented by the different electromyography signals are determined. The probabilities of the gesture information represented by the different electromyography signals are weighted, so as to determine gesture information matching the target object.

Claims (20)

What is claimed is:
1. A gesture information processing method performed by an electronic device, the method comprising:
acquiring an electromyography signal sample generated by an electromyography signal collection target object in connection with performing one of multiple gestures and a corresponding gesture information label;
dividing the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object; and
applying the different electromyography signals to a first neural network model to determine gesture information matching the one of multiple gestures performed by the target object, further including:
recognizing the different electromyography signals based on the gesture information label;
determining probabilities of gesture information represented by the different electromyography signals using the first neural network model; and
weighting the probabilities of the gesture information represented by the different electromyography signals to determine the gesture information matching the one of multiple gestures performed by the target object.
2. The method of claim 1, wherein method further comprises:
determining a dynamic noise threshold associated with the multiple gestures performed by the target object; and
denoising the different electromyography signals according to the dynamic noise threshold to form dynamic noise electromyography signals matching the dynamic noise threshold.
3. The method of claim 1, wherein the method further comprises:
determining features of corresponding electromyography signals when the gesture information usage environment of the gesture recognition model is a powered skeleton;
determining output sizes of different networks in a first neural network model of the gesture recognition model and a quantity of corresponding convolution kernels according to the features of the electromyography signals; and
adjusting quantities of a temporal convolutional layer network, a spatial convolutional layer network, a fully connected layer network and a residual module network of the first neural network model according to features of the powered skeleton, so as to match initialization parameters of the gesture recognition model with the features of the powered skeleton.
4. The method of claim 1, wherein the first neural network model is trained by:
obtaining a first training sample set, wherein the first training sample set comprises at least one group of electromyography signal samples and a corresponding set of gesture information labels;
determining initial parameters of the first neural network model in the gesture recognition model;
iteratively updating the initial parameters of the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model until after a loss function associated with the first neural network model meets a corresponding convergence condition.
5. The method of claim 4, wherein the first training sample set is generated by:
acquiring an electromyography signal sample matching a training sample collection object and a corresponding set of gesture information labels; and
dividing the electromyography signal sample through the sliding window having the fixed window value and the fixed stride to form the first training sample set.
6. The method of claim 4, wherein the determining initial parameters of the first neural network model comprises:
determining the output sizes of different networks in the first neural network model and the quantity of the corresponding convolution kernels according to features of electromyography signals in the first training sample set; and
adjusting the quantities of the different networks in the first neural network model including a temporal convolutional layer network, a spatial convolutional layer network, a fully connected layer network and a residual module network according to an application environment of the gesture recognition model, so as to match initialization parameters of the gesture recognition model with the application environment.
7. The method of claim 4, wherein the iteratively updating the initial parameters of the first neural network model further comprises:
determining a classification loss and a contrastive loss corresponding to the first neural network model;
determining a fusion loss matching the first neural network model according to the classification loss and the contrastive loss as the loss function associated with the first neural network model; and
training the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model iteratively until after the fusion loss indicates that the corresponding convergence condition is reached.
8. An electronic device, comprising:
a memory configured to store executable instructions; and
a processor configured to run the executable instructions stored in the memory to perform a gesture information processing method including:
acquiring an electromyography signal sample generated by an electromyography signal collection target object in connection with performing one of multiple gestures and a corresponding gesture information label;
dividing the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object; and
applying the different electromyography signals to a first neural network model to determine gesture information matching the one of multiple gestures performed by the target object, further including:
recognizing the different electromyography signals based on the gesture information label;
determining probabilities of gesture information represented by the different electromyography signals using the first neural network model; and
weighting the probabilities of the gesture information represented by the different electromyography signals to determine the gesture information matching the one of multiple gestures performed by the target object.
9. The electronic device of claim 8, wherein method further comprises:
determining a dynamic noise threshold associated with the multiple gestures performed by the target object; and
denoising the different electromyography signals according to the dynamic noise threshold to form dynamic noise electromyography signals matching the dynamic noise threshold.
10. The electronic device of claim 8, wherein the method further comprises:
determining features of corresponding electromyography signals when the gesture information usage environment of the gesture recognition model is a powered skeleton;
determining output sizes of different networks in a first neural network model of the gesture recognition model and a quantity of corresponding convolution kernels according to the features of the electromyography signals; and
adjusting quantities of a temporal convolutional layer network, a spatial convolutional layer network, a fully connected layer network and a residual module network of the first neural network model according to features of the powered skeleton, so as to match initialization parameters of the gesture recognition model with the features of the powered skeleton.
11. The electronic device of claim 8, wherein the first neural network model is trained by:
obtaining a first training sample set, wherein the first training sample set comprises at least one group of electromyography signal samples and a corresponding set of gesture information labels;
determining initial parameters of the first neural network model in the gesture recognition model;
iteratively updating the initial parameters of the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model until after a loss function associated with the first neural network model meets a corresponding convergence condition.
12. The electronic device of claim 11, wherein the first training sample set is generated by:
acquiring an electromyography signal sample matching a training sample collection object and a corresponding set of gesture information labels; and
dividing the electromyography signal sample through the sliding window having the fixed window value and the fixed stride to form the first training sample set.
13. The electronic device of claim 11, wherein the determining initial parameters of the first neural network model comprises:
determining the output sizes of different networks in the first neural network model and the quantity of the corresponding convolution kernels according to features of electromyography signals in the first training sample set; and
adjusting the quantities of the different networks in the first neural network model including a temporal convolutional layer network, a spatial convolutional layer network, a fully connected layer network and a residual module network according to an application environment of the gesture recognition model, so as to match initialization parameters of the gesture recognition model with the application environment.
14. The electronic device of claim 11, wherein the iteratively updating the initial parameters of the first neural network model further comprises:
determining a classification loss and a contrastive loss corresponding to the first neural network model;
determining a fusion loss matching the first neural network model according to the classification loss and the contrastive loss as the loss function associated with the first neural network model; and
training the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model iteratively until after the fusion loss indicates that the corresponding convergence condition is reached.
15. A non-transitory computer-readable storage medium, storing executable instructions that, when executed by a processor of an electronic device, cause the electronic device to perform a gesture information processing method including:
acquiring an electromyography signal sample generated by an electromyography signal collection target object in connection with performing one of multiple gestures and a corresponding gesture information label;
dividing the electromyography signal sample through a sliding window having a fixed window value and a fixed stride into different electromyography signals of the target object; and
applying the different electromyography signals to a first neural network model to determine gesture information matching the one of multiple gestures performed by the target object, further including:
recognizing the different electromyography signals based on the gesture information label;
determining probabilities of gesture information represented by the different electromyography signals using the first neural network model; and
weighting the probabilities of the gesture information represented by the different electromyography signals to determine the gesture information matching the one of multiple gestures performed by the target object.
16. The non-transitory computer-readable storage medium of claim 15, wherein method further comprises:
determining a dynamic noise threshold associated with the multiple gestures performed by the target object; and
denoising the different electromyography signals according to the dynamic noise threshold to form dynamic noise electromyography signals matching the dynamic noise threshold.
17. The non-transitory computer-readable storage medium of claim 15, wherein the method further comprises:
determining features of corresponding electromyography signals when the gesture information usage environment of the gesture recognition model is a powered skeleton;
determining output sizes of different networks in a first neural network model of the gesture recognition model and a quantity of corresponding convolution kernels according to the features of the electromyography signals; and
adjusting quantities of a temporal convolutional layer network, a spatial convolutional layer network, a fully connected layer network and a residual module network of the first neural network model according to features of the powered skeleton, so as to match initialization parameters of the gesture recognition model with the features of the powered skeleton.
18. The non-transitory computer-readable storage medium of claim 15, wherein the first neural network model is trained by:
obtaining a first training sample set, wherein the first training sample set comprises at least one group of electromyography signal samples and a corresponding set of gesture information labels;
determining initial parameters of the first neural network model in the gesture recognition model;
iteratively updating the initial parameters of the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model until after a loss function associated with the first neural network model meets a corresponding convergence condition.
19. The non-transitory computer-readable storage medium of claim 18, wherein the first training sample set is generated by:
acquiring an electromyography signal sample matching a training sample collection object and a corresponding set of gesture information labels; and
dividing the electromyography signal sample through the sliding window having the fixed window value and the fixed stride to form the first training sample set.
20. The non-transitory computer-readable storage medium of claim 18, wherein the iteratively updating the initial parameters of the first neural network model further comprises:
determining a classification loss and a contrastive loss corresponding to the first neural network model;
determining a fusion loss matching the first neural network model according to the classification loss and the contrastive loss as the loss function associated with the first neural network model; and
training the first neural network model by applying the first training sample set to the first neural network model according to the updated parameters of the first neural network model iteratively until after the fusion loss indicates that the corresponding convergence condition is reached.
US18/227,857 2020-01-13 2023-07-28 Gesture information processing method and apparatus, electronic device, and storage medium Active 2040-12-09 US12422935B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/227,857 US12422935B2 (en) 2020-01-13 2023-07-28 Gesture information processing method and apparatus, electronic device, and storage medium

Applications Claiming Priority (5)

Application Number Priority Date Filing Date Title
CN202010033904.XA CN111209885B (en) 2020-01-13 2020-01-13 Gesture information processing method, device, electronic equipment and storage medium
CN202010033904.X 2020-01-13
PCT/CN2020/130567 WO2021143353A1 (en) 2020-01-13 2020-11-20 Gesture information processing method and apparatus, electronic device, and storage medium
US17/580,545 US11755121B2 (en) 2020-01-13 2022-01-20 Gesture information processing method and apparatus, electronic device, and storage medium
US18/227,857 US12422935B2 (en) 2020-01-13 2023-07-28 Gesture information processing method and apparatus, electronic device, and storage medium

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US17/580,545 Continuation US11755121B2 (en) 2020-01-13 2022-01-20 Gesture information processing method and apparatus, electronic device, and storage medium

Publications (2)

Publication Number Publication Date
US20230376120A1 US20230376120A1 (en) 2023-11-23
US12422935B2 true US12422935B2 (en) 2025-09-23

Family

ID=70789075

Family Applications (2)

Application Number Title Priority Date Filing Date
US17/580,545 Active 2040-11-20 US11755121B2 (en) 2020-01-13 2022-01-20 Gesture information processing method and apparatus, electronic device, and storage medium
US18/227,857 Active 2040-12-09 US12422935B2 (en) 2020-01-13 2023-07-28 Gesture information processing method and apparatus, electronic device, and storage medium

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US17/580,545 Active 2040-11-20 US11755121B2 (en) 2020-01-13 2022-01-20 Gesture information processing method and apparatus, electronic device, and storage medium

Country Status (3)

Country Link
US (2) US11755121B2 (en)
CN (1) CN111209885B (en)
WO (1) WO2021143353A1 (en)

Families Citing this family (37)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111209885B (en) * 2020-01-13 2023-05-30 腾讯科技（深圳）有限公司 Gesture information processing method, device, electronic equipment and storage medium
CN111844032B (en) * 2020-07-15 2022-04-12 京东科技信息技术有限公司 Electromyographic signal processing and exoskeleton robot control method and device
CN111870242A (en) * 2020-08-03 2020-11-03 南京邮电大学 Intelligent gesture action generation method based on electromyographic signals
CN112132192A (en) * 2020-09-07 2020-12-25 北京海益同展信息科技有限公司 A model training method, device, electronic device and storage medium
CN112507881A (en) * 2020-12-09 2021-03-16 山西三友和智慧信息技术股份有限公司 sEMG signal classification method and system based on time convolution neural network
CN112932508B (en) * 2021-01-29 2022-10-14 电子科技大学 Finger activity recognition system based on arm electromyography network
CN113312994A (en) * 2021-05-18 2021-08-27 中国科学院深圳先进技术研究院 Gesture classification recognition method and application thereof
CN113435340B (en) * 2021-06-29 2022-06-10 福州大学 Real-time gesture recognition method based on improved Resnet
CN113554089B (en) * 2021-07-22 2023-04-18 西安电子科技大学 Image classification countermeasure sample defense method and system and data processing terminal
CN113625882B (en) * 2021-10-12 2022-06-14 四川大学 Myoelectric gesture recognition method based on sparse multichannel correlation characteristics
CN114356076B (en) * 2021-12-13 2023-10-03 中国船舶重工集团公司第七0九研究所 Gesture control method and system
CN114499712B (en) * 2021-12-22 2024-01-05 天翼云科技有限公司 A gesture recognition method, device and storage medium
US11768544B2 (en) * 2022-02-01 2023-09-26 Microsoft Technology Licensing, Llc Gesture recognition based on likelihood of interaction
CN114167996B (en) * 2022-02-14 2022-05-17 浙江强脑科技有限公司 Sensor-based action pre-judging method and device and storage medium
CN114595725B (en) * 2022-03-15 2024-02-20 合肥工业大学 An EEG signal classification method based on additive network and supervised contrastive learning
CN115170831A (en) * 2022-05-24 2022-10-11 重庆邮电大学 Gesture recognition method based on UWB radar technology
CN114936583B (en) * 2022-06-14 2024-03-29 中国科学技术大学 Dual-step field self-adaptive cross-user myoelectricity mode identification method based on teacher-student model
CN115273135B (en) * 2022-06-14 2025-08-05 浙江理工大学 Gesture image classification method based on DC-Res2Net and feature fusion attention module
CN115482587A (en) * 2022-10-17 2022-12-16 厦门大学 Dynamic gesture recognition method based on convolution structure
CN115719514B (en) * 2022-11-23 2023-06-30 南京理工大学 Gesture recognition-oriented field self-adaption method and system
US12073027B2 (en) * 2022-12-20 2024-08-27 Accenture Global Solutions Limited Behavior-based standard operating procedure detection
CN116074835B (en) * 2023-01-16 2024-09-10 上海物骐微电子有限公司 WiFi-based gesture recognition method and device, electronic device, and storage medium
CN116139387B (en) * 2023-04-20 2023-08-29 浙江强脑科技有限公司 Training control method for reaction force training, terminal equipment and storage medium
CN116541775A (en) * 2023-05-11 2023-08-04 北京有竹居网络技术有限公司 Gesture action recognition method, device, electronic device and storage medium
CN116414289B (en) * 2023-06-12 2023-08-22 上海海栎创科技股份有限公司 Gesture recognition system and method based on touch chip
CN117150292B (en) * 2023-08-24 2025-10-17 中国科学院计算技术研究所 Incremental learning-based gesture recognition model training method
CN117216605B (en) * 2023-08-29 2024-10-15 北京国电光宇机电设备有限公司 High-precision electric energy data acquisition method and system
WO2025062565A1 (en) * 2023-09-21 2025-03-27 日本電信電話株式会社 Gesture input device and method
CN117216644B (en) * 2023-11-09 2024-02-02 北京世纪慈海科技有限公司 Human body posture recognition method and device based on electrical digital data processing
CN117238037B (en) * 2023-11-13 2024-03-29 中国科学技术大学 Dynamic action recognition method, device, equipment and storage medium
FR3158383A1 (en) * 2024-01-12 2025-07-18 Idemia Identity & Security France Method and system for classifying dactylograms
CN118445747B (en) * 2024-04-26 2024-12-24 西北工业大学深圳研究院 Multi-mode gesture recognition method based on data enhanced brain electromyographic signal fusion
CN118216937B (en) * 2024-04-30 2024-11-12 北京瑞蜜达国际生物科技有限公司 A method and system for analyzing and processing electromyographic signals based on big data
CN118171118B (en) * 2024-05-13 2024-08-09 浙江强脑科技有限公司 Visual bionic hand training method, training device, storage medium and bionic hand
CN118680584B (en) * 2024-07-08 2024-11-22 岳正检测认证技术有限公司 A method for denoising electromyographic signals based on machine learning
CN118520419B (en) * 2024-07-22 2024-11-26 中山大学 A multimodal gesture recognition method and system based on electromyography and vision fusion
CN119045670B (en) * 2024-10-28 2025-03-18 中国科学院苏州生物医学工程技术研究所 Wheelchair control method, wheelchair and equipment based on sEMG and IMU fusion

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130046715A1 (en) 2010-01-11 2013-02-21 Universite De Mons Method to determine an artificial limb movement from an electroencephalographic signal
US20150156545A1 (en) * 2012-01-04 2015-06-04 Thomson Licensing Method and system for providing media recommendations
CN108491077A (en) 2018-03-19 2018-09-04 浙江大学 A kind of surface electromyogram signal gesture identification method for convolutional neural networks of being divided and ruled based on multithread
CN109213305A (en) * 2017-06-29 2019-01-15 沈阳新松机器人自动化股份有限公司 A kind of gesture identification method based on surface electromyogram signal
CN110399846A (en) 2019-07-03 2019-11-01 北京航空航天大学 A Gesture Recognition Method Based on Correlation of Multi-channel EMG Signals
CN110610172A (en) 2019-09-25 2019-12-24 南京邮电大学 An EMG gesture recognition method based on RNN-CNN architecture
CN110658915A (en) 2019-07-24 2020-01-07 浙江工业大学 A method of EMG gesture recognition based on dual-stream network
CN111209885A (en) 2020-01-13 2020-05-29 腾讯科技（深圳）有限公司 A gesture information processing method, device, electronic device and storage medium
US20200349439A1 (en) * 2019-04-30 2020-11-05 Samsung Electronics Co., Ltd. System and method for convolutional layer structure for neural networks
US20200356178A1 (en) 2019-05-06 2020-11-12 Samsung Electronics Co., Ltd. Methods for gesture recognition and control

Patent Citations (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130046715A1 (en) 2010-01-11 2013-02-21 Universite De Mons Method to determine an artificial limb movement from an electroencephalographic signal
US20150156545A1 (en) * 2012-01-04 2015-06-04 Thomson Licensing Method and system for providing media recommendations
CN109213305A (en) * 2017-06-29 2019-01-15 沈阳新松机器人自动化股份有限公司 A kind of gesture identification method based on surface electromyogram signal
CN108491077A (en) 2018-03-19 2018-09-04 浙江大学 A kind of surface electromyogram signal gesture identification method for convolutional neural networks of being divided and ruled based on multithread
US20200349439A1 (en) * 2019-04-30 2020-11-05 Samsung Electronics Co., Ltd. System and method for convolutional layer structure for neural networks
US20200356178A1 (en) 2019-05-06 2020-11-12 Samsung Electronics Co., Ltd. Methods for gesture recognition and control
CN110399846A (en) 2019-07-03 2019-11-01 北京航空航天大学 A Gesture Recognition Method Based on Correlation of Multi-channel EMG Signals
CN110658915A (en) 2019-07-24 2020-01-07 浙江工业大学 A method of EMG gesture recognition based on dual-stream network
CN110610172A (en) 2019-09-25 2019-12-24 南京邮电大学 An EMG gesture recognition method based on RNN-CNN architecture
CN111209885A (en) 2020-01-13 2020-05-29 腾讯科技（深圳）有限公司 A gesture information processing method, device, electronic device and storage medium
US11755121B2 (en) * 2020-01-13 2023-09-12 Tencent Technology (Shenzhen) Company Limited Gesture information processing method and apparatus, electronic device, and storage medium

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Deep Learning for Electromyographic Hand Gesture Signal Classification Using Transfer Learning Ulysse Côté-Allard , Cheikh Latyr Fall , , IEEE Transactions On Neural Systems and Rehabili (Year: 2019). *
Tencent Technology, IPRP, PCT/CN2020/130567, Jul. 19, 2022, 6 pgs.
Tencent Technology, ISRWO, PCT/CN2020/130567, Feb. 22, 2021, 7 pgs.
Yu Du et al., "Surface EMG-Based Inter-Session Gesture Recognition Enhanced by Deep Domain Adaptation", Sensors 2017, Feb. 24, 2017, MDPI, 22 pgs.

Also Published As

Publication number Publication date
US11755121B2 (en) 2023-09-12
US20220147151A1 (en) 2022-05-12
US20230376120A1 (en) 2023-11-23
WO2021143353A1 (en) 2021-07-22
CN111209885A (en) 2020-05-29
CN111209885B (en) 2023-05-30

Similar Documents

Publication Publication Date Title
US12422935B2 (en) Gesture information processing method and apparatus, electronic device, and storage medium
US11468288B2 (en) Method of and system for evaluating consumption of visual information displayed to a user by analyzing user&#39;s eye tracking and bioresponse data
Martinez-Cagigal et al. An asynchronous P300-based brain-computer interface web browser for severely disabled people
CN111222464B (en) Emotion analysis method and system
CN111631704A (en) Pre-diabetes detection system and method based on the combination of ECG and EEG information
Pirasteh et al. EEG-based brain-computer interface methods with the aim of rehabilitating advanced stage ALS patients
López et al. Comparison of classification techniques for the control of EOG-based HCIs
Khan et al. AT2GRU: A human emotion recognition model with mitigated device heterogeneity
Wang et al. Neural decoding of Chinese sign language with machine learning for brain–computer interfaces
Falcari et al. Evaluation of multi-class support-vector machines strategies and kernel adjustment levels in hand posture recognition by analyzing sEMG signals acquired from a wearable device
Kumar et al. Study towards the analytic approach for human computer interaction using machine learning
Zea et al. Real-time hand gesture recognition: a long short-term memory approach with electromyography
Ferdi et al. Authentication with a one-dimensional CNN model using EEG-based brain-computer interface
Shah et al. Deep learning framework for internet of things for people with disabilities
Alhammadi et al. Cursor Control Using electroencephalogram (EEG) Technology
Yamagami et al. Customized mid-air gestures for accessibility: A $ B recognizer for multi-dimensional biosignal gestures
Al Ajrawi et al. A hierarchical recursive feature elimination algorithm to develop brain computer interface application of user behavior for statistical reasoning and decision making
Su et al. Doctor robot with physical examination for skin disease diagnosis and telemedicine application
Piskozub et al. A sensory glove with a limited number of sensors for recognition of the finger alphabet of Polish Sign Language Jakub Piskozub
Acampora et al. Applying logistic regression for classification in single-channel SSVEP-based BCIs
Liu et al. Dynamic hand gesture recognition for numeral handwritten via a-mode ultrasound
CN114936574A (en) High-flexibility manipulator system based on BCI and implementation method thereof
Faria et al. Analyzing data augmentation methods for convolutional neural network-based brain-computer interfaces
Mahantesh et al. Prosthetic AI Enabled Arm for Rehabilitation and Advanced Dynamics
Khera et al. AI-enabled Hybrid Model for Lower-Limb Movements Recognition using Estimated Muscular Activity from Cortical Brain Signals

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

AS Assignment

Owner name: TENCENT TECHNOLOGY (SHENZHEN) COMPANY LIMITED, CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HONG, XIAOLIN;ZHENG, QINGQING;WANG, XINMIN;AND OTHERS;SIGNING DATES FROM 20220110 TO 20220118;REEL/FRAME:064456/0524

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE