WO2023061037A1 - 隔空手势识别方法及其相关设备 - Google Patents
隔空手势识别方法及其相关设备 Download PDFInfo
- Publication number
- WO2023061037A1 WO2023061037A1 PCT/CN2022/113425 CN2022113425W WO2023061037A1 WO 2023061037 A1 WO2023061037 A1 WO 2023061037A1 CN 2022113425 W CN2022113425 W CN 2022113425W WO 2023061037 A1 WO2023061037 A1 WO 2023061037A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- signal
- target
- myoelectric
- acceleration
- frame
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 122
- 230000001133 acceleration Effects 0.000 claims abstract description 162
- 230000009471 action Effects 0.000 claims abstract description 66
- 108010076504 Protein Sorting Signals Proteins 0.000 claims abstract description 64
- 230000003183 myoelectrical effect Effects 0.000 claims description 204
- 238000002567 electromyography Methods 0.000 claims description 54
- 238000012545 processing Methods 0.000 claims description 21
- 230000015654 memory Effects 0.000 claims description 19
- 238000004590 computer program Methods 0.000 claims description 18
- 230000008859 change Effects 0.000 claims description 11
- 230000005484 gravity Effects 0.000 claims description 7
- 238000009432 framing Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 230000000306 recurrent effect Effects 0.000 claims description 6
- 238000003062 neural network model Methods 0.000 claims description 5
- 230000003993 interaction Effects 0.000 abstract description 76
- 230000000694 effects Effects 0.000 abstract description 10
- 238000005516 engineering process Methods 0.000 description 23
- 238000004891 communication Methods 0.000 description 22
- 230000008569 process Effects 0.000 description 18
- 230000006870 function Effects 0.000 description 17
- 230000033001 locomotion Effects 0.000 description 15
- 238000010586 diagram Methods 0.000 description 10
- 238000010295 mobile communication Methods 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 230000000717 retained effect Effects 0.000 description 4
- 238000004364 calculation method Methods 0.000 description 3
- 125000004122 cyclic group Chemical group 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000007774 longterm Effects 0.000 description 3
- 238000012706 support-vector machine Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000000739 chaotic effect Effects 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000007726 management method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000011218 segmentation Effects 0.000 description 2
- 230000006403 short-term memory Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 241000699670 Mus sp. Species 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 210000004247 hand Anatomy 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000004118 muscle contraction Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 230000002232 neuromuscular Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000013307 optical fiber Substances 0.000 description 1
- 230000036961 partial effect Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000002829 reductive effect Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003238 somatosensory effect Effects 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 210000000707 wrist Anatomy 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
- G06F2203/01—Indexing scheme relating to G06F3/01
- G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
Definitions
- the present application relates to the field of human-computer interaction, in particular to a method for recognizing gestures in the air and related equipment.
- Traditional human-computer interaction devices mainly include keyboards, mice, handwriting tablets, touch screens, game controllers, etc. These devices use the user's hand movements to realize the functions of human-computer interaction.
- air gesture interaction supports more and more natural interaction methods, and provides a human-centered rather than device-centered interaction technology, so that users can focus on what they should do and the content instead of Not focused on equipment.
- air gesture recognition the most important technology in the process of air gesture interaction is air gesture recognition.
- Common air gesture recognition technologies are mainly divided into air gesture recognition technology based on machine vision and air gesture recognition technology based on electromyographic signals.
- the air gesture recognition technology based on machine vision mainly relies on cameras, depth sensors, etc. to collect gesture information, and then uses the model to segment the image corresponding to the human hand, so as to realize gesture detection and recognition.
- This method has relatively high requirements on the environment and vision, and the recognition ability is very limited.
- the types of sensors that can be used for gesture interaction based on myoelectric signals are relatively small, and the functions are very imperfect.
- the present application provides a method for recognizing gestures in the air and related equipment, which jointly determine the gesture action in the air by combining electromyography signals, acceleration signals and angle signals, thereby improving the effect of gesture recognition in the air.
- a method for recognizing gestures in the air which is applied to a first electronic device used by a user, and the first electronic device communicates with a second electronic device, and the method includes:
- the target air gesture corresponding to the user Action Determine a corresponding target operation instruction according to the target air gesture action and the angle signal sequence, where the target operation instruction includes the type and adjustment range of the target operation instruction.
- the embodiment of the present application provides a space gesture recognition technology, which not only collects myoelectric signals and acceleration signals, but also increases the collection of angle signals; Empty gesture action. Since the angle signal is combined, the recognition condition of the gesture action is increased, so that the gesture recognition effect can be effectively improved.
- determining the target air gesture action corresponding to the user according to the myoelectric signal sequence, the acceleration sequence, and the angle signal sequence includes: The signal sequence and the acceleration signal sequence determine the starting moment of the gesture in the air; from the starting moment of the gesture in the air, determine the target myoelectric signal, the target acceleration signal and the target angle signal; according to the target myoelectric signal, The target acceleration signal and the target angle signal use a network model to determine the target air gesture action corresponding to the user.
- the start time of the air gesture is first determined based on the myoelectric signal and the acceleration signal, and then, after the start time of the air gesture is determined, based on the collected target myoelectric signal, target acceleration signal and target Angle signal, using the network model to determine the corresponding target air gesture action. Therefore, by combining angles and increasing recognition conditions, the effect of gesture recognition can be effectively improved.
- determining the starting moment of the air gesture according to the myoelectric signal sequence and the acceleration signal sequence includes: performing sliding window segmentation on the myoelectric signal sequence, and determining The fuzzy entropy corresponding to each frame of myoelectric signal; judging whether the fuzzy entropy corresponding to the sth frame of myoelectric signal to the s+M-1 frame of myoelectric signal is greater than the preset fuzzy entropy threshold, and the corresponding fuzzy entropy of the sth frame acceleration signal Whether the acceleration modulus is greater than the preset acceleration modulus, s and M are both integers greater than 0; if so, the moment corresponding to the sth frame of myoelectric signal is used as the starting moment of the air gesture.
- the degree of confusion of the EMG signal is distinguished by setting the preset fuzzy entropy threshold, and then combined with the acceleration modulus of the acceleration signal, it is used as the judgment condition for the initial moment of the gesture in the air, so that Improve the accuracy of the determined starting moment of the air gesture, so as to facilitate subsequent accurate recognition of the air gesture action.
- performing sliding window segmentation on the EMG signal sequence, and determining the fuzzy entropy corresponding to each frame of EMG signal includes: dividing the EMG signal sequence according to the length of the first sliding window.
- the signal sequence is divided into multiple frames of myoelectric signals; according to the second sliding window length, each frame of myoelectric signals in the multiple frames of myoelectric signals is divided into k subsequences of myoelectric signals, wherein the first sliding window
- the length is N
- the first average value of the fuzzy membership degrees corresponding to each myoelectric signal subsequence; the second sliding window length is updated to m+1, and for each frame of myoelectric signal, determine the relationship between each myoelectric signal subsequence and The second average value of the fuzzy membership degrees corresponding to the other k
- the degree of confusion between the front and rear parts of the waveform corresponding to the electromyographic signal can be evaluated.
- the fuzzy membership degree of each myoelectric signal subsequence corresponding to the other k-1 said myoelectric signal subsequences respectively An average value or the second average value, comprising: for each frame of myoelectric signal, according to the distance formula, determine the distance between each myoelectric signal subsequence and all k myoelectric signal subsequences; , use the fuzzy membership degree formula to determine the corresponding fuzzy membership degrees between each EMG signal subsequence and all k EMG signal subsequences; according to the fuzzy membership degree, use the fuzzy membership degree average formula to determine each The first average value or the second average value of the fuzzy membership degrees corresponding to the myoelectric signal subsequence and other k-1 myoelectric signal subsequences respectively.
- the method further includes: when the fuzzy entropy corresponding to the sth frame to the s+M-1th frame of the myoelectric signal is less than or equal to the When the fuzzy entropy threshold is preset, the fuzzy entropy corresponding to the EMG signal is updated to 0.
- the fuzzy entropy corresponding to the electromyographic signal that does not satisfy the condition can be rectified, so that the fuzzy entropy of the electromyographic signal that meets the condition is different from the fuzzy entropy of the electromyographic signal that does not meet the condition.
- the degree of confusion corresponding to the conditional EMG signal is more significant, which is more beneficial to subsequent processing.
- the method further includes: when the fuzzy entropies corresponding to the sth frame to the s+M-1th frame of the myoelectric signal are not all greater than the When the fuzzy entropy threshold is preset; or, when the acceleration modulus corresponding to the acceleration signal of the sth frame is less than or equal to the preset acceleration modulus, then it is judged that the s+1 frame myoelectric signal is to the s+M frame Whether the fuzzy entropy corresponding to the EMG signal is greater than the preset fuzzy entropy threshold, and whether the acceleration modulus corresponding to the s+1th frame acceleration signal is greater than the preset acceleration modulus.
- the myoelectric signal and the acceleration signal do not meet the conditions for determining the start moment of the air gesture, it can continue to judge whether the subsequent signal meets the preset condition according to the myoelectric signal sequence and the acceleration signal sequence.
- determining the target myoelectric signal, the target acceleration signal, and the target angle signal includes: starting from the start moment of the air gesture, All the myoelectric signals included in the M frame myoelectric signals are used as the target myoelectric signals, all the acceleration signals included in the M frame acceleration signals are used as the target acceleration signals, and all the angle signals included in the M frame angle signals are used as the target acceleration signals.
- the target angle signal or; from the starting moment of the gesture in the air, according to the fuzzy entropy corresponding to each frame of myoelectric signal, determine the end of the gesture in the air; All myoelectric signals included between gesture termination moments are determined as the target myoelectric signal, all included acceleration signals are determined as the target acceleration signal, and all included angle signals are determined as the target angle signal.
- the effective signal can be determined by presetting the number of frames or by determining the start time and end time of the air gesture, so as to facilitate subsequent determination of the air gesture action.
- a network model is used to determine the target air gesture action corresponding to the user, Including: according to the target electromyographic signal, determine the average absolute value of the time domain characteristic, the slope sign change value and the waveform length; according to the target acceleration signal, determine the original data of the acceleration three axes, the distribution of gravity on the three axes and the Acceleration modulus; according to the target angle signal, determine the absolute value and rotation matrix of the three-axis raw data of the gyroscope and the multi-axis raw data of the gyroscope; according to the average absolute value of the time domain feature, the sign change value of the slope and at least one of the waveform length, at least one of the three-axis raw data of the acceleration, at least one of the distribution of the gravity in the three axes and the acceleration modulus, and the three-axis raw data of the gyroscope, The absolute value multiplied
- the network model is a recurrent neural network model, a GRU network model or an LSTM network model.
- the air gesture action includes: open the palm and wave to the left or right, open the palm and wave upward or downward, make a fist and wave upward or downward, make a fist and stretch the index finger upward or at least one of the down swipes.
- determining the corresponding target operation instruction according to the target air gesture action and the angle signal sequence includes: determining the target operation instruction corresponding to the target air gesture action Type, the type of the operation instruction includes at least one of sliding page, volume adjustment, and video progress adjustment; according to the type of the target operation instruction and the target angle signal in the angle signal sequence, determine the target The adjustment range of the operation command.
- the type of the operation instruction is used to indicate what the air gesture action wants to do
- the adjustment range of the operation instruction is used to indicate: based on what the air gesture action wants to do, determine how much it wants to do. Therefore, by determining the type and adjustment range of the target operation instruction, the accuracy of the corresponding operation when the subsequent air gesture action controls the second electronic device can be improved.
- the method further includes: sending the target operation instruction to the second electronic device.
- an air gesture interaction device in a second aspect, includes a unit for performing each step in the above first aspect or any possible implementation manner of the first aspect.
- an electronic device including: a myoelectric electrode, an acceleration sensor, a gyroscope sensor, a processor, and a memory; the myoelectric electrode is used to collect the user's myoelectric signal; the acceleration sensor is used to The user generates an acceleration signal when using the electronic device; the gyro sensor is used to generate an angle signal when the user uses the electronic device; the memory is used to store the A computer program; the processor, configured to execute the processing steps in the air gesture recognition method provided in the first aspect or any possible implementation manner of the first aspect.
- a chip including: a processor, configured to call and run a computer program from a memory, so that a device installed with the chip executes the method provided in the first aspect or any possible implementation manner of the first aspect.
- Air gesture recognition method configured to call and run a computer program from a memory, so that a device installed with the chip executes the method provided in the first aspect or any possible implementation manner of the first aspect. Air gesture recognition method.
- a computer-readable storage medium stores a computer program.
- the computer program includes program instructions.
- the air gesture recognition method provided in any possible implementation manner of the aspect.
- a computer program product includes a computer-readable storage medium storing a computer program, and the computer program enables the computer to execute the isolation program provided in the first aspect or any possible implementation manner of the first aspect. Null gesture recognition method.
- FIG. 1 is a scene diagram for the application of the air gesture interaction system provided by the embodiment of the present application
- FIG. 2 is a schematic flowchart of a method for recognizing gestures in space provided by an embodiment of the present application
- Fig. 3 is a kind of framing of the myoelectric signal sequence provided by the embodiment of the present application, and a schematic diagram of determining the fuzzy entropy corresponding to each frame of the myoelectric signal;
- Fig. 4 is a fuzzy membership table provided by the embodiment of the present application.
- FIG. 5 is a schematic flowchart of a method for inter-space gesture interaction provided by an embodiment of the present application
- FIG. 6 is a schematic structural diagram of an electronic device provided by an embodiment of the present application.
- FIG. 7 is a schematic structural diagram of an air gesture recognition system provided by an embodiment of the present application.
- FIG. 8 is a schematic structural diagram of a chip provided by an embodiment of the present application.
- a relationship means that there may be three kinds of relationships, for example, A and/or B means: A exists alone, A and B exist simultaneously, and B exists alone.
- plural refers to two or more than two.
- first and second are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as “first” and “second” may explicitly or implicitly include one or more of these features. In the description of this embodiment, unless otherwise specified, “plurality” means two or more.
- the air gesture recognition technology based on machine vision mainly uses machine vision to process and recognize gesture image sequences collected by a camera installed on a TV, so as to realize the interaction between the hand and the TV.
- machine vision is used to process and recognize the sequence of gesture images collected by the camera installed on the vehicle center console, so as to realize the interaction between the hand and the vehicle center console.
- This method uses a camera to collect gesture information, and then, for example, uses a skin color model to segment the human hand to achieve gesture detection and recognition, and finally uses the frame difference method to track motion gestures.
- the effect of this method depends on the accuracy of the skin color model.
- the color of human skin is different, and it is difficult to obtain a general and efficient skin color model; moreover, when the movement speed of the human hand is uneven, there will be interruptions in tracking gestures using the frame difference method , thus losing the tracked gesture.
- this method has relatively high requirements on the acquisition environment and field of view, and the recognition ability is very limited.
- the wearable device worn by the user on the arm collects the electromyographic signals corresponding to different gestures and performs gesture recognition processing, and then transmits them to the electronic device connected to the wearable device.
- the device can realize the interaction between the hand and the electronic device.
- the user can collect the electromyographic signals corresponding to different gestures through the ring on the finger of the user, perform gesture recognition processing, and then transmit it to the electronic device connected to the ring to realize hand gesture recognition. interaction with electronic devices.
- there are fewer types of sensors that can be used for air-to-air interaction based on electromyographic signals and the functions are not perfect, and gesture recognition only through electromyographic signals is usually not accurate enough.
- the prior art also provides a technology for gesture recognition based on the combination of myoelectric signals and acceleration signals.
- the air gesture recognition technology collects myoelectric signals and acceleration signals, and then uses support vector machines (SVM) for machine learning to realize gesture recognition.
- SVM support vector machines
- the air gesture recognition technology can make up for the limitations of the environment and field of view, allowing users to get rid of the range limitation and relatively improve the recognition ability, but because the support vector machine makes decisions, it only relies on There are limited samples, so the recognition effect is still not accurate enough to fully meet the needs of users.
- the embodiment of the present application provides an air gesture recognition technology, which not only collects myoelectric signals and acceleration signals, but also increases the collection of angle signals; then, based on the myoelectric signals, acceleration signals and angle signals, the The corresponding target air gesture action. Since the angle signal is combined, the recognition condition of the gesture action is increased, so that the gesture recognition effect can be effectively improved.
- FIG. 1 is an example of a scene diagram applicable to an air gesture interaction system applicable to an embodiment of the present application.
- the air gesture interaction system provided by the embodiment of the present application includes: a first electronic device 11 and a second electronic device 12 .
- the first electronic device 11 is an example of a smart watch used by a user
- the second electronic device 12 is an example of a tablet computer used by a user.
- the air gesture interaction system shown in FIG. 1 may also include more other electronic devices, which is not limited in this embodiment of the present application.
- the first electronic device 11 may include: a smart watch, a smart bracelet, a smart ring, a wearable electronic device on a finger or an arm, etc., which is not limited in the embodiment of the present application.
- the second electronic device 12 may include: a smart phone, a smart TV, a large-screen device, a tablet computer, a personal digital assistant (personal digital assistant, PDA), a computer handheld communication device, a sound box, a vehicle-mounted device (also called a car machine) , laptops, ultra-mobile personal computers (UMPC), handheld computers, netbooks, wearable electronic devices, virtual reality devices and other IOT (internet of things, Internet of Things) devices, electronic devices in 5G networks, etc. , which is not limited in this embodiment of the present application.
- the first electronic device 11 and the second electronic device 12 may be interconnected through a communication network.
- the communication network may be a wired network or a wireless network.
- the aforementioned communication network may be a local area network (local area networks, LAN), or a wide area network (wide area networks, WAN), such as the Internet.
- the above-mentioned communication network can be realized using any known network communication protocol, and the above-mentioned network communication protocol can be various wired or wireless communication protocols, such as Ethernet, universal serial bus (universal serial bus, USB), fire wire (FIREWIRE), Global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), Bluetooth, wireless fidelity (Wi-Fi), Wi-Fi direct (wireless fidelity peer-to-peer, Wi-Fi P2P) connection, NFC, voice over Internet protocol (voice over Internet protocol, VoIP), communication protocol supporting network slicing architecture, or any other suitable letter of agreement.
- Ethernet universal serial bus
- USB fire wire
- FIREWIRE Fire wire
- GSM Global system for mobile communications
- GPRS general packet radio service
- CDMA code division multiple access
- WCDMA wideband code division multiple access
- each electronic device in the air gesture interaction system can build a network (that is, a network) according to a certain communication protocol and networking strategy, so that each electronic device in the air gesture interaction system can communicate with each other.
- the first electronic device 11 needs to transmit the data collected by the first electronic device 11 to the second electronic device 12, and the data of the first electronic device 11 involves user privacy issues. Therefore, before the first electronic device 11 sends data to the second electronic device 12, a trusted relationship can be established between the first electronic device 11 and the second electronic device 12 as a premise, that is, the first electronic device 11 and the A device-level authentication relationship is established between the second electronic devices 12 .
- a device-level authentication relationship can be established between the first electronic device 11 and the second electronic device 12 in various ways, which is not limited in this embodiment of the application.
- the user wears a smart watch on his wrist, and the user wants to operate the tablet computer by waving his hand, so that the applications on the tablet computer can be controlled by the user's hand movements in the air.
- the tablet computer takes a screenshot.
- the air gesture recognition method and the air gesture interaction method provided in this application can be used.
- the user wears a smart ring on his finger, and the user wants to operate the TV by waving his hand, so that the TV can be controlled by the user's hand motion in the air.
- the TV performs page switching processing.
- the air gesture recognition method and the air gesture interaction method provided in this application can be used.
- the user wears a wearable device on his arm, and the user wants to operate the vehicle center console by waving his hand, so that the vehicle center console can be controlled by the user's hand movements in the air. control.
- the vehicle center console performs the processing of making a call.
- the air gesture recognition method and the air gesture interaction method provided in this application can be used.
- the scene shown in FIG. 1 is taken as an example below.
- the first electronic device 11 is a smart watch
- the second electronic device 12 is a tablet computer.
- the smart watch and the tablet computer have been connected through a communication network.
- the user can control the tablet computer through the air to achieve different processing by performing various air gestures.
- the air gesture recognition method provided by the embodiment of the present application can be applied to the first electronic device 11, and the air gesture interaction provided by the embodiment of the present application can be applied between the first electronic device 11 and the second electronic device 12 method.
- FIG. 2 is a schematic flowchart of a method 40 for recognizing gestures in space provided by an embodiment of the present application.
- the air gesture recognition method 40 is applied to smart watches.
- the air gesture recognition method 40 includes: S101 to S113.
- S101 Synchronously collect a user's myoelectric signal, and an acceleration signal and an angle signal generated when the user uses the first electronic device 11 .
- the first electronic device 11 can collect myoelectric signals, acceleration signals and angle signals generated when the user's hands move.
- EMG is a bioelectrical signal related to neuromuscular activity.
- EMG signals can reflect information such as muscle contraction patterns and contraction intensity. Different body movements correspond to different EMG signals.
- the myoelectric electrode can be installed on the smart watch, and the myoelectric electrode can be used to contact the human skin, so that the myoelectric electrode can be used to continuously collect multiple myoelectric signals generated by the user's hand movements, and generate corresponding myoelectric signals sequence.
- the acceleration signal refers to a signal representing acceleration.
- the acceleration signal can be generated by an acceleration sensor.
- the acceleration sensor may be a single-axis acceleration sensor, a two-axis acceleration sensor or a three-axis acceleration sensor.
- the three-axis acceleration sensor in the smart watch will generate multiple acceleration signals according to the motion state of the hand, and collect the acceleration signals.
- a plurality of acceleration signals is used to generate a corresponding acceleration signal sequence.
- the angle signal refers to a signal used to indicate the angle of movement of an object.
- the angle signal can be generated by a gyro sensor.
- the gyro sensor may be a two-degree-of-freedom gyroscope or a three-degree-of-freedom gyroscope.
- the smart watch includes a three-degree-of-freedom gyroscope, then when the user wears the smart watch and waves the hand, the three-degree-of-freedom gyroscope in the smart watch will generate multiple angle signals according to the motion state of the hand, The multiple angle signals are collected to generate a corresponding angle signal sequence.
- Synchronous acquisition means that it is acquired at the same time, and the sequence of the acquired myoelectric signal, acceleration signal and angle signal is consistent.
- the collected myoelectric signal, acceleration signal and angle signal are all the first frame signals in their respective signal sequences, that is to say, the collected myoelectric signal is the first frame of myoelectric signal, and the acceleration signal is the acceleration signal of the first frame, and the angle signal is the angle signal of the first frame.
- the order may be recorded by dots during acquisition, so that the order of the electromyographic signal, the acceleration signal and the angle signal is consistent.
- each frame of myoelectric signal includes one or more myoelectric signals.
- the EMG signal can also be preprocessed before performing sliding window framing on the EMG signal sequence.
- the preprocessing may include: performing signal amplification, power frequency filtering, high-pass filtering and other processing on the EMG signal, which can be set and changed as required, and this embodiment of the present application does not impose any limitation on this.
- fuzzy entropy calculation may be performed on the multiple channels of myoelectric signals collected by different myoelectric electrodes.
- the myoelectric signals of multiple channels collected by multiple myoelectric electrodes can be preprocessed.
- the myoelectric signal sequence includes a plurality of continuous myoelectric signals.
- the length of the sliding window can be set in advance, and the sliding window is used to divide the EMG signal sequence into frames, and then, the following fuzzy entropy calculation formula is used to determine the fuzzy entropy corresponding to each frame of EMG signal.
- Fig. 3 shows a schematic diagram of dividing a sequence of electromyographic signals into frames and determining the fuzzy entropy corresponding to each frame of electromyographic signals.
- the process of dividing the EMG signal sequence into frames and determining the fuzzy entropy corresponding to each frame of EMG signal is as follows:
- Step 1 Framing the EMG signal sequence.
- the electromyographic signal sequence is divided into multiple electromyographic signal segments with a length of N milliseconds.
- each segment of the myoelectric signal corresponds to a frame of the myoelectric signal
- the start time corresponding to each segment of the myoelectric signal is the moment corresponding to the frame of the myoelectric signal.
- the tth moment corresponds to the first frame of myoelectric signal
- the t+M-1 moment corresponds to The EMG signal of the Mth frame.
- the first frame of the myoelectric signal corresponding to the tth moment overlaps with the second frame of the myoelectric signal corresponding to the t+1th moment, and the others are deduced by analogy, which will not be repeated here.
- the third step according to the following distance formula, determine the respective corresponding distances between each EMG subsequence and all k EMG subsequences:
- the distance dij(t) refers to the maximum value of the absolute value of the difference between corresponding elements of the two vectors.
- Step 4 According to the distance dij(t), use the following fuzzy membership degree formula to determine the respective fuzzy membership degrees between each EMG signal subsequence and all k EMG signal subsequences.
- n and r are given parameters, and r is the similarity tolerance.
- FIG. 4 is a fuzzy membership table. As shown in FIG. 4 , the corresponding fuzzy membership degrees between each EMG subsequence and all k EMG subsequences can be calculated.
- Step 5 According to the multiple fuzzy membership degrees obtained above, use the following average formula of fuzzy membership degrees to determine when the length of the second sliding window is m milliseconds, each myoelectric signal subsequence is related to other k-1 myoelectric signal subsequences The first average value of the corresponding fuzzy membership degrees between the sequences.
- Step 6 Increase the length of the second sliding window to m+1 milliseconds, repeat the above-mentioned second to fourth steps, and determine that when the second sliding window length is m+1 milliseconds, each EMG subsequence and other k - the second average value of the respective fuzzy membership degrees corresponding to one EMG signal subsequence.
- Step 7 When the second sliding window length is m milliseconds, the first average value obtained in the fifth step, and when the second sliding window length is m+1 milliseconds, the second average value obtained in the sixth step , use the following fuzzy entropy formula to determine the fuzzy entropy corresponding to each frame of EMG signal with a length of N milliseconds.
- FuzzyEn(t, N) is used to represent the fuzzy entropy corresponding to one frame of EMG signal with length N.
- the size of the fuzzy entropy is related to the values of t and N.
- s is an integer greater than 0.
- the preset fuzzy entropy threshold may be set and adjusted as required, which is not limited in this embodiment of the present application.
- the degree of confusion between the front and rear parts of the waveform corresponding to the EMG signal can be evaluated, and it can also be understood that the repeatability between the front and back waveforms can also be is the frequency.
- the fuzzy entropy is larger, it means that there are more frequencies in the waveform and the more chaotic.
- the fuzzy entropy is smaller, it means that each frequency in the waveform is smaller and less chaotic. Therefore, the degree of confusion can be distinguished by setting a preset fuzzy entropy threshold.
- the fuzzy entropy corresponding to a certain frame of myoelectric signal is greater than the preset fuzzy entropy threshold, the fuzzy entropy corresponding to the frame of myoelectric signal is retained; when the fuzzy entropy corresponding to a certain frame of myoelectric signal is less than or equal to the preset
- the fuzzy entropy threshold is set, the fuzzy entropy corresponding to the frame EMG signal is updated to 0.
- M is used to represent the frame number of the EMG signal, and the size of M can be set and changed as required.
- M is set smaller, it means that starting from the sth frame signal whose fuzzy entropy is preserved, it is expected to compare the fuzzy entropy of a small number of myoelectric signals with the preset fuzzy entropy threshold; when M is set larger , it means that starting from the s-th frame signal whose fuzzy entropy is preserved, it is expected to compare the fuzzy entropy of a larger number of EMG signals with the preset fuzzy entropy threshold.
- the size of the preset M is 9, then after judging that the fuzzy entropy corresponding to the sth frame of the myoelectric signal is greater than the preset fuzzy entropy threshold, determine the subsequent s+1th frame of the myoelectric signal to the s+8th frame of the muscle Whether the fuzzy entropies corresponding to the electric signals are all greater than a preset fuzzy entropy threshold. If the fuzzy entropy corresponding to the s+1th frame to the s+8th frame of the myoelectric signal is greater than the preset fuzzy entropy threshold, then keep the s+1th frame to the s+8th frame of the myoelectric signal The corresponding fuzzy entropy respectively.
- the acceleration signal sequence can be divided into sliding windows with the first sliding window length N, and the time corresponding to each frame of acceleration signals in the divided multi-frame acceleration signals is aligned one-to-one with the time corresponding to each frame of myoelectric signals.
- the acceleration modulus is the root value of the sum of the squares of the components on the x-axis, the components on the y-axis, and the components on the z-axis in the Cartesian coordinate system xyz of the acceleration signal of the sth frame.
- the acceleration modulus is used to represent the magnitude of the acceleration.
- the preset acceleration modulus may be set and adjusted as required, which is not limited in this embodiment of the present application.
- the acceleration signal can be judged and screened first by setting a threshold.
- the starting moment of the air gesture refers to the starting point of the moment corresponding to the s-th frame of the myoelectric signal. That is, the starting point of the time corresponding to the acceleration signal of the sth frame.
- the M-frame myoelectric signal refers to the s-th frame of myoelectric signal corresponding to the starting moment of the gesture in the air, and the M-1 frame of myoelectric signal whose fuzzy entropy is preserved, that is to say, the M-frame
- the myoelectric signals include the myoelectric signals of the sth frame to the s+M-1th frame of the myoelectric signals.
- the moment corresponding to the sth frame of the myoelectric signal is the start moment of the air gesture, then the first signal in the sth frame of the myoelectric signal is the first target myoelectric signal.
- the above S111 may also be:
- the end moment of the air gesture is determined according to the fuzzy entropy corresponding to each frame of the EMG signal.
- All the myoelectric signals included between the start moment of the air gesture and the end moment of the air gesture are determined as target myoelectric signals, all included acceleration signals are determined as target acceleration signals, and all included angle signals are determined as target angle signals.
- the time corresponding to the first frame of myoelectric signals in the Q frames of myoelectric signals is taken as the end time of the air gesture. All the myoelectric signals included between the start time of the air gesture and the end time of the air gesture are used as target myoelectric signals.
- the end moment of the gesture in the air refers to the starting point of the moment corresponding to the first frame of the myoelectric signal in which the fuzzy entropy of the Q frame is 0.
- the last signal in the last frame of myoelectric signal is the last target myoelectric signal.
- the fuzzy entropy corresponding to 10 consecutive frames of myoelectric signal is greater than the preset fuzzy entropy threshold, then the moment corresponding to the 11th frame of myoelectric signal is taken as The starting moment of the air gesture.
- the fuzzy entropy corresponding to the 51st frame of the myoelectric signal to the 70th frame of the myoelectric signal is updated to 0, and it can be determined that the moment corresponding to the 51st frame of the myoelectric signal is the end moment of the air gesture . It can be known that all the myoelectric signals included in the 11th frame to the 50th frame of the myoelectric signal can be used as the target myoelectric signal.
- the target acceleration signal and the target acceleration signal use the network model to determine the corresponding target air gesture action.
- the network model may be a recurrent neural network (recurrent neural network, RNN), a GRU (gated recurrent unit) network model or a long short term memory (long short term memory, LSTM) network model.
- RNN recurrent neural network
- GRU gated recurrent unit
- LSTM long short term memory
- the cyclic neural network model is a neural network that models sequence data, that is, the current output of a sequence is related to the previous output. Since the electromyographic signal is an indefinite long-term sequence signal, the time correlation between the front and rear signals is high, and the acceleration signal and angle signal also have the characteristics of indefinite length and time sequence during the gesture movement process. Therefore, this application can use the cyclic neural network model to analyze The data is classified.
- the network model can also use the GRU network model or the LSTM network model.
- the GRU network model and the LSTM network model are both An improved network model based on recurrent neural network RNN.
- the network model can also be other models, which can be set and modified as needed, and this embodiment of the present application does not impose any limitation on this.
- the above S112 may include the following S1121 to S1124.
- the target myoelectric signal includes the myoelectric signal corresponding to the start moment of the air gesture, and the subsequent M-1 frame myoelectric signal at the start moment of the air gesture; or, the target myoelectric signal includes the start moment of the air gesture All the EMG signals included between the moment when the air gesture terminated.
- the target acceleration signal determine the original data of the three axes of acceleration, the distribution of gravity on the three axes, and the modulus of the acceleration.
- the target acceleration signal includes the acceleration signal corresponding to the initial moment of the air gesture, and the M-1 frame acceleration signal following the initial moment of the air gesture; or, the target acceleration signal includes the initial moment of the air gesture and the acceleration signal of the air gesture All acceleration signals included between the termination instants.
- the target angle signal includes the angle signal corresponding to the start moment of the air gesture, and the subsequent M-1 angle signal at the start moment of the air gesture; or, the target angle signal includes the start moment of the air gesture and the end of the air gesture All angle signals included between moments.
- At least one of the average absolute value of the time-domain characteristics, the slope sign change value, and the waveform length, at least one of the three-axis acceleration raw data, the distribution of gravity on the three axes, and the acceleration modulus, and three gyroscopes At least one of the multi-axis raw data, the absolute value multiplied by the multi-axis raw data of the gyroscope, and the rotation matrix is used to determine the corresponding target air gesture action by using the network model.
- the absolute value multiplied by the multi-axis raw data of the gyroscope is, for example, the absolute value multiplied by the three-axis raw data of the gyroscope.
- the air gesture action may include: at least one of: opening the palm and waving to the left or right, opening the palm and waving upward or downward, clenching a fist and waving upward or downward, clenching a fist and stretching the index finger upward or downward.
- the air gesture action may also include other actions, which may be specifically set and changed according to needs, which is not limited in this embodiment of the present application. It should be understood that the target air gesture action is one of the above air gesture actions.
- the target operation instruction includes the type and adjustment range of the target operation instruction.
- the type of the operation instruction may include: at least one of page sliding, volume adjustment, and video progress adjustment.
- the type of the operation instruction may also include other items, which may be modified and set as required, and this embodiment of the present application does not impose any limitation on this.
- the above S113 may include the following S1131 to S1132.
- the type of the corresponding target operation instruction is determined from a preset operation instruction library, the operation instruction library includes various air gesture actions and the type and adjustment range of the operation instruction corresponding to each air gesture action.
- the target air gesture action is an air gesture action in a preset operation command library.
- all angle signals included in the continuous M frames of angle signals are target angle signals, or, all angle signals from the start moment of the air gesture to the end moment of the air gesture are is the target angle signal.
- angle signal sequence can also be divided into sliding windows with the first sliding window length N, and the time corresponding to each frame of the angle signal in the divided multi-frame angle signals is aligned one-to-one with the time corresponding to each frame of the electromyography signal.
- the change range of the target angle signal can be determined according to the change of the target angle signal in the angle signal, and then, combined with the change range of the target angle signal and the determined type of the target operation command, the range of the target operation command can be determined. Adjustment range.
- the type of the operation instruction is used to indicate what the air gesture action wants to do
- the adjustment range of the operation instruction is used to indicate: based on what the air gesture action wants to do, determine how much it wants to do. In this way, the accuracy of the operation corresponding to the air gesture action can be improved.
- variation range of the target angle can also be determined according to the target angle signal, and then the adjustment range of the target operation command can be calculated according to the variation range of the target angle, that is, how much you want to drag the video or audio progress bar to the left .
- the change range of the target angle signal is ⁇ .
- the embodiment of the present application provides an air gesture recognition technology, which first determines the start time of the air gesture based on the myoelectric signal and the acceleration signal, and then determines the start time of the air gesture based on the collected target muscles.
- the electric signal, target acceleration signal and target angle signal are used to determine the corresponding target gesture action in the air by using the network model. Therefore, by combining angles and increasing recognition conditions, the effect of gesture recognition can be effectively improved.
- FIG. 5 is a schematic flowchart of a method for inter-space gesture interaction provided by an embodiment of the present application.
- the air gesture interaction method 60 is applied to the air gesture interaction system provided in the embodiment of the present application.
- the air gesture interaction method 60 includes: S201 to S205.
- the first electronic device 11 detects a first operation performed by a user.
- the first operation refers to an operation for the user to instruct to perform air gesture interaction.
- the smart watch When the user clicks on the "space interaction function" option, the smart watch responds to the user's click operation and starts to invoke the space interaction function provided by the embodiment of the present application.
- the program corresponding to the gesture recognition method 40 enables the space interaction function.
- the first operation is a click operation.
- the first operation may also be other operations such as voice, which is not limited in this embodiment of the present application.
- the first electronic device 11 sends a target operation instruction to the second electronic device 12.
- the target operation instruction includes the type and adjustment range of the target operation instruction.
- the second electronic device 12 After the second electronic device 12 receives the target operation instruction, the second electronic device 12 performs a second operation according to the target operation instruction, and the second operation refers to an operation indicated by the target air gesture action.
- the mobile phone can take screenshots, slide screens, switch applications, adjust volume, adjust video or audio progress bars, and the like.
- taking screenshots, sliding the screen and how much to slide, switching applications and what applications to switch to, volume adjustment and how much to adjust, video or audio progress bar adjustment and how much to adjust, etc. are the second operations, that is to say, these are different air gestures.
- the TV can perform page switching, pause, volume adjustment, video or audio progress bar adjustment, and the like.
- switching pages and how much to switch, pause and where to pause, volume adjustment and how much to adjust, video or audio progress bar adjustment and how much to adjust, etc. are the second operations, that is to say, these are indicated by different air gestures Actions performed by the TV.
- the vehicle center console can make a call, adjust the volume, open or close an application, and the like.
- making a call and which phone to call, adjusting the volume and how much to adjust, opening or closing the application, etc. are the second operations, that is to say, these are the operations performed by the vehicle center console instructed by different air gestures.
- the type of each target operation instruction is only used to indicate to perform one type of operation, and the adjustment range of the target operation instruction can control how much the second operation performs.
- the adjustment range of the target operation instruction can control how much the second operation performs.
- An embodiment of the present application provides an air gesture interaction method.
- the first electronic device uses the air gesture recognition method provided above to accurately determine the target air gesture action and the target operation instruction corresponding to the target air gesture action. Then, the first electronic device sends the target operation instruction to the second electronic device, and the second electronic device that receives the target operation instruction can perform the second operation, so as to realize the user's control of the second electronic device through the target air gesture action.
- the accuracy of gesture recognition is improved, the accuracy of control is improved accordingly, and the user's interactive experience is also improved accordingly.
- Fig. 6 shows a schematic structural diagram of an electronic device provided by the present application. It should be understood that the electronic device 100 may be the first electronic device 11 or the second electronic device 12 provided in the above embodiments. The electronic device 100 may be used to implement the air gesture recognition method and the air gesture interaction method described in the above method embodiments.
- the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2 , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and A subscriber identification module (subscriber identification module, SIM) card interface 195 and the like.
- SIM subscriber identification module
- the sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, bone conduction sensor 180M, etc.
- the processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
- application processor application processor, AP
- modem processor graphics processing unit
- GPU graphics processing unit
- image signal processor image signal processor
- ISP image signal processor
- controller video codec
- digital signal processor digital signal processor
- baseband processor baseband processor
- neural network processor neural-network processing unit
- the controller may be the nerve center and command center of the electronic device 100 .
- the controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
- a memory may also be provided in the processor 110 for storing instructions and data.
- the memory in processor 110 is a cache memory.
- the memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
- the processor 110 may run the software codes of the air gesture recognition method and the air gesture interaction method provided in the embodiments of the present application to realize the air interaction function.
- the processor 110 may execute S101 to S113 in the air gesture recognition method 40 provided in the embodiment of the present application.
- S201 to S204 in the air gesture interaction method 60 provided by the embodiment of the present application may also be executed.
- the processor 110 may execute S205 in the air gesture interaction method 60 provided in the embodiment of the present application.
- the wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
- Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals.
- Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas.
- Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network.
- the antenna may be used in conjunction with a tuning switch.
- the mobile communication module 150 may provide a wireless communication solution applied to the electronic device 100, such as at least one of the following solutions: a second generation (2th generation, 2G) mobile communication solution, a third generation (3th generation, 3G) Mobile communication solutions, fourth generation (4th generation, 5G) mobile communication solutions, fifth generation (5th generation, 5G), sixth generation (6th generation, 6G) mobile communication solutions.
- a wireless communication solution applied to the electronic device 100 such as at least one of the following solutions: a second generation (2th generation, 2G) mobile communication solution, a third generation (3th generation, 3G) Mobile communication solutions, fourth generation (4th generation, 5G) mobile communication solutions, fifth generation (5th generation, 5G), sixth generation (6th generation, 6G) mobile communication solutions.
- a modem processor may include a modulator and a demodulator.
- the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal.
- the demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing.
- the low-frequency baseband signal is passed to the application processor after being processed by the baseband processor.
- the application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 .
- the modem processor may be a stand-alone device.
- the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.
- the wireless communication module 160 can provide wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (Wireless Fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite, etc. applied on the electronic device 100.
- WLAN wireless local area networks
- System global navigation satellite system, GNSS
- frequency modulation frequency modulation, FM
- near field communication technology near field communication, NFC
- infrared technology infrared, IR
- the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology.
- the internal memory 121 may be used to store computer-executable program codes including instructions.
- the internal memory 121 may include an area for storing programs and an area for storing data.
- the internal memory 121 can also store the software codes of the air gesture recognition method and the air gesture interaction method provided in the embodiment of the present application. When the processor 110 runs the software code, the air gesture recognition method and the air gesture interaction method are executed. The process steps to realize the space interaction function.
- the software codes of the air gesture recognition method and the air gesture interaction method provided in the embodiment of the present application can also be stored in an external memory, and the processor 110 can run the software code through the external memory interface 120 to perform air gesture recognition.
- the method and the process steps of the air gesture interaction method realize the air interaction function.
- the gyro sensor 180B can be used to determine the motion posture of the electronic device 100 .
- the angular velocity of the electronic device 100 around three axes may be determined by the gyro sensor 180B.
- the gyro sensor 180B can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180B detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake.
- the gyro sensor 180B can also be used for navigation and somatosensory game scenes.
- the acceleration sensor 180E can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
- the keys 190 include a power key, a volume key and the like.
- the key 190 may be a mechanical key. It can also be a touch button.
- the electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
- the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 .
- the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components.
- the illustrated components can be realized in hardware, software or a combination of software and hardware.
- FIG. 7 is a schematic diagram of an air gesture interaction system provided by an embodiment of the present application.
- the air gesture interaction system 200 includes a first air gesture interaction device 210 and a second air gesture interaction device 220 .
- the air gesture interaction system 200 can implement the aforementioned air gesture interaction method;
- the first air gesture interaction device 210 includes: an acquisition unit 211 , a first processing unit 212 and a sending unit 213 .
- the second air gesture interaction device 220 includes: a receiving unit 221 and a second processing unit 222 .
- the acquiring unit 211 is configured to detect a first user operation.
- the first operation refers to an operation for the user to instruct to perform air gesture interaction.
- the first processing unit 212 is configured to synchronously collect the user's myoelectric signal and the acceleration signal and angle signal generated by the first air gesture interaction device 210 in response to the first operation.
- the first processing unit 212 is further configured to determine a target air gesture action and a target operation instruction corresponding to the target air gesture action according to the collected myoelectric signal, acceleration signal, and angle signal.
- the sending unit 213 is configured to send a target operation instruction to the second air gesture interaction device 220 .
- the receiving unit 221 is configured to receive the target operation instruction sent by the first space gesture interaction device 210 .
- the second processing unit 222 is configured to perform a second operation according to the target operation instruction.
- the second operation refers to the operation indicated by the target air gesture action.
- first space gesture interaction device 210 and the second space gesture interaction device 220 are embodied in the form of functional units.
- unit here may be implemented in the form of software and/or hardware, which is not specifically limited.
- a "unit” may be a software program, a hardware circuit or a combination of both to realize the above functions.
- the hardware circuitry may include application specific integrated circuits (ASICs), electronic circuits, processors (such as shared processors, dedicated processors, or group processors) for executing one or more software or firmware programs. etc.) and memory, incorporating logic, and/or other suitable components to support the described functionality.
- ASICs application specific integrated circuits
- processors such as shared processors, dedicated processors, or group processors for executing one or more software or firmware programs. etc.
- memory incorporating logic, and/or other suitable components to support the described functionality.
- the units of each example described in the embodiments of the present application can be realized by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.
- the embodiment of the present application also provides a computer-readable storage medium, where computer instructions are stored in the computer-readable storage medium;
- the interaction device executes the aforementioned air gesture interaction method.
- the computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website, computer, server, or data center Transmission to another website site, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.).
- the computer-readable storage medium may be any available medium that can be accessed by a computer, or may be a data storage device including one or more servers, data centers, etc. that can be integrated with the medium.
- the available medium may be a magnetic medium (for example, a floppy disk, a hard disk, or a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk (solid state disk, SSD)) and the like.
- the embodiment of the present application also provides a computer program product including computer instructions, which, when run on an air gesture interaction device, enables the air gesture interaction device to execute the aforementioned air gesture interaction method.
- FIG. 8 is a schematic structural diagram of a chip provided by an embodiment of the present application.
- the chip shown in FIG. 8 may be a general-purpose processor or a special-purpose processor.
- the chip includes a processor 301 .
- the processor 301 is configured to support the air gesture interaction device to execute the technical solution shown in FIG. 5 .
- the chip further includes a transceiver 302, and the transceiver 302 is used to accept the control of the processor 301, and is used to support the air gesture interaction device to execute the technical solution shown in FIG. 5 .
- the chip shown in FIG. 8 may further include: a storage medium 303 .
- the chip shown in Figure 8 can be implemented using the following circuits or devices: one or more field programmable gate arrays (field programmable gate array, FPGA), programmable logic device (programmable logic device, PLD) , controllers, state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.
- field programmable gate array field programmable gate array, FPGA
- programmable logic device programmable logic device
- controllers state machines, gate logic, discrete hardware components, any other suitable circuitry, or any combination of circuitry capable of performing the various functions described throughout this application.
- the electronic equipment, air gesture interaction device, computer storage medium, computer program product, and chip provided by the above-mentioned embodiments of the present application are all used to execute the method provided above. Therefore, the beneficial effects that it can achieve can refer to the above-mentioned The beneficial effects corresponding to the provided method will not be repeated here.
- sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.
- presetting and predefining can be realized by pre-saving corresponding codes, tables or other methods that can be used to indicate related information in devices (for example, including electronic devices) , the present application does not limit its specific implementation.
Landscapes
- Engineering & Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Dermatology (AREA)
- General Health & Medical Sciences (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
本申请提供了一种隔空手势识别方法及其相关设备,涉及人机交互领域,该方法包括:同步采集用户的肌电信号、以及用户使用第一电子设备时,第一电子设备产生的加速度信号和角度信号;利用肌电信号、加速度信号和角度信号,对应分别生成肌电信号序列、加速度信号序列和角度信号序列;根据肌电信号序列、加速度信号序列和角度信号序列,确定用户对应的目标隔空手势动作;根据目标隔空手势动作和角度信号序列,确定对应的目标操作指令。该方法通过结合肌电信号、加速度信号和角度信号来共同确定隔空手势动作,从而提高隔空手势识别的效果。
Description
本申请要求于2021年10月13日提交国家知识产权局、申请号为202111194354.0、申请名称为“隔空手势识别方法及其相关设备”的中国专利申请的优先权,其全部内容通过引用结合在本申请中。
本申请涉及人机交互领域,尤其涉及一种隔空手势识别方法及其相关设备。
传统的人机交互设备主要有键盘、鼠标、手写板、触摸屏、游戏控制器等,这些设备利用使用者的手部运动来实现人机交互的功能。相对于此,隔空手势交互支持更多更自然的交互方式,提供了以人为中心,而不是以设备为中心的交互技术,从而使用户注意力集中在本来该做的事情以及内容上,而不是集中在设备上。
其中,隔空手势交互过程中最重要的技术是隔空手势识别。常见的隔空手势识别技术主要分为基于机器视觉的隔空手势识别技术和基于肌电信号的隔空手势识别技术。目前,基于机器视觉的隔空手势识别技术主要依赖于摄像头、深度传感器等采集手势信息,然后利用模型对人手部分对应的图像进行分割,从而实现手势检测和识别。这种方式对环境、视野的要求比较高,识别能力非常有限。而基于肌电信号进行手势交互所能使用的传感器种类又比较少,功能非常不完善。
为此,亟待一种新的隔空手势识别方法,能有效提高手势识别的效果。
发明内容
本申请提供一种隔空手势识别方法及其相关设备,通过结合肌电信号、加速度信号和角度信号来共同确定隔空手势动作,从而提高隔空手势识别的效果。
为达到上述目的,本申请采用如下技术方案:
第一方面,提供一种隔空手势识别方法,应用于用户使用的第一电子设备,所述第一电子设备与第二电子设备进行通信连接,该方法包括:
同步采集所述用户的肌电信号、以及所述用户使用所述第一电子设备时,所述第一电子设备产生的加速度信号和角度信号;利用所述肌电信号、所述加速度信号和所述角度信号,对应分别生成肌电信号序列、加速度信号序列和角度信号序列;根据所述肌电信号序列、所述加速度信号序列和所述角度信号序列,确定所述用户对应的目标隔空手势动作;根据所述目标隔空手势动作和所述角度信号序列,确定对应的目标操作指令,所述目标操作指令包括所述目标操作指令的类型和调节幅度。
本申请实施例提供了一种隔空手势识别技术,不仅采集肌电信号和加速度信号,还增加采集角度信号;然后,基于肌电信号、加速度信号和角度信号三者来确定出对应的目标隔空手势动作。由于结合了角度信号,增加手势动作的识别条件,从而可以有效提高手势识别的效果。
在第一方面一种可能的实现方式中,根据所述肌电信号序列、所述加速度序列和所述角度信号序列,确定所述用户对应的目标隔空手势动作,包括:根据所述肌电信号序列和所述加速度信号序列,确定隔空手势起始时刻;从所述隔空手势起始时刻开始,确定目标肌电信号、目标加速度信号和目标角度信号;根据所述目标肌电信号、所述目标加速度信号和所述目标角度信号,利用网络模型,确定所述用户对应的所述目标隔空手势动作。
在该实现方式中,先基于肌电信号和加速度信号,确定隔空手势起始时刻,然后,从确定出隔空手势起始时刻之后,再基于采集的目标肌电信号、目标加速度信号和目标角度信号,利用网络模型确定出对应的目标隔空手势动作。由此,通过结合角度,增加识别条件,从而可以有效提高手势识别的效果。
在第一方面一种可能的实现方式中,根据所述肌电信号序列和所述加速度信号序列,确定隔空手势起始时刻,包括:对所述肌电信号序列进行滑窗分帧,确定每帧肌电信号对应的模糊熵;判断第s帧肌电信号至第s+M-1帧肌电信号分别对应的模糊熵是否均大于预设模糊熵阈值,以及第s帧加速度信号对应的加速度模值是否大于预设加速度模值,s、M均为大于0的整数;若是,则将所述第s帧肌电信号对应的时刻作为所述隔空手势起始时刻。
在该实现方式中,通过设定预设模糊熵阈值对肌电信号的混乱程度进行区分,然后再结合加速度信号的加速度模值大小,来共同作为隔空手势起始时刻的判断条件,从而可以提高确定出的隔空手势起始时刻的准确度,便于后续对隔空手势动作进行精准识别。
在第一方面一种可能的实现方式中,对所述肌电信号序列进行滑窗分帧,确定每帧肌电信号对应的模糊熵,包括:根据第一滑窗长度,将所述肌电信号序列划分成多帧肌电信号;根据第二滑窗长度,将所述多帧肌电信号中的每帧肌电信号划分为k个肌电信号子序列,其中,所述第一滑窗长度为N,所述第二滑窗长度为m,k=N-m+1,1≤m<N;针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的第一平均值;将所述第二滑窗长度更新为m+1,针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的第二平均值;根据所述第一平均值和所述第二平均值,确定所述每帧肌电信号对应的模糊熵。
在该实现方式中,基于上述方法,即可评价肌电信号所对应的波形前后部分之间的混乱程度。
在第一方面一种可能的实现方式中,针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个所述肌电信号子序列分别对应的模糊隶属度的第一平均值或第二平均值,包括:针对所述每帧肌电信号,根据距离公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间的距离;根据所述距离,利用模糊隶属度公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间分别对应的模糊隶属度;根据所述模糊隶属度,利用模糊隶属度平均公式,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的所述第一平均值或所述第二平均值。
在第一方面一种可能的实现方式中,所述方法还包括:当所述第s帧肌电信号至 所述第s+M-1帧肌电信号分别对应的模糊熵小于或等于所述预设模糊熵阈值时,将所述肌电信号对应的模糊熵更新为0。
在该实现方式中,可对不满足条件的肌电信号对应的模糊熵进行整流,使得满足条件的肌电信号的模糊熵和不满足条件的肌电信号的模糊熵差异更大,进而使得满足条件的肌电信号对应的混乱度更显著,对后续处理更有利。
在第一方面一种可能的实现方式中,所述方法还包括:当所述第s帧肌电信号至所述第s+M-1帧肌电信号分别对应的模糊熵不是均大于所述预设模糊熵阈值时;或者,当所述第s帧加速度信号对应的加速度模值小于或等于所述预设加速度模值时,则判断第s+1帧肌电信号至第s+M帧肌电信号分别对应的模糊熵是否均大于所述预设模糊熵阈值,以及第s+1帧加速度信号对应的加速度模值是否大于所述预设加速度模值。
在该实现方式中,当肌电信号和加速度信号未满足确定隔空手势起始时刻的条件时,可按照肌电信号序列和加速度信号序列继续判断后续信号是否满足预设条件。
在第一方面一种可能的实现方式中,从所述隔空手势起始时刻开始,确定目标肌电信号、目标加速度信号和目标角度信号,包括:从所述隔空手势起始时刻开始,将M帧肌电信号包括的所有肌电信号均作为所述目标肌电信号,M帧加速度信号包括的所有加速度信号均作为所述目标加速度信号,M帧角度信号包括的所有角度信号均作为所述目标角度信号,或者;从所述隔空手势起始时刻开始,根据每帧肌电信号对应的模糊熵,确定隔空手势终止时刻;将所述隔空手势起始时刻与所述隔空手势终止时刻之间包括的所有肌电信号确定为所述目标肌电信号,包括的所有加速度信号确定为所述目标加速度信号,包括的所有角度信号确定为所述目标角度信号。
在该实现方式中,可以通过预设帧数的方式或者通过确定隔空手势起始时刻和隔空手势终止时刻的方式,来确定有效信号,以便于后续确定隔空手势动作。
在第一方面一种可能的实现方式中,根据所述目标肌电信号、所述目标加速度信号和所述目标角度信号,利用网络模型,确定所述用户对应的所述目标隔空手势动作,包括:根据所述目标肌电信号,确定时域特征平均绝对值、斜率符号变化值和波形长度;根据所述目标加速度信号,确定加速度三轴原始数据、重力在三轴的分布大小和所述加速度模值;根据所述目标角度信号,确定陀螺仪三轴原始数据、陀螺仪多轴原始数据相乘的绝对值和旋转矩阵;根据所述时域特征平均绝对值、所述斜率符号变化值和所述波形长度中的至少一项,所述加速度三轴原始数据、所述重力在三轴的分布大小和所述加速度模值中的至少一项,以及所述陀螺仪三轴原始数据、所述陀螺仪多轴原始数据相乘的绝对值和所述旋转矩阵中的至少一项,利用所述网络模型,确定对应的所述目标隔空手势动作。
在该实现方式中,通过结合有效的目标肌电信号、目标加速度信号和目标角度信号中的多个数据,可以提高确定出的隔空手势动作的准确度。
在第一方面一种可能的实现方式中,所述网络模型为循环神经网络模型、GRU网络模型或LSTM网络模型。
在第一方面一种可能的实现方式中,所述隔空手势动作包括:手掌张开向左或向右挥动、手掌张开向上或向下挥动、握拳向上或向下挥动、握拳伸食指向上或向下挥动中的至少一项。
在第一方面一种可能的实现方式中,根据所述目标隔空手势动作和所述角度信号序列,确定对应的目标操作指令,包括:确定所述目标隔空手势动作对应的目标操作指令的类型,所述操作指令的类型包括滑动页面、音量调节、视频进度调节中的至少一项;根据所述目标操作指令的类型和所述角度信号序列中的所述目标角度信号,确定所述目标操作指令的调节幅度。
在该实现方式中,操作指令的类型用于表示隔空手势动作想要做什么,操作指令的调节幅度用于表示:基于隔空手势动作想要做什么的情况下,确定想要做多少。由此,通过确定目标操作指令的类型和调节幅度,可以提高后续隔空手势动作控制第二电子设备时对应操作的精准度。
在第一方面一种可能的实现方式中,所述方法还包括:向所述第二电子设备发送所述目标操作指令。
第二方面,提供了一种隔空手势交互装置,该装置包括用于执行以上第一方面或第一方面的任意可能的实现方式中各个步骤的单元。
第三方面,提供了一种电子设备,包括:肌电电极、加速度传感器、陀螺仪传感器、处理器和存储器;所述肌电电极用于采集用户的肌电信号;所述加速度传感器用于在所述用户使用所述电子设备时产生加速度信号;所述陀螺仪传感器用于在所述用户使用所述电子设备时产生角度信号;所述存储器,用于存储可在所述处理器上运行的计算机程序;所述处理器,用于执行如第一方面或第一方面的任意可能的实现方式中提供的隔空手势识别方法中进行处理的步骤。
第四方面,提供了一种芯片,包括:处理器,用于从存储器中调用并运行计算机程序,使得安装有芯片的设备执行如第一方面或第一方面的任意可能的实现方式中提供的隔空手势识别方法。
第五方面,提供了一种计算机可读存储介质,计算机可读存储介质存储有计算机程序,计算机程序包括程序指令,程序指令当被处理器执行时,使处理器执行如第一方面或第一方面的任意可能的实现方式中提供的隔空手势识别方法。
第六方面,提供了一种计算机程序产品,计算机程序产品包括存储了计算机程序的计算机可读存储介质,计算机程序使得计算机执行如第一方面或第一方面的任意可能的实现方式中提供的隔空手势识别方法。
图1为本申请实施例提供的隔空手势交互系统适用的场景图;
图2为本申请实施例提供的一种隔空手势识别方法的流程示意图;
图3为本申请实施例提供的一种对肌电信号序列进行分帧,并确定每帧肌电信号对应的模糊熵的示意图;
图4本申请实施例提供的一种模糊隶属度表格;
图5为本申请实施例提供的一种隔空手势交互方法的流程示意图;
图6为本申请实施例提供的电子设备的结构示意图;
图7为本申请实施例提供的一种隔空手势识别系统的结构示意图;
图8为本申请实施例提供的一种芯片的结构示意图。
下面将结合附图,对本申请中的技术方案进行描述。
在本申请实施例的描述中,除非另有说明,“/”表示或的意思,例如,A/B可以表示A或B;本文中的“和/或”仅仅是一种描述关联对象的关联关系,表示可以存在三种关系,例如,A和/或B,可以表示:单独存在A,同时存在A和B,单独存在B这三种情况。另外,在本申请实施例的描述中,“多个”是指两个或多于两个。
以下,术语“第一”、“第二”仅用于描述目的,而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此,限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多个该特征。在本实施例的描述中,除非另有说明,“多个”的含义是两个或两个以上。
常见的隔空手势识别技术主要分为基于机器视觉的隔空手势识别技术和基于肌电信号的隔空手势识别技术。
示例性的,基于机器视觉的隔空手势识别技术,主要通过机器视觉对设置在电视上的摄像头所采集到的手势图像序列处理识别,从而实现手部和电视的交互。或者,基于机器视觉的隔空手势识别技术,通过机器视觉对设置在车载中控台上的摄像头所采集的手势图像序列处理识别,从而实现手部和车载中控台的交互。
这种方法通过使用摄像头采集手势信息,然后,例如利用肤色模型对人手部分进行分割,从而实现手势检测和识别,最后再使用帧间差法实现运动手势的跟踪。该方法的效果取决于肤色模型的准确率,然而人的皮肤颜色不一,难以得到通用、高效的肤色模型;而且,当人手运动速度不均匀时,采用帧间差法跟踪手势会出现中断现象,从而丢失被跟踪手势。此外,这种方式对采集环境、视野的要求也比较高,识别能力非常有限。
示例性的,基于肌电信号的隔空手势识别技术,通过用户带在手臂上的可穿戴设备来采集不同手势对应的肌电信号并进行手势识别处理,再传输给与可穿戴设备连接的电子设备,即可实现手部与电子设备的交互。或者,基于肌电信号的隔空手势识别技术,通过用户带在手指上的指环来采集不同手势对应的肌电信号并进行手势识别处理,再传输给与指环连接的电子设备,即可实现手部与电子设备的交互。但是,现有基于肌电信号隔空交互所能使用的传感器种类较少,功能也不完善,而且仅通过肌电信号进行手势的识别通常也不够准确。
除了上述两种技术之外,现有技术还提供了一种基于肌电信号和加速度信号结合起来进行手势识别的技术。该隔空手势识别技术通过采集肌电信号和加速度信号,然后,利用支持向量机(support vector machines,SVM)进行机器学习来实现手势的识别。虽然相对于上述两种方式,该隔空手势识别技术可以弥补受环境、视场角限制等缺陷,让用户摆脱范围限制,相对提高识别能力,但是由于支持向量机做决策时,仅仅依赖的是有限个样本,所以其识别效果还是不够准确,不能完全满足用户需求。
有鉴于此,本申请实施例提供了一种隔空手势识别技术,不仅采集肌电信号和加速度信号,还增加采集角度信号;然后,基于肌电信号、加速度信号和角度信号三者来确定出对应的目标隔空手势动作。由于结合了角度信号,增加手势动作的识别条件,从而可以有效提高手势识别的效果。
参考图1,图1为一例适用于本申请实施例的隔空手势交互系统适用的场景图。如图1所示,本申请实施例提供的隔空手势交互系统包括:第一电子设备11和第二电子设备12。第一电子设备11以用户使用的智能手表为例,第二电子设备12以用户使用的平板电脑为例。当然,图1所示的隔空手势交互系统中还可以包括其他更多的电子设备,本申请实施例对此不作限制。
在本申请实施例中,第一电子设备11可以包括:智能手表、智能手环、智能指环、手指或手臂上的可穿戴电子设备等,本申请实施例对此并不限定。
第二电子设备12可以包括:智能手机、智能电视、大屏设备、平板电脑、个人数字助理(personal digital assistant,PDA)、电脑手持式通信设备、音箱、车载设备(也可称为车机)、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer,UMPC)、手持计算机、上网本、可穿戴电子设备、虚拟现实设备等IOT(internet of things,物联网)设备、5G网络中的电子设备等,本申请实施例对此并不限定。
应理解,图2中所示的例子不应该对本申请实施例产生任何的限制。
第一电子设备11和第二电子设备12之间可以通过通信网络互联。示例性的,该通信网络可以是有线网络,也可以是无线网络。例如,上述通信网络可以是局域网(local area networks,LAN),也可以是广域网(wide area networks,WAN),例如互联网。上述通信网络可使用任何已知的网络通信协议来实现,上述网络通信协议可以是各种有线或无线通信协议,诸如以太网、通用串行总线(universal serial bus,USB)、火线(FIREWIRE)、全球移动通讯系统(global system for mobile communications,GSM)、通用分组无线服务(general packet radio service,GPRS)、码分多址接入(code division multiple access,CDMA)、宽带码分多址(wideband code division multiple access,WCDMA),时分码分多址(time-division code division multiple access,TD-SCDMA),长期演进(long term evolution,LTE)、蓝牙、无线保真(wireless fidelity,Wi-Fi)、Wi-Fi直连(wireless fidelity peer-to-peer,Wi-Fi P2P)连接、NFC、基于互联网协议的语音通话(voice over Internet protocol,VoIP)、支持网络切片架构的通信协议或任何其他合适的通信协议。
也就是说,隔空手势交互系统内的各个电子设备可按照一定的通信协议和组网策略组建网络(即组网),使得隔空手势交互系统内的各个电子设备之间可以互相通信。
可选地,在本申请中,第一电子设备11需要将第一电子设备11采集的数据传输给第二电子设备12,而第一电子设备11的数据涉及到用户隐私问题。因此,在第一电子设备11向第二电子设备12发送数据之前,第一电子设备11和第二电子设备12之间可建立可信关系并以此为前提,也即第一电子设备11和第二电子设备12之间建立了设备级的认证关系。
在本申请中,第一电子设备11和第二电子设备12之间可以利用各种不同的方式建立设备级的认证关系,本申请实施例对此不作限制。
在一种可能的应用场景中,用户手腕上佩戴有智能手表,用户通过挥舞手部,想要对平板电脑进行操作,以使得平板电脑上的应用等可以隔空受到用户手部动作的控制,例如,用户进行某一隔空手势动作后,平板电脑进行了截屏处理。在这个过程中,可以使用本申请提供的隔空手势识别方法以及隔空手势交互方法。
在另一种可能的应用场景中,用户手指上佩戴有智能指环,用户通过挥舞手部,想要对电视进行操作,以使得电视可以隔空受到用户手部动作的控制。例如,用户进行某一隔空手势动作后,电视进行了切换页面处理。在这个过程中,可以使用本申请提供的隔空手势识别方法以及隔空手势交互方法。
在又一种可能的应用场景中,用户手臂上佩戴有可穿戴设备,用户通过挥舞手部,想要对车载中控台进行操作,以使得车载中控台可以隔空受到用户手部动作的控制。例如,用户进行某一隔空手势动作后,车载中控台进行了拨打电话的处理。在这个过程中,可以使用本申请提供的隔空手势识别方法以及隔空手势交互方法。
下面以图1所示的场景为例进行说明,第一电子设备11为智能手表,第二电子设备12为平板电脑,智能手表和平板电脑已通过通信网络进行了连接。在该场景中,用户携带智能手表后,通过进行各种隔空手势动作就可以隔空控制平板电脑实现不同的处理。
在该过程中,第一电子设备11上可以适用本申请实施例提供的隔空手势识别方法,第一电子设备11和第二电子设备12之间可以适用本申请实施例提供的隔空手势交互方法。
首先,先对本申请实施例提供的隔空手势识别方法进行详细介绍。图2所示为本申请实施例提供的一种隔空手势识别方法40的流程示意图。该隔空手势识别方法40应用于智能手表。
如图2所示,该隔空手势识别方法40包括:S101至S113。
S101、同步采集用户的肌电信号、以及用户使用第一电子设备11时产生的加速度信号和角度信号。
S102、利用采集的肌电信号对应生成肌电信号序列,利用采集的加速度信号对应生成加速度信号序列、利用采集的角度信号对应生成角度信号序列。
应理解,该用户即为使用第一电子设备11的用户,因此,第一电子设备11可采集其使用者手部进行动作时产生的肌电信号、加速度信号和角度信号。
肌电信号是一种与神经肌肉活动相关的生物电信号。肌电信号能反应肌肉的收缩模式以及收缩强度等信息,不同的肢体动作对应不同的肌电信号,通过分析肌电信号可以判别出该肌电信号对应的具体动作。示例性的,可以在智能手表上安装肌电电极,通过肌电电极与人体皮肤接触,从而可以利用肌电电极连续采集用户手部动作时产生的多个肌电信号,生成对应的肌电信号序列。
加速度信号指的是用于表示加速度的信号。加速度信号可以由加速度传感器产生。该加速度传感器可以是单轴加速度传感器、双轴加速度传感器或者三轴加速度传感器。示例性的,若智能手表包括三轴加速度传感器,那么,当用户佩戴智能手表的手部进行挥舞时,智能手表中的三轴加速度传感器将根据手部的运动状态产生多个加速度信号,采集该多个加速度信号,生成对应的加速度信号序列。
角度信号指的是用于表示物体移动角度的信号。角度信号可以由陀螺仪传感器产生。该陀螺仪传感器可以是二自由度陀螺仪或者三自由度陀螺仪。示例性的,若智能手表包括三自由度陀螺仪,那么,当用户佩戴智能手表的手部进行挥舞时,智能手表 中的三自由度陀螺仪将根据手部的运动状态产生多个角度信号,采集该多个角度信号,生成对应的角度信号序列。
同步采集也即是说是同时采集的,且采集的肌电信号、加速度信号和角度信号的次序一致。例如,在同一时间,采集到的肌电信号、加速度信号和角度信号均为各自信号序列中的第1帧信号,也就是说,采集到的肌电信号为第1帧肌电信号,加速度信号为第1帧加速度信号,角度信号为第1帧角度信号。此处,例如可以在采集时通过打点的方式记录次序,使得肌电信号、加速度信号和角度信号三者的次序对应一致。
应理解,当用户手部在隔空进行挥舞,变换各种隔空手势动作时,手势的每一次改变都伴随有肌电信号、加速度信号和角度信号的产生,因此,只有同步采集才能提高采集的信号的准确度,进而提高后续确定出的隔空手势动作的准确度。
S103、对肌电信号序列进行滑窗分帧,确定每帧肌电信号对应的模糊熵。
其中,每帧肌电信号包括1个或多个肌电信号。
应理解,滑窗分帧指的是:针对连续的信号序列,按照一定长度将其分成较短的帧,帧长即为滑窗长度,并且,为了使得两帧信号之间的参数能够平稳过渡,通常相邻两帧信号之间互相有部分重叠。例如,设滑窗长度的wlen,则每帧信号的帧长为wlen。若后一帧信号与前一帧信号的位移量为inc,则该两帧信号的重叠部分为overlap=wlen-inc。
在上述S103之前,由于肌电信号的幅度小、信噪比低,因此,在对肌电信号序列进行滑窗分帧之前,还可以先对肌电信号进行预处理,对肌电信号进行的预处理可以包括:对肌电信号进行信号放大、工频滤波、高通滤波等处理,具体可以根据需要进行设置和更改,本申请实施例对此不进行任何限制。
可选地,作为一种可能实现的方式,当利用多个肌电电极采集到多个通道的肌电信号时,可以对不同肌电电极采集的多个通道的肌电信号进行模糊熵计算。当然,在这之前,可以对多个肌电电极采集到的多个通道的肌电信号均进行预处理。
应理解,肌电信号序列包括连续的多个肌电信号。对此,可以预先设定滑窗的长度,利用该滑窗对肌电信号序列进行分帧,然后,再利用以下模糊熵计算公式,确定出每帧肌电信号对应的模糊熵。
图3示出了一种对肌电信号序列进行分帧,并确定每帧肌电信号对应的模糊熵的示意图。如图3所示,对肌电信号序列进行分帧,并确定每帧肌电信号对应的模糊熵的过程如下:
第一步:对肌电信号序列进行分帧。
示例性的,以N毫秒为第一滑窗长度,对肌电信号序列进行滑窗分帧,将肌电信号序列划分为多个长度为N毫秒的肌电信号片段。其中,每段肌电信号片段对应为1帧肌电信号,每段肌电信号片段对应的起始时刻为该帧肌电信号对应的时刻。
应理解,滑窗分帧后,相邻两帧肌电信号具有交叠。
例如,如图3中的(a)所示,第t时刻对应第1帧肌电信号,第t+1时刻对应的第2帧肌电信号,...,第t+M-1时刻对应第M帧肌电信号。其中,第t时刻对应的第1帧肌电信号与第t+1时刻对应的第2帧肌电信号具有交叠,其他依次类推,在此不再赘述。
第二步:如图3中的(b)所示,以m毫秒为第二滑窗长度,将长度为N毫秒的每帧肌电信号{xi,i=1,2,...,N}按序列顺序分成k=N-m+1个肌电信号子序列。其中,1≤m<N。
由此,肌电信号子序列表示为:Xi(t)={xi(t),xi+1(t),...,xi+m-1(t)}。t表示时间。
应理解,Xi(t)={xi(t),xi+1(t),...,xi+m-1(t)}表示从第i组信号中连续m个信号构成的m维向量。Xj(t)={xj(t),xj+1(t),...,xj+m-1(t)}表示从第j组信号中连续m个信号构成的m维向量。
应理解,第一步和第二步中涉及的滑窗长度均可以根据需要进行设置和修改,本申请实施例对此不进行任何限制。
第三步:根据以下距离公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间分别对应的距离:
d
ij(t)=maxX
i+p(t)-X
j+p(t),p={0,1,...,m-1}
其中,距离dij(t)指的是两向量对应元素差值的绝对值的最大值。
第四步:根据距离dij(t),利用以下模糊隶属度公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间分别对应的模糊隶属度。
其中,n和r为给定参数,r为相似性容限。
示例性的,图4为一种模糊隶属度表格。如图4所示,可以计算得到每个肌电信号子序列与所有k个肌电信号子序列之间分别对应的模糊隶属度。
第五步:根据上述得到的多个模糊隶属度,利用以下模糊隶属度平均公式,确定第二滑窗长度为m毫秒时,每个肌电信号子序列与其他k-1个肌电信号子序列之间分别对应的模糊隶属度的第一平均值。
第六步:将第二滑窗长度增长为m+1毫秒,重复上述第二步至第四步,确定第二滑窗长度为m+1毫秒时,每个肌电信号子序列与其他k-1个肌电信号子序列之间分别对应的模糊隶属度的第二平均值。
第七步:根据第二滑窗长度为m毫秒时,第五步获取到的第一平均值,以及根据第二滑窗长度为m+1毫秒时,第六步获取到的第二平均值,利用以下模糊熵公式,确定长度为N毫秒的每帧肌电信号所对应的模糊熵。
其中,FuzzyEn(t,N)即用于表示长度为N的1帧肌电信号所对应的模糊熵。该模糊熵的大小与t、N的取值均有关。
例如,当t=1时,表示第1时刻对应的1帧肌电信号的模糊熵为FuzzyEn(1,N)。 当t=2时,表示第2时刻对应的1帧肌电信号的模糊熵为FuzzyEn(2,N)。当t=M时,表示第M时刻对应的1帧肌电信号的模糊熵为FuzzyEn(M,N)。
S104、判断第s帧肌电信号对应的模糊熵是否大于预设模糊熵阈值。
S105、当第s帧肌电信号对应的模糊熵小于或等于预设模糊熵阈值时,将该第s帧肌电信号对应的模糊熵更新为0。也就是说,第s帧肌电信号对应的模糊熵将变更为0。
S106、当第s帧肌电信号对应的模糊熵大于预设模糊熵阈值时,保留该第s帧肌电信号对应的模糊熵。
其中,s为大于0的整数。预设模糊熵阈值可以根据需要进行设置和调整,本申请实施例对此不作限制。
应理解,基于上述第一步至第七步确定出的模糊熵,即可评价肌电信号所对应的波形前后部分之间的混乱程度,也可以理解为,前后波形之间的重复性,也就是频率。当模糊熵越大,说明波形中各个频率越多,越混乱。当模糊熵越小,说明波形中各个频率越小,越不混乱。因此,可以通过设定预设模糊熵阈值对混乱程度进行区分。
应理解,将小于或等于预设模糊熵阈值的1帧肌电信号的模糊熵更新为0时,相当于对肌电信号对应的模糊熵进行了整流,使得满足条件的肌电信号的模糊熵和不满足条件的肌电信号的模糊熵差异更大,进而使得满足条件的肌电信号对应的混乱度更显著,对后续处理更有利。
S107、当第s帧肌电信号对应的模糊熵被保留时,继续判断第s帧肌电信号后续连续M-1帧肌电信号分别对应的模糊熵是否均大于预设模糊熵阈值。
应理解,当某一帧肌电信号对应的模糊熵大于预设模糊熵阈值时,则保留该帧肌电信号对应的模糊熵;当某一帧肌电信号对应的模糊熵小于或等于预设模糊熵阈值时,则将该帧肌电信号对应的模糊熵更新为0。
应理解,M用于表示肌电信号的帧数,M的大小可以根据需要进行设置和更改。当M设定的越小,说明从被保留模糊熵的第s帧信号开始,期望将一小部分数量的肌电信号的模糊熵与预设模糊熵阈值进行比较;当M设定的越大,则说明从被保留模糊熵的第s帧信号开始,期望将更多数量的肌电信号的模糊熵与预设模糊熵阈值进行比较。当M设定的太小时,可参考的数据量较小,后续确定隔空手势起始时刻不一定准确;当M设定的太大时,计算量会比较大,因此,一般会根据经验设定一个合适大小的值。
示例性的,预设M的大小为9,则判断到第s帧肌电信号对应的模糊熵大于预设模糊熵阈值之后,确定后续第s+1帧肌电信号至第s+8帧肌电信号分别对应的模糊熵是否均大于预设模糊熵阈值。若第s+1帧肌电信号至第s+8帧肌电信号分别对应的模糊熵均大于预设模糊熵阈值,则保留第s+1帧肌电信号至第s+8帧肌电信号分别对应的模糊熵。
S108、确定第s帧加速度信号对应的加速度模值。
应理解,加速度信号序列可以以第一滑窗长度N进行滑窗分帧,划分出的多帧加速度信号中的每帧加速度信号对应的时刻与每帧肌电信号对应的时刻一一对齐。
示例性的,加速度模值即为第s帧加速度信号在直角坐标系xyz中,x轴上的分量、 在y轴上的分量以及在z轴上的分量的平方和的根值。加速度模值用于表示加速度的大小。
S109、判断第s帧加速度信号对应的加速度模值是否大于预设加速度模值。
应理解,预设加速度模值可以根据需要进行设置和调整,本申请实施例对此不作限制。为了避免手部的一些非必要动作对后续判断进行干扰,因此,可以通过设定阈值的方式先对加速度信号进行判断和筛选。
S110、当第s帧加速度信号对应的加速度模值大于预设加速度模值时,且同步采集的第s帧肌电信号后续连续还有M-1帧肌电信号对应的模糊熵都被保留时,也即,都大于或等于预设模糊熵阈值时,则该第s帧肌电信号对应的时刻为隔空手势起始时刻。
应理解,隔空手势起始时刻指的是第s帧肌电信号对应的时刻起始点。也即,第s帧加速度信号对应的时刻起始点。
当第s帧加速度信号对应的加速度模值小于或等于预设加速度模值时,或者,当第s帧肌电信号后续没有连续M-1个肌电信号对应的模糊熵都被保留时,则在s的基础上增加1,重复上述S104至S108,判断第s+1帧肌电信号至第s+M帧肌电信号对应的模糊熵是否均大于预设模糊熵阈值,以及第s+1帧加速度信号对应的加速度模值是否大于预设加速度模值,后续依次类推,直至确定出隔空手势起始时刻。
S111、从隔空手势起始时刻开始,将M帧肌电信号包括的所有肌电信号均作为目标肌电信号、M帧加速度信号包括的所有加速度信号均作为目标加速度信号、M帧角度信号包括的所有角度信号均作为目标角度信号。
其中,例如,M帧肌电信号指的是包括隔空手势起始时刻对应的第s帧肌电信号,以及后面模糊熵都被保留的M-1帧肌电信号,也就是说,M帧肌电信号包括第s帧肌电信号至第s+M-1帧肌电信号。
应理解,若第s帧肌电信号对应的时刻为隔空手势起始时刻,则第s帧肌电信号中的第1个信号为第1个目标肌电信号。
或者,上述S111还可以为:
从隔空手势起始时刻开始,根据每帧肌电信号对应的模糊熵,确定隔空手势终止时刻。
将隔空手势起始时刻与隔空手势终止时刻之间包括的所有肌电信号确定为目标肌电信号、包括的所有加速度信号确定为目标加速度信号,包括的所有角度信号确定为目标角度信号。
应理解,在确定出隔空手势起始时刻之后,可以判断是否出现连续Q帧肌电信号的模糊熵都为0,来判断隔空手势终止时刻。
例如,若连续有Q帧肌电信号的模糊熵都为0,则该Q帧肌电信号中第1帧肌电信号对应的时刻作为隔空手势终止时刻。隔空手势起始时刻至隔空手势终止时刻之间包括的所有肌电信号均作为目标肌电信号。
应理解,隔空手势终止时刻指的是:Q帧模糊熵都为0的肌电信号中的第1帧肌电信号对应的时刻起始点。
应理解,若某一帧肌电信号对应的时刻为隔空手势终止时刻,则上一帧肌电信号 中的最后1个信号为最后1个目标肌电信号。
例如,从第11帧肌电信号开始,包括第11帧肌电信号一共连续有10帧肌电信号对应的模糊熵都大于预设模糊熵阈值,则将第11帧肌电信号对应的时刻作为隔空手势起始时刻。而从隔空手势起始时刻之后,第51帧肌电信号至第70帧肌电信号对应的模糊熵都更新为0,则可以确定第51帧肌电信号对应的时刻为隔空手势终止时刻。由此可知,第11帧肌电信号至第50帧肌电信号所包括的所有肌电信将均可作为目标肌电信号。
S112、根据目标肌电信号、目标加速度信号和目标加速度信号,利用网络模型,确定对应的目标隔空手势动作。
其中,该网络模型可以为循环神经网络模型(recurrent neural network,RNN)、GRU(gated recurrent unit)网络模型或长短期记忆(long short term memory,LSTM)网络模型。
应理解,循环神经网络模型是一种对序列数据建模的神经网络,即一个序列当前的输出与前面的输出有关。由于肌电信号是不定长时间序列信号,前后信号时间相关性较高,同时手势运动过程中,加速度信号和角度信号也具有不定长、时序的特点,因此,本申请可以采用循环神经网络模型对数据进行分类。
在此基础上,考虑到传统的循环神经网络模型在处理长期依赖时可能会出现问题,为此,该网络模型也可以采用GRU网络模型或LSTM网络模型,GRU网络模型和LSTM网络模型都是一种基于循环神经网络RNN进行改进的网络模型。
当然,该网络模型也可以为其他模型,具体可以根据需要进行设置和修改,本申请实施例对此不进行任何限制。
可选地,上述S112可以包括以下S1121至S1124。
S1121、根据目标肌电信号,确定时域特征平均绝对值、斜率符号变化值和波形长度。
应理解,目标肌电信号包括隔空手势起始时刻对应的肌电信号,以及隔空手势起始时刻后续的M-1帧肌电信号;或者,目标肌电信号包括隔空手势起始时刻与隔空手势终止时刻之间包括的所有肌电信号。
S1122、根据目标加速度信号,确定加速度三轴原始数据、重力在三轴的分布大小和加速度模值。
应理解,目标加速度信号包括隔空手势起始时刻对应的加速度信号,以及隔空手势起始时刻后续的M-1帧加速度信号;或者,目标加速度信号包括隔空手势起始时刻与隔空手势终止时刻之间包括的所有加速度信号。
S1123、根据目标角度信号,确定陀螺仪三轴原始数据、陀螺仪多轴原始数据相乘的绝对值和旋转矩阵。
应理解,目标角度信号包括隔空手势起始时刻对应的角度信号,以及隔空手势起始时刻后续的M-1角度信号;或者,目标角度信号包括隔空手势起始时刻与隔空手势终止时刻之间包括的所有角度信号。
S1124、根据时域特征平均绝对值、斜率符号变化值和波形长度中的至少一项,加速度三轴原始数据、重力在三轴的分布大小和加速度模值中的至少一项,以及陀螺仪 三轴原始数据、陀螺仪多轴原始数据相乘的绝对值和旋转矩阵中的至少一项,利用网络模型,确定对应的目标隔空手势动作。
应理解,陀螺仪多轴原始数据相乘的绝对值例如为:陀螺仪三轴原始数据相乘的绝对值。
其中,隔空手势动作可以包括:手掌张开向左或向右挥动、手掌张开向上或向下挥动、握拳向上或向下挥动、握拳伸食指向上或向下挥动中的至少一项。
当然,隔空手势动作还可以包括其他动作,具体可以根据需要进行设置和改变,本申请实施例对此不进行任何限制。应理解,目标隔空手势动作为上述隔空手势动作中的一种。
S113、根据目标隔空手势动作和目标角度信号,确定对应的目标操作指令。
其中,目标操作指令包括目标操作指令的类型和调节幅度。
可选地,操作指令的类型可以包括:滑动页面、音量调节、视频进度调节中的至少一项。当然,操作指令的类型还可以包括其他项,具体可以根据需要进行修改和设置,本申请实施例对此不进行任何限制。
可选地,上述S113可以包括以下S1131至S1132。
S1131、根据确定出的目标隔空手势动作,确定目标隔空手势动作对应的目标操作指令的类型。
例如,从预设的操作指令库中确定对应的目标操作指令的类型,操作指令库包括多种隔空手势动作以及每种隔空手势动作对应的操作指令的类型和调节幅度。其中,目标隔空手势动作为预设的操作指令库中的一个隔空手势动作。
S1132、根据目标操作指令的类型和角度信号序列中的目标角度信号,确定目标操作指令的调节幅度。
例如,从隔空手势起始时刻开始,连续M帧角度信号所包括的所有角度信号均为目标角度信号,或者,从隔空手势起始时刻至隔空手势终止时刻之间的所有角度信号均为目标角度信号。
应理解,角度信号序列也可以以第一滑窗长度N进行滑窗分帧,划分出的多帧角度信号中的每帧角度信号对应的时刻与每帧肌电信号对应的时刻一一对齐。
应理解,可以根据角度信号中的目标角度信号的变化,确定出目标角度信号的变化范围,然后,再结合目标角度信号的变化范围和已确定出的目标操作指令的类型,确定目标操作指令的调节幅度。
应理解,操作指令的类型用于表示隔空手势动作想要做什么,操作指令的调节幅度用于表示:基于隔空手势动作想要做什么的情况下,确定想要做多少。由此,可以提高隔空手势动作对应操作的精准度。
示例性的,假设操作指令库中预先存储有“手掌张开向左挥动”和“手掌张开向右挥动”两个动作,以及“手掌张开向左挥动”对应的目标操作指令的类型为“向左滑动视频或音频进度条”,“手掌张开向右挥动”对应的目标操作指令的类型为“向右滑动视频或音频进度条”。
由此,若在识别过程中,确定出用户当前的目标隔空手势动作为“手掌张开向左挥动”,则从预设的操作指令库中可以确定出对应的目标操作指令的类型为“向左滑动视 频或音频进度条”。
此外,还可以根据目标角度信号,确定出目标角度的变化范围,进而可以根据目标角度的变化范围计算出目标操作指令的调节幅度,也即,想要将视频或音频进度条向左拖动多少。
例如,设挥动前视频或音频进度条上对应的时长位置为S1,若用户进行了“手掌张开向左挥动”的动作之后,目标角度信号的变化范围为θ。
那么,结合目标操作指令的类型和目标角度信号的变化范围,根据公式:S2=S1*(1-θ/90),可以确定出用户挥动后,视频或音频进度条上对应的时长位置为S2,进而确定出目标操作指令的调节幅度为“从S1至S2”。由此,说明“手掌张开向左挥动”这个指令对应的目标操作指令为:向左滑动视频或音频进度条,并且,调节幅度为:将视频或音频进度条上对应的时长位置从S1拖动至S2处。
本申请实施例提供了一种隔空手势识别技术,先基于肌电信号和加速度信号,确定隔空手势起始时刻,然后,从确定出隔空手势起始时刻之后,再基于采集的目标肌电信号、目标加速度信号和目标角度信号,利用网络模型确定出对应的目标隔空手势动作。由此,通过结合角度,增加识别条件,从而可以有效提高手势识别的效果。
接下来对本申请实施例提供的隔空手势交互方法进行详细介绍。图5所示为本申请实施例提供的一种隔空手势交互方法的流程示意图。该隔空手势交互方法60应用于本申请实施例提供的隔空手势交互系统中。
如图5所示,该隔空手势交互方法60包括:S201至S205。
S201、第一电子设备11检测到用户进行的第一操作。该第一操作是指用户用于指示进行隔空手势交互的操作。
S202、响应于用户的第一操作,开启隔空手势交互功能,第一电子设备11同步采集肌电信号、加速度信号和角度信号。
示例性的,智能手表的显示界面上显示有多个应用选项,其中,当用户点击“隔空交互功能”选项时,智能手表响应于用户的点击操作,开始调用本申请实施例提供的隔空手势识别方法40对应的程序,开启隔空交互功能。其中,第一操作即为点击操作,当然,第一操作也可以为语音等其他操作,本申请实施例对此不作限制。
S203、利用如图2所示的隔空手势识别方法40对用户所进行的隔空手势动作进行识别,确定出目标隔空手势动作,以及目标隔空手势动作对应的目标操作指令和指令调节范围。
具体过程可以参考上述对图2中各个步骤的描述,在此不再赘述。
S204、第一电子设备11向第二电子设备12发送目标操作指令。
其中,目标操作指令包括目标操作指令的类型和调节幅度。
S205、当第二电子设备12接收目标操作指令之后,第二电子设备12根据目标操作指令进行第二操作,该第二操作是指目标隔空手势动作指示进行的操作。
示例性的,若第二电子设备12为手机,根据目标操作指令,手机可以进行截屏、滑动屏幕、切换应用、音量调节、视频或音频进度条调节等。其中,该截屏、滑动屏幕以及滑动多少、切换应用以及切换为什么应用、音量调节以及调节多少、视频或音 频进度条调节以及调节多少等即为第二操作,也就是说,这些为不同隔空手势动作所指示手机进行的操作。
示例性的,若第二电子设备12为电视,根据目标操作指令,电视可以进行切换页面、暂停、音量调节、视频或音频进度条调节等。其中,该切换页面以及切换多少、暂停以及暂停什么位置、音量调节以及调节多少、视频或音频进度条调节以及调节多少等即为第二操作,也就是说,这些为不同隔空手势动作所指示电视进行的操作。
示例性的,若第二电子设备12为车载中控台,根据目标操作指令,车载中控台可以拨打电话、音量调节、打开或关闭应用等。其中,该拨打电话以及拨打哪一个电话、音量调节以及调节多少、打开或关闭应用等即为第二操作,也就是说,这些为不同隔空手势动作所指示车载中控台进行的操作。
应理解,通常每个目标操作指令的类型仅用于指示进行一类操作,而目标操作指令的调节幅度可以控制第二操作进行操作时做多少。由此,在隔空手势交互过程中,不仅可以实现对第二电子设备12的操作,还可以提高操作的精度,改善交互体验。
应理解,目标操作指令与第二操作的对应关系,可以根据需要进行设置和修改,本申请实施例对此不进行任何限制。
本申请实施例提供了一种隔空手势交互方法,通过第一电子设备利用上述提供的隔空手势识别方法,准确确定出目标隔空手势动作,以及目标隔空手势动作对应的目标操作指令,然后,第一电子设备将目标操作指令发送给第二电子设备,接收到目标操作指令的第二电子设备可以进行第二操作,以实现用户通过目标隔空手势动作对第二电子设备进行的控制的目的,并且,在该过程中,由于手势识别的准确度提高,从而控制的精准度随之提高,进而用户的交互体验也随之提升。
上文结合图1至图5,详细描述了本申请实施例的隔空手势识别方法、隔空手势交互方法,下面将结合图6至图8,详细描述本申请适用的电子设备的软件系统、装置以及芯片。应理解,本申请实施例中的软件系统、装置以及芯片可以执行前述本申请实施例的隔空手势识别方法、隔空手势交互方法,即以下各种产品的具体工作过程,可以参考前述方法实施例中的对应过程。
图6示出了本申请提供的一种电子设备的结构示意图。应理解,电子设备100可以为上述实施例提供的第一电子设备11或第二电子设备12。电子设备100可用于实现上述方法实施例中描述的隔空手势识别方法、隔空手势交互方法。
电子设备100可以包括处理器110,外部存储器接口120,内部存储器121,通用串行总线(universal serial bus,USB)接口130,充电管理模块140,电源管理模块141,电池142,天线1,天线2,移动通信模块150,无线通信模块160,音频模块170,扬声器170A,受话器170B,麦克风170C,耳机接口170D,传感器模块180,按键190,马达191,指示器192,摄像头193,显示屏194,以及用户标识模块(subscriber identification module,SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A,陀螺仪传感器180B,气压传感器180C,磁传感器180D,加速度传感器180E,距离传感器180F,接近光传感器180G,指纹传感器180H,温度传感器180J,触摸传感器180K,环境光传感器180L,骨传导传感器180M等。
处理器110可以包括一个或多个处理单元,例如:处理器110可以包括应用处理器(application processor,AP),调制解调处理器,图形处理器(graphics processing unit,GPU),图像信号处理器(image signal processor,ISP),控制器,视频编解码器,数字信号处理器(digital signal processor,DSP),基带处理器,和/或神经网络处理器(neural-network processing unit,NPU)等。其中,不同的处理单元可以是独立的器件,也可以集成在一个或多个处理器中。
其中,控制器可以是电子设备100的神经中枢和指挥中心。控制器可以根据指令操作码和时序信号,产生操作控制信号,完成取指令和执行指令的控制。
处理器110中还可以设置存储器,用于存储指令和数据。在一些实施例中,处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据,可从所述存储器中直接调用。避免了重复存取,减少了处理器110的等待时间,因而提高了系统的效率。
处理器110可以运行本申请实施例提供的隔空手势识别方法、隔空手势交互方法的软件代码,实现隔空交互功能。
示例性的,在本申请的实施例中,当该电子设备为上述第一电子设备11时,处理器110可以运行本申请实施例提供的隔空手势识别方法40中的S101至S113,此外,还可以运行本申请实施例提供的隔空手势交互方法60中的S201至S204。
当电子设备为上述第二电子设备12时,处理器110可以运行本申请实施例提供的隔空手势交互方法60中的S205。
电子设备100的无线通信功能可以通过天线1,天线2,移动通信模块150,无线通信模块160,调制解调处理器以及基带处理器等实现。
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用,以提高天线的利用率。例如:可以将天线1复用为无线局域网的分集天线。在另外一些实施例中,天线可以和调谐开关结合使用。
移动通信模块150可以提供应用在电子设备100上的无线通信的解决方案,例如下列方案中的至少一个:第二代(2th generation,2G)移动通信解决方案、第三代(3th generation,3G)移动通信解决方案、第四代(4th generation,5G)移动通信解决方案、第五代(5th generation,5G)、第六代(6th generation,6G)移动通信解决方案。
调制解调处理器可以包括调制器和解调器。其中,调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后,被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A,受话器170B等)输出声音信号,或通过显示屏194显示图像或视频。在一些实施例中,调制解调处理器可以是独立的器件。在另一些实施例中,调制解调处理器可以独立于处理器110,与移动通信模块150或其他功能模块设置在同一个器件中。
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wireless local area networks,WLAN)(如无线保真(wireless fidelity,Wi-Fi)网络),蓝牙(bluetooth,BT),全球导航卫星系统(global navigation satellite system,GNSS),调频(frequency modulation,FM),近距离无线通信技术(near field communication,NFC),红外技术(infrared,IR)等无线通信的解决方案。
在一些实施例中,电子设备100的天线1和移动通信模块150耦合,天线2和无线通信模块160耦合,使得电子设备100可以通过无线通信技术与网络以及其他设备通信。
内部存储器121可以用于存储计算机可执行程序代码,所述可执行程序代码包括指令。内部存储器121可以包括存储程序区和存储数据区。内部存储器121还可以存储本申请实施例提供的隔空手势识别方法、隔空手势交互方法的软件代码,当处理器110运行所述软件代码时,执行隔空手势识别方法、隔空手势交互方法的流程步骤,实现隔空交互功能。
当然,本申请实施例提供的隔空手势识别方法、隔空手势交互方法的软件代码也可以存储在外部存储器中,处理器110可以通过外部存储器接口120运行所述软件代码,执行隔空手势识别方法、隔空手势交互方法的流程步骤,实现隔空交互功能。
陀螺仪传感器180B可以用于确定电子设备100的运动姿态。在一些实施例中,可以通过陀螺仪传感器180B确定电子设备100围绕三个轴(即,x,y和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。示例性的,当按下快门,陀螺仪传感器180B检测电子设备100抖动的角度,根据角度计算出镜头模组需要补偿的距离,让镜头通过反向运动抵消电子设备100的抖动,实现防抖。陀螺仪传感器180B还可以用于导航,体感游戏场景。
加速度传感器180E可检测电子设备100在各个方向上(一般为三轴)加速度的大小。当电子设备100静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态,应用于横竖屏切换,计步器等应用。
按键190包括开机键,音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备100可以接收按键输入,产生与电子设备100的用户设置以及功能控制有关的键信号输入。
可以理解的是,本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中,电子设备100可以包括比图示更多或更少的部件,或者组合某些部件,或者拆分某些部件,或者不同的部件布置。图示的部件可以以硬件,软件或软件和硬件的组合实现。
下面介绍本申请实施例提供的一种用于实现上述隔空手势交互方法60的隔空手势交互系统。图7是本申请实施例提供的隔空手势交互系统的示意图。
如图7所示,隔空手势交互系统200包括第一隔空手势交互装置210和第二隔空手势交互装置220。
应理解,隔空手势交互系统200可以执行前述所示的隔空手势交互方法;第一隔空手势交互装置210包括:获取单元211、第一处理单元212和发送单元213。第二隔空手势交互装置220包括:接收单元221、第二处理单元222。
获取单元211,用于检测用户的第一操作。第一操作是指用户用于指示进行隔空手势交互的操作。
第一处理单元212,用于响应于第一操作,同步采集用户的肌电信号、以及第一隔空手势交互装置210产生的加速度信号和角度信号。
第一处理单元212,还用于根据采集的肌电信号、加速度信号和角度信号,确定目标隔空手势动作,以及目标隔空手势动作对应的目标操作指令。
发送单元213,用于向第二隔空手势交互装置220发送目标操作指令。
接收单元221,用于接收第一隔空手势交互装置210发送的目标操作指令。
第二处理单元222,用于根据目标操作指令进行第二操作。第二操作是指目标隔空手势动作指示进行的操作。
需要说明的是,上述第一隔空手势交互装置210、第二隔空手势交互装置220以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现,对此不作具体限定。
例如,“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit,ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。
因此,在本申请的实施例中描述的各示例的单元,能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行,取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能,但是这种实现不应认为超出本申请的范围。
本申请实施例还提供一种计算机可读存储介质,所述计算机可读存储介质中存储有计算机指令;当所述计算机可读存储介质在隔空手势交互装置上运行时,使得该隔空手势交互装置执行前述所示的隔空手势交互方法。
所述计算机指令可以存储在计算机可读存储介质中,或者从一个计算机可读存储介质向另一个计算机可读存储介质传输,例如,所述计算机指令可以从一个网站站点、计算机、服务器或者数据中心通过有线(例如同轴电缆、光纤、数字用户线(digital subscriber line,DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可以用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如,软盘、硬盘、磁带),光介质、或者半导体介质(例如固态硬盘(solid state disk,SSD))等。
本申请实施例还提供了一种包含计算机指令的计算机程序产品,当其在隔空手势交互装置上运行时,使得隔空手势交互装置可以执行前述所示的隔空手势交互方法。
图8为本申请实施例提供的一种芯片的结构示意图。图8所示的芯片可以为通用处理器,也可以为专用处理器。该芯片包括处理器301。其中,处理器301用于支持隔空手势交互装置执行图5所示的技术方案。
可选的,该芯片还包括收发器302,收发器302用于接受处理器301的控制,用于支持隔空手势交互装置执行图5所示的技术方案。
可选的,图8所示的芯片还可以包括:存储介质303。
需要说明的是,图8所示的芯片可以使用下述电路或者器件来实现:一个或多个 现场可编程门阵列(field programmable gate array,FPGA)、可编程逻辑器件(programmable logic device,PLD)、控制器、状态机、门逻辑、分立硬件部件、任何其他适合的电路、或者能够执行本申请通篇所描述的各种功能的电路的任意组合。
上述本申请实施例提供的电子设备、隔空手势交互装置、计算机存储介质、计算机程序产品、芯片均用于执行上文所提供的方法,因此,其所能达到的有益效果可参考上文所提供的方法对应的有益效果,在此不再赘述。
应理解,上述只是为了帮助本领域技术人员更好地理解本申请实施例,而非要限制本申请实施例的范围。本领域技术人员根据所给出的上述示例,显然可以进行各种等价的修改或变化,例如,上述检测方法的各个实施例中某些步骤可以是不必须的,或者可以新加入某些步骤等。或者上述任意两种或者任意多种实施例的组合。这样的修改、变化或者组合后的方案也落入本申请实施例的范围内。
还应理解,上文对本申请实施例的描述着重于强调各个实施例之间的不同之处,未提到的相同或相似之处可以互相参考,为了简洁,这里不再赘述。
还应理解,上述各过程的序号的大小并不意味着执行顺序的先后,各过程的执行顺序应以其功能和内在逻辑确定,而不应对本申请实施例的实施过程构成任何限定。
还应理解,本申请实施例中,“预先设定”、“预先定义”可以通过在设备(例如,包括电子设备)中预先保存相应的代码、表格或其他可用于指示相关信息的方式来实现,本申请对于其具体的实现方式不做限定。
还应理解,本申请实施例中的方式、情况、类别以及实施例的划分仅是为了描述的方便,不应构成特别的限定,各种方式、类别、情况以及实施例中的特征在不矛盾的情况下可以相结合。
还应理解,在本申请的各个实施例中,如果没有特殊说明以及逻辑冲突,不同的实施例之间的术语和/或描述具有一致性、且可以相互引用,不同的实施例中的技术特征根据其内在的逻辑关系可以组合形成新的实施例。
最后应说明的是:以上所述,仅为本申请的具体实施方式,但本申请的保护范围并不局限于此,任何在本申请揭露的技术范围内的变化或替换,都应涵盖在本申请的保护范围之内。因此,本申请的保护范围应以所述权利要求的保护范围为准。
Claims (17)
- 一种隔空手势识别方法,其特征在于,应用于用户使用的第一电子设备,所述第一电子设备与第二电子设备进行通信连接,该隔空手势识别方法包括:同步采集所述用户的肌电信号、以及所述用户使用所述第一电子设备时,所述第一电子设备产生的加速度信号和角度信号;利用所述肌电信号、所述加速度信号和所述角度信号,对应分别生成肌电信号序列、加速度信号序列和角度信号序列;根据所述肌电信号序列、所述加速度信号序列和所述角度信号序列,确定所述用户对应的目标隔空手势动作;根据所述目标隔空手势动作和所述角度信号序列,确定对应的目标操作指令,所述目标操作指令包括所述目标操作指令的类型和调节幅度。
- 根据权利要求1所述的隔空手势识别方法,其特征在于,根据所述肌电信号序列、所述加速度序列和所述角度信号序列,确定所述用户对应的目标隔空手势动作,包括:根据所述肌电信号序列和所述加速度信号序列,确定隔空手势起始时刻;从所述隔空手势起始时刻开始,确定目标肌电信号、目标加速度信号和目标角度信号;根据所述目标肌电信号、所述目标加速度信号和所述目标角度信号,利用网络模型,确定所述用户对应的所述目标隔空手势动作。
- 根据权利要求2所述的隔空手势识别方法,其特征在于,根据所述肌电信号序列和所述加速度信号序列,确定隔空手势起始时刻,包括:对所述肌电信号序列进行滑窗分帧,确定每帧肌电信号对应的模糊熵;判断第s帧肌电信号至第s+M-1帧肌电信号分别对应的模糊熵是否均大于预设模糊熵阈值,以及第s帧加速度信号对应的加速度模值是否大于预设加速度模值,s、M均为大于0的整数;若是,则将所述第s帧肌电信号对应的时刻作为所述隔空手势起始时刻。
- 根据权利要求3所述的隔空手势识别方法,其特征在于,对所述肌电信号序列进行滑窗分帧,确定每帧肌电信号对应的模糊熵,包括:根据第一滑窗长度,将所述肌电信号序列划分成多帧肌电信号;根据第二滑窗长度,将所述多帧肌电信号中的每帧肌电信号划分为k个肌电信号子序列,其中,所述第一滑窗长度为N,所述第二滑窗长度为m,k=N-m+1,1≤m<N;针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的第一平均值;将所述第二滑窗长度更新为m+1,针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的第二平均值;根据所述第一平均值和所述第二平均值,确定所述每帧肌电信号对应的模糊熵。
- 根据权利要求4所述的隔空手势识别方法,其特征在于,针对所述每帧肌电信号,确定每个肌电信号子序列与其他k-1个所述肌电信号子序列分别对应的模糊隶属 度的第一平均值或第二平均值,包括:针对所述每帧肌电信号,根据距离公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间的距离;根据所述距离,利用模糊隶属度公式,确定每个肌电信号子序列与所有k个肌电信号子序列之间分别对应的模糊隶属度;根据所述模糊隶属度,利用模糊隶属度平均公式,确定每个肌电信号子序列与其他k-1个肌电信号子序列分别对应的模糊隶属度的所述第一平均值或所述第二平均值。
- 根据权利要求3至5中任一项所述的隔空手势识别方法,其特征在于,所述方法还包括:当所述第s帧肌电信号至所述第s+M-1帧肌电信号分别对应的模糊熵小于或等于所述预设模糊熵阈值时,将所述肌电信号对应的模糊熵更新为0。
- 根据权利要求3至6中任一项所述的隔空手势识别方法,其特征在于,所述方法还包括:当所述第s帧肌电信号至所述第s+M-1帧肌电信号分别对应的模糊熵不是均大于所述预设模糊熵阈值时;或者,当所述第s帧加速度信号对应的加速度模值小于或等于所述预设加速度模值时,则判断第s+1帧肌电信号至第s+M帧肌电信号分别对应的模糊熵是否均大于所述预设模糊熵阈值,以及第s+1帧加速度信号对应的加速度模值是否大于所述预设加速度模值。
- 根据权利要求1至7中任一项所述的隔空手势识别方法,其特征在于,从所述隔空手势起始时刻开始,确定目标肌电信号、目标加速度信号和目标角度信号,包括:从所述隔空手势起始时刻开始,将M帧肌电信号包括的所有肌电信号均作为所述目标肌电信号,M帧加速度信号包括的所有加速度信号均作为所述目标加速度信号,M帧角度信号包括的所有角度信号均作为所述目标角度信号,或者;从所述隔空手势起始时刻开始,根据每帧肌电信号对应的模糊熵,确定隔空手势终止时刻;将所述隔空手势起始时刻与所述隔空手势终止时刻之间包括的所有肌电信号确定为所述目标肌电信号,包括的所有加速度信号确定为所述目标加速度信号,包括的所有角度信号确定为所述目标角度信号。
- 根据权利要求2至8中任一项所述的隔空手势识别方法,其特征在于,根据所述目标肌电信号、所述目标加速度信号和所述目标角度信号,利用网络模型,确定所述用户对应的所述目标隔空手势动作,包括:根据所述目标肌电信号,确定时域特征平均绝对值、斜率符号变化值和波形长度;根据所述目标加速度信号,确定加速度三轴原始数据、重力在三轴的分布大小和所述加速度模值;根据所述目标角度信号,确定陀螺仪三轴原始数据、陀螺仪多轴原始数据相乘的绝对值和旋转矩阵;根据所述时域特征平均绝对值、所述斜率符号变化值和所述波形长度中的至少一项,所述加速度三轴原始数据、所述重力在三轴的分布大小和所述加速度模值中的至 少一项,以及所述陀螺仪三轴原始数据、所述陀螺仪多轴原始数据相乘的绝对值和所述旋转矩阵中的至少一项,利用所述网络模型,确定对应的所述目标隔空手势动作。
- 根据权利要求2或9所述的隔空手势识别方法,其特征在于,所述网络模型为循环神经网络模型、GRU网络模型或LSTM网络模型。
- 根据权利要求1至10中任一项所述的隔空手势识别方法,其特征在于,所述隔空手势动作包括:手掌张开向左或向右挥动、手掌张开向上或向下挥动、握拳向上或向下挥动、握拳伸食指向上或向下挥动中的至少一项。
- 根据权利要求2至11中任一项所述的隔空手势识别方法,其特征在于,根据所述目标隔空手势动作和所述角度信号序列,确定对应的目标操作指令,包括:确定所述目标隔空手势动作对应的目标操作指令的类型,所述操作指令的类型包括滑动页面、音量调节、视频进度调节中的至少一项;根据所述目标操作指令的类型和所述角度信号序列中的所述目标角度信号,确定所述目标操作指令的调节幅度。
- 根据权利要求1至12中任一项所述的隔空手势识别方法,其特征在于,所述方法还包括:向所述第二电子设备发送所述目标操作指令。
- 一种电子设备,其特征在于,包括:肌电电极、加速度传感器、陀螺仪传感器、处理器和存储器;所述肌电电极用于采集用户的肌电信号;所述加速度传感器用于在所述用户使用所述电子设备时产生加速度信号;所述陀螺仪传感器用于在所述用户使用所述电子设备时产生角度信号;所述存储器,用于存储可在所述处理器上运行的计算机程序;所述处理器,用于执行如权利要求1至13中任一项所述的隔空手势识别方法中进行处理的步骤。
- 一种芯片,其特征在于,包括:处理器,用于从存储器中调用并运行计算机程序,使得安装有所述芯片的设备执行如权利要求1至13中任一项所述的隔空手势识别方法。
- 一种计算机可读存储介质,其特征在于,所述计算机可读存储介质存储有计算机程序,所述计算机程序包括程序指令,所述程序指令当被处理器执行时,使所述处理器执行如权利要求1至13中任一项所述的隔空手势识别方法。
- 一种计算机程序产品,其特征在于,所述计算机程序产品包括存储了计算机程序的计算机可读存储介质,所述计算机程序使得计算机执行如权利要求1至13中任一项所述的隔空手势识别方法。
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202111194354.0A CN114995628B (zh) | 2021-10-13 | 2021-10-13 | 隔空手势识别方法及其相关设备 |
CN202111194354.0 | 2021-10-13 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023061037A1 true WO2023061037A1 (zh) | 2023-04-20 |
Family
ID=83018877
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/CN2022/113425 WO2023061037A1 (zh) | 2021-10-13 | 2022-08-18 | 隔空手势识别方法及其相关设备 |
Country Status (2)
Country | Link |
---|---|
CN (1) | CN114995628B (zh) |
WO (1) | WO2023061037A1 (zh) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN115562472B (zh) * | 2022-02-11 | 2023-09-22 | 荣耀终端有限公司 | 手势交互方法、介质及电子设备 |
CN116449967A (zh) * | 2023-06-20 | 2023-07-18 | 浙江强脑科技有限公司 | 仿生手教具及其控制方法、主控设备 |
CN117075742B (zh) * | 2023-10-17 | 2024-01-30 | 深圳市魔样科技有限公司 | 基于手势识别的智能戒指控制方法 |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030132974A1 (en) * | 2002-01-15 | 2003-07-17 | International Business Machines Corporation | Free-space gesture recognition for transaction security and command processing |
CN103645804A (zh) * | 2013-12-18 | 2014-03-19 | 三星电子(中国)研发中心 | 识别人体手势的方法、装置以及使用其的手表 |
CN103853333A (zh) * | 2014-03-21 | 2014-06-11 | 上海威璞电子科技有限公司 | 一种手势控制玩具的方案 |
CN104536558A (zh) * | 2014-10-29 | 2015-04-22 | 三星电子(中国)研发中心 | 一种智能指环和控制智能设备的方法 |
Family Cites Families (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN103019378B (zh) * | 2012-12-07 | 2016-09-07 | 无锡清华信息科学与技术国家实验室物联网技术中心 | 一种移动电子设备手势控制交互方法、装置及移动终端 |
CN104111733B (zh) * | 2014-07-29 | 2017-03-08 | 上海交通大学 | 一种手势识别系统及方法 |
CN106294485B (zh) * | 2015-06-05 | 2019-11-01 | 华为技术有限公司 | 确定显著地点的方法及装置 |
CN109192007A (zh) * | 2018-09-21 | 2019-01-11 | 杭州电子科技大学 | 一种基于肌电运动感知的ar手语识别方法及教学方法 |
-
2021
- 2021-10-13 CN CN202111194354.0A patent/CN114995628B/zh active Active
-
2022
- 2022-08-18 WO PCT/CN2022/113425 patent/WO2023061037A1/zh active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030132974A1 (en) * | 2002-01-15 | 2003-07-17 | International Business Machines Corporation | Free-space gesture recognition for transaction security and command processing |
CN103645804A (zh) * | 2013-12-18 | 2014-03-19 | 三星电子(中国)研发中心 | 识别人体手势的方法、装置以及使用其的手表 |
CN103853333A (zh) * | 2014-03-21 | 2014-06-11 | 上海威璞电子科技有限公司 | 一种手势控制玩具的方案 |
CN104536558A (zh) * | 2014-10-29 | 2015-04-22 | 三星电子(中国)研发中心 | 一种智能指环和控制智能设备的方法 |
Non-Patent Citations (1)
Title |
---|
CHEN, WEITING ET AL.: "Characterization of Surface EMG Signal Based on Fuzzy Entropy", IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, vol. 15, no. 2, 30 June 2007 (2007-06-30), pages 266 - 269, XP011348034, DOI: 10.1109/TNSRE.2007.897025 * |
Also Published As
Publication number | Publication date |
---|---|
CN114995628B (zh) | 2023-08-11 |
CN114995628A (zh) | 2022-09-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2023061037A1 (zh) | 隔空手势识别方法及其相关设备 | |
Wang et al. | Joint activity recognition and indoor localization with WiFi fingerprints | |
Ling et al. | Ultragesture: Fine-grained gesture sensing and recognition | |
EP3965003A1 (en) | Image processing method and device | |
CN104679246B (zh) | 一种交互界面中人手漫游控制的穿戴式设备及控制方法 | |
Lu et al. | A hand gesture recognition framework and wearable gesture-based interaction prototype for mobile devices | |
US11383166B2 (en) | Interaction method of application scene, mobile terminal, and storage medium | |
US10061389B2 (en) | Gesture recognition system and gesture recognition method | |
WO2021115210A1 (zh) | 一种触控区域调整方法及装置 | |
US10521018B2 (en) | Human body-based interaction method and interaction apparatus | |
CN108989672B (zh) | 一种拍摄方法及移动终端 | |
WO2022179376A1 (zh) | 手势控制方法与装置、电子设备及存储介质 | |
WO2023051411A1 (zh) | 一种识别触摸操作的方法及电子设备 | |
KR20150088599A (ko) | 이동 단말기 및 이의 제어방법 | |
US11249179B2 (en) | Motion detection system and motion detection device | |
CN111387978B (zh) | 一种表面肌电信号的动作段检测方法、装置、设备及介质 | |
JPWO2016024424A1 (ja) | 情報処理装置、情報処理方法、及び情報処理システム | |
Costante et al. | Personalizing a smartwatch-based gesture interface with transfer learning | |
CN117130469B (zh) | 一种隔空手势识别方法、电子设备及芯片系统 | |
WO2021170062A1 (zh) | 隔空手势的调节方法及终端 | |
WO2019024718A1 (zh) | 防伪处理方法、防伪处理装置及电子设备 | |
WO2017134732A1 (ja) | 入力装置、入力支援方法および入力支援プログラム | |
US20170212590A1 (en) | User action activated voice recognition | |
JP7341324B2 (ja) | 目標ユーザのロック方法および電子デバイス | |
Jhang et al. | Sensor based dynamic hand gesture recognition by PairNet |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22879984 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 22879984 Country of ref document: EP Kind code of ref document: A1 |