RU201260U1 - A device for recognizing gestures by detecting and analyzing muscle activity by electromyography - Google Patents

Abstract

The utility model relates to a device for gesture recognition by detecting and analyzing muscle activity by electromyography. The technical result is to improve the accuracy and reliability of gesture recognition. The device is made in the form of an elastic band or bracelet, worn on the forearm, which contains a block for converting analog signals from electromyographic sensors, EMG sensors to a digital code, inertial sensors, a power supply, a control module for external devices connected to a microcontroller that processes data, programmed by a type of built-in neural network, at the output of which a control signal is generated by external objects. The device additionally contains a module for determining a reference electrode - an EMG sensor having the lowest signal level, resistance sensors are used as EMG sensors, and electrical signals of muscle contractions between the reference electrode and the electrodes of the EMG sensors are recorded. 2 ill.

Description

The utility model relates to input-output devices for user interaction with a computer, and in particular to a device for recognizing gestures, a wearable human-machine interface (HMI, English Human-machine interface, HMI), based on the registration of electrical signals from muscle contraction, by surface electromyography.

Surface electromyography (EMG) is a method of recording electrical signals generated during muscle contractions between a grounded electrode and an EMG sensor. The use of this method in special wearable devices allows the HMI to be implemented in the form of a gesture interface, or myointerface, to replace the usual mechanical input-output devices, such as a computer mouse, touch panel, remote control, and control various objects without physically touching them. ... Such objects can be prostheses, rehabilitation simulators, a telephone, a toy, an aircraft.

One of the variants of this design can be an HMI device worn on the arm in the form of a bandage or bracelet, capable of reading electrical signals from muscles, analyzing muscle activity and recognizing hand gestures, converting them into control signals for various devices and technical objects.

Muscle electrical signals represent an analog signal in the form of a rather chaotic sequence of pulses of small amplitude, up to millivolts, and cannot be directly used as an input signal for HMI devices.

One of the limitations of detecting the incoming signal from muscle contraction is that EMG sensors are very sensitive to changes in operating conditions. The signals generated by these sensors can be influenced by sweating of the skin, the amount of hair and fat in the skin, the effects of activity from other muscles, movement of the skin through the muscles, electromagnetic interference in the environment, etc.

Two types of EMG sensors are used to register surface EMG signals: resistive EMG sensors and capacitive EMG sensors. The electrode of a resistive EMG sensor typically contains a plate of electrically conductive material that is in direct physical contact with the user's skin, while the electrode of a capacitive EMG sensor typically contains a plate of electrically conductive material that is electrically isolated from the user's skin by at least one thin intermediate a layer of dielectric material.

There are documents in the art that describe a wearable HMI device including improved capacitive EMG sensors.

US 20 190 033 967 discloses the advantage of capacitive EMG sensors.

The proposed improved capacitive EMG sensors provide increased reliability due to an additional dielectric layer based on ceramics, which covers the sensor electrode plates. This dielectric layer acts as a barrier to protect the sensor electrode plates from moisture on the skin surface, and at the same time, beneficially contributes to increasing the capacitance between the EMG sensor electrode and the user's body, while still being sensitive to moisture. The capacitive EMG sensor necessarily additionally includes a ground electrode, the surface of which is not covered with a dielectric layer, this also differs from the resistive EMG sensor.

Capacitive coupling provides a relatively high resistance between the skin and the sensor, and this can complicate the circuitry needed to amplify the detected EMG signals, making it more expensive.

US patent application US 20 180 344 195 discloses a wearable EMG device, which is an HMI and includes such capacitive EMG sensors that combine elements of traditional capacitive and resistive EMG sensors. For example, capacitive EMG sensors are described that are adapted to resistively couple to the user's skin. In this case, the resistive connection between the sensor electrode and the consumer's skin is galvanically isolated from the electrical circuits of the sensor by a capacitor included in the electrical circuit connecting the sensor electrodes with the amplifier. The combination of resistively coupled skin electrode and capacitor provides the corresponding advantages of traditional resistive and capacitive EMG sensor designs while mitigating the corresponding disadvantages of each.

Skin resistive coupling provided by a resistive EMG sensor provides a relatively low impedance between the skin and the sensor compared to capacitive coupling, and this can greatly simplify the circuitry required to amplify detected EMG signals. However, since the resistive coupling is essentially galvanic, it can supply the amplification circuit with a parasitic DC voltage applied to the user's skin. Capacitive gain sensors do not have this effect, since they galvanically isolate the amplification circuit from the skin. Both of these effects potentially affect the quality of the EMG signals detected. Obviously, no type of surface EMG sensor is ideal.

Therefore, along with the task of improving EMG sensors, other methods are needed to create a device capable of separating a useful signal from the noise level.

The signal-to-noise ratio is significantly influenced by such factors as the position of the electrodes on the human body, the density of pressing the electrodes to the skin, the noise of microcircuit elements, and the incorrect interpretation of electrical signals sent by the muscles.

US Pat. No. 9,299,248 describes an EMG-based muscle control device that uses combinations of different types of sensors: capacitive EMG sensors, inertial and vibration sensors to control objects using gestures. Additional sensors are mechanical, do not affect the signal, and can provide reliable control. The device is a hand-worn stretch band that is capable of remaining stationary on the user's forearm. The strip contains combinations of different types of sensors adapted to detect a variety of gestures performed by the user. The detected user gestures obtained from the analysis of signals from the sensors are processed, after which a control signal is generated that allows the user to interact with the connected controlled device. The device also contains a module for filtering and processing signals received from sensors, an analog-to-digital converter for converting an analog signal into a digital one, a calibration module with a subroutine for calibration, and a processing unit capable of recognizing user gestures based on the results of analyzing processed signals from capacitive EMGs. sensors, inertial and vibration sensors.

Capacitive EMG sensors do not require direct contact with the user's skin, and therefore are more resistant to signal changes that are characteristic of resistive sensors, but capacitive coupling provides a relatively high resistance between the skin and the sensor, and to amplify the signal, each such sensor must be additionally equipped amplifier and additional grounded electrode. This complicates the device, making the amplification circuit more expensive; in addition, the useful area of the electrodes for removing the surface EMG signals is reduced, since it is necessary to have three electrodes on the sensor, namely, two signal electrodes and a grounded electrode.

Preparing and pre-setting the device for operation through correct pre-placement of sensors and accurate calibration, preferably in automatic mode, contributes to the possibility of obtaining the best signal-to-noise ratio.

This prior art is known from US Pat. No. 8,170,656, which describes a wearable electromyographic controller that includes a plurality of capacitive EMG sensors and provides a wired or wireless human-machine interface for interacting with computing systems and connected devices using electrical signals generated by specific movements of the user's muscles. Measurement and interpretation of electrical signals generated by muscles is carried out after an automated process of self-calibration and positional localization, by sampling signals from EMG sensors. When donned, the wearable electromyography controller is positioned on the user's skin surface. The user is then instructed or given automatic signals to fine-tune the placement of the wearable electromyography controller and the controller electrodes or sensors. The controller article may be in the form of an armband, a wristwatch, or a garment having a plurality of integrated EMG sensor assemblies and associated electronics.

The user calibrates to find the optimal location for the wearable controller and to verify the accuracy of the placement of the sensors. Certain gestures are performed during calibration. Once calibrated, the user may be instructed to move or move the electrodes manually to provide gesture recognition. Manual, and possibly repeated, change in the location of the electrodes is not entirely convenient. This device also uses capacitive sensors, the advantage of which is that there is no direct contact with the skin.

However, due to the fact that the potentials and currents on the skin surface are very small, and in addition, there is a large amount of interference in the environment in the form of electromagnetic noise, which affects the measurement of the potential in a non-contact way, the recognition accuracy with a remote location of the sensors is comparable to classical sensors resistance located on the surface of the skin.

Thus, taking into account the advantages and disadvantages of devices known from the prior art, the task to be solved by the present utility model is to develop a device for recognizing gestures by detecting and analyzing muscle activity by electromyography, worn on the arm, increasing the reliability of gesture recognition, ensuring accuracy the transmitted command to the managed object.

The technical result achieved by the above set of essential features is to simplify the device, increase the accuracy and reliability of gesture recognition by reducing the effect of the noise level on the signal of electrical muscle activity.

The specified technical result, in the implementation of the utility model, is achieved due to the fact that a device for recognizing gestures by detecting muscle activity by electromyography, made in the form of an elastic band or bracelet, worn on the forearm, in which a unit for converting analog signals into a digital code is located, including series-connected electrodes of EMG sensors, low-pass filters and an analog-to-digital converter with an amplifier, inertial sensors, a power supply, a module for controlling external devices connected to a microcontroller that processes data and is programmed according to the type of an embedded neural network, which contains a calibration module with adjustment of the built-in neural networks connected to it a module for storing a fixed set of predefined user gestures and a training module in the form of a memory of the learning process for adapting the device to a certain gesture from a fixed set, and exiting d of the calibration module is connected by feedback to adjust the neural network with the input of the training module, as well as the input of the recognition module, which, based on preliminary processing of information from EMG sensors, inertial sensors and commands from the storage module for a fixed set of user gestures, generates a signal that carries information about a gesture, compares it with the information from a set of gestures coming from the memory of the training module and generates a control signal at the output, and the output of the recognition module is connected to the input of the external device control module, the microcontroller additionally contains a module for determining the reference electrode, EMG sensor having the lowest signal level , while the reference electrode determination module is connected to the output of the ADC and to the output of the storage module for a fixed set of predetermined user gestures, which additionally has feedback with the user to expand the functionality of the device. By means of a calibration module with neural network adjustment, resistance sensors are used as EMG sensors, and electrical signals of muscle contractions are recorded between the reference electrode and the electrodes of the EMG sensors.

Resistive coupling to the skin, implemented by a resistive EMG sensor, provides a relatively low impedance compared to capacitive coupling, which greatly simplifies the amplification of the detected EMG signals. The circuit is also simplified due to the absence of the need to use a reference electrode with zero potential as part of each sensor, since any resistive sensor becomes a reference electrode, the location of which is determined at the beginning of the device operation and corresponds to the position above the ulna or radius.

During the development of the device, the number of electrodes placed on the wearable device was experimentally determined, which gives an optimal and sufficient contribution to improving gesture recognition. The number of electrodes, taking into account their size and forearm sizes in different people, is at least 6 and no more than 10. A smaller number of electrodes will give insufficient information, which will reduce the accuracy and reliability of gesture recognition, and the number of electrodes more than 10 will entail an increase in data processing costs and will be redundant, since the signals from adjacent electrodes will correlate with each other.

When choosing the optimal number of electrodes, within the declared range, all electrodes are taken into account. Compared to the prototype, it becomes possible to make each electrode of a larger area, both due to the choice of the optimal number of electrodes, and due to the absence of an additional reference electrode in each resistive EMG sensor, as a result of which the transition resistance of the "electrode-skin" contact can be reduced, and thereby increasing the signal-to-noise ratio, accuracy and reliability of gesture recognition.

The feedback of the neural network calibration and adjustment module with the storage module for a fixed set of predetermined user gestures makes it possible to expand the functionality of the device and use it for people with disabilities who have poor or no muscle function at all.

The utility model is illustrated by the example of implementation given below and figures of drawings, which show:

FIG. 1 - Block diagram of a gesture recognition device.

FIG. 2 - Functional diagram illustrating the preparation and operation of the gesture recognition device.

In the device, made in the form of an elastic band or bracelet 1, worn on the forearm, electrodes of EMG 2 sensors are placed. Electrodes of EMG 2 sensors remove signals from the muscles on the skin surface that are fed to low-pass filters 3, which cut off frequencies above 1.5 kHz since there is no useful information for gesture recognition in the high frequency region. After the low-pass filters 3, the signals go to the analog-to-digital converter, ADC 4. With the help of the ADC 4, the signals from the electrodes of the EMG sensors are amplified, and then additionally filtered by a high-pass filter, to cut off frequencies below 10 Hz, where there is also no useful information, and converted to digital form. The digital signal enters the microcontroller 5, which is used to process signals from EMG sensors and recognize gestures using a built-in neural network.

Power supply 6, which may contain a battery, supplies power to the device and converts the input voltage into a set of voltages required for the operation of the device. The device contains 7 inertial sensors: gyroscope, accelerometer, which serve to obtain additional information about possible movements of the user's hand and to activate the power saving mode. The device can transmit a control signal to external devices using the external device control module 8 via Bluetooth, USB or other interfaces.

The microcontroller 5, Fig. 2, includes: a module for determining the reference electrode of the EMG sensor 9, which, when putting on the bracelet, occupies a position above the bone and has a zero potential, a storage module for a fixed set of predetermined user gestures 10, which serves to transmit information to the user about performance certain gestures in the amount of 5 to 7 gestures, a calibration module with adjustment of the built-in neural network 11, training module 12.

The memory of the training module 12 contains the averaged coefficients of a neural network trained on data from 20 people. The learning process is necessary; it consists in adapting the device to specific movements of the user's hands from a fixed set of predefined gestures. The learning process is connected to adapt the device so that the device responds with the necessary responses to certain external influences. Recognition module 13.

The description of the preliminary setting of the device and its operation is carried out according to the explanatory functional diagram of Fig. 2.

The user puts on the device, for example, a bracelet, on the forearm, so that one of the electrodes of the EMG 2 sensors is located above the ulna or radius.

After the device is fixed on the hand, the user turns on the device, or gives a ready command from an external device, for example, from a phone, laptop.

After receiving the command in the device, the storage module for a fixed set of predetermined gestures of the user 10 begins to work in the mode of joint operation with the module for determining the reference electrode of the EMG sensors 9. Instructions are given to the user about which gesture or gestures must be performed, for example, five clenches into a fist at intervals in five seconds. These instructions can be given, for example, using information on the screen of an external device of a telephone, laptop. While the user is making gestures, the module for determining the reference electrode of the EMG sensor 9 starts to work to determine which electrode of the EMG sensors installed on the bracelet in total from 6 to 10 is located above the bone. During gestures, the device evaluates from which electrode the smallest signal is received and concludes that this electrode is a reference and is located above the bone. To clarify the position, data from inertial sensors can also be analyzed: accelerometer, gyroscope. When the device selects the reference electrode in this way, the signal amplitude increases and hence the signal-to-noise ratio improves. In addition, the convenience for the user increases, since he does not need to select the placement of the bracelet on the forearm in a certain position, but it is enough to fulfill only one condition - one of the electrodes must be above the bone just below the elbow.

After the device determines which electrode is the reference, the calibration module begins to work with the adjustment of the built-in neural network 11, with the storage module for a fixed set of user gestures 10. The user is given instructions which gestures should be performed, for example, 5 folds of the palm to the right, 5 folds of the palm to the left.

Module 11 compares signals from EMG 2 sensors and from inertial sensors 7 with data that come from the memory of training module 12, after which it issues a signal about the correspondence or inconsistency of the data, as well as the need to adjust the built-in neural network. Module 11 provides the calculation of new coefficients and their recording in the memory of the training module 12.

After the end of the calibration, the device is ready for operation and recognition module 13 begins to work. It analyzes signals from EMG sensors, as well as inertial sensors 7, commands from module 10 and, using the memory of training module 12, recognizes the gesture performed by the user using the built-in neural network, and then issues a control signal to the external devices control module 8, which transmits a control command to an external controlled object or device. Module 8 can be represented by one or more wireless interfaces for data transmission, Wi-Fi, Bluetooth or others, or, if necessary, a wired interface. An externally controlled device can be a prosthesis, telephone, computer, toy, aircraft, etc.

The device has the ability to work for users with disabilities, who, due to surgical consequences, have poor or no muscle function at all. For this, if the user cannot make distinguishable gestures from the standard set, these gestures are assigned from the available common set of gestures, in the amount from 5 to 7 and distinguishable from each other.

When, during the operation of the calibration module 11, the signals from the EMG sensors, inertial sensors are compared with the data transmitted from the memory of the training module 12 from some electrodes of the EMG 2, the signal is practically absent and this is very different from the data stored in the training module 12, the adjustment of the coefficients and the signal is not transmitted to the recognition module 13, but a feedback signal is transmitted from the calibration module 11 to the storage module of the fixed set of gestures 10 and another gesture is selected to be performed next.

Thus, it is possible to replace 3-5 gestures from the training module 12 with gestures from a fixed set of predefined gestures of the user of module 10. Further, similarly to the above, if the calibration module 11 compares signals from sensors with data, training module 12 and the result above the triggering threshold, a signal is issued to adjust the neural network coefficients in the training module 12. After the calibration is completed, the device is ready for operation and the recognition module 13 starts working.

EMG electrodes are metal plates made of medical steel or other metal, enclosed in a plastic case. To use differential signals at the input of ADC 4, two electrodes are used - positive and negative, which are two metal plates. Two electrodes make up one EMG signal sensor. These metal plates in the sensor are connected by soldering or mechanically using wires or flexible printed circuit board to the myointerface board.

This device can be manufactured using modern electronic components and software. As an ADC in the device, you can use an ADC from Texas Instruments of the ADS1299 family or another with similar characteristics. The microcontroller in the device can be used by STMicroelectronics of the STM32F405 family or another with better characteristics.

It can be seen from the given example of implementation that the utility model is a simple device, randomly dressed on the forearm just below the elbow, in which traditional resistance sensors are used, an optimal automatic presetting mode that provides reliable information for recognizing gestures used for remote control of various devices.

Claims (1)

A device for recognizing gestures by detecting muscle activity by electromyography, made in the form of an elastic band or bracelet, worn on the forearm, which houses a block for converting analog signals into a digital code, including series-connected electrodes of EMG sensors, low-frequency filters and an analog-to-digital converter with amplifier, inertial sensors, a power supply, a module for controlling external devices connected to a microcontroller that processes data and programmed as an embedded neural network, which contains a calibration module with tuning an embedded neural network, a storage module for a fixed set of predefined user gestures and a training module connected to it a module in the form of a memory of the learning process for adapting the device to a certain gesture from a fixed set, and the output of the calibration module is connected by feedback to adjust the neural network to the input of the training module, as well as the same input of the recognition module, which, on the basis of preliminary processing of information from EMG sensors, inertial sensors and commands of the storage module for a fixed set of user gestures, generates a signal that carries information about the gesture, compares it with information from the set of gestures coming from the memory of the training module and generates at the output a control signal, and the output of the recognition module is connected to the input of the external device control module, characterized in that the microcontroller additionally contains a module for determining the reference electrode of the EMG sensor having the lowest signal level, while the module for determining the reference electrode is connected to the output of the ADC and to the output storage module for a fixed set of predetermined user gestures, which additionally has feedback with the calibration module with adjustment of the neural network, resistance sensors are used as EMG sensors, and electrical signals of muscle contractions are recorded between the reference electrode and electrodes of EMG sensors.

Images