RU2683859C1 - Method and system for controlling electronic devices by electromyographic reading device - Google Patents

Abstract

FIELD: medicine.SUBSTANCE: method of controlling electronic devices is carried out using an electromyographic reading device. EMG signal of the user is recorded using an electromyographic reading device. Obtained signal primary processing is performed by non-overlapping EMG signal segmentation. For each EMG signal segment obtained, a set of sign features is formed based on EMG signal dispersion square for classification of gestures. Set of features of EMG-signal of each segment is transmitted along the data transmission channel to the network of the neural network and to the computing part of the device. Based on the obtained set of characteristics, a neuron network hash function in the form of an artificial neural network generates a secondary feature data space which is transmitted to the metric classifier and describes the gesture. Intention of user in part of gesture is determined. Type of gesture is determined based on the generated set of features of the EMG signal by using a neuronetwork hash function and a metric classifier. Control signal is generated based on a certain type of gesture. Generated control signal and additional information are transmitted from gyroscope and accelerometer of electromyographic reading device via data transmission channel to actuator. Actuator action is initiated based on the obtained control signal and additional information from the sensors. Receiving feedback from actuator. Vibration motor is started at electromyographic reading device. Actuator may look in the form of a helmet or glasses with a controller on a virtual reality device, a radio-controlled model, a controlled mobile phone, a computer.EFFECT: higher accuracy of positioning and decision on hand grip of object, as well as registration of more accurate foot motion, limb position in space or determination of other motor function.20 cl, 15 dwg

Description

FIELD OF TECHNOLOGY

This technical solution relates to the field of gaming devices and medical equipment, namely, rehabilitation, in particular, to methods and systems for controlling various devices, and is intended for gaming devices and rehabilitation of people with loss of limb, with diseases leading to musculoskeletal disorders apparatus and musculoskeletal system, as well as for using parts of the product independently in various tasks to improve the quality of life.

BACKGROUND

Existing virtual reality or gaming devices management systems typically include motors, sensors, controllers, power supplies, mechanical and electrical connectors, interfaces and feedback mechanisms, all of which are optional accessories.

A known method and system for identifying gestures described in US 20150370333 A1, 12.24.2015. The system for identifying gestures includes at least one sensor responsive to user-made gestures and a processor communicatively connected to at least one sensor. A gesture identification method includes: receiving at least one signal from at least one sensor per processor; segmenting at least one signal into data windows; for each data window, at least one subset of data windows: determining the window class for the data window by the processor, the window class is selected by the processor from the window class library, where each window class in the window class library exclusively characterizes at least one property of the data window; determination by the processor of the corresponding probability that each gesture in the gesture library is a user-executed gesture based on a) the window class for the data window, when i> 1, b) the window class for the data window, where j <i; identification of a high probability of a gesture for the data window by the processor, corresponding to a gesture in the gesture library, which has the highest probability that the user performs a certain gesture. A distinctive feature of the current patent is the lack of use of neural networks, the scope of the device, as well as the technology of work, including wireless data transmission, since the current patent involves processing data on a recognition device.

Also known is a method and system for controlling a gesture-based device described in US 2015-0261306 A1, 09/17/2015. The gesture-based device control system is configured to establish a wireless connection between the gesture-based control device and the specific receiving device, the gesture-based control device including a processor, at least one sensor coupled to interact with the processor, and a wireless transmitter communicatively connected to the processor. A gesture-based device control method comprises the steps of: detecting a first gesture performed by a user of a gesture-based control device by at least one sensor, the first gesture is registered by the first receiving device, identified by the processor; a first receiving device with which the user wishes to interact based on the recognized first gesture; Configuring, using a processor, a first signal for use only by the first receiving device; and a wireless method for transmitting a first signal at a first receiver using a wireless transmitter. A distinctive feature of the current patent is the lack of use of neural networks, the scope of the device, as well as the operation technology, since the current patent involves the implementation of data processing on a recognition device.

There is also a known method and control system for intellectual bionic limb described in RU 2 635 632 C1, 12/14/2016, based on this patent, improvements were made in terms of the algorithm for recognition of grips, sensors, as well as the architecture of data processing, the use of a new device, namely control not of a bionic limb, but as a gaming device, for virtual reality and not only, new studies formed the basis of the current patent.

Currently, there is a constant need to improve control systems and create new approaches in the management of gaming devices and virtual reality, which would be able to be used in rehabilitation, virtual reality management or a gaming system; the proposed solutions should cope with this task.

ESSENCE OF TECHNICAL SOLUTION

The technical problem solved in this technical solution consists in the implementation of an autonomous full-featured control for performing low-level tasks of manipulating any electronic device adapted for this, such as virtual reality, radio-controlled models, computers, etc.

The technical task of the technical solution is to develop

- a reliable functional device in the form of a bracelet or other similar system design with feedback controlled by neuromuscular signals (data processing and recognition of grips can occur both with the participation of a computer or phone, or autonomously, on the device itself);

- a method and system for controlling an auxiliary device, such as a gaming device, virtual reality, computer, telephone, etc.

The technical result that manifests itself in solving the indicated technical problem and problem is to increase the accuracy of positioning and make decisions about the capture of a subject by hand, as well as registering more precise foot movements, the position of a limb in space, or the determination of another motor function.

Also an additional technical result is a decrease in the cognitive load on a person, an increase in the efficiency of device management, and a decrease in the response time of actuators.

Another additional technical result is the simplification of the installation and configuration of the system "actuator-reader" due to the modularity of the system - the reader is portable, and requires approximate positioning during placement, and is easily removable. The actuator can come in various forms: it can be a gaming device, such as a radio-controlled machine or drone, virtual reality, or a software device, computer or system for rehabilitation, connected to the reader only with a wireless data transmission channel.

A decrease in the cognitive load on a person and an increase in the efficiency of controlling the actuator are achieved through the use of a hybrid control system that combines decoding methods of electroneuromyosignals (myo-electric signals) and the use of a neural network hash function with a teacher for performing glad human limb movements. Also, additional sensors in the bracelet help better actuators after calibration to determine the position in space of a person’s limbs, speed and direction of movement.

The transmitted data array to the microcontroller in the actuator with the ability to position the limb in space in real time allows you to better position the limb and perform the necessary manipulations in terms of control, as well as receive feedback from the actuator if necessary. Also, for more accurate positioning of the limb in space, additional sensors are used, such as for example a gyroscope and accelerometer.

The use of feedback, implemented by means of a vibrating motor reading system, allows you to secure the use of the actuator and transmit the necessary information from the actuator.

The use of wireless data transfer between the actuator and the reading system provides the convenience of using the system and the interchangeability of components, in addition, the possibility of their use independently of each other.

The specified technical result is achieved due to the method of controlling the actuator, in which at least one EMG signal of the patient is received by means of a myoelectric reader; processing at least one user EMG signal by overlapping segmentation of the EMG signal; for each segment obtained in the previous step, a set of features of the EMG signal is formed based on the dispersion square and the neural network hash function of the EMG signal for classifying gestures; transmitting a set of features of the EMG signal of each segment over the data channel to the actuator control system; determine the type of gesture based on a set of signs of the EMG signal by using an artificial neural network and a metric classifier; generating a control signal based on a certain type of gesture; transmit the generated control signal to the actuator; receive feedback from the control system by obtaining information from external sensors or an actuator algorithm.

System

BRIEF DESCRIPTION OF THE DRAWINGS

The signs and advantages of this technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

FIG. 1 (a, b) presents an example of the configuration and functioning of the actuator control system, in particular, paintings are presented by hand, but possibly from the foot and other places where EMG is read.

FIG. 2 describes the general structure of a technical solution.

FIG. 3 (a, b, c) presents a diagram of the implementation of the control system, including for the actuator, for a) rehabilitation b) of the gaming device, as an example, a radio-controlled machine, as well as a block diagram of this implementation, c) virtual reality management .

Fig 4 is an example of the implementation of the settings of the bracelet in the presence of the necessary functionality in the bracelet itself, in the absence of additional functional settings.

FIG. 5 presents an example of a scheme for obtaining data in the form of a myoelectric reading system, a design in the form of a bracelet with replaceable electrodes (item 1 - a reading system, item 2 - a removable biopotential registration sensor, the number of sensors can be adjusted, item 3 - an elastic cuff), but the design may be different, depending on need and limb.

FIG. 6 shows the components of the reading system in the form of a myoelectric reading system (pos. 1 and pos. 2) and their approximate location on the limb, as an example, hands on the shoulder (pos. 1) or forearm (pos. 2).

FIG. 7 shows an example of an electromyographic signal recorded with volitional force from the muscles of the forearm.

FIG. 8 shows an exemplary embodiment of an EMG sensor using a dry contact more preferred for long-term use.

FIG. 9 is a block diagram of an input circuit.

FIG. 10 is a block diagram of a gesture recognition system.

FIG. 11 shows the result before. signal processing (squared dispersion followed by contrast using a sigmoid).

FIG. 12 (a, b). Examples of the actuator: a) virtual reality glasses, an example, Oculus b) a machine running Bluetooth, an example on the Arduino platform.

FIG. 13 shows an example of processing a signal from a reading system.

FIG. 14 shows time functions for extracting features of an EMG signal.

FIG. 15 shows data capture on an orthonormal basis using rotation. Left: neuron trying to adapt to cover new data. Right: end position of a neuron after a new data coverage.

DETAILED DESCRIPTION OF THE TECHNICAL SOLUTION

Below will be described the concepts and definitions necessary for the detailed disclosure of the ongoing technical solution.

The technical solution can be implemented as a distributed computer system.

In this solution, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems, and any other devices that can perform a given, clearly defined sequence of operations (actions, instructions).

By a command processing device is meant an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

The command processing device reads and executes machine instructions (programs) from one or more data storage devices. Storage devices may include, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical media (CD, DVD, etc.).

A program is a sequence of instructions intended for execution by a control device of a computer or a device for processing commands.

Electromyography (EMG, ENMG, myography, electroneuromyography) - a method for studying the bioelectric potentials that occur in the skeletal muscles of humans and animals upon excitation of muscle fibers; registration of muscle electrical activity.

Electroneuromyography (ENMG) is a comprehensive electrophysiological study necessary to determine the functional state of the peripheral nervous system and muscles [1].

Actuator is a device in the system of automatic regulation and control that directly performs mechanical movement (or rotation) of the regulatory body of the control object, including the virtual object. It can be either a complex of devices or a single device receiving commands from the receiver and processor of EMG data and not only the device, and capable of transmitting feedback information.

The method of physiological control of the actuator based on gesture recognition (processing the control signal) is as follows.

The signal is pre-processed and the input vector of EMG signal attributes is created to classify user gestures, which can be performed in a similar way during setup and during system operation.

In order to distinguish and classify muscle movements, the most significant parts of EMG (signs), which are characteristics with sufficient information for classification, must be extracted. According to research, to classify EMG signals of the upper extremities, the dispersion square on each channel is calculated, after which information from all channels is fed to a neural network hash function that identifies a number of uncorrelated most important signs.

More suitable for analysis are signs of an EMG signal in the time domain based on signal amplitudes. Such features can be easily identified, have high stability for pattern recognition using EMG. To achieve better results, the function (feature) should contain enough information to represent the essential properties of the EMG signal, and should be simple enough for quick processing and classification. In some embodiments, the following time-domain values, measured as functions of time, may be used as features: integral EMG; average; the average value of the module; finite differences; sum of elementary areas; dispersion; standard deviation; signal length; the maximum value of the EMG signal, in more detail, information on the selection of signs for classification is disclosed in the authors [2].

Formulas for calculating these values are shown in FIG. fourteen.

Evaluation of the effectiveness of signs is carried out according to two main parameters - the performance of the neural network and operating time - as the most important for real-time use. According to previous studies described in the information source [3], for the conditions for fulfilling the requirements of real-time operation, the signal recognition time should take no more than 25 ms (but these parameters are regulated by the neural grid, window width, and other parameters, which allows the recognition time to be adjusted signal, increasing both up and down). For comfortable work of the user, the productivity or recognition accuracy (the percentage of correct classification cases to all cases considered) should be at least 95%, as shown in the source [4]. An example implementation of signal processing is shown in FIG. 10 and 13. In order to increase the accuracy of gesture classification and prevent false triggering of the bionic limb actuator at the signal pre-processing stage, simultaneously with calculating the maximum value of the EMG signal segment, the average duration of exceeding the threshold value of the EMG signal in the real-time data window is calculated, then, after centering and normalizing the received signal, a decision is made about the occurrence of a control signal.

Neuromuscular activity data (matrix data of electromyograph channels) are obtained from the myoelectric reader and digitized by ADC, then the condition for the appearance of the active signal (corresponding to the gesture) is checked, for example, 30% of the amplitude specified in the settings. Next, the received digitized signal is broken by a non-overlapping window into segments, for example, 25 ms long. In some embodiments, the implementation of the technical solution uses the function of the maximum value of the segment, as the most effective and easily calculated. For each segment, its maximum value is calculated by the formula:

where x _k is the k-th segment of the signal, x _i is the i-th value of the segment. Further, the obtained values are normalized as follows:

where X is the signal data array, X _i is the i-th signal value;

And center:

– среднее арифметическое сигнала Х.Where

- the arithmetic mean of the signal X.

Thus, an input feature vector (by which gestures are classified) is obtained for each channel with a lower dimension than the original signal (depending on the window width): a vector of length k instead of the original vector of length k * i, which reduces the computational load to the microcontroller. The dimension is reduced due to the fact that they find one value among i values.

It is important to note that the myoelectric reader also has an electromyograph operating mode and the EMG data is output to the dial of the myoelectric reader or to the actuator screen, such as a computer or telephone, which makes it possible to better determine the signal level from each muscle or muscle group.

At the stage of system setup, after creating an input vector of signs of an EMG signal, an artificial neural network (ANN) is trained. In some embodiments, an ANN is used based on a radial basis function (as an example, do not limit it, since the best option is to use a neural network hash function). In some embodiments, a multilayer perceptron of a neural network, as well as other types of neural networks or a support vector method, can be used.

In order to recognize gestures, the extracted features must be classified into distinctive classes. The classifier should be able to cope with factors that have a noticeable effect on EMG patterns over time, such as a significant change in EMG signals, the location of the electrodes, sweat and fatigue described in the information source [5].

The above method can use a three-tier management architecture (database - server - client) when processing a set of signs of an EMG signal.

In the input layer, the number of neurons is equal to the dimension of the feature vector (which in this case is equal to the number of data channels through which data from the sensors are transmitted), which in the example implementation of the invention can be eight: x _i , i = 1, 2 ... 8. The hidden layer, where the number of neurons was not determined in advance, since they were formed during the training procedure, was divided into fourteen sub-hidden layers. In the output layer, the number of neurons was equal to the number of classes in the training data set (the number of gestures is the number of neurons).

The basic function of the neural network in the hidden layer is the Gauss function, and the output of the kth neuron in the hidden layer for each given input X = [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , x ₆ x ₇ , x ₈ ] ^T can be calculated using the following formula:

(4)

(four)

This equation describes an 8-dimensional Gaussian centered at C = [C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C _6, C ₇ , C ₈ , C ₃ ] ^T and rotates along the orthonormal basis {U ₁ , U ₂ , U ₃ , U ₄ , U ₅ , U ₆ U ₇ , U ₈ }, which allows the neuron to cover the data field of the neighbor without offset or any size change. The width of this Gaussian along each axis is a _i , i = 1, 2 ... 8.

Since the input feature vectors for each image are eight-dimensional, the coordinates corresponding to these vectors represent a basis of the form [1,0,0,0,0,0,0,0,0] ^T , [0,1,0,0,0, 0,0,0] ^T ... [0,0,0,0,0,0,0,0,1] ^T. Thus, the component x _{i of} each input vector X with respect to the new axes can be calculated as:

(5)

(5)

Rotation along the basis vectors allows neurons to cover all nearby data without increasing the radius. In FIG. Figure 15 shows how a neuron tries to adapt to cover new data; the right side of the figure shows the location of the data by the neuron.

During system setup, a user signal is recorded in the database, the data in which is accumulated, and used to retrain the network, but there is not always a need for this retraining. For each user, there is a feature vector with a size of 8 × N (where 8 is the number of channels, N is the number of training data) obtained using the method described above (segmentation by non-overlapping windows). Before the training procedure, each data set is mixed and then divided into sets in a ratio of 2: 1: 1 (according to the number of training data N) with data for the stages of training, testing and control, respectively.

The orthonormal basis is calculated through the eigenvector of the covariance matrix. Since the training data is entered into the network sequentially, the mean vector and covariance matrix are calculated recursively. For N samples X = {x ₁ , x ₂ , ..., x _N }, in which x _j = ℜ ³ , j = 1, ..., N, the vector of average values is calculated as follows:

(6)

(6)

– это вектор средних значений множества данных X; X_N+1 – новый вектор данных, добавляемый во множество данных X.Where

Is the vector of average values of the data set X; X _{N +} 1 is a new data vector added to the data set X.

Then the covariance matrix is calculated as:

(7)

(7)

(8)

(8)

To find an orthonormal basis for RBF, the concept of principal component analysis is used. The eigenvalues {λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , λ ₆ λ ₇ , λ ₈ } and the corresponding eigenvectors {U ₁ , U ₂ , U ₃ , U ₄ , U ₅ , U ₆ U ₇ , U ₈ } are calculated from the level of the covariance matrix. Further, the set of eigenvalues of the orthogonal vectors form an orthonormal basis, as shown in the information source [6].

The procedure for training a neural network during system setup can be described in more detail as follows.

Let X = {(x _j , t _j ) | 1 ≤ j ≤ N} be the set N of training data (training set), where x _j is a vector function (component x _j ϵ ℜ ⁸ , where ℜ ⁸ is the set of training data ) and t _j is the class with which the gesture needs to be associated. Let Ω = {Ω _k | 1 ≤ k ≤ m} be a collection of m neurons. Each neuron has five parameters:

Ω _k = (C ^k , S ^k , N _k , A _k , d _k ), (9)

– это количество данных, соответствующих k-му нейрону,

– ширина вектора k-го нейрона, и d_k – класс-метка k-го нейрона.where C ^k is the center of the kth neuron, S ^k is the covariance matrix of the kth neuron,

Is the amount of data corresponding to the k-th neuron,

Is the width of the kth neuron vector, and d _k is the label class of the kth neuron.

Since new neurons can be automatically added to the network and located very close to each other, it is possible to implement a strategy of merging several neurons to prevent the network from growing to its maximum structure (one neuron for each data unit), which is described in detail in the information source [7].

A trained neural network allows you to precisely adapt to the user. As a result of training, ANNs store new values of the neural network weighting coefficients in the microprocessor of the actuator directly or in the data warehouse.

However, it should be borne in mind that other types of ANNs, like other classification methods, can also be implemented in this invention by a specialist in their field.

During the operation of the system, the result of gesture recognition is converted into the appropriate control command (for example, mental compression into a “fist” by the user - “fist”), and transmitted, for example, to the actuator — it is reflected on the computer screen or the radio-controlled model stops moving.

A physiological control system based on gesture recognition (processing a control signal) is carried out as follows.

The above system consists of a myoelectric biopotential reading device, which can be structurally made in the form of an easily removable elastic bracelet or similar design (shown in Fig. 5, item 1). This device contains electrodes for recording the activity of the neuromuscular system (item 2) that occurs in response to the phantom movement of the user. Moreover, the bracelet can be placed on the forearm on the shoulder or other places on the human body and is configured to transmit a control signal to the actuator.

This bracelet may include a wireless data sensor, and, without limitation, Bluetooth or Wi-Fi, or ZigBee wireless technology can be used.

In an easily removable bracelet or other similar design worn on the shoulder, forearm, or other places on the human body, at least one electromyogram sensor (hereinafter referred to as EMG sensor) is placed, with an operational amplifier, a low-pass and high-pass filter, and an analog-to-digital converter ( ADC), microprocessor-microcontroller, voltage converter and batteries, vibration motor, gyroscope and accelerometer.

The EMG sensor in some embodiments is a set of electrode plates placed on a dielectric substrate. In various embodiments of the technical solution, the electrodes can be made of stainless steel or of slightly polarizing conductive materials (for example, stainless steel grade 12X18H10). In some embodiments, the electrodes can be made in the following sizes: 1x1 cm square for the receiving plate (2 pieces) and 1x0.5 cm rectangle for the referent), not limited to. In some cases, there is no need to use a rectangle; one rectangle is used for all sensors. One skilled in the art will appreciate that the size of the electrodes can be made in a different size, given that the plate size affects the signal amplitude and crosstalk.

On top of the electrode substrate, a biopotential amplifier board can be installed, which can consist of a tool (differential) amplifier (for example, such as INA114), low-pass and high-pass filters made on passive elements in the form of RC circuits or on active elements. Through flexible loops, the sensors are connected to the microprocessor board, where digitization by analog-to-digital conversion can take place and transfer it to the microcontroller (for example, 2x 16-bit ADS1115 ADCs and signal preprocessing on the Cortex M4 microprocessor ARM controller) FIG. 8.

With voluntary muscle contraction by the user with active electrodes, the potential difference in the low-frequency range that occurs in the neuromuscular tissue is recorded.

It should be emphasized the need for a reliable way to remove the control signal. The frequency composition of the electromyogram for users, according to studies, is represented by a highly noisy signal in the range of 0.5-300 Hz and a signal amplitude of 1 μV to 2 mV (Fig. 6).

There is also a completely new solution method in terms of sensors, since sensors can be without amplifiers. The input circuits of measuring instruments and processing biopotentials are high requirements, because the level of the electromyosignal is only tens of microvolts - units of millivolts, and it is also much less than the induced noise, which can reach tens or even hundreds of millivolts.

Under the input circuits (CC) of the device with digital processing of the electromy signal, we mean the circuit of the analog signal from the sensor electrodes to the output of the analog-to-digital converter (ADC).

The following basic requirements are made to the EC:

1. Providing input impedance of 100 megohms or more;

2. Pre-filtering the input signal based on the sampling frequency of the ADC;

3. Suppression of the in-phase component of the input signal at least 100 dB;

4. ADC resolution sufficient to convert a signal level of 10 μV;

5. The presence of an active common mode rejection circuit for the input signal.

In addition, to ensure a high resolution for grip recognition in a system using a neural network as a computer, additional requirements are imposed. Namely, the number of simultaneously processed channels should be as large as possible.

Based on these requirements, the ADS chip was selected, which includes:

- preamplifier with adjustable gain up to 24;

- the suppression ratio of the component of the input signal is not less than 115 dB;

- delta-sigma ADC with a bit capacity of 24 bits;

- active common mode rejection circuit;

- voltage generator Vcm;

- 8 differential, simultaneously processed channels;

- sampling frequency of the input signal up to 32000 b./s.

ADS input impedance is more than 1 GΩ.

Two electrodes of each channel are connected directly to the preliminary filter, forming a differential channel.

To actively suppress the common-mode component of the input signal, the amplified and inverted common-mode signal is fed to the third electrode of each sensor.

An RC filter of the first order was used as a preliminary filter.

The ADS chip is controlled via the SPI interface.

The digitized signal is fed to the system processor for further processing.

A block diagram of the input circuit is shown in FIG. 9.

The implementation of the technical solution (Fig. 1A) can occur as follows. The user performs the specified gestures, while the myoelectric reader transmits user data to the data processing device via a wireless data transmission channel (for example, using the Bluetooth 4.0 standard, but not limited to it), where the user profile is created. Then, the created profile is recorded in the memory of the microcontroller.

The myoelectric reader performs registration and filtering of the electromyogram (EMG) and data transmission via a wireless communication channel (for example, using the Bluetooth 4.0 standard, but not limited to it) to the control board of the actuator, the arrangement of which is shown in FIG. 12. The signal processing method is described in more detail above.

FIG. 12 is a top view of an actuator that is different for virtual reality or for a radio-controlled model.

The management of virtual reality is as follows: a myoelectric reader in the form of a bracelet (or a device similar in design) is put on the forearm or shoulder, or another part of the limb, as an example, legs, the bracelet is calibrated and adjusted, this can be done on a computer, phone ( Fig. 3c), as well as without them (Fig. 4). Virtual reality glasses consist of at least different types of lenses, a display, housing parts, a computer system, an accelerometer, a gyroscope, and a wireless sensor (Bluetooth) (Fig. 12a). The connection between the bracelet and virtual reality glasses is organized using any wireless connection, in our case, Bluetooth is used, with the Bluetooth of the bracelet configured as a slave, and the Bluetooth of virtual reality glasses as a master. A mioelectric reader, in the form of a bracelet (or a device similar in design), registers and filters an electromyogram (EMG), determines the position of the hand and, depending on it, sends a corresponding command to virtual reality glasses, or receives information by feedback. It is important that the myoelectric reader, in addition to data on the numbers of the grips, sends information to the glasses from the gyroscope and accelerometer, which allows you to better determine the position of the hand in space. It is possible that the myoelectric reader is located not only on the hands, but also on the legs, back, neck, which will make it possible to better determine the location of other body parts in space. Also, virtual reality glasses are able to transmit feedback information and run a vibromotor on the myoelectric reader, which allows you to create feedback, it is also possible to additionally position other feedback sensors and sensors.

The radio-controlled model is controlled as follows: a myoelectric reader in the form of a bracelet (or a similar device design) is put on the forearm or shoulder, the device is calibrated and configured, this can be done on a computer, phone (Fig. 3b), as well as without them (Fig. . four). The radio-controlled model consists of at least a microcircuit, a wireless communication sensor (Bluetooth), and engines (Fig. 12b). The connection between the bracelet and the radio-controlled model is organized using any wireless connection, in our case, Bluetooth is used, with the Bluetooth of the bracelet configured as a master, and the Bluetooth model as a slave. After starting both devices, with connection without intermediaries, the wireless device of the bracelet is connected to the communication device of the actuator in our case, for communication, the master needs to know the name and password of the communication device of the executive mechanism. In the event that the devices are connected through an intermediary (computer, telephone), the intermediary device is involved in establishing communication between the devices. The myoelectric reader performs registration and filtering of the electromyogram (EMG), determines the position of the hand and, depending on it, sends the appropriate command to the model. List of commands: F - the model starts moving forward until it receives the next command, B - the model starts moving back until it receives the next command, L - the model starts moving left until this command arrives, after its completion continues the action that performed until the command was received ( moving forward, backward, inactivity), R - the model starts moving to the right until this command arrives, after its completion continues the action that performed until the command was received (moving forward, backward, inaction), S - stop the model.

The model receives a command, the chip processes it depending on the command, performs manipulations on the motors: start rotation, change direction, stop. After executing the current command, the device is ready to perform the following. Also, feedback information can come from the model to the bracelet and start the vibration motor on the bracelet or other types of sensors.

It is also possible that a rehabilitation system is implemented with or without virtual reality glasses (Fig. 3a). Namely, when data is being taken, for example, from grips from the forearm or shoulder of the hand, and transmission to a computer or phone to perform certain grips or movements in space, movements, a program is launched on the computer or phone, which receives information and signals success or failure completing the task, it is possible to feed a feedback signal to the bracelet.

The management system also includes:

controller network (controller network), which provides a modular, reliable and fault-tolerant interaction scheme for data exchange between the controller and other control and executive bodies of the actuator, and which ensures the transfer of intentions to the controller from the user's neuromuscular system;

source of power;

multi-colored surface light-emitting diode, vibration and sound indicators (LEDs) located in the prosthesis, with the help of which the user feedback is provided through the controller.

Exemplary embodiments of the control system include a controller 301 (not more than 2x3 cm), which receives data about the recognition result, exchanges data, and controls the actuator.

A controller in some embodiments may include a processing unit,

an interface unit, which is physically separated from the processing unit, and the above units can be connected by a flexible cable, which allows for flexible construction of the interface without changing the corresponding software control element.

In FIG. 3 shows a block diagram of an actuator control system that includes a prosthesis controller 301, a tire 302, and at least one tire 303 (only one of them is shown in FIG. 3). A bus 302 and a finger bus 303 are connected to the controller 301 via an I / O interface 304.

The controller 301 may be one of the components of a mechanism (not shown in this diagram). The controller 301 is the central control unit of the actuator. Thus, the controller 301, which can be structurally located inside the actuator, is responsible for the high level of coordinated control of the actuator, as well as a variety of functions associated with the internal nuances of the actuator.

Also, if necessary, in some embodiments, the controller 301 may be located outside the actuator.

One of the functions of controller 301 is to control the movement of the actuator.

Accordingly, commands about the user’s intention are transmitted by the ENM unit 305 of the interface (electroneuromiointerface), which interacts with the controller 301 via the bus 302 and the input-output interface 304, as shown in FIG. 3. Commands from the ENM unit 305 of the interface, as well as information from the large manipulation drive controller 306 and at least one small manipulation drive controller 307 (one of them is shown in FIG. 3), which are received through the bus 302 and the finger bus 303, allows the control method the actuator described above, in the controller 301 generate new commands for controlling large manipulation drives and small manipulation drives.

The controller 301 also provides for the transmission of sensory data from sensors or external software using a vibration sensor, or using a weak electrical pulse.

This feedback allows the user to sense external conditions without being limited only by visual or audio feedback.

In further embodiments, the controller 301 uses bus 302 to communicate with additional devices and / or accessories, such as radio frequency identification devices (RFIDs), smartphones, and personal digital assistants, although additional or alternative embodiments of the invention are not limited to these examples.

The functioning of the system is as follows. A mioelectric reader in the form of a bracelet or a similar system design is worn on the forearm and shoulder and connected to a computer or phone, and the bracelet and actuator are set up and calibrated as follows. The training sample is formed as follows. Using the calibration program, the types of gestures are sequentially displayed on the screen, for example, “fist”, “pinch with three fingers”, “grip” key ”,“ cylindrical grip ”, and others, which are proposed to be repeated by the user several times over a certain time period, for example , 5 times in one second, after which the obtained data are used to train a recognition algorithm based on a neural network, as described above. A user profile is created with new weights, which is recorded in the microcontroller of the bracelet, or recording immediately occurs in the bracelet. Further, the actuator and bracelet can work autonomously, based on wireless communications, for example, via Bluetooth 4.0.

The myoelectric reader can be worn before or after adjusting the actuator thanks to the elastic cuff.

The user can use the movements recorded during calibration. After the movement, the signal is recorded, converted and classified according to the steps described above. Due to the presence of sensors transmitting additional information from the actuator, feedback is provided with the user, as described above.

EXAMPLES OF IMPLEMENTATION OF THE TECHNICAL SOLUTION

A mioelectric reader is placed on the user's hand and simultaneously connects to a personal computer or telephone, which is a data processing device. The user, executing commands on the screen, makes 14 gestures (for example, “Fist”, “Incomplete fist, or cylindrical grip”, “Opening the palm”, “Pinch with thumb and forefinger”, “Pinch with thumb and ring finger”, “Turn the palm on yourself ”,“ Turn the palm away from you ”,“ Pistol ”- the forefinger and thumb are straightened, the rest are pressed”, “Fist with the straightened thumb”, “Rotate the forearm onto yourself”, “Rotate the forearm away from you”, “Grip computer mouse ”,“ Turnkey Capture ”,“ Relaxation ”), each for one second und, repeating them according to the images on the screen, thus forming a training sample of gestures.

Another option, when the myoelectric reader is placed on the user's hand and simultaneously performs not only data reading, but also processing and training of the neural network and not only any further mechanism can be used, such as a personal computer or telephone, which are reflection devices of processed data by grips. In the learning process, you can simply use instructions on the sequence of contractions or an actuator for clarity. In the future, the user performs gestures that, for example, are reflected on the screen, for example, 14 gestures (for example, “Fist”, “Incomplete fist, or cylindrical grip”, “Opening the palm”, “Pinch with thumb and forefinger”, “Pinch with thumb and ring finger "," Turn the palm toward yourself "," Turn the palm away from you "," Pistol "- the index and thumb are straightened, the rest are pressed", "Fist with the straightened thumb", "Rotate the forearm onto yourself", "Rotate the forearm from yourself ”,“ Capture a computer mouse ”,“ Capture on a turn-key basis ”,“ Ra weakening ").

 The sensor electrodes of the myoelectric reader record the potential difference that occurs during muscle contraction, then the signal is amplified 1000 times (it is important to note that the amplification can be both directly on the electrode and on the ADC, in the latter case, the electrode represents only protected plates that transmit data to ADC) is filtered in the 5-500 Hz band and digitized in the ADC. Further, in the setup mode, the digitized signal is transmitted to the computer. Each digital signal sample for each gesture is associated with the number of the performed gesture for further training. The data for each gesture are connected into a matrix, respectively, at a sampling frequency of, for example, 500 or 1000 Hz, we have a data set of size 14x1000x8x14000 (14 gestures per second x 1000 Hertz, 8 - the number of channels, 14000 - target vector from 1 to 14, a gesture number assigned to each sample, 1000 values for each target), the signal amplitude is in the range from 100 μV to 1 mV. Then the signal is segmented by windows with a width of 25 ms, 25 samples per segment, respectively, for a sampling frequency of 500 or 1000 Hz. Next, a maximum is found on each segment, an output vector of attributes of size 14x40x8x350 is obtained (the dimension of the target vector also decreases). Then the data in the resulting vector are centered and normalized, as described above. The amplitude is reduced to the range [-1: 1] or remains in the dimension of the potential difference.

Thus, an input feature vector for training a neural network is obtained. Training takes place according to the algorithm described above on a reader or on another processor. Further, the new values of the network weights are written to the controller of the reading system. Then the user can use the system. The data before segmentation and the data after segmentation are stored in a database on a remote computer to allow further research and to replenish the training sample of the neural network. It is also further possible to adjust the correspondence of gestures performed by the user and the grips implemented by the actuator in response to the user’s gesture by connecting the actuator to a computer or directly to the reading system.

It is important to note that the number of recognizable grips depends primarily on the step / overlap of the window, and on the averaging algorithm. For example, if we average over 2 seconds, then we recognize 1 gesture in 2 seconds, but how much data we send to averaging depends on the step, if in the samples we write n = fd / m, where n is the number of recognized grips per second, fd is the sampling frequency, m is the window step in samples. You can write this down after a while, then n = 1 / t, where t is the window shift in seconds.

Then, during the operation of the system, the user performs any of fourteen gestures, then the signal is recorded and processed. The sensor electrodes record the potential difference that occurs during muscle contraction, then the signal is amplified 1000 times, filtered in the 5-500 Hz band and digitized in the ADC. Further, the signal is fed to the microcontroller of the reading system. A number of data is dialed equal to the width of the window (25 ms), respectively, at a sampling frequency of, for example, 500 or 1000 Hz, we have a data vector of size 25x8 (25 samples in 25 ms, 8 is the number of channels), the signal amplitude is in the range from 100 μV up to 1 mV. Next, find the maximum of the current segment, get the output size of 1x8. Then the data in the resulting vector are centered and normalized, as described above. The amplitude is reduced to the range [-1: 1] or remains in the dimension of the potential difference. Next, the vector is fed to the input of the neural network, which determines the class of gesture. Based on the classification result, a control command is created and transmitted to the actuator. The resulting grip, which the actuator translates into the desired action, can be either similar to the executed user or otherwise specified in the settings initially. It is important to note that for children a different number of sensors can be used, the number of channels, for example 4 or 5. In addition to the EMG data, the reader also uses data from the gyroscope and accelerometer and, if necessary, they can be transmitted separately or via a neural network hash function to the actuator , which, if necessary, can transmit feedback information and start the vibromotor on the myoelectric reader and signal the user about something.

Received information on contractions and additional information is transmitted to the actuators. An example of managing virtual reality is as follows: data on grips and additional information from a myoelectric device, that is, a more accurate determination of the position of an arm or leg, transmit a corresponding command to virtual reality glasses, or receive information by feedback. It is important that the myoelectric reader, in addition to data on the numbers of the grips, sends information to the glasses from the gyroscope and accelerometer, which allows you to better determine the position of the hand in space. Also, virtual reality glasses are able to transmit feedback information and run a vibromotor on the myoelectric reader, which allows you to create feedback, it is also possible to additionally position other feedback sensors and sensors.

The radio-controlled model is controlled as follows: after starting both devices, without connecting to intermediaries, the wireless communication device, the myoelectric reading system, is connected to the communication device of the actuator, in our case, communication requires the master to know the name and password of the communication device of the actuator. In the event that the devices are connected through an intermediary (computer, telephone), the intermediary device is involved in establishing communication between the devices. The myoelectric reader performs registration and filtering of the electromyogram (EMG), determines the position of the hand and, depending on it, sends the appropriate command to the model. List of commands: F - the model starts moving forward until it receives the next command, B - the model starts moving back until it receives the next command, L - the model starts moving left until this command arrives, after its completion continues the action that performed until the command was received ( moving forward, backward, inactivity), R - the model starts moving to the right until this command arrives, after its completion continues the action that performed until the command was received (moving forward, backward, inaction), S - stop the model. The model receives a command, the chip processes it depending on the command, performs manipulations on the motors: start rotation, change direction, stop. After executing the current command, the device is ready to perform the following. Also, feedback information can come from the model to the bracelet and start the vibration motor on the bracelet or other types of sensors.

It is also possible that a rehabilitation system is implemented with or without virtual reality goggles, namely, when data is taken, for example, from the grips of the forearm or shoulder of the arm, and transferred to a computer or phone to perform certain grips or movements in space, movements, A program is launched on the computer or phone that receives information and signals the successful or unsuccessful completion of the task; a feedback signal can be sent to the myoelectric device.

It is important to note that the myoelectric reader also has an electromyograph operating mode and the EMG data is output to the dial of the myoelectric reader or to the actuator screen, such as a computer or telephone, which makes it possible to better determine the signal level from each muscle or muscle group.

USED INFORMATION SOURCES

1. Cowan X. L., Brumlik J. Guide to electromyography and electrodiagnostics: Per. from English.-M.: Medicine, 2005.- 192 p.

2. Ivaniuk Natallia, Ponimash Zahar, Karimov Vladimir “Art of Recognition the Electromyographic Signals for Control of the Bionic Artificial Limb of the Hand”, International Conference of Artificial Intelligence, Medical Engineering, Education AIMEE 2017: Advances in Artificial Systems for Medicine and Education , pp 176-181 https://link.springer.com/chapter/10.1007/978-3-319-67349-3_16.

3. Englehart K, Hudgins B: A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans Biomed Eng 2003, 50 (7): 848-854.

4. Van den Broek EL, Lis’y V, Janssen JH, Westerink JHDM, Schut MH, Tuinenbreijer K: Affective Man – machine Interface: Unveiling human emotions through biosignals. Biomedical Engineering Systems and Technologies: Communications in Computer and Information Science. Berlin, Germany: Springer Verlag; 2010.

5. Maria C. Selection of suitable hand gestures for reliable myoelectric human computer interface / Maria Castro, Sridhar P Arjunan, Dinesh K Kumar // BioMedical Engineering OnLine. 2015. -14: 30 (April 9, 2015) available at: http://www.biomedical-engineering-online.com/content/14/1/30.

6. Huihui L. Relationship of EMG / SMG features and muscle strength level: an exploratory study on tibialis anterior muscles during plantar-flexion among hemiplegia patients / Huihui Li, Guoru Zhao, Yongjin Zhou, Xin Chen, Zhen Ji, Lei Wang // BioMedical Engineering OnLine. 2014. 13: 5 (January 27, 2014) available at: http://www.biomedical-engineering-online.com/content/13/1/5.

7. Saichon J, Chidchanok L, Suphakant P: A very fast neural learning for classification using only new incoming datum. IEEE Trans Neural Netw 2010.

Claims (34)

1. A method of controlling electronic devices using an electromyographic reader, comprising the steps of: - registering at least one EMG user signal using an electromyographic reader; - carry out the initial processing of the received at least one EMG signal of the user through non-overlapping segmentation of the EMG signal; - for each received segment of the EMG signal, a set of gestural signs is formed based on the square of the dispersion of the EMG signal for classifying gestures; - transmit a set of signs of the EMG signal of each segment on the data channel to the neural network database and to the computing part of the device; - on the basis of the obtained set of features, a neural network hash function, which is an artificial neural network, forms a secondary feature space of data that is supplied to the metric classifier and describe at least one gesture; - determine the intention of the user in terms of the gesture, in which the type of gesture is determined based on the generated set of signs of the EMG signal by using a neural network hash function and a metric classifier; - form a control signal based on a certain type of gesture; - transmit the generated control signal and also additional information from sensors such as a gyroscope and an accelerometer of an electromyographic reading device via a data transmission channel to an actuator; - initiate the action of the actuator based on the received control signal and additional information from the sensors; - receive feedback from the actuator, while the vibromotor is started on the electromyographic reader. 2. The method according to p. 1, characterized in that when registering an EMG signal, a potential difference is recorded in the low-frequency range that occurs in the neuromuscular tissue. 3. The method according to p. 1, characterized in that upon receipt of the EMG signal, the myoelectric reader performs filtering of the signal. 4. The method according to p. 1, characterized in that when segmenting the EMG signal for each segment, the square of the dispersion of the EMG signal and its maximum value are determined. 5. The method according to p. 1, characterized in that the processing of the EMG signal occurs in three levels. 6. The method according to p. 1, characterized in that they use an artificial neural network. 7. The method according to p. 1, characterized in that in the input layer for an artificial neural network the number of neurons is equal to the number of channels. 8. The method according to p. 1, characterized in that in the output layer for an artificial neural network, the number of neurons is equal to the dimension of the feature space for the metric classifier. 9. The method according to claim 1, characterized in that the metric classifier is trained on an electromyographic reader, but the actuator can be activated if necessary to visually reflect gestures. 10. The method according to p. 1, characterized in that it contains functionality for additional data collection from the gyroscope and accelerometer and their use in conjunction with a neural network hash function. 11. The method according to p. 1, characterized in that it contains a feedback functionality with an actuator that allows using a vibrator to transmit information. 12. A control system for electronic devices using an electromyographic reader, comprising: an electromyographic device for reading the user's EMG signals, configured to register and initially process the user's EMG signal by means of non-overlapping segmentation of the EMG signal; wherein the electromyographic device for reading the user's EMG signals comprises at least one myoelectric sensor configured to register EMG signals, a data transmission device, an ADC, a gyroscope, an accelerometer, a vibromotor, batteries and a microcontroller, configured to form a set of gestural signs based on the square of the dispersion of the EMG signal and the neural network hash function of the EMG signal for classifying gestures according to the steps of the method according to claim 1; an actuator configured to implement actions based on the received control signal and additional information from the gyroscope and accelerometer and provide feedback to the electromyographic device for reading the user's EMG signals; while the actuator may look like a helmet or glasses with a controller on a virtual reality device, a radio-controlled model, a controlled mobile phone, a computer. 13. The system according to p. 12, characterized in that the electromyographic device for reading the user's EMG signals is an elastic bracelet. 14. The system according to claim 13, characterized in that the electromyographic device for reading the user's EMG signals contains electrodes for recording activity of the neuromuscular system, which can be with a separate amplifier on each sensor or have a common amplifier on the ADC, and the sensors themselves may not use an amplifier. 15. The system according to claim 13, characterized in that the electromyographic device for reading the user's EMG signals is located on the forearm or shoulder for the arm, or on the thigh or calves of the leg. 16. The system according to p. 13, characterized in that the myoelectric reader includes a wireless data transmission sensor. 17. The system according to p. 13, characterized in that the electromyographic device for reading EMG user signals contains an electromyogram sensor with or without operational amplifiers, a low-pass and high-pass filter, an analog-to-digital converter and a microprocessor, a voltage converter, a battery, a vibromotor, and a gyroscope and accelerometer. 18. The system according to p. 13, characterized in that the electromyographic reader includes a vibromotor for feedback from the actuator. 19. The system according to p. 13, characterized in that the controller includes a processing unit and an interface unit that is physically separated from the processing unit, and the above blocks can be connected by a flexible cable. 20. The system according to claim 13, characterized in that the electromyographic device for reading the user's EMG signals can operate in the mode of the electromyograph and outputting the EMG data from the electromyographic reader to the dial of the electromyographic reader or to the screen of the actuator.

Images