RU2762775C1 - Method for deciphering electromyosignals and apparatus for implementation thereof - Google Patents

Abstract

FIELD: decoding.

SUBSTANCE: group of inventions relates to deciphering of electromyosignals and an apparatus for implementation thereof. Proposed is an apparatus for implementation of the method, containing a myoelectric reading apparatus, an EMG signal processing unit, an on-board processor deciphering electromyosignals by means of a neural network classifier, and a servo engine controller, connected in series. The electromyosignal deciphering apparatus therein includes a comparator unit, an informative feature computer unit, a multiplexer, a first neural network, a memory unit, and a second neural network, the outputs whereof are intended for connection to the servo engine controller, and a synchroniser connected by the outputs to the control inputs of the computer unit, the multiplexer, the memory unit and the servo engine controller, connected in series.

EFFECT: group of inventions minimises the error rate during positioning of the exoskeleton by means of servo engines controlled by commands received based on the results of decoding of the electromyosignal.

5 cl, 14 dwg

Description

This technical solution relates to the field of medical rehabilitation devices and robotics, in particular, to methods and systems for controlling the exoskeleton in the rehabilitation of people with diseases leading to disorders of the musculoskeletal system and musculoskeletal system, as well as for use as a man-machine interfaces of industrial exoskeletons.

In order to distinguish and classify muscle movements, the most significant parts of the EMG (signs) must be extracted, which represent the current signal in the form of a code that carries information about a specific movement of the limbs. This information is decrypted in a decoder or classifier. According to research data, various coding methods are used to classify EMG signals: spectral coefficients, autoregression coefficients, wavelet coefficients. However, given the time constraints on decoding the command, preference is given to coding an EMG signal in the time domain based on the analysis of signal amplitudes. Such features can be easily identified, have high stability for pattern recognition using neural networks. To achieve the best results in command recognition, the code (feature vector) must contain a sufficient amount of information to reflect the essential properties of the EMG signal. Consequently, the main requirement for the code is its ease of obtaining and the speed of its decoding (command classification). The following values in the time domain, measured as a function of time, are used as signs: integral EMG; average; the average value of the module; finite differences; the sum of elementary areas; dispersion; standard deviation; signal length; the maximum value of the EMG signal. More detailed information on the choice of features for classification is presented in the work [Budko R.Yu., Starchenko I.B. Creation of a classifier of facial movements based on the analysis of electromyogram // Proceedings of SPIIRAS. 2016. Issue. 46. S. 76-89.].

The known method and system for identifying gestures, described in US 20150370333 A1, 24.12.2015. The system for identifying gestures includes at least one sensor responsive to gestures performed by the user, and a processor communicatively connected to at least one sensor. A gesture identification method includes: receiving at least one signal from at least one sensor; segmenting at least one signal into data windows; processor definition of the data window class from the window class library; identifying a high likelihood of a gesture for the data window by the processor corresponding to a gesture in the gesture library that has the highest likelihood that a particular gesture is being performed by the user.

The disadvantage of the system and method is the data classification technology, which involves the correlation of one property of the data window with a certain class, which leads to the binding of several classes to one window and the need to calculate the most probable class for the current window.

As a prototype, a method and a control system for an intelligent bionic limb described in [RF Patent RU 2 635 632 C1. Method and control system of intellectual bionic limb / N.M. Ivanyuk, V.R. Karimov, R. Yu. Budko, P.V. Gronsky, S.M. Clayman. Published on November 14th, 2017. Bul. No. 32]. The method comprises the steps of: receiving at least one EMG signal from a patient by means of a myoelectric reader; processing at least one EMG signal of the patient by non-overlapping segmentation of the EMG signal; for each segment obtained in the previous step, a set of EMG signal features is generated based on the amplitude of the EMG signal to classify gestures; transmitting a set of features of the EMG signal of each segment via the data transmission channel to the intelligent bionic limb control system; determining the type of gesture based on a set of features of the EMG signal by using an artificial neural network; generating a control signal based on a certain type of gesture; transmit the generated control signal to the motors driving the fingers of the bionic limb; receive feedback from the intelligent bionic limb control system by receiving information from external sensors. The invention improves the positioning accuracy and decision-making on the capture of the object.

The main disadvantage of the prototype is that when forming the vector of informative features based on the analysis of the EMG signal, only the amplitude characteristics of the EMG readings are used in the window, which leads to a loss of information content of the EMG signal and a high level of errors in the classification of commands. It is proposed to increase the accuracy of the classification of commands in the prototype by duplicating EMG signals on several channels and decoding the vector of informative signs formed in this way by means of a trained neural network. However, such a technical solution can only be used when picking up signals from the upper or lower extremities and is unacceptable for decoding the EMG of the back muscles, the decoding of which is necessary when controlling rehabilitation exoskeletons in the stand-up mode or industrial exoskeletons in the load lifting mode. In addition, the use of duplicate channels does not lead to an increase in the information content of the signal, but is most likely aimed at increasing its noise immunity. This circumstance entails a decrease in the reliability of the neural network classifier, the inputs of which are fed highly correlated signs obtained on the basis of the analysis of signals that depend not only on the EMG signal, but also on the location of the electrode in the current experiment or on the different dynamics of changes in the skin resistance under the electrodes in the experiment, which cannot be taken into account in the learning process of the neural network.

The technical task of the proposed method and device is to minimize errors in positioning the exoskeleton by means of servo motors controlled by commands obtained from the results of decoding the electromyosignal.

,

, где N-число уровней компарации, вычисляются согласно выражениюThe task is achieved by the fact that in the known method, which consists in obtaining at least one EMG signal from a patient by means of a myoelectric reader; processing at least one EMG signal of the patient by stepwise segmentation of the EMG signal into intersecting or non-intersecting windows; forming for each segment obtained in the previous step, a set of EMG signal features (a vector of informative features); transmitting a set of features of the EMG signal of each segment via a data transmission channel to a classifier that controls a servo motor controller, which is made in the form of a trained neural network; the formation of a vector of informative features for a neural network in it is carried out by means of a multilevel comparator, the number of levels of which is determined by the dimension of the vector of informative features, and the components of the vector of informative features

,

, where N is the number of levels of comparison, are calculated according to the expression

,

,

where TW is the width of the window in samples,

i is the number of the comparator level,

τ-number of the sample in the i-th window;

Θ _i is the value of the i-th level of comparison;

x _τ - readout of the EMG signal with the number τ in the TW window;

and the classifier uses a second neural network designed to generalize the data obtained during the classification of the vector of informative features of the current window, the number of inputs of which is determined by the number of EMG windows used when deciding whether to turn on / off the corresponding servo motor.

In order to adapt the functional state of the patient and the process of verticalization by means of an exoskeleton, the decoder uses many duplicate channels of EMG signals associated with a muscle or muscle groups that control the movement of the same joint of the extremities, as a result of which, at the output of the classifier of each channel, we obtain a number corresponding to confidence in the command to rotate the exoskeleton servo motor, and for aggregation of solutions through the channels of the classifiers, all outputs of the channel classifiers are fed to a fuzzy neural network, the defuzzifier of which generates a control signal to the servo motor controller, as a result of which the controller determines the speed and direction of rotation.

at the decision of the decision maker, when classifying the EMG signal, at each decision-making step, the data from the first neural network is updated completely, or only the output of the neural network that came first is updated, and all other components of the vector of informative features of the second neural network are shifted by one step.

In order to adapt the patient and the exoskeleton to the control device for the servo motors of the exoskeleton, which contains a serially connected myoelectric reader, an EMG signal processing unit, an onboard processor that decrypts electromyosignals by means of a neural network classifier, and a servo motor controller, an electromyosignal decoder is additionally introduced into it, which includes serially connected block of comparators, block of calculators of informative signs, multiplexer, first neural network, memory block and second neural network, the outputs of which are intended to be connected to the controller of servo motors, and a synchronizer, outputs connected to the control inputs of the block of calculators, multiplexer, memory block and controller of servo motors. In order to be able to adapt the control device to the functional state of the patient, additional channels are introduced into it, each of which is an EMG signal classifier associated with a specific muscle or a group of muscles that control the movement of the same joint of the limbs, and a third neural network, connected to the outputs of the channel classifiers, and the output to the servo motor controller.

Figure 1 shows a block diagram of a device that implements the proposed method.

FIG. 2 shows a block diagram of an EMG signal processing unit.

FIG. 3 shows a block diagram of an EMG signal window classifier.

Figure 4 shows a diagram of an algorithm for implementing the method for classifying EMG using one channel as an example.

FIG. 5 illustrates the transition from the EMG signal x (t) (Fig. 5a) to the EMG signal y (t) (Fig. 5b).

Figure 6 shows an illustration of the number of analysis windows: a) for windows without overlapping, b) for overlapping windows

FIG. 7 shows the diagrams of the formation of control pulses at the output of the synchronizer.

FIG. 8 shows the process of rehabilitation by means of verticalization of the patient, carried out by means of a simple device.

FIG. 9 shows simplified kinematic diagrams of the exoskeleton for the verticalization mode: 1 - lower leg, 2 - thigh, 3 - spine, 4 - foot; fig. 9a - the initial state of verticalization, FIG. b - intermediate state of verticalization, Fig. 9c - final state of verticalization.

FIG. 10 shows the anatomical atlas of the muscles of the lower extremities, from which EMG is taken - signals that control the work of the joints during the verticalization of the patient by means of the exoskeleton - front view.

FIG. 11 shows the anatomical atlas of the muscles of the lower extremities, from which EMG is taken - signals that control the work of the joints during the verticalization of the patient by means of the exoskeleton - side view.

FIG. 12 shows the anatomical atlas of the muscles of the lower extremities, from which EMG is taken - signals that control the work of the joints during the verticalization of the patient by means of the exoskeleton - rear view.

FIG. 13 shows a block diagram of the EMG classifier when using duplicate channels.

FIG. 14 shows the graphs of membership functions for the control terms of the first servo motor: 1 - "sit down quickly", 2 - "sit down at an average pace", 3 - "sit down slowly", 4 - "get up slowly", 4 - "get up slowly", 5 - "Get up at an average pace", 6 - "get up quickly."

The method is carried out using a device, the structural diagram of which is shown in Fig. 1. The device of FIG. 1 consists of a microelectrode reader 1, an EMG processing unit - signal 2; onboard processor 3, and servo motor controller 4.

The EMG signal processing unit of FIG. 2 consists of a series-connected block of comparators 5, a block of calculators 6 and a multiplexer 7.

Classifier 8 of FIG. 3 decrypts the electromyosignal, as a result of which control commands are generated to the controller 4. It is implemented by means of the software of the on-board processor 3 and consists of a series-connected first neural network 9, a memory unit 10 and a second neural network 11, the interaction of which is controlled by the synchronizer 12, the first output which is connected to the second input by the memory unit 10, and the second output strobes the inputs of the controller 4.

The method is implemented by means of a sequence of program modules 13 ... 38 (Fig. 4). The essence of the method lies in the segmentation of the current EMG signal into intersecting or non-intersecting windows of width TW, followed by the formation of an informative feature FD _{from the samples of each window x τ.} For this purpose, a transition is made from the current discrete EMG reading x _τ to the y _{τ reading} obtained by comparing the current reading with the threshold Θ and calculated according to the expression

(1)

(one)

Based on the readings (1), the informative feature in the TW window is calculated as

. (2)

... (2)

The plots in Fig. 5 shows how from the EMG signal x (t) of FIG. 5a, the signal y (t) of FIG. 5b for one threshold Θ. A set of thresholds Θ can be used on the window width and, as a consequence, a set of informative features FD can be obtained.

The procedure for transforming the EMG signal samples according to (1) is carried out by a block of comparators 5. Thus, one _{sample x τ} is associated with N samples of y _τ , where N is the number of comparison thresholds Θ. This procedure can be carried out both at the hardware and software levels. In the flow diagram of Fig. 4, it is represented by block 24 in the form of parallel computation (1), which corresponds to the hardware implementation level. In the calculators of the block of calculators 6, the procedure for calculating the informative feature (2) is implemented. In the flow chart of FIG. 4 is block 25. Through multiplexer 7, the resulting N-dimensional vector of informative features is fed to the input of the first neural network NET1. After the classification of the windowed electromyosignal in NET1, its corresponding solution is entered into the corresponding memory cell of the memory unit 10. After that, it is necessary to skip W samples and perform a similar procedure (cycle block 19 in the algorithm diagram of Fig. 4). The number of analyzed windows MW required for making a decision is entered in block 15 (Fig. 4). If W> TW, then non-overlapping windows are used. Otherwise, overlapped windows are used.

These informative features are the inputs of the learning network NET1, which is used as a decoder of EMG signals (block 11 in Fig. 3).

Given the EMG aperture (blocks 14 and 15 of Fig. 4), on which a decision is made to turn on the corresponding servo motor, we can select on it a set of windows, and, consequently, a set of NET1 outputs. Since the outputs of NET1 are spaced in time, to make a decision on the totality of the results of these outputs, memory devices (block 10 of FIG. 3) are needed, in which these results are stored. After analyzing the corresponding number of windows, the results of this analysis are fed to the trained neural network NET2 (block 11 of Fig. 3), the outputs of which are connected to the controller of the servo motors 4 (Fig. 3).

The number of analysis windows, that is, NET2 inputs, depends on the decryption time of the command encrypted in the EMG signal and on the minimum information quantum carried by the EMG signal. According to the literature, the time aperture for decoding the EMG signal is 250 ms, and the minimum information quantum, that is, the minimum segment of the EMG signal that carries the relevant information, is 25 ms. Thus, when using non-overlapping windows, MW = int (250 / W). If overlapped windows are used, then MW = int ((250-TW) / W). The process for determining the number of analysis windows is illustrated in FIG. 6. In FIG. 6a shows a process for forming windows without overlapping, and FIG. 6b - the process of forming overlapped windows.

, где

. Так как формирование этого вектора информативных признаков разнесено во времени, то на контроллер 4 должен быть подан стробирующий сигнал E2, сигнализирующий о завершении формирования этого вектора. The synchronizer 12 (Fig. 3) synchronizes the operation of the EMG signal processing unit 2 and the classifier 8, so that a sequence of vectors of informative signs characterizing the time windows is formed. At the observation aperture of the EMG signal corresponding to the MW analysis windows, at the input NET2, it is necessary to form a vector of informative features

, where

... Since the formation of this vector of informative features is spaced apart in time, a strobe signal E2 must be sent to the controller 4, signaling the completion of the formation of this vector.

выходами NET1 и разрешает контроллеру серводвигателей 4 исполнять команды, поступающие с NET2.Plots of signals at the output of the synchronizer are shown in Fig. 7. In FIG. 7a shows the windows of samples of the x (t) EMG signal. This example shows the case with non-intersecting windows. FIG. 7b shows strobe signals supplied to the multiplexer and the memory unit. These signals toggle the multiplexer address registers and the memory block. FIG. 6c shows a strobe signal that signals that the relevant information has appeared on the NET2 outputs. The strobe is formed after the vector is formed

outputs NET1 and enables the servo motor controller 4 to execute commands from NET2.

After turning on the corresponding servomotor, the user can choose two options for further work (block 16 of Fig. 4). The choice of these options is carried out by means of the CASE statement in block 30. If CASE is equal to 3, then the transition to block 19 is carried out, in which the following MW windows are analyzed. Therefore, the next command to the servomotor can only be sent through TWxMW counts. This slows down the decision-making process. Therefore, the algorithm provides for branch 2, according to which, when making decisions in the neural network NET2, only the input corresponding to the first result of the analysis of the neural network NET1 changes. The rest of the NET2 inputs are shifted by one, as a result of which the last report is replaced by the first sample of the next window, and the first sample of the previous window is replaced by the second sample of the previous window. This procedure is implemented by blocks 35 and 36 of the algorithm of FIG. 4.

An example of the implementation of a technical solution

In recent years, devices called exoskeletons (exoskeletons), that is, external skeletons, have become more common. An exoskeleton is a device designed to expand human functionality through an external frame. The main task of such devices is to assist a person when moving in space, including when walking. With the help of an exoskeleton, the tasks of expanding the functional capabilities of patients with disabilities of the musculoskeletal system, who have suffered as a result of accidents or various diseases that exclude normal human movement, are solved. One of the rehabilitation methods is verticalization with the help of an exoskeleton, that is, the transition from a "sitting" position to a "standing" position.

The clinical use of verticalization has proven to be highly effective in carrying out rehabilitation therapy for a wide variety of neurological disorders. Verticalization can effectively improve and stabilize the indicators of the cardiovascular system, normalize the breathing process, improve muscle mobility, activate work and improve the innervation of the musculoskeletal system. At the same time, verticalization not only speeds up the rehabilitation process, but also significantly reduces the risk of secondary complications associated with prolonged immobility of the patient. It is also prescribed for patients with cerebral palsy, spinal dysraphism, muscular dystrophy, multiple sclerosis, etc. with a tendency to form flexion contractures in the hip and knee joints in children who walk little or do not walk at all, but move by crawling; the presence of hip dysplasia or a condition after hip surgery; cognitive impairment, lack of motivation of the child to an upright position; spasticity and violation of the muscle tone of the lower extremities, which do not allow the child to stand independently without making pathological movements and adopting pathological poses. Verticalizers also contribute to the development of motor actions and the development of the functions of the upper extremities. These devices are shown for young children - from 9 to 12 months.

With the systematic use of training devices, a passive and active effect on the musculoskeletal system (muscles, joints) is exerted, the activity of brain structures (hypothalamus, motor centers of the cerebral cortex and other departments) is stimulated or normalized, the activation of which contributes to the maintenance of an upright posture, locomotor acts , manipulating objects, etc.

With the correct and systematic use of verticalizers, the balance and mobility of the processes of inhibition and excitation in the central nervous system increases, motor-visceral reflex connections are normalized, muscle tone decreases, and coordination of movements improves.

Many foreign and domestic companies engaged in the production of rehabilitation equipment offer a wide range of verticalizers designed for children of different ages, as well as adults with different levels of physical activity, locomotor disorders, etc.

Thus, the use of verticalization allows you to simultaneously achieve several goals:

• prevent the formation of flexion contractures in the hip and knee joints;

• help in the prevention of cognitive impairment, lack of motivation of the child to an upright position, spasticity and impaired muscle tone of the lower extremities, which do not allow the child to stand on his own without making pathological movements and adopting pathological postures;

• help to master motor actions and develop the functions of the upper extremities;

• restore the muscle strength of the legs, avoid muscle atrophy;

• restore walking skills;

• smoothly adapt the cardiovascular system to physical activity;

• to prevent bedsores, pneumonia, urinary disorders, defecation and other complications associated with prolonged immobility.

Despite the numerous rehabilitation exoskeletons on the market of medical services, today there are no effective algorithms that ensure stable movement of the patient in the exoskeleton in the process of verticalization; the questions of mathematical modeling of the patient's behavior in the exoskeleton remain open. The main complaint to medical exoskeletons on the part of specialists is the lack of the ability to adapt the exoskeleton to a specific patient; therefore, they often prefer cheaper and more effective means of rehabilitation based on the technology of patient verticalization.

FIG. 8 shows the process of verticalizing a patient with a hand support, which acts as the simplest rehabilitation product. Despite the prostate of such a rehabilitating device, very often they give preference to it and refuse rehabilitating exoskeletons due to the problems of their control and the difficulty of adapting to the indicators of the patient's functional state.

FIG. 9 shows a simplified kinematic diagram of an exoskeleton for verticalization mode. For simplicity of calculations, we assume that φ1 = 90 ° and does not change during verticalization (this case is shown in Fig. 9), then the verticalization process can be carried out using two servomotors SD1 and SD2, controlling the angles φ2 and φ3, respectively.

To control the process of verticalization, electromyosignals are used, received by means of electrodes installed in the area of those muscles that are involved in verticalization. An anatomical atlas of these muscles with marked places for possible electrode placement is shown in Fig. 10, figs. 11, fig. 12.

To control the servomotors during verticalization, we use five EMG channels, which read the EMG signal from the gluteus maximus muscle, the biceps femoris muscle, the semimembranosus muscle, the semitendinosus muscle, and the adductorus major muscle. The verticalization process is controlled by decoding the EMG in the channels and the verticalization rate depends on the intensity of the EMG signals in the channels. With certain combinations of EMG intensities in the channels, the servomotors can stop at any phase of verticalization. In this case, feeling that he “does not have the strength to stand up,” the patient will want to sit up. In this case, it is possible to generate appropriate EMG signals that can be used to reverse the servomotors. To implement the "sit down" mode, EMG channels from the iliopsoas muscle are used (Fig. 12). Thus, for the verticalizer, as a decoder of EMG signals, it is necessary to use at least six EMG channels and one more neural network structure used as an aggregator of decoders of channel EMG. The block diagram of such a decoder is shown in Fig. thirteen.

The neural network NET3 is a fuzzy neural network. Its outputs are calculated by defuzzifying six membership functions, which for the first servo motor are shown in FIG. 14. The second servo motor is controlled by the servo motor controller based on the model of movement of the exoskeleton in the "stand up and sit down" mode. The parameters of the model are adapted to the physical condition of the patient, which eliminates the main disadvantage of the known rehabilitating exoskeletons.

Claims (13)

1. A method for decoding electromyosignals, which consists in receiving at least one electromyosignal of a patient by means of a myoelectric reader; processing at least one electromyosignal of the patient by stepwise segmentation of the electromyosignal into intersecting or non-intersecting windows; forming for each segment obtained in the previous step, a set of electromyosignal features; transmitting a set of features of the EMG signal of each segment via a data transmission channel to a classifier that controls a servo motor controller, which is made in the form of a trained neural network; characterized in that the formation of the vector of informative features for the neural network is carried out by means of a multilevel comparator, the number of levels of which is determined by the dimension of the vector of informative features, and the components of the vector of informative features

where N is the number of levels of comparison, calculated according to the expression

where TW is the width of the window in counts, i - number of the comparator level, τ is the number of the sample in the i-th window;

Θ _i is the value of the i-th level of comparison; х _τ - readout of the EMG signal with the number τ in the TW window; and the classifier uses a second neural network designed to generalize the data obtained during the classification of the vector of informative features of the current window, the number of inputs of which is determined by the number of EMG windows used when deciding whether to turn on / off the corresponding servo motor. 2. The method according to claim 1, characterized in that the decoder uses duplicate channels for the classification of electromyosignals associated with a muscle or muscle groups that control the movement of the same joint of the limbs, as a result of which at the output of the classifier of each channel a number corresponding to confidence in to a command to rotate the exoskeleton servo motor, all outputs of the channel classifiers are fed to a fuzzy neural network, the defuzzifier of which generates a control signal to the servo motor controller, as a result of which the controller determines the speed and direction of rotation. 3. The method according to claim 1, characterized in that, at the request of the decision-maker, when classifying the EMG signal at each decision-making step, the data from the first neural network is completely updated, or only the output of the neural network that came first is updated, and all other components of the vector of informative features of the second neural network are shifted by one step. 4. A device for controlling servo motors of an exoskeleton with decoding an electromyosignal according to the method of claim 1, comprising a serially connected myoelectric reader, an EMG signal processing unit, an onboard processor that decrypts electromyosignals by means of a neural network classifier, and a servo motor controller, characterized in that the decoder of electromyosignals is connected in series connected block of comparators, block of calculators of informative signs, multiplexer, first neural network, memory block and second neural network, the outputs of which are intended to be connected to the controller of servo motors, and a synchronizer, outputs connected to the control inputs of the block of calculators, multiplexer, memory block and controller of servo motors. 5. The device according to claim 4, characterized in that additional channels are introduced into it, each of which is a classifier of an EMG signal associated with a specific muscle or a group of muscles that control the movement of the same joint of the limbs, and the third neural network, inputs connected to the outputs of the channel classifiers, and the output to the servo motor controller.

Images