RU2635632C1 - Method and system of intellectual bionic limb control - Google Patents

Abstract

FIELD: medicine.

SUBSTANCE: method of intelligent bionic limb control contains the stages of: receiving at least one EMG signal of the patient via myoelectric reader; processing at least one EMG signal of the patient via a non-overlapping segmentation of EMG signal; for each segment, obtained in the previous step, forming the set of features of EMG signal based on the amplitude of the EMG signal for classify gestures; transmitting the set of features of EMG signal of each segment on the data channel to the intelligent bionic limb control system; determining a gesture type based on sets of features of EMG signal through the use of an artificial neural network; forming a control signal based on the determined type of gesture; and transmitting the generated control signal to the motors driving the fingers of bionic limbs; receiving feedback from the intelligent bionic limb control system by retrieving information from external sensors.

EFFECT: invention allows to increase the accuracy of positioning and making decisions about capturing an object.

13 cl, 13 dwg

Description

FIELD OF TECHNOLOGY

[0001] This technical solution relates to the field of medical technology, namely to prosthetics, in particular to methods and systems for controlling an intellectual bionic limb, and is intended for prosthetics of people with loss of limb, as well as for using parts of the product independently in various tasks of improving quality of life.

BACKGROUND

[0002] The patent of the Russian Federation No. 2506931 "Hand prosthesis" is known from the prior art, date of publication: 10.11.2013. This solution describes a brush prosthesis containing a receiving sleeve, body, frames of the first finger, second to fourth fingers, the drive system of the mobility function "grip-opening of the brush", the joint of flexion in the wrist joint, the cuff on the forearm, three potentiometers, three signal converters, an adder, two functional converters and four power amplifiers. This solution has four-finger micro drives (except for the little finger) with the drives attached to the frame and a worm gear on the fingers. The disadvantage of this prosthesis is a mechanical control method, because it requires significant muscle effort, in addition, this method of control is not physiological for humans. For example, the task of taking a coin from the table, which is not difficult for a person with an intact hand, is impossible for such a decision.

[0003] Chinese Patent Application No. CN1582866 (A) “Myoelectric bionic artificial hand with thigmesthesia and its control”, patent holder: HANGZHOU ELECTRONIC SCIENCE AN [CN], publication date: 23.02.2005 is also known in the art. This solution describes a myoelectric bionic artificial arm with touch control, containing measuring electrodes mounted on the residual muscles of the stump, analog-to-digital conversion data collected by a computer to complete the output of the motion recognition of the control signal by the EMG control circuit for the electric drive of the artificial arm, processing system signal and touch control system by means of a piezoceramic plate. In the present invention, to obtain sensory information according to the claims, the following characteristics in the time domain are used: arithmetic mean and variance. But this solution requires a lot of concentration on the part of the user to accurately position and capture the subject. In addition, for a more complete feedback requires the transfer of a larger number of parameters about the subject, for example, its temperature.

[0004] Current prosthetic control systems typically include motors, sensors, controllers, power supplies, mechanical and electrical connectors, interfaces and feedback mechanisms, all of which are optional accessories.

[0005] Currently, there is a continuing need to improve prostheses of the upper limb that would be able to restore the full motor functions and sensory abilities of the amputated limb.

ESSENCE OF TECHNICAL SOLUTION

[0006] The technical problem solved in this technical solution consists in the implementation of an autonomous, fully functional control for performing low-level tasks of manipulating an intelligent bionic limb prosthesis, which would make it possible to perceive the specified prosthesis as a natural limb.

[0007] The technical task of the technical solution is to develop:

[0008] a reliable functional intelligent bionic limb prosthesis with tactile feedback controlled by neuromuscular signals,

[0009] - a method and system for controlling an auxiliary device, such as a limb prosthesis.

[00010] The technical result manifested in solving the specified technical problem and problem is to increase the accuracy of positioning and making decisions about the capture of the subject.

[00011] Also, an additional technical result is a decrease in the cognitive load on a person, an increase in the efficiency of controlling the prosthesis, a reduction in the response time of the actuators, and prevention of false responses of the prosthesis due to the signal preprocessing method.

[00012] Another additional technical result is the simplification of the installation and configuration of the prosthesis-reader system due to the modularity of the system — the reader is portable, and requires approximate positioning when placed, and is easily removable. The prosthesis, in turn, is worn through a body receiver, and is connected to the reader only with a wireless data channel.

[00013] Reducing the cognitive load on a person and increasing the efficiency of controlling the prosthesis is achieved through the use of a hybrid control system that combines methods for decoding electroneuromyosignals with elements of autonomous robotic manipulations to capture an object when a threshold distance to it is reached by making a decision that does not require participation the user about the beginning or termination of the grip using information from additional sensors (temperature and distance ’), and about the compression force of an object using information from sensors.

[00014] The accuracy of the classification and prevention of false alarms of the prosthesis is achieved by simultaneously generating the maximum value and the average duration of exceeding the threshold value of the neuromial signal in the data window in real time, then after centering and normalization, a decision is made about the occurrence of a control signal.

[00015] The location of the microcontroller, made with the possibility of modular assembly in the prosthesis, allows you to bring the artificial limb in weight and size to natural.

[00016] Using feedback provided by temperature, distance, moment, and position sensors located in the prosthesis allows you to secure the use of the prosthesis and provide maximum control capabilities.

[00017] The use of wireless data transfer between the prosthesis and the reading system provides the usability of the system and the interchangeability of components, in addition, the possibility of their use independently of each other.

[00018] The specified technical result is achieved due to the method of controlling the intellectual bionic limb, in which at least one EMG signal of the patient is received by means of a myoelectric reader; processing at least one patient EMG signal by means of non-overlapping segmentation of the EMG signal; for each segment obtained in the previous step, a set of features of the EMG signal is formed based on the amplitude of the EMG signal for classifying gestures; transmitting a set of features of the EMG signal of each segment over the data channel to the intelligent bionic limb control system; determine the type of gesture based on a set of signs of the EMG signal by using an artificial neural network; generating a control signal based on a certain type of gesture; transmit the generated control signal to the engines, driving the fingers of the bionic limb; receive feedback from the control system of the intellectual bionic limb by receiving information from external sensors.

BRIEF DESCRIPTION OF THE DRAWINGS

[00019] The features and advantages of this technical solution will become apparent from the following detailed description and the accompanying drawings, in which:

[00020] FIG. 1 (a, b) presents an example of the configuration and functioning of the control system of the intellectual bionic limb.

[00021] FIG. 2 describes the general structure of a technical solution.

[00022] FIG. 3 is a block diagram of an implementation of an intelligent bionic limb control system.

[00023] FIG. 4 represents an example of a data acquisition implementation scheme in the form of a myoelectric reading system (item 14 — reading system, item 15 — biopotential recording sensor, item 16 — elastic cuff).

[00024] FIG. 5 shows the components of a bionic prosthesis of the hand in the form of a myoelectric reading system (pos. 17), a body receiver (pos. 18) and a hand prosthesis (pos. 19) and their approximate location on the limb.

[00025] FIG. 6 shows an example of an electromyographic signal recorded with a volitional effort from the muscles of the forearm.

[00026] FIG. 7 shows an exemplary embodiment of an EMG sensor using a dry contact more preferred for long-term use.

[00027] FIG. 8 is an example of placing feedback sensors on an intelligent bionic limb.

[00028] FIG. 9 (a, b) represents the finger drive mechanism in a straight and bent state.

[00029] position 6 gear.

[00030] position 7 Threaded plunger.

[00031] pos. 8 Nut.

[00032] position 9 Lever actuator.

[00033] Pos. 10 The gear is large.

[00034] pos. 11 Micromotor gear.

[00035] Pos. 12 Joint-knot of rotation fixed on the metacarpus.

[00036] pos. 13 The first phalanx of the finger.

[00037] position 14 The knot of rotation of the finger.

[00038] position 15 The second phalanx of the finger.

[00039] FIG. 10 shows the placement of the control board and drives in the brush of an intellectual bionic limb.

[00040] FIG. 11 shows an example of processing a signal from a reading system.

[00041] FIG. 12 shows time functions for extracting features of an EMG signal.

[00042] FIG. 13 shows data capture on an orthonormal basis using rotation. Left: neuron trying to adapt to cover new data. Right: end position of a neuron after a new data coverage.

DETAILED DESCRIPTION OF THE TECHNICAL SOLUTION

[00043] Below will be described the concepts and definitions necessary for the detailed disclosure of the ongoing technical solution.

[00044] The technical solution may be implemented as a distributed computer system.

[00045] In this solution, a system means a computer system, a computer (electronic computer), CNC (numerical control), PLC (programmable logic controller), computerized control systems, and any other devices that can perform a given, well-defined sequence of operations (actions, instructions).

[00046] An instruction processing device is understood to mean an electronic unit or an integrated circuit (microprocessor) that executes machine instructions (programs).

[00047] An instruction processing device reads and executes machine instructions (programs) from one or more data storage devices. Storage devices may include, but are not limited to, hard disks (HDD), flash memory, ROM (read only memory), solid state drives (SSD), optical media (CD, DVD, etc.).

[00048] A program is a sequence of instructions for execution by a computer control device or an instruction processing device.

[00049] Electromyography (EMG, ENMG, myography, electroneuromyography) is a method for studying the bioelectric potentials that arise in the skeletal muscles of humans and animals upon excitation of muscle fibers; registration of muscle electrical activity.

[00050] A culture receptacle is a part of a limb prosthesis intended to house a stump.

[00051] Electroneuromyography (ENMG) is a comprehensive electrophysiological study necessary to determine the functional state of the peripheral nervous system and muscles [1].

[00052] A method of physiological control of an intellectual bionic limb based on gesture recognition (processing a control signal) is as follows.

[00053] Pre-processing the signal and creating an input vector of signs of the EMG signal to classify the patient's gestures, which can be performed in a similar way during setup and during system operation.

[00054] In order to distinguish and classify muscle movements, the most significant parts of EMG (signs), which are characteristics with sufficient information for classification, must be extracted. According to studies, various types of functions in the frequency domain are used to classify EMG signals of the upper extremities, such as autoregressive coefficients, cosine Fourier transform coefficients, and wavelet coefficients.

[00055] More suitable for analysis are signs of an EMG signal in the time domain based on signal amplitudes. Such features can be easily identified, have high stability for pattern recognition using EMG. To achieve better results, the function (feature) should contain enough information to represent the essential properties of the EMG signal, and should be simple enough for quick processing and classification. In some embodiments, the following time-domain values, measured as functions of time, may be used as features: integral EMG; average; the average value of the module; finite differences; sum of elementary areas; dispersion; standard deviation; signal length; the maximum value of the EMG signal, in more detail, information on the selection of signs for classification is disclosed in the authors [2].

[00056] Formulas for calculating these values are shown in FIG. 12.

[00057] The evaluation of the effectiveness of signs is carried out according to two main parameters - the performance of the neural network and the operating time - as the most important for real-time use. According to previous studies described in the information source [3], for the conditions for fulfilling the requirements of real-time operation, the signal recognition time should take no more than 25 ms. For comfortable work of the user, the productivity or recognition accuracy (the percentage of correct classification cases to all cases considered) should be at least 95%, as shown in the source [4]. An example implementation of signal processing is shown in FIG. 11. In order to improve the accuracy of the classification of the gesture and prevent false triggering of the bionic limb at the signal pre-processing stage, simultaneously with calculating the maximum value of the EMG signal segment, the average duration of exceeding the threshold value of the EMG signal in the data window in real time is calculated, then after centering and normalization of the received signal, a decision is made on the occurrence of a control signal.

[00058] Neuromuscular activity data (matrix data of electromyograph channels) are obtained from the myoelectric reader and digitized by ADC, then the condition for the appearance of the active signal (corresponding to the gesture) is checked, for example, 30% of the amplitude specified in the settings. Next, the received digitized signal is broken by a non-overlapping window into segments, for example, 25 ms long. In some embodiments, the implementation of the technical solution uses the function of the maximum value of the segment, as the most effective and easily calculated. For each segment, its maximum value is calculated by the formula:

[00059]

[00060] where x _k is the kth segment of the signal, x _i is the value of the segment. Further, the obtained values are normalized as follows:

[00061]

[00062] where X is the signal data array, X _i is the signal value;

[00063] And center:

[00064]

- среднее арифметическое сигналах X.[00065] Where

- the arithmetic mean of the signals X.

[00066] Thus, an input feature vector (by which gestures are classified) is obtained for each channel with a lower dimension than the original signal (depending on the window width): a vector of length k instead of the original vector of length k * i, which reduces the computing load on the microcontroller. The dimension is reduced due to the fact that they find one value among i values.

[00067] At the stage of tuning the system after creating the input vector of signs of the EMG signal, training of the artificial neural network (ANN) takes place. In some embodiments, an ANN is used based on, but not limited to, the radial basis function. In some embodiments, a multilayer perceptron of a neural network or a support vector technique may be used.

[00068] In order to recognize gestures, the extracted features must be classified into distinctive classes. The classifier should be able to cope with factors that have a noticeable effect on EMG patterns over time, such as a significant change in EMG signals, the location of the electrodes, sweat and fatigue described in the information source [5].

[00069] The above method uses a three-tier management architecture (database - server - client) when processing a set of features of the EMG signal.

[00070] In the input layer, the number of neurons is equal to the dimension of the feature vector (which in this case is equal to the number of data channels through which data from the sensors are transmitted), which in the example implementation of the invention can be eight: x _i , i = 1, 2 ... 8. The hidden layer, where the number of neurons was not determined in advance, since they were formed during the training procedure, was divided into fourteen sub-hidden layers (fourteen gestures by the number of classes in the training data). In the output layer, the number of neurons was equal to the number of classes in the training data set (fourteen neurons).

The basic function of the neural network in the hidden layer is the Gaussian function, and the output of the kth neural in the hidden layer for each given input X = [x ₁ , x ₂ , x ₃ , x ₄ , x ₅ , x ₆ , x ₇ , x ₈ ] ^T can be calculated using the following formula:

This equation describes an 8-dimensional Gaussian centered at C = [C ₁ , C ₂ , C ₃ , C ₄ , C ₅ , C ₆ , C ₇ , C ₈ , C ₃ ] ^T and rotates along the orthonormal basis {U ₁ , U ₂ , U ₃ , U ₄ , U ₅ , U ₆ , U ₇ , U ₈ }, which allows the neuron to cover the data field of the neighbor without offset or any size change. The width of this Gaussian along each axis is a _i , i = 1, 2 ... 8.

Since the input feature vectors for each image are eight-dimensional, the coordinates corresponding to these vectors represent a basis of the form [1,0,0,0,0,0,0,0,0] ^T , [0,1,0,0,0, 0,0,0] ^T ... [0,0,0,0,0,0,0,0,1] ^T. Thus, the component x _{i of} each input vector X with respect to the new axes can be calculated as:

Rotation along the basis vectors allows neurons to cover all nearby data without increasing the radius. In FIG. Figure 13 shows how a neuron tries to adapt to cover new data; the right side of the figure shows the location of the data by the neuron.

During the setup of the system, the user signal is recorded in the database, the data in which are accumulated, and used to retrain the network. For each user, there is a feature vector with a size of 8 × N (where 8 is the number of channels, N is the number of training data) obtained using the method described above (segmentation by non-overlapping windows). Before the training procedure, each data set is mixed and then divided into sets in a ratio of 2: 1: 1 (according to the number of training data N) with data for the stages of training, testing and control, respectively.

, j=1, …, N вектор средних значений рассчитывается следующим образом:The orthonormal basis is calculated through the eigenvector of the covariance matrix. Since the training data is entered into the network sequentially, the mean vector and covariance matrix are calculated recursively. For N samples, X = {x ₁ , x ₂ , ..., x _N }, in which

, j = 1, ..., N, the vector of average values is calculated as follows:

- это вектор средних значений множества данных X;Where

is the vector of average values of the data set X;

X _{N + 1} is a new data vector added to the data set X.

Then the covariance matrix is calculated as:

To find an orthonormal basis for RBF, the concept of principal component analysis is used. The eigenvalues {λ ₁ , λ ₂ , λ ₃ , λ ₄ , λ ₅ , λ _b , λ ₇ , λ ₈ } and the corresponding eigenvectors {U ₁ , U ₂ , U ₃ , U ₄ , U ₅ , U ₆ , U ₇ , U ₈ } are calculated from the level of the covariance matrix. Further, the set of eigenvalues of the orthogonal vectors form an orthonormal basis, as shown in the information source [6].

The procedure for training a neural network during system setup can be described in more detail as follows.

, где

- множество обучающих данных) и t_j является классом, с которым необходимо соотнести жест. Пусть Ω_k={Ω_k|1≤k≤m} это набор из m нейронов. Каждый нейрон имеет пять параметров:[00071] Let X = {(x _j , t _j ) | 1≤j≤N} be the set N of training data (training set), where x _j is a vector function (component

where

is the set of training data) and t _j is the class with which the gesture needs to be associated. Let Ω _k = {Ω _k | 1≤k≤m} be a set of m neurons. Each neuron has five parameters:

[00072]

[00073] where C ^k is the center of the kth neuron, S ^k is the covariance matrix of the kth neuron, N _k is the amount of data corresponding to the kth neuron, A _k is the width of the vector of the kth neuron, and d _k is class label of the k-th neuron.

[00074] The entire training procedure can be reduced to the following steps.

[00075] The space width vector is initialized. In this case, an eight-dimensional vector of features is used. For simplicity of calculations, we assume the radius of the eight-dimensional Gaussian sphere to be 1: A ₀ = [1; one; one; one; one; one; one; 1] ^T.

,

, N_k=1, d_k=t_j, A_k=A₀. Затем обучающие данные сбрасываются. Если K≠0, ближайший нейрон в скрытом слое Ω_k∈Ω находится так, что d_k=t_j, и

,

; и вектор средних значений и ковариационная матрица обновляются.[00076] Then, a lot of training data (x _j , t _j ) is input to the ANN. When a neuron is not in the network (K = 0), K = K + 1, a new neuron Ω _k is formed taking into account the following parameters:

,

, N _k = 1, d _k = t _j , A _k = A ₀ . Then the training data is discarded. If K ≠ 0, the nearest neuron in the hidden layer Ω _k ∈Ω is such that d _k = t _j , and

,

; and the mean vector and covariance matrix are updated.

[00077] Next, determine the orthonormal basis Ω _k , after which the value is determined at the output of the k-th neuron:

[00078]

- центр нового нейрона,

- координата ортонормированного базиса,

- координата вектора ширины гауссиана.where X _j - je is the value of the input and the target,

is the center of the new neuron,

- coordinate of the orthonormal basis,

- coordinate of the Gaussian width vector.

[00079] If the value of ψ _k (X _j ) ≤0, then the neuron spans the data so that the temporal parameters are tuned to its main parameters. Otherwise, for ψ _k (X _j )> 0, a new neuron is created.

[00080] Since new neurons can be automatically added to the network and located very close to each other, it is possible to implement a strategy of merging several neurons to prevent the network from growing to its maximum structure (one neuron for each data unit), which is described in detail in the information source [7 ].

[00081] A trained neural network allows you to precisely adapt to the patient. As a result of training, ANNs store new values of the neural network weighting coefficients in the prosthesis microprocessor directly or in the data warehouse.

[00082] However, it should be borne in mind that other types of ANNs, as well as other classification methods, can also be carried out in this invention by a specialist in their field.

[00083] During the operation of the system, the result of gesture recognition is converted into the appropriate control command (for example, mental compression into a “fist” by the user - a “fist” on the prosthesis) and transmitted to the bionic prosthesis drives of the hand.

[00084] The physiological control system of an intellectual bionic limb based on gesture recognition (processing a control signal) is as follows.

[00085] The above system consists of a myoelectric biopotential reader, which can be structurally designed as an easily removable elastic bracelet (shown in Fig. 4, item 14). This device contains electrodes for recording the activity of the neuromuscular system (pos. 15) that occurs in response to the phantom movement of the patient. Moreover, the bracelet can be placed on the forearm or on the shoulder and is configured to transmit a control signal to the brush prosthesis.

[00086] This bracelet may include a wireless data sensor, and, without limitation, Bluetooth or Wi-Fi, or ZigBee wireless technology may be used.

[00087] At least one electromyogram sensor (hereinafter referred to as EMG sensor), with operational amplifiers, low-pass and high-pass filters, analog-to-digital converters (ADCs), a microprocessor, a microcontroller, are placed in an easily removable bracelet worn on a shoulder or forearm; voltage converter and batteries.

[00088] The EMG sensor in some embodiments is a set of electrode plates placed on a dielectric substrate. In various embodiments of the technical solution, the electrodes can be made of stainless steel or of slightly polarizing conductive materials (for example, stainless steel grade 12X18H10). In some embodiments, the electrodes can be made in the following sizes: 1 × 1 cm square for the receiving plate (2 pieces) and 1 × 0.5 cm rectangle for the referent, not limited to. One skilled in the art will appreciate that the size of the electrodes can be made in a different size, given that the plate size affects the signal amplitude and crosstalk.

[00089] A biopotential amplifier board is installed on top of the electrode substrate, which may consist of a tool (differential) amplifier (for example, such as INA114), low-pass and high-pass filters made on passive elements in the form of RC circuits or on active elements. Through flexible loops, the sensors are connected to the microprocessor board, where they can be digitized by analog-to-digital conversion and transferred to a microcontroller (for example, 2x 16-bit ADS1115 ADCs and signal preprocessing on the Cortex M4 microprocessor ARM controller).

[00090] In a voluntary contraction of a muscle by a patient with active electrodes, a potential difference in the low frequency range that occurs in the neuromuscular tissue is recorded.

[00091] It should be emphasized the need for a reliable method of removing the control signal. The frequency composition of the electromyogram in individuals with amputation, according to studies, is represented by a highly noisy signal in the range of 0.5-300 Hz and a signal amplitude of 1 μV to 2 mV (Fig. 6).

[00092] The implementation of the technical solution (Fig. 1A) can occur as follows. The patient performs the specified gestures, while the myoelectric reader transmits user data to the data processing device via a wireless data transmission channel (for example, using the Bluetooth 4.0 standard), where the user profile is created. Then, the created profile is recorded on the microcontroller.

[00093] A myoelectric reader performs registration and filtering of an electromyogram (EMG) and data transmission over a wireless communication channel (for example, using the Bluetooth 4.0 standard, but not limited to it) to the brush prosthesis control board, the placement of which is shown in FIG. 10 (item 20). The signal processing method is described in more detail above.

[00094] FIG. 10 is a plan view of a prosthesis of the hand that includes the palm of the hand, wrist and lever portion.

[00095] As shown in FIG. 8, indicators are located in the palmar part of the prosthesis. The prosthesis control board is connected to LED indicators that display different colors: red, green, orange and other colors, to provide various information / feedback to the prosthesis user (for example, if necessary, perform stronger compression or repeat a gesture, or if it is impossible to capture an object from due to inappropriate temperature).

[00096] An intelligent bionic limb control system also includes:

[00097] a controller network (controller network), which provides a modular, reliable and fault-tolerant interaction scheme for exchanging data between the controller and the rest of the control and executive bodies of the bionic prosthesis, and which transfers intentions from the user's neuromuscular system to the controller;

[00098] a power source;

[00099] multi-colored surface light emitting diode, vibration and sound indicators (LEDs) located in the prosthesis, by which through the controller provides feedback to the patient.

[000100] Exemplary embodiments of an intelligent bionic limb control system include a controller 301 (no more than 2 × 3 cm) that receives data about a recognition result, exchanges data and controls finger motors, temperature, position, contact sensors and torque sensors, as well as vibration and color indicators in an intelligent bionic prosthesis.

[000101] The controller, in some embodiments, may include a processing unit,

[000102] an interface unit that is physically separated from the processing unit, wherein the above units can be connected by a flexible cable, which allows for flexible construction of the interface without changing the corresponding software control element.

[000103] This architecture also provides flexible power management for the prosthesis, significantly increasing battery life, as Thanks to such an organization, the currently unused unit is in a state of standby and low power consumption.

[000104] In FIG. 3 shows a block diagram of an intelligent bionic limb control system, which includes a prosthesis controller 301, a tire 302, and at least one finger tire 303 (only one of them is shown in FIG. 3). A bus 302 and a finger bus 303 are connected to the prosthesis controller 301 via an I / O interface 304.

[000105] The prosthesis controller 301 may be one of the components of the prosthesis (not shown in this diagram). The prosthesis controller 301 is the central link for controlling the prosthesis. Thus, the prosthesis controller 301, which can be structurally placed inside the prosthesis (and, in one embodiment, can be completely placed in the palm of the hand), is responsible for the high level of coordinated control of the hand and large motor joints, as well as various functions, related to the internal nuances of the prosthesis.

[000106] Also, if necessary, in some embodiments, the prosthesis controller 301 may be placed outside the prosthesis.

[000107] One of the functions of the prosthesis controller 301 is to control the movement of the prosthesis.

[000108] Accordingly, user intention commands are transmitted by the ENM unit 305 of the interface (electroneuromiointerface), which communicates with the limb controller 301 via the bus 302 and the I / O interface 304, as shown in FIG. 3. Commands from the ENM unit 305 of the interface, as well as information from the large manipulation drive controller 306 and at least one small manipulation drive controller 307 (one of them is shown in FIG. 3), which are received through the bus 302 and the finger bus 303, allows the control method intellectual bionic limb described above, in the controller 301 to generate new commands for controlling large manipulation drives and small manipulation drives.

[000109] The prosthesis controller 301 also provides sensory data transmission from the sensors in the prosthesis using a vibration sensor or a weak electrical pulse.

[000110] This tactile feedback allows the patient to sense external conditions without being limited only by visual or auditory feedback.

[000111] The prosthesis controller 301 also receives and processes sensory information, such as temperature of an object, prosthesis position, torque, and vibration. For this, a temperature sensor 308, a position sensor 309, a contact sensor 310 and a torque sensor 311, respectively, are placed on the surface of the prosthesis.

[000112] As shown in FIG. 3, the small manipulator drive controller 307 receives information from temperature sensors 308, a position sensor 309, a contact sensor 310, and a torque sensor 311.

[000113] In some embodiments, two position sensors 309 (or more) may be connected to the controller 306. In addition, the large manipulator drive controller 306 may have sensors connected, for example, one or more temperature sensors 308 or torque sensors 311, as shown in FIG. 3.

[000114] As shown in FIG. 3, an intelligent bionic limb control system in accordance with an embodiment includes at least one prosthesis fingertip sensor assembly 312 connected to the prosthesis controller 301 via a finger bus 303, and a socket controller 313 connected to the connector of the controller 301 prosthesis through a splint 302.

[000115] The fingertip sensor assembly 312 has a set of sensors (not shown in the figure), including but not limited to: position sensors (eg, potentiometer or Hall sensors), contact sensors (eg, polyvinylidene difluoride contact sensor), and temperature sensors, each of which provides information about a different position, torque, and / or tactile feedback inputs to the prosthesis controller 301. These sensors, distributed throughout the limb, provide inputs for advanced control methods running on the prosthesis controller 301, as well as for tactile user feedback.

[000116] In further embodiments, the prosthesis controller 301 uses a bus 302 to communicate with additional devices and / or accessories, such as radio frequency identification devices (RFID sensors), smartphones, and personal digital assistants, although additional or alternative embodiments of the invention are not limited to these examples.

[000117] The prosthesis of the hand may be a metal-plastic metacarpus frame with five electric motors and five metal frames of the fingers associated with the metacarpus lever-plunger system (mechatronics). The input feature vector is classified by the neural network algorithm as “learning with a teacher” in the Cortex M4 microprocessor installed in the control board pos. 20 cm. FIG. 9a, where each set of EMG signs obtained from volitional muscle contraction is assigned a corresponding control signal for transmission to the prosthesis drive mechanism. When a signal is supplied from the control board 20 (see Fig. 9a) to the motor 11, the nut 8, together with the drive lever, extends (retracts) to the position shown in FIG. 9b, which ensures the state of "gripping" or opening the palm. Each finger has a separate drive (see Fig. 10).

[000118] The operation of the system is as follows. The mioelectric reader in the form of a bracelet is put on the surviving part of the forearm and connected to the computer, the computer is configured, calibrated, and the bionic prosthesis as follows. The training sample is formed as follows. Using the calibration program, the types of gestures, for example, “fist”, “pinch with three fingers”, “grip” key “,” cylindrical grip “, and others that are proposed to be repeated several times by the user for a certain time, are displayed on the screen for example 5 once in one second, after which the obtained data are used to train the recognition algorithm based on the neural network, as described above. A user profile is created with new weights, which is recorded in the microcontroller of the bracelet. Further, the prosthesis and bracelet can work autonomously, based on wireless communications, for example, via Bluetooth 4.0.

[000119] A brush prosthesis is installed in the body kit. The myoelectric reader can be worn before or after installing the prosthesis thanks to the elastic cuff.

[000120] The user can use the movements recorded during calibration. After the movement, the signal is recorded, converted and classified according to the steps described above. Due to the presence of sensors transmitting additional information from the prosthesis, user feedback is performed as described above.

EXAMPLES OF IMPLEMENTATION OF THE TECHNICAL SOLUTION

[000121] The data collection means is located on the user's hand and is simultaneously connected to a personal computer, which is a data processing device. The user, executing commands on the screen, makes 14 gestures (for example, “Fist”, “Incomplete fist, or cylindrical grip”, “Opening the palm”, “Pinch with thumb and forefinger”, “Pinch with thumb and ring finger”, “Turn the palm on yourself ”,“ Turn the palm away from you ”,“ Pistol ”- the forefinger and thumb are straightened, the rest are pressed”, “Fist with the straightened thumb”, “Rotate the forearm onto yourself”, “Rotate the forearm away from you”, “Grip computer mouse ”,“ Turnkey Capture ”,“ Relaxation ”), each for one second und, repeating them according to the images on the screen, thus forming a training sample of gestures.

[000122] The sensor electrodes record the potential difference that occurs when muscles contract, then the signal is amplified 1000 times, filtered in the 5-500 Hz band and digitized in the ADC. Further, in the setup mode, the digitized signal is transmitted to the computer. Each digital signal sample for each gesture is associated with the number of the performed gesture for further training. The data for each gesture are connected into a matrix, respectively, at a sampling frequency of, for example, 1000 Hz, we have a data set of size 14 × 1000 × 8 × 14000 (14 gestures per second x 1000 Hertz, 8 - the number of channels, 14000 - target vector from 1 to 14, a gesture number assigned to each sample, 1000 values for each target), the signal amplitude is in the range from 100 μV to 1 mV. Then the signal is segmented by windows with a width of 25 ms, 25 samples per segment, respectively, for a sampling frequency of 1000 Hz. Next, find the maximum on each segment, get the output of a feature vector of size 14 × 40 × 8 × 350 (the dimension of the target vector also decreases). Then the data in the resulting vector are centered and normalized, as described above. The amplitude is reduced to the range [-1: 1].

[000123] Thus, an input feature vector for training a neural network is obtained. Training takes place according to the algorithm described above. Further, the new values of the network weights are written to the controller of the reading system. Then the user can use the system. The data before segmentation and the data after segmentation are stored in a database on a remote computer to allow further research and to replenish the training sample of the neural network. It is also further possible to adjust the correspondence of gestures performed by the user and the grip realized by the prosthesis drives in response to the user's gesture by connecting the prosthesis to a computer.

[000124] Then, during the operation of the system, the user performs any of fourteen gestures, then the signal is recorded and processed. The sensor electrodes record the potential difference that occurs during muscle contraction, then the signal is amplified 1000 times, filtered in the 5-500 Hz band and digitized in the ADC. Further, the signal is fed to the microcontroller of the reading system. A number of data is dialed equal to the width of the window (25 ms), respectively, at a sampling frequency of, for example, 1000 Hz, we have a data vector of size 25 × 8 (25 samples in 25 ms, 8 is the number of channels), the signal amplitude is in the range from 100 μV up to 1 mV. Next, find the maximum of the current segment, get the output vector size of 1 × 8. Then the data in the resulting vector are centered and normalized, as described above. The amplitude is reduced to the range [-1: 1]. Next, the vector is fed to the input of the neural network, which determines the class of gesture. Based on the classification result, a control team is created and transmitted to the prosthesis drives. The resulting grip of the prosthesis can be either similar to that performed by the user or otherwise specified in the settings initially.

USED INFORMATION SOURCES

1. Cowan XL, Brumlik J. Guide to electromyography and electrodiagnostics: Per. from English - M.: Medicine, 2005 .-- 192 p.

2. Budko R.Yu., Starchenko IB Creation of a classifier of facial movements based on the analysis of an electromyogram // Transactions of SPIIRAS. 2016. Issue. 46.S. 76-89.

3. Englehart K, Hudgins B: A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans Biomed Eng 2003, 50 (7): 848-854.

4. Van den Broek EL, Lis'y V, Janssen JH, Westerink JHDM, Schut MH, Tuinenbreijer K: Affective Man-machine Interface: Unveiling human emotions through biosignals. Biomedical Engineering Systems and Technologies: Communications in Computer and Information Science. Berlin, Germany: Springer Verlag; 2010.

5. Maria C. Selection of suitable hand gestures for reliable myoelectric human computer interface / Maria Castro, Sridhar P Arjunan, Dinesh K Kumar // BioMedical Engineering OnLine. - 2015. - 14:30 (April 9, 2015) available at: http://www.biomedical-engineering-online.com/content/14/1/30.

6. Huihui L. Relationship of EMG / SMG features and muscle strength level: an exploratory study on tibialis anterior muscles during plantar-flexion among hemiplegia patients / Huihui Li, Guoru Zhao, Yongjin Zhou, Xin Chen, Zhen Ji, Lei Wang // BioMedical Engineering OnLine. - 2014.13: 5 (January 27, 2014) available at: http://www.biomedical-engineering-online.com/content/13/1/5.

7. Saichon J, Chidchanok L, Suphakant P: A very fast neural learning for classification using only new incoming datum. IEEE Trans Neural Netw 2010.

Claims (35)

1. A method for controlling an intelligent bionic prosthesis of the upper limb, comprising the following steps: A) receiving at least one patient EMG signal arising in response to the phantom movement of the patient by means of a reader; B) process at least one received patient EMG signal by means of non-overlapping segmentation of the EMG signal; B) for each segment obtained in the previous step, form a set of features of the EMG signal based on the amplitude of the EMG signal for classifying gestures; D) transmit a set of features of the EMG signal of each segment over the data channel to the database of the neural network for accumulation and further training of the neural network and to the reader; D) if the patient intends to make a gesture, determine the type of gesture based on the generated set of signs of the EMG signal through the use of an artificial neural network; E) form a control signal based on a certain type of gesture; G) transmit the generated control signal to the controller of the prosthesis of the upper limb to transmit the corresponding signals to the engines, which move the fingers of the bionic limb; H) receive feedback from the control system of the intellectual bionic limb by receiving information from external sensors. 2. The method according to p. 1, characterized in that upon receipt of an EMG signal, a potential difference in the low frequency range is recorded that occurs in the patient’s neuromuscular tissue with the intention of making a gesture. 3. The method according to p. 1, characterized in that upon receipt of the EMG signal, the reading device filters the signal. 4. The method according to p. 1, characterized in that during segmentation of the EMG signal for each segment determine its maximum value. 5. The method according to claim 1, characterized in that they use an artificial neural network based on a radial basis function. 6. The method according to claim 1, characterized in that in the input layer for an artificial neural network the number of neurons is equal to the dimension of the feature vector. 7. The method according to claim 1, characterized in that in the output layer for an artificial neural network, the number of neurons is equal to the number of classes in the training data set. 8. The control system of the bionic prosthesis of the upper limb using an artificial neural network, containing: an EMG reader for patient signals arising in response to a phantom movement of the patient, and primary processing of said signals; at least one data processing device for creating a patient profile in which a set of signs of an EMG signal is generated for classifying gestures according to steps AB of the method according to claim 1; a neural network database, including at least data on EMG signals after initial processing and on a set of signs of EMG signals for classifying gestures; upper limb prosthesis that has - a controller configured to control independent finger drives, - one or more sensors connected to the controller and selected from the following: temperature, position, contact, torque sensor; - light emitting diodes of various colors (LED), vibration and sound indicators connected to the controller, wherein the device for reading EMG signals and primary processing of these signals has at least one myoelectric sensor for recording EMG signals, a microprocessor for primary processing of these signals and a microcontroller; the specified microcontroller is configured to - data exchange with a data processing device, - transmitting the control signal to the controller of the prosthesis of the right limb to control the movement of the fingers, - receiving feedback signals from the controller of the prosthesis of the upper limb, wherein the processing of the control signal is carried out according to the method according to p. 1. 9. The system according to claim 8, characterized in that the patient's EMG reader is an easily removable elastic bracelet. 10. The system according to claim 9, characterized in that the bracelet is located on the forearm or on the shoulder of the patient. 11. The system according to p. 8, characterized in that the device for reading the patient's EMG signals contains a wireless data transmission sensor. 12. The system according to p. 8, characterized in that the controller of the prosthesis of the upper limb is placed inside the prosthesis. 13. The system according to p. 12, characterized in that said controller uses data buses to communicate with external devices.

Images