RU2415642C1 - Method of classification of electroencephalographic signals in interface brain-computer - Google Patents

Abstract

FIELD: medicine.

SUBSTANCE: invention relates to field of human brain communication with computer and is intended for EEG registration, analysis and interpretation of brain signals for controlling external execution units. From EEG signal isolated are positive maximums of amplitude of EEG signals from all deviations. If values of two neighbouring positive peaks differ by less than threshold of human psychophysiological perception, they are considered equal and the second peak is excludes from further analysis. Simultaneously with isolation of the first positive peak from support deviation values of amplitudes of EEG signals by all remaining deviations are registered. In teaching multilayer neural network (MNN) additionally formed is array of indices of classes of mental movements, performed by user, who represents outlet array for MNN teaching, weighting coefficients of classification by back-propagation algorithm are calculated. In identification of mental movement array of inlet vectors is supplied to MNN for calculation of outlet vector, used for determination of user's mental movement class.

EFFECT: method makes it possible to reduce time of mental command identification and simultaneously increases accuracy of their identification.

2 cl, 20 dwg, 4 tbl

Description

The invention relates to bioengineering and computer technology, in particular to the field of communication between the human brain and a computer, and is intended for electroencephalographic (EEG) recording, analysis and interpretation of brain signals to control external actuators, for example, a mouse cursor on a monitor screen, wheelchair or specialized prostheses, and can be used in medical diagnostics, camera work, in security systems, in the entertainment industry (computer nye games).

The hardware-software complex brain-computer-interface (Brain-Computer-Interface (BCI)) converts the electrical activity of the brain - impulse and total into a specific action in which the human body does not take any part. Through a computer, a completely paralyzed person can establish contact with others, join actively in the surrounding social environment and, possibly, perform certain tasks, engage in useful and interesting work. In BCI systems based on electroencephalography recorded from the scalp, EEG signals are measured and the necessary areas are selected, while the user presents various movements, for example, movements with the left or right hand. Depending on the BCI, specific pre-treatment and methods of extracting features are applied to the EEG sample of a certain length in order to reliably detect (isolate) limited brain states from EEG signals or EEG fragment patterns with a certain level of accuracy. The problem that arises in this case is based, at least in part, on a limited understanding of the human brain and its inherent electrical activity, and that the accuracy of detecting the state of mental activity deteriorates as the number of mental states increases. For example, it is currently not possible to recognize words that the user may think of, which might be desirable in order to accomplish the dictation task. However, the current state of the art allows reliable detection of several emotional states and / or motor intentions, such as states of relaxation (relaxation) or stress, movement of the right or left hand.

The challenge of creating a reliable BCI has received considerable attention in recent years. Previous research has focused on applying EEG signals to simple command control tasks, such as moving the cursor on a computer screen or controlling the movement of a robot or robotic arm. This type of linear control of the command was created to detect the mental state of the test person from the EEG and convert the detected state into a control signal. The specific parameters of brain biopotentials in the known BCI systems use mu, alpha and beta brain rhythms and evoked potentials (Event Related Potentials - ERPs).

The well-known encephalolexanalizer (US 5840040, IPC AB 5/00, US Cl. 600/545, 600/544, 24-11-1998) / 1 / is designed to detect, interpret and use brain wave signals for communication and control for a thought, such as a mental keyboard typing. The main embodiment of the invention are two control systems based on the movement rehearsal paradigm. The invention measures the decrease in the mu wave, which is separated from other events not related to movement or thinking about movement. The invention detects brain waves, that is, an EEG signal for determining the state of a person: moving or thinking about moving or not moving and not thinking about moving. A pair of electrodes is placed along the motor cortex over the central region of the scalp on both sides of the head. Encephalograph records EEG - potential differences between these two electrodes. When a person is resting, that is, does not move and does not think about moving, a wave occurs, known as the mu wave, usually presented in the 8-13 Hz range. These power suppressions are known as event-related synchronization - ERS (Event Related Synchronization) or event-related desynchronization - ERD (Event Related Desynchronization). When a person moves or thinks about moving any part of the body, mu rhythm is significantly reduced. the system operates on the basis of the attenuation of the mu wave caused by the representation of motion (thinking about moving) or real motion Processing of the digital signal of the EEG wave by the level of spectral power, separated from other waves, is used to generate control The signal can be used for communication or driving various machines.The invention provides two-way communication between the user and the wheelchair, and binary signals are generated from two separately excitable sections of the electrode, using the communication time sequentially. Codex: After enough practice, codes become subconscious in the same way that an experienced typist is not consciously bothered and details of the finger. In the simplest implementation, a dual or on-off system is provided. Although many different body parts could be used, body parts such as arms and legs are preferred. Fingers - a particularly suitable part of the body, the beginning of the movement or rehearsal of the movement of the fingers produces a reliable signal (a big change in the mu wave). In additional implementations, more than two electrodes can be used; measurements can be made on both sides of the head to provide independent left-right control.

In the communication method and the system using brain waves for multidimensional control (US 5638826, IPC АВВ 5/0476, US Cl. 128/732, 340 / 825.19, 345/157, 463/36, 17-06-1997) / 2 / , the control signals for the vertical and horizontal cursor movement on the display screen are isolated by spatio-temporal and frequency analysis and classified using probabilistic statistical processing of digital EEG signals from the accumulated database.

Evoked potentials (ERPs) hidden in a continuous EEG are brain responses that are timed to coincide with the start of an external event. An external event may be a sensory stimulus, such as a visual flash or sound, a mental event, such as recognition of a specific target stimulus, or a missed stimulus, such as an increased inter-stimulus interval. ERPs recorded in response to visual stimuli are called visual evoked potentials (VRPs). Like EEGs, ERPs are recorded by placing electrodes on various parts of the scalp. The high millisecond temporal resolution of ERPs is ideal for exploring aspects of timing in both normal and abnormal cognitive processes such as learning, attention, decision making, reading, language processing, and memory. In clinical studies, ERPs are used to help diagnose brain dysfunction in patients with dyslexia, schizophrenia, Alzheimer's, Parkinson's disease, aphasia, and alcoholism.

The evoked potentials of ERPs are used as specific informative parameters in BCI systems: (WO 2009057260 (A1). G06F 3/01, 2009-05-07) / 3 /, (WO 2009057278 (A1), G06F 3/01, А61В 5 / 0484, А61В 5/0476, 2009-05-07) / 4 /, (WO2008059878 (A1), G06F 3/01, А61В 5/0476, 2008-05-22) / 5 /, (US2005085744, А61В 5 / 00, G06F 3/00, G06F 3/01, 2005-04-21) / 6 /. In apparatus for classifying an unknown multi-channel biopotential signal (WO 2006026548, G06K 9/00, 2006-03-09) / 7 /, biopotential waveforms such as ERPs, EEGs, cardiograms or EMGs are classified by dynamically linking classification information from a plurality of electrodes, tests or other data warehouses. These various data warehouses or channels are evaluated at different points in time according to their respective one-dimensional classification accuracy. The channel ranking is determined during the training phase, in which the accuracy of classification of each channel at each time point is determined. Classifiers are simple one-dimensional classifiers that require only a one-dimensional parameter estimate. Using classification information, a criterion is formulated to dynamically select different channels at different points in time during the testing phase. Independent decisions of the selected channels at different points in time are combined into a vector that is optimally classified using a discrete Bayes classifier. The dynamic system provides high classification accuracy, is very flexible in use and overcomes the main limitations of the classifiers currently used in studies of the biopotential waveform in clinical trials.

A known method for classifying brain signals in the BCI system (WO 2008117145, IPC G06F 3/01, 2008-10-12) / 8 / includes providing a hierarchical multi-level structure of the decision tree, consisting of internal nodes and leaf nodes, where the structure of the decision tree represents the problem. The information obtained from the detected mental states of the user through the levels of the structure of the decision tree is used to reach the leaf node and to complete the task. The operations of the classification method include selecting, using information obtained from the detected mental states of the user, between the values associated with the internal nodes of the decision tree structure by the product of the computer program, and the device that are responsive to the detected mental states of the user in order to perform selection processes to complete the task. The device may be a communication device, and the task may be a call to a task name number or a command / control task. However, during the execution of the task by means of the hierarchical structure of the decision tree, the mental state of the user changes, which reduces accuracy and increases time by repeatedly converting the user's intentions into a signal that controls an external device.

In the improvement related to BCI (WO 2009/044325 A1, 6 MKI G06F 3/01, 09-04-2009) / 9 /, to receive a wheelchair control signal, the user is trained in five commands (left, right, back, forward, stop ) The processor selects five out of ten properties that match the mental description of the task in the user profile, which includes the user's brain signals mapped. The processing function is designed to measure the weighted fatigue component of the user during the training exercise and adapt the BCI accordingly.

In a method and apparatus for determining intentions and monitoring a signal (US 6349231 B1, IPC A61B 5/0482, A61B 5/0484, A61B 5/16, G06F 17/00, A61B 5/0476, 2002-02-19) / 10 / Alpha, beta, theta, delta and mu rhythms of brain activity with combinations with an electrocardiogram, muscle potential, eye movements are used as specific information parameters in BCI. To classify biosignals from 63 inputs, an artificial neural network was used, which has input and output layers that performs linear classification, which is necessary for working with input vectors of 63 received vectors.

The closest in technical essence to the claimed invention is a method for classifying brain signals in the BCI system (WO 2008/097200 A1, 6 IPC A61B 5/0476, C06F 19/00, A61F 2/72, 14-08-2008) / 11 /, taken as a prototype. A known method for classifying brain signals in the BCI system includes the following sequence of actions on EEG signals:

- highlighting features from N1 training tests;

- Calculation of the classification based on the distinguished features of the CSP (Common Spatial Pattern- a common spatio-temporal pattern);

- classification of N2 test tests and repetition;

- preliminary selection of features from all N1, N2 tests;

- preliminary classification calculation based on preliminary allocation of CSP features;

- preliminary classification of N2 test tests;

- and determination of changes (differences) in the preliminary classification from N2 test tests in comparison with the previous repetition equally or a decrease of three times.

The classification calculation is based on the allocation of CSP features in a Bayesian linear classifier.

A known method for classifying electroencephalographic signals includes the following steps: testing a user, extracting specific information components from a common spatio-temporal pattern, creating a sample of digitized EEG fragments from a variety of leads for classifier training, calculating weight coefficients and classifying EEG fragments to identify classes of user mental commands corresponding to control signals.

The well-known BCI system can be used as an alternative way of communication and control for people with severe motor impairments. In particular, the BCI system can also be used to input test information to create a mathematical method for processing a multi-channel signal to extract a control signal from brain activity. For example, a non-invasive EEG-based BCI system can measure specific components of brain activity through EEG signals, highlight features of these signals, and transmit these features to an external device, such as a drive, mouse cursor, or robot arm device.

As specific features of the CSP, the mu rhythm (8-12 Hz) and beta rhythm (18-22 Hz) frequency ranges (p. 7 of the application description) are allocated by the frequency-band filter.

With weak signals of brain waves, it is desirable to increase the classification accuracy using data from all electrodes, eras, and stimuli in the classification process. The main limitations of existing methods for classifying ERP (or classifying other biopotentials associated with specific events) are caused by the following reasons. First of all, most ERP or other measurements of biopotentials require registration data from many electrodes, epochs, stimuli and / or tests (collectively called channels). Each data channel contains weights. The distinctiveness between classes varies from channel to channel as a function of time. Most classifiers are focused on classifying ERPs of each channel independently and completely do not use different, but additional information, hidden in multichannel registration of biopotentials. Due to the poor signal-to-noise ratio, analysis and classification are typically performed on ERPs averaged over a large number of trials. Collecting a large number of trials of ERPs to form averaged ERPs causes users to become tired and lose concentration during long data collection sessions. Currently, the need for repeated experiments is the main reason limiting the use of ERPs in the management of external devices and medical diagnostics. Third, most classifiers for assessing the shape of brain waves are multidimensional classifiers and require the assignment of covariance matrices. In order for the covariance matrices to be used in the calculations, it is necessary that the number of ERPs samples be greater than the dimension of the covariance matrix. Since in a separate test this condition is not fulfilled, it is impossible to evaluate multidimensional parameters. Fourth, classifier formulations are typically limited to dichotomous classification (division into 2 classes) and are not generalized to a larger number of classes.

Thus, in spite of individual achievements, there are still many unsolved problems in non-invasive BCI. A small number of conditions are reliably recognized (2-3); the recognition performance of these states is low, especially when using evoked potentials or the total EEG (recognition of a single state usually takes tens of seconds); training a subject to manage their biopotentials can take several days or even weeks; it is necessary to take into account the functional state and tuning to the individual characteristics of the subject. A solution to these problems is necessary in order to bring the characteristics of BCI-based communication and control systems to a level of practical interest.

The problem to be solved in the claimed invention is to reduce the time for identification of user mental commands in the BCI system to 2 seconds while increasing the accuracy of identification of user mental commands to 90% due to the use of previously unknown specific information components of a common spatio-temporal pattern and processing adapted to solve this tasks to the classification algorithm for the allocation of specific fragments of an EEG signal by a neural multilayer network (NMS).

The indicated technical result is achieved by the fact that the method of classifying electroencephalographic signals in the brain-computer interface consists in testing the user, extracting specific information components from a common spatio-temporal pattern, creating a sample of digitized EEG fragments from a variety of leads for classifier training, calculating weight coefficients and classifying fragments EEG to identify classes of mental commands of the user corresponding to the control signal lamas.

According to the invention, local positive maxima of the amplitude of the EEG signals from all leads are used as specific information components, while if the values of two adjacent positive maxima differ by less than the psychophysiological threshold of human perception, they are considered equal and the second maximum is excluded from the subsequent analysis, simultaneously with the allocation the first positive local maximum from the reference leads record the values of the amplitudes of the EEG signals for all other leads, in the result is a set of amplitudes representing the first input vector for the neural multilayer network (NMS), and the procedure for generating the input vectors is repeated for each subsequent positive maximum of the reference lead and for each individual lead, each time taken as a reference lead, until each it doesn’t fulfill the reference function from the leads, as a result, a multidimensional array of input vectors is received from a specific user, and when training the NMS networks, they additionally form an array of indexes of the classes of mental movements performed by the user, which is the output array for training the NMS, and calculate the weighting coefficients of classification by the algorithm of back propagation of error, and when identifying the mental movement, the array of input vectors is fed to the NMS to calculate the output vector by which the class of mental movement of the user is determined .

In a specific embodiment of the invention, the values of two adjacent positive local maximums of the amplitudes of the EEG signals differ by at least 5%.

The present invention is based on the ideas of I. Prigogine about the intrinsic time of nonlinear systems - inanimate and living. (Prigogine I. From the existing to the arising. M., Nauka, 1985, 326 s) / 12 /. This time differs from astronomical one, although it can be measured by a wristwatch or using some kind of dynamic device, but it has a completely different meaning, because it arises from the random behavior of trajectories encountered in unstable dynamic systems.

In psychological studies (Phress P. Perception and time evaluation. Experimental psychology. Issue VI, M., Progress, 1978, 88-135) / 13 / it was shown that, depending on the functional state of the subject, his subjective time flows differently and, therefore, it can act as a dependent variable. A similar situation occurs not only at the psychological, but also at the physiological level. The fact that it is proper time, set by physiologically significant events, must be taken into account when analyzing information processing processes in the nervous system, for example, numerous facts related to stimulation effects associated with certain phases of the dynamics of bioelectrical activity indicators of various subsystems of the body indicate. Thus, we can assume that time in physiological systems is both “contextually” dependent and determined by the functional state. Experience in various fields of research shows that in cases where it is possible to correctly determine the system’s own time, its use significantly increases the efficiency of the description of the dynamic properties and behavior of this system. This is manifested in a significant reduction in the mathematical description of such a system without loss of information and an improvement in the predictive properties of such a description. Based on the available facts and theoretical generalizations, the idea was formed that to describe the specifics and structure of biological processes, it is necessary to introduce the concept of biological (physiological) time (J. Wheatrow. The structure and nature of time. M., Nauka, 1984, 64 s) / 14 /, that is, the time associated with the internal rhythm of the functioning and development of biological objects and with the random behavior of the trajectories of biological processes. The time construction used now is Newton’s absolute time, for the measurement of which there are a variety of clocks. On the other hand, the point of view on the functioning of living organisms as a sequence of events - functional quanta, such as quanta of elementary physiological processes, quanta of homeostasis, quanta of behavior, is universally recognized. Any of these quanta, ending with a certain result and being functionally the same, can have different durations in a commonly used time scale. Consequently, the natural elements of physiological and behavioral processes are not equivalent to generally accepted metric units of time, but determine a heterogeneous flow of events that determine the proper time of a process.

It is known that the main contribution to electrical activity, recorded in total in the form of EEG, is made by slowly developing potentials of neural membranes. The total record reflects changes in the electric potential that occur in the plexus of nerve fibers. When the activity of a large number of neurons is synchronized under the action of external stimuli, there is a correspondence between impulse and total activity. Therefore, it can be assumed that the total potentials reflect the probability with which individual neurons will be discharged upon presentation of stimuli that synchronize large groups of neurons.

It has now been established that slow potentials form dynamic structures that play an important role not only in the transmission of nerve impulses, but also in the analogue “calculation” of interactions between neurons, including those that are spaced apart over considerable distances. In other words, there is a two-process mechanism in which the local microstructure — increasing and decreasing — of slow potentials, recorded as an EEG, is coupled with propagating nerve impulses. Man and animals perceive not time, but processes, changes, sequences. As natural elements of behavior are sequences of physiologically significant events. Such an event is a change in sign (from positive to negative) of the derivative of the EEG signal. At the time of each such event, a separate EEG lead acts as a reference in relation to the other leads. All living systems respond only to changes in the incoming information, and there are thresholds of perception, determined in accordance with the psychophysical law of Weber-Fechner (Harvey Schiffman. Sensation and perception of time. Transl. From English, edition 5, chap. 19, publ. "Peter", 2003, p. 772-789) / 15 /. Based on this, in the present invention, it is assumed that two EEG events differ from each other if the amplitude of the next event differs from the amplitude of the previous event by at least 5%, which corresponds to the threshold of a person's psychophysiological perception.

The invention is illustrated by the figures of drawings, graphs and tables.

Figure 1 presents a General diagram of the system interface brain - computer.

Figure 2 presents the block diagram of the encephalograph.

Figure 3 presents a block diagram of a processor for training a multilayer neural network and classifying an EEG signal.

Figure 4 presents a block diagram of the training of a multilayer neural network.

Figure 5 presents a block diagram illustrating the memorization of training time fragments, presented in the form of matrices S ⁱ EEG in computer memory.

Figure 6 presents a block diagram of the formation of training arrays of input vectors {a} and output string variables {b}.

Figure 7 presents a block diagram of the preprocessing of an array of input vectors {a} into an array of vectors {α}.

On Fig presents a block diagram of the preprocessing array of output string variables {b} into an array of vectors {β}.

Figure 9 presents the block diagram of the algorithm for the back propagation of the classification error of the EEG signal.

On figa presents a structural diagram of a multilayer neural network.

On figv presents a diagram of the nodal element of the NMS.

Figure 11 presents a block diagram of a direct-flow calculation algorithm of the output vector θ in a multilayer neural network.

On Fig presents a block diagram of the algorithm for the reverse flow calculation of errors of the weight coefficients Δw in a multilayer neural network.

13 shows an illustration of the advantages of a nonlinear classification over a linear one in the case of nonlinearly separable classes, where a) an example of nonlinearly separable classes, light and dark dots indicate the areas of vectors x = (x1, x2) belonging to conditionally distinguished classes I and II, respectively, b a) a linear classification of nonlinearly separable classes, c) a nonlinear classification of nonlinearly separable classes.

On Fig presents a flowchart illustrating the process of neural network classification of the EEG signal.

On Fig presents a block diagram of the formation of an array of input vectors {a}.

On Fig presents a block diagram illustrating the interpretation of the output vector θ in a neural multilayer network.

On Fig presents EEG fragments illustrating the process of forming an array of input vectors {a}.

On Fig shows a diagram of the search and selection of positive local maxima of the amplitude on the fragment of the EEG signal.

On Fig presents a General view of the used headset for recording EEG.

On Fig presents the used electrode system EEG.

Table 1 shows the time interval F ^{i of the} EEG signal, presented as an array of S ⁱ vectors S ⁱ _j amplitudes S ⁱ _jk .

Table 2 shows the values of the string variable b.

Table 3 shows the encoding of the values of the string variable b into the output vector β.

Table 4 shows the coding of classes of mental movements in control teams.

An example of a preferred embodiment of the invention.

The general BCI circuit (FIG. 1) contains a portable encephalograph 2 connected at the input via electrodes to the scalp of user 1, and connected to a computer 3 with its output, the output of which is connected to a controlled device 4 (wheelchair). To record the EEG signal, the 19-channel EEG recorder Encephalan-EEGR-19/26, manufactured by the Medicom company, Taganrog, was used. The headset for collecting EEG signals includes a set of silver chloride electrodes and a fabric helmet with special sockets for fixing the electrodes (Figs. 1, 2). The working element of the EEG electrodes is Ag / AgCl. The tubular design of the electrodes allows you to refuel and add contact material without reinstalling the electrode. In the body of the electrode there is a hole for the nozzle of a standard syringe (without a needle), designed to refill the electrode gel. The colors of the electrodes are identical with the colors of the corresponding slots on the helmet.

- The total resistance of the EEG electrodes is no more than 2 kOhm.

- The difference of the electrode potentials of the EEG is within ± 20 mV.

- The readiness time of the EEG electrodes is not more than 10 minutes.

- The drift of the potential difference of the EEG electrodes is not more than 15 μV.

- The noise voltage of the EEG electrodes is not more than 3 μV.

The Encephalograph (figure 2) contains electrodes 5, an EEG amplifier 6, an ADC 7 and a frequency-band filter 7. An analog EEG signal is recorded using electrodes 5 placed on patient's scalp 1 at 12 leads according to the standard 10-20 scheme: F3 , F4, C3, C4, T3, T4, T5, T6, P3, P4, O1, O2 using two reference electrodes.

The processor 3 (figure 3) contains a learning unit NMS 9 and a classification block NMS 10.

The training unit of the NMS 9 (Fig. 4) contains blocks for storing time periods F ⁱ EEG for each i-th mental movement 11, the formation of training arrays of input vectors {a} and output string variables {b} 12, preprocessing an array of input vectors {a} 13, preprocessing the array of output string variables {b} 14, executing the back-propagation algorithm for error 15.

The block diagram of the memorization of the training EEG segments for each mental movement 11 (Fig. 5) contains blocks for giving the test subject a command to make ie mental movement 16, memorizing the time interval F ^{i of the} EEG signal of duration τ in time Δt after command 17, memorizing the textual name of the mental movement b ⁱ 18, cessation of command and EEG recording 19.

The block diagram of the formation of training arrays of input vectors {a} and output string variables {b} 12 (Fig. 6) contains blocks for finding positive local amplitude maxima in the memorized time intervals F ⁱ EEG 20, selection of maxima 21 that differ from the previous maximum for this assignment of not less than 5%, the formation for each segment F ⁱ EEG of an array of vectors x ⁱ from the values of the amplitudes for all leads at the time of detection of maxima 22, the combination of vectors x ⁱ _j from all arrays x ⁱ into a training array of input vectors {a}: a = x ^{1 2} ∪x 23 ... ∪x ⁿ forming an array output training string variables {b} of text names mental movements corresponding to each vector and array {a} 24.

The block diagram of the preprocessing of the array of input vectors {α} 13 (Fig. 7) linearly transforms the array of input vectors a = (a ₁ , a ₂ , ..., a ₁₂ ) into the array of vectors α = (α ₁ , a ₂ , ..., α ₁₂ ).

The block diagram of the preprocessing of the array of output string variables 14 (Fig. 8) encodes an array of string variables b into an array of output vectors β.

The flowchart of the algorithm for the back propagation of error 15 (Fig. 9) contains blocks for performing direct-flow calculations of output vectors θ 25, performing back-flow calculations of errors Δw of weight coefficients w, and their modifications w = w + Δw 26.

The structural diagram of the used neural multilayer network (NMS) contains the nodal elements of the input layer, the hidden layer and the output layer (figa).

Functional diagram of the nodal element of the hidden layer and the output layer of the NMS contains an adder 27 that calculates the scalar product of the input signal by the vector of weight coefficients w, and a nonlinear converter 28 (Fig.10b).

The flowchart for performing straight-through calculations with input vectors α 25 (Fig. 11) calculates the output vector θ layer-by-layer in the NMS.

The flowchart for performing back-flow calculations of errors Δw of the weight coefficients w and their modifications w = w + Δw 26 (Fig. 12) calculates layer-by-layer weight coefficients w in the NMS.

The classification flowchart 10 (Fig. 14) contains blocks for storing an EEG segment F ^{class of} duration τ 29, generating an array of input vectors {a} 30, preprocessing an array of input vectors {a} 13, performing direct-flow calculations of output vectors θ 24, interpreting output vectors θ NMS 31, determining the execution of the mental movement 32, encoding the class of mental movement in the control command 33.

The block diagram of the formation of an array of input vectors {a} 30 (Fig. 15) contains blocks for searching for positive local amplitude maxima in the memorized segment F ^class EEG 34, selection of maxima 35, differing from the previous maximum by this lead by at least 5%, formation array of vectors {a} from the values of the amplitudes for all leads at the moments of detection of maxima 36.

The interpretation scheme of the output vectors θ of the NMS 31 (Fig. 16) converts the calculated output vector θ into the binary vector β = (β ₁ , ..., β _i , ... β ₅ ) by initializing the component β _{i with} unity if the ith component of the vector θ the maximum of all the values of the components of the vector θ, and the initialization of the remaining components of the vector β by zeros, and the conversion of the binary vector β to the string variable b.

The EEG fragments on a scale of 100 ms / cm show the process of forming an array of input vectors {a} = a ₁ , a ₂ , a ₃ , ..., where vertical dashed lines show the allocation of positive local maximums of amplitude a _{1 1} , a _{2 1} , and _{3 1} , ... from the reference lead T4 of the EEG signal and fixing the amplitudes a _{1 2} , a ₂₂ , a _{3 2} , ... - a _{1 12} , a _{2 12} , and _{3 12} , ... EEG signals for all other leads (Fig. 17 and figa - continuation of Fig.17).

A fragment of the EEG signal (Fig. 18) illustrates the selection of local positive maxima of the amplitude of the EEG signal, where the vertical dashed lines indicate selected local maxima that differ from neighboring local maxima by at least 5%.

The analog signals from the electrodes 5, corresponding to the twelve leads from the scalp of the user 1, are fed to an encephalograph 2 (Fig. 1), where they are amplified by a multi-channel amplifier 6 and converted into digital signals with a sampling frequency of 250 Hz and a bit weight of 25 μV in ADC 7 and filtered -pass filter 8 with a passband in the range from 1.6 to 70 Hz to eliminate low-frequency and high-frequency interference. The filtered signal is transmitted to the processor 3 (Fig. 1) for training a neural multilayer network (NMS) if the NMS is not trained or for the classification of the NMS 10 if the NMS is trained (Fig. 3). The training of NMS 9 begins with the memorization of training time segments F ⁱ EEG signal for each i-th mental movement and the names of mental movements, designated as string variables b ⁱ in the block for memorizing time segments F ⁱ EEG for each i-th mental movement 11 (Fig. four).

In the submission unit of the test subject to make ie mental movement 16, the test subject is given a command from the computer screen in the form of a verbal transparency or visual image to make ie of five mental movements: squeeze the left fist, squeeze the right fist, stick out the tongue, squeeze the fingers of the left foot, squeeze the fingers of the right foot .

In the block for storing the time interval F ^{i of the} EEG signal of duration τ after a time Δt after giving the command 17, the time interval F ^{i of the} EEG signal of duration τ = 400 ms with a delay interval Δt = 50 ms from the moment the command was sent is stored. Each time interval F ^{i of the} EEG signal is presented in the form of an array S ⁱ of amplitude vectors, where each vector S ⁱ _j consists of the values of the amplitudes S ⁱ _jk for all 12 leads at the same time. Index i denotes the number of mental movement, index j denotes the serial number of the vector S ⁱ _j in the array S ^{i of} vectors, and index k denotes the lead number. The structure of the array S ⁱ of amplitude vectors is illustrated in table 1. Each column of the table corresponds to one lead, and the row to one vector. The vector S ⁱ _j in the table is highlighted in gray, and the value of the amplitude S ⁱ _jk is in bold. At the same time, in the memory unit for the textual name of the mental movement b ⁱ 18, the textual names of the mental movements in the string variable b ^{i are} stored (Fig. 5), which are pointers to the classes of mental movements performed by the user (table 2).

The issuance of commands and recording of the EEG signal is stopped in the block for stopping the flow of commands and recording the EEG 19, if the condition “Are all 5 commands sent?” Is fulfilled. If this condition is not fulfilled, the next command is sent to the submission unit of the test subject to make ie mental movement 16 (Fig. 5). After memorization, training arrays of input vectors {a} and output string variables {b} are formed in the block for the formation of training arrays of input vectors {a} and output string variables {b} 12. For this, in the search block for positive local amplitude maxima in the stored time intervals F ⁱ EEG 20 select positive local amplitude maxima in each of the time intervals F ^{i of the} EEG signal stored in arrays of S ⁱ vectors. In the block of selection of maxima 21 that differ from the previous maximum by a given lead by at least 5%, from the found positive local maximums of the amplitude, the maxima M ⁱ ₁ are selected, which differ in amplitude from the previous selected maximum by this lead by at least 5%. For this, from the array S ⁱ select the values of the amplitudes S ⁱ _jk that satisfy the conditions:

where L _k is the previous selected maximum at the kth lead.

. Процесс отбора максимумов проиллюстрирован на фиг.18. Отобранные максимумы Мⁱ _l отмечены вертикальными пунктирными линиями. Локальные максимумы с амплитудой меньше нуля, а также максимумы, не отличающиеся на 5% от предыдущего максимума, не отбираются и не учитываются в последующем анализе. В блоке формирования для каждого отрезка Fⁱ ЭЭГ массива векторов хⁱ из значений амплитуд по всем отведениям в моменты обнаружения максимумов 22 из каждого массива Sⁱ векторов, соответствующего запомненному отрезку Fⁱ ЭЭГ, выбирают все вектора Sⁱ _j, содержащие хотя бы один отобранный на предыдущем этапе положительный локальный максимум Mⁱ _l амплитуды. Из этих векторов для каждого запомненного временного отрезка Fⁱ ЭЭГ сигнала формируют массив векторов хⁱ для i-го мысленного движения, где i=1, 2, 3, 4, 5.If the condition is met, the value of L _{k is} updated:

. The process of selecting highs is illustrated in FIG. The selected maxima M ⁱ _l are marked by vertical dashed lines. Local maxima with an amplitude less than zero, as well as maxima that do not differ by 5% from the previous maximum, are not selected and are not taken into account in the subsequent analysis. In the formation unit for each segment F ⁱ EEG of an array of vectors x ⁱ from the values of the amplitudes for all leads at the moments of detection of maximums 22 from each array S ^{i of} vectors corresponding to the stored segment F ^{i of the} EEG, select all vectors S ⁱ _j containing at least one selected at the previous stage, the positive local maximum M ⁱ _l amplitude. From these vectors for each stored time interval F ^{i the} EEG signal forms an array of vectors x ⁱ for the i-th mental movement, where i = 1, 2, 3, 4, 5.

In the block of combining vectors x ⁱ _j from all arrays x ⁱ into the training array of input vectors {a}: a = x ¹ ∪x ² ∪ ... ∪x ⁿ 23 vectors x ⁱ _j from the formed arrays x ¹ , x ² , x ³ , x ⁴ , x ⁵ are combined into a training array of input vectors {a}:

In the block for the formation of the training array of output string variables {b} from the text names of mental movements corresponding to each vector a of the array {a} 24, a training array of output string variables {b} containing the text names of mental movements b ⁱ corresponding to each vector of the training array is formed input vectors {a}.

In the preprocessing unit of the array of input vectors {a} 13 (Fig. 7), the preprocessing of the array of input vectors {a} is performed, which consists in the linear transformation of each ith component of the input vector a into the components of the vector α in accordance with the formula:

where a _{i min} and a _{i max} respectively the minimum and maximum value of the i-th component of the vector a of the array {a}.

In the preprocessing block of the array of output string variables {b} 14 (Fig. 8), the preprocessing of the array of output string variables {b} is performed, which consists in encoding the textual names of classes b ¹ , b ² , b ³ , b ⁴ , b ⁵ into five-component numeric vectors β ¹ = (1, 0, 0, 0, 0), β ² = (0, 1, 0, 0, 0), β ³ = (0, 0, 1, 0, 0), β ⁴ = ( 0, 0, 0, 1, 0), β ⁵ = (0, 0, 0, 0, 1) (table 3).

After preprocessing, the training arrays of the input {α} and output vectors {β} are used to train the NMS in the execution unit of the back propagation algorithm for error 15, which consists in adjusting the weight coefficients w carried out by the back propagation algorithm [Rumelhart D.E., Hinton G.E. and Williams R.J. Learning internal representations by error propagation. In Parallel Distributed Processing, London: MIT Press, vol. 1, 1986, 550 p.]. The error back-propagation algorithm consists of an iterative procedure for performing direct-flow calculations with input vectors α in the unit for performing direct-flow computations of output vectors θ 25 and then performing back-flow computations of errors Δw weight coefficients w and their modifications in the unit for performing back-flow computations of errors Δw weight coefficients w and their modifications w = w + Δw 26 according to the formula:

To classify EEG signals, an artificial neural multilayer network (NMS) was used, which contains computational nodal elements organized in layers: input layer, hidden layer, and output layer (Figs. 10a, 10c). Each node of the input layer of the NMS does not perform any calculations and serves to distribute each component of the input vector α to all the node elements of the hidden layer having a weight matrix w ₁ . Similarly, the nodal elements of the output layer are associated with all elements of the hidden layer having a matrix of weight coefficients w ₂ . The calculations produce nodal elements in the hidden and output layer of the NMS.

In direct flow calculation, each nodal element ij of the hidden i = 1 and output i = 2 layers, where index j is the ordinal number of the nodal element in the layer, contains a weight vector w _ij , an adder _мат 27 and a non-linear transducer σ 28 (Fig. 11). The adder 27 calculates the scalar product of the input signal vector of the node element of the NMS with the vector of weights w _ij . Nonlinear transducer 28 calculates a sigmoid function:

As a result of layer-by-layer calculations carried out by the nodal elements of the hidden and output layer above the input vector α, we obtain the calculated output vector θ at the output of the NMS.

When the reverse flow calculation (Fig), each nodal element of the output layer of the NMS calculates the function δ ₂ according to the formula:

where θ is the calculated output vector of the neural network, β is the output vector, σ ₂ is the sigmoid function at the output of the nodal element of the output layer of the neural network,

and modifies its weighting factors w ₂ by the error Δw ₂ = δ ₂ · σ ₁ according to the formula:

Next, each nodal element of the hidden layer of the NMS calculates the function δ ₁ by the formula:

where σ ₁ is the sigmoid function at the output of the nodal element of the hidden layer of the neural network,

and modifies its weighting factors w ₁ by the error Δw ₁ = δ ₁ · σ ₀ according to the formula:

where σ ₀ = α. The indices 0,1 and 2 relate to the input, hidden and output layer, respectively.

The iterative procedure of direct and reverse flow calculations continues until the calculation limit is reached, which consists in changing Δw weight coefficients w: Δw <Aw _min , where Δw _min = 0.001.

The error back propagation algorithm used in teaching the NMS provides the construction of nonlinear boundaries separating the given classes of mental movements represented by string variables b ⁱ . On Fig shows a case of non-linear arrangement of examples of two classes, depicted by points of light and dark color (figa). The use of a linear classifier constructing linear dividing boundaries, such as discriminant analysis, does not allow us to separate the domains of belonging of examples of different classes (Fig.13b). The use of a nonlinear classifier that constructs nonlinear dividing boundaries, such as a NMS with an error back propagation algorithm, allows us to separate the domains of belonging of examples of different classes (Fig.13c).

After training the NMS, the processor 3 carries out the classification 10 (Fig.14). During the classification, the time interval F ^{class of the} EEG signal with a duration of τ = 400 ms is stored in the memory unit of the EEG segment F ^{class with a} duration of τ 29. The time interval F ^{class of the} EEG signal is presented as an array S ^class of amplitude vectors S ^class _j , where each vector consists of values of amplitudes S ^class _jk for all 12 leads at the same time. Index j denotes the sequence number of the amplitude vector S ^class _j in the array of vectors S ^class , and index k denotes the lead number. The structure of the S ^class array is similar to the structure of the S ⁱ array shown in Table 1.

The array of input vectors {a} in block 30 is formed by searching for positive local amplitude maxima in the search unit for positive local amplitude maxima in the stored segment F ^class EEG 34 and selecting the maxima M ⁱ _l that differ from the previous selected maximum by this lead by at least 5%, in block 35 in the memorized time interval F ^class EEG signal.

To do this, from the S ^class array, select the values of the S ^class _jk amplitudes that satisfy the conditions:

where L _k is the previous selected maximum at the kth lead. If the condition is met, the value of L _{k is} updated: L _k = S ^class _jk .

In the block for generating an array of vectors {a} from the values of the amplitudes for all leads at the moments of detection of maxima 36 from the array S ^{class of} vectors corresponding to the stored time interval F ^{class of the} signal, all vectors S ^class _j containing at least one positive local selected at the previous stage are selected maximum M ⁱ _l amplitude. Of these vectors, a training input array of vectors {a} is formed (Fig. 15).

Then, the array of input vectors {a} is pre-processed in block 13 and neural network direct-flow calculations of the output vector θ in block 24 are performed.

In the interpretation block of the output vectors θ of the NMS 31 (Fig. 16), the output vectors θ of the NMS are converted into binary vectors β = (β ₁ , ..., β _i , ..., β ₅ ) in accordance with the following formula:

and interpret the binary vectors β into string variables b (table 2).

As a result of interpretation of the array of output vectors {θ}, an array of string variables {b} with the text names of the classes of mental movements is obtained. String variables b indicate that the input vectors a belong to the classes of mental movements b ⁱ . The determination of the execution of mental movement is carried out in block 32 (Fig. 14) by determining the class of mental movement b ⁱ with the maximum scoop of occurrence of the value of the string variable b ⁱ in the array {b} according to the formula:

The probability of performing the mental movement ε of the i-th class is calculated as the ratio of the number n = ∑ b ^{i of} output string variables b with the value b ⁱ to the total number N = ∑ b ¹ + ∑ b ² + ∑ b ³ + ∑ b ⁴ + ∑ b ⁵ string variables b in the array {b} ε = n / N · 100%.

It is accepted that mental movement occurs if the probability ε is greater than the threshold value

where ε ^* = 80%, which is consistent with previously conducted experiments. Ten series of experiments show that in nine cases out of ten, the probability of identifying classes of mental movements corresponding to the values of the string variable b is 80-95%. Thus, when condition (13) is satisfied, we have an accuracy of identification of mental commands of 90%, which is comparable with an achievable accuracy of 88% and 94% for two separate sessions, which was achieved by the user with the well-known two-dimensional control of the mental movement of the mouse cursor when registering an electrocorticogram (ECoG) of signals brain activity.

If the execution of a mental movement is detected, in block 33, the class of the mental movement is encoded into the control command: b ¹ → c ¹ , b ² → c ² , b ³ → c ³ , b ⁴ → c ⁴ , b ⁵ → c ⁵ (table 4 ) Such control commands can be commands to move the cursor on a computer screen or press a computer mouse button, to wheelchair drives, toggle switches of the operator’s workplace.

In the present invention there is no need for multiple accumulation and processing of fragments of EEG signals to identify informative parameters, which reduces the time for identification of mental commands. During testing, the user was asked to actually compress the fists of the left and right hands and compress the fingers of the right and left legs, which were recorded using the electromyographic electrodes included in the encephalograph kit. The electromyogram signal appeared after 400-600 ms from the moment the command to execute the movements was presented. The appearance of this signal means that a mental command to execute the movement during this time interval was formed in the user's brain. It is these fragments of EEGs with a duration of 400-600 ms (Figs. 17, 17a) that were used in the process of training the NMS. During testing, the time of computational operations is significantly less than 1 second, so that the total time for identifying a mental command and generating a control signal does not exceed 2 seconds, which was not achieved in well-known patent analogues and publications.

The processor unit 3 is implemented in the Microsoft Windows Vista operating system on an IBM PC, having the configuration:

- A motherboard with an Intel processor architecture of the Core2Duo family;

- microprocessor clock frequency of 2000 MHz;

- RAM 1024 MB;

- a hard drive with a capacity of 80 GB;

- video adapter supported by the operating system;

- LCD monitor;

- keyboard and mouse manipulator.

Information sources

1. US 5840040, IPC A61B 5/00, U.S. Cl. 600/545, 600/544, 24-11-1998.

2. US 5638826, IPC A61B 5/0476, U.S. Cl. 128/732, 340 / 825.19, 345/157, 463/36, 17-06-1997.

3. WO 2009057260 (A1), IPC G06F 3/01, 2009-05-07.

4. WO 2009057278 (A1), IPC G06F 3/01, A61B 5/0484, A61B 5/0476, 2009-05-07.

5. WO 2008059878 (A1), IPC G06F 3/01, A61B 5/0476, 2008-05-22.

6. US 2005085744, A61B 5/00, IPC G06F 3/00, G06F 3/01, 2005-04-21.

7. WO 2006026548, IPC G06K 9/00, 2006-03-09.

8. WO 2008117145, IPC G06F 3/01, 2008-10-12.

9. WO 2009/044325 A1, IPC G06F 3/01, 04/09/2009.

10. US 6349231 B1, IPC A61B 5/0482, A61B 5/0484, A61B 5/16, G06F 17/00, A61B 5/0476, 2002-02-19.

11. WO 2008/097200 A1, 6 IPC A61B 5/0476, C06F 19/00, A61F 2/72, 08/14/2008 - prototype.

12. Prigogine I. From the existing to the emerging, M., Science, 1985, 326 p.

13. Fress P. Perception and assessment of time. Experimental psychology. Vol. V1, M., Progress, 1978, pp. 88-135.

14. Wheatrow J. The structure and nature of time. M., Science, 1984, 64 p.

15. Harvey Schiffman. Sensation and perception of time. Per. from English., 5th edition, ch. 19, publishing house "Peter", 2003, S. 772-789.

Claims (2)

1. A method for classifying electroencephalographic signals in the brain-computer interface, which consists of testing the user, extracting specific information components from a common spatio-temporal pattern, creating a sample of digitized EEG fragments from multiple leads for classifier training, calculating weight coefficients and classifying EEG fragments for class identification user mental commands corresponding to control signals, characterized in that as a specific x information components use local positive maxima of the amplitude of the EEG signals from all leads, while if the values of two adjacent positive maxima differ by less than the threshold of the psychophysiological perception of a person, they are considered equal and the second maximum is excluded from the subsequent analysis, simultaneously with the allocation of the first positive maximum from the reference lead, the values of the amplitudes of the EEG signals for all other leads are recorded, as a result, a set of amplitudes representing the first input vector for a neural multilayer network (NMS), and the procedure for generating input vectors is repeated for each subsequent positive maximum of the reference lead and for each individual lead, each time taken as a reference lead, until each of the leads fulfills the function of the reference , as a result, a multidimensional array of input vectors is received from a particular user, and during training (NMS), an array of pointers of classes of mental movements performed using the body, which is the output array for training the NMS, and calculate the weighting coefficients of classification according to the algorithm for the back propagation of error, and when identifying the mental movement, an array of input vectors is fed to the NMS to calculate the output vector, which determines the class of mental movement of the user. 2. The method according to claim 1, characterized in that the values of two adjacent positive local maximums of the amplitudes of the EEG signals differ from each other by at least 5%.

Images