KR101149002B1 - Adaptive brain-computer interface device - Google Patents

Abstract

The present invention relates to an adaptive brain-computer interface device, the technical problem to be solved is an adaptive brain that can adaptively determine the order of stimulation according to the situation so that the user can accurately find the desired command as soon as possible To provide a computer interface device.
To this end, the adaptive brain-computer interface device according to the present invention comprises a stimulus generating device for generating a stimulus and applying a stimulus to the user, a signal collection device for recording an EEG signal of the user generated by the stimulus, A preprocessing device for extracting the characteristics of P300 for the stimulus, an analysis device for determining the presence or absence of the P300 from the extraction signal of the preprocessing device, and an inference of the current state from the observation of the analysis device and an optimal stimulus for the current state It is characterized in that it comprises a stimulation order determination device for selecting the stimulation order of the stimulus generating device.

Description

ADAPTIVE BRAIN-COMPUTER INTERFACE DEVICE

The present invention relates to an adaptive brain-computer interface device, and more particularly, to an adaptive brain-computer interface device for improving the number of commands and messages that can be input per unit time.

In general, the brain-computer interface refers to a device that uses commands of the brain to send commands and messages to external devices without using physical actions. In particular, the brain-computer interface can be very helpful for the expression and control of the physically handicapped.

In constructing the brain-computer interface, an invasive method for measuring brain activity by directly inserting electrodes into the brain and a non-invasive method for measuring brain activity using brain waves observed from outside the skull are used. .

Electroencephalograms (EEGs), measured using non-invasive methods, are electrical signals generated by neurons in the brain.

These electrical signals represent the sum of the simultaneous electrical activity of thousands to millions of neurons, which exist around the electrode measuring brain waves.

Given the somatic stimulus that the user is focused on, EEG generates Event-Related Potentials (ERPs) according to the stimulus.

P300 is the peak towards the + potential at the event linkage potential that occurs about 300 ms after the stimulus is given, and P300 is known for reliable brain activity in constructing the brain-computer interface.

An embodiment that has been actively studied to construct a brain-computer using the P300 is a device for inputting a keyboard using brain waves with a P300 speller.

In the P300 speller, the characters are arranged in a 6x6 matrix, and the user watches the characters of the 36 characters. The user is given a stimulus in which the characters in each row and column light up and darken in a random order for a short time.

Given this stimulus, if the stimulus is given to the character staring at the user, the event-linked potential appears in the EEG, and P300 is generated in the EEG after about 300 ms given the stimulus.

The brain-computer interface recognizes the P300 signal and can find out which character the user is staring at. That is, if P300 is present in the brain signal corresponding to a stimulus, the user may interpret that the user wants to input a character corresponding to the stimulus. If P300 is not detected, the user inputs a character corresponding to the stimulus. It can be interpreted as not desired.

In addition, the number of characters that can be input can be changed by adjusting the size of the matrix in the P300 speller, and the characters can be replaced with commands to be applied to the control of devices such as wheelchairs.

4 is a block diagram of a conventional adaptive brain-computer interface device.

Meanwhile, a typical structure of the brain-computer interface device based on P300 is shown in FIG. 4.

The stimulus generating device 10 is a device for generating a stimulus so that P300 may occur under a condition desired by the user, and the signal collection device 20 is a device for recording the brain waves of the user with respect to a given stimulus. In addition, the preprocessor 30 performs preprocessing such as extraction of P300 from a given EEG signal, and the interpreter 40 determines whether P300 is present in the extracted signal given from the preprocessor, and the brain-computer interface. It serves to give a command to the external device 50 connected to.

The conventional brain-computer interface device as described above gives the user the same number of stimuli for all kinds of stimuli in order to find a command to be input by the user, and the order of giving each kind of stimuli to the user is arbitrary ( random).

As described above, giving the same number of stimuli to all kinds of stimuli is an unnecessary process, and arbitrarily setting the order of stimulation also brings limitations to the performance improvement of the brain-computer interface.

For example, in the P300 speller, given the stimulus given so far and the EEG signals collected for it, if the user's intention is very unlikely to be present in the first row, the reason to re-stimulate the first row is none.

Also, in situations where there is a high likelihood of user intent in the second and third columns, it may be better to reduce the uncertainty by repeatedly stimulating these two columns.

That is, if the order of the stimuli to give to the user can be effectively determined, the user's intention can be grasped with only a small number of stimuli.

The present invention has been made to solve the above problems, an adaptive brain-computer interface device capable of adaptively determining the order of stimulation according to the situation so that the user can accurately find the desired command as soon as possible The purpose is to provide.

Another object of the present invention is to provide an adaptive brain-computer interface device that can grasp a large number of commands and messages per unit time and can be applied to external device control even in a situation where it is difficult to use a handicapped person or a hand or a foot.

In order to achieve the above object, the adaptive brain-computer interface device according to the present invention comprises a stimulus generating device for generating a stimulus and applying a stimulus to a user, and collecting a signal for recording an EEG signal of the user generated by the stimulus. A preprocessing device for extracting features of the P300 for the stimulus from the EEG signal, an analysis device for determining the presence or absence of the P300 from the extraction signal of the preprocessing device, and inferring the current state from the observations of the analysis device And a stimulus order determination device for selecting an optimal stimulus for the current state to determine the stimulation order of the stimulus generating device.

In addition, the adaptive brain-computer interface device may be subjected to a partial observation Markov decision model (POMDP) to determine the optimal behavioral policy.

In addition, the stimulus order determination device may be configured to generate a confidence state update unit for inferring a probability distribution for the current state from a previous stimulus given to the user and observations corresponding to each stimulus, and an optimum stimulus corresponding to the confidence state of the confidence state update unit. It may include an optimal stimulus selection unit to select and perform, and to deliver a command or a message to an external device.

In addition, the adaptive brain-computer interface device may apply a delayed observation POMDP to determine the optimal behavior policy.

In addition, the adaptive brain-computer interface device may determine the optimal behavioral policy based only on the behavior except for the behavior performed within the time when the repetition blindness due to the overlapping stimulus occurs.

In addition, the optimal behavior policy may be calculated by a value function defined by the following equation.

(Where A is the set of actions and A 'is the set of actions performed within 500ms)

In addition, the preprocessor may remove noise by obtaining an average of EEG signals, or may use a P300 extraction algorithm such as a spatial filter algorithm or a Mexican hat wavelet.

In addition, the analysis device may be composed of a P300 classifier using a classification algorithm, such as Fisher's linear discriminant, stepwise linear discriminant analysis, support vector machine.

As described above, according to the adaptive brain-computer interface device according to the present invention, it is possible to adaptively determine the order of stimulation according to the situation so that the user can accurately find the desired command as quickly as possible.

In addition, by identifying a large number of commands and messages per unit time, there is an effect that can be applied to the control of the external device even in a situation where it is difficult to use the handicapped or hands and feet.

1 is a block diagram of an adaptive brain-computer interface device in accordance with an embodiment of the present invention.
2 is another configuration diagram of an adaptive brain-computer interface device according to an embodiment of the present invention.
3A is a diagram showing an experimental result of a success rate in a [2 × 2] matrix.
Fig. 3b shows the experimental result of the success rate in the [2x3] matrix.
4 is a schematic diagram of a conventional adaptive brain-computer interface device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, it should be noted that the same components or parts in the drawings represent the same reference numerals as much as possible. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted so as not to obscure the subject matter of the present invention.

1 is a block diagram of an adaptive brain-computer interface device according to an embodiment of the present invention.

Adaptive brain-computer interface device according to an embodiment of the present invention, as shown in Figure 1, the stimulus generating device 100, the signal collection device 200, the preprocessing device 300, the analysis device ( 400 and the stimulation order determination device 500.

The stimulus generating device 100 generates a stimulus and applies a stimulus to a user.

Specifically, the stimulus generating device 100 provides a stimulus corresponding to each command to the user so that a peak in the + direction of the event-linked potential occurring after P300, that is, about 300 ms after the stimulus is given, under a condition desired by the user. It is a device to generate stimulation.

The stimulus generating method in the stimulus generating device 100 follows the general P300 speller system.

That is, the user gazes at the character to select, and the characters in the matrix blink 250ms one character at a time. At this time, the text becomes bright for the first 125ms among the 250ms, and the text becomes dark for the remaining 125ms.

When the task of finding a single character intended by the user is called test, there may be a 2.5 second pause between two consecutive tests.

The signal collection device 200 is a device for recording the brain wave signal of the user generated by the stimulus.

The EEG signal may be collected at 1 kHz from 16 channels through the Biopac MP 150 system.

Since P300 occurs 300ms after the stimulus, the EEG signal corresponding to one blinking stimulus may consist of data from 200ms to 450ms after the stimulus.

The preprocessor 300 is a device for extracting the characteristics of P300 for the stimulus from the EEG signal.

Since the EEG signal includes all the active states of the brain resulting from various causes, the P300 is not directly exposed by the EEG signal generated from other causes. In other words, since the EEG signal includes a lot of noise, direct measurement of the P300 is difficult.

To solve this problem, various techniques of machine learning can be used.

Obtaining purified EEG using a filter to facilitate the detection of P300 is called P300 feature extraction. To do this, remove the noise by calculating the average of EEG, or extract P300 such as spatial filter algorithm or Mexican hat wavelet. An algorithm can be used to remove the noise.

The analysis device 400 is a device for determining the presence of the P300 from the extraction signal of the preprocessing device 100.

The analysis device 400 may include a P300 classifier. The P300 classifier may be Fisher's linear discriminant, stepwise linear discriminant analysis (SWLDA), and support vector machin (SVM).

In order to generate the preprocessor 300 and the classifier, first, a stimulus is given to a target, that is, a character gazing at the user, and a stimulus is generated on a non-target, ie, a character not gazed at the user. In this case, training data consisting of EEG signals can be collected.

Since the raw EEG signals contain a lot of noise, the EEG signals can be band-pass filtered (0.5-30 Hz) through a 6th-order Butterworth filter and down-sampled to 100 Hz.

In order to extract a feature from the collected EEG signal, the spatial projection algorithm may be used.

This algorithm calculates filters that transform the EEG signal collected from each channel into a signal collected from one virtual channel, semantically, so that the EEG signal corresponding to the target and non-target can be distinguished as much as possible.

Meanwhile, a LIBLINEAR package can be used as a classifier to determine whether a feature extracted from a preprocessor (Preprocessor, 300) is a target in which P300 exists. You can use L2-regularized logistic regression to get a mistake.

In this case, the mistake means a probability that the feature is a target, and the parameters of the classifier may be determined through 5-fold cross-validation from training data.

The stimulation order determination device 500 is a device for inferring the current state from the observation of the analysis device 400, and selecting an optimal stimulus for the current state to determine the stimulation order of the stimulus generating device 100.

In general, the partially observable Markov decision process (POMDP) is a general decision framework for partially observable problems.

It is a suitable model for real problems with uncertainty, assuming an accurate model of the real world, and finding the optimal policy of action in a given model.

The POMDP is defined by eight elements of <S, A, Z, b ₀ , T, O, R, γ>.

S is the set of states, A is the set of actions, Z is the set of observations, b ₀ is the initial confidence state, and b ₀ (s) represents the probability that the state of the initial environment is s.

In addition, T is the state transition probability T (s, a, s ') is the probability of transition from state s to s' through action a, O is the observed probability, O (s, a, z) is the action The probability of reaching observation state z by reaching state s through a.

In addition, R is the compensation function R (s, a) represents the reward obtained when taking action a in state s, and γ has a real value between 0 and 1 as the discount rate.

The actor cannot directly know the current state of the environment, but can only obtain observations. Therefore, the actor maintains the probability distribution of the current environmental state through all the behaviors up to now and the corresponding observations. This is called a trust state, and in the case of b _t (s), it represents the probability that the state at time t is s.

Trust status at time t, that is, when performing the action in a _t b _t and scored observation z _{t +1,} the trust state of the next time, b _{t +1} = τ _(t b, _t a, z _{t + 1} ) is calculated by Equation 1 below by Bayes rule.

이 되도록 하기 위한 정규화 상수이다.
Where P (z _{t +1 |} b _t , a _t )

Normalization constant to ensure that

The action policy determines the action to be performed by the actor, which is expressed as a response from trust state to action.

Every behavioral policy has a corresponding value function, which represents the expected value of a discounted reward function that can be obtained from a given trust state when an action is performed infinitely according to the behavioral policy.

Thus, solving POMDP is to calculate the behavioral policy that will get the most value from each trust state.

The maximum value that can be obtained in any confidence state is calculated recursively as in Equation 2 below.

In addition, an optimal behavior policy corresponding to a given optimal value function is calculated as in Equation 3 below.

In practice, it is impossible to calculate the optimal behavioral policy because it takes too long, so optimal behavioral policy approximation algorithms such as Point-Based Value Iteration can be used.

2 is a block diagram of an adaptive brain-computer interface device according to an embodiment of the present invention.

The stimulation order determination device 500 may apply a partial observation Markov decision model (POMDP) to determine an optimal behavior policy.

In this case, the environment of a given brain-computer interface should be modeled with POMDP and the optimal behavior policy should be calculated from the given POMDP model. Thereafter, as shown in FIG. 2, the trust state updater 510 and the optimum stimulus selector 520 may be configured by using the POMDP model and the optimal behavior policy of the POMDP model.

The optimal behavior policy is the same as the optimal behavior sequence in the brain-computer interface, which determines the sequence of stimuli that can grasp the user's intended instruction as quickly as possible with maximum accuracy in the case of noisy observations. do.

The confidence state updater 510 may infer a probability distribution of the current state by using a confidence state update method of the general POMDP from previous stimuli given to the user and observations corresponding to each stimulus.

That is, the confidence state updater 510 is a part of inferring a probability distribution as to whether a current user wants to input the command.

The optimal stimulus selector 520 may select and execute an optimal stimulus corresponding to the trusted state of the trusted state updater 510, and may transmit a command or a message to the external device 600.

That is, the optimal stimulus selector 520 selects and performs an action corresponding to the current trust state through a predetermined optimal behavior policy or an approximate optimal behavior policy.

As described above, when the environment of the brain-computer interface is modeled with POMDP, the model should be able to obtain an optimal behavior policy that accurately finds the command intended by the user with the smallest number of stimuli.

This is very similar to the tiger problem in the field of POMDP, where the number of statements in the tiger problem can be thought of as being expanded by the number of instructions in the brain-computer interface.

When N is the number of commands, each command can be configured with each state (N states) in POMDP, giving the user a stimulus corresponding to each command, or selecting each command (2 * N Dog's behavior), and real numbers from classifiers can be composed of continuous observations, or discretized, each of which consists of observations (K observations).

You can also set low rewards for stimulating behaviors, set high rewards when you take action to select commands that you are watching, and very low for actions that users are not watching. Compensation can be set.

In addition, the state transition function may be the same as the previous state for the stimulating action, and the state transition occurs with the same probability for all characters for the action that the user selects as the intended command.

In addition, the observation function contains a lot of noise in the EEG, so it is impossible to model the actual brain-computer interface in the same way as the operating environment, but pre-experimental experiments are performed from the existing non-invasive brain-computer interface without POMDP. The data can be collected and used for beta distribution, exponential distribution or arbitrary probability distribution.

In addition, the discount rate can be adjusted so that the actor acts properly by mistake between 0 and 1, and the initial trust state is a probability distribution, and can be adjusted appropriately according to the initial environment state.

On the other hand, the state in the POMDP can be thought of as a command that the current user wants to select, but the brain-computer interface does not know this state directly.

Therefore, from the stimuli given to the user and the observations corresponding to each stimulus, it is necessary to infer the command that the user wants to input as the probability distribution for each state, that is, the confidence state.

When the brain-computer interface gives a user a stimulus, if the stimulus is a stimulus for a command that the user wants to enter, the observations given are frequently observed when a stimulus is given for the command the user wants to enter. The probability increases.

On the contrary, if a given stimulus is not a stimulus for a command that a user wants to input, the probability that a given observation is not frequently generated when a stimulus for a command that a user wants to input is given.

That is, from the act of giving one stimulus and corresponding observations, the probability that the command corresponding to each stimulus is the command desired by the user can be inferred as a confidence state in the POMDP, and the specific command is the command desired by the user. If the probability is relatively high, the brain-computer interface may further generate a stimulus for the command, thereby determining whether the command is a command desired by the user.

Then, when the probability that a particular instruction is the user's desired instruction rises above a certain level, rather than taking a stimulus action, choose the instruction to act to maximize the expected value of the value obtained from a given POMDP model. do.

On the other hand, calculating the optimal behavior policy from the above-described POMDP model can be obtained using existing optimal behavior policy calculation algorithms.

However, in general, the problem of determining the optimal behavior policy of POMDP is known as PSPACE-Complete, and for problems with a large number of states, behaviors, and observations, it takes too long to calculate the exact behavior policy, so it cannot be calculated. . To solve this problem, optimal behavior policy approximation algorithms such as Point-Based Value Iteration and Heuristic Search Value Iteration can be used.

Using this approximation algorithm, we can sufficiently calculate the approximate optimal behavior policy for the POMDP model of appropriate size.

Meanwhile, the stimulation order determination device 500 may determine an optimal behavior policy by applying a delayed observation POMDP.

The basic POMDP may assume that an observation is given after performing an action and before performing the next action, but this assumption may not hold true in an actual brain-computer interface.

For example, in the case of giving the stimulus to the user at intervals of 250 ms, the length of the EEG to be used as the observation corresponding to the first stimulus may exceed 250 ms. In particular, in the brain-computer interface using P300, P300 can be detected only 300ms after the user is stimulated. Therefore, observation of the first stimulus can be obtained at least after the second stimulus is given.

This is called delayed observation, and this constraint does not affect the actual POMDP model process, but problems may arise in calculating the optimal behavior policy of the POMDP model. This constraint can be solved using the POMDP with delayed observations.

In addition, the stimulation order determination apparatus 500 may determine an optimal behavior policy based only on the behavior except the behavior performed within the time when the repetition blindness phenomenon due to the overlap stimulus occurs.

Repetition blindness due to overlapping stimuli is, for example, when the user's intended command is "A", the "A" is returned to the user again within 500ms after giving the user a stimulus corresponding to "A". If you give a stimulus corresponding to P300 does not occur in the EEG for the second stimulus.

A simple way to solve this is to not give back the stimulus that was given to the user within 500 ms.

This also has no effect on modeling the actual POMDP, but there is a problem in calculating the optimal behavior policy of the POMDP model. To solve this problem, when A is a set of behaviors and A 'is a set of actions performed within 500 ms, it can be calculated by defining a value function as shown in Equation 4 below.

Hereinafter, an experimental example using an adaptive brain-computer interface device according to an embodiment of the present invention will be described.

In the following Experimental Example, the experiment was conducted through nine persons without physical and mental abnormalities, and in the case of Baseline, the same as the present invention except that the order of stimulation was arbitrary, PWSA indicates a device to which POMDP was applied. .

For the fair comparison of the baseline and PWSA, repetition blindness was solved by randomly selecting a stimulus among two stimuli different from the previous two stimuli.

<Experimental Example>

First, nine POMDP models with different observation probabilities for the [2x2] matrix were created, and eleven different models for the [2x3] matrix were created, and the behavioral policy of the model was calculated before the experiment.

In this experiment, training data was first collected to generate preprocessor and classifier for each subject.

Subsequently, baseline and PWSA experiments were performed on [2x2] and [2x3] matrices, respectively. In the case of PWSA, we found the POMDP model that most closely resembled the distribution of observations for the subject and determined the order of the stimuli using the behavioral policy of the model.

For evaluation, baseline and PWSA were compared for success rate and bit rate. At this time, the accuracy is defined as the ratio of the user's intention to a given number of stimuli, and in the case of bit rate, it indicates the amount of information transmitted per unit time.

In fact, nine experiments were carried out, but two of them were excluded from the experimental results because the observation probabilities were very different from the POMDP model.

FIG. 3A is a diagram showing an experimental result of a success rate in the [2x2] matrix, and FIG. 3B is a diagram showing an experimental result of a success rate in the [2x3] matrix.

As shown in FIGS. 3A and 3B, it can be seen that the probability of success of the PWSA is higher for any number of stimuli than the baseline, and the difference in accuracy becomes larger as the size of the matrix increases. The PWSA also converges to higher accuracy at a faster rate.

On the other hand, the experimental results of the bit rate is as shown in Table 1 below.

[2x2] Matrix

[2x3] Matrix

Baseline

10.065 (96.4%)
8.052 (92.9%)
PWSA

24.368 (98.2%)
21.367 (97.6%)

 As shown in [Table 1], the PWSA showed 24.368 bits / min with 98.2% accuracy for the [2x2] matrix and the bit rate of 21.367 bits / min with 97.6% with the [2x3] matrix. Indicated.

For each matrix, the baseline showed up to 96.4% and 92.9% accuracy, and the bit rates for the accuracy were only 10.065 bits / min and 8.052 bits / min.

As described above with reference to the drawings illustrating an adaptive brain-computer interface device according to the present invention, the present invention is not limited by the embodiments and drawings disclosed herein, it is within the scope of the technical spirit of the present invention Of course, various modifications can be made by those skilled in the art.

100, 10: stimulation generator 200, 20: signal acquisition device
300, 30: pretreatment device 400, 40: analysis device
410: classifier 500: stimulation order determination device
510: Trust status update unit 520: Optimal stimulus selector
600,50: external device

Claims (8)

A stimulus generating device for generating a stimulus and applying a stimulus to a user;
A signal collection device for recording an EEG signal of the user generated by the stimulus;
A pre-processing device for extracting characteristics of P300 for the stimulus from the EEG signal;
An analysis device for determining whether P300 is present from an extraction signal of the preprocessor; And
An adaptive brain-computer interface device for inferring a current state from the observation of the analysis device, and selecting an optimal stimulus for the current state to determine a stimulation order of the stimulus generating device. .
The method of claim 1,
The stimulation order determining device is adapted to determine the optimal behavior policy by applying a partial observation Markov decision model (POMDP).
The method of claim 2,
The stimulation order determination device,
A confidence state updater for inferring a probability distribution of the current state from previous stimuli given to the user and observations corresponding to each stimulus; And
And an optimum stimulus selector configured to select and perform an optimal stimulus corresponding to the trusted state of the trusted state updater, and to transmit a command or a message to an external device.
The method of claim 1,
The apparatus for determining stimulation order is an adaptive brain-computer interface device, characterized in that a delayed observation POMDP is applied to determine an optimal behavior policy.
The method of claim 1,
The apparatus for determining stimulation is an adaptive brain-computer interface device, characterized in that for determining the optimal behavioral policy based only on the behavior except for the behavior performed within a time when repetition blindness due to overlapping stimulation occurs.
6. The method of claim 5,
The optimal behavior policy,
Adaptive brain-computer interface device, characterized in that it is calculated by a value function defined by the following equation.

(Where A is the set of actions and A 'is the set of actions performed within 500ms)
The method of claim 1,
The pretreatment device,
Adaptive brain-computer interface device, characterized in that by removing the noise by averaging the EEG signal, or using a P300 extraction algorithm, such as spatial filter algorithm, Mexican hat wavelet.
The method of claim 1,
The analysis device,
Using classification algorithms such as Fisher's linear discriminant, stepwise linear discriminant analysis, and support vector machine Adaptive brain-computer interface device, characterized in that consisting of a P300 classifier.

Images