KR101939363B1 - Brain-computer interface apparatus adaptable to use environment and method of operating thereof - Google Patents

Abstract

The present invention discloses a brain-computer interface device. According to the present invention, a brain signal based control program is stored and a brain signal data of a user, which is received in real time according to execution of a program stored in the memory, is filtered by a predetermined filtering frequency band, And extracts a feature vector and inputs the extracted feature vector to the learned brain signal classifier to determine whether the user's intention is one of a plurality of motions, , A processor performing at least one of filtering frequency band update, common spatial filter adjustment, and brain signal classifier adjustment.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a brain-computer interface device adaptable to a use environment, and a method of operating the same. 2. Description of the Related Art [

The present invention relates to a brain-computer interface apparatus adaptable to a use environment and a method of operating the apparatus.

Brain-computer interface (BCI) is a technology that allows external devices to be controlled using various types of signals generated in the brain of a subject. Brain-computer interface technology enables control of devices using brain signals, which can be used as a rehabilitation and life support tool for patients with Lou Gehrig's disease, which is difficult to move the body. In addition, the brain-computer interface technique can be used as a training tool for enhancing entertainment or concentration such as game using the brain signal of the subject. Brain signals are composed of electroencephalography (EEG) using electrical signals according to brain activities, magnetic resonance imaging (MEG) using magnetic signals induced by electrical signals, functional magnetic resonance imaging (MEG) using changes in blood oxygen saturation functional magnetic resonance imaging (fMRI) or near-infrared spectroscopy (NIRS). EEG signals having excellent portability and time resolution are widely used. A brain-computer interface device using an EEG is a device for interrogating the movement of a specific body part of a subject by means of an intentional (or imaginary) motion of the subject, or a temporal and spatial change of the brain signal pattern , Or a change in the brain signal responding to visual stimulation (i.e., a brain-computer interface device based on a steady state visually evoked potential (SSVEP)) can be used to control an external device such as a computer .

Specifically, the motion intent awareness-based brain-computer interface device is designed to allow a subject to move a specific body part (e.g., a right hand, a left hand, etc.) to cause an event- related desynchronization (ERD) (ERD) / ERS (event-related synchronization) pattern generated in a specific body part of the sensory-motor area of the brain, thereby controlling the external device. On the other hand, the SSVEP-based brain-computer interface device arranges a plurality of light sources for generating light of different frequencies, and as the subject gazes at a specific light source among a plurality of light sources, And the external device is controlled by recognizing the signal. In addition, an event-related potential (ERP) -based brain-computer interface device recognizes the brain stimulation of the subject generated from at least one stimulus of a specific time, And the external device is controlled by recognizing the brain signal generated after a predetermined time (for example, 300 ms).

Generally, the brain-computer interface device repeatedly performs a predetermined task (for example, a subject's intention to move a predetermined body part, a subject concentrates on a predetermined stimulus, etc.) before using the user, And collect brain signals to learn the Classifier. Specifically, the brain-computer interface device extracts characteristic values capable of distinguishing types of tasks from an arbitrary brain signal based on the collected brain signals, and learns the classifier using the extracted characteristic values.

However, the brain-computer interface device using the EEG has a problem that the brain signal pattern changes over time due to changes in the state of the subject, changes in the position of the electrodes, noise due to movement, and the like. As a result, the classification performance of the classifier of the brain-computer interface device decreases as the examination time is continued. Therefore, when a user uses the brain-computer interface device for a long time, a need arises for a brain-computer interface device adapted to the environment of use.

Korean Patent No. 10-1581895 entitled " P300 Brain-Induced Potential Inputting Brain-Computer Interface System " and " P300 Brain-induced Potential Inputting Method "

Disclosure of Invention Technical Problem [8] Accordingly, the present invention has been made to solve the above problems occurring in the prior art, and it is an object of the present invention to provide a brain-computer interface apparatus and a method of operating the same that do not reduce classification performance even when the brain- have.

As a technical means for achieving the above technical object, a first aspect of the present invention provides a brain-computer interface apparatus including a memory for storing a brain signal-based control program, and a processor for executing a program stored in the memory . At this time, according to the execution of the program, the processor filters the user's brain signal data received in real time in a predetermined filtering frequency band, extracts a feature vector by applying a common spatial filter from the filtered data, And the vector is input to the pre-learned brain signal classifier to determine whether the user's intention is one of a plurality of motions. Based on the difference between the user's test data received at predetermined intervals, Filter adjustment, and brain signal classifier adjustment.

According to a second aspect of the present invention, there is provided a method for processing a brain signal, comprising: filtering brain signal data of a user received in real time to a predetermined filtering frequency band; Extracting a feature vector by applying a common spatial filter from the filtered data; And inputting the extracted feature vector to the learned brain signal classifier, thereby determining whether the user's intention is one of a plurality of motions. At this time, at least one of the filtering frequency band update, the common spatial filter adjustment, and the brain signal classifier adjustment is performed on the basis of the difference between the user test data received at regular intervals.

A third aspect of the present invention also provides a computer-readable recording medium having recorded thereon a program for performing the method of the second aspect on a computer.

According to the above-mentioned problem solving means, it is possible to provide a brain-computer interface device capable of providing an optimal use environment for a user even if the user uses the notebook computer for a long time.

1 is a block diagram illustrating a configuration of a brain-computer interface apparatus according to an embodiment of the present invention.
FIG. 2 is a flow chart for explaining how a brain-computer interface device adaptively detects a movement intention of a user.
3 is a flowchart detailing how the processor of FIG. 1 performs at least one of filtering frequency band update, common spatial filter adjustment, and brain signal classifier adjustment based on differences between test data, in accordance with an embodiment of the present invention. to be.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when a part is referred to as " including " an element, it does not exclude other elements unless specifically stated otherwise.

Further, the " motor imagery " referred to below may be referred to as an operational image, a moving image, a dynamic image, and the like.

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings.

1 is a block diagram of a brain-computer interface device 100 according to an embodiment of the present invention.

The brain-computer interface device 100 can receive the user's brain signal and determine the user's intention. And the brain-computer interface device 100 may control the brain-computer interface based device based on the user's intention. At this time, the brain-computer interface-based device may be a living assistance device such as a wheelchair of the user 200 who can not move the body. Alternatively, the brain-computer interface-based device may be, but is not limited to, an Internet appliance for a smart home.

The brain-computer interface device 100 includes a brain-and-signal measuring unit 110, a memory 120, and a processor 130.

The brain-signal measuring unit 110 is attached to a user's head and can measure a brain signal such as an electroencephalogram (EEG) according to the operation of the user 200. For example, a brain signal may be collected from a plurality of regions determined based on a 10-20 system.

The memory 120 stores control programs for controlling the brain-computer interface device 100. For example, the memory 120 may store a brain signal based control program. At this time, the memory 120 is collectively referred to as a non-volatile storage device that keeps stored information even when power is not supplied, and a volatile storage device that requires power to maintain stored information.

The processor 130 controls the overall operation of the brain-computer interface device 100. The processor 130 may include at least one component for controlling the operation of the brain-signal measuring unit 110 and the memory 120. For example, the processor 810 may include at least one of a RAM (Random Access Memory) (not shown), a ROM (Read Only Memory) (not shown), a CPU (not shown) and a GPU (Graphic Processing Unit One can be included.

In addition, the processor 130 can identify the user's motion intention from the user ' s brain signal data by executing programs stored in the memory 120. [ At this time, the motion intention may be a motion of the body thought by the user or an action thought by the user. For example, the motion intention may be "left hand motion", "right hand motion", "left foot motion", and the like.

The processor 130 detects the user's motion intention from the brain signal data received in real time. At this time, the processor 130 performs a preliminary work for setting a suitable use environment for the user before detecting the motion intention of the user 200. [ The processor 130 sets and / or sets an optimal frequency band for detecting the user's motion intention based on data (hereinafter, referred to as 'test data') measured according to a predetermined motion (or behavior) Or adjusts the brain signal classifier, and performs learning and / or adjustment of the brain signal classifier to suit the user.

However, if the use time is increased, the brain-computer interface device 100 may be able to detect the brain (s) measured by the brain-computer interface device 100 due to a change in the user's state, The aspect of the signal may change. Accordingly, the brain-computer interface device 100 according to an embodiment of the present invention operates adaptively to the user's environment by repeatedly measuring the test data at a predetermined time interval (or a time interval). Hereinafter, this will be described in detail with reference to FIG.

On the other hand, the brain-computer interface device 100 can be implemented by more components than the components shown in FIG. For example, the brain-computer interface device 100 may further include a display (not shown) for outputting various information and a communication unit (not shown) for communicating with other devices.

2 is a flowchart for explaining how the brain-computer interface apparatus 100 adaptively detects a movement intention of a user.

Referring to FIG. 2, first, the brain signal measuring unit 110 acquires brain signal data of a user in real time (S100). The acquired brain signal data is provided to the processor 130.

The processor 130 filters brain signal data using a predetermined filtering frequency range (S110). Here, the filtering frequency range may be a frequency band in which the primary motor cortex of the brain responds to activation of the brain according to the user's motional intention, and may be adjusted to be optimized for the user during the pre-work. The filtering frequency range may be, for example, about 8 to 12 Hz.

Thereafter, the processor 130 applies a common spatial filter from the filtered data to extract a feature vector (S120). Specifically, the processor 130 may calculate a covariance matrix of the filtered data based on a common spatial pattern (CSP) algorithm, and then apply a common spatial filter thereto to extract a feature vector. Here, the common space pattern algorithm is an EEG feature extraction algorithm using the Fukunaga-Koontz transform, and is used to search for a feature vector that best reveals the difference between signals collected under two different conditions.

Thereafter, the processor 130 determines whether the user's intention is one of a plurality of motions by inputting the feature vector extracted from the brain signal into the learned brain signal classifier (S130). Here, the brain signal classifier is at least one program (or instruction set) including a class (or category) for identifying a user's motion intention and a feature vector of each class, for example, a linear classifier, A quadratic classifier, and the like.

Then, the brain signal measuring unit 110 acquires test data of the user at regular intervals (S140). Here, the test data may be brain signal data measured as the user considers the promised motion (or behavior) in order to evaluate the performance of the brain-computer interface device 100. The test data may be acquired at predetermined time intervals, or repeatedly each time a predetermined number of real-time brain signal data is received. On the other hand, the test data can be obtained after a certain time from the use time of the user, but is not limited thereto.

Thereafter, the processor 130 performs at least one of filtering frequency band update, common spatial filter adjustment, and brain signal classifier adjustment based on the difference between the test data (S150). That is, the processor 130 can adapt to the user's use environment by updating and / or adjusting at least one of the steps S110, S120, and S130 based on the difference between the test data. To this end, the processor 130 determines which process to optimize based on the difference between the test data based on the frequency difference of the current test data and the previous test data, the difference in the fisher ratio, the difference in the feature distribution, . At this time, the previous test data may be test data immediately before the current test data, but is not limited thereto. For example, the previous test data may be an average value of the previous plurality of test data. Hereinafter, this will be described in detail with reference to FIG.

FIG. 3 is a flowchart detailing how the processor 130 of FIG. 1 performs at least one of filtering frequency band update, common spatial filter adjustment, and brain signal classifier adjustment based on differences between test data.

First, the processor 130 calculates the Fisher's ratio of the signal within the currently used filtering frequency band for each test data (S151). Thereafter, the processor 130 updates the filtering frequency band when the deviation of the Fisher's ratio between the current test data and the previous test data exceeds a preset first threshold value (S152) (S153). At this time, the first threshold value is an experimentally determined value, and can be calculated based on a case where the classification performance in the preliminary operation is reduced to about 70%.

)은 아래의 수학식 1에 의해 산출될 수 있다. On the other hand, the frequency-by-frequency Fisher ratio (

) Can be calculated by the following equation (1).

및

는 각각 클래스 내의 분산 행렬 및 클래스 간 분산 행렬을 나타낸다. 또한,

는 n 번째 시행의 이산 시간(t)-주파수(f) 밀도 패턴이며,

는 클래스 c (c=1, 2, ..., C)에 대한 평균 시간(t)-주파수(f) 밀도를 나타낸다. 또한,

는 전체 클래스에 대한 평균 시간(t)-주파수(f) 밀도를 나타내고,

는 클래스 c의 시행 회수를 나타낸다. In the above equation,

And

Represent the dispersion matrix and the inter-class dispersion matrix in the class, respectively. Also,

Is the discrete-time (t) -frequency (f) density pattern of the nth trial,

Represents the average time (t) - frequency (f) density for class c (c = 1, 2, ..., C). Also,

Represents the mean time (t) - frequency (f) density for the entire class,

Represents the number of trials of class c.

Thereafter, the processor 130 calculates the difference of feature distribution of each test data (S154). For example, the processor 130 may calculate the difference of feature distribution of each test data by Equation (2) using a Kullback-Leibler divergence (KLD) algorithm.

는 이전 테스트 데이터의 가우시안 분포를 나타내며,

은 현재 테스트 데이터의 가우시안 분포를 나타낸다. 또한,

는 i 클래스의 평균을 나타내며,

는 i 클래스의 공분산을 나타낸다. 또한, d는 데이터의 차원(dimension)을 나타낸다. In the above equation,

Represents the Gaussian distribution of the previous test data,

Represents the Gaussian distribution of the current test data. Also,

Represents the average of the i classes,

Denotes the covariance of i-class. Also, d represents the dimension of the data.

The processor 130 may adjust the common spatial filter and / or the brain signal classifier if the feature distribution difference between the current test data and the previous test data exceeds the second threshold (S155) (S156). At this time, the second threshold value is an experimentally determined value, and can be calculated on the basis of a case where the classification performance in the pre-work is reduced to about 70%.

A general brain-computer interface device uses a common spatial filter and a brain signal classifier using a pre-calculated covariance matrix at the time of preliminary work. Therefore, if the user's environment is changed, it may not be reflected. The processor 130 according to an embodiment of the present invention can solve the above problem by adjusting the common spatial filter and the brain signal classifier when the feature distribution difference between the test data exceeds the second threshold value.

)을 갱신하는 일례이다. Specifically, in order to adjust the common spatial filter, the processor 130 may assign a predetermined feature vector weight to the previously calculated covariance matrix. Equation (3) below indicates that the processor 130 assigns a feature vector weight to the covariance matrix < RTI ID = 0.0 >

Is updated.

는 특징 벡터를 나타내며,

은 기 설정된 특징 벡터 가중치를 나타낸다. k는 이산적인 시간을 나타낸다. In the above equation,

Represents a feature vector,

Represents a predetermined feature vector weight. k represents a discrete time.

)가 부여된 특징 벡터를 상기 평균 및 공분산 중 적어도 하나에 적용함으로써 뇌 신호 분류기의 파라미터를 갱신할 수 있다. 아래의 수학식 4는 각 클래스의 특징 분포의 평균(

)을 갱신하는 일례이며, 수학식 5는 각 클래스의 특징 분포의 공분산(

)을 갱신하는 일례이다. In addition, in order to adjust the brain signal classifier, the processor 130 may update the parameters of the brain signal classifier. Here, the parameter may include, for example, an average of characteristic distributions of each class, a covariance, and the like. The processor 130 determines a predetermined feature vector weight

) May be applied to at least one of the mean and the covariance to update the parameters of the brain signal classifier. Equation 4 below represents the average of the feature distribution of each class (

(5) is an example of updating the covariance of the feature distribution of each class

Is updated.

Equation (6) below is an example of adjusting the brain signal classifier as the parameter is updated. Equation (6) is based on a linear discriminant analysis (LDA) algorithm.

는 선형 판별 분석(LDA)의 사상/가중치 벡터를 나타내며,

는 LDA의 바이어스(bias)를 나타낸다. 또한,

는 i 클래스의 평균을 나타내며,

는 클래스 간 공통 평균을 나타낸다. In the above equation

Represents the mapping / weight vector of linear discriminant analysis (LDA)

Represents the bias of the LDA. Also,

Represents the average of the i classes,

Represents the common average among the classes.

As described above, the brain-computer interface device 100 according to an exemplary embodiment of the present invention can acquire real-time brain signal data and acquire test data at regular intervals to adapt to a user-friendly usage environment. Accordingly, even if the user uses the brain-computer interface device 100 for a long time, it is possible to provide an optimal use environment for the user.

One embodiment of the present invention may also be embodied in the form of a recording medium including instructions executable by a computer, such as program modules, being executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. The computer-readable medium may also include computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

While the methods and systems of the present invention have been described in connection with specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It will be understood by those skilled in the art that the foregoing description of the present invention is for illustrative purposes only and that those of ordinary skill in the art can readily understand that various changes and modifications may be made without departing from the spirit or essential characteristics of the present invention. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

The scope of the present invention is defined by the appended claims rather than the detailed description and all changes or modifications derived from the meaning and scope of the claims and their equivalents are to be construed as being included within the scope of the present invention do.

100: Brain-computer interface device
110: brain signal measuring unit
120: processor
130: memory

Claims (9)

A brain-computer interface device,
A memory in which a brain signal based control program is stored, and
And a processor for executing a program stored in the memory,
The processor, upon execution of the program,
The user's brain signal data received in real time is filtered by a predetermined filtering frequency band,
Extracting feature vectors by applying a common spatial filter from the filtered data,
Determining whether the intention of the user is one of a plurality of motions by inputting the extracted feature vector into the learned brain signal classifier,
Performing at least one of the filtering frequency band update, the common spatial filter adjustment, and the brain signal classifier adjustment based on a difference between test data of a user received at regular intervals,
Wherein the test data is brain signal data measured as the user considers the promised motion or behavior. delete The method according to claim 1,
The difference between the test data
Wherein the difference is based on at least one of a deviation of the fisher ratio and a difference of the feature distribution by the frequency range of the current test data and the previous test data. The method according to claim 1,
The processor
And updates the filtering frequency band if the deviation of the Fisher's ratio by frequency range of the current test data and the previous test data exceeds a first threshold value. The method according to claim 1,
The processor
Wherein the common spatial filter is adjusted by assigning a predetermined feature vector weight to a previously calculated covariance matrix if the difference of the feature distribution of the current test data and the previous test data exceeds a second threshold value, Device. The method according to claim 1,
The processor comprising:
And updates the parameters of the brain signal classifier if the difference in the feature distribution of the current test data and the previous test data exceeds a second threshold value. The method according to claim 6,
The parameters of the brain signal classifier are the mean and covariance of the feature distribution of each class,
The processor comprising:
Wherein the parameters of the brain signal classifier are updated by applying a feature vector having a predetermined feature vector weight to at least one of the mean and the covariance. A method for adaptively detecting a user's movement intention by a brain-computer interface device,
Filtering brain signal data of a user received in real time in a predetermined filtering frequency band;
Extracting a feature vector by applying a common spatial filter from the filtered data; And
And inputting the extracted feature vector to the learned brain signal classifier to determine whether the user's intention is one of a plurality of motions,
Performing at least one of the filtering frequency band update, the common spatial filter adjustment, and the brain signal classifier adjustment based on a difference between test data of a user received at regular intervals,
Wherein the test data is brain signal data measured as the user considers the promised motion or behavior. 9. A computer-readable recording medium recording a program for performing the method of claim 8 on a computer.

Images