KR20190030611A - Method for integrated signal processing of bci system - Google Patents

Abstract

The present invention relates to a method for processing an integrated signal of a brain-computer interface system. The method for processing an integrated signal performed by a brain-computer interface (BCI) includes the steps of: a) constructing an integrated paradigm by integrating at least one paradigm for acquiring a user′s brain signal and collecting at least one kind of brain signals through the integrated paradigm; b) providing an integrated framework for performing feature extraction and pattern classification of brain signals collected from the integrated paradigm; c) analyzing the brain signals collected from the integrated paradigm to generate input information based on time information, frequency information, and spatial information and providing the input information to the integrated framework; and d) classifying the brain signals based on information learned through the integrated framework to recognize user′s intention and outputting a control signal of an external device according to the recognized user intention.

Description

[0001] METHOD FOR INTEGRATED SIGNAL PROCESSING OF BCI SYSTEM [0002]

The present invention relates to an integrated signal processing method based on a brain-computer interface system capable of integrating various paradigms into a single paradigm and extracting characteristic pattern information of a user's brain signal irrespective of the type of paradigm.

The Brain-Computer Interface (BCI) is an interface system that controls external devices without direct body movement through the connection between the human brain and the machine. These BCI technologies use brain waves induced by various paradigms.

The brain signals are classified into electroencephalography (EEG) using electrical signals according to brain activity, magnetoencephalography (MEG) using magnetic signals induced together, functional Magnetic Resonance (MEG) using changes in blood oxygen saturation Imaging, fMRI) or near-infrared spectroscopy (NIRS). EEG signals, which are inexpensive, portable and time-resolved, are widely used as signal sources for BCI.

EEG-induced brain waves can be divided into involuntary methods in which specific external stimuli are projected on the brain or voluntary methods in which the user concentrates on a specific body part or imagines movement.

In particular, steady-state visual evoked potential (SSVEP) and event-related potential (ERP) have been studied as characteristics of involuntary EEG caused by external stimuli. SSVEP is a brain signal feature that is triggered by the user projecting a specific frequency (3.5 ~ 75Hz) to the brain when the user strobes a visual stimulus that blinks at a certain frequency.

In other words, SSVEP is an EEG signal with frequency characteristics. An ERP is a brain signal feature that responds to a stimulus when the user is repeatedly presented with a particular stimulus. In the case of ERP, a specific visual / auditory / tactile stimulus is expressed at a certain time interval. In this case, the brain recognizes the stimulus and uses the brain signal generated after 300 ms. ERP is an EEG signal with temporal characteristics. The disadvantage of such involuntary EEG features is that the subject's point of view must be constantly fixed, and the subject's fatigue can be caused by stimulation.

In contrast, there is the Motor Imagery (MI), a feature of the brain wave that is spontaneously expressed when the user concentrates or imagines the motion without any external stimulus. MI is the amplitude change of the brain signal in a specific frequency range (mu-band (8 to 12 Hz), beta-band (18 to 25 Hz)) while imagining the motion of each part (eg right hand, left hand, Can be observed. Representative features include Event-Related Desynchronization (ERD) and Event-Related Synchronization (ERS). ERD is a phenomenon in which the mu-band wave temporarily decreases in the sensorimotor cortex when intended or imagined, and ERS is a transient increase in beta-band wave after movement.

1 is a flowchart illustrating a process of processing an EEG in the conventional ERP, MI, and SSVEP paradigms.

Referring to FIG. 1, the paradigm of ERP, MI and SSVEP consists of five data processing processes. (S110) for collecting brain signals, a signal processing step (S120) for preprocessing the measured data, a feature extraction step (S130) for extracting meaningful data from brain signals, a decision boundary based on the information obtained from the data (S140), and a feedback step (S150) for delivering the given result to the user.

The brain-computer interface (BCI) system allows the user to experiment through one of paradigms of SSVEP, ERP, and MI. The brain signal obtained through this experiment is processed through the above-described five steps of data processing, And the like.

Illustratively, in the case of an SSVEP analyzing based on an external stimulus first, when striking one of several different frequencies (for example, 7 Hz, 9 Hz, 11 Hz, 13 Hz, etc.) It can be seen that the brain signal changes at one frequency (7 Hz). The EEG obtained based on the external frequency stimulus extracts the brain signal characteristics related to the frequency through the frequency analysis and classifies the brain signals based on the extracted brain signal characteristics.

ERP is a brain signal characteristic that appears after a certain period of time (300 ms, etc.) in response to external stimuli. In case of ERP, unlike Steady-State Visual Evoked Potential (SSVEP), which extracts frequency EEG features, it extracts EEG features based on time information and classifies brain signals based on this.

Alternatively, a frequency magnitude change within a specific frequency range of 8 to 30 Hz may be used in the case of an operating image that is spontaneously expressed without external stimulation, that is, in the case of MI (for example, left, right, And classifies brain signals based on the extracted signals.

The existing BCI system has two limitations. First, experiments should be conducted for each paradigm to collect different kinds of brain signals (SSVEP, ERP, MI, etc.). Second, each paradigm needs different brain signal processes to extract brain signal features and patterns, which is different from the practical and popular BCI intrinsic purpose.

Korean Patent No. 10-1389015 (entitled "Electroencephalogram Analysis System Using Amplitude Modulated Steady State Visual Induced Potential Visual Stimulation")

One embodiment of the present invention integrates various paradigms into one paradigm and provides an integrated framework for feature extraction and pattern classification of the user's brain signals acquired through an integrated paradigm, To provide a method of integrated signal processing based on a brain-computer interface system capable of providing a popular brain-computer interface technology that does not need to go through a process.

It is to be understood, however, that the technical scope of the present invention is not limited to the above-described technical problems, and other technical problems may exist.

According to an aspect of the present invention, there is provided a method for processing an integrated signal based on a brain-computer interface system, the method comprising the steps of: generating a combined signal, which is performed by a brain-computer interface (BCI) A method comprising: a) constructing an integrated paradigm by incorporating at least one paradigm for acquiring a user's brain signal, and collecting at least one type of brain signal through the integrated paradigm; b) providing an integrated framework for performing feature extraction and pattern classification of brain signals collected from the integrated paradigm; c) analyzing brain signals collected from the integrated paradigm to generate input information based on time information, frequency information, and spatial information, and providing the input information to the integrated framework; And d) classifying the brain signal based on the information learned through the integration framework to recognize the user's intention and outputting a control signal of the external device according to the recognized user's intention.

According to the present invention, it is possible to eliminate the hassle of processing different brain signals for each paradigm and to perform feature extraction and pattern classification for brain signals regardless of the paradigm, Can be greatly improved.

The integrated signal processing method based on the brain-computer interface system according to an embodiment of the present invention can be applied to rehabilitation and motion aids for persons with disabilities who are inconvenienced by physical movements or to use for entertainment and education for normal persons .

1 is a flowchart illustrating a process of processing an EEG in the conventional ERP, MI, and SSVEP paradigms.
2 is a diagram illustrating a configuration of a brain-computer interface system for integrated signal processing according to an embodiment of the present invention.
3 is a block diagram illustrating the configuration of the processor of Fig.
4 is a flowchart illustrating an integrated signal processing method of a brain-computer interface system according to an embodiment of the present invention.
FIG. 5 is a view for explaining a feature extraction and pattern classification process according to an integrated framework according to an embodiment of the present invention.
FIG. 6 is a view for explaining a CNN learning model applied to the integration framework of FIG. 5. FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "including" an element, it is to be understood that the element may include other elements as well as other elements, And does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, or combinations thereof.

In this specification, the term " part " means a unit realized by hardware or software, a unit realized by using both, and one unit may be realized by using two or more hardware, The above units may be realized by one hardware.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

2 is a diagram illustrating a configuration of a brain-computer interface system for integrated signal processing according to an embodiment of the present invention.

Referring to FIG. 2, a brain-computer interface system 100 for integrated signal processing includes a memory 110, a processor 120, and a database 130.

The memory 110 integrates various paradigms into one, and a program for performing an integrated signal processing method capable of extracting brain signal pattern information irrespective of the type of paradigm without performing different brain signal processing processes for each paradigm is recorded do. In addition, the processor 120 performs a function of temporarily or permanently storing data to be processed. Herein, the memory 110 may include volatile storage media or non-volatile storage media, but the scope of the present invention is not limited thereto.

The processor 120 controls the entire process of providing an integrated signal processing method of a brain-computer interface system. Each step performed by the processor 120 will be described later with reference to FIG. 3 and FIG.

Here, the processor 120 may include any kind of device capable of processing data, such as a processor. Herein, the term " processor " may refer to a data processing apparatus embedded in hardware, for example, having a circuit physically structured to perform a function represented by a code or an instruction contained in the program. As an example of the data processing apparatus built in hardware, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, an application-specific integrated circuit (ASIC) circuit, and a field programmable gate array (FPGA), but the scope of the present invention is not limited thereto.

The database 130 stores data accumulated while the brain-computer interface system performs the integrated signal processing method. In particular, the database 130 stores and manages the learned feature values based on the CNN learning model using brain signals acquired from the paradigms SSVEP, MI, and ERP.

In addition to the above configuration, the brain-computer interface system 100 may further include communication interface means for collecting brain signals or outputting control signals of external devices, a process for integrated signal processing, and a display means for outputting results have.

3 is a block diagram illustrating the configuration of the processor of Fig.

Referring to FIG. 3, the processor 120 includes a paradigm integration unit 121, an integration analysis unit 122, a signal input unit 123, and a control unit 124.

The paradigm integrator 121 integrates a plurality of paradigms, such as ERP, SSVEP, and MI, into an integrated paradigm. Here, the paradigm means the experimental environment for acquiring the brain signal.

The integration analyzer 122 performs feature extraction and pattern classification of the user's brain signal by providing an integrated framework for performing feature extraction and pattern classification of the brain signals collected from the integrated paradigm. Here, the integrated framework means a method of simultaneously performing the feature extraction and the pattern classification processing. In other words, the integrated framework constructs the feature extraction and pattern classification of acquired brain signals based on deep learning, which is one of the machine learning methods, instead of going through different signal processing processes for each paradigm, regardless of the type of paradigm. The most important advantage of such a deep run is that the feature extraction and pattern classification of the brain signal are automatically extracted rather than directly extracted by the human.

The signal input unit 123 transmits the input information based on the time information, the frequency information, and the spatial information extracted from the brain signal received from the paradigm integration unit 121 to the integration analysis unit 122.

When the input information is input to the integrated framework, the control unit 124 recognizes the user intention through pattern classification based on the brain signal in the integrated framework, and outputs the control signal of the external device according to the user's intention thus recognized.

FIG. 4 is a flowchart illustrating an integrated signal processing method of a brain-computer interface system according to an exemplary embodiment of the present invention. FIG. 5 is a flowchart illustrating a feature extraction and pattern classification process according to an integrated framework according to an exemplary embodiment of the present invention FIG. 6 is a view for explaining a CNN learning model applied to the integrated framework of FIG. 5. FIG.

Referring to FIGS. 4 to 6, the brain-computer interface system provides an integrated paradigm by configuring different paradigms such as ERP, SSVEP, and MI as one integrated paradigm (S10).

The brain-computer interface system acquires the user's brain wave based on the integrated paradigm (S20), and classifies the extracted feature and pattern of the brain signal irrespective of the type of paradigm in the integrated framework (S30). As shown in FIG. 5, when the user's brain waves are acquired, the integration framework outputs pattern classification results of brain signals through processes of brain signal preprocessing, feature extraction, and classifier generation, And selects one of the paradigm algorithms of SSVEP.

At this time, the brain-computer interface system analyzes the brain signals received from the integrated paradigm and constructs input information based on time information, frequency information, and spatial information, and then transmits the input information to the integrated framework. That is, the signal input unit 123 collects brain signals using a plurality of EEG channel data according to a predetermined electrode arrangement method, and performs a fifth-order Butterworth frequency filtering of the collected brain signals to 0.5 to 40 Hz Remove noise.

The signal input unit 123 selects only a brain signal (for example, 2500 ms) for a predetermined measurement time (for example, 1 second to 3 seconds) from a point of time when the start signal for the EEG task is recognized in at least one paradigm And forms a three-dimensional tensor-type input information including time information, spatial information, and frequency information of the extracted brain signal. For example, the input information may be defined as a three-dimensional tensor of time information (2500 seconds) X spatial information (32 channels) X frequency information (30).

On the other hand, the arrangement of the electrodes is according to 10-20 International Nomenclature (10-20 International Nomenclature), and is an international well-known arrangement method for arranging the electrodes placed on the head during EEG (Electroencephalogram) test or experiment. The 10-20 arrangement is based on the relationship between the electrodes and the underlying cortex below the electrode. The numbers "10" and "20" indicate that the distance between adjacent electrodes is the front-to-back or the left- 1/2, F-7/3 / z / 4/8, T-7/8, TP-9/10, CP-5/1/2, 2/6, P-7/3 / z / 4/8, PO-9/10, 0-1 / Z / 2.

The brain-computer interface system outputs a control signal of an external device capable of controlling an external device based on the learned information via the integration framework (S40).

6, the integrated framework performs feature extraction and pattern classification of a brain signal using a CNN (Convolution Neural Network) learning model. The CNN learning model includes a convolution layer and a pooling Pooling It is possible to divide a feature extraction part (Fully Connected Layer) which repeatedly stacks a layer and a classification part which applies Softmax to the final output layer.

In order to classify EEG tasks, the CNN learning model reduces the input information of the 3D tensor type into low dimensional feature vectors for each frequency.

The CNN learning model extracts the peak value through the max pooling process that calculates the maximum value from the reduced low dimensional feature vector in the pooling layer and extracts the significant feature through the sum pooling process that combines all the information .

The CNN learning model extracts a frequency value having the highest peak value and the lowest peak value, a frequency value having the highest island value and the lowest island value, and tracks the gradient of the feature vector for each extracted frequency value Weights are applied to feature vectors with negative slopes or positive slopes to extract meaningful feature vectors.

The CNN learning model determines the ranking of the feature vectors by projecting the significant feature vectors into the regression space and compares the convolution feature values selected with the highest priority in the ranking of the feature vectors to the previously learned information And selects one paradigm among the integrated paradigms.

Based on the paradigm of ERP, SSVEP, and MI, the brain-computer interface system acquires the brain signals of a plurality of subjects, and uses the brain signals of a plurality of subjects to predict brain signals of a plurality of subjects through a CNN learning model And the feature pattern information on the brain signals of the plurality of extracted subjects is stored and managed in the database as learned information capable of classifying each paradigm.

The integration framework also projects the convolution feature values on the feature space, clusters similar feature information based on the Euclidean distance relation between the mean of each convolution feature, fully-connected layer), outputting a total of N class results, and applying a soft max to classify the pattern of brain signals.

The present invention can integrate various paradigms into one paradigm and provide a paradigm independent technology capable of extracting features of a brain signal and classifying patterns regardless of the type of paradigm, have. For example, the brain-computer interface system displays the stimulus for various objects in the augmented reality environment when the user expresses the intention to use through the MI, and detects the intention of the user to select the specific object through the ERP SSVEP enables quick interaction with the selected object according to the user's intention.

The integrated signal processing method of the brain-computer interface system according to an embodiment of the present invention can be applied to an overall BCI system such as rehabilitation and motion jamming apparatus for persons with physical disabilities, . In addition, the present invention can be applied to a remote object control based on a bio-signal for a smart home, and can be applied as a technology for communicating a severely disabled person in a hospital or as a technology for a general public.

The integrated signal processing method of the brain-computer interface system according to the embodiment of the present invention described above may be implemented in the form of a recording medium including instructions executable by a computer such as a program module executed by a computer . Such a recording medium includes a computer-readable medium, which may be any available medium that can be accessed by a computer, and includes both volatile and non-volatile media, both removable and non-removable media. The computer-readable medium may also include computer storage media, which may be volatile and non-volatile, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data , Both removable and non-removable media.

It will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention as defined by the following claims. will be. It is therefore to be understood that the above-described embodiments are illustrative in all aspects and not restrictive. For example, each component described as a single entity may be distributed and implemented, and components described as being distributed may also be implemented in a combined form.

Furthermore, while the methods and systems of the present invention have been described in terms of specific embodiments, some or all of those elements or operations may be implemented using a computer system having a general purpose hardware architecture.

It is intended that the present invention covers the modifications and variations of this invention provided they come within the scope of the appended claims and their equivalents. .

100: Brain-computer interface system
110: Memory
120: Processor
130: Database

Claims (8)

In an integrated signal processing method performed by a Brain Computer Interface (BCI) system,
a) constructing an integrated paradigm by integrating at least one paradigm for acquiring a user's brain signal, and collecting at least one kind of brain signal through the integrated paradigm;
b) providing an integrated framework for performing feature extraction and pattern classification of brain signals collected from the integrated paradigm;
c) analyzing brain signals collected from the integrated paradigm to generate input information based on time information, frequency information, and spatial information, and providing the input information to the integrated framework; And
d) classifying brain signals based on the learned information through the integrated framework to recognize user intent and outputting a control signal of an external device according to the recognized user intention. Integrated signal processing method based on interface system. The method according to claim 1,
The integrated paradigm includes multiple paradigms including the Event-Related Potential (ERP) paradigm, the Steady-State Visual Evoked Potential (SSVEP) paradigm, and the Motor Imagery Integrated signal processing method based on a brain-computer interface system. The method according to claim 1,
The step a)
Collecting a brain signal of a user using a plurality of EEG channel data according to a predetermined electrode arrangement method; And
And removing noise from the collected brain signals through Butterworth frequency filtering having a predetermined frequency range. The integrated signal processing method based on a brain-computer interface system. The method according to claim 1,
The step c)
The method includes the steps of: selectively extracting only a brain signal for a predetermined measurement time from a time point at which a start signal for an EEG task is recognized in at least one paradigm; and extracting the extracted brain signal from three- Wherein the input information is formed of input information in the form of a tensor. The method according to claim 1,
Wherein the integrated framework performs feature extraction and pattern classification of a brain signal using a CNN (Convolution Neural Network) learning model. 6. The method of claim 5,
The integrated framework includes:
Performing a convolution operation for each frequency information of the input information to output a low-dimensional feature vector to reduce the dimension;
Dimensional feature vector, a peak pool is extracted through a max pooling process for calculating a maximum value in the low-dimensional feature vector, and a sum pooling process for combining all the information of the low-dimensional feature vector ;
A frequency value having a maximum peak value or a minimum peak value in the max-pulling process, a frequency value having a maximum sum value or a minimum sum value in the island pulling process, Tracking the slope of the vector;
Extracting a significant feature vector by imposing a weight on a feature vector having a negative slope or a positive slope; And
Determining a ranking of the feature vectors through projection of the extracted feature vectors into a regression space, comparing the convolution feature selected as the highest priority in the ranking of the feature vectors with previously learned information, And selecting a paradigm of the brain-computer interface system. The method according to claim 6,
The integrated framework includes:
Projecting the convolution feature on a feature space and clustering similar feature information based on the Euclidean distance relation between the mean of each convolution feature; And
Further comprising the steps of: learning the fully-connected layer based on the clustered similar feature information, outputting a total of N class results, and classifying the result through a classifier; Based integrated signal processing method. The method according to claim 6,
Comparing the convolution feature value selected as the highest priority in the ranking of the feature vectors with the previously learned information to select any paradigm among the integrated paradigms,
Acquiring brain signals of a plurality of subjects through at least one paradigm and extracting brain signal of the plurality of subjects obtained from the plurality of subjects based on the CNN learning model; And
And storing characteristic pattern information on brain signals of the plurality of extracted subjects as learned information in a database and managing the integrated pattern processing information.

Images