KR102290243B1 - Classification method for Electroencephalography motor imagery - Google Patents

Abstract

The present invention relates to an electroencephalography motor imagery classification method. The present invention relates to an electroencephalography motor imagery classification method, which is performed by at least one processor in a computing device or computing network. The method includes: an embedding step of receiving support data and query data and extracting support features and query features for classification; an attention step of calculating a classification representative feature using the support features and query features obtained in the embedding step; and a relation step of associating the classification representative feature calculated in the attention step with the query feature, and predicting a label. Therefore, it is possible to effectively perform electroencephalography motor imagery classification for a new object by using a relatively small number of data.

Description

Classification method for Electroencephalography motor imagery

The present invention relates to a motor imaginary EEG classification method, and more particularly, to a motor imaginary EEG classification method capable of performing accurate classification while using small-scale data.

In general, Electroencephalography (EEG) motor imagery (MI) classification is one of the EEG based Brain-computer interface (BCI). It is applied in many fields.

Recently, as deep learning techniques have been applied to EEG MI classification, classification performance has been greatly improved.

For example, in Republic of Korea Patent Registration No. 10-16659739 (registered on September 6, 2016, EEG-based control device and method of wearable robot), each EEG signal is classified by movement command to generate learning data and learn. This is described.

However, since these conventional methods do not consider the variance of each object, there is a problem in that classification cannot be performed well unless the data of the object used for learning is used.

As the cause of these problems, electroencephalography has characteristics of low signal-to-noise ratio (SNR) and limited spatial resolution.

In particular, powerful deep learning models have a problem in that there must be large-scale, diversely labeled data for different behaviors.

Conventional motion imaginary (MI) classification methods based on deep learning showed high performance, and a detailed example of the conventional classification method is as follows.

“SU Amin, M. Alsulaiman, G. Muhammad, MA Bencherif, and MS Hossain, “Multilevel weighted feature fusion using convolutional neural networks for EEG motor imagery classification” IEEE Access, vol. 7, pp. 18 940-18950, 2019. 1D CNN was applied to extract multidimensional features and predict labels.

See also “M. Dai, D. Zheng, R. Na, S. Wang, and S. Zhang, “EEG classification of motor imagery using a novel deep learning framework,” Sensors, vol. 19, no. 3, p. 551 , 2019.", after converting the time domain signal to the frequency domain using STFT, applied CNN to the frequency intensity.

and “R. Zhang, Q. Zong, L. Dou, and X. Zhao, “A novel hybrid deep learning scheme for four-class motor imagery classification,” Journal of neural engineering, vol. 16, no. 6, p. 066004, 2019." proposes a network including both CNN and long-term short-term memory model (LSTM) to process sequential time-domain data.

As a follow-up study, “G. Dai, J. Zhou, J. Huang, and N. Wang, “HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification,” Journal of neural engineering, vol. 17, no . 1, p. 016025, 2020.", a hybrid convolution that uses bandpass filters in the frequency bands 4 to 7 Hz, 8 to 13 Hz, and 13 to 32 Hz to separate the signal into three signals and feed them to convolutional layers with different filter sizes. A solution scale CNN is proposed.

Although the above studies have improved classification performance compared to previous traditional classification methods, deep learning methods can often fail when the training sample for a subject (subject) is limited.

Therefore, in order to train a robust model, there is a problem in that it is necessary to secure a large number of training samples for each subject.

An object of the present invention to be solved in consideration of the above problems is to provide a classification method capable of learning using a relatively small amount of data and performing accurate classification according to the learning.

The present invention motor imaginary electroencephalogram classification method for solving the above technical problem is a motor imaginary electroencephalogram classification method performed by at least one processor in a computing device or computing network, and includes support data and query data. data) and an embedding step of extracting support features and query features for classification, and classification representative features using the support features and query features obtained in the embedding step It includes an attention step of calculating, and a relation step of associating the classification representative feature calculated in the attention step with the query feature, and predicting a label.

In an embodiment of the present invention, the embedding step receives a pair of support data and query data as inputs, and extracts semantic features for classification using convolutional layers. , it is possible to separate the EEG data for each frequency band classified according to a specific criterion, and obtain features of the separated frequency band data using the first convolution layer.

In an embodiment of the present invention, the first convolutional layer may have the same number as the number of the divided frequency bands and may have different kernel sizes.

In an embodiment of the present invention, the embedding step includes a process of associating features extracted using the convolutional layer through a second convolutional layer, and a max pooling layer having strides and kernels of the same size again. (max pooling layer) may further include a process of obtaining an embedding feature correlating all features.

In an embodiment of the present invention, the attention step predicts an attention score by associating the set signal characteristics of each support data with the query signal characteristics in the channel direction in order to mainly reflect important signals in the set of support data in the determination The process of obtaining the attention-weighted characteristic expressed by Equation 1 below using the attention score, and the classification representative characteristic of Equation 2 below for each classification by associating the attention-weighted characteristic with the query data characteristic It may include a retrieval process.

[Equation 1]

[Equation 2]

In an embodiment of the present invention, in the relation step, a label may be predicted by learning a relational network using few-shot learning.

In an embodiment of the present invention, a classification having the largest relationship score in the relationship network learning may be defined as a label.

The present invention has the effect of effectively performing motor imaginary EEG classification for a new object using a relatively small number of data.

1 is a flow chart of the present invention motor imaginary electroencephalogram classification method.
2 is a flowchart of an embedding step.
3 is an accuracy graph for each 9 iteration process of the present invention and the conventional classification method.

In order to fully understand the configuration and effects of the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various forms and various modifications may be made. However, the description of the present embodiment is provided so that the disclosure of the present invention is complete, and to fully inform those of ordinary skill in the art to which the present invention belongs, the scope of the invention. In the accompanying drawings, components are enlarged in size than actual for convenience of description, and ratios of each component may be exaggerated or reduced.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above term may be used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first component' may be termed a 'second component', and similarly, a 'second component' may also be termed a 'first component'. can Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those of ordinary skill in the art.

Also, the present invention relates to a method for classifying motor imaginary electroencephalogram, and to a method for learning and classifying electroencephalogram data using the proposed framework.

The present invention provides a processor such as a microprocessor for performing a framework, a storage means for storing EEG data, a memory means for temporarily calling data when classification and learning are performed, an input means for inputting a user's control command, It may be performed in a computing device including a display means for displaying the classification result.

In addition, it may further include an electroencephalogram detecting means for acquiring electroencephalogram data.

The computing device may be a single device, such as a personal computer, a server, a smart pad, or a smart phone, as well as a network device including terminals for collecting EEG data and a server for learning and classifying the collected EEG data.

That is, the present invention is a method using a computing system constituting a single or network, and in particular, may be understood as a process processed by a processor.

Therefore, even if there is no special mention, the subject performing each step constituting the present invention may have a difference in expression, but may be a normal processor.

1 is a flowchart of a motion imaginary electroencephalogram classification method according to a preferred embodiment of the present invention.

1 shows the framework performed by the processor described above, the motor imaginary electroencephalogram classification method of the present invention includes three characteristic steps (framework module).

Specifically, the present invention consists of an embedding step (module), S10), an attention step (module), S20, and a relation step (module), S30).

The embedding step S10 receives support data and query data and extracts features for classification.

Next, in the attention step ( S20 ), a plurality of support features and query features obtained in the embedding step are respectively concatenated and an attention score is predicted.

Then, a weighted average of each of a plurality of support features is obtained based on the attention score, and a class representative feature is calculated.

Next, in the relation step (S30), a relation score for each class is predicted by associating the obtained classification representative feature with the query feature.

In the test stage, the classification with the highest relation score is predicted as a label.

Hereinafter, the characteristics of each step constituting the present invention will be described in more detail.

First, F(x) of the embedding step (S10) takes a pair of support data and query data as inputs, and extracts semantic features for classification using convolutional layers. do.

In the case of using a conventional 3X3 kernel CNN block, it is difficult to learn filters for extracting meaningful features when the same CNN block is applied to a noisy one-dimensional signal.

In the present invention, a technique of extracting a significant signal feature for each frequency and correlating the extracted result is used as a feature.

2 is a flowchart of the embedding step ( S10 ).

For the explanation of the present invention, electroencephalogram data in the left and right and central lobe (C3, C4, CZ) channels were taken as an example. EEG data obtained from the left and right and central lobe (C3, C4, CZ) channels are processed by dividing the frequency domain into 4~7Hz, 8~13Hz, and 14~32Hz.

Characteristics are obtained by passing the convolutional layer for the divided frequency-domain signals. That is, without using the method of directly applying the convolutional layer to the data, the EEG data is separated for each frequency band classified according to a specific criterion, and the data of the separated frequency band is obtained using the convolution layer.

At this time, data (x) obtained from the C3, C4, and CZ electrodes is obtained as frequency band data (xb) by using a fourth-order Butterworth bandpass filter.

Here, b is the divided frequency range of 4-7Hz, 8-13Hz, and 14-32Hz.

For each xb, features are extracted using three convolutional layers having different kernel sizes (45x1, 65x1, 85x1).

The three features obtained through this process are passed through another convolutional layer with a kernel size of 1x3 in order to extract higher-level features between C3, C4, and CZ.

Then, the stride is 6x1 and the max pooling layer is passed through a max pooling layer with a kernel size of 6x1 to finally obtain an embedding feature (z=F(x)) that relates all features. there will be

Next, in the attention step (S20), a classification representative feature useful for predicting a query signal is obtained from the features extracted in the embedding step (S10).

Assuming that the support data set includes k data for each label, l-way and k-shot include lxk data.

Therefore, query from the features (f(Sij), i=1, 2, 3, ..., l, j=1, 2, 3, ..., k) of each data extracted in the embedding step (S10). A classification representative feature f"(Sij) for predicting a signal is obtained.

Conventionally, in order to obtain a classification representative feature, a method of extracting a feature from a signal of each support data and obtaining an element wise sum or average is used.

However, the conventional method could not effectively use data useful for prediction of query signals among data in the support data set because the signals of all support data equally act on the classification representative features.

The conventional method is an easy example to explain why a given query data set belongs to a particular classification, but if a noisy data set sample is present in the support data set, the prediction can become inaccurate.

To solve this problem, in the present invention, it is more advantageous to focus on the support samples using the attention-based approach.

In the present invention, the signal characteristics of each support data set ((Z ^S _ij ) i=1, 2, 3, ..., N j=1, 2, 3 , ..., k) and the query data feature (z ^Q ) are correlated in the channel direction, and then an attention score (a ^S _ij ) is predicted.

In this case, the relational expression can be expressed as Equation 1 below.

)는 연관을 나타낸다.In Equation 1 above, the operation symbol (

) indicates an association.

Accordingly, the attention score a ^S _ij is obtained by the attention function A(Z ^SQ _{ij ).}

To this end, the attention step S20 uses two convolutional layers conv_a1 and conv_a2 and two pulley connection layers fc_a1 and fc_a2.

The two convolutional layers conv_a1 and conv_a2 use a 16x1 kernel and a 4x1 kernel, respectively.

The two pulley connection layers fc_a1 and fc_a2 are 64 and 1-dimensional fully connected layers, taking into account both global and local features.

For each classification, the attention-weighted feature may be expressed by Equation 2 below.

And by associating the attention-weighted feature with Z ^Q to obtain the final classification representative feature including the vector of Equation 3 below for each class, it is supplied to the relation step S30 to finally predict the label. .

Next, in the relation step S30, the label is predicted using few-shot learning.

Conventionally, some pew-shot learning methods predict a label based on a distance metric between a classification representative feature and a query feature, but in the present invention, the label is predicted by learning a relational network.

The function R(x) of the relation step is predicted as a ^{relation score r S} _i =R(Z ^SQ i) derived from the attention-weighted characteristic of Equation (2).

Finally, the classification with the highest predicted relationship score becomes the label.

The relation step (S30) for estimating the label predicted from the 1-dimensional feature vector uses two convolutional layers having kernels of 30x1 and 15x1, respectively, and two pulley connection layers having 256 and 100 dimensions.

For training, the present invention uses a cross entropy loss function as in Equation 4 below.

In Equation 4, y is 1 if the class of the support data and the query data are the same, and is 0 if not the same.

The Adam optimizer mentioned in “D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” “arXiv preprint arXiv: 1412.6980, 2014.” was used to optimize the parameters.

Also, the batch size was set to 100 and the initial learning rate was set to ^{10 -4.}

With each repetition of learning, the learning rate is reduced by an exponential decay ratio of 0.033%.

The model of the present invention was saved when the validation loss was minimal.

Hereinafter, the effect of the present invention will be described with reference to a more specific embodiment.

a. data set

In order to verify the effect of the present invention, "R. Leeb, C. Brunner, G. Muller-Putz, A. Schlogl, and G. Pfurtscheller, "BCI competition 2008-Graz data set B," Graz University of Technology, Austria, pp. 1-6, 2008.” was used.

The data set contains about 120 raw EGG data per experiment for 9 subjects for MI classification, each subject imagines moving to the right or left as instructed.

Five experiments were performed for each subject, and a total of 45 experimental data were collected for nine subjects.

The collected samples were measured at a sampling frequency of 250 Hz at three electrodes (C3, C4, CZ) according to the protocol of the International 10-20 System.

In the present invention, out of 5400 experiments (120x45), some rejected tests were ignored, and signals from 3.5s to 7s were used in the remaining tests.

875 pieces of data were obtained for the three electrodes, and data of an 875x3 matrix was used as input data.

b. test setup

For the experiments with the above data set, 9 cross-validation methods using a model trained with 8 subject data samples were used, and the remainder per validation episode was tested.

Each training set consisted of the first four experiments per subject, and the fifth experiment was used as the validation set.

The test set consists of the full set of the remaining subjects.

During training, support and query data samples were set to be randomly selected from the training and validation sets at each iteration.

For the test, the entire sample was divided into two groups in each classification. Specifically, 20 samples were divided into support data and the remaining samples were divided into query data.

In order to evaluate the effect of the attention of the present invention, the performance of the pew-shot learning model of k = [1, 5, 10, 20] shots without attention is also evaluated.

In addition, in the present invention, HS-CNN trained with only 40 samples in the target's support data set is compared with HS-CNN using all training samples of 8 targets.

As the results may vary as the support data set changes, the same experiment is repeated 10 times and the mean is used as the final accuracy in all cases.

Accuracy is measured based on the number of positives and negatives for the entire data set.

c. result

3 is a graph of the accuracy of each of 9 iterations of the present invention and the conventional classification method, and Table 1 shows the overall accuracy with respect to the standard deviation of the present invention and the conventional method.

3 and Table 1, the present invention exhibits higher performance characteristics than the conventional method in all cases.

In particular, "G. Dai, J. Zhou, J. Huang, and N. Wang, "HS-CNN: a CNN with hybrid convolution scale for EEG motor imagery classification," "Journal, which describes a conventional supervised learning-based method of neural engineering, vol. 17, no. 1, p. 016025, 2020." explains that the classification performance reaches 87.6% when sufficient training data is used, but the training performance drops significantly to 65.3% when the model is trained in a cross-object setting. That is, the model is trained with a small number of samples. However, the accuracy of the data obtained from the subject (20 samples per classification) is very limited.

On the other hand, the present invention can improve performance, with a 1.7% improvement over the prior art in a one-shot setup with only one labeled sample per subject used for training, and 4.9% in a 5-shot. , 6.4% in 10-shot and 8.6% in 20-shot.

Finally, in the present invention, it can be confirmed that the accuracy of the 20-shot setting is an average of 74.6%, which is increased by 9.3% compared to the prior art.

It can be seen that the performance of the attention step proposed in the present invention is also consistently improved.

In particular, in all few-shots (f=1, 5, 10, 20), an average improvement of 1.5% was observed in the present invention.

It has been confirmed that in the attention phase, higher performance can be achieved by suppressing the function of noisy signals while concentrating on the expression of support data set features important for query prediction.

In addition, as shown in FIG. 3 , it was confirmed that the accuracy scores of the prior art (HS-CNN-All) and the present invention were relatively low in the second and third repetition training.

This result is due to the characteristics of the EEG data, because it is difficult to classify the query data with only a few support data sets because the signal-to-noise ratio (SNR) is large.

If the noise in the data is sufficiently reduced, the performance can be greatly improved.

When comparing the average prediction accuracy (avg in Fig. 3) excluding the 2nd and 3rd iterations for a more effective comparison, the present invention shows that the accuracy is improved by 67.3% in the case of the prior art and 79.5% in the case of 20 shots. can

Similar results are obtained in all iterations except for every second and third time, and it can be seen that the performance is improved by more than 12.2%.

As such, the classification method according to the present invention can provide a bidirectional few-shot classification network that performs well on invisible objects by applying the few-shot learning concept to motor imaginary electroencephalogram classification.

The present invention has demonstrated that classification accuracy is improved compared to the prior art, and provides evidence that EEG data can be classified while using a limited sample.

In addition, by emphasizing information in support data that is highly related to query data, performance can be further improved in the attention stage.

Although the embodiments according to the present invention have been described above, these are merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent ranges of embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the following claims.

Claims (7)

A method for classifying motor imaginary electroencephalograms performed by at least one processor in a computing device or computing network, the method comprising:
an embedding step of receiving support data and query data and extracting support features and query features for classification;
an attention step of calculating a classification representative feature using the support features and query features obtained in the embedding step; and
and a relation step of associating the classification representative feature calculated in the attention step with the query feature, and predicting a label. According to claim 1,
The embedding step is
With a pair of support data and query data as inputs, semantic features for classification are extracted using convolutional layers,
A motor imaginary electroencephalogram classification method, characterized in that the electroencephalogram data is separated for each frequency band classified according to a specific criterion, and features of the separated frequency band data are obtained using a first convolution layer. 3. The method of claim 2,
The first convolutional layer is
The motor imaginary electroencephalogram classification method, characterized in that the same number as the number of the separated frequency bands and the size of each kernel is different. 4. The method of claim 3,
The embedding step is
associating features extracted using the convolutional layer through a second convolutional layer; and
Again, passing through a max pooling layer having a stride and a kernel of the same size, and obtaining an embedding feature correlating all features. 5. The method of claim 4,
The attention step is
predicting an attention score by associating the set signal features of each support data with the query signal features in a channel direction in order to mainly reflect an important signal in the set of support data in decision;
obtaining an attention-weighted feature expressed by Equation 1 below using the attention score; and
and a process of associating the attention-weighted feature with the query data feature to obtain the classification representative feature of Equation 2 below for each classification.
[Equation 1]

[Equation 2]

6. The method according to any one of claims 1 to 5,
The relation step is
A motor imaginary electroencephalogram classification method to predict labels by relational network learning using few-shot learning. 7. The method of claim 6,
A motor imaginary electroencephalogram classification method in which a classification having the greatest relational score in the relational network learning is designated as a label.

Images