CN118228129B - Motor imagery electroencephalogram signal classification method based on deep migration learning - Google Patents

Abstract

The invention provides a motor imagery electroencephalogram signal classification method based on deep transfer learning, which relates to the field of brain-computer interface data classification processing of neural networks and machine learning, and comprises the following steps: s1, acquiring brain electrical signals and preprocessing brain electrical signal data; s2, constructing a EEGNet-attribute-Resnet classification model; s3, realizing parameter sharing of the shallow network of the source domain and the target domain by using European alignment, and realizing domain adaptation of the deep network of different subjects by using the maximum mean difference MMD measurement difference. Compared with the prior art, the method and the device for classifying the electroencephalogram signals based on the data migration from fine adjustment and sharing of parameters of the shallow network and domain adaptation of the deep network can solve the problem of low classification accuracy caused by insufficient data quantity, improve the classification accuracy of the electroencephalogram signals of the cross-subjects, complete good classification tasks of different subjects, and can be widely applied to the fields of medical health and the like.

Description

A motor imagery EEG signal classification method based on deep transfer learning

Technical Field

The field of the present invention belongs to the field of brain-computer interface data classification and processing using neural networks and machine learning, and more specifically, to a motor imagery EEG signal classification method based on deep transfer learning.

Background technique

Brain-computer interface has created a new way of human-computer communication for people. It converts the brain's thoughts into actual commands to control external devices. Motor imagery is one of the classic brain-computer interface paradigms and is an imaginary movement behavior. Through motor imagery or certain movement intentions, brain activity can be converted into control signals, which can be captured and identified through electroencephalography (EEG). According to current research, when people imagine their body movements, brain signals will change significantly due to the influence of motor imagery tasks. At the same time, the decoding of motor imagery EEG signals has potential application value.

There are three major challenges in the current research on brain-computer interfaces based on motor imagery: low signal-to-noise ratio, inherent non-stationarity in recording EEG signals, and specificity between different subjects. Many machine learning-based methods have been proposed to address the above three challenges. The good use of machine learning requires a large amount of data to learn the distribution of data. However, in the application of brain-computer interfaces, it is time-consuming and troublesome to re-collect the required training data. Therefore, if you want to use fewer training samples to accurately deal with the problem of differences between individuals, it is far from enough to use traditional machine learning methods alone. Transfer learning is a method in machine learning that focuses on storing the knowledge gained when solving a problem and applying it to different but related problems.

In the fields of image processing and natural language processing, and even in the research of brain-computer interfaces, transfer learning has shown its vital value and influence. Transfer learning can map features from different fields to a unified feature space, increase the amount of data, overcome the problem of limited data, and solve the problem of distribution differences. Transfer learning can not only solve the overfitting problem caused by insufficient data, but also improve the generalization ability of EEG signal recognition models. Although there are many mature algorithms that can accurately decode motor imagery EEG signals, there is still room for improvement in their classification accuracy. For patients who need to be used in medical rehabilitation, the significant differences in classification accuracy between different individuals actually limit its maturity and scope of application.

Summary of the invention

In order to overcome the shortcomings of the prior art, and to address the problems of long training time and poor cross-subject classification effect in the field of motor imagery EEG signal classification, the present invention aims to provide a motor imagery EEG signal classification method based on deep transfer learning, and designs a deep transfer learning algorithm based on the EEGNet-Attention-ResNet model, so that the transfer module can be integrated into the EEGNet-Attention-ResNet model, thereby using the source domain information to help the target domain train a reliable classification model, thereby reducing the number of training samples for subjects in the target domain and shortening the training time for subjects in the target domain.

To achieve the above object, the present invention provides the following technical solution: a motor imagery EEG signal classification method based on deep transfer learning, comprising:

Step S1: Obtain a data set of motor imagery EEG signals and perform preprocessing.

The preprocessing module uses a bandpass filter to preprocess the EEG signal to remove artifacts of the EEG signal to obtain a preprocessed data set, which is divided into a training set and a test set based on the source domain subject data set.

Step S11: Assume that EEG data from different subjects are considered to be different domains. Given a labeled source domain represented as and the unlabeled target domain ,here n channels and m sampling points represent multi-channel EEG data. Represents the source label of the i-th sample. The deep transfer learning model established in the present invention can absorb knowledge from the source domain _Ds to enhance the more accurate prediction and classification capabilities on the target domain _Dt , effectively reducing the distribution drift between data domains.

Step S12: Use a bandpass filter to preprocess the EEG signal in the EEG source domain data set in step S11 to remove artifacts of the EEG signal and obtain preprocessed EEG data.

Step S13: In step S12, the present invention divides the preprocessed EEG data set into a training set and a test set.

Step S2: By applying the EEGNet-Attention-ResNet algorithm to train the classification model, the present invention performs feature extraction on the source domain data set of the preprocessed motor imagery EEG signal, and further processes it through the fully connected layer, and finally obtains and outputs the classification result. Wherein, the classification model is based on the EEGNet neural network, adding a spatial, channel attention mechanism and a ResNet residual network, that is, the EEGNet-Attention-ResNet model, which trains the EEGNet-based model through the training set, and evaluates the performance of the trained model through the test set to obtain a convolutional neural network model.

Step S21: Add a dual self-attention mechanism, namely, spatial and channel attention mechanism, based on the EEGNet neural network layer. Since EEG signals contain rich spatial and EEG channel information, extracting more useful information will greatly improve the classification accuracy.

Step S211: Input the EEG features extracted by the EEGNet network, learn the importance of channels in the EEG signals through the ECA module, and multiply the learned channel weights with the input features to obtain channel attention features. Among them, the ECA channel attention module first uses the global average pooling layer to aggregate the convolution features to obtain the feature vector, then adaptively determines the size of the kernel K, uses one-dimensional convolution to learn the channel weights, and completes the information interaction across channels. This module does not reduce the local cross-channel interaction strategy of dimensionality, which effectively avoids the adverse effects caused.

Step S212: Take the channel attention output feature as input and send it to the CBAM module to learn the importance of spatial position. The learned spatial position weight is multiplied by the channel attention feature to obtain the final result. Among them, the CBAM attention module first performs maximum pooling and average pooling operations on the feature information, and then connects the generated different feature maps and performs convolution operations. The weights are normalized by the Sigmoid function, and finally the weights are multiplied by the input feature map to obtain the final result. By learning the spatial attention weights, we focus on important spatial feature information, suppress repeated and non-critical information, and improve the performance and generalization ability of the entire network.

The weights of the above-mentioned dual-channel sub-attention mechanism are feature-adaptive, which strengthens the correlation between channels and the importance of spatial positions, suppresses non-critical information, and improves the representation ability of EEG data and the accuracy of network perception.

Step S22: Adding the ResNet residual network on the basis of step S21, as the number of network model layers increases, there will be problems such as a sharp drop in recognition rate, increased consumption of computing resources, and gradient vanishing or exploding. By introducing the ResNet residual network, using shortcuts to directly connect, learning the feature mapping extracted from the attention mechanism and the output residual mapping, it does not require a complete mapping, and solves the related gradient vanishing and network degradation problems.

Step S23: Extract the features of the EEG signal through the above model and classify it through the fully connected layer.

Step S24: Finally, update the network weights based on the Adam optimization algorithm.

Step S3: Build a transfer learning model.

Perform model adaptation and build a deep transfer learning model so that the model of the source domain EEG signal can be applied to the motor imagery EEG signal in the target domain, thereby completing the accurate feature extraction and classification of the motor imagery EEG signal.

Step S31: First, input the source domain, the preprocessed channel data of the target domain and the EEG signal data of the target domain.

Step S32: In the shallow network layer, the features of EEG signals are often universal in the field. The universal features are aligned by the Euclidean alignment method, that is, a feature alignment module is added after the time and space filters of EEGNet. The module is composed of filters and feature alignment modules, namely the Euclidean alignment module, to achieve data alignment. The feature extraction model trained in step S2 realizes the sharing of shallow parameters of source domain data and parameters of target domain data. The expression for executing Euclidean alignment is as follows:

, ,

represents the mean of n covariance matrices, represents the covariance function, represents a sample of EEG signal, represents the transposition of EEG signal sample data, n represents the total number of signal samples, Represents the sample representation after center alignment.

Step S33: As the network depth increases, the feature representations of different subjects become more specific. The present invention achieves more refined domain adaptation of the feature extractor output features by adding a domain adaptation module.

An adaptation layer is added between the output features of the source domain data and the target domain data feature extractor, and the maximum mean difference MMD measurement function is added between the adaptation layers to measure the difference in the output features of the source domain data and the target domain data. The multi-core maximum mean difference MK-MMD maps the source domain data and the target domain data to the reproducing kernel Hilbert space through multiple Gaussian kernel functions, and measures the distance between the two distributions p and q in the reproducing kernel Hilbert space. The kernel function K defined by multiple kernels is described by the formula as follows:

,

in is the weight of different Gaussian kernels. The weight of Gaussian kernels with large contributions is large, and the weight of Gaussian kernels with small contributions is small. _Ku is the uth Gaussian kernel, and it is added to the loss of the network to continue training. m represents the number of Gaussian kernels, and k represents the combination of different kernel { _ku } functions. MK-MMD can be expressed by the following formula:

,

In the formula, represents the distance of the reproducing kernel Hilbert space H _k , where , They represent the mapping of source domain data _Ds and target domain data _Dt in the reproducible kernel Hilbert space respectively, and _Ep and _Eq represent the mathematical expectations of the feature outputs of source domain data and target domain data respectively.

The entire deep transfer learning model consists of two main parts: on the one hand, the Euclidean alignment method is used in the shallow network for parameter sharing; on the other hand, the maximum mean difference (MMD) is used in the deep network to achieve domain adaptation, thereby minimizing domain shift.

Step S34: Finally, the Adam optimization algorithm is used to update the network weights.

The present invention has the following beneficial effects:

The present invention constructs an EEGNet-Attention-ResNet model, which sets EEGNet, Attention, and ResNet in series. EEGNet inherits the advantages of CNN while solving the disadvantage that the same CNN cannot process signals of different experimental paradigms, and can obtain more accurate frequency domain features; by introducing a dual-channel Attention mechanism and a ResNet residual network for feature extraction and fusion, the classification results of EEG signals finally take into full consideration the factors of the time-frequency domain and the spatial domain, making the classification results more accurate; introducing a deep transfer learning module, respectively, from the fine-tuning of the shallow network, shared parameters and the domain adaptation of the deep network to perform model adaptation, the constructed deep transfer learning model can complete good classification tasks for different subjects. Further understand the operating mechanism of the human brain, and have important practical significance in medical health and biomedical engineering.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the implementation modes of the present invention or the technical solutions in the prior art, the drawings required for describing the implementation modes or the prior art are briefly introduced below.

FIG1 is a flow chart of the classification method of MI-EEG signals of the present invention.

FIG2 is a diagram showing the architecture of the attention module of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

Referring to FIG1 , which is an overall flow chart of the implementation of the present invention, the present invention provides a method for classifying motor imagery EEG signals based on deep transfer learning, which specifically includes the following processes: data acquisition and preprocessing; a classification model trained using EEGNet, Attention mechanism, and Resnet residual network; and transfer learning of target domain data based on the metric function of Euclidean distance and maximum mean MMD.

See Figure 2 for the dual self-attention mechanism of the present invention, which consists of a channel attention module and a spatial attention module, which respectively learn important information of the channel and spatial position and suppress non-critical information, so as to improve the network's perception of key targets and improve classification accuracy.

As shown in FIG1 , a motor imagery EEG signal classification method based on deep transfer learning includes the following steps:

Step S1: Obtain a motor imagery dataset and preprocess the dataset.

Step S11: Assume that EEG data from different subjects are considered to be different domains. Given a labeled source domain represented as and the unlabeled target domain ,here n channels and m sampling points represent multi-channel EEG data. Represents the source label of the i-th sample. The deep transfer learning model established in the present invention can absorb knowledge from the source domain _Ds to enhance the more accurate prediction and classification capabilities on the target domain _Dt , effectively reducing the distribution drift between data domains.

Step S12: Use a bandpass filter to preprocess the EEG signal in the EEG source domain data set in step S11 to remove artifacts of the EEG signal and obtain preprocessed EEG data.

Step S13: In step S12, the present invention divides the preprocessed EEG data set into a training set and a test set.

Step S2: Feature extraction and classification.

Based on the EEGNet model, EEGNet inherits the advantages of CNN while solving the shortcoming that the same CNN cannot process signals of different experimental paradigms, and can obtain more accurate frequency domain features. By introducing the dual-channel Attention mechanism and ResNet residual network, the dual-channel attention mechanism includes the channel attention mechanism and the spatial attention mechanism, and the ResNet residual network, a feature extraction and classification model of EEG signals is constructed. The constructed model consists of source domain data input, feature extraction and classification output layers, and specifically includes the following steps:

Step S21: A dual-channel Attention mechanism, namely, spatial and channel attention mechanism, is added based on the EEGNet neural network layer, as shown in Figure 2. Since EEG signals contain rich channel and spatial information, extracting more useful information will greatly improve the classification accuracy.

Step S211: First, input the EEG features extracted by the EEGNet network, learn the importance of channels in the EEG signal through the ECA module, and multiply the learned channel weights by the input features to obtain channel attention features. The ECA channel attention module first uses the global average pooling layer to aggregate the convolution features to obtain the feature vector. Then the size of the kernel K is adaptively determined, and the channel weights are learned using one-dimensional convolution to complete the information interaction across channels. At the same time, this module does not reduce the local cross-channel interaction strategy of dimensionality, which effectively avoids the adverse effects.

Step S212: Then, the channel attention output feature is used as input and sent to the CBAM module to learn the importance of spatial position. The learned spatial position weight is multiplied by the channel attention feature to obtain the final result. The CBAM attention module first performs maximum pooling and average pooling operations on the feature information, and then connects the generated different feature maps and performs convolution operations. The weights are normalized by the Sigmoid function, and finally the weights are multiplied by the input feature map to obtain the final result. By learning the spatial attention weights, focusing on important feature information, suppressing repeated and non-critical information, and improving the performance and generalization ability of the entire network.

The dual-channel Attention mechanism weights are feature-adaptive, which strengthens the correlation between channels and the importance of spatial positions, suppresses non-critical information, and improves the representation ability of EEG data and the accuracy of network perception.

Step S22: Based on step S21, a ResNet residual network is added. As the number of network model layers increases, not only will the recognition rate drop significantly, the computing resource consumption increases, and the gradient disappears or explodes. By introducing the ResNet residual network, the shortcut is used to directly connect, learn the feature mapping extracted from the attention mechanism and the output residual mapping, and the complete mapping is not required, which can solve the related gradient disappearance and network degradation problems.

Step S23: extract the features of the EEG signal through the above model and classify it through the fully connected layer.

Step S24: Update network weights based on the Adam optimization algorithm.

Step S25: Based on the above steps as a cross-validation, repeat steps S21 to S24.

Step S3: Model adaptation, building a deep transfer learning model so that the classification model of the source domain EEG signal can be applied to the target domain EEG signal, specifically including the following steps:

Step S31: First, input the pre-processed channel data of the source domain and the target domain.

Step S32: In the shallow network layer, features are often universal in the field. The universal feature alignment is achieved through the Euclidean alignment method, that is, a feature alignment module is added after the second convolution filter of the EEGNet network for fusion, that is, the Euclidean alignment module, to achieve data alignment. The model trained in step S2 realizes the sharing of shallow parameters of source domain data and parameters of target domain data. The expression for executing Euclidean alignment is as follows:

, ,

represents the mean of n covariance matrices, represents the covariance function, represents a sample of EEG signal, represents the transposition of EEG signal sample data, n represents the total number of signal samples, Represents the sample representation after center alignment.

Step S33: As the network depth increases, the feature representations of different subjects become more specific, and the present invention performs more sophisticated domain adaptation on the output features of the feature extractor.

An adaptation layer is added between the output features of the source domain data and the target domain data feature extractor, and the maximum mean difference MMD measurement function is added between the adaptation layers to measure the difference in the output features of the source domain data and the target domain data. The multi-core maximum mean difference MK-MMD maps the source domain data and the target domain data to the reproducing kernel Hilbert space through multiple Gaussian kernel functions, and measures the distance between the two distributions p and q in the reproducing kernel Hilbert space. The kernel function K defined by multiple kernels is described by the formula as follows:

,

in is the weight of different Gaussian kernels. The weight of Gaussian kernels with large contributions is large, and the weight of Gaussian kernels with small contributions is small. _Ku is the uth Gaussian kernel, and it is added to the loss of the network to continue training. m represents the number of Gaussian kernels, and k represents the combination of different kernel { _ku } functions. MK-MMD can be expressed by the following formula:

,

In the formula, represents the distance of the reproducing kernel Hilbert space H _k , where , They represent the mapping of source domain data _Ds and target domain data _Dt in the reproducible kernel Hilbert space respectively, and _Ep and _Eq represent the mathematical expectations of the feature outputs of source domain data and target domain data respectively.

The entire deep transfer learning model mainly consists of two parts: the Euclidean alignment method is used for parameter sharing in the shallow network, and the maximum mean difference (MMD) is used in the deep network to achieve domain adaptation and minimize domain shift.

Step S34: Finally, the Adam optimization algorithm is used to update the network weights.

Step S35: Based on the above steps as a cross-validation, repeat steps S31 to S34.

Claims (2)

1. A motor imagery electroencephalogram signal classification method based on deep migration learning is characterized by comprising the following steps:

step S1: acquiring brain electrical signals and preprocessing brain electrical signal data;

the step S1 of collecting the brain electrical signals and preprocessing the brain electrical signal data comprises the following steps:

step S11: acquiring a motor imagery electroencephalogram data set, and dividing electroencephalogram data of different subjects into a source domain and a target domain;

Step S12: preprocessing the brain power domain data set in the step S11 by using a band-pass filter to remove artifacts of the brain electrical signals and obtain preprocessed brain electrical data;

Step S13: dividing the electroencephalogram data set preprocessed in the step S12 into a training set and a testing set;

Step S2: a EEGNet network is taken as a basic framework, a double-channel Attention mechanism, namely a space, a channel Attention model and a ResNet residual network, is added on the basis, a motor imagery electroencephalogram signal classification network is constructed, the motor imagery electroencephalogram signal classification network is used for extracting the characteristics of source domain data, classifying the characteristics through a full connection layer, updating weights through an Adam optimization algorithm, and obtaining and outputting a model of a classification result;

Step S21: on the basis of EEGNet neural network, a channel and a spatial attention mechanism, namely an ECA module and a CBAM module are added in series, so that the performance of relevant important channels and spaces for extracting electroencephalogram signals by a network model is enhanced;

Step S22: on the basis of the step S21, a ResNet residual network is added in series, and a EEGNet-Attention-ResNet model is built;

Step S23: extracting characteristics of the preprocessed source domain brain electrical data through EEGNet-Attention-ResNet model;

Step S24: classifying the electroencephalogram features extracted in the step S23 through the full connection layer;

step S25: updating the weight through an Adam optimization algorithm;

step S3: constructing a model-adaptive deep migration learning model;

step S31: constructing a shallow model adaptation module, and realizing the alignment of general characteristics of brain electrical signals of different subjects based on European alignment;

Step S32: and constructing a deep model adaptation module, measuring the difference between the output characteristics of the source domain data and the target domain data based on MMD mean value difference measurement, and realizing the field adaptation of the deep characteristics of different subjects.

2. The motor imagery electroencephalogram classification method based on deep transfer learning according to claim 1, wherein the construction model-adapted deep transfer learning model in step S3 is implemented specifically according to the following steps:

step S31: the EEGNet-Attention-ResNet model is used for constructing a shallow model adaptation module, realizing characteristic alignment of general characteristics of electroencephalogram signals of different subjects based on European alignment, performing model adaptation, constructing a deep migration learning model and performing parameter sharing of a general characteristic layer;

Based on the European alignment module, the European alignment module is added after the time and space convolution filter, the shallow parameters of the electroencephalogram signal classification model trained in the step S23 in the claim 1 are shared, and the European alignment expression is executed as follows:

，，

representing the mean of the n covariance matrices, The covariance function is represented by a function of the covariance,A segment of an EEG signal sample is represented,Representing a transpose of the EEG signal sample data, n representing the total number of signal samples,A sample representation after center alignment is performed;

step S32: the EEGNet-Attention-ResNet model is used for constructing a deep model adaptation module, measuring the difference between the output characteristics of the source domain data and the target domain data based on MMD mean value difference measurement, and realizing the field adaptation of the deep characteristics of different subjects;

Based on the maximum mean difference MMD metric function, after the feature extractor is set up in step S23 in claim 1, adding a domain adaptive module to measure the difference between the output features of the source domain and the target domain data, where the maximum mean difference MMD maps the electroencephalogram data to the regenerated kernel hilbert space through a plurality of gaussian kernel functions, measures the distribution distance of two data feature outputs p and q in the regenerated kernel hilbert space, where the p and q distributions represent the feature outputs of the source domain data and the target domain data, and the kernel function K expression defined by the plurality of kernels is as follows:

，

Wherein the method comprises the steps of For the weights contributed by different Gaussian kernels, the weights of the contributing large Gaussian kernels are large, the weights of the contributing small Gaussian kernels are small, k _u is the u-th Gaussian kernel, and the u-th Gaussian kernel is added into the loss of the network to continue training, m represents the number of Gaussian kernels, k represents the combination of functions of different kernels { k _u }, and MK-MMD is expressed by the following formula:

，

in the method, in the process of the invention, Representing the distance of the regenerated kernel Hilbert space H _k, where，Mapping of the characteristic outputs of the source domain data D _s and the target domain data D _t in the renewable kernel hilbert space is represented, and E _p、E_q represents the mathematical expectations of the characteristic outputs of the source domain data and the target domain data, respectively.