CN113010013A - Wasserstein distance-based motor imagery electroencephalogram migration learning method - Google Patents
Wasserstein distance-based motor imagery electroencephalogram migration learning method Download PDFInfo
- Publication number
- CN113010013A CN113010013A CN202110263235.XA CN202110263235A CN113010013A CN 113010013 A CN113010013 A CN 113010013A CN 202110263235 A CN202110263235 A CN 202110263235A CN 113010013 A CN113010013 A CN 113010013A
- Authority
- CN
- China
- Prior art keywords
- data
- motor imagery
- domain data
- domain
- target domain
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F18/2155—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/088—Non-supervised learning, e.g. competitive learning
Abstract
The invention discloses a motor imagery electroencephalogram transfer learning method based on Wasserstein distance, which comprises the following steps: 1) adopting motor imagery electroencephalogram signals as a data set; 2) the collected data set is subjected to preprocessing operation and is divided into a source domain and a target domain according to different subjects and labeled or unlabeled data, and the migration learning aims at improving the classification accuracy of unlabeled target domain data by using the labeled source domain data; 3) designing a deep migration learning model, pre-training source domain data by using the deep migration learning model, and using the deep migration learning model obtained by training for the next stage of migration learning; 4) carrying out countermeasure training on a specific source domain and a specific target domain by using Wasserstein distance to obtain a deep migration learning model which can be used for target domain classification; 5) and calculating the classification accuracy and kappa coefficient of the deep migration learning model on the target domain. The method effectively utilizes the existing data, has strong generalization capability and has good migration effect.
Description
Technical Field
The invention relates to the technical field of electroencephalogram signal detection, in particular to a motor imagery electroencephalogram migration learning method based on Wasserstein distance.
Background
A brain-computer interface (BCI) is a direct connection established between the brain of a human or animal and an external device or a computer to realize information exchange between the brain and the device. Research on brain-computer interfaces has continued for over 40 years. The brain-computer interface for sports has experienced rapid development since the mid 90 s of the 20 th century. At present, a brain-computer interface as a novel human-computer interaction mode is gradually becoming a hot topic of brain science research, and has great application prospects in the fields of rehabilitation training, high-risk operation, psychological cognition and the like.
The brain-computer interface utilizes neurophysiological signals originating from the brain to control an external device or computer without any actual action. In short, the brain-computer interface can extract the electroencephalogram signal of a user, and the electroencephalogram signal is converted into an output control signal through technologies such as specific signal processing and mode recognition, so that a specific computer system executes corresponding operation. According to the arrangement mode of the detection signal sensor, the brain-computer interface is divided into an invasive brain-computer interface with an invasive implanted electrode and a non-invasive brain-computer interface with a non-invasive scalp electrode. In a non-invasive brain-computer interface, eeg (electroencephalography) brain electrical signals can be recorded relatively easily and using equipment is relatively inexpensive. Thus, studies based on EEG brain electrical signals are gaining wide attention.
The EEG signal generated by the subject performing the motor imagery task belongs to a spontaneous EEG signal, and is mainly closely related to the mu rhythm and the beta rhythm of the sensory motor cortex. The real movement or hypothetical movement process of the limbs of the subject is accompanied by the change in the energy of the brain electricity caused by the ERD/ERS phenomenon of mu rhythm and beta rhythm in the cerebral cortex. Specifically, when a unilateral limb is subjected to motor imagery or real movement, the sensory motor cortex of the contralateral cerebral hemisphere is activated, the amplitude and frequency of alpha waves in the area are reduced, the energy is obviously reduced, and namely mu rhythm is inhibited. In response, the ERS phenomenon is embodied as: the sensory motor cortex of the ipsilateral cerebral hemisphere is not activated and the alpha wave in this region becomes active with a significant increase in energy, i.e. mu rhythm is enhanced. Theoretically, ERD/ERS energy changes based on mu and beta rhythms can be used as electroencephalogram features to distinguish different motor imagery tasks.
However, due to differences in physiological structures and physiological conditions of subjects, the characteristic distribution of the brain electrical signals may have significant differences. EEG data acquired by different subjects can be regarded as a data set originating from different distributions, and migratory learning of EEG data between two different subjects is a challenging problem. Moreover, because the acquisition of a large number of EEG data labels in the real-world situation is difficult, the existing large number of labeled EEG data are used for improving the classification accuracy of the unlabeled EEG data of a new subject, and the method has high research value. The problem of using labeled data from one subject to improve the classification accuracy of unlabeled data from another subject, namely unsupervised migratory learning, is that the labeled subject data set is the source domain data set and the unlabeled subject data set is the target domain data set.
The method is characterized in that the anti-migration learning is a form of unsupervised deep migration learning, the anti-migration learning is influenced by an anti-generation network, and an anti-migration learning network model is formed by using a feature extractor, a classifier and a domain discriminator, wherein the feature extractor and the classifier form a classification detection part, and the feature extractor and the domain discriminator form a domain discrimination part. The optimization targets of the feature extractor and the domain discriminator are opposite, the domain discriminator tries to judge that the example is from a source domain or a target domain, the feature extractor tries to enable the domain discriminator not to judge the source of the example, and the classifier can accurately classify the features of the source domain and the target domain simultaneously by enabling the final source domain and the target domain to be close to the feature distribution of an output intermediate layer of the feature extractor through countermeasures on the optimization targets of the antibody and the antibody.
The Wasserstein distance is a distance for measuring the difference between two different distributions, has the advantage of smoothness compared with the distance between two different distributions of a weighing quantity such as KL divergence and JS divergence, and can provide a more meaningful gradient in deep learning training. The distance of two different distributions is measured, the two distributions can be subjected to counterlearning through countertransfer learning, the difference of characteristic output of the middle layer of the characteristic extractor is reduced, and the classification effect of the model on the target domain is improved. At present, the countermeasure migration learning is mostly applied to the image field, the countermeasure migration learning is used on the motor imagery electroencephalogram, an end-to-end depth model is applied to the motor imagery electroencephalogram field, the method is suitable for the non-label motor imagery data classification scene, the existing large amount of labeled motor imagery data is better utilized, the extraction of the classification information of cross-subjects is facilitated, and the method is a feasible motor imagery electroencephalogram migration learning method.
Disclosure of Invention
The invention aims to overcome the defects and shortcomings of the prior art, and provides a Wasserstein distance-based motor imagery electroencephalogram migration learning method which can apply different subjects with labeled data, flexibly select a source domain, and has high classification accuracy and strong generalization capability.
In order to achieve the purpose, the technical scheme provided by the invention is as follows: a motor imagery electroencephalogram migration learning method based on Wasserstein distance comprises the following steps:
1) electroencephalogram data acquisition
Guiding the subject to carry out motor imagery experiments by adopting a visual reminding method, and extracting motor imagery electroencephalogram signals of the subject as a data set;
2) data pre-processing
Preprocessing the data set acquired in the step 1) to be used as a data set of a deep migration learning model;
3) pre-training motor imagery electroencephalogram signals using depth migration learning model
The method comprises the steps that transfer learning applied to motor imagery electroencephalogram signals is unsupervised transfer learning, used data are divided into source domain data and target domain data, the source domain data and the target domain data refer to motor imagery electroencephalogram signal data of two different subjects, a deep transfer learning model is used for pre-training on the source domain data, and the deep transfer learning model obtained through training is used for the next stage of the transfer learning;
4) training deep migration learning model using Wasserstein distance
Obtaining intermediate layer output characteristics of source domain data and target domain data by using a deep migration learning model pre-trained by source domain data, and finding out a source domain suitable for a specific target domain by calculating and comparing Wasserstein distances of the source domain characteristics and the target domain characteristics; carrying out countermeasure training on specific source domain data and target domain data by using Wasserstein distance to obtain a deep migration learning model which can be used for target domain data classification;
5) calculating the classification accuracy and kappa coefficient of the deep migration learning model on the target domain data
And the output result of the deep migration learning model on the target domain data is the probability of the class to which the target domain data belongs, the class with the highest probability is taken as the classification result of the target domain data, and the classification accuracy and the kappa coefficient of the target domain data are further calculated.
In the step 1), a BCI system is adopted in the motor imagery experiment, 22 lead electrode channels are provided, and the sampling frequency is 250 Hz; the specific process of the motor imagery experiment is as follows: the subject watches a given screen for 0-2s, and a symbol is displayed on the screen to remind the subject to prepare for starting the test; 2-3.25s, an indication arrow appears on the screen, and the direction is up, down, left or right; 3-6s, the subject performing either tongue, or both feet, or left hand, or right hand motor imagery tasks as instructed; 6-7.5s, the indicator arrow on the screen disappears, and the subject relaxes to rest; the motor imagery electroencephalogram signals of the single motor imagery task are used for transfer learning as a single example, and are classified into four types which respectively represent motor imagery of a subject on four body parts, namely a tongue, two feet, a left hand, a right hand and a right hand.
In step 2), the data preprocessing comprises the following steps:
2.1) intercepting the motor imagery electroencephalogram signal of 2.5-6s of a single motor imagery task as input, namely a single instance NelecA plurality of channels, each channel NtSampling points;
2.2) using 3-order low-pass filtering of 0-38Hz for each channel of the input signal to extract frequency characteristics suitable for classification, and then using moving average denoising filtering for each channel of the input signal.
In step 3), for unsupervised transfer learning, deep transfer learning model and pre-training, the following explanations exist:
unsupervised migratory learning data Source is denoted XsAnd XtWherein X issAs a source domain, XtThe target domain is obtained, wherein the source domain data has a classification label, and the target domain data has no classification label; the task of unsupervised transfer learning is to utilize the source domain data to improve the classification effect of the model on the target domain data;
the deep migration learning model comprises a feature extractor, a classifier and a domain discriminator, wherein the feature extractor takes motor imagery electroencephalogram signals as input and extracts high-dimensional abstract data features, namely high-dimensional data features; the classifier takes the high-dimensional data features output by the feature extractor as input to realize classification of the motor imagery electroencephalogram signals; the domain discriminator takes the high-dimensional data features output by the feature extractor as input to realize the classification of the motor imagery electroencephalogram signals from a source domain or a target domain;
the pre-training is to use the source domain data to perform supervised training on a feature extractor and a classifier of the deep migration learning model to obtain the deep migration learning model capable of effectively classifying the source domain data, and prepare for the next migration learning of the target domain data.
Further, the feature extractor, the classifier and the domain discriminator applied to the depth migration learning model of the motor imagery electroencephalogram signal are explained as follows:
feature extractor for filtered motor imagery electroencephalogram signals
As input, the representation input is two dimensions, where NelecIs the number of channels, NtThe number of sampling points; the feature extractor extracts time domain and space domain features by using a layer of time domain convolution and a layer of space domain convolution, and extracts higher layer features by using a convolution module to obtain fixed-dimension featuresOutput characteristic of output NfTensor of x 1, NfThe value is a reasonable value for design;
the classifier is a three-layer full-connection layer, the middle layer feature output of the feature extractor is used as input, the output is different classification probabilities, and the classification with the maximum probability is taken as the classification of model prediction;
the domain discriminator is a three-layer full-connection layer, takes the middle layer feature output of the feature extractor as input, and outputs the classification judgment of whether the data comes from a source domain or a target domain; wherein N is usedcPersonal area discriminator, NcIs the motor imagery category total.
In step 4), the Wasserstein distance training deep migration learning model is used, and the method comprises the following steps:
4.1) inputting the source domain data and the target domain data into a feature extractor of a deep migration learning model to obtain intermediate layer feature output; the deep migration learning model comprises a feature extractor, a classifier and a domain discriminator;
4.2) the classifier takes the middle layer characteristic output of the characteristic extractor as input; for the classification output of the classifier, a back propagation algorithm is used as a training method of the deep migration learning model, and the classifier and the feature extractor are trained simultaneously, so that the deep migration learning model can extract classification information of the source domain data;
4.3) computing Wasserstein distance using the mid-level feature output of the source domain data and the target domain data, the Wasserstein distance being computed as follows:
W(μ,ν)＝minΓ∈∑(μ,ν)<C,Γ>
the above formula is a discrete representation form of Wasserstein distance, wherein mu and v are two distributed probability vectors, W (mu, v) is the Wasserstein distance between mu and v, and the matrix
Is a cost matrix in which
Representing a matrix of size m x n with elements that are non-negative, cijThe distance between the ith supporting point representing mu and the jth supporting point representing nu, wherein Γ is the optimal solution for minimizing the Wasserstein distance, and the sign<·,·>Representing the dot product between the two matrices,
wherein gamma 1nRepresentative matrices Γ and 1nMultiplication by 1nA vector representing n dimensions having all values of 1;
the method comprises the steps that an IPOT method is used for calculating a sparse gamma matrix, wherein the IPOT method is a method capable of calculating an accurate Wasserstein distance, source domain data corresponding to each target domain data are obtained according to the calculated sparse gamma matrix, pseudo labels are configured for the target domain data according to labels of the source domain data corresponding to the target domain data, and then the source domain data and the target domain data of each category can be separated;
4.4) outputting the source domain data and the target domain data of each category through a middle feature layer of the feature extractor to serve as the input of the domain discriminator of each category to obtain the output of the domain discriminator of each category; the output result is trained in a mode of countermeasure training, a feature extractor and a domain discriminator are regarded as a whole, the domain discriminator needs to judge whether the data is from a source domain or a target domain, the feature extractor needs to confuse the judgment of the domain discriminator, so that the domain discriminator cannot correctly judge whether the data is from the source domain or the target domain, and finally the feature extractor can extract middle layer feature output with consistent distribution, so that the middle layer feature output of the source domain and the middle layer feature output of the target domain have the same distribution; when the inverse gradient propagation is carried out, the domain discriminator uses gradient descent training, and the feature extractor uses gradient ascent training.
In step 5), calculating the classification accuracy and kappa coefficient of the deep migration learning model to the target domain data, specifically as follows:
calculating kappa coefficient:
in the formula, P0To classify accuratelyRate, PeIs an index measuring the imbalance of the classification, PeExpressed as:
in the formula, aiRepresenting the number of i-th real samples, bjRepresenting the number of samples predicted from the jth class, and n representing the total number of classes;
the kappa coefficient is an index for measuring classification accuracy, and the higher the value of the kappa coefficient, the higher the classification accuracy achieved by the representative model.
Compared with the prior art, the invention has the following advantages and beneficial effects:
1. the model adopted by the invention realizes transfer learning by using an end-to-end deep neural network model, namely a deep transfer learning model, and omits the characteristic extraction step of the traditional method.
2. The invention uses accurate Wasserstein distance to measure the distribution difference between a source domain and a target domain in the problem of unsupervised transfer learning, provides a method for screening the source domain of a specific target domain by using the Wasserstein distance as a criterion, and is a rapid, reliable and accurate source domain screening method.
3. The invention uses a multi-domain discriminator to better fit the edge distribution compared with a single-domain discriminator.
4. The invention combines the Wasserstein distance and the multi-domain discriminator for the first time, proposes a method for distributing a pseudo label to target domain data by using the Wasserstein distance for the first time, and combines the multi-domain discriminator to carry out domain adaptation, thereby improving the domain adaptation capability of the distribution of a source domain and a target domain and ensuring that the effect of transfer learning is more obvious.
Drawings
Fig. 1 is a schematic diagram of the working process of motor imagery electroencephalogram transfer learning.
Fig. 2 is a diagram of a deep migration learning model structure according to the present invention.
FIG. 3 is a schematic diagram of the distribution of the middle layer feature output before and after the anti-migration learning according to the present invention.
Detailed Description
The present invention will be described in further detail with reference to examples and drawings, but the present invention is not limited thereto.
Referring to fig. 1, the motor imagery electroencephalogram migration learning method based on Wasserstein distance provided in this embodiment includes the following steps:
1) electroencephalogram data acquisition
Guiding the subject to carry out motor imagery experiments by adopting a visual reminding method, and extracting motor imagery electroencephalogram signals of the subject as a data set; the BCI system is adopted in the motor imagery experiment, 22 EEG lead electrode channels are adopted, and the sampling frequency is 250 Hz. The specific process of the motor imagery experiment is as follows: the subject watches a given screen for 0-2s, and a symbol is displayed on the screen to remind the subject to prepare for starting the test; 2-3.25s, an indication arrow appears on the screen, and the direction is up, down, left or right; 3-6s, the subject performing either tongue, or both feet, or left hand, or right hand motor imagery tasks as instructed; 6-7.5s, the indicator arrow on the screen disappears and the subject relaxes to rest. The motor imagery electroencephalogram signals of the single motor imagery task are used for transfer learning as a single example, and are classified into four types which respectively represent motor imagery of a subject on four body parts, namely a tongue, two feet, a left hand, a right hand and a right hand.
2) Data pre-processing
Intercepting 2.5-6s motor imagery electroencephalogram signals of a single motor imagery task as input, namely 22 channels of a single instance, and 876 sampling points of each channel, so that the format of single sample data is 22x 876;
and 3-order low-pass filtering of 0-38Hz is used for each channel of the input signal, frequency characteristics suitable for classification are extracted, and moving average denoising filtering is used for each channel of the input signal.
3) Pre-training motor imagery electroencephalogram signals using depth migration learning model
The transfer learning applied to the motor imagery electroencephalogram signals is unsupervised transfer learning, the used data are divided into source domain data and target domain data, the source domain data and the target domain data refer to the motor imagery electroencephalogram signal data of two different subjects, a deep transfer learning model is used for pre-training on the source domain data, and the deep transfer learning model obtained through training is used for the next stage of the transfer learning.
Unsupervised migratory learning data Source is denoted XsAnd XtWherein X issAs a source domain, XtThe target domain is obtained, wherein the source domain data has a classification label, and the target domain data has no classification label; the task of unsupervised transfer learning is to utilize the source domain data to improve the classification effect of the model on the target domain data;
the deep migration learning model comprises a feature extractor, a classifier and a domain discriminator, wherein the feature extractor takes a motor imagery electroencephalogram signal as input and extracts high-dimensional abstract data features, namely high-dimensional data features; the classifier takes the high-dimensional data features output by the feature extractor as input to realize classification of the motor imagery electroencephalogram signals; the domain discriminator takes the high-dimensional data features output by the feature extractor as input to realize the classification of the motor imagery electroencephalogram signals from a source domain or a target domain, and the whole model is as shown in figure 2;
the feature extractor takes the filtered motor imagery electroencephalogram signal as input, converts the number of input channels into a fixed value by using a layer of standard convolution, extracts time domain and space domain features by performing a layer of time domain convolution and a layer of space domain convolution, and extracts features of a higher layer by two dense blocks, wherein the dense blocks are convolution modules which are connected in pairs of convolution layers in the modules to obtain output features of fixed dimensions;
the classifier is a three-layer full-connection layer, the middle layer feature output of the feature extractor is used as input, the output is different classification probabilities, and the classification with the maximum probability is taken as the classification of model prediction;
the domain discriminator is a three-layer fully-connected layer, takes the middle-layer feature output of the feature extractor as input, and outputs the input as the classification judgment of whether the data comes from a source domain or a target domain. The method uses a plurality of domain discriminators, the number of which is the total number of motor imagery categories, namely four domain discriminators.
The pre-training is to use the source domain data to perform supervised training on a feature extractor and a classifier of the deep migration learning model to obtain the deep migration learning model capable of effectively classifying the source domain data, so as to prepare for the next migration learning of the target domain data.
4) Training deep migration learning model using Wasserstein distance
And inputting the source domain data and the target domain data into a feature extractor of the deep migration learning model to obtain intermediate layer feature output.
The classifier takes the middle layer characteristic output of the characteristic extractor as input; and (3) for the classification output of the classifier, using a back propagation algorithm as a training method of the deep migration learning model, and simultaneously training the classifier and the feature extractor to enable the deep migration learning model to extract classification information of the source domain data.
The Wasserstein distance is calculated using the mid-level feature outputs of the source domain data and the target domain data, and is calculated as follows:
W(μ,ν)＝minΓ∈∑(μ,ν)<C,Γ>
the above formula is a discrete representation form of Wasserstein distance, wherein mu and v are two distributed probability vectors, W (mu, v) is the Wasserstein distance between mu and v, and the matrix
Is a cost matrix in which
Representing a matrix of size m x n with elements that are non-negative, cijThe distance between the ith supporting point representing mu and the jth supporting point representing nu, wherein Γ is the optimal solution for minimizing the Wasserstein distance, and the sign<·,·>Represents the dot product between two matrices, wherein
Wherein gamma 1nRepresentative matrices Γ and 1nMultiplication by 1nA vector representing all values of 1 in the n dimensions.
And (2) calculating a sparse gamma matrix by using an IPOT method, wherein the IPOT method is a method capable of calculating an accurate Wasserstein distance, obtaining source domain data corresponding to each target domain data according to the calculated sparse gamma matrix, configuring a pseudo label for the target domain data according to the label of the source domain data corresponding to the target domain data, and then separating the source domain data and the target domain data of each category.
And outputting the source domain data and the target domain data of the four categories through the intermediate feature layer of the feature extractor as the input of the domain discriminators of the four categories to obtain the output of the domain discriminators of the four categories. The output result is trained in a mode of countertraining, the feature extractor and the domain discriminator are regarded as a whole, the domain discriminator needs to judge whether the data is from a source domain or a target domain, the feature extractor needs to confuse the judgment of the domain discriminator, so that the domain discriminator cannot correctly judge whether the data is from the source domain or the target domain, and finally the feature extractor can extract middle-layer feature output with consistent distribution, so that the middle-layer feature output of the source domain and the middle-layer feature output of the target domain have the same distribution. When the inverse gradient propagation is carried out, the domain discriminator uses gradient descent training, and the feature extractor uses gradient ascent training.
5) Calculating the classification accuracy and kappa coefficient of the deep migration learning model in the target domain
And the output result of the deep migration learning model on the target domain data is the probability of the class to which the target domain data belongs, the class with the highest probability is taken as the classification result of the motor imagery electroencephalogram signal, and the classification accuracy and kappa coefficient of all data of the target domain are further calculated.
Calculating kappa coefficient:
wherein, P0To classify accuracy, PeIs an index measuring the imbalance of the classification, PeExpressed as:
in the formula, aiRepresenting the number of i-th real samples, bjRepresents the number of samples predicted by the jth class, and n represents the total number of classes.
The Kappa coefficient is an index for measuring classification accuracy, and the higher the value of the Kappa coefficient, the higher the classification accuracy achieved by the representative model.
The ideal target effect achieved before and after the adoption of the depth antagonistic migration learning based on the Wasserstein distance is shown in FIG. 3, wherein D1And D2In two different distributions, where S1And S2In two different classes, the curve is D1Middle S1And S2Classification boundary of (D), classification boundary before transfer learning2S in (1)1And S2In distinction, after transfer learning, D1Can also be applied to D2。
In conclusion, the invention provides an effective method for the field of motor imagery electroencephalogram detection transfer learning, does not need manual feature extraction, effectively utilizes the existing data, has obvious transfer learning effect and strong generalization capability, has actual popularization value and is worthy of popularization.
The above embodiments are preferred embodiments of the present invention, but the present invention is not limited to the above embodiments, and any other changes, modifications, substitutions, combinations, and simplifications which do not depart from the spirit and principle of the present invention should be construed as equivalents thereof, and all such changes, modifications, substitutions, combinations, and simplifications are intended to be included in the scope of the present invention.
Claims (7)
1. A motor imagery electroencephalogram migration learning method based on Wasserstein distance is characterized by comprising the following steps:
1) electroencephalogram data acquisition
Guiding the subject to carry out motor imagery experiments by adopting a visual reminding method, and extracting motor imagery electroencephalogram signals of the subject as a data set;
2) data pre-processing
Preprocessing the data set acquired in the step 1) to be used as a data set of a deep migration learning model;
3) pre-training motor imagery electroencephalogram signals using depth migration learning model
The method comprises the steps that transfer learning applied to motor imagery electroencephalogram signals is unsupervised transfer learning, used data are divided into source domain data and target domain data, the source domain data and the target domain data refer to motor imagery electroencephalogram signal data of two different subjects, a deep transfer learning model is used for pre-training on the source domain data, and the deep transfer learning model obtained through training is used for the next stage of the transfer learning;
4) training deep migration learning model using Wasserstein distance
Obtaining intermediate layer output characteristics of source domain data and target domain data by using a deep migration learning model pre-trained by source domain data, and finding out a source domain suitable for a specific target domain by calculating and comparing Wasserstein distances of the source domain characteristics and the target domain characteristics; carrying out countermeasure training on specific source domain data and target domain data by using Wasserstein distance to obtain a deep migration learning model which can be used for target domain data classification;
5) calculating the classification accuracy and kappa coefficient of the deep migration learning model on the target domain data
And the output result of the deep migration learning model on the target domain data is the probability of the class to which the target domain data belongs, the class with the highest probability is taken as the classification result of the target domain data, and the classification accuracy and the kappa coefficient of the target domain data are further calculated.
2. The Wasserstein distance-based motor imagery electroencephalogram migration learning method according to claim 1, wherein: in the step 1), a BCI system is adopted in the motor imagery experiment, 22 lead electrode channels are provided, and the sampling frequency is 250 Hz; the specific process of the motor imagery experiment is as follows: the subject watches a given screen for 0-2s, and a symbol is displayed on the screen to remind the subject to prepare for starting the test; 2-3.25s, an indication arrow appears on the screen, and the direction is up, down, left or right; 3-6s, the subject performing either tongue, or both feet, or left hand, or right hand motor imagery tasks as instructed; 6-7.5s, the indicator arrow on the screen disappears, and the subject relaxes to rest; the motor imagery electroencephalogram signals of the single motor imagery task are used for transfer learning as a single example, and are classified into four types which respectively represent motor imagery of a subject on four body parts, namely a tongue, two feet, a left hand, a right hand and a right hand.
3. The Wasserstein distance-based motor imagery electroencephalogram migration learning method according to claim 1, wherein: in step 2), the data preprocessing comprises the following steps:
2.1) intercepting the motor imagery electroencephalogram signal of 2.5-6s of a single motor imagery task as input, namely a single instance NelecA plurality of channels, each channel NtSampling points;
2.2) using 3-order low-pass filtering of 0-38Hz for each channel of the input signal to extract frequency characteristics suitable for classification, and then using moving average denoising filtering for each channel of the input signal.
4. The Wasserstein distance-based motor imagery electroencephalogram migration learning method according to claim 1, wherein: in step 3), for unsupervised transfer learning, deep transfer learning model and pre-training, the following explanations exist:
unsupervised migratory learning data Source is denoted XsAnd XtWherein X issAs a source domain, XtThe target domain is obtained, wherein the source domain data has a classification label, and the target domain data has no classification label; the task of unsupervised transfer learning is to utilize the source domain data to improve the classification effect of the model on the target domain data;
the deep migration learning model comprises a feature extractor, a classifier and a domain discriminator, wherein the feature extractor takes motor imagery electroencephalogram signals as input and extracts high-dimensional abstract data features, namely high-dimensional data features; the classifier takes the high-dimensional data features output by the feature extractor as input to realize classification of the motor imagery electroencephalogram signals; the domain discriminator takes the high-dimensional data features output by the feature extractor as input to realize the classification of the motor imagery electroencephalogram signals from a source domain or a target domain;
the pre-training is to use the source domain data to perform supervised training on a feature extractor and a classifier of the deep migration learning model to obtain the deep migration learning model capable of effectively classifying the source domain data, and prepare for the next migration learning of the target domain data.
5. The Walserstein distance-based motor imagery electroencephalogram migration learning method according to claim 4, wherein: the feature extractor, the classifier and the domain discriminator applied to the depth migration learning model of the motor imagery electroencephalogram signal are explained as follows:
feature extractor for filtered motor imagery electroencephalogram signals
As input, the representation input is two dimensions, where NelecIs the number of channels, NtThe number of sampling points; the feature extractor extracts time domain and space domain features by using a layer of time domain convolution and a layer of space domain convolution, and extracts higher layer features by using a convolution module to obtain output features with fixed dimensionality, wherein the output is NfTensor of x 1, NfThe value is a reasonable value for design;
the classifier is a three-layer full-connection layer, the middle layer feature output of the feature extractor is used as input, the output is different classification probabilities, and the classification with the maximum probability is taken as the classification of model prediction;
the domain discriminator is a three-layer full-connection layer, takes the middle layer feature output of the feature extractor as input, and outputs the classification judgment of whether the data comes from a source domain or a target domain; wherein N is usedcPersonal area discriminator, NcIs the motor imagery category total.
6. The Wasserstein distance-based motor imagery electroencephalogram migration learning method according to claim 1, wherein: in step 4), the Wasserstein distance training deep migration learning model is used, and the method comprises the following steps:
4.1) inputting the source domain data and the target domain data into a feature extractor of a deep migration learning model to obtain intermediate layer feature output; the deep migration learning model comprises a feature extractor, a classifier and a domain discriminator;
4.2) the classifier takes the middle layer characteristic output of the characteristic extractor as input; for the classification output of the classifier, a back propagation algorithm is used as a training method of the deep migration learning model, and the classifier and the feature extractor are trained simultaneously, so that the deep migration learning model can extract classification information of the source domain data;
4.3) computing Wasserstein distance using the mid-level feature output of the source domain data and the target domain data, the Wasserstein distance being computed as follows:
W(μ,ν)＝minΓ∈∑(μ,ν)<C,Γ>
the above formula is a discrete representation form of Wasserstein distance, wherein mu and v are two distributed probability vectors, W (mu, v) is the Wasserstein distance between mu and v, and the matrix
Is a cost matrix in which
Representing a matrix of size m x n with elements that are non-negative, cijThe distance between the ith supporting point representing mu and the jth supporting point representing nu, wherein Γ is the optimal solution for minimizing the Wasserstein distance, and the sign<·,·>Representing the dot product between the two matrices,
wherein gamma 1nRepresentative matrices Γ and 1nMultiplication by 1nA vector representing n dimensions having all values of 1;
the method comprises the steps that an IPOT method is used for calculating a sparse gamma matrix, wherein the IPOT method is a method capable of calculating an accurate Wasserstein distance, source domain data corresponding to each target domain data are obtained according to the calculated sparse gamma matrix, pseudo labels are configured for the target domain data according to labels of the source domain data corresponding to the target domain data, and then the source domain data and the target domain data of each category can be separated;
4.4) outputting the source domain data and the target domain data of each category through a middle feature layer of the feature extractor to serve as the input of the domain discriminator of each category to obtain the output of the domain discriminator of each category; the output result is trained in a mode of countermeasure training, a feature extractor and a domain discriminator are regarded as a whole, the domain discriminator needs to judge whether the data is from a source domain or a target domain, the feature extractor needs to confuse the judgment of the domain discriminator, so that the domain discriminator cannot correctly judge whether the data is from the source domain or the target domain, and finally the feature extractor can extract middle layer feature output with consistent distribution, so that the middle layer feature output of the source domain and the middle layer feature output of the target domain have the same distribution; when the inverse gradient propagation is carried out, the domain discriminator uses gradient descent training, and the feature extractor uses gradient ascent training.
7. The Wasserstein distance-based motor imagery electroencephalogram migration learning method according to claim 1, wherein: in step 5), calculating the classification accuracy and kappa coefficient of the deep migration learning model to the target domain data, specifically as follows:
calculating kappa coefficient:
in the formula, P0To classify accuracy, PeIs an index measuring the imbalance of the classification, PeExpressed as:
in the formula, aiRepresenting the number of i-th real samples, bjRepresenting the number of samples predicted from the jth class, and n representing the total number of classes;
the kappa coefficient is an index for measuring classification accuracy, and the higher the value of the kappa coefficient, the higher the classification accuracy achieved by the representative model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110263235.XA CN113010013A (en) | 2021-03-11 | 2021-03-11 | Wasserstein distance-based motor imagery electroencephalogram migration learning method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110263235.XA CN113010013A (en) | 2021-03-11 | 2021-03-11 | Wasserstein distance-based motor imagery electroencephalogram migration learning method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113010013A true CN113010013A (en) | 2021-06-22 |
Family
ID=76404752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110263235.XA Pending CN113010013A (en) | 2021-03-11 | 2021-03-11 | Wasserstein distance-based motor imagery electroencephalogram migration learning method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113010013A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113486827A (en) * | 2021-07-13 | 2021-10-08 | 上海中科辰新卫星技术有限公司 | Multi-source remote sensing image transfer learning method based on domain confrontation and self-supervision |
| CN113705464A (en) * | 2021-08-30 | 2021-11-26 | 哈尔滨工业大学 | Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface |
| CN114444374A (en) * | 2021-11-29 | 2022-05-06 | 河南工业大学 | Multi-source to multi-target domain self-adaption method based on similarity measurement |
| CN114972299A (en) * | 2022-06-16 | 2022-08-30 | 沈阳工业大学 | Railway track defect detection method based on deep migration learning |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110414383A (en) * | 2019-07-11 | 2019-11-05 | 华中科技大学 | Convolutional neural networks based on Wasserstein distance fight transfer learning method and its application |
| CN110907176A (en) * | 2019-09-30 | 2020-03-24 | 合肥工业大学 | Wasserstein distance-based fault diagnosis method for deep countermeasure migration network |
-
2021
- 2021-03-11 CN CN202110263235.XA patent/CN113010013A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110414383A (en) * | 2019-07-11 | 2019-11-05 | 华中科技大学 | Convolutional neural networks based on Wasserstein distance fight transfer learning method and its application |
| CN110907176A (en) * | 2019-09-30 | 2020-03-24 | 合肥工业大学 | Wasserstein distance-based fault diagnosis method for deep countermeasure migration network |
Non-Patent Citations (1)
| Title |
|---|
| 刘政等: "基于深度学习的头皮脑电信息解码研究进展", 《中国生物医学工程学报》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113486827A (en) * | 2021-07-13 | 2021-10-08 | 上海中科辰新卫星技术有限公司 | Multi-source remote sensing image transfer learning method based on domain confrontation and self-supervision |
| CN113486827B (en) * | 2021-07-13 | 2023-12-08 | 上海中科辰新卫星技术有限公司 | Multi-source remote sensing image migration learning method based on domain countermeasure and self supervision |
| CN113705464A (en) * | 2021-08-30 | 2021-11-26 | 哈尔滨工业大学 | Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface |
| CN113705464B (en) * | 2021-08-30 | 2024-01-09 | 哈尔滨工业大学 | Domain adaptation method for solving characteristic migration problem in motor imagery brain-computer interface |
| CN114444374A (en) * | 2021-11-29 | 2022-05-06 | 河南工业大学 | Multi-source to multi-target domain self-adaption method based on similarity measurement |
| CN114972299A (en) * | 2022-06-16 | 2022-08-30 | 沈阳工业大学 | Railway track defect detection method based on deep migration learning |
| CN114972299B (en) * | 2022-06-16 | 2024-03-26 | 沈阳工业大学 | Railway track defect detection method based on deep migration learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113627518B (en) | Method for realizing neural network brain electricity emotion recognition model by utilizing transfer learning | |
| CN107844755B (en) | Electroencephalogram characteristic extraction and classification method combining DAE and CNN | |
| CN113010013A (en) | Wasserstein distance-based motor imagery electroencephalogram migration learning method | |
| CN106886792B (en) | Electroencephalogram emotion recognition method for constructing multi-classifier fusion model based on layering mechanism | |
| CN107924472B (en) | Image classification method and system based on brain computer interface | |
| Lewicki | A review of methods for spike sorting: the detection and classification of neural action potentials | |
| CN110353702A (en) | A kind of emotion identification method and system based on shallow-layer convolutional neural networks | |
| Yuan et al. | Patients’ EEG data analysis via spectrogram image with a convolution neural network | |
| CN112244873A (en) | Electroencephalogram time-space feature learning and emotion classification method based on hybrid neural network | |
| CN110236533A (en) | Epileptic seizure prediction method based on the study of more deep neural network migration features | |
| CN109330613A (en) | Human body Emotion identification method based on real-time brain electricity | |
| CN110367976B (en) | Brain wave signal detection method, related device and storage medium | |
| Manjunath et al. | A low-power lstm processor for multi-channel brain eeg artifact detection | |
| Zhang et al. | Instance transfer subject-dependent strategy for motor imagery signal classification using deep convolutional neural networks | |
| CN112370066A (en) | Brain-computer interface method of stroke rehabilitation system based on generation of countermeasure network | |
| CN112884063B (en) | P300 signal detection and identification method based on multi-element space-time convolution neural network | |
| CN112674782B (en) | Device and method for detecting epileptic-like electrical activity of epileptic during inter-seizure period | |
| CN114983343A (en) | Sleep staging method and system, computer-readable storage medium and electronic device | |
| CN113951883B (en) | Gender difference detection method based on electroencephalogram signal emotion recognition | |
| Jiang et al. | Analytical comparison of two emotion classification models based on convolutional neural networks | |
| CN113011330B (en) | Electroencephalogram signal classification method based on multi-scale neural network and cavity convolution | |
| CN114145744A (en) | Cross-device forehead electroencephalogram emotion recognition method and system | |
| Zhao et al. | E3GCAPS: Efficient EEG-based multi-capsule framework with dynamic attention for cross-subject cognitive state detection | |
| CN116821764A (en) | Knowledge distillation-based multi-source domain adaptive EEG emotion state classification method | |
| CN116421200A (en) | Brain electricity emotion analysis method of multi-task mixed model based on parallel training |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210622 |
|
| RJ01 | Rejection of invention patent application after publication |