CN111695500A - Method and system for recognizing motor imagery task of stroke patient based on transfer learning - Google Patents
Method and system for recognizing motor imagery task of stroke patient based on transfer learning Download PDFInfo
- Publication number
- CN111695500A CN111695500A CN202010527624.4A CN202010527624A CN111695500A CN 111695500 A CN111695500 A CN 111695500A CN 202010527624 A CN202010527624 A CN 202010527624A CN 111695500 A CN111695500 A CN 111695500A
- Authority
- CN
- China
- Prior art keywords
- model
- classification
- parameters
- module
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G16—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
- G16H—HEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
- G16H50/00—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
- G16H50/20—ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Public Health (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Medical Informatics (AREA)
- Databases & Information Systems (AREA)
- Signal Processing (AREA)
- Pathology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Epidemiology (AREA)
- Primary Health Care (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention discloses a method and a system for recognizing motor imagery tasks of stroke patients based on transfer learning, belonging to the field of brain-computer interfaces and aiming at solving the technical problems that under the condition of insufficient training data, a training model is constructed by using transfer learning, electroencephalogram signal features are effectively extracted and efficient mode classification is realized, and the adopted technical scheme is as follows: the method comprises the following specific steps: collecting and preprocessing electroencephalogram signals: collecting an electroencephalogram signal containing MI of a stroke patient, and preprocessing the electroencephalogram signal; feature extraction and classification: combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals; and (3) evaluating classification performance: and inputting the test set into each model, comparing the classification accuracy of each model in the subject, and evaluating the classification performance of each model. The system comprises an acquisition and preprocessing module, a feature extraction and classification module and a classification performance evaluation module.
Description
Technical Field
The invention relates to the field of brain-computer interfaces, in particular to a method and a system for recognizing motor imagery tasks of stroke patients based on transfer learning.
Background
The cerebral apoplexy is an acute cerebrovascular disease, is a common disease seriously threatening human beings, and has the characteristics of high morbidity, high disability rate, high mortality and the like. The fatality rate of cardiovascular disease patients in China shows an increasing trend, and the burden of cardiovascular diseases is gradually increased, so that the cardiovascular disease patients become a great public health problem. At present, cardiovascular disease patients in China are 2.9 hundred million, and stroke patients are 1300 million. The stroke patients have poor daily life capability, and heavy economic burden is brought to families and society. More and more stroke patients urgently hope to receive rehabilitation therapy to regain limb motor functions. In patients with motor dysfunction, motor ability of patients can be restored by repeatedly stimulating damaged motor cortex portions by means of Motor Imagery (MI) therapy based on the BCI (Brain-Computer Interface, BCI) system to reactivate damaged peripheral motor nerve cells.
The BCI system collects electroencephalogram signals from cerebral cortex through signal collection equipment, converts the electroencephalogram signals into signals which can be identified by a computer through processes of amplification, filtering, analog/digital conversion and the like, then preprocesses the signals, extracts characteristic signals, utilizes the characteristics to identify modes, and finally converts the signals into specific instructions for controlling external equipment to realize the control of the external equipment.
Aiming at different electroencephalogram signal acquisition modes, BCI systems are classified into invasive and non-invasive. The invasive method is to place the electrode under the cerebral cortex, and the electroencephalogram signal acquired by the method has higher signal-to-noise ratio, and has the defects that the stability of the structure and the function of the electrode in the brain can not be ensured for a long time, and potential safety hazards exist when the electrode is implanted into the cerebral cortex. The brain electrical signals generated by the brain activities of a human can be directly obtained by attaching the electrodes to the scalp, and the characteristics of easy acquisition, non-invasiveness and the like make the brain electrical signals which are acquired by a non-invasive method become the main direction of the BCI technical research. The BCI system based on MI stimulates the brain motor cortex to sense the movement rhythm change through MI, so as to realize the communication and control of external equipment.
With the rapid development of deep learning, the electroencephalogram signal feature extraction and classification identification method based on the deep learning network is widely concerned. Theoretically, deep learning can realize more effective feature extraction and higher-precision pattern classification of electroencephalogram (EEG), but actually, because a stroke patient has a poor state and a complex disease condition, EEG data acquisition and labeling costs are high, and it is very difficult to construct a large-scale data set, which limits the development of a deep learning method in a brain stroke patient-based motor imagery BCI system.
Aiming at the characteristics that the electroencephalogram signal collection including motor imagery is carried out on a cerebral apoplexy patient, the characteristics that the patient insists on short time, the imagination action is not in place, the data of the EEG data including MI is scarce, the signal to noise ratio is low, the classification and identification are difficult and the like exist, the conventional methods for solving the data scarcity have the defects of traditional semi-supervised learning, collaborative training, active learning and the like, but the methods have high requirements on data quantity, so that how to construct a training model by utilizing transfer learning under the condition of insufficient training data, the electroencephalogram signal characteristics are effectively extracted, and the efficient mode classification is the technical problem to be solved urgently at present.
Disclosure of Invention
The invention provides a method and a system for recognizing a motor imagery task of a stroke patient based on transfer learning, and aims to solve the problems that how to construct a training model by using transfer learning under the condition of insufficient training data, the electroencephalogram signal characteristics are effectively extracted, and efficient mode classification is realized.
The technical task of the invention is realized according to the following mode, the method for identifying the motor imagery task of the stroke patient based on the transfer learning combines an EEGNet framework with a Finetune (fine tuning) technology in the transfer learning, realizes the classification and identification of the motor imagery task of the stroke patient, effectively extracts the characteristics of electroencephalogram signals and realizes high-efficiency mode classification; the method comprises the following specific steps:
collecting and preprocessing electroencephalogram signals: acquiring an electroencephalogram (EEG) signal (EEG signal) containing MI of a stroke patient, wherein the information of the EEG signal comprises data of a subject and a corresponding class label, and dividing the EEG signal into a test set and a training set; preprocessing the electroencephalogram signals at the same time;
feature extraction and classification: combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals;
and (3) evaluating classification performance: and inputting the test set into each model, comparing the classification accuracy of each model in the subject, and evaluating the classification performance of each model.
Preferably, the preprocessing includes down-sampling and filtering.
More preferably, the EEGNet model is specified as follows:
in the input layer and module one: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology, wherein the value of Dropout is preferably 0.25;
in the module two: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
in module three: in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
in the classification block: features are passed directly into the softmax classification with N units; where N is the number of classes in the data, the softmax function is used because EEGNet is a multi-class model.
Preferably, the finnetune technology in the transfer learning specifically includes:
modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
because the classification number of the original data classification is different from the classification of the target number, the last full-connection layer of the original network is modified according to the classification of the data, parameters of all the previous layers are frozen in a modification mode, the classification A of the softmax layer of the original network is deleted, and the parameters N needing to be classified are changed into the parameters N needing to be updated;
loading the existing parameters of the EEGNet model, and training the model according to own data;
after the Finetune succeeds, selectively opening all layers to update the small-step-length parameters; in the invention, the layers in the original structure are not removed or added, because the network structure is conducted layer by layer, and the unexpected result can not be obtained in the process of conducting in the future by changing the number of the layers;
for the fine adjustment of the learning rate of the model, the learning rate should not be set too large, because the premise of the Finetune is that the weight of the model is significant, but if the learning rate is too large, the updating is too fast, the original good weight information is destroyed, and therefore the learning rate adjustment is low;
adjusting configuration parameters of the model, generally learning rate and step length, and reducing iteration times, specifically as follows: because the selected pre-training model is a parameter with good effect obtained by repeated training, if the pre-training model is adjusted too much, the loss of characteristic information can be caused, so that the parameters of other convolutional layers are frozen when Finetune is carried out, the size of the kernel of the pooling layer is slowly changed, whether the accuracy is improved or not is observed by repeated training, and finally, when the size of the pooling layer is reduced to 1/2, the obtained accuracy effect is optimal; meanwhile, parameters of other layers are fixed, the size of the inner core of the convolution layer is modified, but the accuracy is not improved, and the accuracy is reduced, so that the classification performance effect of the model cannot be improved due to excessive adjustment of the parameters in the Finetune; the EEGNet model is a well-trained model applied to a large amount of electroencephalogram data, and has a good data processing effect, so that the Finetune can enable the model to obtain a good effect after a small number of iterations, the iterations can be reduced to 1/10, the accuracy is observed to be improved or not, the results are output in a proper iteration process, whether a stable state is presented or not is observed, if the stable state is improved, the number of iterations can be considered to be increased, and if the stable state is basically stable, the number of iterations does not need to be increased;
training is started and the parameters of the pre-trained model are loaded.
More preferably, the evaluation classification performance is specifically as follows:
inputting the training set data into a selected EEGNet model and a Convolutional Neural Network (CNN) model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
after the corresponding model is trained, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and comparing the predicted label of the test set with the real label to obtain the classification accuracy of the test.
A brain stroke patient motor imagery task identification system based on transfer learning comprises,
the acquisition and preprocessing module is used for acquiring electroencephalogram signals (EEG signals) containing MI of a stroke patient, wherein the information of the electroencephalogram signals comprises data of a subject and a corresponding class label, and the electroencephalogram signals are divided into a test set and a training set; preprocessing the electroencephalogram signals at the same time; wherein the preprocessing comprises down-sampling and filtering;
the characteristic extraction and classification module is used for combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals;
and the classification performance evaluation module is used for inputting the test set into each model, comparing the classification accuracy of each model in the subject and evaluating the classification performance of each model.
Preferably, the EEGNet model is specifically as follows:
(1) and in the input layer and the module I: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology, wherein the value of Dropout is preferably 0.25;
(2) and in the module II: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
(3) and in module three: in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
(4) in the classification block: features are passed directly into the softmax classification with N units; where N is the number of classes in the data, the softmax function is used because EEGNet is a multi-class model;
the finenetune technology in the transfer learning is specifically as follows:
modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
firstly, modifying the last full-connection layer of the original network, freezing parameters of the previous layers in a modification mode, deleting the category A of the softmax layer of the original network, and replacing the category A with the parameter N needing to be classified for updating;
secondly, loading the existing parameters of the EEGNet model, and training the model according to own data;
thirdly, after the Finetune succeeds, all layers are selected to be opened to carry out small-step parameter updating;
fourthly, fine adjustment of the learning rate of the model;
and (II) adjusting configuration parameters of the model, usually learning the rate and the step length, and reducing the iteration times, which is specifically as follows: freezing parameters of other convolutional layers during Finetune, slowly changing the size of the kernel of the pooling layer, observing whether the accuracy is improved or not through repeated training, and finally reducing the size of the pooling layer to 1/2, wherein the obtained accuracy effect is optimal;
and (III) starting training, and loading the parameters of the pre-training model.
Preferably, the working process of the classification performance evaluation module is as follows:
inputting training set data into the selected EEGNet model and CNN model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
(II) after the corresponding model is trained, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and (III) comparing the predicted test set label with a real label to obtain the classification accuracy of the test.
An electronic device, comprising: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executes the computer-executable instructions stored by the memory to cause the at least one processor to perform a method for stroke patient motor imagery task identification based on transfer learning as described above.
A computer-readable storage medium having stored thereon computer-executable instructions, which, when executed by a processor, implement the method for identifying a motor imagery task for stroke patients based on transfer learning as described above.
The method and the system for identifying the motor imagery task of the stroke patient based on the transfer learning have the following advantages:
the EEGNet model and the Finetune technology are combined and applied to the acquired electroencephalogram signals of the cerebral apoplexy patient, so that the problem of insufficient training data caused by poor states and complex illness conditions of the cerebral apoplexy patient can be effectively solved, and the cerebral apoplexy patient has better robustness and generalization in classification performance;
the deep transfer learning of the invention is different from the traditional deep learning algorithm in that the transfer learning does not need a large amount of calculation time and calculation cost to train a model, and the transfer learning can obtain better learning effect by using scarce data;
the invention (III) combines the EEGNet framework and the Finetune technology, so that the advantages of improving the algorithm speed and reducing the algorithm complexity without completely retraining the model are achieved, the Finetune can obtain a satisfactory effect after a few iterations, and the Finetune is selected to construct a deep learning algorithm framework under the condition that the EEG data volume of a cerebral apoplexy patient containing MI is scarce.
Drawings
The invention is further described below with reference to the accompanying drawings.
Fig. 1 is a flow chart of a stroke patient motor imagery task identification method based on transfer learning;
FIG. 2 is a schematic diagram of a one-time experimental feature extraction process;
FIG. 3 is a block diagram of the EEGNet;
FIG. 4 is a histogram of classification results;
fig. 5 is a block diagram of a system for recognizing motor imagery tasks of stroke patients based on transfer learning.
Detailed Description
The method and system for identifying motor imagery tasks of stroke patients based on transfer learning of the invention are described in detail below with reference to the drawings and specific embodiments of the specification.
Example 1:
as shown in fig. 1, the method for identifying motor imagery tasks of stroke patients based on transfer learning combines an EEGNet framework with a Finetune (fine tuning) technology in transfer learning to realize classification and identification of the motor imagery tasks of stroke patients, effectively extracts electroencephalogram signal characteristics and realizes efficient mode classification; the method comprises the following specific steps:
s1, acquiring and preprocessing electroencephalogram signals: acquiring an electroencephalogram (EEG) signal (EEG signal) containing MI of a stroke patient, wherein the information of the EEG signal comprises data of a subject and a corresponding class label, and dividing the EEG signal into a test set and a training set; preprocessing the electroencephalogram signals at the same time;
an MI-containing EEG acquisition experiment based on stroke patients is performed in a hospital, and data sets of stroke patients are acquired by using NeuroScan electroencephalogram experimental equipment, as shown in fig. 2, and during a BCI experiment, a subject must perform imaginary actions of grasping by the left hand, grasping by the right hand, and grasping by a wooden nail. A64-channel EEG signal acquisition device is adopted, the test time is 3 seconds each time, one type of imaginary movement is contained, and the interval between two tests is 4 seconds; in each run, the trials were assigned to 30 trials for right-hand grip, 30 trials for left-hand grip and 10 trials for gripping a dowel; to avoid the visual evoked potentials being reflected by the data, the recording interval started 1 second after the end of the visual cue and the EEG sampling frequency was 1000 Hz.
The pre-processing includes down-sampling and filtering. And reducing the sampling frequency of the data from 1000HZ to 100HZ, filtering the E electroencephalogram signals subjected to the down-sampling by using a Butterworth band-pass filter, and filtering noise and interference frequency.
S2, feature extraction and classification: combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals;
the transfer learning method adopted by the invention is a Finetune technology, and the pre-trained model can be transferred to the model thereof. The pre-trained network weights are typically used to initialize their own network weights, rather than the original random initialization.
The EEGNet model is a compact CNN architecture for EEG-based BCI that can be applied to multiple different BCI paradigms, can be trained with very limited data, and can produce neurophysiologically interpretable features. This model was used for our EEG classification test with C-channel and M-time sampling. We fit a model using Adam optimizer with default parameters to minimize the classification cross-entropy loss function, the model used was created in a pytorech environment. As shown in fig. 3, the EEGNet model is embodied as follows:
s201-01, in an input layer and a module I: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology, wherein the value of Dropout is preferably 0.25;
s201-02, in the module II: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
s201-03, in module III: in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
s201-4, in the classification block: features are passed directly into the softmax classification with N units; where N is the number of classes in the data, the softmax function is used because EEGNet is a multi-class model.
The transfer learning comprises the following steps:
domain, denoted by D ═ (X, p (X)), includes two parts: a feature space X and an edge distribution p (X), where X ═ X1,x2,...,xnBelongs to X;
a task, denoted by T ═ { Y, P (Y | X) }, includes two parts: the label space Y and a target prediction function f () can be regarded as a conditional probability P (Y | X), and the target prediction function cannot be directly and accurately observed but can be obtained through training;
mathematical definition, given a Source Domain DSSource domain learning task TSTarget domain DtAnd an objectDomain task Tt，DSIs not equal to DtAnd TSIs not equal to Tt。
Transfer learning utilizes knowledge enhancement of the source domain to improve or optimize the learning efficiency of the predictive objective function of the target domain. As is well known, the strategy of the migration learning depends on various factors, but most importantly, the similarity between the original data and the target data is smaller, and the better the effect of the adopted finenet technology is. The first few layers of the EEGNet network obtain features that are general features based on comparison (e.g., EEGNet extracts spatial filters of specific frequencies), and the last few layers extract features that are related to specific classes (e.g., models can summarize the kernels of each feature map separately and find the best combined feature map). In the invention, the database of the task is relatively small, in order to avoid overfitting, the adopted method is to reserve the first layers of the network and change the output category of the last layer, and the adopted optimal Finetune technology is to take the first layers of the EEGNet network as a feature extractor and change the last layer into the output category for classification. The electroencephalogram feature extraction principle packaged in the pre-training model is utilized, and the Finetune technology is adopted, so that the model is better in robustness and generalization in experiments. In brief, the method is a process of training by using the optimal parameters to which the initialization parameters are updated continuously and the modified network to adapt the parameters to the electroencephalogram data.
The difference from the prior art is that: generally speaking, the traditional electroencephalogram classification method, such as support vector machine, fourier transform, linear discriminant analysis, common spatial mode and other algorithms, can achieve better effects. However, depth network-based electroencephalography classification tends to defeat traditional methods on large data sets. On the other hand, the deep network must rely on a large amount of training data, but in practice, due to high data acquisition and labeling cost, it is very difficult to construct a large-scale well-annotated data set, which limits the development of deep learning on electroencephalogram signal processing. The cerebral apoplexy electroencephalogram signals are processed by utilizing transfer learning, and the extracted features have similarity and inheritance. The characteristics are not only specific to a certain data set, but also can be used in other related data sets, so that the effectiveness of transfer learning on the electroencephalogram signals by using the depth network is ensured.
In summary, the finenetune technology in the migration learning is specifically as follows:
s202-01, modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
s202-01-01, because the classification number of the original data classification is different from the classification of the target number, the last full connection layer of the original network is modified according to the classification of the data, the parameters of the previous layers are all frozen in a modification mode, the classification A of the softmax layer of the original network is deleted, and the parameter N needing to be classified is changed into the parameter N needing to be updated;
s202-01-02, loading the existing parameters of the EEGNet model, and training the model according to own data;
s202-01-03 and Finetune succeed, then all layers are selected to be opened to carry out small step parameter updating; in the invention, the layers in the original structure are not removed or added, because the network structure is conducted layer by layer, and the unexpected result can not be obtained in the process of conducting in the future by changing the number of the layers;
s202-01-04, for the fine adjustment of the learning rate of the model, the learning rate should not be set too large, because the premise of the Finetune is that the weight of the model is much meaningful, but if the learning rate is too large, the updating is too fast, the original good weight information is damaged, and therefore the adjustment of the learning rate is lower;
s202-02, adjusting configuration parameters of the model, generally learning the rate and the step length, and reducing the iteration times, wherein the method specifically comprises the following steps: because the selected pre-training model is a parameter with good effect obtained by repeated training, if the pre-training model is adjusted too much, the loss of characteristic information can be caused, so that the parameters of other convolutional layers are frozen when Finetune is carried out, the size of the kernel of the pooling layer is slowly changed, whether the accuracy is improved or not is observed by repeated training, and finally, when the size of the pooling layer is reduced to 1/2, the obtained accuracy effect is optimal; meanwhile, parameters of other layers are fixed, the size of the inner core of the convolution layer is modified, but the accuracy is not improved, and the accuracy is reduced, so that the classification performance effect of the model cannot be improved due to excessive adjustment of the parameters in the Finetune; the EEGNet model is a well-trained model applied to a large amount of electroencephalogram data, and has a good data processing effect, so that the Finetune can enable the model to obtain a good effect after a small number of iterations, the iterations can be reduced to 1/10, the accuracy is observed to be improved or not, the results are output in a proper iteration process, whether a stable state is presented or not is observed, if the stable state is improved, the number of iterations can be considered to be increased, and if the stable state is basically stable, the number of iterations does not need to be increased;
s202-03, starting training and loading parameters of a pre-training model.
S3, evaluating classification performance: inputting the test set into each model, comparing the classification accuracy of each model in the subject, and evaluating the classification performance of each model; the method comprises the following specific steps:
s301, inputting training set data into the selected EEGNet model and the selected CNN model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
s302, after training the corresponding model, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and S303, comparing the predicted test set label with the real label to obtain the classification accuracy of the test.
The present invention proposes two migratory learning methods between the subject itself and the subject to evaluate the performance of the proposed framework and avoid the time consuming training process, as follows:
1) a migration learning method based on the subject: extracting the first 60 trials of each subject, and dividing the trials into a training set of 40 × 64 × 10000 and corresponding labels of 40 × 1, a test set of 20 × 64 × 10000 and corresponding labels of 20 × 1; the predictive model is then trained and tested.
2) A method of learning based on migration between subjects: experimental design all subjects were divided into two groups, the first group consisting of an array of 9 subjects as a training prediction model, assuming a total of 10 subject data were obtained. The second set consisted of the remaining 1 subject data for evaluation of the performance of the model. In turn, 1 subject was selected as test data, each subject being able to be selected for performance assessment. By the method, the average classification precision of 65.91% can be achieved.
Comparing the EEGNet model with the existing CNN model, the CNN model is a model formed by combining a convolution layer with a convolution kernel size of (30, 1), an average pooling layer with a convolution kernel size of (5, 1) and a long-time and short-time memory neural network, and evaluates better model classification performance. The results of the two models were compared according to different subjects and are shown in figure 4. Compared with other CNN models, the average accuracy of the invention is improved by about 7.27%, so that the EEGNet model can be used, and the advantages can be expressed as follows:
1) generalization can be performed across different BCI paradigms in the presence of limited data, and interpretable features can be generated;
2) the prediction performance of EEGNet is better than that of other models, and the training model can be simplified to reduce the complexity of the algorithm and improve the performance of the algorithm.
Example 2:
as shown in fig. 5, the system for recognizing motor imagery task of stroke patient based on transfer learning of the present invention comprises,
the acquisition and preprocessing module is used for acquiring electroencephalogram signals (EEG signals) containing MI of a stroke patient, wherein the information of the electroencephalogram signals comprises data of a subject and a corresponding class label, and the electroencephalogram signals are divided into a test set and a training set; preprocessing the electroencephalogram signals at the same time; wherein the preprocessing comprises down-sampling and filtering;
the characteristic extraction and classification module is used for combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals; the EEGNet model is specifically as follows:
(1) and in the input layer and the module I: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology, wherein the value of Dropout is preferably 0.25;
(2) and in the module II: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
(3) and in module three: in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
(4) in the classification block: features are passed directly into the softmax classification with N units; where N is the number of classes in the data, the softmax function is used because EEGNet is a multi-class model;
the finenetune technology in the transfer learning is specifically as follows:
modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
firstly, modifying the last full-connection layer of the original network, freezing parameters of the previous layers in a modification mode, deleting the category A of the softmax layer of the original network, and replacing the category A with the parameter N needing to be classified for updating;
secondly, loading the existing parameters of the EEGNet model, and training the model according to own data;
thirdly, after the Finetune succeeds, all layers are selected to be opened to carry out small-step parameter updating;
fourthly, fine adjustment of the learning rate of the model;
and (II) adjusting configuration parameters of the model, usually learning the rate and the step length, and reducing the iteration times, which is specifically as follows: freezing parameters of other convolutional layers during Finetune, slowly changing the size of the kernel of the pooling layer, observing whether the accuracy is improved or not through repeated training, and finally reducing the size of the pooling layer to 1/2, wherein the obtained accuracy effect is optimal;
and (III) starting training, and loading the parameters of the pre-training model.
And the classification performance evaluation module is used for inputting the test set into each model, comparing the classification accuracy of each model in the subject and evaluating the classification performance of each model. The working process of the classification performance evaluation module is as follows:
inputting training set data into the selected EEGNet model and CNN model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
(II) after the corresponding model is trained, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and (III) comparing the predicted test set label with a real label to obtain the classification accuracy of the test.
Example 3:
an embodiment of the present invention further provides an electronic device, including: a memory and a processor;
wherein the memory stores computer-executable instructions;
the one processor executes the computer-executable instructions stored in the memory, so that the one processor executes the stroke patient motor imagery task identification method based on the transfer learning in embodiment 1.
Example 6:
the embodiment of the invention also provides a computer-readable storage medium, wherein a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the method for identifying the motor imagery task of the stroke patient based on the transfer learning in the embodiment 1 of the invention. Specifically, a system or an apparatus equipped with a storage medium on which software program codes that realize the functions of any of the above-described embodiments are stored may be provided, and a computer (or a CPU or MPU) of the system or the apparatus is caused to read out and execute the program codes stored in the storage medium.
In this case, the program code itself read from the storage medium can realize the functions of any of the above-described embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.
Examples of the storage medium for supplying the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (e.g., CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-R stroke patient motor imagery task recognition method and system M, DVD-RW, DVD + RW based on migration learning), a magnetic tape, a nonvolatile memory card, and a ROM. Alternatively, the program code may be downloaded from a server computer via a communications network.
Further, it should be clear that the functions of any one of the above-described embodiments may be implemented not only by executing the program code read out by the computer, but also by causing an operating system or the like operating on the computer to perform a part or all of the actual operations based on instructions of the program code.
Further, it is to be understood that the program code read out from the storage medium is written to a memory provided in an expansion board inserted into the computer or to a memory provided in an expansion unit connected to the computer, and then causes a CPU or the like mounted on the expansion board or the expansion unit to perform part or all of the actual operations based on instructions of the program code, thereby realizing the functions of any of the above-described embodiments.
Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; and the modifications or the substitutions do not make the essence of the corresponding technical solutions depart from the scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A stroke patient motor imagery task recognition method based on transfer learning is characterized in that the method combines an EEGNet framework with a Finetune technology in transfer learning to realize motor imagery task classification recognition of stroke patients, effectively extracts electroencephalogram signal characteristics and realizes efficient mode classification; the method comprises the following specific steps:
collecting and preprocessing electroencephalogram signals: acquiring an electroencephalogram signal containing MI of a stroke patient, wherein the information of the electroencephalogram signal comprises data of a subject and a corresponding class label, and dividing the electroencephalogram signal into a test set and a training set; preprocessing the electroencephalogram signals at the same time;
feature extraction and classification: combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals;
and (3) evaluating classification performance: and inputting the test set into each model, comparing the classification accuracy of each model in the subject, and evaluating the classification performance of each model.
2. The method for identifying motor imagery tasks of stroke patients based on migratory learning of claim 1, wherein the preprocessing comprises down sampling and filtering.
3. The method for identifying motor imagery tasks of stroke patients based on transfer learning of claim 1 or 2, wherein the EEGNet model is specifically as follows:
in the input layer and module one: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology;
in the module two: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
in the classification block: features are passed directly into the softmax classification with N units; where N is the number of classes in the data.
4. The method for identifying motor imagery tasks of stroke patients based on transfer learning of claim 3, wherein the Finetune technique in transfer learning is as follows:
modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
modifying the last full-connection layer of the original network, freezing parameters of the previous layers in a modification mode, deleting the category A of the softmax layer of the original network, and replacing the category A with the parameter N needing to be classified for updating;
loading the existing parameters of the EEGNet model, and training the model according to own data;
after the Finetune succeeds, selectively opening all layers to update the small-step-length parameters;
learning rate fine-tuning for the model;
adjusting configuration parameters of the model, generally learning rate and step length, and reducing iteration times, specifically as follows: freezing parameters of other convolutional layers during Finetune, slowly changing the size of the kernel of the pooling layer, observing whether the accuracy is improved or not through repeated training, and finally reducing the size of the pooling layer to 1/2, wherein the obtained accuracy effect is optimal;
training is started and the parameters of the pre-trained model are loaded.
5. The method for identifying motor imagery tasks of stroke patients based on transfer learning of claim 4, wherein the evaluation classification performance is as follows:
inputting the training set data into the selected EEGNet model and CNN model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
after the corresponding model is trained, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and comparing the predicted label of the test set with the real label to obtain the classification accuracy of the test.
6. A brain stroke patient motor imagery task recognition system based on transfer learning is characterized by comprising,
the acquisition and preprocessing module is used for acquiring the electroencephalogram signals containing MI of a stroke patient, wherein the information of the electroencephalogram signals comprises data of a subject and a corresponding class label, and the electroencephalogram signals are divided into a test set and a training set; preprocessing the electroencephalogram signals at the same time; wherein the preprocessing comprises down-sampling and filtering;
the characteristic extraction and classification module is used for combining an EEGNet model with a Finetune technology in transfer learning to extract and classify the characteristics of the preprocessed electroencephalogram signals;
and the classification performance evaluation module is used for inputting the test set into each model, comparing the classification accuracy of each model in the subject and evaluating the classification performance of each model.
7. The system for recognizing motor imagery task for stroke patients based on transfer learning of claim 6, wherein the EEGNet model is specifically as follows:
(1) and in the input layer and the module I: starting from the input layer, performing convolution with a size (C, 1), convolution for reducing the number of trainable parameters to be fitted, and partial convolution layers are connected to the previous feature map; carrying out standardized modeling by adopting a Dropout technology;
(2) and in the module II: maximum pooling is performed using a deep convolution of size (2, 32) and kernel size (2, 2);
(3) and in module three: performing convolution by adopting deep convolution with the size of (8, 4), and selecting the same size of (2, 2) for pooling; the deep convolution effect of the second module and the third module is to reduce the number of parameters for fitting, and the kernel of each feature map can be well understood, and meanwhile, the maximum pooling layer is used for reducing the size;
(4) in the classification block: features are passed directly into the softmax classification with N units; wherein N is the number of classes in the data;
the finenetune technology in the transfer learning is specifically as follows:
modifying the output category of the last layer of the network, and accelerating the parameter learning rate of the last layer, specifically as follows:
firstly, modifying the last full-connection layer of the original network, freezing parameters of the previous layers in a modification mode, deleting the category A of the softmax layer of the original network, and replacing the category A with the parameter N needing to be classified for updating;
secondly, loading the existing parameters of the EEGNet model, and training the model according to own data;
thirdly, after the Finetune succeeds, all layers are selected to be opened to carry out small-step parameter updating;
fourthly, fine adjustment of the learning rate of the model;
and (II) adjusting configuration parameters of the model, usually learning the rate and the step length, and reducing the iteration times, which is specifically as follows: freezing parameters of other convolutional layers during Finetune, slowly changing the size of the kernel of the pooling layer, observing whether the accuracy is improved or not through repeated training, and finally reducing the size of the pooling layer to 1/2, wherein the obtained accuracy effect is optimal;
and (III) starting training, and loading the parameters of the pre-training model.
8. The system for identifying motor imagery tasks of stroke patients based on transfer learning of claim 6 or 7, wherein the classification performance evaluation module specifically operates as follows:
inputting training set data into the selected EEGNet model and CNN model, continuously performing iterative training through a network, and adjusting network parameters to finally obtain a trained network model;
(II) after the corresponding model is trained, inputting the test set into the model to obtain the final classification accuracy of the motor imagery task, and further evaluating the classification performance of the model;
and (III) comparing the predicted test set label with a real label to obtain the classification accuracy of the test.
9. An electronic device, comprising: a memory and at least one processor;
wherein the memory stores computer-executable instructions;
the at least one processor executing the memory-stored computer-executable instructions cause the at least one processor to perform the stroke patient motor imagery task identification method based on transfer learning of any one of claims 1 to 5.
10. A computer-readable storage medium having stored thereon computer-executable instructions which, when executed by a processor, implement the method for identifying motor imagery tasks of stroke patients based on migratory learning as claimed in claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010527624.4A CN111695500A (en) | 2020-06-11 | 2020-06-11 | Method and system for recognizing motor imagery task of stroke patient based on transfer learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010527624.4A CN111695500A (en) | 2020-06-11 | 2020-06-11 | Method and system for recognizing motor imagery task of stroke patient based on transfer learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111695500A true CN111695500A (en) | 2020-09-22 |
Family
ID=72480167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010527624.4A Pending CN111695500A (en) | 2020-06-11 | 2020-06-11 | Method and system for recognizing motor imagery task of stroke patient based on transfer learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111695500A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113283414A (en) * | 2021-07-26 | 2021-08-20 | 深圳市安软科技股份有限公司 | Pedestrian attribute identification method, related equipment and computer readable storage medium |
| CN113408397A (en) * | 2021-06-16 | 2021-09-17 | 西安电子科技大学 | Domain-adaptive cross-subject motor imagery electroencephalogram signal identification system and method |
| CN113951898A (en) * | 2021-10-15 | 2022-01-21 | 浙江大学 | P300 electroencephalogram signal detection method and device for data migration, electronic device and medium |
| CN114305452A (en) * | 2021-12-15 | 2022-04-12 | 南京航空航天大学 | Cross-task cognitive load identification method based on electroencephalogram and field adaptation |
| CN114767120A (en) * | 2022-04-25 | 2022-07-22 | 上海韶脑传感技术有限公司 | Depth learning-based selection method for motor imagery electroencephalogram channels of unilateral limb patients |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106485251A (en) * | 2016-10-08 | 2017-03-08 | 天津工业大学 | Egg embryo classification based on deep learning |
| CN106991409A (en) * | 2017-04-14 | 2017-07-28 | 山东建筑大学 | A kind of Mental imagery EEG feature extraction and categorizing system and method |
| CN109377441A (en) * | 2018-08-20 | 2019-02-22 | 清华大学 | Tongue with privacy protection function is as acquisition method and system |
| CN110765920A (en) * | 2019-10-18 | 2020-02-07 | 西安电子科技大学 | Motor imagery classification method based on convolutional neural network |
| CN110837807A (en) * | 2019-11-11 | 2020-02-25 | 内蒙古大学 | Identification method and system for fake-licensed vehicle |
-
2020
- 2020-06-11 CN CN202010527624.4A patent/CN111695500A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106485251A (en) * | 2016-10-08 | 2017-03-08 | 天津工业大学 | Egg embryo classification based on deep learning |
| CN106991409A (en) * | 2017-04-14 | 2017-07-28 | 山东建筑大学 | A kind of Mental imagery EEG feature extraction and categorizing system and method |
| CN109377441A (en) * | 2018-08-20 | 2019-02-22 | 清华大学 | Tongue with privacy protection function is as acquisition method and system |
| CN110765920A (en) * | 2019-10-18 | 2020-02-07 | 西安电子科技大学 | Motor imagery classification method based on convolutional neural network |
| CN110837807A (en) * | 2019-11-11 | 2020-02-25 | 内蒙古大学 | Identification method and system for fake-licensed vehicle |
Non-Patent Citations (1)
| Title |
|---|
| VERNON J.LAWHERN等: ""EEGNet: A Compact Convolutional Neural Network for EEG-based Brain-Computer Interfaces"" * |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113408397A (en) * | 2021-06-16 | 2021-09-17 | 西安电子科技大学 | Domain-adaptive cross-subject motor imagery electroencephalogram signal identification system and method |
| CN113283414A (en) * | 2021-07-26 | 2021-08-20 | 深圳市安软科技股份有限公司 | Pedestrian attribute identification method, related equipment and computer readable storage medium |
| CN113951898A (en) * | 2021-10-15 | 2022-01-21 | 浙江大学 | P300 electroencephalogram signal detection method and device for data migration, electronic device and medium |
| CN114305452A (en) * | 2021-12-15 | 2022-04-12 | 南京航空航天大学 | Cross-task cognitive load identification method based on electroencephalogram and field adaptation |
| CN114767120A (en) * | 2022-04-25 | 2022-07-22 | 上海韶脑传感技术有限公司 | Depth learning-based selection method for motor imagery electroencephalogram channels of unilateral limb patients |
| CN114767120B (en) * | 2022-04-25 | 2024-05-10 | 上海韶脑传感技术有限公司 | Single-side limb patient motor imagery electroencephalogram channel selection method based on deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111695500A (en) | Method and system for recognizing motor imagery task of stroke patient based on transfer learning | |
| Sun et al. | A review of adaptive feature extraction and classification methods for EEG-based brain-computer interfaces | |
| CN110070105B (en) | Electroencephalogram emotion recognition method and system based on meta-learning example rapid screening | |
| CN108256629B (en) | EEG signal unsupervised feature learning method based on convolutional network and self-coding | |
| CN109784023B (en) | Steady-state vision-evoked electroencephalogram identity recognition method and system based on deep learning | |
| CN111709267B (en) | Electroencephalogram signal emotion recognition method of deep convolutional neural network | |
| CN108090509B (en) | Data length self-adaptive electrocardiogram classification method | |
| CN110555468A (en) | Electroencephalogram signal identification method and system combining recursion graph and CNN | |
| CN111329474A (en) | Electroencephalogram identity recognition method and system based on deep learning and information updating method | |
| Yedukondalu et al. | Cognitive load detection using circulant singular spectrum analysis and Binary Harris Hawks Optimization based feature selection | |
| Zhao et al. | Deep CNN model based on serial-parallel structure optimization for four-class motor imagery EEG classification | |
| CN112488002B (en) | Emotion recognition method and system based on N170 | |
| CN112450885B (en) | Epileptic electroencephalogram-oriented identification method | |
| CN110558975A (en) | Electrocardiosignal classification method and system based on LRF-ELM and BLSTM | |
| CN110543831A (en) | brain print identification method based on convolutional neural network | |
| Yan et al. | A novel exploitative and explorative GWO-SVM algorithm for smart emotion recognition | |
| Lian et al. | The improved ELM algorithms optimized by bionic WOA for EEG classification of brain computer interface | |
| Joshi et al. | Classification of brainwaves using convolutional neural network | |
| Ince et al. | Automated patient-specific classification of premature ventricular contractions | |
| Alsuwaiket | Feature Extraction of EEG Signals for Seizure Detection Using Machine Learning Algorthims | |
| CN113128353B (en) | Emotion perception method and system oriented to natural man-machine interaction | |
| CN111428601B (en) | P300 signal identification method, device and storage medium based on MS-CNN | |
| CN110738093B (en) | Classification method based on improved small world echo state network electromyography | |
| Liu et al. | Automated Machine Learning for Epileptic Seizure Detection Based on EEG Signals. | |
| CN114699093A (en) | Electroencephalogram seizure signal detection method based on convolutional neural network and long-term and short-term memory |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |