CN116746947A - Cross-subject electroencephalogram signal classification method based on online test time domain adaptation - Google Patents
Cross-subject electroencephalogram signal classification method based on online test time domain adaptation Download PDFInfo
- Publication number
- CN116746947A CN116746947A CN202310719543.8A CN202310719543A CN116746947A CN 116746947 A CN116746947 A CN 116746947A CN 202310719543 A CN202310719543 A CN 202310719543A CN 116746947 A CN116746947 A CN 116746947A
- Authority
- CN
- China
- Prior art keywords
- model
- electroencephalogram
- source
- electroencephalogram signal
- formula
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 230000006978 adaptation Effects 0.000 title claims abstract description 20
- 238000012360 testing method Methods 0.000 title claims abstract description 19
- 238000012549 training Methods 0.000 claims abstract description 27
- 210000004556 brain Anatomy 0.000 claims abstract description 17
- 239000012634 fragment Substances 0.000 claims abstract description 7
- 230000006870 function Effects 0.000 claims description 27
- 230000004913 activation Effects 0.000 claims description 15
- 125000003275 alpha amino acid group Chemical group 0.000 claims description 13
- 150000001875 compounds Chemical class 0.000 claims description 13
- 238000012545 processing Methods 0.000 claims description 13
- 238000011176 pooling Methods 0.000 claims description 9
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 7
- 238000013145 classification model Methods 0.000 claims description 6
- 238000002372 labelling Methods 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 238000004590 computer program Methods 0.000 claims description 5
- 238000005070 sampling Methods 0.000 claims description 5
- 239000000126 substance Substances 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 2
- 230000009467 reduction Effects 0.000 claims description 2
- 238000010276 construction Methods 0.000 claims 1
- 238000007781 pre-processing Methods 0.000 abstract 1
- 230000009466 transformation Effects 0.000 abstract 1
- 238000013527 convolutional neural network Methods 0.000 description 6
- 230000009286 beneficial effect Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 241000282472 Canis lupus familiaris Species 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 238000007619 statistical method Methods 0.000 description 3
- 238000000692 Student's t-test Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000012353 t test Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000007635 classification algorithm Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000000537 electroencephalography Methods 0.000 description 1
- 230000008909 emotion recognition Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 238000007917 intracranial administration Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007427 paired t-test Methods 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7271—Specific aspects of physiological measurement analysis
- A61B5/7285—Specific aspects of physiological measurement analysis for synchronising or triggering a physiological measurement or image acquisition with a physiological event or waveform, e.g. an ECG signal
- A61B5/7289—Retrospective gating, i.e. associating measured signals or images with a physiological event after the actual measurement or image acquisition, e.g. by simultaneously recording an additional physiological signal during the measurement or image acquisition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/096—Transfer learning
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Theoretical Computer Science (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Psychiatry (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- Mathematical Physics (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physiology (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Signal Processing (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Psychology (AREA)
- Fuzzy Systems (AREA)
Abstract
The invention discloses a method for classifying brain electrical signals of a cross-subject based on online test time domain adaptation, which comprises the following steps: 1, preprocessing original EEG data, including removing noise, segmenting fragments, extracting time-frequency characteristics by utilizing short-time Fourier transformation, and obtaining source domain data and target domain data; 2, constructing a source model, a student model and a teacher model based on a CNN network, and training the source model by inputting data to obtain a pre-training source model; 3. initializing a student model and a teacher model by using an existing pre-training invention model; and 4, online tuning and optimizing the student model and the teacher model on the target data stream based on a mutual learning strategy, and realizing classification of the electroencephalogram signals. The invention can realize the rapid classification of the electroencephalogram signals under the condition of protecting the privacy of the patient, thereby meeting the real-time requirement of a classification system of the electroencephalogram signals in an actual scene.
Description
Technical Field
The invention relates to the field of electroencephalogram signal classification, in particular to a cross-subject electroencephalogram signal classification method based on online test time domain adaptation.
Background
Electroencephalography (EEG) is a physiological technique for recording brain electrical activity. The recognition and prediction of physiological and psychological states from patterns of neural activity observed in scalp and intracranial electroencephalograms is widely used in the field of brain-computer interfaces for emotion recognition, motor imagery, medical health, etc. While Deep Neural Networks (DNNs) have had unprecedented success in various brain-computer interface applications, such as seizure prediction, existing approaches typically train models in a patient-specific manner, i.e., training and testing data from the same subject. Because of the time-varying characteristics and heterogeneity of the electroencephalogram signals of different subjects, when the source domain trained model is directly used for the electroencephalogram data of the target domain subjects, the model performance can be seriously degraded, so that the system has poor robustness.
Although some domain adaptation techniques have been used to solve the above problems, most of the existing domain adaptation techniques require access to source domain data, which may cause privacy protection problems due to the inclusion of sensitive physiological information of the user in the electroencephalogram signal, and recently, some passive domain adaptation (SFDA) -based methods have been proposed by researchers. SFDA focuses on modifying well-learned source models to target source domain data without using any source domain data, which is beneficial to protecting the privacy of the source domain data.
However, existing SFDA methods focus only on offline applications, which means that all of the target subject's brain electrical data is available prior to training. In the training phase, the offline SFDA optimizes the source pre-training model through a plurality of epochs using a certain amount of target data, and then evaluates the performance of the task model using the target test data. This arrangement has a number of limitations: first, waiting for a sufficient collection of all unlabeled electroencephalogram samples of the target subject in advance is inefficient and time consuming. Second, since the offline algorithm requires multiple accesses to the target subject's electroencephalogram data and multiple epoch iterations to train the classifier to achieve reliable performance, this further increases the cost of algorithm computation time. Third, when new electroencephalogram data needs to be classified, the task model needs to be retrained, which is costly. Finally, because the electroencephalogram signal has time-varying characteristics and unstable characteristics, the classifier is required to be continuously adjusted to adapt to new electroencephalogram data distribution, and the offline model cannot meet the requirement.
Disclosure of Invention
In order to overcome the defects, the invention provides a cross-subject electroencephalogram signal classification method based on online test time domain adaptation, so that dynamic prediction of a target subject electroencephalogram signal can be realized on the premise of protecting privacy of a patient, and the real-time requirement of an electroencephalogram signal classification system in practical application can be met.
In order to achieve the aim of the invention, the invention adopts the following technical scheme:
the invention discloses a method for classifying brain electrical signals of a cross-subject based on online test time domain adaptation, which is characterized by comprising the following steps:
step 1.1, acquiring an electroencephalogram signal source domain data set with labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal source domain data set to obtain source domain electroencephalogram signals of C channels; after slicing the C-channel source domain electroencephalogram signals through a sliding window, extracting the time-frequency characteristics of the sliced source domain electroencephalogram signals by adopting short-time Fourier transform, and reconstructing the input shape of the source domain electroencephalogram signals after short-time Fourier processingThereby obtaining N S C for a time step of t S The electroencephalogram signal sample in the source-like domain is recorded as a source training set
wherein ,/>Represents the ith source domain brain electrical signal sample in the jth brain electrical signal, and is/are added with>For brain electric signal sample->The corresponding label; c (C) S Representing the category number of the electroencephalogram signals, c represents the channel number of the electroencephalogram signal samples, w represents the width of a sliding window, and h represents the height of the electroencephalogram signal samples;
step 1.2, acquiring an electroencephalogram signal target domain data set without labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal target domain data set to obtain target domain electroencephalogram signals of C channels; after the C-channel target domain electroencephalogram signals are sliced through a sliding window, the time-frequency characteristics of the sliced target domain electroencephalogram signals are extracted by adopting short-time Fourier transform, and the input shape of the target domain electroencephalogram signals after short-time Fourier processing is reconstructed, so that N is obtained T Target domain data set composed of individual electroencephalogram signal fragments wherein ,Segn Representing a target domain dataset D T An electroencephalogram signal of the nth section, and-> wherein ,/>Representing a target domainData set D T N-th electroencephalogram fragment Seg n A small batch of samples at time step t +.>The b-th electroencephalogram signal sample in (a);
step 2, constructing a source model f and a student model f based on a CNN network s Teacher model f t ；
The CNN network comprises: first convolution module Conv3D and second convolution module Conv2D 1 Third convolution module Conv2D 2 A classification module;
the first convolution module Conv3D sequentially comprises: the first batch of normalized layers, step size (1, s) 1 ，s 1 ) And the convolution kernel is (c, k) 1 ，k 1 ) A first convolution layer, a first ReLU nonlinear activation function layer, and a convolution kernel of (1, d) 1 ，d 1 ) Is a maximum pooling layer of (2);
the second convolution module Conv2D 1 The method sequentially comprises the following steps: a second batch of normalization layers with step length s 2 And the convolution kernel is k 2 A second convolution layer, a second ReLU nonlinear activation function layer, and a convolution kernel of d 2 Is a maximum pooling layer of (2);
the third convolution module Conv2D 2 The method sequentially comprises the following steps: the third batch of normalization layers has a step length s 3 And the convolution kernel is k 3 A third convolution layer, a third ReLU nonlinear activation function layer, and a convolution kernel of d 3 Is a maximum pooling layer of (2);
the classification module comprises: two connection layers FC 1 and FC2 A first Sigmoid nonlinear activation function;
step 2.1, initializing the weights of all convolution kernels in the CNN network by using a kaiming initializer;
step 2.2, sampling the ith source domain brain electrical signalInputting the initial characteristic into the source model f and performing initial characteristic through a first convolution module Conv3D in the source model fExtracting and reducing feature dimension to obtain a first feature sequenceWherein h' represents the first characteristic sequence +.>Is of a height of (2);
step 2.3, the first feature sequenceSequentially passing through the second convolution module Conv2D 1 After treatment of (2) a second characteristic sequence is obtained>Wherein c' represents a second characteristic sequence +.>The number of channels, w' represents the second characteristic sequence +.>Is the width of the second characteristic sequence +.>Is of a height of (2);
the second characteristic sequenceSequentially passing through a third convolution module Conv2D 2 After the processing of (2) outputting the third characteristic sequence +.>Wherein h' "represents the third characteristic sequence +.>W "represents the third characteristic sequence +.>C "represents the third characteristic sequence +.>The number of channels;
step 2.4 for the third feature sequenceThe fourth characteristic sequence +.>
Fourth characteristic sequenceIs input into the classification module and passes through the first full connection layer FC 1 And processing the first Sigmoid nonlinear activation function to obtain an ith section of electroencephalogram signal sample +.>Fifth characteristic sequence of (2)Wherein a represents a fifth characteristic sequence +.>B represents a fifth characteristic sequence +.>Is a width of (2);
the fifth characteristic sequenceInputting the second full connection layer FC 2 The (1) processing is carried out to obtain a source model f for the ith section of electroencephalogram signal sample +.>Final Cs logic output values +.> wherein ,/>Representation of source model f prediction of i-th segment electroencephalogram signal sample +.>Logit values belonging to class j, b' representing +.>And b' < b;
step 2.5, establishing a counter-propagating loss function L using the rebalancing-like loss function of formula (1) CE ：
In the formula (1), L i,j Represents the ith electroencephalogram signal sample in the jth electroencephalogram signalAnd is obtained from formula (2):
in the formula (2), the amino acid sequence of the compound,represents the i-th electroencephalogram signal sample +.>Probability values belonging to the j-th class and obtained by the formula (3):
step 2.6, based on the source training set D S Training the source model f by using an Adam optimizer, and calculating a loss function L CE When the training iteration number reaches the set number or the loss error is smaller than the set threshold, the training is stopped, so that the optimal source classification model f is obtained 0 ；
Step 3, initializing n=1;
step 4, initializing t=1;
step 4.0, utilizing the optimal source classification model f 0 Respectively initializing chemical raw model f s And teacher model f t Obtaining a student model at a time step tAnd teacher model f t t ；
Step 4.1, sample a small batchRespectively inputting teacher model f under time step t t t Sum student model->Respectively obtain teacher models f t t Output Cs logic values +.>Student model->Output Cs logic values +.> wherein ,/>Representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->A logic value belonging to the j-th class; />Representing student model f t s Predicting a small sample +.>B-th electroencephalogram signal sample in (B)A logic value belonging to the j-th class;
step 4.1 calculating loss unsupervised loss L using equation (4) hybrid ：
L hybrid ＝L ent +δL div +γL sce (4)
In the formula (4), delta and gamma both represent regularization coefficients, L ent Represents the test entropy and is obtained from formula (5), L div Represents KL divergence and is obtained from formula (6), L sce Represents symmetrical cross entropy and is obtained by the formula (6);
in the formula ,representing student model->Predicting a small sample +.>B of (b)Personal brain signal sample->Probability values belonging to the j-th class and obtained by the formula (6):
in the formula ,representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->Probability values belonging to the j-th class and obtained by equation (8):
step 4.2 is based onStudent model at time step t using Adam optimizer +.>Training and minimizing an unsupervised loss function L hybrid When student model->After one epoch, the student model +.1 time step is obtained>Meanwhile, the teacher model f is used for the teacher model f by the method (7) t t Parameter +.>Updating to obtain a teacher model f t t Parameter +.about.1 at time step t+1>
In the formula (7), the amino acid sequence of the compound,representing student model->Parameters at time step t+1; alpha is a slip factor and has:
in the formula (8), the amino acid sequence of the compound, and />Respectively refer to student models->For a small batch of samples at time step tBook (I)>The average prediction probability of the most confidence and the less confidence obtained in the prediction is as follows:
in the formula (9), the amino acid sequence of the compound, and />Representing student model->Next sample of small lot +.>Sample b of (b)
The most and less confidence prediction probabilities;
step 4.3 student model at time step t+1For a small sample lot->Classifying to obtain a classification result;
step 4.4 after assigning t+1 to t, if t>T represents Seg n The samples in (a) are all classified, the step 4.5 is executed, otherwise, the step 4.0 is returned to be executed in sequence;
step 4.5 after n+1 is assigned to n, if n>N T Then the target domain dataset is representedAll the samples in the model are classified; otherwise, returning to the execution step 4 for sequential execution.
The invention provides an electronic device comprising a memory and a processor, wherein the memory is used for storing a program for supporting the processor to execute the method for classifying the electroencephalogram signals of the subjects, and the processor is configured to execute the program stored in the memory.
The invention relates to a computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and is characterized in that the computer program is executed by a processor to execute the steps of the inter-subject electroencephalogram signal classification method.
Compared with the prior art, the invention has the beneficial effects that:
1. the invention provides an electroencephalogram signal state classification framework (TSOTTA) based on an online test time domain adaptation technology, which starts from an existing source model and continuously adapts to current test electroencephalogram data, and the source domain data is not available, so that privacy of a patient can be protected; the target data arrive in sequence, each target sample only needs to be accessed once, and classification and self-adaption are carried out simultaneously, so that the real-time requirement of an electroencephalogram classification system is met.
2. Aiming at the problems of time-varying characteristics and heterogeneity of electroencephalogram data, the invention designs an online mutual learning strategy for teachers and students, and updates a student model by optimizing mixing loss, wherein the teacher model is an average value of continuous student models. The teacher mode and the student mode are updated simultaneously, and the student mode is supervised on line; this strategy is beneficial to maintaining the classification ability of the student model on the electroencephalogram signals of the source domain subjects. The proximity update strategy is beneficial to mining the correlation between electroencephalogram samples, and better electroencephalogram classification performance is obtained.
Drawings
FIG. 1 is a schematic diagram of a paradigm comparison of three field-adaptive algorithms involved in the present invention;
FIG. 2 is a schematic diagram of an electroencephalogram signal classification method according to the present invention;
FIG. 3 is a schematic diagram of a data partitioning process in the present invention;
FIG. 4 is a schematic diagram of the model self-training process in the present invention.
Detailed Description
In this embodiment, an electroencephalogram signal classification method based on online test time domain adaptation is mainly designed, and an online electroencephalogram signal classification framework with privacy protection is provided, wherein the framework performs classification and adaptation on electroencephalogram signals of a target domain at the same time, and a specific paradigm is shown in a section (c) in fig. 1. Unlike part (b) of fig. 1, the previous domain adaptation method shown in part (c) of fig. 1, the framework uses only one pre-trained source model and label-free target data, improves the performance of the pre-trained source model on the target domain with continuously changing distribution in an online tuning manner, the adaptation process does not use the source domain data, and the target domain data arrives online. The overall framework of the method is shown in fig. 2; specifically, the method comprises the following steps:
the method uses two published electroencephalogram datasets: CHB-MIT and kagle; as shown in fig. 3, to maximize the potential of available electroencephalographic data, the present invention employs a strategy to leave a subject to evaluate TSOTTA. Specifically, on the CHB-MIT dataset, the present invention selects one patient as the target new patient and another 19 patients as the source patients. In the kagle data set, since the number of channels of the electroencephalogram signals of the dog 5 is different from the number of channels of the rest 4 dogs, the dog 5 is discarded. Furthermore, the present invention divides the source domain data into two subsets: 9/10 of the data is used as training set x s tr The remaining 1/10 is assigned as verification set x s val . Meanwhile, in order to simulate a scene that test data arrive in sequence, only one electroencephalogram sample without label information is provided in each step, and the sample can only be accessed once.
Step 1.1, acquiring an electroencephalogram signal source domain data set with labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal source domain data set to obtain source domain electroencephalogram signals of C channels; through sliding windowAfter slicing the C-channel source domain electroencephalogram signals, extracting the time-frequency characteristics of the sliced source domain electroencephalogram signals by adopting short-time Fourier transform, and reconstructing the input shape of the short-time Fourier processed source domain electroencephalogram signals to obtain N S C for a time step of t S The electroencephalogram signal sample in the source-like domain is recorded as a source training set
wherein ,/>Represents the ith source domain brain electrical signal sample in the jth brain electrical signal, and is/are added with>For brain electric signal sample->The corresponding label; c (C) S Representing the category number of the electroencephalogram signals, c represents the channel number of the electroencephalogram signal samples, w represents the width of a sliding window, and h represents the height of the electroencephalogram signal samples;
the method comprises the steps of taking a channel number of 22, a sliding window length of 30s and an electroencephalogram signal sampling rate of 256HZ per second on a CHB-MIT electroencephalogram data set; on the Kaggle electroencephalogram data set, the number of channels is 16, the sliding window length is 30s, and the sampling rate of electroencephalogram signals per second is 400HZ.
Step 1.2, acquiring an electroencephalogram signal target domain data set without labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal target domain data set to obtain target domain electroencephalogram signals of C channels; after the C-channel target domain electroencephalogram signals are sliced through the sliding window, the time-frequency characteristics of the sliced target domain electroencephalogram signals are extracted by adopting short-time Fourier transform, and the input shape of the target domain electroencephalogram signals after short-time Fourier processing is reconstructed, so that N is obtained T The electroencephalogram signal fragments form a target domain data set wherein Segn Representing a target domain dataset D T An electroencephalogram signal of the nth section, and-> wherein ,/>Representing a target domain dataset D T N-th electroencephalogram fragment Seg n A small batch of samples at time step t +.>The b-th electroencephalogram signal sample in (a);
step 2, constructing a source model f and a student model f based on a CNN network s Teacher model f t ；
The CNN network comprises: first convolution module Conv3D and second convolution module Conv2D 1 Third convolution module Conv2D 2 A classification module;
the first convolution module Conv3D sequentially comprises: the first batch of normalized layers, step size (1, s) 1 ，s 1 ) And the convolution kernel is (c, k) 1 ，k 1 ) A first convolution layer, a first ReLU nonlinear activation function layer, and a convolution kernel of (1, d) 1 ，d 1 ) Is a maximum pooling layer of (2);
second convolution module Conv2D 1 The method sequentially comprises the following steps: a second batch of normalization layers with step length s 2 And the convolution kernel is k 2 A second convolution layer, a second ReLU nonlinear activation function layer, and a convolution kernel of d 2 Is a maximum pooling layer of (2);
third convolution module Conv2D 2 The method sequentially comprises the following steps: the third batch of normalization layers has a step length s 3 And the convolution kernel is k 3 A third convolution layer, a third ReLU nonlinear activation function layer, and a convolution kernel of d 3 Is a maximum pooling layer of (2);
the classification module comprises: two connection layers FC 1 and FC2 A first Sigmoid nonlinear activation function;
step 2.1, initializing the weights of all convolution kernels in the CNN network by using a kaiming initializer;
step 2.2, sampling the ith source domain brain electrical signalInputting the initial feature extraction and feature dimension reduction into a source model f through a first convolution module Conv3D in the source model f, and obtaining a first feature sequenceWherein h' represents the first characteristic sequence +.>Is of a height of (2);
step 2.3, first feature sequenceSequentially passing through a second convolution module Conv2D 1 After treatment of (2) a second characteristic sequence is obtained>Wherein c' represents a second characteristic sequence +.>The number of channels, w' represents the second characteristic sequence +.>Is the width of the second characteristic sequence +.>Is of a height of (2);
second characteristic sequenceSequentially passing through a third convolution module Conv2D 2 After the processing of (a), outputting a third characteristic sequenceWherein h' "represents the third characteristic sequence +.>W "represents the third characteristic sequence +.>C "represents the third characteristic sequence +.>The number of channels;
step 2.4 for the third feature sequenceThe fourth characteristic sequence +.>
Fourth characteristic sequenceInput into the classification module and pass through the first full connection layer FC 1 And processing the first Sigmoid nonlinear activation function to obtain an ith section of electroencephalogram signal sample +.>Fifth characteristic sequence of (2)Wherein a represents a fifth characteristic sequence +.>B represents a fifth characteristic sequence +.>Is a width of (2);
the fifth characteristic sequenceInputting the second full connection layer FC 2 The (1) processing is carried out to obtain a source model f for the ith section of electroencephalogram signal sample +.>Final Cs logic output values +.> wherein ,/>Representation of source model f prediction of i-th segment electroencephalogram signal sample +.>Logit values belonging to class j, b' representing +.>And b' < b;
step 2.5, establishing a counter-propagating loss function L using the rebalancing-like loss function of formula (1) CE ：
In the formula (1), L i,j Represents the ith electroencephalogram signal sample in the jth electroencephalogram signalAnd is obtained from formula (2):
in the formula (2), the amino acid sequence of the compound,represents the i-th electroencephalogram signal sample +.>Probability values belonging to the j-th class and obtained by the formula (3):
in the formula (3), the amino acid sequence of the compound,representing the source model f for the ith section of electroencephalogram signal sample +.>An output logic value;
step 2.6, source training set D based S Training the source model f by using an Adam optimizer, and calculating a loss function L CE The batch size in this example was set to 64 and the initial learning rate of the adam optimizer was set to 0.001. In addition, an early stopping technology is applied, and when the verification error lasts for 10 training periods, model training is stopped, so that an optimal source classification model f is obtained 0 ；
Step 3, the online model tuning process is shown in fig. 4, and n=1 is initialized;
step 4, initializing t=1;
step 4.0, utilizing the optimal source classification model f 0 Respectively initializing chemical raw model f s And teacher model f t Obtaining a student model at a time step tAnd teacher model f t t ；
Step 4.1, sample a small batchTeacher model for respectively inputting time step tf t t Sum student model->Respectively obtain teacher models f t t Output Cs logic values +.>Student model->Output Cs logic values +.> wherein ,/>Representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->A logic value belonging to the j-th class; />Representing student model f t s Predicting a small sample +.>B-th electroencephalogram signal sample in (B)The logic value belonging to the j-th class.
Step 4.1 calculating loss unsupervised loss L using equation (4) hybrid ：
L hybrid ＝L ent +δL div +γL sce (4)
In the formula (4), delta and gamma are shown in the tableShow regularization coefficient, L ent Represents the test entropy and is obtained from formula (5), L div Represents KL divergence and is obtained from formula (6), L sce Represents symmetrical cross entropy and is obtained by the formula (6);
wherein ,representing student model->Predicting a small sample +.>B-th electroencephalogram signal sample->Probability values belonging to the j-th class and obtained by the formula (6):
wherein ,representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->Probability values belonging to class j and obtained by formula (8)To:
wherein j represents the class number of the electroencephalogram sample;
step 4.2 is based onStudent model at time step t using Adam optimizer +.>Training and minimizing an unsupervised loss function L hybrid When student model->After one epoch, the student model +.1 time step is obtained>Meanwhile, the teacher model f is used for the teacher model f by the method (7) t t Parameter +.>Updating to obtain a teacher model f t t Parameter +.about.1 at time step t+1>
In the formula (7), the amino acid sequence of the compound,representing student model->Parameters at time step t+1; alpha is a slip factor and has:
in the formula (8), the amino acid sequence of the compound, and />Respectively refer to student models->For a small batch of samples at time step t +.>The average prediction probability of the most confidence and the less confidence obtained in the prediction is as follows:
in the formula (9), the amino acid sequence of the compound, and />Representing student model->Next sample of small lot +.>Sample b of (b)
The most and less confidence prediction probabilities;
step 4.3 student model at time step t+1For a small sample lot->Classifying to obtain a classification result;
step 4.4 after assigning t+1 to t, if t>T represents Seg n The samples in (a) are all classified, the step 4.5 is executed, otherwise, the step 4.0 is returned to be executed in sequence;
step 4.5 after n+1 is assigned to n, if n>N T Then the target domain dataset is representedAll the samples in the model are classified; otherwise, returning to the execution step 4 for sequential execution.
In this embodiment, an electronic device includes a memory for storing a program supporting the processor to execute the above method, and a processor configured to execute the program stored in the memory.
In this embodiment, a computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the method described above.
The online test time domain adaptation technology in the invention expands experiments on two public data sets of CHB-MIT and Kaggle respectively, and the invention can not be directly compared with the existing electroencephalogram signal classification algorithm due to different experimental conditions and settings. In order to evaluate the effectiveness of TSOTTA, the present invention compares it to an on-line test time domain adaptation method typical of the image field. In this example, four widely used evaluation indicators are used to measure model performance. Sensitivity (SEN) refers to the proportion of positive classes correctly classified in all positive class samples, the greater SEN, the more accurate positive class classification capability the classifier has; specificity (SPEC represents the proportion of correctly classified negative samples in all negative samples, and represents the capability of the classifier to correctly classify the negative samples, AUC is an important index for accurately balancing the prediction performance of the model, AUC value of a random classifier is 0.5, and a perfect classifier can reach an AUC value of 1, according to the invention, paired t test statistical analysis is carried out on the methods on the basis of the AUC at the significance level of 0.05, and can measure whether the model is superior to a contrast method classifier, when p-value (p-value) is smaller than 0.05, the model is obviously superior to the contrast classifier, and specific statistical results on CHB-MIT and Kaggle data sets are respectively shown in table 3 and table 4.
Table 1TSOTTA and four baseline methods performance comparisons on the CHB-MIT database
Table 2TSOTTA and four baseline methods performance comparisons on kagle database
TABLE 3 t-test statistical analysis of different domain adaptation methods on the CHB-MIT dataset at a significance level of 0.05
Table 4 t-test statistical analysis of different domain adaptation methods on Kaggle dataset at a significance level of 0.05
In summary, the invention provides an online test time domain adaptation framework aiming at the problems that the data privacy of the tested person is revealed and the offline system cannot learn online in the clinical practical application, and the framework not only considers the problem of patient privacy protection, but also realizes online tuning and optimization of a task model through a teacher-student mutual learning strategy, thereby executing online classification setting of the electroencephalogram signals and achieving the faster and more accurate electroencephalogram signal classification effect.
Claims (3)
1. The method for classifying the inter-subject electroencephalogram signals based on the online test time domain adaptation is characterized by comprising the following steps of:
step 1.1, acquiring an electroencephalogram signal source domain data set with labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal source domain data set to obtain source domain electroencephalogram signals of C channels; after slicing the C-channel source domain electroencephalogram signals through a sliding window, extracting the time-frequency characteristics of the sliced source domain electroencephalogram signals by adopting short-time Fourier transform, and reconstructing the input shape of the source domain electroencephalogram signals after short-time Fourier processing, thereby obtaining N S C for a time step of t S The electroencephalogram signal sample in the source-like domain is recorded as a source training set
wherein ,/>Represents the ith source domain brain electrical signal sample in the jth brain electrical signal, and is/are added with>For brain electric signal sample->The corresponding label; c (C) S Representing the category number of the electroencephalogram signals, c represents the channel number of the electroencephalogram signal samples, w represents the width of a sliding window, and h represents the height of the electroencephalogram signal samples;
step 1.2, acquiring an electroencephalogram signal target domain data set without labeling category information, and carrying out channel data selection on original electroencephalogram signals in the electroencephalogram signal target domain data set to obtain target domain electroencephalogram signals of C channels; after the C-channel target domain electroencephalogram signals are sliced through a sliding window, the time-frequency characteristics of the sliced target domain electroencephalogram signals are extracted by adopting short-time Fourier transform, and the input shape of the target domain electroencephalogram signals after short-time Fourier processing is reconstructed, so that N is obtained T Target domain data set composed of individual electroencephalogram signal fragments wherein ,Segn Representing a target domain dataset D T An electroencephalogram signal of the nth section, and-> wherein ,/>Representing a target domain dataset D T N-th electroencephalogram fragment Seg n A small batch of samples at time step t +.>The b-th electroencephalogram signal sample in (a);
step 2, based onCNN network construction source model f and student model f s Teacher model f t ；
The CNN network comprises: first convolution module Conv3D and second convolution module Conv2D 1 Third convolution module Conv2D 2 A classification module;
the first convolution module Conv3D sequentially comprises: the first batch of normalized layers, step size (1, s) 1 ，s 1 ) And the convolution kernel is (c, k) 1 ，k 1 ) A first convolution layer, a first ReLU nonlinear activation function layer, and a convolution kernel of (1, d) 1 ，d 1 ) Is a maximum pooling layer of (2);
the second convolution module Conv2D 1 The method sequentially comprises the following steps: a second batch of normalization layers with step length s 2 And the convolution kernel is k 2 A second convolution layer, a second ReLU nonlinear activation function layer, and a convolution kernel of d 2 Is a maximum pooling layer of (2);
the third convolution module Conv2D 2 The method sequentially comprises the following steps: the third batch of normalization layers has a step length s 3 And the convolution kernel is k 3 A third convolution layer, a third ReLU nonlinear activation function layer, and a convolution kernel of d 3 Is a maximum pooling layer of (2);
the classification module comprises: two connection layers FC 1 and FC2 A first Sigmoid nonlinear activation function;
step 2.1, initializing the weights of all convolution kernels in the CNN network by using a kaiming initializer;
step 2.2, sampling the ith source domain brain electrical signalInputting the initial feature extraction and feature dimension reduction into the source model f through a first convolution module Conv3D in the source model f, and obtaining a first feature sequenceWherein h' represents the first characteristic sequence +.>Is of a height of (2);
step 2.3, the first feature sequenceSequentially passing through the second convolution module Conv2D 1 After treatment of (2) a second characteristic sequence is obtained>Wherein c' represents a second characteristic sequence +.>And w' represents the second characteristic sequenceIs the width of the second characteristic sequence +.>Is of a height of (2);
the second characteristic sequenceSequentially passing through a third convolution module Conv2D 2 After the processing of (a), outputting a third characteristic sequenceWherein h' "represents the third characteristic sequence +.>W "represents the third characteristic sequence +.>C "represents the third characteristic sequence +.>The number of channels;
step 2.4 for the third feature sequenceThe fourth characteristic sequence +.>
Fourth characteristic sequenceIs input into the classification module and passes through the first full connection layer FC 1 And processing the first Sigmoid nonlinear activation function to obtain an ith section of electroencephalogram signal sample +.>Fifth characteristic sequence of (2)Wherein a represents a fifth characteristic sequence +.>B represents a fifth characteristic sequence +.>Is a width of (2);
the fifth characteristic sequenceInputting the second full connection layer FC 2 The (1) processing is carried out to obtain a source model f for the ith section of electroencephalogram signal sample +.>Final Cs logit output value +.> wherein ,representation of source model f prediction of i-th segment electroencephalogram signal sample +.>Logit values belonging to class j, b' representing +.>And b' < b;
step 2.5, establishing a counter-propagating loss function L using the rebalancing-like loss function of formula (1) CE ：
In the formula (1), L i,j Represents the ith electroencephalogram signal sample in the jth electroencephalogram signalAnd is obtained from formula (2):
in the formula (2), the amino acid sequence of the compound,represents the i-th electroencephalogram signal sample +.>Probability values belonging to the j-th class and obtained by the formula (3):
step 2.6, based on the source training set D S Training the source model f by using an Adam optimizer, and calculating a loss function L CE When the training iteration number reaches the set number or the loss error is smaller than the set threshold, the training is stopped, so that the optimal source classification model f is obtained 0 ；
Step 3, initializing n=1;
step 4, initializing t=1;
step 4.0, utilizing the optimal source classification model f 0 Respectively initializing chemical raw model f s And teacher model f t Obtaining a student model f at a time step t s t And teacher model f t t ；
Step 4.1, sample a small batchRespectively inputting teacher model f under time step t t t Sum student model f s t Respectively obtain teacher models f t t Output Cs logic values +.>Student model f s t Output Cs logic values wherein ,/>Representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->A logic value belonging to the j-th class; />Representing student model f t s Predicting a small sample +.>B-th electroencephalogram signal sample->A logic value belonging to the j-th class;
step 4.1 calculating loss unsupervised loss L using equation (4) hybrid ：
L hybrid ＝L ent +δL div +γL sce (4)
In the formula (4), delta and gamma both represent regularization coefficients, L ent Represents the test entropy and is obtained from formula (5), L div Represents KL divergence and is obtained from formula (6), L sce Represents symmetrical cross entropy and is obtained by the formula (6);
in the formula ,representing student model f s t Predicting a small sample +.>B-th electroencephalogram signal sample->Probability values belonging to the j-th class and obtained by the formula (6):
in the formula ,representing teacher model f t t Predicting a small sample +.>B-th electroencephalogram signal sample->Probability values belonging to the j-th class and obtained by equation (8):
step 4.2 is based onStudent model at time step t using Adam optimizer +.>Training and minimizing an unsupervised loss function L hybrid When student model->After one epoch, a student model f at a time step t+1 is obtained s t+1 The method comprises the steps of carrying out a first treatment on the surface of the Meanwhile, the teacher model f is used for the teacher model f by the method (7) t t Parameter +.>Updating to obtain a teacher model f t t Parameter +.about.1 at time step t+1>
In the formula (7), the amino acid sequence of the compound,representing student model f s t+1 Parameters at time step t+1; alpha is a slip factor and has:
in the formula (8), the amino acid sequence of the compound, and />Respectively refer to student models->For a small batch of samples at time step t +.>The most self-contained one obtained when the prediction is performedAverage prediction probabilities of confidence and secondary confidence, and:
in the formula (9), the amino acid sequence of the compound, and />Representing student model->Next sample of small lot +.>Sample b of (b)The most and less confidence prediction probabilities;
step 4.3 student model f at time step t+1 s t+1 For a small batch of samplesClassifying to obtain a classification result;
step 4.4 after assigning t+1 to t, if t>T represents Seg n The samples in (a) are all classified, the step 4.5 is executed, otherwise, the step 4.0 is returned to be executed in sequence;
step 4.5 after n+1 is assigned to n, if n>N T Then the target domain dataset is representedAll the samples in the model are classified; otherwise, returning to the execution step 4 for sequential execution.
2. An electronic device comprising a memory and a processor, wherein the memory is configured to store a program that supports the processor to perform the method of cross-subject electroencephalogram classification of claim 1, the processor being configured to execute the program stored in the memory.
3. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor performs the steps of the inter-subject electroencephalogram classification method of claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310719543.8A CN116746947A (en) | 2023-06-16 | 2023-06-16 | Cross-subject electroencephalogram signal classification method based on online test time domain adaptation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310719543.8A CN116746947A (en) | 2023-06-16 | 2023-06-16 | Cross-subject electroencephalogram signal classification method based on online test time domain adaptation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116746947A true CN116746947A (en) | 2023-09-15 |
Family
ID=87954797
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310719543.8A Pending CN116746947A (en) | 2023-06-16 | 2023-06-16 | Cross-subject electroencephalogram signal classification method based on online test time domain adaptation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116746947A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390542A (en) * | 2023-10-13 | 2024-01-12 | 上海韶脑传感技术有限公司 | Domain generalization-based cross-test motor imagery electroencephalogram modeling method |
-
2023
- 2023-06-16 CN CN202310719543.8A patent/CN116746947A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390542A (en) * | 2023-10-13 | 2024-01-12 | 上海韶脑传感技术有限公司 | Domain generalization-based cross-test motor imagery electroencephalogram modeling method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Abdelhameed et al. | A deep learning approach for automatic seizure detection in children with epilepsy | |
| Tuncer et al. | Classification of epileptic seizures from electroencephalogram (EEG) data using bidirectional short-term memory (Bi-LSTM) network architecture | |
| CN112766355B (en) | Electroencephalogram signal emotion recognition method under label noise | |
| CN114176607B (en) | Electroencephalogram signal classification method based on vision transducer | |
| Qiao et al. | Ternary-task convolutional bidirectional neural turing machine for assessment of EEG-based cognitive workload | |
| WO2023059746A1 (en) | Estimating uncertainty in predictions generated by machine learning models | |
| Singhal et al. | Detection of alcoholism using EEG signals and a CNN-LSTM-ATTN network | |
| CN112932501B (en) | Method for automatically identifying insomnia based on one-dimensional convolutional neural network | |
| CN116746947A (en) | Cross-subject electroencephalogram signal classification method based on online test time domain adaptation | |
| CN114595725B (en) | Electroencephalogram signal classification method based on addition network and supervised contrast learning | |
| Zeynali et al. | Classification of EEG signals using Transformer based deep learning and ensemble models | |
| CN115422973A (en) | Electroencephalogram emotion recognition method of space-time network based on attention | |
| CN116671919B (en) | Emotion detection reminding method based on wearable equipment | |
| Liu et al. | Automated Machine Learning for Epileptic Seizure Detection Based on EEG Signals. | |
| Williams | Deep learning and transfer learning in the classification of EEG signals | |
| CN115017960B (en) | Electroencephalogram signal classification method based on space-time combined MLP network and application | |
| CN114841216B (en) | Electroencephalogram signal classification method based on model uncertainty learning | |
| Satyender et al. | ESIMD: epileptic seizure identification using metaheuristic deep learning technique | |
| Al-hajjar et al. | Epileptic seizure detection using feature importance and ML classifiers | |
| CN114091530A (en) | Electrocardiogram classification method and domain-adaptive-based electrocardiogram classification model training method | |
| Chang et al. | A semi-supervised classification RBM with an improved fMRI representation algorithm | |
| Mardi et al. | Temporal Convolutional Learning: A New Sequence-based Structure to Promote the Performance of Convolutional Neural Networks in Recognizing P300 Signals | |
| Sinha et al. | Automatic epileptic signal classification using deep convolutional neural network | |
| Rosenman et al. | Pre-training transformers for fingerprinting to improve stress prediction in fmri | |
| Omar | EEG-Based Autism Detection Using Multi-Input 1D Convolutional Neural Networks. |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |