CN117113015A - Electroencephalogram signal identification method and device based on space-time deep learning - Google Patents
Electroencephalogram signal identification method and device based on space-time deep learning Download PDFInfo
- Publication number
- CN117113015A CN117113015A CN202311137978.8A CN202311137978A CN117113015A CN 117113015 A CN117113015 A CN 117113015A CN 202311137978 A CN202311137978 A CN 202311137978A CN 117113015 A CN117113015 A CN 117113015A
- Authority
- CN
- China
- Prior art keywords
- deep learning
- space
- manual
- dimensional
- time deep
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013135 deep learning Methods 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 31
- 210000004556 brain Anatomy 0.000 claims abstract description 65
- 239000011159 matrix material Substances 0.000 claims abstract description 49
- 238000013136 deep learning model Methods 0.000 claims abstract description 38
- 238000012549 training Methods 0.000 claims abstract description 14
- 238000007781 pre-processing Methods 0.000 claims abstract description 9
- 239000013598 vector Substances 0.000 claims description 34
- 238000010606 normalization Methods 0.000 claims description 22
- 238000004364 calculation method Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 14
- 238000004422 calculation algorithm Methods 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 6
- 238000012880 independent component analysis Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 5
- 238000012545 processing Methods 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000003475 lamination Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims 2
- 230000004913 activation Effects 0.000 description 7
- 230000009467 reduction Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000005070 sampling Methods 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 208000027089 Parkinsonian disease Diseases 0.000 description 2
- 206010034010 Parkinsonism Diseases 0.000 description 2
- 206010015037 epilepsy Diseases 0.000 description 2
- 210000002569 neuron Anatomy 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002570 electrooculography Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 210000001152 parietal lobe Anatomy 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
- G06F18/15—Statistical pre-processing, e.g. techniques for normalisation or restoring missing data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/047—Probabilistic or stochastic networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Probability & Statistics with Applications (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The application discloses an electroencephalogram signal identification method and device based on space-time deep learning, comprising the steps of preprocessing an acquired original electroencephalogram signal to obtain a preprocessed electroencephalogram signal; calculating manual characteristics of the preprocessed brain electrical signals and combining the spatial distribution of brain electrodes of the original brain electrical signals to establish a three-dimensional manual characteristic matrix; constructing a space-time deep learning model, taking the preprocessed electroencephalogram signals and the three-dimensional manual feature matrix as the input of the space-time deep learning model, and training the space-time deep learning model; and finally, inputting the electroencephalogram signals into a trained space-time deep learning model to obtain the recognition result of the electroencephalogram signals. Compared with the prior art, the brain electrode spatial distribution and manual characteristics of the brain electrical signals are combined, precious information contained in the brain electrical signals is more comprehensively mined, priori knowledge is more fully combined, and therefore the accuracy of brain electrical signal identification is remarkably improved.
Description
Technical Field
The application belongs to the field of electroencephalogram signal processing, and particularly relates to an electroencephalogram signal identification method and device based on space-time deep learning.
Background
The electroencephalogram signal acquisition cost is low, contains rich information, and has unique value in scientific research and disease diagnosis. However, the electroencephalogram signal is a non-stationary signal with high randomness, has extremely weak amplitude, and is easily interfered by other physiological signals such as electrocardiograms, electrooculography and the like, so that the electroencephalogram signal is covered by artifacts. In addition, because the electroencephalogram signals are collected in real time, the signal time is long, and a large number of experts are difficult to configure in a hospital scene at present to read the real-time electroencephalogram signals with extremely high rhythms.
In recent years, there have been some researches on recognition of electroencephalogram signals by artificial intelligence means such as deep learning, for example, CN2023104777954, which is an electroencephalogram monitoring system and method for neurology, CN2021116144703, which is an electroencephalogram recognition method based on a time-channel cascade transform network, and CN2021100488180, which is an electroencephalogram recognition method and system based on a graph rolling and gating circulation unit. However, the existing researches including the patent application above ignore the spatial information of brain electrode distribution related to the brain electrical signal, and the time sequence characteristics of the brain electrical signal are not combined with the traditional manual characteristics, and high-dimensional characteristics and priori knowledge are not fully considered, so that ideal performance is not achieved.
Disclosure of Invention
The application aims to solve the problems in the background technology and provides an electroencephalogram signal identification method and device based on space-time deep learning.
In order to achieve the above purpose, the technical scheme adopted by the application is as follows:
the application provides an electroencephalogram signal identification method based on space-time deep learning, which comprises the following steps:
preprocessing the acquired original brain electrical signals to obtain preprocessed brain electrical signals;
calculating manual characteristics of the preprocessed brain electrical signals and combining the spatial distribution of brain electrodes of the original brain electrical signals to establish a three-dimensional manual characteristic matrix;
constructing a space-time deep learning model, taking the preprocessed electroencephalogram signals and the three-dimensional manual feature matrix as the input of the space-time deep learning model, and training the space-time deep learning model;
and finally, inputting the electroencephalogram signals into a trained space-time deep learning model to obtain the recognition result of the electroencephalogram signals.
Preferably, preprocessing the acquired original electroencephalogram signal to obtain a preprocessed electroencephalogram signal, including:
and sequentially processing the original electroencephalogram signal through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed electroencephalogram signal.
Preferably, calculating the manual feature of the preprocessed brain electrical signal and combining the spatial distribution of brain electrodes of the original brain electrical signal to establish a three-dimensional manual feature matrix, including:
respectively calculating the manual characteristics of the time domain, the manual characteristics of the frequency domain and the manual characteristics of the nonlinear entropy of the preprocessed electroencephalogram signals;
calculating the manual characteristics of the time domain, wherein the manual characteristics comprise respectively calculating an average value, a variance, a standard deviation, a normalized first-order difference and a normalized second-order difference of the preprocessed electroencephalogram signals to obtain corresponding manual characteristics;
calculating the manual characteristics of the frequency domain, wherein the manual characteristics comprise respectively calculating the band power of delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band for the preprocessed electroencephalogram signal to obtain corresponding manual characteristics;
calculating the manual characteristics of the nonlinear entropy, wherein the manual characteristics comprise respectively calculating the approximate entropy, the sample entropy, the Raney entropy, the Tsallis entropy and the differential entropy of the preprocessed electroencephalogram signal to obtain the corresponding manual characteristics;
according to the spatial distribution of brain electrodes related to the original brain signals, a two-dimensional matrix is constructed, and a three-dimensional manual characteristic matrix is obtained by combining the manual characteristic lamination obtained through calculation.
Preferably, a space-time deep learning model is constructed, the preprocessed electroencephalogram signals and the three-dimensional manual feature matrix are used as input of the space-time deep learning model, and the space-time deep learning model is trained, and the method comprises the following steps:
the space-time deep learning module comprises a preprocessed electroencephalogram signal branch, a three-dimensional manual feature matrix branch, a full-connection layer and a Softmax layer;
the preprocessed electroencephalogram signal branches adopt a plurality of convolution layers to extract high-dimensional characteristics of the input preprocessed electroencephalogram signal, the extracted high-dimensional characteristics are leveled into one-dimensional vectors, then input embedding and position coding are carried out, the data after the position coding is completed is extracted by adopting a preset number of coding blocks, and each coding block comprises a multi-head attention layer, a first normalization layer, a forward feedback layer and a second normalization layer;
the three-dimensional manual feature matrix branches adopt a preset number of convolution blocks and a maximum pooling layer to carry out high-dimensional feature extraction on the input three-dimensional manual feature matrix, and each convolution block comprises a convolution layer, a batch normalization layer and a random discarding layer;
respectively flattening the output of each branch into one-dimensional vectors, and combining and connecting the two one-dimensional vectors into one-dimensional vector;
finally, sequentially inputting the one-dimensional vectors after combination and connection into a full connection layer and a Softmax layer to obtain a prediction result;
and calculating a loss function, and performing back propagation to complete training of the space-time deep learning model.
Preferably, a binary cross entropy function is used as a Loss function of the space-time deep learning model, and a Loss function Loss calculation formula is as follows:
wherein N' represents the number of samples of the training space-time deep learning module, y u Binary label representing the u-th sample, p (y u ) Representing that the output of the space-time deep learning module belongs to the binary label y u Is a probability of (2).
The application also provides an electroencephalogram signal recognition device based on space-time deep learning, which comprises a processor and a memory storing a plurality of computer instructions, wherein the computer instructions realize the steps of the electroencephalogram signal recognition method based on space-time deep learning when being executed by the processor.
Compared with the prior art, the application has the beneficial effects that:
compared with the prior art, the brain electrode spatial distribution and manual characteristics of the brain electrical signals are combined, precious information contained in the brain electrical signals is more comprehensively mined, priori knowledge is more fully combined, and therefore the accuracy of brain electrical signal identification is remarkably improved.
Drawings
FIG. 1 is a block diagram of an electroencephalogram signal recognition method and device based on space-time deep learning;
FIG. 2 is a schematic representation of a three-dimensional manual feature matrix of the present application;
FIG. 3 is a schematic diagram of a space-time deep learning model according to the present application;
FIG. 4 is a schematic view of a combination of different electrodes according to the present application;
fig. 5 is a class activation diagram of different electrode reduction schemes of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It will be understood that when an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein in the description of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.
In one embodiment, as shown in fig. 1-3, an electroencephalogram signal identification method based on space-time deep learning includes:
step 1, preprocessing the acquired original brain electrical signals to obtain preprocessed brain electrical signals.
Specifically, preprocessing the acquired original electroencephalogram signals to obtain preprocessed electroencephalogram signals, wherein the preprocessing comprises the following steps:
and sequentially processing the original electroencephalogram signal through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed electroencephalogram signal.
In this embodiment, the object of the acquired original electroencephalogram signal is not limited. Filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation, the following specific operations are adopted:
filtering high-frequency noise and baseline drift of the original electroencephalogram signal by adopting a band-pass filter of 1-60 Hz;
filtering the influence of power frequency interference by adopting a 50Hz notch filter;
downsampling the original 1kHz signal to 250Hz;
removing the components of the electro-oculogram signal and the electromyogram signal contained in the original electroencephalogram signal by adopting an Independent Component Analysis (ICA) algorithm;
the signal is normalized by adopting a Min-Max method, and the specific calculation formula is as follows:
wherein b i Is the ith element of the sequence B of the amplitude of the electroencephalogram, min { B } represents the minimum value of the sequence B, and max { B } represents the maximum value of the sequence B;
the normalized signal is divided by sampling sliding windows, the length of each sliding window is 5 seconds, and the length of each section of signal is 1250 sampling points after the sampling rate of 250Hz is combined.
And 2, calculating manual characteristics of the preprocessed brain electrical signals and establishing a three-dimensional manual characteristic matrix by combining the spatial distribution of brain electrodes of the original brain electrical signals.
Specifically, the method for calculating the manual characteristics of the preprocessed brain electrical signals and combining the spatial distribution of brain electrodes of the original brain electrical signals to establish a three-dimensional manual characteristic matrix comprises the following steps:
respectively calculating the manual characteristics of the time domain, the manual characteristics of the frequency domain and the manual characteristics of the nonlinear entropy of the preprocessed electroencephalogram signals;
calculating the manual characteristics of the time domain, wherein the manual characteristics comprise respectively calculating an average value, a variance, a standard deviation, a normalized first-order difference and a normalized second-order difference of the preprocessed electroencephalogram signals to obtain corresponding manual characteristics;
calculating the manual characteristics of the frequency domain, wherein the manual characteristics comprise respectively calculating the band power of delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band for the preprocessed electroencephalogram signal to obtain corresponding manual characteristics;
calculating the manual characteristics of the nonlinear entropy, wherein the manual characteristics comprise respectively calculating the approximate entropy, the sample entropy, the Raney entropy, the Tsallis entropy and the differential entropy of the preprocessed electroencephalogram signal to obtain the corresponding manual characteristics;
according to the spatial distribution of brain electrodes related to the original brain signals, a two-dimensional matrix is constructed, and a three-dimensional manual characteristic matrix is obtained by combining the manual characteristic lamination obtained through calculation.
The normalized first order difference δ in the manual feature of the calculation time domain is calculated ξ And normalized second order differenceThe calculation formula of (2) is as follows:
wherein xi represents a standard deviation sigma ξ And a time series of length T', T representing the time instant T of the time series.
In this embodiment, when the manual characteristics of the frequency domain are calculated, the band powers of the delta band (1-4 Hz), the theta band (4-8 Hz), the alpha band (8-12 Hz), the beta band (12-30 Hz) and the gamma band (30-45 Hz) are calculated.
In this embodiment, when the manual feature of the nonlinear entropy is calculated, the approximate entropy ApEn is used to measure the randomness of the time sequence, and the specific calculation formula is as follows:
where n=n-m+1, N represents the number of data points in the time series, m is the embedding dimension,the probability of approximation at the embedding dimension m and the similarity tolerance r is expressed, and 1.ltoreq.i.ltoreq.n.
The sample entropy is expanded on the basis of the approximate entropy, and the randomness and regularity can be measured.
The Raney entropy RenyiEn is an expansion of shannon entropy, and a specific calculation formula is as follows:
wherein α represents an order and is a non-negative real number not equal to 1, p i The probability is calculated by the energy of the original electroencephalogram signal, and i is more than or equal to 1 and less than or equal to n.
The specific calculation formula of the Tsallis entropy TsEn aiming at the complexity of a non-superposition system is as follows:
the differential entropy DiffEn is also expanded by shannon entropy, and the specific calculation formula is as follows:
in this embodiment, since the brain electrodes collect the activities of cortex in different regions of the brain during the electroencephalogram signal acquisition process, it is necessary to fully consider the spatial position distribution of the brain electrodes, the spatial distribution of the brain electrodes is three-dimensional distribution, firstly, the three-dimensional distribution of the brain electrodes is mapped to a two-dimensional plane to construct a two-dimensional matrix, such as a 7*7 two-dimensional matrix, and a three-dimensional manual feature matrix of 7×7×15 is constructed by combining 15 manual features (such as an average value, a variance, a standard deviation, a normalized first-order difference, a normalized second-order difference, a delta frequency band, a theta frequency band, an alpha frequency band, a beta frequency band, a gamma frequency band, an approximate entropy, a sample entropy, a Raney entropy, a Tsallis entropy and a manual feature of differential entropy), as shown in fig. 2.
And 3, constructing a space-time deep learning model, taking the preprocessed electroencephalogram signals and the three-dimensional manual feature matrix as input of the space-time deep learning model, and training the space-time deep learning model.
Specifically, the space-time deep learning module comprises a preprocessed electroencephalogram signal branch, a three-dimensional manual feature matrix branch, a full-connection layer and a Softmax layer;
the preprocessed electroencephalogram signal branches adopt a plurality of convolution layers to extract high-dimensional characteristics of the input preprocessed electroencephalogram signal, the extracted high-dimensional characteristics are leveled into one-dimensional vectors, then input embedding and position coding are carried out, the data after the position coding is completed is extracted by adopting a preset number of coding blocks, and each coding block comprises a multi-head attention layer, a first normalization layer, a forward feedback layer and a second normalization layer;
the three-dimensional manual feature matrix branches adopt a preset number of convolution blocks and a maximum pooling layer to carry out high-dimensional feature extraction on the input three-dimensional manual feature matrix, and each convolution block comprises a convolution layer, a batch normalization layer and a random discarding layer;
respectively flattening the output of each branch into one-dimensional vectors, and combining and connecting the two one-dimensional vectors into one-dimensional vector;
finally, sequentially inputting the one-dimensional vectors after combination and connection into a full connection layer and a Softmax layer to obtain a prediction result;
and calculating a loss function, and performing back propagation to complete training of the space-time deep learning model.
The parameters of the spatiotemporal deep learning model are shown in table 1:
TABLE 1
It should be noted that, the brain electrical signal after preprocessing branches: in this embodiment, the size of the preprocessed brain electrical signal input to the preprocessed brain electrical signal branch is 21×1250×1 (e.g. input_2 in table 1), and six convolution layers with identical parameters are adopted (e.g. two-dimensional convolution_5, two-dimensional convolution_6, two-dimensional convolution_7, two-dimensional convolution_8, two-dimensional convolution_9 and two-dimensional convolution_10 in table 1, and the dimensions output by the six convolution blocks are 21×618×32, 21×302×32, 21×144×16, 21×65×16, 21×25×8 and 21×5×4 in sequence), so that the high-dimensional feature extraction is performed on the input signal, and the convolution layers are composed of a plurality of convolution kernels, wherein the size of each convolution kernel is 1×16, and the step size is 1*2; after the extracted high-dimensional features are leveled into one-dimensional vectors (such as leveling_2 in table 1 and leveling dimension is 420), input embedding and position coding are performed, the extracted high-dimensional features are defined as X, and the vectors after input embedding are expressed as X embedding (as embedded in table 1, the output dimension is 420×32), and the position coding is performed by using a sine-cosine function (the position coding originates from the natural language processing field, and different arrangements of words in the sentence can lead to different meanings of the sentence, and the electroencephalogram signal is similar to the text information, and also emphasizes the time sequence), and the specific calculation formula is as follows:
wherein,PE (pos,2q) 、PE (pos,2q+1) the positions pos of the data points representing the current time sequence, the corresponding encodings in the even case (2 q) and the odd case (2q+1), d model Representing the input embedding dimension of the encoded vector, the output after input embedding and position encoding is represented as X embedded ＝X embedding +X pos And X is pos Is (PE) (pos,2q) ，PE (pos,2q+1) )。
In this embodiment, four coding blocks (for example, coding block_1, coding block_2, coding block_3, and coding block_4 in table 1, and the dimensions output by the four coding blocks are 420×32) with identical structures are used to extract features from the data after position coding, where each coding block includes a multi-head attention layer, a first normalization layer, a forward feedback layer, and a second normalization layer:
the multi-head attention layer needs to complete the data X after the position coding embedded The input map is a query vector Q, a key value vector K and a value vector V, and the specific calculation formula is as follows:
Q＝X embedded ×W Q
K＝X embedded ×w K
V＝X embedded ×W V
wherein w is Q 、w K 、w V Respectively representing the weight matrix of the corresponding vector.
The attention information is calculated on the query vector, the key value vector and the value vector which are mapped by adopting a dot product attention mechanism with scaling, and the specific calculation formula is as follows:
Multihead(Q，K，V)＝Concat(H 1 ，H 2 ，...，H h )
H s ＝Attention(Q s ，K s ，V s )
wherein, multi head represents multi-head, concat represents merging connection operation, H represents the number of heads in multi-head attention, H s Representing the s-th head, Q s ，K s ，V s Respectively representing a query vector matrix, a key value vector matrix and a value vector matrix of the s-th head, T represents a matrix biasing operation, d k Representing balance parameters for preventing dimension from becoming too high.
After passing through the multi-head attention layer, the output X of a coding block is obtained by adopting a first normalization layer, a forward feedback layer and a second normalization layer output The specific calculation formula is as follows:
X add&norm1 ＝Layernorm(X embedded +Multihead(Q，K，V))
X ff ＝max(0，W 1 X add&norm1 +b 1 )w 2 +b 2
X output ＝Layernorm(X add&norm1 +X rr )
wherein X is add&norm1 Representing the output of the first normalization layer, layernorm representing the layer normalization operation (operation of the first normalization layer and the second normalization layer), X ff Representing the output of the feed-forward layer, w 1 And W is 2 Representing a weight matrix, b 1 And b 2 Representing the bias matrix.
Output X of coding block output And input X of coding block embedded The dimensions are consistent so that a preset number of code blocks can be directly connected end to end.
Three-dimensional manual feature matrix branching:
in this embodiment, the three-dimensional manual feature matrix branches perform high-dimensional feature extraction on the input three-dimensional manual feature matrix (for example, input_1 in table 1, with dimensions of 7×7×15) by using four convolution blocks (four convolution blocks, for example, two-dimensional convolution_1, two-dimensional convolution_2, two-dimensional convolution_3 and two-dimensional convolution_4 in table 1, and dimensions output by the four convolution blocks are 7×7×32, 7×7×64, 7×7×128 and 7×7×32 in sequence), where each convolution block includes a convolution layer, a batch normalization layer and a random discarding layer; the convolution kernel size of the convolution layer in the first convolution block is 1*1, and the discarding rate of the random discarding layer is 0.2; the convolution kernel size of the convolution layers in the second, third and fourth convolution blocks is 3*3, and the discarding rate of the random discarding layer is 0.2; and (3) performing dimension reduction operation on the output of the fourth convolution block by using a maximum pooling layer (such as two-dimensional maximum pooling in table 1, wherein the dimension of the output through the layer is 7×7×32), and the size of the pooling is 2×2.
Respectively flattening the outputs of all branches (the output of the preprocessed electroencephalogram branches and the output of the three-dimensional manual feature matrix branches) into one-dimensional vectors (the output of the three-dimensional manual feature matrix branches is flattened into one-dimensional vectors such as flattened_1 in table 1 and the flattened dimension is 1568; the output of the preprocessed electroencephalogram branches is flattened into one-dimensional vectors such as flattened_3 in table 1 and the flattened dimension is 13440), and combining and connecting the two one-dimensional vectors into one-dimensional vector (such as combination in table 1 and the output dimension is 15008); finally, the one-dimensional vectors after merging and connection are sequentially input into a full connection layer (such as full connection_1 in table 1 and output dimension is 512) and a Softmax layer (such as full connection_2 in table 1 and output dimension is 2) to obtain a prediction result:
in this example, the fully connected layer has 512 neurons and the Softmax layer has 512 neurons.
In the training process of the space-time deep learning model, the accuracy of the model is improved by reasonably setting training super parameters of the model, and the method comprises the following specific steps:
adopting an Adam algorithm as an optimizer, wherein the learning rate is 0.01, and the weight attenuation is 1e-6;
the maximum iteration round is 100;
batch size 64;
the binary cross entropy function is used as a Loss function of the space-time deep learning model, and the Loss function Loss has the following calculation formula:
wherein N' represents the number of samples of the training space-time deep learning module, y u Binary label representing the u-th sample, p (y u ) Representing that the output of the space-time deep learning module belongs to the binary label y u Is a probability of (2).
And step 4, finally, inputting the electroencephalogram signals into a trained space-time deep learning model to obtain an electroencephalogram signal recognition result.
It should be noted that the method is applicable to various situations, such as recognition of pain, for example, whether the recognition result of the electroencephalogram signal is pain (i.e. pain or no pain); if the result of the recognition is that symptoms such as depression, epilepsy, parkinsonism and the like occur in the scenes such as depression, epilepsy, parkinsonism and the like.
Compared with the prior art, the brain electrode spatial distribution and manual characteristics of the brain electrical signals are combined, precious information contained in the brain electrical signals is more comprehensively mined, priori knowledge is more fully combined, and therefore the accuracy of brain electrical signal identification is remarkably improved.
In one embodiment, an electroencephalogram signal recognition device based on space-time deep learning is further provided, and the electroencephalogram signal recognition device comprises a processor and a memory storing a plurality of computer instructions, wherein the computer instructions realize the steps of the electroencephalogram signal recognition method based on space-time deep learning when being executed by the processor.
Note that, for specific limitation of the electroencephalogram signal recognition apparatus based on the space-time deep learning, reference may be made to the limitation of the electroencephalogram signal recognition method based on the space-time deep learning hereinabove, and the description thereof will not be repeated here.
Evaluating a space-time deep learning model:
and adopting cross-person ten-fold cross-validation to evaluate the constructed space-time deep learning model. Randomly disturbing the sequence of the subjects and equally dividing the subjects into ten groups according to the number of people, wherein eight groups are used as training sets of the space-time deep learning model, one group is used as verification sets of the space-time deep learning model, and the other group is used as test sets of the space-time deep learning model; and (3) storing the model with the best performance on the verification set in the training process, and testing the model on the test set and outputting the accuracy of the electroencephalogram signal identification.
To demonstrate the effectiveness and innovativeness of the present application, table 2 shows:
TABLE 2
The accuracy and F1 score of the application and other existing common machine learning algorithms are compared aiming at the preprocessed electroencephalogram branches and the three-dimensional manual feature matrix branches, the preprocessed electroencephalogram branches (adopting multi-head attention) are compared with the traditional one-dimensional convolution neural network, the three-dimensional manual feature matrix branches (adopting two-dimensional convolution) are compared with the existing support vector machine, random forest and K nearest neighbor algorithm, the preprocessed electroencephalogram branches and the three-dimensional manual feature matrix branches are combined (adopting multi-head attention + two-dimensional convolution) and compared with the existing one-dimensional convolution + two-dimensional convolution, and the accuracy and F1 score of the application are both high, so that the rationality of the application is embodied.
In one embodiment, two electrode reduction schemes based on the electroencephalogram pain recognition method are as follows:
in this embodiment, gradient weighted class-activated heat maps (Grad-CAM) are used to measure the contribution of each electroencephalogram channel to pain recognition; the class activation heat map with gradient weighting is calculated mainly by using the feature map output by the last convolution block (namely the fourth convolution block, two-dimensional convolution_4 in table 1) in the three-dimensional manual feature matrix branch, because the feature map contains the most representative high-dimensional feature, and the specific calculation formula is as follows:
wherein A represents the feature map output by the last convolution block in the three-dimensional manual feature matrix branch, k represents the kth channel of the feature map A, c represents the category,representing weights, reLU representing a nonlinear activation function,>representing global average pooling operations,/->Representing the calculation of the gradient by back propagation.
The method comprises the steps of carrying out region division on 21 brain electrodes involved in acquisition of original brain signals, wherein the division situation is shown in fig. 4; based on the division, the contribution degree of different areas is evaluated in a targeted manner by using the gradient weighted class activation heat map, and 15 electrode combinations, 9 electrode combinations, 6 electrode combinations and 6 electrode combinations are mainly considered. The class activation heat map obtained under different electrode combinations is shown in fig. 5, wherein fig. 5 (a) shows the contribution degree of the 21-channel electroencephalogram signals; a total of 15 electrode contributions from frontal, temporal, medial and parietal lobes in fig. 5 (a) are relatively higher, these 15 channels can be used as the first electrode reduction scheme, for which class activation heat maps are calculated, as shown in fig. 5 (b); combining fig. 5 (a) and fig. 5 (b), the contribution of the left half brain part of the frontal, temporal, middle and top lobe areas is relatively higher, so the 9 electrodes are used as the second electrode reduction scheme, and the activation heat map is calculated for this type, and the result is shown in fig. 5 (c); fig. 5 (d), 5 (e) and 5 (f) show the respective electrode contributions in the case of 9 electrode combinations, 6 electrode combinations.
To quantitatively compare the automatic pain recognition accuracy of children under different electrode reduction schemes, the following table 3 shows:
TABLE 3 Table 3
Aiming at five different electrode reduction combination forms, the recognition accuracy and F1 score of the constructed space-time deep learning model (21 electrode) are compared, and the result shows that the accuracy and F1 score of the space-time deep learning model (21 electrode) are the highest, so that the advancement and effectiveness of the application are embodied.
The technical features of the above-described embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above-described embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above-described embodiments represent only the more specific and detailed embodiments of the present application, but are not to be construed as limiting the claims. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of protection of the present application is to be determined by the appended claims.
Claims (6)
1. An electroencephalogram signal identification method based on space-time deep learning is characterized by comprising the following steps of: the electroencephalogram signal identification method based on space-time deep learning comprises the following steps:
preprocessing the acquired original brain electrical signals to obtain preprocessed brain electrical signals;
calculating manual characteristics of the preprocessed brain electrical signals and combining the spatial distribution of brain electrodes of the original brain electrical signals to establish a three-dimensional manual characteristic matrix;
constructing a space-time deep learning model, taking the preprocessed electroencephalogram signals and the three-dimensional manual feature matrix as the input of the space-time deep learning model, and training the space-time deep learning model;
and finally, inputting the electroencephalogram signals into a trained space-time deep learning model to obtain the recognition result of the electroencephalogram signals.
2. The brain electrical signal recognition method based on space-time deep learning according to claim 1, wherein: the preprocessing of the collected original brain electrical signals to obtain preprocessed brain electrical signals comprises the following steps:
and sequentially processing the original electroencephalogram signal through filtering, downsampling, independent component analysis algorithm, normalization and sliding window segmentation to obtain the preprocessed electroencephalogram signal.
3. The brain electrical signal recognition method based on space-time deep learning according to claim 1, wherein: the step of calculating the manual characteristics of the preprocessed brain electrical signals and combining the spatial distribution of brain electrodes of the original brain electrical signals to establish a three-dimensional manual characteristic matrix comprises the following steps:
respectively calculating the manual characteristics of the time domain, the manual characteristics of the frequency domain and the manual characteristics of the nonlinear entropy of the preprocessed electroencephalogram signals;
calculating the manual characteristics of the time domain, wherein the manual characteristics comprise respectively calculating an average value, a variance, a standard deviation, a normalized first-order difference and a normalized second-order difference of the preprocessed electroencephalogram signals to obtain corresponding manual characteristics;
calculating the manual characteristics of the frequency domain, wherein the manual characteristics comprise respectively calculating the band power of delta frequency band, theta frequency band, alpha frequency band, beta frequency band and gamma frequency band for the preprocessed electroencephalogram signal to obtain corresponding manual characteristics;
calculating the manual characteristics of the nonlinear entropy, wherein the manual characteristics comprise respectively calculating the approximate entropy, the sample entropy, the Raney entropy, the Tsallis entropy and the differential entropy of the preprocessed electroencephalogram signal to obtain the corresponding manual characteristics;
according to the spatial distribution of brain electrodes related to the original brain signals, a two-dimensional matrix is constructed, and a three-dimensional manual characteristic matrix is obtained by combining the manual characteristic lamination obtained through calculation.
4. The brain electrical signal recognition method based on space-time deep learning according to claim 1, wherein: the construction of a space-time deep learning model takes the preprocessed electroencephalogram signals and a three-dimensional manual feature matrix as the input of the space-time deep learning model, and trains the space-time deep learning model, and the construction comprises the following steps:
the space-time deep learning module comprises a preprocessed electroencephalogram signal branch, a three-dimensional manual feature matrix branch, a full-connection layer and a Softmax layer;
the preprocessed electroencephalogram signal branch adopts a plurality of convolution layers to extract high-dimensional characteristics of the input preprocessed electroencephalogram signal, the extracted high-dimensional characteristics are flattened into one-dimensional vectors, then input embedding and position coding are carried out, the data after the position coding is completed is extracted by adopting a preset number of coding blocks, and each coding block comprises a multi-head attention layer, a first normalization layer, a forward feedback layer and a second normalization layer;
the three-dimensional manual feature matrix branches adopt a preset number of convolution blocks and a maximum pooling layer to carry out high-dimensional feature extraction on the input three-dimensional manual feature matrix, and each convolution block comprises a convolution layer, a batch normalization layer and a random discarding layer;
respectively flattening the output of each branch into one-dimensional vectors, and combining and connecting the two one-dimensional vectors into one-dimensional vector;
finally, sequentially inputting the one-dimensional vectors after merging and connecting into a full-connection layer and a Softmax layer to obtain a prediction result;
and calculating a loss function, and performing back propagation to complete training of the space-time deep learning model.
5. The brain electrical signal recognition method based on space-time deep learning according to claim 4, wherein: the binary cross entropy function is adopted as a Loss function of the space-time deep learning model, and the Loss function Loss has the following calculation formula:
wherein N' represents the number of samples of the training space-time deep learning module, y u Binary label representing the u-th sample, p (y u ) Representing that the output of the space-time deep learning module belongs to the binary label y u Is a probability of (2).
6. An electroencephalogram signal recognition device based on space-time deep learning comprises a processor and a memory storing a plurality of computer instructions, and is characterized in that: the computer instructions, when executed by a processor, implement the steps of the method of any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311137978.8A CN117113015A (en) | 2023-09-05 | 2023-09-05 | Electroencephalogram signal identification method and device based on space-time deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311137978.8A CN117113015A (en) | 2023-09-05 | 2023-09-05 | Electroencephalogram signal identification method and device based on space-time deep learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117113015A true CN117113015A (en) | 2023-11-24 |
Family
ID=88794650
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311137978.8A Pending CN117113015A (en) | 2023-09-05 | 2023-09-05 | Electroencephalogram signal identification method and device based on space-time deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117113015A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117473303A (en) * | 2023-12-27 | 2024-01-30 | 小舟科技有限公司 | Personalized dynamic intention feature extraction method and related device based on electroencephalogram signals |
-
2023
- 2023-09-05 CN CN202311137978.8A patent/CN117113015A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117473303A (en) * | 2023-12-27 | 2024-01-30 | 小舟科技有限公司 | Personalized dynamic intention feature extraction method and related device based on electroencephalogram signals |
| CN117473303B (en) * | 2023-12-27 | 2024-03-19 | 小舟科技有限公司 | Personalized dynamic intention feature extraction method and related device based on electroencephalogram signals |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Kant et al. | CWT based transfer learning for motor imagery classification for brain computer interfaces | |
| CN108776788B (en) | Brain wave-based identification method | |
| Jang et al. | EEG-based video identification using graph signal modeling and graph convolutional neural network | |
| CN108256629B (en) | EEG signal unsupervised feature learning method based on convolutional network and self-coding | |
| CN111956221B (en) | Temporal lobe epilepsy classification method based on wavelet scattering factor and LSTM neural network model | |
| CN111407243A (en) | Pulse signal pressure identification method based on deep learning | |
| CN117113015A (en) | Electroencephalogram signal identification method and device based on space-time deep learning | |
| CN116304815A (en) | Motor imagery electroencephalogram signal classification method based on self-attention mechanism and parallel convolution | |
| CN113569997A (en) | Emotion classification method and system based on graph convolution neural network | |
| CN112244868A (en) | Epilepsia electroencephalogram signal classification method based on ANFIS | |
| Abbas et al. | FBCSP-based multi-class motor imagery classification using BP and TDP features | |
| CN113128384B (en) | Brain-computer interface software key technical method of cerebral apoplexy rehabilitation system based on deep learning | |
| CN110192864A (en) | A kind of cross-domain electrocardiogram biometric identity recognition methods | |
| Wang et al. | A shallow convolutional neural network for classifying MI-EEG | |
| Narayana et al. | ECG based biometric authentication system using deep learning methods | |
| CN116522106A (en) | Motor imagery electroencephalogram signal classification method based on transfer learning parallel multi-scale filter bank time domain convolution | |
| CN115270847A (en) | Design decision electroencephalogram recognition method based on wavelet packet decomposition and convolutional neural network | |
| CN110443276A (en) | Time series classification method based on depth convolutional network Yu the map analysis of gray scale recurrence | |
| Foumani et al. | Eeg2rep: enhancing self-supervised EEG representation through informative masked inputs | |
| CN114081492A (en) | Electroencephalogram emotion recognition system based on learnable adjacency matrix | |
| CN114287937A (en) | Emotion recognition method based on multi-mode convolutional neural network | |
| KR20220158462A (en) | EMG signal-based recognition information extraction system and EMG signal-based recognition information extraction method using the same | |
| Wang | A modified motor imagery classification method based on eegnet | |
| Drage et al. | A novel deep learning approach using AlexNet for the classification of electroencephalograms in Alzheimer's Disease and Mild Cognitive Impairment | |
| CN117643475B (en) | Feature extraction method based on KL divergence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |