CN116712074A - Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning - Google Patents
Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning Download PDFInfo
- Publication number
- CN116712074A CN116712074A CN202310656838.5A CN202310656838A CN116712074A CN 116712074 A CN116712074 A CN 116712074A CN 202310656838 A CN202310656838 A CN 202310656838A CN 116712074 A CN116712074 A CN 116712074A
- Authority
- CN
- China
- Prior art keywords
- cross
- frequency
- frequency coupling
- channel
- electroencephalogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000008878 coupling Effects 0.000 title claims abstract description 139
- 238000010168 coupling process Methods 0.000 title claims abstract description 139
- 238000005859 coupling reaction Methods 0.000 title claims abstract description 139
- 238000010586 diagram Methods 0.000 title claims abstract description 73
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 56
- 238000000034 method Methods 0.000 title claims abstract description 45
- 238000013135 deep learning Methods 0.000 title claims abstract description 29
- 210000004556 brain Anatomy 0.000 claims abstract description 67
- 230000006870 function Effects 0.000 claims abstract description 31
- 230000008451 emotion Effects 0.000 claims abstract description 22
- 238000012549 training Methods 0.000 claims abstract description 14
- 238000013528 artificial neural network Methods 0.000 claims abstract description 10
- 230000007246 mechanism Effects 0.000 claims abstract description 10
- 239000013598 vector Substances 0.000 claims description 33
- 238000007781 pre-processing Methods 0.000 claims description 14
- 238000013527 convolutional neural network Methods 0.000 claims description 13
- 230000007704 transition Effects 0.000 claims description 11
- 238000001914 filtration Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 210000004761 scalp Anatomy 0.000 claims description 8
- 230000021615 conjugation Effects 0.000 claims description 6
- 238000010606 normalization Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 5
- 238000000605 extraction Methods 0.000 claims description 4
- 230000004913 activation Effects 0.000 claims description 3
- 238000005259 measurement Methods 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 230000005540 biological transmission Effects 0.000 claims description 2
- 238000012880 independent component analysis Methods 0.000 claims description 2
- 230000003183 myoelectrical effect Effects 0.000 claims description 2
- 238000012800 visualization Methods 0.000 claims 2
- 238000012545 processing Methods 0.000 abstract description 2
- 230000000694 effects Effects 0.000 description 4
- 230000005611 electricity Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 238000012952 Resampling Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 230000007547 defect Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 210000004727 amygdala Anatomy 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000036992 cognitive tasks Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 230000000875 corresponding effect Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 238000013136 deep learning model Methods 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 230000006397 emotional response Effects 0.000 description 1
- 210000001061 forehead Anatomy 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000010355 oscillation Effects 0.000 description 1
- 230000006461 physiological response Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000008448 thought Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G06N3/0455—Auto-encoder networks; Encoder-decoder networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/048—Activation functions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/02—Preprocessing
- G06F2218/04—Denoising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Psychiatry (AREA)
- Pathology (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- Computing Systems (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychology (AREA)
- Fuzzy Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Physiology (AREA)
- Signal Processing (AREA)
- Child & Adolescent Psychology (AREA)
- Social Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Developmental Disabilities (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention relates to the technical field of signal processing and emotion recognition, and discloses an emotion recognition method and device based on an electroencephalogram cross-frequency coupling diagram and deep learning, wherein electroencephalogram signals of N channels are collected and preprocessed; selecting two channels of electroencephalogram signals, calculating cross-frequency phase amplitude coupling coefficients between sub-bands, and drawing two channels of cross-frequency coupling diagrams to obtain N multiplied by N two channels of cross-frequency coupling diagrams; determining an optimal ordering mode of N multiplied by N two-channel cross-frequency coupling graphs by utilizing a gradient iterative algorithm to minimize an objective function, and splicing the N multiplied by N two-channel cross-frequency coupling graphs into a whole brain channel cross-frequency coupling graph in a form of N rows and N columns; training a neural network based on an attention mechanism by using a full brain channel cross-frequency coupling diagram to form an emotion recognition model; and classifying by using the trained emotion recognition model to obtain an emotion recognition result. Compared with the prior art, the method and the device can mine potential characteristics in the whole brain cross-frequency coupling diagram under different emotion states, and effectively decode the emotion states of the user.
Description
Technical Field
The invention relates to the technical fields of signal processing and emotion recognition, in particular to an emotion recognition method and device based on an electroencephalogram cross-frequency coupling diagram and deep learning.
Background
Emotion is a comprehensive state of human feeling, thought and behavior, and emotion recognition refers to recognition of a corresponding emotional state through human behavior and physiological response. The dorsolateral forehead, amygdala, orbitum, etc. in the brain are the key locations responsible for emotion. The brain electricity is a physiological signal generated by the brain, is noninvasive, has high time resolution, is convenient to use and difficult to disguise, and is a better choice for identifying emotional response.
The emotion recognition based on the electroencephalogram needs to adopt a proper method to extract meaningful information from the electroencephalogram, and the prior art at least has the following defects:
1. the brain is a complex communication network, and is cooperatively operated by a plurality of brain regions in advanced cognitive tasks such as emotion memory decision making and the like. Most of the existing brain electricity emotion recognition methods extract time, frequency, time-frequency and other characteristics from single lead signals, cannot show information exchange activities among brain regions, and cannot reveal multi-frequency interactions of emotion brain electricity.
2. The traditional method of calculating the cross-frequency phase-amplitude coupling is to first bandpass filter to obtain the frequency band of interest and then Hilbert transform to extract the instantaneous amplitude and instantaneous phase. In the process of using band-pass filtering, due to the need of selecting band-pass filter parameters including filter order, transition band and the like, systematic deviation can occur. The use of hilbert transform to estimate the amplitude and phase depends on the narrowband assumption of the signal, i.e. the signal is close to a sinusoidal signal, whereas the signal in the band pass filtered electroencephalogram Beta, gamma band does not meet this precondition. In addition, we do not know a priori information of the most useful oscillation components that distinguish each user's different emotions, so the conventional bandpass filtering+hilbert transform method is unreliable.
3. Traditional machine learning methods for classifying images, such as support vector machines and decision trees, all require manual extraction of image features. At present, deep learning based on convolutional neural network has become a promising method for classifying and detecting images, which can automatically learn optimal features without artificially designing features. However, the local connection characteristic of the convolutional neural network makes the result of the image convolutional operation generally affected by the arrangement sequence of elements in the image. Likewise, the arrangement and the stitching sequence of the electroencephalogram cross-frequency coupling diagrams among different leads also influence the image classification effect of the convolutional neural network. Due to the volume conduction effect, the electroencephalogram signals obtained from adjacent brain regions tend to be similar, and the electroencephalogram cross-frequency coupling maps from physically adjacent electrodes can be arranged together to construct a smoother stitched image. This approach to ordering based on channel physical distance is somewhat reasonable, but not optimal, because channels that are spatially far from each other may be highly correlated.
4. The region where the key information appears in the image usually only occupies a small part of the whole image, and the traditional deep learning image identification method based on the convolutional neural network is relatively low in efficiency. Due to the limitation of convolution kernel size, convolutional neural networks lack sensitivity to global information of input data, and it is difficult to capture long-distance dependent features of global images.
Based on this, it is necessary to study a new emotion recognition method and apparatus to solve the above-mentioned problems of the conventional brain-electric emotion recognition method.
Disclosure of Invention
The invention aims to: aiming at the defects existing in the prior art, the invention aims to provide an emotion recognition method and device based on an electroencephalogram cross-frequency coupling diagram and deep learning, so as to comprehensively represent the coupling relations between channels of the electroencephalogram and between different frequencies of the electroencephalogram, and mine potential characteristics in the whole brain cross-frequency coupling diagram under different emotion states through the deep learning method, thereby effectively decoding the emotion state of a user.
The technical scheme is as follows: the invention provides an emotion recognition method based on an electroencephalogram cross-frequency coupling diagram and deep learning, which comprises the following steps of:
step 1: collecting brain electrical signals of N channels based on a sensor on the scalp surface of a user, and preprocessing the brain electrical signals to obtain preprocessed signals;
step S2: selecting electroencephalogram signals of two channels, calculating cross-frequency phase amplitude coupling coefficients between brain electric sub-bands of the two channels, drawing a cross-frequency coupling diagram of the two channels, replacing channel combinations, and repeating until all channel combinations are calculated to obtain N multiplied by N cross-frequency coupling diagrams of the two channels;
step S3: determining an optimal ordering mode of N multiplied by N two-channel cross-frequency coupling graphs by utilizing a gradient iterative algorithm to minimize an objective function, and splicing the N multiplied by N two-channel cross-frequency coupling graphs into a smooth transition full brain channel cross-frequency coupling graph in a form of N rows and N columns;
step S4: training a neural network based on an attention mechanism by using the whole brain channel cross-frequency coupling diagram in the step S3 to form an emotion recognition model;
step S5: and converting the electroencephalogram signals to be identified into full brain channel cross-frequency coupling diagrams according to the step S2 and the step S3, and inputting the full brain channel cross-frequency coupling diagrams into the emotion identification model trained in the step S4 for classification to obtain emotion identification results.
Further, the preprocessing the electroencephalogram signal in the step S1 specifically includes:
s1.1, acquiring brain electrical signals of N channels of the brain based on a sensor on the scalp surface of a user;
s1.2, performing downsampling and re-referencing on the electroencephalogram signals, and then removing baseline drift, high-frequency noise and power frequency interference through band-pass filtering and notch filtering;
s1.3, independent component analysis is applied to the electroencephalogram signal obtained in the step S1.2, and electro-oculogram and myoelectric artifacts are removed, so that a preprocessing signal is obtained.
Further, the step S2 specifically includes:
s2.1, selecting brain electrical signals of two channels from the preprocessing signals, setting the brain electrical signals as a channel p and a channel a, presetting the brain electrical signals of the channel p to m sub-frequency bands, and presetting the brain electrical signals of the channel a to n sub-frequency bands;
s2.2 regarding the channel a signal as x a (t) calculating the distribution C of signal energy over time t and frequency f a (t, f) the calculation method is:
wherein ,is a kernel function of the Rihaczek time-frequency distribution,/->To filter cross-term Choi-Williams kernels, A a (θ, τ) is the signal x a The fuzzy function of (t) is calculated by the following steps: /> wherein ,/>Is x a Conjugation of (t);
s2.3 calculating the instantaneous amplitude of the channel a subband signal wherein ,/> and />Defining a sub-band center frequency f a Surrounding bandwidth, for resolving signals->Fourier transform thereof>There is->Namely:
wherein , and θa (f) Representing the phase of the channel a signal in the time and frequency domains, respectively;
s2.4 regarding the channel p signal in S2.1 as x p (t) calculating the distribution C of signal energy over time t and frequency f p (t, f) the calculation method is:
wherein ,Ap (θ, τ) is the signal x p The fuzzy function of (t) is calculated by wherein />Is x p Conjugation of (t);
s2.5 calculating the instantaneous phase of the channel p subband signalf p Is the center frequency of the sub-band, & gt for the resolved signal>Fourier transform thereof>Has the following components
wherein , and θp (f) Representing the phase of the channel p signal in the time and frequency domains, respectively;
s2.6, calculating a cross-frequency phase-amplitude coupling coefficient MVL between every two sub-bands of the channel p and the channel a:
obtaining cross-frequency phase amplitude coupling coefficients between m multiplied by n sub-bands, drawing two-channel cross-frequency coupling graphs, wherein the size of each two-channel cross-frequency coupling graph is m multiplied by n pixels, namely, each cross-frequency phase amplitude coupling coefficient corresponds to one pixel in the two-channel cross-frequency coupling graph;
s2.7, replacing the channel combination, and repeating the steps S2.1 to S2.6 until all the channel combinations are calculated, so as to obtain N multiplied by N two-channel cross-frequency coupling diagrams.
Further, the electroencephalogram signals of two channels are selected in the pre-processing signal, which may be the same channel.
Further, the step S3 specifically includes:
s3.1 three-dimensional space position X of N brain electrical channels 1 ,X 2 ,...,X n Projecting the images into a one-dimensional space by using a one-dimensional scaling method to obtain new positions of N brain electrical channels in the one-dimensional space, and setting |l i -l k I represents the euclidean distance between the ith and kth channels projected into one-dimensional space, defining the parallax function δ (i, k) =2 (1-c i,k), wherein ci,k And expressing the measurement of the cross-frequency coupling relation between the ith and kth channels, namely substituting the average value of all coupling coefficients in the cross-frequency coupling graph of the two channels into a normalized stress function:solving an objective function minStress (l) using a gradient iterative algorithm 1 ,...,l N ) Solution obtained->The optimal channel arrangement sequence is used for enabling the parallax function values and the distances in the one-dimensional space to be similar as much as possible, namely channels with strong cross-frequency coupling relations are arranged closer;
and S3.2, arranging N multiplied by N two-channel cross-frequency coupling graphs in the form of N rows and N columns in the horizontal direction and the vertical direction according to the sequence solved in the step S3.1, and splicing the N multiplied by N two-channel cross-frequency coupling graphs into a smooth transition full-brain cross-frequency coupling graph, wherein the size of each full-brain cross-frequency coupling graph is mN multiplied by N pixels.
Further, the step S4 specifically includes:
s4.1, dividing the whole brain channel cross-frequency coupling diagram into M non-overlapping image blocks with uniform sizes, and generating a position code c of the image blocks;
s4.2, constructing a convolutional neural network image feature extractor, wherein the convolutional neural network image feature extractor comprises a convolutional layer, a maximum pooling layer, a batch normalization layer and a Dropout layer;
s4.3 synchronously feeding the M image blocks in the step S4.1 into a convolutional neural network image feature extractor to generate M feature graphs, and stretching each feature graph into a vector v i Linearly projecting to obtain embedded vectors v of M image blocks;
s4.4, combining the position code c of the image block with the embedded vector v of the image block to form an embedded vector e with position information, and adding a classification vector for learning the category information in the process of converting training to obtain a vector e in ；
S4.5: vector e in An input transducer encoder module comprising a multi-headed self-attention sub-module and a multi-layered perceptron sub-module, with layer normalization applied before each sub-module, residual connection applied after each sub-module, and with the multi-headed self-attention sub-module used to calculate the embedding vector e in Representation e with self-attention weighting attention The calculation method comprises the following steps:
Q,K,V＝Linear(e in ,(W Q ,W K ,W V ))
wherein ,WQ ,W K ,W V Weights representing Linear projections, linear () represents the sum of the weights of the embedded vector e in Doing linear projection, selfAttention (Q, K, V) is the output result of the self-attention machine submodule, d k Is the dimension of K, K T Representing the transpose of K, softmax () is the Softmax activation function, embedding vector e in And representation e with self-attention weighting attention In combination, e' =e attention +e in Inputting the multi-layer perception machine submodule and outputting e out ；
S4.6: extraction e out The classification vector in (1) is linearly projected and then input into a Softmax full-connection layer to obtain emotion classification probability;
s4.7: training the neural network based on the attention mechanism by using the whole brain channel cross-frequency coupling diagram in the step S3, optimizing a loss function by using cross entropy loss and adopting an AdamW optimization algorithm until convergence to obtain an emotion recognition model.
The invention also discloses a device based on the emotion recognition method based on the electroencephalogram cross-frequency coupling diagram and the deep learning, wherein a program is stored in the device, and the program is executed to realize the steps in the emotion recognition method based on the electroencephalogram cross-frequency coupling diagram and the deep learning, and the method specifically comprises the following steps:
the electroencephalogram signal acquisition and preprocessing module is used for acquiring the electroencephalogram signals of N channels on the surface of the scalp of the user, preprocessing the electroencephalogram signals and then sending the preprocessed electroencephalogram signals to the cross-frequency coupling diagram drawing module;
the cross-frequency coupling diagram drawing module is used for calculating cross-frequency phase amplitude coupling coefficients MVL between the brain electric sub-bands of the two channels and drawing N multiplied by N two-channel cross-frequency coupling diagrams;
the cross-frequency coupling diagram sequencing and splicing module is used for solving the optimal sequencing mode of N multiplied by N two-channel cross-frequency coupling diagrams and splicing the N multiplied by N two-channel cross-frequency coupling diagrams into a smooth transition whole brain channel cross-frequency coupling diagram;
the deep learning emotion recognition model building training module is used for training a neural network based on an attention mechanism by using the whole brain channel cross-frequency coupling diagram to form an emotion recognition model;
and the emotion recognition and classification module is used for converting the electroencephalogram signals to be recognized into a whole brain channel cross-frequency coupling diagram, and then classifying by using the emotion recognition model to obtain emotion recognition results.
Further, the device also comprises a result visual feedback transmission module, which is used for visually feeding back the emotion recognition result obtained by the emotion recognition module, transmitting the result visual feedback, and storing the result visual feedback to the remote server.
Advantageous effects
1. The invention extracts the amplitude and the phase of the signal based on the high-resolution complex time-frequency distribution method, does not need to carry out band-pass filtering on the signal, and solves the problem of frequency identification errors caused by selecting the bandwidth and the transition band of the band-pass filter.
2. Through drawing, sequencing and splicing of the cross-frequency coupling diagrams, the invention comprehensively shows the coupling relations among the channels of the electroencephalogram signals and among different frequencies, forms a full-brain cross-frequency coupling diagram with smooth transition, and excavates the potential characteristics of the electroencephalogram in different emotion states through a deep learning method so as to effectively decode the emotion states of the user.
3. According to the invention, by introducing a self-attention mechanism in the transducer module, key information in the cross-frequency coupling diagram is automatically focused, so that the model can more intensively acquire the characteristics related to the specific emotion types, interference of irrelevant information can be reduced to a certain extent, meanwhile, the receptive field can be enlarged, long-distance dependent characteristics of the global image can be captured, and the efficiency of emotion classification tasks is improved.
4. The device can realize remote emotion monitoring based on the deep learning emotion recognition model for recognizing the emotion state of the user, and has a certain practical significance.
Drawings
Fig. 1 is a flowchart of an emotion recognition method based on an electroencephalogram cross-frequency coupling diagram and deep learning.
Fig. 2 is an electroencephalogram signal of FP1 lead and C3 lead obtained by pretreatment in the embodiment of the present invention.
FIG. 3 is a diagram of a two-channel cross-frequency coupling between FP1 lead and C3 lead at 0-100Hz in an embodiment of the invention.
Fig. 4 is a full brain cross-frequency coupling diagram composed of 3600 two-channel cross-frequency coupling diagrams in an embodiment of the present invention.
Fig. 5 is a deep learning model structure for 7-class emotion recognition in an embodiment of the present invention.
Fig. 6 is a schematic structural diagram of an emotion recognition device based on an electroencephalogram cross-frequency coupling diagram and deep learning.
Best mode for carrying out the invention
The invention will now be described in further detail with reference to the drawings and to specific examples.
Referring to fig. 1, the invention provides an emotion recognition method based on an electroencephalogram cross-frequency coupling diagram and deep learning, which comprises the following steps:
step S1: and acquiring 60 channels of electroencephalogram signals based on a sensor on the surface of the scalp of the user, and preprocessing the electroencephalogram signals to obtain preprocessed signals.
In this embodiment, step S1 collects an electroencephalogram signal based on a sensor on the scalp surface of the user, the sampling rate of the original signal is 1000Hz, resampling the signal to reduce the operand and thereby increase the operation speed, setting the resampling frequency to 200Hz, and re-referencing to an average reference, performing 0.1-100Hz band-pass filtering on the electroencephalogram signal to remove baseline drift and high frequency noise, and performing 50Hz notch filtering to remove power frequency interference.
Step S2: and selecting an electroencephalogram signal of the FP1 and C3 channels, calculating a cross-frequency phase amplitude coupling coefficient between the brain electric sub-bands of the FP1 and C3 channels, wherein the FP1 channel is used for extracting a phase, the C3 channel is used for extracting an amplitude, drawing a two-channel cross-frequency coupling diagram, replacing a channel combination, and repeating until all channel combinations are calculated to obtain 60 multiplied by 60 two-channel cross-frequency coupling diagrams.
S2.1 referring to FIG. 2, the EEG signal of the FP1 channel is preset to 100 sub-bands, and the EEG signal of the C3 channel is preset to 100 sub-bands, each sub-band having a bandwidth of 1Hz.
S2.2 regarding the FP1 channel signal as x a (t) calculating the distribution C of signal energy over time t and frequency f a (t, f) the calculation method is:
wherein ,is a kernel function of the Rihaczek time-frequency distribution,/->Is a Choi-Williams core for filtering cross terms, A a (θ, τ) is the signal x a The fuzzy function of (t) is calculated by the following steps: /> wherein />Is x a Conjugation of (t).
S2.3 calculating the instantaneous amplitude of each sub-band signal of the FP1 channel brain electrical signal wherein fa1 and fa2 Defining a sub-band center frequency f a Surrounding bandwidth, for resolving signals->Fourier transform thereof>There is->Namely:
wherein , and θa (f) The phases of the FP1 channel electroencephalogram signals in the time domain and the frequency domain are represented, respectively.
S2.4 regarding the EEG signal of the C3 channel as x p (t)，Calculating the distribution C of signal energy over time t and frequency f p (t, f) the calculation method is:
wherein ,Ap (θ, τ) is the signal x p The fuzzy function of (t) is calculated by wherein />Is x p Conjugation of (t).
S2.5 calculating the instantaneous phase of the brain electrical signal of the C3 channelf p Is the center frequency of the sub-band, & gt for the resolved signal>Fourier transform thereof>The method comprises the following steps:
wherein , and θp (f) The phases of the brain electrical signals of the C3 channel in the time domain and the frequency domain are represented, respectively.
S2.6 referring to FIG. 3, the cross-frequency phase-amplitude coupling coefficient MVL between the frequency bands of the FP1 and C3 channels of the EEG signals is calculated:
the cross-frequency phase amplitude coupling coefficients between 100×100 sub-bands are obtained, two-channel cross-frequency coupling diagrams are drawn, the size of each two-channel cross-frequency coupling diagram is 100×100 pixels, namely, each cross-frequency phase amplitude coupling coefficient corresponds to one pixel in the two-channel cross-frequency coupling diagram.
S2.7, replacing the channel combination, and repeating the steps S2.1, S2.2, S2.3, S2.4, S2.5 and S2.6 until all the channel combinations are calculated, so as to obtain 60 multiplied by 60 two-channel cross-frequency coupling diagrams.
Step S3: and (3) minimizing an objective function by using a gradient iterative algorithm, determining an ordering mode of 60 multiplied by 60 two-channel cross-frequency coupling graphs, and splicing the 60 multiplied by 60 two-channel cross-frequency coupling graphs into a smooth transition full brain channel cross-frequency coupling graph in a 60-row 60-column mode.
S3.1 three-dimensional space position X of 60 brain electrical channels 1 ,X 2 ,...,X n Projecting the images into a one-dimensional space by using a one-dimensional scaling method to obtain new positions of N electroencephalogram channels in the one-dimensional space, |l i -l k I represents the euclidean distance between the ith and kth channels projected into one-dimensional space, defining the parallax function δ (i, k) =2 (1-c i,k), wherein ci,k And expressing the measurement of the cross-frequency coupling relation between the ith and kth channels, namely substituting the average value of all coupling coefficients in the cross-frequency coupling graph of the two channels into a normalized stress function:solving an objective function minStress (l) using a gradient iterative algorithm 1 ,...,l N ) Solution obtained->The optimal channel arrangement sequence is obtained. The optimal channel arrangement order makes the parallax function values and distances in the one-dimensional space as similar as possible, i.e. channels with a strong cross-frequency coupling relationship are arranged closer.
S3.2 referring to FIG. 4, 60×60 two-channel cross-frequency coupling graphs are arranged in the horizontal direction and the vertical direction in the form of 60 rows and 60 columns according to the sequence solved in the step S3.1, and are spliced into a smooth transition full-brain cross-frequency coupling graph, wherein the size of each full-brain cross-frequency coupling graph is 6000×6000 pixels.
Step S4: neural networks based on the attention mechanism are trained by using the whole brain channel cross-frequency coupling diagram to form an emotion recognition model, and reference is made to fig. 5.
S4.1, dividing the whole brain channel cross-frequency coupling diagram into 25 non-overlapping image blocks with uniform sizes, and generating a position code c of the image blocks.
S4.2, constructing a convolutional neural network image feature extractor, which comprises a convolutional layer, a max pooling layer, a batch normalization layer and a Dropout layer. The convolution kernel of the convolution neural network image feature extractor has a size of 5×5, a step length of 2×2, a learning rate of 0.001 in the training process, a batch size of 16, and an iteration number of 1000.
S4.3 synchronously feeding the 25 image blocks of step S4.1 into a convolutional neural network image feature extractor, generating 25 feature maps, i=1,..m, and then stretching each feature map into a vector v i The linear projection results in embedded vectors v for 25 image blocks.
S4.4, combining the position code c of the image block with the embedded vector v of the image block to form an embedded vector e with position information, and adding a classification vector for learning the category information in the process of converting training to obtain a vector e in 。
S4.5: vector e in An input transducer encoder module comprising a multi-headed self-attention sub-module and a multi-Layer perceptron sub-module, layer normalization (Layer Norm) being applied before each sub-module, residual connection (Residual Connection) being applied after each sub-module, an embedding vector e being calculated using the multi-headed self-attention sub-module in Representation e with self-attention weighting attention The calculation method comprises the following steps:
wherein ,WQ ,W K ,W V Representing linear projectionWeight, linear () represents the number of embedded vectors e in Doing linear projection, selfAttention (Q, K, V) is the output result of the self-attention machine submodule, d k Is the dimension of K, K T Representing the K transpose, softmax () is the Softmax activation function, embedding the vector e in And representation e with self-attention weighting attention In combination, e' =e attention +e in Inputting the multi-layer perception machine submodule and outputting e out 。
S4.6: extraction e out The classification vectors in (1) are linearly projected and then input into the Softmax full connectivity layer to obtain emotion classification probabilities.
S4.7: training the neural network based on the attention mechanism by using the whole brain channel cross-frequency coupling diagram in the step S3, optimizing the loss function by using cross entropy loss and adopting an AdamW optimization algorithm until convergence to obtain an emotion recognition model.
Step S5: and converting the electroencephalogram signals to be identified into full brain channel cross-frequency coupling diagrams according to the step S2 and the step S3, and inputting the full brain channel cross-frequency coupling diagrams into the emotion recognition model trained in the step S4 for classification to obtain emotion recognition results.
Referring to fig. 6, as an implementation of the method shown in the above embodiment, another embodiment of the present invention further provides an electroencephalogram cross-frequency coupling map and deep learning-based emotion recognition apparatus, on which a program is stored, which when executed implements the steps in the electroencephalogram cross-frequency coupling map and deep learning-based emotion recognition method as described above. The embodiment of the device corresponds to the embodiment of the method, and for convenience of reading, details of the embodiment of the method are not repeated one by one, but it should be clear that the device in the embodiment can correspondingly realize all the details of the embodiment of the method. Fig. 6 shows a structure of an emotion recognition device based on an electroencephalogram cross-frequency coupling diagram and deep learning according to an embodiment of the present invention.
The foregoing has outlined preferred embodiments of the present invention so that those skilled in the art may better understand and utilize the invention. The present invention is not limited to the embodiments described above, and equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the present invention, and these equivalent modifications and substitutions are intended to be included in the scope of the present invention as defined in the appended claims.
Claims (8)
1. An emotion recognition method based on an electroencephalogram cross-frequency coupling diagram and deep learning is characterized by comprising the following steps of:
step 1: collecting brain electrical signals of N channels based on a sensor on the scalp surface of a user, and preprocessing the brain electrical signals to obtain preprocessed signals;
step S2: selecting electroencephalogram signals of two channels, calculating cross-frequency phase amplitude coupling coefficients between brain electric sub-bands of the two channels, drawing a cross-frequency coupling diagram of the two channels, replacing channel combinations, and repeating until all channel combinations are calculated to obtain N multiplied by N cross-frequency coupling diagrams of the two channels;
step S3: minimizing an objective function by utilizing a gradient iterative algorithm, determining an optimal ordering mode of N multiplied by N two-channel cross-frequency coupling graphs, and splicing the N multiplied by N two-channel cross-frequency coupling graphs into a smooth transition full brain channel cross-frequency coupling graph in the form of N rows and N columns;
step S4: training a neural network based on an attention mechanism by using the whole brain channel cross-frequency coupling diagram in the step S3 to form an emotion recognition model;
step S5: and converting the electroencephalogram signals to be identified into full brain channel cross-frequency coupling diagrams according to the step S2 and the step S3, and inputting the full brain channel cross-frequency coupling diagrams into the emotion identification model trained in the step S4 for classification to obtain emotion identification results.
2. The emotion recognition method based on the electroencephalogram cross-frequency coupling diagram and deep learning according to claim 1, wherein the preprocessing of the electroencephalogram signal in step S1 specifically includes:
s1.1, acquiring brain electrical signals of N channels of the brain based on a sensor on the scalp surface of a user;
s1.2, performing downsampling and re-referencing on the electroencephalogram signals, and then removing baseline drift, high-frequency noise and power frequency interference through band-pass filtering and notch filtering;
s1.3, independent component analysis is applied to the electroencephalogram signal obtained in the step S1.2, and electro-oculogram and myoelectric artifacts are removed, so that a preprocessing signal is obtained.
3. The emotion recognition method based on the electroencephalogram cross-frequency coupling map and deep learning according to claim 1, wherein the step S2 specifically includes:
s2.1, selecting brain electrical signals of two channels from the preprocessing signals, setting the brain electrical signals as a channel p and a channel a, presetting the brain electrical signals of the channel p to m sub-frequency bands, and presetting the brain electrical signals of the channel a to n sub-frequency bands;
s2.2 regarding the channel a signal as x a (t) calculating the distribution C of signal energy over time t and frequency f a (t, f) the calculation method is:
wherein ,is a kernel function of the Rihaczek time-frequency distribution,/->To filter cross-term Choi-Williams kernels, A a (θ, τ) is the signal x a The fuzzy function of (t) is calculated by the following steps: /> wherein ,/>Is x a Conjugation of (t);
s2.3 calculating the instantaneous amplitude of the channel a subband signal wherein ,/> and />Defining a sub-band center frequency f a Surrounding bandwidth, for resolving signals->Fourier transform thereof>There is->Namely:
wherein , and θa (f) Representing the phase of the channel a signal in the time and frequency domains, respectively;
s2.4 regarding the channel p signal in S2.1 as x p (t) calculating the distribution C of signal energy over time t and frequency f p (t, f) the calculation method is:
wherein ,Ap (θ, τ) is the signal x p The fuzzy function of (t) is calculated by wherein />Is x p Conjugation of (t);
s2.5 calculating the instantaneous phase of the channel p subband signalf p Is the center frequency of the sub-band, & gt for the resolved signal>Fourier transform thereof>Has the following components
wherein , and θp (f) Representing the phase of the channel p signal in the time and frequency domains, respectively;
s2.6, calculating a cross-frequency phase-amplitude coupling coefficient MVL between every two sub-bands of the channel p and the channel a:
obtaining cross-frequency phase amplitude coupling coefficients between m multiplied by n sub-bands, drawing two-channel cross-frequency coupling graphs, wherein the size of each two-channel cross-frequency coupling graph is m multiplied by n pixels, namely, each cross-frequency phase amplitude coupling coefficient corresponds to one pixel in the two-channel cross-frequency coupling graph;
s2.7, replacing the channel combination, and repeating the steps S2.1 to S2.6 until all the channel combinations are calculated, so as to obtain N multiplied by N two-channel cross-frequency coupling diagrams.
4. A method of emotion recognition based on an electroencephalogram cross-frequency coupling map and deep learning as claimed in claim 3, wherein two channels of electroencephalogram signals are selected in the pre-processed signal, which may be the same channel.
5. The emotion recognition method based on the electroencephalogram cross-frequency coupling diagram and deep learning according to claim 1, wherein the step S3 specifically includes:
s3.1 three-dimensional space position X of N brain electrical channels 1 ,X 2 ,...,X n Projecting the images into a one-dimensional space by using a one-dimensional scaling method to obtain new positions of N brain electrical channels in the one-dimensional space, and setting |l i -l k I represents the euclidean distance between the ith and kth channels projected into one-dimensional space, defining the parallax function δ (i, k) =2 (1-c i,k), wherein ci,k And expressing the measurement of the cross-frequency coupling relation between the ith and kth channels, namely substituting the average value of all coupling coefficients in the cross-frequency coupling graph of the two channels into a normalized stress function:solving an objective function minStress (l) using a gradient iterative algorithm 1 ,...,l N ) Solution obtained->The optimal channel arrangement sequence is used for enabling the parallax function values and the distances in the one-dimensional space to be similar as much as possible, namely channels with strong cross-frequency coupling relations are arranged closer;
and S3.2, arranging N multiplied by N two-channel cross-frequency coupling graphs in the form of N rows and N columns in the horizontal direction and the vertical direction according to the sequence solved in the step S3.1, and splicing the N multiplied by N two-channel cross-frequency coupling graphs into a smooth transition full-brain cross-frequency coupling graph, wherein the size of each full-brain cross-frequency coupling graph is mN multiplied by N pixels.
6. The emotion recognition method based on the electroencephalogram cross-frequency coupling map and deep learning according to claim 1, wherein the step S4 specifically includes:
s4.1, dividing the whole brain channel cross-frequency coupling diagram into M non-overlapping image blocks with uniform sizes, and generating a position code c of the image blocks;
s4.2, constructing a convolutional neural network image feature extractor, wherein the convolutional neural network image feature extractor comprises a convolutional layer, a maximum pooling layer, a batch normalization layer and a Dropout layer;
s4.3 synchronously feeding the M image blocks in the step S4.1 into a convolutional neural network image feature extractor to generate M feature graphs, and stretching each feature graph into a vector v i Linearly projecting to obtain embedded vectors v of M image blocks;
s4.4, combining the position code c of the image block with the embedded vector v of the image block to form an embedded vector e with position information, and adding a classification vector for learning the category information in the process of converting training to obtain a vector e in ；
S4.5: vector e in An input transducer encoder module comprising a multi-headed self-attention sub-module and a multi-layered perceptron sub-module, with layer normalization applied before each sub-module, residual connection applied after each sub-module, and with the multi-headed self-attention sub-module used to calculate the embedding vector e in Representation e with self-attention weighting attention The calculation method comprises the following steps:
Q,K,V＝Linear(e in ,(W Q ,W K ,W V ))
wherein ,WQ ,W K ,W V Weights representing Linear projections, linear () represents the sum of the weights of the embedded vector e in Doing linear projection, selfAttention (Q, K, V) is the output result of the self-attention machine submodule, d k Is the dimension of K, K T Representing the transpose of K, softmax () is the Softmax activation function, embedding vector e in And representation e with self-attention weighting attention In combination, e' =e attention +e in Inputting the multi-layer perception machine submodule and outputting e out ；
S4.6: extraction e out The classification vector in (1) is linearly projected and then input into a Softmax full-connection layer to obtain emotion classification probability;
s4.7: training the neural network based on the attention mechanism by using the whole brain channel cross-frequency coupling diagram in the step S3, optimizing a loss function by using cross entropy loss and adopting an AdamW optimization algorithm until convergence to obtain an emotion recognition model.
7. An apparatus based on the electroencephalogram cross-frequency coupling map and deep learning emotion recognition method according to any one of claims 1 to 6, wherein a program is stored in the apparatus, and the program is executed to implement the steps in the electroencephalogram cross-frequency coupling map and deep learning emotion recognition method according to any one of claims 1 to 6, specifically comprising:
the electroencephalogram signal acquisition and preprocessing module is used for acquiring the electroencephalogram signals of N channels on the surface of the scalp of the user, preprocessing the electroencephalogram signals and then sending the preprocessed electroencephalogram signals to the cross-frequency coupling diagram drawing module;
the cross-frequency coupling diagram drawing module is used for calculating cross-frequency phase amplitude coupling coefficients MVL between the brain electric sub-bands of the two channels and drawing N multiplied by N two-channel cross-frequency coupling diagrams;
the cross-frequency coupling diagram sequencing and splicing module is used for solving the optimal sequencing mode of N multiplied by N two-channel cross-frequency coupling diagrams and splicing the N multiplied by N two-channel cross-frequency coupling diagrams into a smooth transition whole brain channel cross-frequency coupling diagram;
the deep learning emotion recognition model building training module is used for training a neural network based on an attention mechanism by using the whole brain channel cross-frequency coupling diagram to form an emotion recognition model;
and the emotion recognition and classification module is used for converting the electroencephalogram signals to be recognized into a whole brain channel cross-frequency coupling diagram, and then classifying by using the emotion recognition model to obtain emotion recognition results.
8. The device according to claim 7, further comprising a result visualization feedback transmission module, configured to perform visualization feedback on the emotion recognition result obtained by the emotion recognition module, and transmit and store the result to a remote server.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310656838.5A CN116712074B (en) | 2023-06-05 | 2023-06-05 | Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310656838.5A CN116712074B (en) | 2023-06-05 | 2023-06-05 | Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116712074A true CN116712074A (en) | 2023-09-08 |
| CN116712074B CN116712074B (en) | 2024-06-14 |
Family
ID=87872735
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310656838.5A Active CN116712074B (en) | 2023-06-05 | 2023-06-05 | Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116712074B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170202475A1 (en) * | 2016-01-19 | 2017-07-20 | Washington University | Depression brain computer interface for the quantitative assessment of mood state and for biofeedback for mood alteration |
| WO2022020538A1 (en) * | 2020-07-22 | 2022-01-27 | Reversal Solutions, Inc. | Systems and methodologies for performing brainwave entrainment using nested waveforms |
| CN115770044A (en) * | 2022-11-17 | 2023-03-10 | 天津大学 | Emotion recognition method and device based on electroencephalogram phase amplitude coupling network |
-
2023
- 2023-06-05 CN CN202310656838.5A patent/CN116712074B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170202475A1 (en) * | 2016-01-19 | 2017-07-20 | Washington University | Depression brain computer interface for the quantitative assessment of mood state and for biofeedback for mood alteration |
| WO2022020538A1 (en) * | 2020-07-22 | 2022-01-27 | Reversal Solutions, Inc. | Systems and methodologies for performing brainwave entrainment using nested waveforms |
| CN115770044A (en) * | 2022-11-17 | 2023-03-10 | 天津大学 | Emotion recognition method and device based on electroencephalogram phase amplitude coupling network |
Non-Patent Citations (5)
| Title |
|---|
| LI, M., QIU, M., ZHU, L. ET AL.: "Emotion recognition using spatial-temporal EEG features through convolutional graph attention network", 《JOURNAL OF NEURAL ENGINEERING》, pages 1 - 13 * |
| MAREIKE J. HÜLSEMANN，ETC.: "Quantification of Phase-Amplitude Coupling in Neuronal Oscillations: Comparison of Phase-Locking Value, Mean Vector Length, Modulation Index, and Generalized-Linear- Modeling-Cross-Frequency-Coupling", FRONTIERS IN NEUROSCIENCE, pages 1 - 13 * |
| SEONG-EUN MOON, ETC.: "Emotional EEG classification using connectivity features and convolutional neural networks", NEURAL NETWORKS, pages 99 - 106 * |
| WANG W: "《Brain network features based on theta-gamma cross-frequency coupling connections in EEG for emotion recognition》", 《NEUROSCI LETT》, pages 1 - 8 * |
| XIAOHU WANG, ETC: "Deep learning-based EEG emotion recognition: Current trends and future perspectives", FRONTIERS IN PSYCHOLOGY, pages 1 - 12 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116712074B (en) | 2024-06-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111012336B (en) | Parallel convolutional network motor imagery electroencephalogram classification method based on spatio-temporal feature fusion | |
| Aznan et al. | Simulating brain signals: Creating synthetic eeg data via neural-based generative models for improved ssvep classification | |
| CN112244873B (en) | Electroencephalogram space-time feature learning and emotion classification method based on hybrid neural network | |
| CN113887513B (en) | Motor imagery electroencephalogram signal classification method based on parallel CNN-transducer neural network | |
| Li et al. | Deep learning methods in real-time image super-resolution: a survey | |
| CN112120694A (en) | Motor imagery electroencephalogram signal classification method based on neural network | |
| Guo et al. | Deep neural networks on graph signals for brain imaging analysis | |
| KR20190033972A (en) | Method and apparatus for an automatic artifact removal of EEG based on a deep leaning algorithm | |
| CN114947883A (en) | Time-frequency domain information fusion deep learning electroencephalogram noise reduction method | |
| CN115105094B (en) | Motor imagery classification method based on attention and 3D dense connection neural network | |
| KR20190073330A (en) | Method and apparatus for an automatic artifact removal of EEG based on a deep leaning algorithm | |
| CN115770044A (en) | Emotion recognition method and device based on electroencephalogram phase amplitude coupling network | |
| CN114167982A (en) | Brain-computer interface system based on tensor space-frequency coupling filtering | |
| CN114841192A (en) | Electroencephalogram signal feature enhancement method based on reinforcement learning combined denoising and space-time relation modeling | |
| CN116584902B (en) | Heart sound classification device based on feature optimization and visualization | |
| CN116712074B (en) | Emotion recognition method and device based on electroencephalogram cross-frequency coupling diagram and deep learning | |
| CN117694904A (en) | Self-supervision pre-training method for multi-source electroencephalogram signals | |
| CN115374831B (en) | Dynamic and static combination velocity imagery classification method for multi-modal registration and space-time feature attention | |
| CN116662782A (en) | MSFF-SENET-based motor imagery electroencephalogram decoding method | |
| CN116421200A (en) | Brain electricity emotion analysis method of multi-task mixed model based on parallel training | |
| Chen et al. | Compressive sensing with side information: How to optimally capture this extra information for GMM signals? | |
| Mukadam et al. | Fusion of ESRGAN, Adaptive NSCT, and Multi-Attention CNN with Wavelet Transform for Histopathological Image Classification | |
| CN118709050B (en) | Whale signal classification method and system based on space heterogeneous fusion | |
| CN114533083B (en) | Motor imagery state identification method based on multi-fusion convolutional neural network | |
| Liu et al. | One to multiple mapping dual learning: Learning multiple sources from one mixed signal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |