CN113297981A - End-to-end electroencephalogram emotion recognition method based on attention mechanism - Google Patents
End-to-end electroencephalogram emotion recognition method based on attention mechanism Download PDFInfo
- Publication number
- CN113297981A CN113297981A CN202110584519.9A CN202110584519A CN113297981A CN 113297981 A CN113297981 A CN 113297981A CN 202110584519 A CN202110584519 A CN 202110584519A CN 113297981 A CN113297981 A CN 113297981A
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- sample
- attention
- matrix
- lead
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention discloses an end-to-end electroencephalogram emotion recognition method based on an attention mechanism, which comprises the steps of firstly determining the length of an electroencephalogram sample, the length of a slice in the sample and the window shift of the slice, and slicing the electroencephalogram sample; calculating the characteristics of the electroencephalogram signals, and rearranging the characteristics according to the distribution of leads of the electroencephalogram acquisition equipment on the head area to obtain electroencephalogram signal characteristics similar to thermodynamic diagrams; then, inputting the electroencephalogram signal characteristics into a convolutional neural network based on an attention system to obtain electroencephalogram signal short-time characteristic expression; then, taking the short-time feature representation of the electroencephalogram signal as input, learning the importance of the electroencephalogram signal in different time periods in the sample to emotion classification by utilizing an LSTM network added with an attention mechanism, and obtaining the long-time feature representation of the electroencephalogram sample; and finally, taking the long-term feature expression of the whole sample as input, and entering a full-connection layer for emotion classification. The method improves the accuracy of emotion classification based on the electroencephalogram, and provides reference for the duration of maintenance of the electroencephalogram paradigm when the emotion changes.
Description
Technical Field
The invention belongs to the technical field of pattern recognition, and particularly relates to an electroencephalogram emotion recognition method.
Background
In recent years, attention is paid to the field of automatic emotion recognition, for example, in the field of human-computer interaction, a machine can automatically recognize the emotion of an observed person and respond accordingly. At present, the emotion recognition field is mainly divided into two types, one type is discrete emotion recognition, namely, the emotion of a person is classified into a plurality of common states of happiness, sadness, anger and the like; the other is continuous emotion recognition, which expresses the emotional state of a person in two dimensions, wherein Arousal represents excitement degree and value represents pleasure degree. The different angles of the descriptions of the two fields are different, and the information conveyed to people is also different, which is the focus of the research of the emotional field.
In the past years, emotion classification through electroencephalogram signal states has achieved a lot of important results, and a lot of research has proved that the effect on emotion classification or mental stress is superior to that of a model using global features of electroencephalogram signals as input by paying attention to the spatiotemporal characteristics of electroencephalogram signals at the same time. In the documents "EEG-Based emission Recognition using 3D conditional Neural Networks" International Journal of Advanced Computer Science and Applications (IJACSA),9(8),2018, the 3D-CNN is used for recognizing human Emotion from multi-channel EEG data of DEAP data sets, and 87.44% and 88.49% of Recognition accuracy documents "Spatial-Temporal current Neural Network for Emotion Recognition," in IEEE Transactions on Cybernetics, vol.49, No.3, pp.839-847 propose a new deep learning framework, called spatio-Temporal Recurrent Neural Network (STRNN), which integrates the characteristics of Spatial information and Temporal information of signal sources into a unified system, and makes reasonable classification of discrete electroencephalogram information by using spatio-Temporal characteristics.
Space-time correlation researchers have achieved certain achievements in emotion classification models, however continuous dimension emotion estimation still faces the following challenges due to emotion complexity and individual variability:
1) the "key frame" problem. In the long-time sequence emotion classification task, the emotion state of each moment has strong correlation with the emotion state of the latest moment and has stronger correlation with the emotion information of some key moments, and meanwhile, the emotion state of each moment has less possible relation with the emotion information of a long time before. In the past electroencephalogram-based emotion classification research, when the emotional state at each moment is estimated, past emotion information is processed in an equally important mode, so that the model is difficult to acquire key context information, and the generalization capability and accuracy of the model are influenced.
2) And (4) sample length regulation. Input samples of a traditional electroencephalogram-based emotion classification model are often not the same in length, emotion related features are generally calculated in 1 second or 4 seconds for emotion classification, if a time sequence model is not used, the sample length is often set to be 1s or 4s, electroencephalogram related features such as power spectral density are calculated in 10s medically, and a unified window length determination rule for calculating the sample length and the emotion related features is not provided. The classification accuracy calculated by different sample lengths also varies.
3) The model addresses the force mechanism problem. The traditional electroencephalogram-based emotion classification method generally utilizes global characteristics or Principal Component Analysis (PCA) and other methods to perform characteristic dimension reduction, uses the reduced characteristics as the input of a model, or divides different electroencephalogram leads into different areas, calculates the electroencephalogram emotion related characteristics in a partitioning mode as the input of the model, and finally performs result comparison. No weights are assigned to the different brain regions, frequency bands and time periods. It is difficult to mine the characteristics of electroencephalogram signals and the feature importance of different time and space from the machine learning angle.
In conclusion, the conventional electroencephalogram-based emotion classification method is limited by the traditional method, key space-time information is difficult to find, and the problems of low emotion classification precision, poor generalization capability and the like are easily caused.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an end-to-end electroencephalogram emotion recognition method based on an attention mechanism, wherein firstly, the length of an electroencephalogram sample, the length of a slice in the sample and the window shift of the slice are determined, and the electroencephalogram sample is sliced; calculating the characteristics of the electroencephalogram signals, and rearranging the characteristics according to the distribution of leads of the electroencephalogram acquisition equipment on the head area to obtain electroencephalogram signal characteristics similar to thermodynamic diagrams; then, inputting the electroencephalogram signal characteristics into a convolutional neural network based on an attention system to obtain electroencephalogram signal short-time characteristic expression; then, taking the short-time feature representation of the electroencephalogram signal as input, learning the importance of the electroencephalogram signal in different time periods in the sample to emotion classification by utilizing an LSTM network added with an attention mechanism, and obtaining the long-time feature representation of the electroencephalogram sample; and finally, taking the long-term feature expression of the whole sample as input, and entering a full-connection layer for emotion classification. The method improves the accuracy of emotion classification based on the electroencephalogram, and provides reference for the duration of maintenance of the electroencephalogram paradigm when the emotion changes.
The technical scheme adopted by the invention for solving the technical problem comprises the following steps:
step 1: preprocessing an electroencephalogram signal;
setting the length of an electroencephalogram sample, the length of a slice in the electroencephalogram sample and the slice window shift of the electroencephalogram sample, and slicing an electroencephalogram signal to obtain an electroencephalogram sample slice;
dividing each lead in an electroencephalogram sample slice into 5 different wave bands to calculate power spectral density and differential entropy characteristics, and forming a matrix by the power spectral density and the differential entropy characteristics to form electroencephalogram signal characteristics;
step 2: constructing an emotion classification model based on an electroencephalogram signal, wherein the emotion classification model comprises a convolutional neural network based on an attention mechanism and an LSTM network based on the attention mechanism;
step 2-1: a convolutional neural network based on an attention mechanism;
the attention-based convolutional neural network comprises a band attention module and a lead attention module;
step 2-1-1: inputting the electroencephalogram signal characteristics into a wave band attention module, firstly pooling, taking the pooled one-dimensional characteristics as the input of a convolutional neural network, and then outputting the convolutional neural network through a Softmax function; the band attention module generates a band attention weight vector by using the characteristic relation among frequency bands, and the formula is as follows:
Wb(E)＝σ(f(Pool(E)))
wherein E represents electroencephalogram signal characteristics, Pool (.), f (.) represents a convolutional neural network, and sigma (.) represents a Softmax function;
step 2-1-2: inputting the wave band attention weight vector into a lead attention module, and generating a lead attention matrix by utilizing the spatial relation of the electroencephalogram energy, wherein the lead attention matrix is a two-dimensional square matrix;
firstly, performing one-dimensional pooling operation on a wave band attention weight vector, splicing pooled characteristic values into a two-dimensional square matrix according to the original position of a lead in a brain area, then performing convolution operation on the two-dimensional square matrix by using convolution check, and ensuring the size of the square matrix of the lead dimension to be unchanged by using padding and step length; the calculation formula of the lead attention matrix is as follows:
WC(Wb)＝σ(filter(Pool(Wb)))
wherein, filter (.) represents the convolution operation;
step 2-1-3: and adding the lead attention matrix and the electroencephalogram signal characteristics to obtain short-time characteristics of the electroencephalogram signals:
Wf＝(E+WC(Wb(E)))
step 2-2: an LSTM network based on an attention mechanism;
adding a time sequence attention module in the LSTM network for acquiring the electroencephalogram signal time sequence context;
definition of QtIs the characteristic vector of the electroencephalogram sample at the time t, and Q is equal to (Q)1,...Qt,...Qn) The method comprises the steps that a feature set of a single sample is obtained, and n represents the number of slices in the single sample, namely the number of short-time features; defining K, Q, V, and K and V are respectively a key name and mapping of electroencephalogram characteristics;
q, K, V is linearly projected to a plurality of subspaces, attention weights at different moments in time sequence are calculated in each subspace, weighting is carried out to obtain a feature vector of the whole sample, and finally the feature vector of the whole sample is linearly projected again to obtain final feature representation; the calculation formula of the whole time sequence attention module is as follows:
Q＝QW(T)Q
K＝KW(T)K
V＝VW(T)V
Attention(Q,K,V)＝AV
wherein Q is weighted EEG signal characteristic Wf；W(T)Q,W(T)K,W(T)VRespectively representing a query matrix, a K matrix and a key value matrix; t denotes the matrix transposition, dkIs a constant;
inputting the short-term characteristics of the electroencephalogram sample obtained in the step 2-1-3 into an LSTM network based on an attention mechanism to obtain long-term characteristics of the electroencephalogram sample;
step 2-3: a forward propagation module;
the forward propagation module comprises two linear mappings and a nonlinear activation function, and the calculation formula is as follows:
Cls(x)＝Softmax((xW1+b)+b)
where x represents the long-term characteristics of the brain electrical sample, where W1B is a parameter of the neural network, W1Representing neuron weights, b representing bias;
inputting the long-term characteristics of the electroencephalogram sample into a forward propagation module, and outputting an emotion classification result of the electroencephalogram signal;
and step 3: and training the emotion classification model based on the electroencephalogram signals, and obtaining a final emotion classification model after the training is finished, wherein the final emotion classification model is used for carrying out emotion judgment on the input electroencephalogram signals.
Preferably, the step of slicing the electroencephalogram signal is as follows:
setting 3 different sample lengths, dividing the samples into different slice lengths and window shifts in the samples, and processing and slicing the samples as dynamic sequences; the power spectral density and differential entropy characteristics are then calculated for 5 different bands separately for each lead in the sample slice. And (3) putting the characteristic values of all leads on the same wave band into a 9 x 9 two-dimensional square matrix according to the distribution mode of the leads on the electroencephalogram acquisition equipment, stacking the 5 lead characteristic square matrices according to the wave bands, and splicing the 5 lead characteristic square matrices into a three-dimensional 5 x 9 matrix as the input of the emotion classification model.
The invention has the following beneficial effects:
1. according to the invention, an attention mechanism is added in the CNN and LSTM models, so that the models can pay more attention to physical characteristics of a certain time and a certain point, thereby improving the accuracy of electroencephalogram-based emotion classification, and providing reference for the segmentation length of an electroencephalogram sample during emotion classification, namely the duration of maintenance of an electroencephalogram paradigm during emotion change.
2. The time periods, leads and wave bands with large energy fluctuation when being stimulated by emotional inducing factors are mined. The method is helpful for simplifying the calculation of the electroencephalogram signal emotion related characteristics and improving the real-time performance and generalization capability of electroencephalogram-based emotion classification.
Drawings
FIG. 1 is a diagram of an emotion classification model structure based on electroencephalogram signals.
FIG. 2 is a block diagram of a band attention module according to the present invention.
Figure 3 is a block diagram of a lead attention module of the present invention.
FIG. 4 is a block diagram of a short-term feature extraction module according to the present invention
Detailed Description
The invention is further illustrated with reference to the following figures and examples.
The invention provides an emotion classification model consisting of a Convolutional Neural Network (CNN), a long-short term memory network (LSTM), an Attention mechanism (Attention mechanism) and a time sequence convolutional network (TCN), which comprises two sub-networks connected in series, wherein each slice recombination characteristic in an electroencephalogram sample is firstly input into the convolutional neural network based on the Attention mechanism to obtain a new weighted and dimensionality-reduced electroencephalogram signal short-time characteristic expression; and then, taking the short-time feature representation of the electroencephalogram signal as input, learning the importance of the electroencephalogram signal in different time periods in the sample to emotion classification by using an LSTM network added with an attention mechanism, and obtaining the long-time feature representation of the electroencephalogram sample. And (4) taking long-term feature expression of the whole sample as input, and entering a full-connection layer for emotion classification.
As shown in fig. 1, an end-to-end electroencephalogram emotion recognition method based on an attention mechanism includes the following steps:
step 1: preprocessing an electroencephalogram signal;
setting the length of an electroencephalogram sample, the length of a slice in the electroencephalogram sample and the slice window shift of the electroencephalogram sample, and slicing an electroencephalogram signal to obtain an electroencephalogram sample slice; rearranging the characteristics according to the distribution of the leads of the electroencephalogram acquisition equipment on the head area to obtain electroencephalogram signal characteristics similar to thermodynamic diagrams;
setting 3 different sample lengths, dividing the samples into different slice lengths and window shifts in the samples, and processing and slicing the samples as dynamic sequences; the power spectral density and differential entropy characteristics are then calculated for 5 different bands separately for each lead in the sample slice. And (3) putting the characteristic values of all leads on the same wave band into a 9 x 9 two-dimensional square matrix according to the distribution mode of the leads on the electroencephalogram acquisition equipment, stacking the 5 lead characteristic square matrices according to the wave bands, and splicing the 5 lead characteristic square matrices into a three-dimensional 5 x 9 matrix as the input of the emotion classification model.
Step 2: constructing an emotion classification model based on an electroencephalogram signal, wherein the emotion classification model comprises a convolutional neural network based on an attention mechanism and an LSTM network based on the attention mechanism;
step 2-1: a convolutional neural network based on an attention mechanism;
the attention-based convolutional neural network comprises a band attention module and a lead attention module;
step 2-1-1: as shown in fig. 2, the electroencephalogram signal features are input into a waveband attention module, and the spatial dimension of the input features is compressed; pooling, taking the pooled one-dimensional features as input of a convolutional neural network, and performing a Softmax function on output of the convolutional neural network; the band attention module generates a band attention weight vector by using the characteristic relation among frequency bands, and the formula is as follows:
Wb(E)＝σ(f(Pool(E)))
wherein E represents electroencephalogram signal characteristics, Pool (.), f (.) represents a convolutional neural network, and sigma (.) represents a Softmax function;
step 2-1-2: as shown in fig. 3, the wave band attention weight vector is input into the lead attention module, and a lead attention matrix is generated by using the spatial relationship of the electroencephalogram energy, wherein the lead attention matrix is a two-dimensional square matrix;
firstly, performing one-dimensional pooling operation on a wave band attention weight vector, splicing pooled characteristic values into a two-dimensional square matrix according to the original position of a lead in a brain area, then performing convolution operation on the two-dimensional square matrix by using convolution check, and ensuring the size of the square matrix of the lead dimension to be unchanged by using padding and step length; the calculation formula of the lead attention matrix is as follows:
WC(Wb)＝σ(filter(Pool(Wb)))
wherein, filter (.) represents the convolution operation;
step 2-1-3: as shown in fig. 4, the lead attention matrix and the electroencephalogram signal characteristics are added to obtain the short-time characteristics of the electroencephalogram signal:
Wf＝(E+WC(WB(E)))
step 2-2: an LSTM network based on an attention mechanism;
in a limited sample length, different short-time stimulation segments have different influences on emotion, for example, different video pictures and different music tunes have different feelings to people, and in order to quantify the stimulation influence of time dimension, a time sequence attention module is added in an LSTM network for acquiring the time sequence context relationship of electroencephalogram signals;
definition of QtIs the characteristic vector of the electroencephalogram sample at the time t, and Q is equal to (Q)1,...Qt,...Qn) The method comprises the steps that a feature set of a single sample is obtained, and n represents the number of slices in the single sample, namely the number of short-time features; defining K, Q, V, and K and V are respectively a key name and mapping of electroencephalogram characteristics;
q, K, V is linearly projected to a plurality of subspaces, attention weights at different moments in time sequence are calculated in each subspace, weighting is carried out to obtain a feature vector of the whole sample, and finally the feature vector of the whole sample is linearly projected again to obtain final feature representation; the calculation formula of the whole time sequence attention module is as follows:
Q＝QW(T)Q
K＝KW(T)K
V＝VW(T)V
Attention(Q,K,V)＝AV
wherein Q is weighted EEG signal characteristic Wf；W(T)Q,W(T)K,W(T)VRespectively representing a query matrix, a K matrix and a key value matrix; t denotes the matrix transposition, dkIs a constant;
inputting the short-term characteristics of the electroencephalogram sample obtained in the step 2-1-3 into an LSTM network based on an attention mechanism to obtain long-term characteristics of the electroencephalogram sample;
step 2-3: a forward propagation module;
the forward propagation module comprises two linear mappings and a nonlinear activation function, and the calculation formula is as follows:
Cls(x)＝Softmax((xW1+b)+b)
where x represents the long-term characteristics of the brain electrical sample, where W1B is a parameter of the neural network, W1Representing neuron weights, b representing bias;
inputting the long-term characteristics of the electroencephalogram sample into a forward propagation module, and outputting an emotion classification result of the electroencephalogram signal;
and step 3: and training the emotion classification model based on the electroencephalogram signals, and obtaining a final emotion classification model after the training is finished, wherein the final emotion classification model is used for carrying out emotion judgment on the input electroencephalogram signals.
The specific embodiment is as follows:
the embodiment mainly designs an electroencephalogram lead wave band attention module, inserts the module into a CNN encoder, combines with an LSTM model based on an attention mechanism, expands the encoder into a model capable of acquiring time sequence context dependency relation for multi-modal characteristics at the same time, and realizes an emotion classification framework by utilizing the model. Through experimental inspection, the accuracy of classifying the emotion dimension Arousal can reach 0.91, the accuracy of classifying the emotion dimension Valence can reach 0.89, and the accuracy of classifying the active, neutral and passive discrete emotions can reach 0.92.
The invention designs an emotion classification model based on a Convolutional Neural Network (CNN), a long-short term memory network (LSTM) and an Attention Mechanism (Attention Mechanism):
1) according to the distribution form of the brain electricity leads on the brain electricity equipment, a new brain electricity characteristic organization form is designed, the new characteristic organization form is used as input, and the brain electricity characteristic organization form is more suitable for the time-space characteristics of brain electricity signals collected by the brain electricity collection equipment. And the influence of the sample duration, the length of a slice inside the sample and the slice time window shift on emotion classification is explored on a new electroencephalogram characteristic tissue form.
2) A band channel attention module is provided on the basis of a new electroencephalogram characteristic organization form, the importance of related characteristics of different bands of different leads on emotion classification is learned, and meanwhile, the relationship between stimulation content and electroencephalogram signal change of a subject when receiving emotion stimulation is learned together by combining with an attention mechanism of time dimension. At each time in the brain electrical sample, the feature input in a picture-like form automatically learns the importance of each feature point to the emotion classification within the time slice. The model then learns the interaction information between different moments and the importance of each moment to the emotion classification by means of an attention mechanism. And finally, the output of the weighted sum of the importance is used as the input of a full connection layer to obtain the emotion classification probability value.
3) A set of complete emotion classification models based on electroencephalogram signals is provided, and the models can adapt to continuous dimension emotion classification and discrete emotion classification by changing the number of nodes of the last layer of the models. The model contains two sub-networks in total: firstly, a wave band channel characteristic learning network extracts short time sequence characteristic expression of an electroencephalogram signal by using CNN and Attention, and the short time sequence characteristic expression is used as input of a long time sequence network model; secondly, a long-time-sequence electroencephalogram emotion prediction network is trained to obtain high-level feature expression fused with electroencephalogram time sequence context information from input short-time electroencephalogram features by using LSTM and Attention, and classification of emotion states is carried out.
Claims (2)
1. An end-to-end electroencephalogram emotion recognition method based on an attention mechanism is characterized by comprising the following steps:
step 1: preprocessing an electroencephalogram signal;
setting the length of an electroencephalogram sample, the length of a slice in the electroencephalogram sample and the slice window shift of the electroencephalogram sample, and slicing an electroencephalogram signal to obtain an electroencephalogram sample slice;
dividing each lead in an electroencephalogram sample slice into 5 different wave bands to calculate power spectral density and differential entropy characteristics, and forming a matrix by the power spectral density and the differential entropy characteristics to form electroencephalogram signal characteristics;
step 2: constructing an emotion classification model based on an electroencephalogram signal, wherein the emotion classification model comprises a convolutional neural network based on an attention mechanism and an LSTM network based on the attention mechanism;
step 2-1: a convolutional neural network based on an attention mechanism;
the attention-based convolutional neural network comprises a band attention module and a lead attention module;
step 2-1-1: inputting the electroencephalogram signal characteristics into a wave band attention module, firstly pooling, taking the pooled one-dimensional characteristics as the input of a convolutional neural network, and then outputting the convolutional neural network through a Softmax function; the band attention module generates a band attention weight vector by using the characteristic relation among frequency bands, and the formula is as follows:
Wb(E)＝σ(f(Pool(E)))
wherein E represents electroencephalogram signal characteristics, Pool (.), f (.) represents a convolutional neural network, and sigma (.) represents a Softmax function;
step 2-1-2: inputting the wave band attention weight vector into a lead attention module, and generating a lead attention matrix by utilizing the spatial relation of the electroencephalogram energy, wherein the lead attention matrix is a two-dimensional square matrix;
firstly, performing one-dimensional pooling operation on a wave band attention weight vector, splicing pooled characteristic values into a two-dimensional square matrix according to the original position of a lead in a brain area, then performing convolution operation on the two-dimensional square matrix by using convolution check, and ensuring the size of the square matrix of the lead dimension to be unchanged by using padding and step length; the calculation formula of the lead attention matrix is as follows:
WC(Wb)＝σ(filter(Pool(Wb)))
wherein, filter (.) represents the convolution operation;
step 2-1-3: and adding the lead attention matrix and the electroencephalogram signal characteristics to obtain short-time characteristics of the electroencephalogram signals:
Wf＝(E+WC(Wb(E)))
step 2-2: an LSTM network based on an attention mechanism;
adding a time sequence attention module in the LSTM network for acquiring the electroencephalogram signal time sequence context;
definition of QtIs the characteristic vector of the electroencephalogram sample at the time t, and Q is equal to (Q)1,...Qt,...Qn) The method comprises the steps that a feature set of a single sample is obtained, and n represents the number of slices in the single sample, namely the number of short-time features; defining K, Q, V, and K and V are respectively a key name and mapping of electroencephalogram characteristics;
q, K, V is linearly projected to a plurality of subspaces, attention weights at different moments in time sequence are calculated in each subspace, weighting is carried out to obtain a feature vector of the whole sample, and finally the feature vector of the whole sample is linearly projected again to obtain final feature representation; the calculation formula of the whole time sequence attention module is as follows:
Q＝QW(T)Q
K＝KW(T)K
V＝VW(T)V
Attention(Q,K,V)＝AV
wherein Q is weighted EEG signal characteristic Wf；W(T)Q,W(T)K,W(T)VRespectively representing a query matrix, a K matrix and a key value matrix; t denotes the matrix transposition, dkIs a constant;
inputting the short-term characteristics of the electroencephalogram sample obtained in the step 2-1-3 into an LSTM network based on an attention mechanism to obtain long-term characteristics of the electroencephalogram sample;
step 2-3: a forward propagation module;
the forward propagation module comprises two linear mappings and a nonlinear activation function, and the calculation formula is as follows:
Cls(x)＝Softmax((xW1+b)+b)
where x represents the long-term characteristics of the brain electrical sample, where W1B is a parameter of the neural network, W1Representing neuron weights, b representing bias;
inputting the long-term characteristics of the electroencephalogram sample into a forward propagation module, and outputting an emotion classification result of the electroencephalogram signal;
and step 3: and training the emotion classification model based on the electroencephalogram signals, and obtaining a final emotion classification model after the training is finished, wherein the final emotion classification model is used for carrying out emotion judgment on the input electroencephalogram signals.
2. The attention mechanism-based end-to-end electroencephalogram emotion recognition method as recited in claim 1, wherein the step of slicing the electroencephalogram signal is as follows:
setting 3 different sample lengths, dividing the samples into different slice lengths and window shifts in the samples, and processing and slicing the samples as dynamic sequences; then, calculating power spectral density and differential entropy characteristics of each lead in the sample slice for 5 different wave bands respectively; and (3) putting the characteristic values of all leads on the same wave band into a 9 x 9 two-dimensional square matrix according to the distribution mode of the leads on the electroencephalogram acquisition equipment, stacking the 5 lead characteristic square matrices according to the wave bands, and splicing the 5 lead characteristic square matrices into a three-dimensional 5 x 9 matrix as the input of the emotion classification model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110584519.9A CN113297981B (en) | 2021-05-27 | 2021-05-27 | End-to-end electroencephalogram emotion recognition method based on attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110584519.9A CN113297981B (en) | 2021-05-27 | 2021-05-27 | End-to-end electroencephalogram emotion recognition method based on attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113297981A true CN113297981A (en) | 2021-08-24 |
| CN113297981B CN113297981B (en) | 2023-04-07 |
Family
ID=77325577
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110584519.9A Active CN113297981B (en) | 2021-05-27 | 2021-05-27 | End-to-end electroencephalogram emotion recognition method based on attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113297981B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113616209A (en) * | 2021-08-25 | 2021-11-09 | 西南石油大学 | Schizophrenia patient discrimination method based on space-time attention mechanism |
| CN114224342A (en) * | 2021-12-06 | 2022-03-25 | 南京航空航天大学 | Multi-channel electroencephalogram emotion recognition method based on space-time fusion feature network |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109770925A (en) * | 2019-02-03 | 2019-05-21 | 闽江学院 | A kind of fatigue detection method based on depth time-space network |
| CN109793528A (en) * | 2019-01-28 | 2019-05-24 | 华南理工大学 | A kind of mood classification method based on dynamic brain function network |
| KR20190130808A (en) * | 2018-05-15 | 2019-11-25 | 연세대학교 산학협력단 | Emotion Classification Device and Method using Convergence of Features of EEG and Face |
| CN110515456A (en) * | 2019-08-14 | 2019-11-29 | 东南大学 | EEG signals emotion method of discrimination and device based on attention mechanism |
| CN110610168A (en) * | 2019-09-20 | 2019-12-24 | 合肥工业大学 | Electroencephalogram emotion recognition method based on attention mechanism |
| CN110897648A (en) * | 2019-12-16 | 2020-03-24 | 南京医科大学 | Emotion recognition classification method based on electroencephalogram signal and LSTM neural network model |
| CN111222464A (en) * | 2020-01-07 | 2020-06-02 | 中国医学科学院生物医学工程研究所 | Emotion analysis method and system |
| CN111259761A (en) * | 2020-01-13 | 2020-06-09 | 东南大学 | Electroencephalogram emotion recognition method and device based on migratable attention neural network |
| CN111353390A (en) * | 2020-01-17 | 2020-06-30 | 道和安邦（天津）安防科技有限公司 | Micro-expression recognition method based on deep learning |
| CN111709267A (en) * | 2020-03-27 | 2020-09-25 | 吉林大学 | Electroencephalogram signal emotion recognition method of deep convolutional neural network |
| CN112101152A (en) * | 2020-09-01 | 2020-12-18 | 西安电子科技大学 | Electroencephalogram emotion recognition method and system, computer equipment and wearable equipment |
| CN112244873A (en) * | 2020-09-29 | 2021-01-22 | 陕西科技大学 | Electroencephalogram time-space feature learning and emotion classification method based on hybrid neural network |
| CN112800998A (en) * | 2021-02-05 | 2021-05-14 | 南京邮电大学 | Multi-mode emotion recognition method and system integrating attention mechanism and DMCCA |
-
2021
- 2021-05-27 CN CN202110584519.9A patent/CN113297981B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190130808A (en) * | 2018-05-15 | 2019-11-25 | 연세대학교 산학협력단 | Emotion Classification Device and Method using Convergence of Features of EEG and Face |
| CN109793528A (en) * | 2019-01-28 | 2019-05-24 | 华南理工大学 | A kind of mood classification method based on dynamic brain function network |
| CN109770925A (en) * | 2019-02-03 | 2019-05-21 | 闽江学院 | A kind of fatigue detection method based on depth time-space network |
| CN110515456A (en) * | 2019-08-14 | 2019-11-29 | 东南大学 | EEG signals emotion method of discrimination and device based on attention mechanism |
| CN110610168A (en) * | 2019-09-20 | 2019-12-24 | 合肥工业大学 | Electroencephalogram emotion recognition method based on attention mechanism |
| CN110897648A (en) * | 2019-12-16 | 2020-03-24 | 南京医科大学 | Emotion recognition classification method based on electroencephalogram signal and LSTM neural network model |
| CN111222464A (en) * | 2020-01-07 | 2020-06-02 | 中国医学科学院生物医学工程研究所 | Emotion analysis method and system |
| CN111259761A (en) * | 2020-01-13 | 2020-06-09 | 东南大学 | Electroencephalogram emotion recognition method and device based on migratable attention neural network |
| CN111353390A (en) * | 2020-01-17 | 2020-06-30 | 道和安邦（天津）安防科技有限公司 | Micro-expression recognition method based on deep learning |
| CN111709267A (en) * | 2020-03-27 | 2020-09-25 | 吉林大学 | Electroencephalogram signal emotion recognition method of deep convolutional neural network |
| CN112101152A (en) * | 2020-09-01 | 2020-12-18 | 西安电子科技大学 | Electroencephalogram emotion recognition method and system, computer equipment and wearable equipment |
| CN112244873A (en) * | 2020-09-29 | 2021-01-22 | 陕西科技大学 | Electroencephalogram time-space feature learning and emotion classification method based on hybrid neural network |
| CN112800998A (en) * | 2021-02-05 | 2021-05-14 | 南京邮电大学 | Multi-mode emotion recognition method and system integrating attention mechanism and DMCCA |
Non-Patent Citations (5)
| Title |
|---|
| ANUBHAV 等: "An Efficient Approach to EEG-Based Emotion Recognition using LSTM Network", 《CSPA 2020》 * |
| TONG ZHANG 等: "Spatial–Temporal Recurrent Neural Network for Emotion Recognition", 《IEEE TRANSACTIONS ON CYBERNETICS》 * |
| XIAOBIN ZHENG: "Research on Video Emotion Recognition Based on Attention Mechanism LSTM Model", 《NCCE 2018》 * |
| 汤宇豪 等: "基于层次注意力机制的维度情感识别方法", 《计算机工程》 * |
| 陶威: "基于注意力机制的脑电情绪识别方法研究", 《中国优秀硕士学位论文全文数据库 医药卫生科技辑(月刊)》 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113616209A (en) * | 2021-08-25 | 2021-11-09 | 西南石油大学 | Schizophrenia patient discrimination method based on space-time attention mechanism |
| CN113616209B (en) * | 2021-08-25 | 2023-08-04 | 西南石油大学 | Method for screening schizophrenic patients based on space-time attention mechanism |
| CN114224342A (en) * | 2021-12-06 | 2022-03-25 | 南京航空航天大学 | Multi-channel electroencephalogram emotion recognition method based on space-time fusion feature network |
| CN114224342B (en) * | 2021-12-06 | 2023-12-15 | 南京航空航天大学 | Multichannel electroencephalogram signal emotion recognition method based on space-time fusion feature network |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113297981B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112244873A (en) | Electroencephalogram time-space feature learning and emotion classification method based on hybrid neural network | |
| Kollias et al. | Training deep neural networks with different datasets in-the-wild: The emotion recognition paradigm | |
| CN113297981B (en) | End-to-end electroencephalogram emotion recognition method based on attention mechanism | |
| Asadi et al. | A convolution recurrent autoencoder for spatio-temporal missing data imputation | |
| CN114052735B (en) | Deep field self-adaption-based electroencephalogram emotion recognition method and system | |
| CN113749657B (en) | Brain electricity emotion recognition method based on multi-task capsule | |
| CN110135244B (en) | Expression recognition method based on brain-computer collaborative intelligence | |
| CN111079658A (en) | Video-based multi-target continuous behavior analysis method, system and device | |
| CN113288146A (en) | Electroencephalogram emotion classification method based on time-space-frequency combined characteristics | |
| Lu et al. | Multi-object detection method based on YOLO and ResNet hybrid networks | |
| Shao et al. | EEG-based emotion recognition with deep convolution neural network | |
| CN112507893A (en) | Distributed unsupervised pedestrian re-identification method based on edge calculation | |
| CN114578967A (en) | Emotion recognition method and system based on electroencephalogram signals | |
| CN114488069A (en) | Radar high-resolution range profile identification method based on graph neural network | |
| CN107045624A (en) | A kind of EEG signals pretreatment rolled into a ball based on maximum weighted and sorting technique | |
| CN113558644A (en) | Emotion classification method, medium and equipment for 3D matrix and multidimensional convolution network | |
| CN116956222A (en) | Multi-complexity behavior recognition system and method based on self-adaptive feature extraction | |
| CN115801152B (en) | WiFi action recognition method based on hierarchical transformer model | |
| Uddin et al. | A convolutional neural network for real-time face detection and emotion & gender classification | |
| CN116421200A (en) | Brain electricity emotion analysis method of multi-task mixed model based on parallel training | |
| Afrin et al. | AI based facial expression recognition for autism children | |
| Jayakumar et al. | CNN based Music Recommendation system based on Age, Gender and Emotion | |
| CN114742107A (en) | Method for identifying perception signal in information service and related equipment | |
| Jamal et al. | Cloud-Based Human Emotion Classification Model from EEG Signals | |
| Jyoti et al. | A single hierarchical network for face, action unit and emotion detection |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |