CN116861211A - Electroencephalogram emotion recognition method and system integrating space-time interaction neural network - Google Patents
Electroencephalogram emotion recognition method and system integrating space-time interaction neural network Download PDFInfo
- Publication number
- CN116861211A CN116861211A CN202310674320.4A CN202310674320A CN116861211A CN 116861211 A CN116861211 A CN 116861211A CN 202310674320 A CN202310674320 A CN 202310674320A CN 116861211 A CN116861211 A CN 116861211A
- Authority
- CN
- China
- Prior art keywords
- time
- emotion
- space
- electroencephalogram
- emotion recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 60
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000003993 interaction Effects 0.000 title claims abstract description 37
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 26
- 230000008451 emotion Effects 0.000 claims abstract description 56
- 239000011159 matrix material Substances 0.000 claims abstract description 8
- 238000010586 diagram Methods 0.000 claims description 27
- 230000006870 function Effects 0.000 claims description 25
- 238000012549 training Methods 0.000 claims description 25
- 210000004556 brain Anatomy 0.000 claims description 24
- 238000012360 testing method Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 13
- 210000002569 neuron Anatomy 0.000 claims description 12
- 238000010606 normalization Methods 0.000 claims description 10
- 230000004913 activation Effects 0.000 claims description 6
- 230000005611 electricity Effects 0.000 claims description 5
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 4
- 239000000470 constituent Substances 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 230000002452 interceptive effect Effects 0.000 claims description 3
- 230000004927 fusion Effects 0.000 claims 3
- 230000003213 activating effect Effects 0.000 claims 1
- 230000002708 enhancing effect Effects 0.000 abstract description 5
- 230000002596 correlated effect Effects 0.000 abstract description 3
- PIZHFBODNLEQBL-UHFFFAOYSA-N 2,2-diethoxy-1-phenylethanone Chemical compound CCOC(OCC)C(=O)C1=CC=CC=C1 PIZHFBODNLEQBL-UHFFFAOYSA-N 0.000 description 6
- 238000013135 deep learning Methods 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 5
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000013136 deep learning model Methods 0.000 description 4
- 230000036651 mood Effects 0.000 description 4
- 230000008569 process Effects 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 238000002679 ablation Methods 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 230000002996 emotional effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000037007 arousal Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000000875 corresponding effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 210000003205 muscle Anatomy 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000003997 social interaction Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Animal Behavior & Ethology (AREA)
- Evolutionary Biology (AREA)
- Medical Informatics (AREA)
- Heart & Thoracic Surgery (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Pathology (AREA)
- Surgery (AREA)
- Psychology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Social Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Educational Technology (AREA)
- Developmental Disabilities (AREA)
- Child & Adolescent Psychology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The application relates to the technical field of emotion recognition, in particular to an electroencephalogram emotion recognition method integrating space-time interaction neural networks, which comprises the following steps: inputting electroencephalogram data into an electroencephalogram emotion recognition model, and constructing a 2D matrix representation of the electroencephalogram according to the position of an electroencephalogram acquisition electrode; capturing common characteristics of a plurality of continuous time steps from a low-dimensional layer and a high-dimensional layer through time step sharing convolution, and enhancing the information expression of the original input; by downsampling convolution, the time dimension is reduced and time-space features highly correlated to emotion categories are extracted; information interaction among a plurality of subspace regions is realized through a space interaction multi-layer perceptron, and the space dependence of the features is enhanced; and constructing emotion classification tasks, obtaining the probability of emotion categories in each task, and completing emotion recognition. The application is based on fewer electroencephalogram acquisition electrodes, enhances the time expression and potential space dependence of electroencephalogram signals, and further improves the classification precision and generalization capability of electroencephalogram emotion recognition.
Description
Technical Field
The application relates to the technical field of emotion recognition, in particular to an electroencephalogram emotion recognition method and system integrating space-time interaction neural networks.
Background
Emotion is one of the highest levels of cognitive activity in humans, directly affecting aspects of public daily life, including learning, decision making, and social interactions. The physiological signals truly and objectively reflect the emotional state of a person, wherein brain waves (EEG) directly record the signals of brain cortex-related mood-related neural activity. In the emotion recognition task, the brain electrical signal can accurately reflect the emotional state of the human. With the rapid development of brain-computer interface technology, electroencephalogram signals are widely applied to emotion recognition at present.
The electroencephalogram signal is a time sequence of multiple channels and multiple electrodes, so that the electroencephalogram signal has the characteristics of time, space, frequency spectrum and the like. In the prior art, methods such as convolutional neural networks, recurrent neural networks, and Transformer architectures, which are popular in deep learning, have been successfully applied to electroencephalogram emotion recognition. However, convolutional neural networks can effectively capture local information, but ignore global information; the recurrent neural network and transducer architecture can effectively mine the time information of the EEG, but ignores the spatial information.
Therefore, the existing deep learning-based electroencephalogram emotion recognition method has a defect of utilizing space information and time information of electroencephalogram signals, so that emotion recognition accuracy is limited, complexity of a model is increased, and arrangement and application in a real scene are not utilized. Thus, in the electroencephalogram emotion recognition method, how to adequately capture the temporal and spatial features of the EEG remains a challenge to be solved.
Disclosure of Invention
Therefore, the technical problem to be solved by the application is to overcome the defects of the prior art that the deep learning-based electroencephalogram emotion recognition method has the defects of space information and time information of the electroencephalogram, so that the emotion recognition precision is limited, the complexity of a model is increased, and the problems of arrangement and application in a real scene are not utilized.
In order to solve the technical problems, the application provides an electroencephalogram emotion recognition method integrating space-time interaction neural networks, which comprises the following specific steps:
s1, inputting electroencephalogram data into an electroencephalogram emotion recognition model, and according to the positions of electroencephalogram acquisition electrodes, putting the electroencephalogram acquired by each electroencephalogram electrode into a 2D matrix of h multiplied by w to construct a 2D matrix representation of the electroencephalogram; dividing the electroencephalogram signal of each channel into samples with the time length T according to the time window length T to obtain the size of each sample as T multiplied by h multiplied by w, and carrying out 0-1 normalization on the electroencephalogram data of each channel;
s2, dividing each sample into M continuous time steps according to the time dimension, and marking the continuous time steps as P t ，t∈[1,M]The method comprises the steps of carrying out a first treatment on the surface of the Will P t Respectively sequentially inputting to a time step sharing convolution, capturing common features of a plurality of continuous time steps from a low-dimensional layer and a high-dimensional layer to obtain common features F of continuous M time steps in the sample t The method comprises the steps of carrying out a first treatment on the surface of the Splicing the common features F according to the time dimension t Taking the sample as a residual error, and adding the residual error and the spliced characteristic to obtain a characteristic MF;
s3, constructing a downsampling convolution, inputting the feature MF into the downsampling convolution, reducing the time dimension of data, and extracting the time-space feature highly related to emotion typesObtaining a time-space characteristic diagram C epsilon R (T/8)×h×w ；
S4, operating the space dimension of the time-space characteristic diagram C to divide the time-space characteristic diagram C into N sub-areas with the size of p multiplied by p, wherein the sub-areas are expressed asWhere i=1, 2,..n, j=1, 2,., T/8; according to the time dimension of the time-space feature map C, (T/8) feature maps are spliced together and are expressed as C i ∈R 1×[(T/8)×p×p] Therefore, N spliced feature maps C i Final constituent feature map C 0 ∈R N×[(T/8 ) ×p×p] ；
Map the characteristic diagram C 0 Transposing, and performing multi-layer perceptron operation on each transposed sub-region to enable spatial information interaction between each sub-region, further enhancing spatial expression of features, and obtaining a feature map C 1 The method comprises the steps of carrying out a first treatment on the surface of the Again for the characteristic diagram C 1 Transposed to obtain a feature map C 2 The characteristic diagram C is processed 0 As residual and feature map C 2 Adding and carrying out layer normalization to obtain a characteristic diagram C A ；
S5, constructing three emotion classification tasks according to the valence, awakening and dominant three dimensions of emotion, wherein each emotion classification task comprises two categories; using full connection layer to the feature map C A And respectively carrying out two classification in three emotion classification tasks to obtain the probability of emotion classification in each task, and completing emotion recognition.
In one embodiment of the present application, in S1, the electroencephalogram data selects 14 channels to be input into an electroencephalogram emotion recognition model, including channels AF3, AF4, F7, F3, F4, F8, FC5, FC6, T7, T8, P7, P8, O1, O2.
In one embodiment of the present application, in S2, the time-step shared convolution includes two 2D convolution layers, a first 2D convolution reducing the time dimension and a second 2D convolution enlarging the time dimension to an initial size.
In one embodiment of the application, in S3, the downsampling convolution comprises 3 2D convolution layers, each 2D convolution layer reduces the temporal dimension of the feature MF to 1/2 of the initial state, leaves the spatial dimension unchanged, and simultaneously extracts the temporal-spatial feature highly correlated to the emotion classification.
In one embodiment of the application, each of the 2D convolution layers is provided with an activation function, a batch normalization layer, and a Dropout layer.
In one embodiment of the application, the multi-layered perceptron includes two fully connected layers and a GELU activation function.
In one embodiment of the application, in S5, the feature map C is used with a full connection layer A When two classification is respectively carried out in three classification tasks, the emotion category with the highest probability is calculated by using a Softmax function, and the formula is as follows:
wherein x is i Representing the output value of the ith neuron, x c The output value of the C-th neuron is represented, C is the number of neurons,representing the output values of all neurons.
In one embodiment of the present application, the step of training the electroencephalogram emotion recognition model includes:
collecting brain electrical information of a subject, constructing an brain electrical emotion data set, enabling the subject to evaluate emotion of the subject, and scoring from three dimensions of valence, awakening and domination of the emotion, so as to determine a real label of emotion type; dividing the electroencephalogram emotion data set into a training set and a testing set according to a preset proportion;
inputting electroencephalogram signal data of the training set into an electroencephalogram emotion recognition model to obtain probability of emotion categories in each task; during training, the cross entropy loss function is used for minimizing the difference between the classification result and the real label, and the Adam optimizer is used for optimizing the loss function and updating the learning rate;
and performing performance evaluation on the electroencephalogram emotion recognition model by using classification accuracy, standard deviation, training and test time on a test set.
In one embodiment of the present application, the cross entropy loss function L is formulated as:
where N represents the number of samples and M represents the number of categories; if the classification category is the same as the true category, y ic 1, otherwise 0; p is p ic Representing the probability that sample i belongs to category c.
The application also provides an electroencephalogram emotion recognition system fusing the space-time interaction neural network, which comprises the following steps:
a memory for storing a computer program;
and the processor is used for realizing the step of the electroencephalogram emotion recognition method fusing the space-time interaction neural network when executing the computer program.
Compared with the prior art, the technical scheme of the application has the following advantages:
according to the electroencephalogram emotion recognition method integrating the space-time interaction neural network, the time step sharing convolution is used, the time dimension is divided into a plurality of continuous time steps, common features of the continuous time steps are captured from the low-dimension layer and the high-dimension layer and spliced, and the time expression of electroencephalogram signals is enhanced. And the space interaction multi-layer perceptron is used for dividing the feature map from the space dimension and splicing the feature map from the time dimension so as to perform information interaction among a plurality of subspace regions, further strengthen the space expression of the features and strengthen the space dependence of the electroencephalogram signals.
Therefore, the method is based on a time-step shared convolution and space interaction multi-layer perceptron, so that the time and space characteristics of the electroencephalogram emotion characteristics are further mined, the problem that the utilization of the electroencephalogram space information and the time information is insufficient in the existing electroencephalogram emotion recognition method is solved to a certain extent, and the method has the advantages of being high in calculation efficiency, high in recognition precision and strong in generalization capability.
In addition, fewer electroencephalogram acquisition electrodes are used, so that the training and testing time of the model is shortened while the recognition accuracy is ensured, and the realizability of the model in a practical application scene is improved.
Drawings
In order that the application may be more readily understood, a further description of the application will be rendered by reference to specific embodiments thereof that are illustrated in the appended drawings, in which
FIG. 1 is a flow chart of an electroencephalogram emotion recognition method fused with a space-time interaction neural network;
FIG. 2 is an overall frame diagram of an electroencephalogram emotion recognition method fused with a space-time interaction neural network;
FIG. 3 is a schematic diagram of a specific method for constructing a 2D matrix representation of an electroencephalogram signal in the present application;
FIG. 4 is a schematic illustration of the spatial zone division of the present application;
FIG. 5 is a block diagram of a spatially interactive multilayer perceptron of the present application;
fig. 6 is a comparative graph of the results of an ablation experiment in an embodiment of the present application.
Detailed Description
The present application will be further described with reference to the accompanying drawings and specific examples, which are not intended to be limiting, so that those skilled in the art will better understand the application and practice it.
Referring to fig. 1 and 2, the application discloses an electroencephalogram emotion recognition method fusing a space-time interaction neural network, which comprises the following specific steps:
s1, inputting electroencephalogram data into an electroencephalogram emotion recognition model, and referring to FIG. 3, 14 channels which are highly related to emotion are selected, wherein the 14 channels comprise channels AF3, AF4, F7, F3, F4, F8, FC5, FC6, T7, T8, P7, P8, O1 and O2, and the aim is to reduce the number of the channels so as to improve the realizability in an actual application scene. And carrying out 0.1-45hz band-pass filtering on the electroencephalogram signals, and then placing the electroencephalogram signals acquired by each electroencephalogram electrode into a 2D matrix of h multiplied by w according to the positions of the electroencephalogram acquisition electrodes to construct a 2D matrix representation of the electroencephalogram signals.
In this embodiment, the time window T takes 1s. Dividing the electroencephalogram signals of each channel into samples with the time length T according to the time window length T, namely, arranging the electroencephalogram signals of all time nodes into T h×w 2D matrixes according to the time sequence, obtaining the size of each sample as T×h×w, and carrying out 0-1 normalization on the electroencephalogram data of each channel.
S2, extracting common features of the continuous time steps of the sample through time step sharing convolution, capturing the common features of a plurality of continuous time steps from the low-dimensional and high-dimensional layers, and enhancing the information expression of the original input.
Dividing each sample into M continuous time steps according to the time dimension, and marking the time steps as P t Wherein t is [1, M ]]. Will P t Respectively sequentially inputting to a time step sharing convolution, capturing common features of a plurality of continuous time steps from a low-dimensional layer and a high-dimensional layer to obtain common features F of continuous M time steps in the sample t . Splicing the common features F according to the time dimension t And taking the sample as a residual error, adding the residual error with the spliced characteristic, and enhancing the information expression of the original input to obtain the characteristic MF.
The time-step shared convolution includes two 2D convolution layers, a first 2D convolution reducing the time dimension and a second 2D convolution enlarging the time dimension to an initial size. Each 2D convolution layer is provided with an activation function, a batch normalization layer and a Dropout layer.
S3, constructing downsampling convolution, reducing the time dimension, and extracting time-space characteristics highly relevant to emotion categories.
Inputting the feature MF into downsampling convolution, reducing the time dimension of data, and extracting time-space features highly related to emotion categories to obtain a time-space feature map C epsilon R (T/8)×h×w 。
The downsampling convolution comprises 3 2D convolution layers, each 2D convolution layer reduces the time dimension of the feature MF to 1/2 of the initial state, keeps the space dimension unchanged, and simultaneously extracts time-space features highly correlated to emotion categories. Each 2D convolution layer is provided with an activation function, a batch normalization layer and a Dropout layer.
S4, space region division is carried out on the time-space feature map C, a space interaction multi-layer perceptron is constructed, information interaction in a plurality of subspace regions is achieved, and the space dependence of the features is enhanced.
And carrying out space region division on the time-space characteristic diagram C. Referring to FIG. 4, the spatial dimension of the time-space feature map C is operated on to divide it into N sub-regions of p size, denoted asWhere i=1, 2,..n, j=1, 2,..t/8. According to the time dimension of the time-space feature map C, (T/8) feature maps are spliced together and are expressed as C i ∈R 1×[(T/8)×p×p] . Thus, N spliced feature maps C i Final constituent feature map C 0 ∈R N ×[(T/8)×p×p] 。
And constructing a space interaction multi-layer perceptron. Referring to FIG. 5, the characteristic diagram C 0 Transposing, and performing multi-layer perceptron operation on each transposed sub-region, namely performing spatial information interaction between each sub-region, further enhancing spatial expression of the features, and obtaining a feature map C 1 . The multi-layer perceptron comprises two full connection layers and a GELU activation function. Again for the characteristic diagram C 1 Transposed to obtain a feature map C 2 The characteristic diagram C is processed 0 As residual and feature map C 2 Adding and carrying out layer normalization to obtain a characteristic diagram C with enhanced spatial dependence A 。
S5, constructing emotion classification tasks, obtaining the probability of emotion categories in each task, and completing emotion recognition.
According to the valence, awakening and dominant three dimensions of emotion, three emotion classification tasks are constructed, and each emotion classification task comprises two categories; using full connection layer to the feature map C A And respectively carrying out two classification in three emotion classification tasks to obtain the probability of emotion classification in each task, and completing emotion recognition.
Using full connection layer to the feature map C A When two classification is respectively carried out in three classification tasks, the emotion category with the highest probability is calculated by using a Softmax function, and the formula is as follows:
wherein x is i Representing the output value of the ith neuron, x c The output value of the C-th neuron is represented, C is the number of neurons,representing the output values of all neurons.
In order to solve the problems that in the existing electroencephalogram emotion recognition method, the time features and the space features of electroencephalogram data are not captured sufficiently, so that the recognition accuracy is limited, the complexity of a model is increased, and arrangement and application in a real scene are not utilized, the method further excavates the time and the space features of the emotion features in the electroencephalogram data, the time expression of the electroencephalogram is increased by capturing common features of continuous time steps, the space dependence of the electroencephalogram is enhanced by local space region interaction of the electroencephalogram emotion features, the classification performance of the electroencephalogram emotion recognition model is improved, and the accurate recognition of the electroencephalogram emotion is realized. In addition, the application is based on fewer electroencephalogram acquisition electrodes, and the realizability of the application in the actual application scene is improved.
The application also provides another specific embodiment for clarifying the process of training the electroencephalogram emotion recognition model, and comparing the training result with the current advanced deep learning algorithms MLF-CapsNet, NAS and ECNN-C to further verify the beneficial effects of the application.
The electroencephalogram data for training the electroencephalogram emotion recognition model can be an existing electroencephalogram data set based on emotion recognition recorded in literature data, or can be electroencephalogram information of a tested person collected from a site.
The embodiment uses the DEAP brain electrical emotion data set to train the brain electrical emotion recognition model. The dataset was collected from 32 subjects, with a 50% male and female ratio each, aged 19-37 years. The dataset recorded brain electrical signals, which were recorded primarily by 32 electrode channels, of 40 60s of music video viewed by each subject, along with peripheral physiological signals, with a baseline signal of 3s before viewing the video. After viewing the video, the subject scored 1 to 9 from three dimensions of mood, arousal and dominance to evaluate his mood and determine the true signature of the mood category.
The original brain electrical signal recorded mainly 32 electrode channels, sampled at 512Hz, and passed through noise of muscle movements, 4-45Hz band pass filter and 128Hz downsampling.
Dividing the electroencephalogram emotion data set into a training set and a testing set according to a preset proportion.
And inputting the electroencephalogram data of the training set into the electroencephalogram emotion recognition model to obtain the probability of emotion categories in each task.
The cross entropy loss function is used for minimizing the difference between the classification result and the corresponding real label during training, and the Adam optimizer is used for optimizing the loss function and updating the learning rate.
Using the cross entropy as a loss function, calculating a loss index in the training process, wherein the loss function L is defined as:
where N represents the number of samples and M represents the number of categories; if the classification category is the same as the true category, y ic 1, otherwise 0; p is p ic Representing the probability that sample i belongs to category c.
And finally, performing performance evaluation on the electroencephalogram emotion recognition model by using classification accuracy, standard deviation, training and test time as indexes on a test set.
To further illustrate the benefits of the present application, in this embodiment, the method of the present application is compared with the current advanced deep learning algorithms MLF-CapsNet, NAS and ECNN-C.
Parameters set in the experiment include: adam optimizer is used to minimize class cross-over entropy loss function with a learning rate of 5e-4. The number of heads of the multi-head self-attention mechanism was set to 4, and a total of 80 epochs were trained. When training is completed for 5 epochs and loss is not reduced, the learning rate is reduced by 80%, the value of Dropout takes 0.3, and an early-stop strategy is used in the training process.
The classification result of the method of the application is compared with an advanced deep learning algorithm based on DEAP emotion recognition brain electricity data set, and the specific result is shown in table 1.
TABLE 1 classification results of DEAP-based different methods for emotion recognition of brain electrical data sets
Table 1 shows the average classification accuracy and standard deviation over 32 subjects for the current advanced deep learning model and DEAP-based emotion recognition electroencephalogram dataset of the method of the present application. The application can be found that the highest average classification precision of 98.56%,98.70% and 98.63% are obtained in all three classification tasks, and meanwhile, the lower standard deviation is also obtained. Compared with the current advanced deep learning model, the method provided by the application improves the classification performance of the electroencephalogram emotion recognition model and realizes more accurate recognition of the electroencephalogram emotion.
Meanwhile, in the embodiment, the effectiveness analysis is carried out on the time-step sharing convolution and space interaction multi-layer perceptron by using an ablation experiment method.
The model containing only downsampled convolutions was designated as baseline and was named Base-CNN. The results of the ablation experiments are shown in FIG. 6, which shows the average classification accuracy and standard deviation of Base-CNN and the application in three classification tasks. Compared with a baseline model, the model provided by the application has higher classification precision and lower standard deviation, and proves that the time-step sharing convolution and space interaction multi-layer perceptron can more comprehensively capture the time characteristics and the space characteristics of the electroencephalogram signals, and further improve the classification precision.
Experimental results show that the electroencephalogram emotion recognition method based on the time-step sharing convolution and space interaction multi-layer perceptron has good classification performance and strong generalization capability.
In addition, we also compare the training and testing time of the method of the present application with the current advanced deep learning model, and the specific results are shown in table 2.
TABLE 2 training and test time for different methods based on DEAP dataset
Table 2 shows training and testing times of the present advanced deep learning model and the method of the present application on the DEAP dataset. It can be found that the method of the application has shorter training and testing time, which makes it possible to apply the method to actual scenes.
The experiment shows that the application realizes accurate identification of the brain electricity emotion, improves the classification performance and the realizability in the practical application scene, and can provide reference for the development of the brain electricity emotion identification technical field.
The specific embodiment of the application also provides an electroencephalogram emotion recognition system fusing the space-time interaction neural network, which comprises the following steps:
a memory for storing a computer program;
and the processor is used for realizing the step of the electroencephalogram emotion recognition method fusing the space-time interaction neural network when executing the computer program.
It will be appreciated by those skilled in the art that embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It is apparent that the above examples are given by way of illustration only and are not limiting of the embodiments. Other variations and modifications of the present application will be apparent to those of ordinary skill in the art in light of the foregoing description. It is not necessary here nor is it exhaustive of all embodiments. And obvious variations or modifications thereof are contemplated as falling within the scope of the present application.
Claims (10)
1. The brain electricity emotion recognition method integrating the space-time interaction neural network is characterized by comprising the following specific steps of:
s1, inputting electroencephalogram data into an electroencephalogram emotion recognition model, and according to the positions of electroencephalogram acquisition electrodes, putting the electroencephalogram acquired by each electroencephalogram electrode into a 2D matrix of h multiplied by w to construct a 2D matrix representation of the electroencephalogram; dividing the electroencephalogram signal of each channel into samples with the time length T according to the time window length T to obtain the size of each sample as T multiplied by h multiplied by w, and carrying out 0-1 normalization on the electroencephalogram data of each channel;
s2, dividing each sample into M continuous time steps according to the time dimension, and marking the continuous time steps as P t ，t∈[1,M]The method comprises the steps of carrying out a first treatment on the surface of the Will P t Respectively sequentially inputting to a time step sharing convolution, capturing common features of a plurality of continuous time steps from a low-dimensional layer and a high-dimensional layer to obtain common features F of continuous M time steps in the sample t The method comprises the steps of carrying out a first treatment on the surface of the Splicing the common features F according to the time dimension t Taking the sample as a residual error, and adding the residual error and the spliced characteristic to obtain a characteristic MF;
s3, constructing a downsampling convolution, inputting the feature MF into the downsampling convolution, reducing the time dimension of data, extracting time-space features highly related to emotion categories, and obtaining a time-space feature map C epsilon R (T/8)×h×w ；
S4, operating the space dimension of the time-space characteristic diagram C to divide the time-space characteristic diagram C into N sub-areas with the size of p multiplied by p, wherein the sub-areas are expressed asWhere i=1, 2,..n, j=1, 2,., T/8; according to the time dimension of the time-space feature map C, (T/8) feature maps are spliced together and are expressed as C i ∈R 1×[(T/8)×p×p] Therefore, N spliced feature maps C i Final constituent feature map C 0 ∈R N×[T/8)×p×p] ；
Map the characteristic diagram C 0 Transpose and pair transposeThen each sub-region is operated by a multi-layer perceptron to enable the space information interaction between each sub-region to further strengthen the space expression of the features and obtain a feature map C 1 The method comprises the steps of carrying out a first treatment on the surface of the Again for the characteristic diagram C 1 Transposed to obtain a feature map C 2 The characteristic diagram C is processed 0 As residual and feature map C 2 Adding and carrying out layer normalization to obtain a characteristic diagram C A ；
S5, constructing three emotion classification tasks according to the valence, awakening and dominant three dimensions of emotion, wherein each emotion classification task comprises two categories; using full connection layer to the feature map C A And respectively carrying out two classification in three emotion classification tasks to obtain the probability of emotion classification in each task, and completing emotion recognition.
2. The brain-electrical emotion recognition method based on the fusion space-time interaction neural network according to claim 1, wherein in S1, 14 channels are selected for inputting brain-electrical signal data to a brain-electrical emotion recognition model, and the brain-electrical emotion recognition model comprises channels AF3, AF4, F7, F3, F4, F8, FC5, FC6, T7, T8, P7, P8, O1 and O2.
3. The method for recognizing brain waves fused with space-time interactive neural network according to claim 1, wherein in S2, the time-step sharing convolution comprises two 2D convolution layers, the first 2D convolution reduces the time dimension, and the second 2D convolution enlarges the time dimension to the initial size.
4. The method for recognizing brain waves fused with space-time interactive neural network according to claim 1, wherein in S3, the downsampling convolution comprises 3 2D convolution layers, each 2D convolution layer reduces the time dimension of the feature MF to 1/2 of the initial state, keeps the space dimension unchanged, and simultaneously extracts the time-space feature highly related to the emotion classification.
5. The method for recognizing brain electric emotion integrating space-time interaction neural network according to any one of claims 3 and 4, wherein each 2D convolution layer is provided with an activation function, a batch normalization layer and a Dropout layer.
6. The method for recognizing brain electricity emotion fusing space-time interaction neural network as claimed in claim 1, wherein in S4, said multi-layer perceptron comprises two full connection layers and a gel activating function.
7. The brain-electrical emotion recognition method based on the fusion of space-time interaction neural network according to claim 1, wherein in S5, the feature map C is obtained by using a full connection layer A When two classification is respectively carried out in three classification tasks, the emotion category with the highest probability is calculated by using a Softmax function, and the formula is as follows:
wherein x is i Representing the output value of the u-th neuron, x c The output value of the C-th neuron is represented, C is the number of neurons,representing the output values of all neurons.
8. The method for recognizing brain waves by fusing a space-time interaction neural network according to claim 1, wherein the step of training the brain wave recognition model comprises the steps of:
collecting brain electrical information of a subject, constructing an brain electrical emotion data set, enabling the subject to evaluate emotion of the subject, and scoring from three dimensions of valence, awakening and domination of the emotion, so as to determine a real label of emotion type; dividing the electroencephalogram emotion data set into a training set and a testing set according to a preset proportion;
inputting electroencephalogram signal data of the training set into an electroencephalogram emotion recognition model to obtain probability of emotion categories in each task; during training, the cross entropy loss function is used for minimizing the difference between the classification result and the real label, and the Adam optimizer is used for optimizing the loss function and updating the learning rate;
and performing performance evaluation on the electroencephalogram emotion recognition model by using classification accuracy, standard deviation, training and test time on a test set.
9. The brain electrical emotion recognition method based on the fusion space-time interaction neural network of claim 8, wherein the cross entropy loss function L formula is:
where N represents the number of samples and M represents the number of categories; if the classification category is the same as the true category, y ic 1, otherwise 0; p is p ic Representing the probability that sample i belongs to category c.
10. An electroencephalogram emotion recognition system integrating space-time interaction neural networks, which is characterized by comprising:
a memory for storing a computer program;
a processor for implementing the steps of a method for identifying brain waves fusing space-time interaction neural networks according to any one of claims 1 to 9 when executing the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310674320.4A CN116861211B (en) | 2023-06-08 | 2023-06-08 | Electroencephalogram emotion recognition method and system integrating space-time interaction neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310674320.4A CN116861211B (en) | 2023-06-08 | 2023-06-08 | Electroencephalogram emotion recognition method and system integrating space-time interaction neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116861211A true CN116861211A (en) | 2023-10-10 |
| CN116861211B CN116861211B (en) | 2024-09-06 |
Family
ID=88218070
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310674320.4A Active CN116861211B (en) | 2023-06-08 | 2023-06-08 | Electroencephalogram emotion recognition method and system integrating space-time interaction neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116861211B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111297380A (en) * | 2020-02-12 | 2020-06-19 | 电子科技大学 | Emotion recognition method based on space-time convolution core block |
| EP3716152A1 (en) * | 2019-03-26 | 2020-09-30 | Robert Bosch GmbH | Reusing of neural network training for higher dimensional data |
| CN113128353A (en) * | 2021-03-26 | 2021-07-16 | 安徽大学 | Emotion sensing method and system for natural human-computer interaction |
| CN113723557A (en) * | 2021-09-08 | 2021-11-30 | 山东大学 | Depression electroencephalogram classification system based on multiband time-space convolution network |
| CN113749657A (en) * | 2021-09-10 | 2021-12-07 | 合肥工业大学 | Brain wave emotion recognition method based on multitask capsules |
| CN114209342A (en) * | 2022-01-28 | 2022-03-22 | 南京邮电大学 | Electroencephalogram signal motor imagery classification method based on space-time characteristics |
| CN114224342A (en) * | 2021-12-06 | 2022-03-25 | 南京航空航天大学 | Multi-channel electroencephalogram emotion recognition method based on space-time fusion feature network |
| CN115474899A (en) * | 2022-08-17 | 2022-12-16 | 浙江大学 | Basic taste perception identification method based on multi-scale convolution neural network |
| WO2023060721A1 (en) * | 2021-10-11 | 2023-04-20 | 北京工业大学 | Emotional state displaying method, apparatus and system based on resting-state cerebral functional image |
-
2023
- 2023-06-08 CN CN202310674320.4A patent/CN116861211B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3716152A1 (en) * | 2019-03-26 | 2020-09-30 | Robert Bosch GmbH | Reusing of neural network training for higher dimensional data |
| CN111297380A (en) * | 2020-02-12 | 2020-06-19 | 电子科技大学 | Emotion recognition method based on space-time convolution core block |
| CN113128353A (en) * | 2021-03-26 | 2021-07-16 | 安徽大学 | Emotion sensing method and system for natural human-computer interaction |
| CN113723557A (en) * | 2021-09-08 | 2021-11-30 | 山东大学 | Depression electroencephalogram classification system based on multiband time-space convolution network |
| CN113749657A (en) * | 2021-09-10 | 2021-12-07 | 合肥工业大学 | Brain wave emotion recognition method based on multitask capsules |
| WO2023060721A1 (en) * | 2021-10-11 | 2023-04-20 | 北京工业大学 | Emotional state displaying method, apparatus and system based on resting-state cerebral functional image |
| CN114224342A (en) * | 2021-12-06 | 2022-03-25 | 南京航空航天大学 | Multi-channel electroencephalogram emotion recognition method based on space-time fusion feature network |
| CN114209342A (en) * | 2022-01-28 | 2022-03-22 | 南京邮电大学 | Electroencephalogram signal motor imagery classification method based on space-time characteristics |
| CN115474899A (en) * | 2022-08-17 | 2022-12-16 | 浙江大学 | Basic taste perception identification method based on multi-scale convolution neural network |
Non-Patent Citations (2)
| Title |
|---|
| PHAM TD ET AL.,: "Enhancing Performance of EEG-based Emotion Recognition Systems Using Feature Smoothing", 《SPRINGER-VERLAG BERLIN》, vol. 9492, 31 December 2015 (2015-12-31), pages 95 - 102 * |
| 张慧港,: "基于脉冲神经网络和脑功能分析的情绪识别研究", 《中国优秀硕士学位论文全文数据库 (基础科学辑)》, vol. 2023, no. 2, 15 February 2023 (2023-02-15), pages 006 - 1377 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116861211B (en) | 2024-09-06 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112932502B (en) | Electroencephalogram emotion recognition method combining mutual information channel selection and hybrid neural network | |
| CN113158964B (en) | Sleep stage method based on residual error learning and multi-granularity feature fusion | |
| Kan et al. | Self-supervised group meiosis contrastive learning for eeg-based emotion recognition | |
| CN117195099A (en) | Electroencephalogram signal emotion recognition algorithm integrating multi-scale features | |
| CN115659207A (en) | Electroencephalogram emotion recognition method and system | |
| Bai et al. | Sect: A method of shifted eeg channel transformer for emotion recognition | |
| Xu et al. | AMDET: Attention based multiple dimensions EEG transformer for emotion recognition | |
| Li et al. | GCNs–FSMI: EEG recognition of mental illness based on fine-grained signal features and graph mutual information maximization | |
| Li et al. | Dynamical graph neural network with attention mechanism for epilepsy detection using single channel EEG | |
| CN116861211B (en) | Electroencephalogram emotion recognition method and system integrating space-time interaction neural network | |
| CN117407748A (en) | Electroencephalogram emotion recognition method based on graph convolution and attention fusion | |
| CN110507288A (en) | Vision based on one-dimensional convolutional neural networks induces motion sickness detection method | |
| CN116115240A (en) | Electroencephalogram emotion recognition method based on multi-branch chart convolution network | |
| CN115316955A (en) | Light-weight and quick decoding method for motor imagery electroencephalogram signals | |
| CN115859185A (en) | Electroencephalogram emotion recognition method based on pulse convolution neural network | |
| CN114638253A (en) | Identity recognition system and method based on emotion electroencephalogram feature fusion optimization mechanism | |
| Wu et al. | A novel emotion recognition method based on 1D-DenseNet | |
| Gao et al. | Multi-Head Self-Attention Enhanced Convolutional Neural Network for Driver Fatigue Detection using EEG Signals | |
| Zhou et al. | Dual-Branch Convolution Network with Efficient Channel Attention for EEG-Based Motor Imagery Classification | |
| CN118467934A (en) | Brain region characteristic-based electroencephalogram signal representation learning method and system | |
| Neverlien et al. | Decoding Emotions From EEG Responses Elicited by Videos Using Machine Learning Techniques on Two Datasets | |
| Wang et al. | EEG Emotion Classification Using 2D-3DCNN | |
| CN117158912B (en) | Sleep stage detection system based on graph attention mechanism and space-time graph convolution | |
| CN117332317B (en) | EEG emotion recognition method combining attention residual error network with LSTM | |
| CN118490232B (en) | Brain depression diagnosis method based on multi-frequency domain decomposition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |