CN112990008A - Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network - Google Patents
Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network Download PDFInfo
- Publication number
- CN112990008A CN112990008A CN202110272735.XA CN202110272735A CN112990008A CN 112990008 A CN112990008 A CN 112990008A CN 202110272735 A CN202110272735 A CN 202110272735A CN 112990008 A CN112990008 A CN 112990008A
- Authority
- CN
- China
- Prior art keywords
- electroencephalogram
- wavelet
- emotion recognition
- signal
- emotional state
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 52
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000013527 convolutional neural network Methods 0.000 title claims abstract description 34
- 238000010586 diagram Methods 0.000 title claims abstract description 18
- 230000002996 emotional effect Effects 0.000 claims abstract description 57
- 230000009466 transformation Effects 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 20
- 230000008030 elimination Effects 0.000 claims description 7
- 238000003379 elimination reaction Methods 0.000 claims description 7
- 238000003860 storage Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 238000000537 electroencephalography Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 10
- 239000013598 vector Substances 0.000 description 9
- 230000008451 emotion Effects 0.000 description 8
- 210000002569 neuron Anatomy 0.000 description 8
- 238000004590 computer program Methods 0.000 description 7
- 238000012545 processing Methods 0.000 description 7
- 230000004913 activation Effects 0.000 description 5
- 238000009826 distribution Methods 0.000 description 4
- 238000001914 filtration Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 239000011159 matrix material Substances 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 238000013145 classification model Methods 0.000 description 2
- 238000000354 decomposition reaction Methods 0.000 description 2
- 230000004927 fusion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000001054 cortical effect Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000004070 electrodeposition Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000001671 psychotherapy Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000000946 synaptic effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012549 training Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Software Systems (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Signal Processing (AREA)
- Dermatology (AREA)
- Neurology (AREA)
- Neurosurgery (AREA)
- Human Computer Interaction (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The emotion recognition method and system based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the disclosure comprise the following steps: acquiring an electroencephalogram signal to be identified; extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified; decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal; acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal; arranging the electroencephalogram features to form a feature cube; and inputting the feature cube into the trained CNN model for emotion recognition. The accuracy of emotion recognition is improved.
Description
Technical Field
The invention relates to the technical field of emotion state recognition, in particular to an emotion recognition method and system based on a three-dimensional characteristic diagram and a convolutional neural network.
Background
The statements in this section merely provide background information related to the present disclosure and may not necessarily constitute prior art.
Emotion is a state that integrates human feelings, ideas, and behaviors, and is very important in human decision processing, interaction, and cognition. Currently, most emotional state classification studies use electroencephalography (EEG) and facial expressions to classify emotional states. EEG is a signal that records cortical surface activity and is the result of synaptic activation of neurons in the brain. In recent years, it has been shown that EEG is a suitable signal for biometric authentication and has an important function. Therefore, emotional state recognition based on EEG comes to mind, namely, the EEG is used for emotional state recognition, so that online psychotherapy and medical diagnosis are realized. The relationship between emotional state and brain activity is recorded in the electroencephalogram signal, and very subtle changes in emotional state are reflected with high time resolution. However, the electroencephalogram signals have the defects of time asymmetry and instability, low signal-to-noise ratio, incapability of directly determining brain area reactions and the like. Therefore, EEG-based emotional state recognition remains a difficult task. Many researchers have proposed methods for emotion state recognition based on EEG, such as those based on convolutional neural networks, deep belief networks, convolutional neural networks, and the like.
In recent years, CNN is gradually adopted in the field, but there is a problem in how to convert a one-dimensional electroencephalogram signal into a form of a 3D feature map and to effectively combine the one-dimensional electroencephalogram signal with CNN, thereby achieving accurate recognition of human emotion.
The prior art has the following technical problems:
the single-class electroencephalogram feature emotion recognition accuracy is low, and the traditional classification model is easy to suffer from dimensional disasters and low in emotion recognition accuracy. Compared with a neural network, the extracted features determine the classification performance of the traditional classification model. The convolutional neural network in the neural network has the advantages of weight sharing, insusceptibility to influence of dimensional disasters, self-selection of features and the like.
The convolutional neural network has the capability of learning a local connection structure and developing a multi-scale hierarchical mode, and the efficiency of image processing, video processing and voice recognition tasks is improved. Electroencephalograms are composed of one-dimensional signals generated by each channel, and cannot provide spatial information for emotion recognition. At present, CNN is applied to the field of emotion recognition based on electroencephalogram, but the recognition accuracy is relatively low, and one reason for the CNN is that the information of the spatial relative position between electroencephalogram channels cannot be provided; the data volume of the two data is relatively small, and the overfitting probability of the model is increased; thirdly, the extracted electroencephalogram features have small contribution to emotion recognition.
Disclosure of Invention
In order to solve the problems, the emotion recognition method and system based on the three-dimensional feature map and the convolutional neural network are provided, the feature cube capable of reflecting electroencephalogram signal space information is obtained, emotion recognition is carried out through the feature cube, and the accuracy of emotion recognition is improved.
In order to achieve the purpose, the following technical scheme is adopted in the disclosure:
in a first aspect, a method for emotion recognition based on a three-dimensional feature map and a convolutional neural network is provided, which includes:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram features to form a feature cube;
and inputting the feature cube into the trained CNN model for emotion recognition.
In a second aspect, a emotion recognition system based on a three-dimensional feature map and a convolutional neural network is provided, which includes:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing electroencephalogram signals without a basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
and the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition.
In a third aspect, an electronic device is provided, which includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the emotion recognition method based on the three-dimensional feature map and the convolutional neural network.
In a fourth aspect, a computer-readable storage medium is provided for storing computer instructions, which when executed by a processor, perform the steps of the emotion recognition method based on a three-dimensional feature map and a convolutional neural network.
Compared with the prior art, the beneficial effect of this disclosure is:
1. the electroencephalogram signal is subjected to wavelet packet transformation, the wavelet energy ratio and the wavelet entropy of each frequency band signal are extracted, the wavelet energy ratio can reflect the energy of each frequency band in the electroencephalogram signal, the wavelet entropy can reflect the order or disorder of signal spectrum energy distribution in each space, the approximate entropy can reflect the complexity of a time sequence and is used for measuring the complexity of the electroencephalogram signal, and a feature cube consisting of the approximate entropy, the wavelet energy ratio and the wavelet entropy provides space information for emotion recognition based on electroencephalogram, so that the emotion recognition accuracy is improved.
2. The method extracts the electroencephalogram signals without the basic emotional states from the electroencephalogram signals to be recognized, the recognition effect is better when the emotional states are recognized based on the electroencephalogram signals without the basic emotional states, and the accuracy of emotional recognition is guaranteed when the emotion recognition is performed by using the feature cube in the feature cube further acquired.
Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application.
FIG. 1 is a flow chart of a method disclosed in example 1 of the present disclosure;
FIG. 2 is a diagram showing an arrangement of features disclosed in example 1 of the present disclosure;
fig. 3 is a diagram of a structure of a CNN model disclosed in embodiment 1 of the present disclosure.
The specific implementation mode is as follows:
the present disclosure is further described with reference to the following drawings and examples.
It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the disclosure. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.
It is noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of example embodiments according to the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, and it should be understood that when the terms "comprises" and/or "comprising" are used in this specification, they specify the presence of stated features, steps, operations, devices, components, and/or combinations thereof, unless the context clearly indicates otherwise.
In the present disclosure, terms such as "upper", "lower", "left", "right", "front", "rear", "vertical", "horizontal", "side", "bottom", and the like indicate orientations or positional relationships based on those shown in the drawings, and are only relational terms determined for convenience in describing structural relationships of the parts or elements of the present disclosure, and do not refer to any parts or elements of the present disclosure, and are not to be construed as limiting the present disclosure.
In the present disclosure, terms such as "fixedly connected", "connected", and the like are to be understood in a broad sense, and mean either a fixed connection or an integrally connected or detachable connection; may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present disclosure can be determined on a case-by-case basis by persons skilled in the relevant art or technicians, and are not to be construed as limitations of the present disclosure.
Example 1
In this embodiment, an emotion recognition method based on a three-dimensional feature map and a convolutional neural network is disclosed, which includes:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram features to form a feature cube;
and inputting the feature cube into the trained CNN model for emotion recognition.
Further, a method for eliminating the basic emotional state is adopted to extract the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified.
Further, extracting a wavelet coefficient of each frequency band signal;
and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient of each frequency band signal.
Furthermore, the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state is calculated by adopting an approximate entropy algorithm.
Further, the CNN model includes an input layer, a convolutional layer, a reconstruction layer, a full link layer, and an output layer.
Further, a cross entropy function is adopted to define a loss function, and the CNN model is trained.
Further, the acquired electroencephalogram features are arranged according to the positions of the corresponding channels on the 2D plane graph to form a feature cube.
The emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the embodiment is explained in detail, and the emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network disclosed by the embodiment weakens the dependence of an electroencephalogram experiment on a test by using a basic emotion state elimination method, so that the experiment system has higher universality; the wavelet energy ratio may reflect the energy of each band in the EEG signal; the wavelet entropy is an extension of the wavelet energy ratio and can reflect the order or disorder of the signal spectrum energy distribution in each space; the approximate entropy can reflect the complexity of the time series and is used to measure the complexity of the EEG signal; the feature cube provides spatial information for electroencephalogram-based emotion recognition, and specifically comprises the following steps as shown in fig. 1:
s1: acquiring an electroencephalogram signal to be recognized, and acquiring an electroencephalogram signal without a basic emotional state by adopting a basic emotional state elimination method for the electroencephalogram signal to be recognized.
In specific implementation, a basic emotion state elimination method is adopted for the electroencephalogram signal to be recognized, and the specific process of acquiring the electroencephalogram signal without the basic emotion state is as follows:
initializing an input original electroencephalogram data set D, and for each tested electroencephalogram signal S of the original electroencephalogram signal to be identified in the data set DiIntercepting an experimental trial number i;
eliminating the basic emotional state for each trial i, and extracting a calm electroencephalogram signal SciAnd the EEG signal S after the experimenttiFrom StiMinus SciObtaining the electroencephalogram signal S without the basic emotional state after emotional stimulationsi。
S2: decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining the wavelet energy ratio and the wavelet entropy of each frequency band signal.
In specific implementation, wavelet packet transformation is adopted for the electroencephalogram signals without the basic emotional state to be divided into five frequency bands of Delta, Theta, Alpha, Beta and Gamma, and frequency band signals are obtained; and acquiring the wavelet coefficient of each frequency band signal, and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient. The method specifically comprises the following steps:
method for decomposing non-basic emotional state electroencephalogram signal S by wavelet packetsiDecomposing and reconstructing to obtain 2 by decomposing j-layer wavelet packet of the EEG signaliA wavelet node;
reconstructing wavelet packet coefficients of nodes of the ith layer to obtain a reconstructed signal S of each nodej,mWherein m is 1, 2, …, 2jExtracting wavelet coefficient C from five frequency bands of Delta, Theta, Alpha, Beta and Gammai;
By CiAnd calculating the wavelet energy ratio and the wavelet entropy of each frequency band signal in each channel.
The wavelet energy ratio and the wavelet entropy are calculated as follows:
hypothesis CiExtracting wavelet coefficients after wavelet packet transformation, wherein i ═ Delta, Theta, Alpha, Beta and Gamma; the calculation of the total energy of the wavelet coefficients is defined as follows:
wavelet energy ratio eta of ith frequency bandiThe calculation is defined as follows:
the wavelet entropy calculation method of the ith frequency band is defined as follows:
Entropyi=-ηiln(ηi) (3)
in this embodiment, wavelet packet decomposition is employed as db6 wavelet basis wavelet packet decomposition.
S3: the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state is obtained, and the electroencephalogram characteristics are formed by the wavelet energy ratio and the wavelet entropy of each frequency band signal.
In specific implementation, an approximate entropy algorithm is adopted to have no complexity of each channel electroencephalogram signal in the electroencephalogram signals in the basic emotional state, and the approximate entropy is specifically calculated by the following steps:
b) Determining parameters m and r, wherein m is an integer and represents the length of a vector to be reconstructed, and r is a non-negative real number and represents a similar threshold value between the reconstructed vectors;
d) Calculating distance between arbitrary reconstruction vectorsWherein the value range of i, j is 1 to N-m + 1;
e) counting the number of similar vectors of each reconstructed vectorWherein the value range of i, j is 1 to N-m + 1;
g) According to the steps a) to f), and then calculating phim+1;
h)ApEn=Φm-Φm+1。
And (4) forming the electroencephalogram characteristics by the acquired complexity of each channel electroencephalogram signal, the wavelet energy ratio and the wavelet entropy of each frequency band signal.
S4: and arranging the electroencephalogram characteristics to form a characteristic cube.
In specific implementation, the extracted electroencephalogram features are subjected to feature arrangement according to a feature arrangement rule to form a feature cube; the electroencephalogram electrode selects 32 electrode positions according to the international 10-20 system, and is respectively Fp1, AF3, F3, F7, FC5, FC1, C3, T7, CP5, CP1, P3, P7, PO3, O1, Oz, Pz, Fp2, AF4, Fz, F4, F8, FC6, FC, Cz, C4, T8, Cp6, Cp2, P4, P8, PO4 and O2.
The extracted electroencephalogram features are arranged, namely the positions of the electrode channels are mapped to the positions on the 2D plane graph, and then the extracted electroencephalogram features are arranged according to the positions of the 2D plane graph to form a 9 x 11 feature cube.
S5: and inputting the feature cube into the trained CNN model for emotion recognition.
In specific implementation, a CNN model for feature fusion and classification recognition is constructed for a feature cube, and as shown in fig. 3, 1 input layer, 3 convolutional layers, 1 reconstruction layer, 2 full-link layers, and 1 output layer are designed in the model. The input layer is used for receiving the feature cube; the convolutional layer is used for extracting spatial information among electroencephalogram channels and performing weighted summation on characteristics among specific channels to form classification characteristics which are easy to identify by a model; the reconstruction layer is used for arranging the calculated results of the adjacent convolution layers to form a row vector or column vector form and forming a classifier on the full-connection layer and the output layer so as to achieve emotion recognition; the full connection layer is used for carrying out data dimension transformation on the data of the reconstruction layer to provide high-dimension identification information for emotion identification and classification; the output layer is used for judging the calculation result of the full connection layer and outputting the emotion recognition result.
Designing (setting) each layer function of CNN modelIs nth layer two-dimensional feature map size and number):
(1) input layer (L1): the input for each sample is l1A three-dimensional matrix of W H FN, where W H is the arrangement of the simulation electrodes on the scalp and FN is the number of features extracted from each channel.
(2) Convolutional layer (L2): the main function of this layer is to perform spatial filtering and fusion on the input element graph, so the connection between this layer and the input layer is a local connection. The size and the number of the filters are set asW is the width of convolution kernel, H is the height of convolution kernel, Z is the number of characteristic maps in the previous layer, FN is the number of two-dimensional characteristic maps (or called as the number of filters) to be output, and the size of each obtained characteristic map isThe number of the two-dimensional characteristic graphs isThe convolution kernel is set as a matrixRather than vectors, because of the need to fuse spatial information to form abstract features that are easily recognized by the models designed by this disclosure.
(3) Convolutional layer (L3): the main function of this layer is to integrate the new features of L2 and achieve the dimensionality reduction effect. The size and the number of the filters of the third layer are set asThe size of each feature map obtained isThe number of the characteristic graphs is
(4) Convolutional layer (L4): the main function of this layer is the same as L3. The size and the number of the filters of the third layer are set asThe size of each feature map obtained isThe number of the characteristic graphs is
(5) Reconstituted layer (L5): this layer will receive the new feature computed by L4, setting the number of neurons toLayers 6 to 7 constitute the classifier.
(6) Full tie layer (L6): this layer upscales the features of L5 to provide a high dimensional space for the classification of the last layer. The present disclosure sets the neurons of this layer to
(7) Output layer (L7): this layer outputs the emotional state using the calculation results of the L6 layer.
In this embodiment, the calculation modes of each layer of the model are designed:
is provided withIs the output of the jth neuron in the nth feature map of the ith layer,then it is the input of the jth neuron in the nth feature map of the ith layer, and the relationship between the two is shown below.
Where σ (-) is the activation function.
(1) Input layer (L1): l1WXHFN is a three-dimensional feature map and is also an input to the present model.
(2) Convolutional layer (L2): by usingA filter pair l1A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution,is of sizeThe filter of (2) is preferably a filter,is an offset.
(3) Convolutional layer (L3): by usingA filter pair l2A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution,is of sizeThe filter of (2) is preferably a filter,is an offset.
(4) Convolutional layer (L4): by usingA filter pair l3A filtering operation is performed and then a new feature map is obtained by calculation of the activation function.
Wherein is the operation of convolution,is of sizeThe filter of (2) is preferably a filter,is an offset.
(5) Reconstituted layer (L5): the output elements of L4 are reconstructed as a column vector, i.e. o5(j)。
(6) Full tie layer (L6): all neurons in this layer are fully connected to the element of L5.
o6(j)=σ(o5(j)W6+b6(j)) (8)
(7) Output layer (L7): all neurons in this layer were fully connected to L6 neurons.
p=σ(o6(j)W7+b7(j)) (9)
The loss function of the CNN model constructed in this example is defined as follows:
Loss=cross_entropy(p,l)+α||W||2 (10)
wherein p is the output value of the model, W represents all the parameters to be trained in the model, and alpha is a regular term coefficient.
In order to accelerate the network convergence speed, the weights and deviations of all layers of the network are randomized into normal distribution, the mean value is 0, and the variance is 1/NinputIn which N isinputIs the number of upper features. Except for the L5 layer and the L7 layer, the Batch-Norm technology is added to all other layers, so that the training time can be saved, and the possibility of overfitting can be effectively reduced. Since the classification label is a one-hot encoding mode, a cross entropy function is used to define a loss function. Meanwhile, Adam optimizer and gradient descent algorithm are used in TensorFlow to adjust connection weights and bias. The maximum number of iterations is set to 10000 and the loss threshold is set to 0.01.
The method disclosed by the embodiment comprises 5 parts: acquiring an electroencephalogram signal to be identified, eliminating a basic emotional state, acquiring a wavelet energy ratio, a wavelet entropy, acquiring the complexity of the electroencephalogram signal, arranging characteristics and identifying an emotional state. By analysis, it was found that the wavelet energy ratio can reflect the energy of each band in the EEG signal; the wavelet entropy is an extension of the wavelet energy ratio and can reflect the order or disorder of the signal spectrum energy distribution in each space; the approximate entropy can reflect the complexity of the time series and is used to measure the complexity of the EEG signal; the feature cube (3D feature map) provides spatial information for electroencephalogram-based emotion recognition; the method comprises the steps of obtaining the electroencephalogram without the basic emotional state by adopting a basic emotional state elimination method, reconstructing and extracting wavelet coefficients of electroencephalogram frequency bands, obtaining corresponding wavelet characteristics by adopting wavelet energy ratio and wavelet entropy calculation modes according to the wavelet coefficients of five frequency bands of the electroencephalogram eliminated by the basic emotional state, estimating the overall complexity of the electroencephalogram by using approximate entropy, performing characteristic arrangement on the extracted characteristics to build characteristic cubes, and finally inputting the characteristic cubes into a CNN (computer network node) model for classification and prediction to further identify emotional states.
Example 2
In this embodiment, an emotion recognition system based on a three-dimensional feature map and a convolutional neural network is disclosed, including:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing electroencephalogram signals without a basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
and the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition.
Example 3
In this embodiment, an electronic device is disclosed, which comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the emotion recognition method based on the three-dimensional feature map and the convolutional neural network disclosed in embodiment 1.
Example 4
In this embodiment, a computer readable storage medium is disclosed for storing computer instructions which, when executed by a processor, perform the steps described in the emotion recognition method based on a three-dimensional feature map and a convolutional neural network disclosed in embodiment 1.
The above description is only a preferred embodiment of the present application and is not intended to limit the present application, and various modifications and changes may be made by those skilled in the art. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application shall be included in the protection scope of the present application.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solutions of the present invention and not for limiting the same, and although the present invention is described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: modifications and equivalents may be made to the embodiments of the invention without departing from the spirit and scope of the invention, which is to be covered by the claims.
Claims (10)
1. The emotion recognition method based on the three-dimensional characteristic diagram and the convolutional neural network is characterized by comprising the following steps of:
acquiring an electroencephalogram signal to be identified;
extracting an electroencephalogram signal without a basic emotional state from the electroencephalogram signal to be identified;
decomposing and reconstructing the electroencephalogram signals without the basic emotional state by wavelet packet transformation to obtain a plurality of frequency band signals, and obtaining a wavelet energy ratio and a wavelet entropy of each frequency band signal;
acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
arranging the electroencephalogram features to form a feature cube;
and inputting the feature cube into the trained CNN model for emotion recognition.
2. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein a method for eliminating a basic emotional state is adopted to extract an electroencephalogram signal without a basic emotional state from an electroencephalogram signal to be recognized.
3. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein wavelet coefficients of each band signal are extracted;
and calculating the wavelet characteristics of each frequency band signal according to the wavelet coefficient of each frequency band signal.
4. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein the degree of complexity of each channel electroencephalogram signal in the electroencephalogram signal without the basic emotional state is calculated by adopting an approximate entropy algorithm.
5. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network of claim 1, wherein the CNN model includes an input layer, a convolutional layer, a reconstruction layer, a fully-connected layer, and an output layer.
6. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network of claim 1, wherein the CNN model is trained by defining a loss function using a cross entropy function.
7. The emotion recognition method based on a three-dimensional feature map and a convolutional neural network as claimed in claim 1, wherein the acquired electroencephalogram features are arranged according to the positions of the corresponding channels on the 2D plane map to form a feature cube.
8. The emotion recognition system based on the three-dimensional characteristic diagram and the convolutional neural network is characterized by comprising:
the data acquisition module is used for acquiring an electroencephalogram signal to be identified;
the basic emotional state elimination module is used for extracting the electroencephalogram signals without the basic emotional state from the electroencephalogram signals to be identified;
the electroencephalogram characteristic acquisition module is used for decomposing and reconstructing electroencephalogram signals without a basic emotional state by adopting wavelet packet transformation, acquiring a plurality of frequency band signals, acquiring the wavelet energy ratio and the wavelet entropy of each frequency band signal, acquiring the complexity of each channel electroencephalogram signal in the electroencephalogram signals without the basic emotional state, and forming electroencephalogram characteristics with the wavelet energy ratio and the wavelet entropy of each frequency band signal;
the characteristic cube acquisition module is used for arranging the acquired electroencephalogram characteristics to form a characteristic cube;
and the emotion recognition module is used for inputting the feature cube into the trained CNN model for emotion recognition.
9. An electronic device comprising a memory and a processor, and computer instructions stored on the memory and executed on the processor, wherein the computer instructions, when executed by the processor, perform the steps of the method for emotion recognition based on a three-dimensional feature map and a convolutional neural network of any of claims 1-7.
10. A computer-readable storage medium storing computer instructions which, when executed by a processor, perform the steps of the method for emotion recognition based on a three-dimensional feature map and a convolutional neural network according to any of claims 1 to 7.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110272735.XA CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202110272735.XA CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Publications (2)
Publication Number | Publication Date |
---|---|
CN112990008A true CN112990008A (en) | 2021-06-18 |
CN112990008B CN112990008B (en) | 2022-06-17 |
Family
ID=76335410
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202110272735.XA Expired - Fee Related CN112990008B (en) | 2021-03-13 | 2021-03-13 | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN112990008B (en) |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113780134A (en) * | 2021-08-31 | 2021-12-10 | 昆明理工大学 | Motor imagery electroencephalogram decoding method based on ShuffleNet V2 network |
CN113974627A (en) * | 2021-10-26 | 2022-01-28 | 杭州电子科技大学 | Emotion recognition method based on brain-computer generated confrontation |
CN116269386A (en) * | 2023-03-13 | 2023-06-23 | 中国矿业大学 | Multichannel physiological time sequence emotion recognition method based on ordinal division network |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104538027A (en) * | 2014-12-12 | 2015-04-22 | 复旦大学 | Method and system for calculating emotion spreading of voice social contact media |
CN111079465A (en) * | 2018-10-18 | 2020-04-28 | 北京入思技术有限公司 | Emotional state comprehensive judgment method based on three-dimensional imaging analysis |
CN111616721A (en) * | 2020-05-31 | 2020-09-04 | 天津大学 | Emotion recognition system based on deep learning and brain-computer interface and application |
CN112294341A (en) * | 2020-10-29 | 2021-02-02 | 武汉大学 | Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network |
-
2021
- 2021-03-13 CN CN202110272735.XA patent/CN112990008B/en not_active Expired - Fee Related
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN104538027A (en) * | 2014-12-12 | 2015-04-22 | 复旦大学 | Method and system for calculating emotion spreading of voice social contact media |
CN111079465A (en) * | 2018-10-18 | 2020-04-28 | 北京入思技术有限公司 | Emotional state comprehensive judgment method based on three-dimensional imaging analysis |
CN111616721A (en) * | 2020-05-31 | 2020-09-04 | 天津大学 | Emotion recognition system based on deep learning and brain-computer interface and application |
CN112294341A (en) * | 2020-10-29 | 2021-02-02 | 武汉大学 | Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network |
Non-Patent Citations (1)
Title |
---|
YI WANG.ETC: ""EmotioNet: A 3-D Convolutional Neural Network for EEG-based Emotion Recognition"", 《2018 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN)》 * |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113780134A (en) * | 2021-08-31 | 2021-12-10 | 昆明理工大学 | Motor imagery electroencephalogram decoding method based on ShuffleNet V2 network |
CN113974627A (en) * | 2021-10-26 | 2022-01-28 | 杭州电子科技大学 | Emotion recognition method based on brain-computer generated confrontation |
CN116269386A (en) * | 2023-03-13 | 2023-06-23 | 中国矿业大学 | Multichannel physiological time sequence emotion recognition method based on ordinal division network |
CN116269386B (en) * | 2023-03-13 | 2024-06-11 | 中国矿业大学 | Multichannel physiological time sequence emotion recognition method based on ordinal division network |
Also Published As
Publication number | Publication date |
---|---|
CN112990008B (en) | 2022-06-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN112990008B (en) | Emotion recognition method and system based on three-dimensional characteristic diagram and convolutional neural network | |
CN109389059B (en) | P300 detection method based on CNN-LSTM network | |
CN113693613B (en) | Electroencephalogram signal classification method, electroencephalogram signal classification device, computer equipment and storage medium | |
CN114266276B (en) | Motor imagery electroencephalogram signal classification method based on channel attention and multi-scale time domain convolution | |
CN108960299B (en) | Method for identifying multi-class motor imagery electroencephalogram signals | |
CN110598793A (en) | Brain function network feature classification method | |
WO2022183966A1 (en) | Electroencephalogram signal classification method and apparatus, device, storage medium and program product | |
CN114533086A (en) | Motor imagery electroencephalogram decoding method based on spatial domain characteristic time-frequency transformation | |
CN112450885B (en) | Epileptic electroencephalogram-oriented identification method | |
CN117503057B (en) | Epileptic seizure detection device and medium for constructing brain network based on high-order tensor decomposition | |
CN113133769A (en) | Equipment control method, device and terminal based on motor imagery electroencephalogram signals | |
CN115919330A (en) | EEG Emotional State Classification Method Based on Multi-level SE Attention and Graph Convolution | |
CN113712571A (en) | Abnormal electroencephalogram signal detection method based on Rinyi phase transfer entropy and lightweight convolutional neural network | |
CN112155549A (en) | ADHD disease diagnosis aid decision-making system based on deep convolution pulse neural network | |
CN114027786B (en) | Sleep breathing disorder detection method and system based on self-supervision type memory network | |
Ramakrishnan et al. | Epileptic eeg signal classification using multi-class convolutional neural network | |
CN112259228B (en) | Depression screening method by dynamic attention network non-negative matrix factorization | |
CN117574059A (en) | High-resolution brain-electrical-signal deep neural network compression method and brain-computer interface system | |
CN107256408B (en) | Method for searching key path of brain function network | |
Huang et al. | Modeling task fMRI data via mixture of deep expert networks | |
Sridhar et al. | A Neural Network Approach for EEG classification in BCI | |
Khalkhali et al. | Low latency real-time seizure detection using transfer deep learning | |
CN114428555B (en) | Electroencephalogram movement intention recognition method and system based on cortex source signals | |
Immanuel et al. | Analysis of different emotions with bio-signals (EEG) using deep CNN | |
CN115721323A (en) | Brain-computer interface signal identification method and system and electronic equipment |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant | ||
CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20220617 |
|
CF01 | Termination of patent right due to non-payment of annual fee |