CN113951883A - Gender difference detection method based on electroencephalogram signal emotion recognition - Google Patents
Gender difference detection method based on electroencephalogram signal emotion recognition Download PDFInfo
- Publication number
- CN113951883A CN113951883A CN202111335757.2A CN202111335757A CN113951883A CN 113951883 A CN113951883 A CN 113951883A CN 202111335757 A CN202111335757 A CN 202111335757A CN 113951883 A CN113951883 A CN 113951883A
- Authority
- CN
- China
- Prior art keywords
- data
- long
- short
- neural network
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 17
- 238000001514 detection method Methods 0.000 title description 7
- 230000015654 memory Effects 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 46
- 238000013528 artificial neural network Methods 0.000 claims abstract description 41
- 230000008451 emotion Effects 0.000 claims abstract description 35
- 210000004556 brain Anatomy 0.000 claims abstract description 24
- 238000010586 diagram Methods 0.000 claims abstract description 20
- 230000003925 brain function Effects 0.000 claims abstract description 12
- 239000000284 extract Substances 0.000 claims abstract description 5
- 238000003062 neural network model Methods 0.000 claims abstract description 3
- 238000012549 training Methods 0.000 claims description 33
- 239000010410 layer Substances 0.000 claims description 26
- 230000006870 function Effects 0.000 claims description 15
- 239000011159 matrix material Substances 0.000 claims description 13
- 230000008569 process Effects 0.000 claims description 12
- 238000002474 experimental method Methods 0.000 claims description 10
- 230000002996 emotional effect Effects 0.000 claims description 8
- 230000008447 perception Effects 0.000 claims description 8
- 238000011176 pooling Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 230000007787 long-term memory Effects 0.000 claims description 6
- PIZHFBODNLEQBL-UHFFFAOYSA-N 2,2-diethoxy-1-phenylethanone Chemical compound CCOC(OCC)C(=O)C1=CC=CC=C1 PIZHFBODNLEQBL-UHFFFAOYSA-N 0.000 claims description 5
- 241000997494 Oneirodidae Species 0.000 claims description 5
- BQJCRHHNABKAKU-KBQPJGBKSA-N morphine Chemical compound O([C@H]1[C@H](C=C[C@H]23)O)C4=C5[C@@]12CCN(C)[C@@H]3CC5=CC=C4O BQJCRHHNABKAKU-KBQPJGBKSA-N 0.000 claims description 5
- 230000006403 short-term memory Effects 0.000 claims description 5
- 238000001914 filtration Methods 0.000 claims description 4
- 238000012360 testing method Methods 0.000 claims description 4
- 238000002790 cross-validation Methods 0.000 claims description 3
- 230000000116 mitigating effect Effects 0.000 claims description 3
- 238000005457 optimization Methods 0.000 claims description 3
- 238000013526 transfer learning Methods 0.000 claims description 3
- 230000004913 activation Effects 0.000 claims description 2
- 238000012937 correction Methods 0.000 claims description 2
- 238000007405 data analysis Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 claims description 2
- 238000007781 pre-processing Methods 0.000 claims description 2
- 239000004576 sand Substances 0.000 claims description 2
- 239000002356 single layer Substances 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 230000001133 acceleration Effects 0.000 claims 1
- 210000003484 anatomy Anatomy 0.000 claims 1
- 230000001419 dependent effect Effects 0.000 claims 1
- 238000013461 design Methods 0.000 claims 1
- 238000005538 encapsulation Methods 0.000 claims 1
- 238000005070 sampling Methods 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 8
- 230000014509 gene expression Effects 0.000 description 7
- 238000011160 research Methods 0.000 description 7
- 208000019022 Mood disease Diseases 0.000 description 4
- 238000012706 support-vector machine Methods 0.000 description 4
- 208000017194 Affective disease Diseases 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 210000004720 cerebrum Anatomy 0.000 description 2
- 238000003745 diagnosis Methods 0.000 description 2
- 238000009499 grossing Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000036651 mood Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000012795 verification Methods 0.000 description 2
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 description 1
- 206010063659 Aversion Diseases 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 230000003930 cognitive ability Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000000763 evoking effect Effects 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 230000002349 favourable effect Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000036544 posture Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
- A61B5/374—Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
- A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Psychiatry (AREA)
- Molecular Biology (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Signal Processing (AREA)
- Physiology (AREA)
- Psychology (AREA)
- Mathematical Physics (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Developmental Disabilities (AREA)
- Social Psychology (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Child & Adolescent Psychology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
A brain electrical characteristic recognition method based on a long-and-short-term memory diagram neural network extracts differential entropy characteristics from brain electrical data in a centralized mode, then the differential entropy characteristics are converted into characteristic matrixes of representation diagrams, then a long-and-short-term diagram neural network model is trained, brain function connection information and time sequence relation of the characteristic data are collected at the same time, and finally emotion recognition is achieved through the trained network model. According to the invention, the long-time memory diagram neural network fully utilizes the brain function connection information and the time sequence information, so that a plurality of commonly used and representative data sets are identified, the most common difference performance is analyzed, and the gender difference characteristic in emotion-related electroencephalogram activities is verified.
Description
Technical Field
The invention relates to a technology in the field of electroencephalogram signals, in particular to a gender difference detection method based on electroencephalogram signal emotion recognition.
Background
Compared with the signal recognition such as facial expressions, voice, body postures and the like frequently used by researchers in the past, the electroencephalogram signal can represent the brain activities related to the emotion of the tested person more delicately, so that the electroencephalogram signal is considered to be the most effective signal form for emotion recognition, and therefore, the research of emotion recognition by adopting the electroencephalogram signal is more and more concerned by academic circles and industrial circles. However, because there are not only physical structure differences such as scalp impedance and head shape, but also mental mechanism differences such as thinking way, psychological state and cognitive ability among different individuals, the characteristic pattern of the brain electrical signal highly depends on the individual characteristics of the tested individual, and the problem of the individual difference seriously hinders the practical application of the emotional brain-computer interface.
In order to solve the problem of individual difference of electroencephalogram signals, one method is to use transfer learning to process the difference between tested individuals. Another research method is to find out the reasons for the individual differences of the electroencephalogram signals, understand the mechanism and action range behind the electroencephalogram signals and then solve the problem of the corresponding individual differences according to the properties of the electroencephalogram signals. Sex is a simple and clear individual difference, and the incidence rate of affective disorder diseases has obvious sex difference. For example, statistical data indicate that the number of female patients with depression is approximately 1.7 times greater than that of males. Affective disorders are closely related to the emotional state of the patient. Therefore, the detection of whether gender difference exists in electroencephalogram signal emotion recognition provides important theoretical and technical support for constructing an accurate emotion model and developing an assistant diagnosis and treatment system for the affective disorder diseases based on an emotional brain-computer interface.
Although the research result of the difference of emotion recognition and gender of the electroencephalogram signals exists, the existing work has great limitations: firstly, most of the data only use data of a single data set, and the model performance is insufficient, so that the conclusion is not universal; secondly, the processing method and the operation steps of the data in the researches are different, and the same conclusion can not be stably obtained under other experimental configurations; in addition, although various electroencephalogram difference expression forms are found in the results, the results are inconsistent among different researches, and the results cannot be accepted as general results. Therefore, a new method needs to be developed to find more stable gender difference expression and verify the gender difference of the brain electrical activity in emotional experience.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a gender difference detection method based on electroencephalogram signal emotion recognition, which fully utilizes brain function connection information and time sequence information through a long-time memory map neural network, realizes recognition of a plurality of commonly used and representative data sets, analyzes the most common difference expressions at the same time, and verifies the gender difference characteristics in emotion-related electroencephalogram activities.
The invention is realized by the following technical scheme:
the invention relates to a brain electrical characteristic identification method based on a long-time and short-time memory diagram neural network, which comprises the following steps: the method comprises the steps of extracting differential entropy characteristics from electroencephalogram data set, converting the differential entropy characteristics into characteristic matrixes of representation diagrams, training a neural network model of a long-time diagram and a short-time diagram, simultaneously acquiring brain function connection information and time sequence relation of the characteristic data, and finally realizing emotion recognition by using the trained network model.
The training was performed using 5 common, more representative data sets to develop the experiment, the data sets involved were: china (SEED, SEED-IV, SEED-V), Europe (DEAP, DREAMER) data set. They all acquire EEG electroencephalogram modality data and all use video material to induce emotion.
The training is to extract the differential entropy frequency domain characteristics as the input data of the experiment by preprocessing the original data. Designing a universal same-sex training strategy and an opposite-sex training strategy, training a recognizer for each tested data of each data set by using a leave-one-cross verification method to obtain a corresponding same-sex model and an opposite-sex model, adjusting the hyper-parameters of each group of models in iteration, and obtaining a final experimental result.
On the basis of the method, various problems of differential expression are solved. The invention analyzes and compares the experimental results on each data set with common gender difference expression forms, such as: the neural mode, the key brain area and the key frequency band further obtain a more objective and stable conclusion, and the sex difference characteristic of the emotional experience electroencephalogram signal is verified.
The pretreatment is as follows: the baseline correction is carried out on the original data of the electroencephalogram signals in the data set, and the data are down-sampled to 200Hz, so that the data analysis process is accelerated. Then, band-pass filtering is performed in the range of 1-75Hz to filter noise and artifacts in the data.
The differential entropy feature extraction is as follows: carrying out short-time Fourier transform on the preprocessed data, and extracting differential entropy characteristics of each lead on 5 frequency bands, wherein the characteristics are as follows: delta is 1-4Hz, theta is 4-8Hz, alpha is 8-14Hz, beta is 14-31Hz, and gamma is 31-50 Hz; because the SEED, SEED-IV and SEED-V use 62-lead electroencephalogram caps, the differential entropy characteristics of 310-dimensional electroencephalogram are shared; and because the original electroencephalogram data of DEAP and DREAMER filter delta frequency range and 32-lead and 14-lead electroencephalogram caps are used, 128-dimensional and 56-dimensional differential entropy characteristics are obtained respectively. And finally, smoothing the extracted electroencephalogram characteristics by using a linear power system, and eliminating rapid jitter information irrelevant to emotion.
The characteristic transformation means that: the InMemoryDataset base class of the PyTorch Geometric tool library is used, the extracted leads of the frequency domain feature are used as points, the connection among the leads is an edge, the feature data are converted into a feature matrix in the form of g ═ v, epsilon, and then the feature matrix is converted into time sequence data through a time window T to construct input data of an experiment.
The recognizer model is a long-time and short-time memory map neural network which is a map convolution neural network added with a memory module, and the network comprises: memory module, a plurality of picture convolution module, domain classifier, gradient reversal layer, emotion perception learner, pooling layer, full tie-layer when long and short term, wherein: the long-time and short-time memory module captures time sequence dependence information among the characteristic matrixes, the image volume module extracts brain function connection characteristic information related to emotional experience, the domain classifier is used for solving the efficiency problem of a scene crossing a tested scene, the gradient of the domain classifier is reversed by the gradient reversing layer during back propagation, the emotion perception learner aims at data label noise, the pooling layer pools output characteristics, and finally the full-connection layer is used for decoding the pooled characteristics and predicting an emotion label.
The long-time and short-time memory module comprises one or more self-connected memory cells and three gate units, and for data of each time step, the memory cells of the neural network can extract information from the result of the previous step, so that the memory module can store the time sequence dependence information of the data in a long time. Because the electroencephalogram signals are also timing sequence data, the memory module can capture and utilize timing sequence information in the electroencephalogram signals, and the identification accuracy is improved.
The graph convolution module captures local connection and global connection information among different leads by adopting a sparse adjacency matrix attached to an intracerebral network structure, the sparse adjacency matrix is obtained by utilizing reciprocal calculation of physical distances among lead channels, the local connection shows the anatomical connectivity of a brain area, and the global connection shows the emotion-related functional connectivity of left and right hemispheres.
The domain classifier combines a transfer learning and confrontation training method, reduces the difference between a source domain and a target domain, enhances the generalization capability of the model, and solves the problem of poor recognition efficiency in a cross-tested scene.
The emotion perception learner is as follows: the noise level factor is used to transform the individual tags into a prior probability distribution according to the mood-evoked properties, thereby mitigating the tag noise problem inside the data set.
The training process comprises the following specific contents:
1) firstly, information packaging is carried out on each data set, experimental configuration is convenient to switch, then each data set is divided into male data and female data, and the experimental data are normalized and used as input of model training.
2) Secondly, initializing the adjacency matrix learned by the graph convolution network:
and sets the global connection initial value to: a. theij＝AijAnd-1, the global connection is totally 9 pairs, and the global connection spans the left and right cerebral hemispheres and can maximize the lateralization of the brain electrical signals and find the functional connectivity between the hemispheres.
3) Training same-sex model and different-sex model of each tested data by using leave-one-out cross-validation methodSpecifically, any piece of tested data is taken as a test set, and all other tested electroencephalogram data with the same sex are taken as a training set Xi_sameTraining a same sex model, and simultaneously using all the different sex data as a training set Xi_crossAnd training a heterology model.
4) In a long-short time memory diagram neural network, a characteristic matrix sequence is formed
And a long-time input memory module. For the elements in the input sequence, the updating method of the memory module is respectively as follows: an input gate: i.e. it＝σ(Wixxt+bii+Wihht-1+bi) And forget to close the door: f. oft＝σ(Wfxxt+bif+Wfhht-1+bf) And a memory gate: gt＝tanh(Wcxxt+Wchht-1+bc) Memory cell status:
an output gate: ot＝σ(Woxxt+Wohht-1+bo)，ht＝ot*tanh(ct) Wherein: h istHidden layer state at time t, corresponding to ht-1σ refers to the logistic sigmoid function, representing the Hadamard product, at time t-1 or the initial hidden layer state.
5) For output X of long-and-short-time memory moduleiEach graph convolution module calculates: zi＝SLXiW, output ZiAnd learn the importance of functional connections of the various brains.
6) Then the output of the graph convolution module passes through the pooling layer and the full-connection layer, and the emotion label Y is outputiProbability distribution of (2):
wherein,
the full connection layer takes a softmax function as an activation function, and pool (phi) represents that global sum pooling is carried out, wherein the sigma (Z)i) Is to ZiEach element of (a) is non-linearly transformed: σ (x) is max (0, x).
7) In the node representation learning process, a domain classifier is trained to learn domain invariant features, and the source domain X is reducedSAnd a target domain XTThe difference between them. The main task of the domain classifier is to minimize the cross-entropy loss function of two binary classification tasks:
the method enhances the generalization capability of the model and improves the robustness in the cross-tested experiment.
8) Then, a learning process of the gradient inversion layer auxiliary domain classifier is utilized, and a calculation function of a gradient inversion layer factor is as follows:
wherein p is [0, 1 ]]Representing the progress of model training.
9) The emotion perception learner converts a single emotion label into prior probability distribution by using the noise level factor, and replaces the optimization problem of the graph convolution module with the problem of the minimized KL divergence function:
mitigating tag noise issues within the data set.
10) To this end, the loss function of the entire model becomes a calculation: phi ″ ═ phi' + phiDAnd finally, outputting a prediction result by using a single-layer full-connection network, respectively calculating the recognition accuracy of the isotropic model and the special-shaped model, and iterating the training process.
Technical effects
The invention combines a long-time memory map neural network and a short-time memory map neural network, simultaneously captures brain function connection information and time sequence information to perform signal pattern recognition, introduces a domain classifier and an emotion perception learner, reduces the difference between a training set and a test set, reduces data label noise, enhances the performance of a network model, and improves the pattern recognition accuracy based on electroencephalogram signals. Compared with the prior art, the method and the device realize emotion recognition tasks on a plurality of commonly used representative data sets, integrally solve the problem of limitation of the existing research work, improve the recognition rate of the electroencephalogram signal mode, and verify the gender difference of the electroencephalogram activity in emotion experience.
Drawings
FIG. 1 is a schematic flow diagram of the present invention;
FIG. 2 is a network structure diagram of a neural network of a long and short term memory diagram according to the present invention;
FIG. 3 is a statistical chart of the experimental results of the support vector machine;
FIG. 4 is a statistical graph of experimental results of neural networks based on long-term and short-term memory;
FIG. 5 is an analysis chart of the experimental results of the neural network based on long-term and short-term memory under different moods in the invention;
fig. 6 is a topological diagram for visualizing energy difference of male and female brains of 5 emotions in 5 frequency bands.
Detailed Description
As shown in fig. 1, the present embodiment relates to a gender difference detection method based on electroencephalogram emotion recognition, which specifically includes:
the method comprises the following steps: the configuration information of 5 data sets including brain electricity cap leads, tested information, label content and the like is sorted and packaged, and experimental configuration is convenient to switch. Acquiring original data of the electroencephalogram signals in the data set.
Step two: the raw data is pre-processed, down-sampled to 200Hz and band-pass filtered at 1 to 75Hz, filtering noise and artifacts.
Step three: and (3) extracting differential entropy characteristics, calculating short-time Fourier transform on the preprocessed data, extracting the differential entropy characteristics on 5 frequency bands on each lead in a non-overlapping time window of 4 seconds, and then performing characteristic smoothing by using a linear power system to eliminate rapid jitter to obtain experimental data of 310 dimension, 128 dimension and 56 dimension respectively.
Step four: the experimental data of the 5 data sets are converted into a feature matrix of a representation diagram, and then converted into time series data of the feature matrix, wherein the time series data are used as input data of a neural network of a training long-time and short-time memory diagram, and the network structure is shown in fig. 2.
Step five: and training the same-sex model and the opposite-sex model of each tested data by using a leave-one cross verification method, and respectively outputting the result data of the two models.
Step six: and (3) iterating the model training process, adjusting the hyper-parameters of the model, correcting the neural network and optimizing the performance of the model.
Step seven: fixing the neural network parameters of the same-sex model and the opposite-sex model, inputting the tested data into the neural network, outputting a predicted emotion label, and calculating the classification accuracy.
Step eight: and (4) counting the final identification accuracy, analyzing and evaluating the experimental result, analyzing the electroencephalogram mode and the brain area of the gender difference according to the brain energy topological graph, evaluating the key brain area and the key frequency band by utilizing network weight distribution and the like.
As shown in fig. 3 and 4, the classification accuracy of the support vector machine and the long-term memory-based neural network isotropic model is generally higher than that of the opposite sex model, which indicates that the isotropic data has more consistent data distribution, and there is a misclassification between the opposite sex data and the test set. And the accuracy of the neural network based on long-time and short-time memory is generally higher than that of a support vector machine, which shows that the time sequence information is fully utilized to be beneficial to improving the performance of the recognizer, namely: with the improvement of model performance, the difference between the same-sex model and the different-sex model becomes more obvious. The graph neural network is known to realize excellent performance in emotion recognition, so that the method is combined with a long-time memory module and the graph neural network, utilizes brain function connection information and time sequence information, and is more favorable for detecting gender difference of male and female emotion electroencephalogram signals.
As shown in fig. 5, the accuracy of the same sex model is higher than that of the opposite sex model under different emotions, which indicates that the electroencephalogram signals of men and women are different under different emotions. The electroencephalogram data of men and women with different emotions can show different signal modes, and the method can be applied to scenes such as auxiliary diagnosis and treatment of affective disorder diseases of men and women.
From fig. 6, it can be seen that the brain of the male is activated more in a limited area than the brain of the female, and a larger difference in brain energy can be found in aversion to the emotion. Gender differences related to emotion recognition are more significant in high frequency bands, such as beta and gamma bands, and can assist in the development of recognition tasks.
Through specific experiments, a PyTorch neural network framework is used, the learning rate is set to be 0.01 under the specific environment setting based on a plurality of data sets and unified experimental configuration, the Adam optimization method is applied, the classification accuracy of a same-sex model in the obtained experimental data is generally higher than that of a different-sex model, and the situation of large misclassification between the different-sex model and the tested data is explained; under most emotions, the accuracy of the same-sex model is higher than that of the opposite-sex model, and the same-sex data are proved to have higher similarity.
Compared with the prior art, the performance index of the method is improved as follows: the long-time and short-time memory module is combined with the graph neural network, brain function connection information and time sequence information are collected simultaneously, network model performance is enhanced, mode identification accuracy based on electroencephalogram signals is improved, and gender difference in electroencephalogram data is highlighted.
According to the invention, the long-time memory map neural network and the short-time memory map neural network are used for simultaneously capturing brain function connection information and time sequence information, the domain classifier is introduced to reduce the difference between a source domain and a target domain, emotion labels are converted into probability distribution to reduce label noise, the emotion recognition task of a plurality of commonly used and representative data sets is realized, the limitation problem existing in the existing research work is integrally solved, and the sex difference of electroencephalogram activities in emotion experience is verified.
Compared with the prior art, the unique new functions/effects of the invention comprise:
1) firstly, the electroencephalogram feature recognition method based on the long-time memory diagram neural network is provided, brain function connection information and time sequence information of feature data are collected simultaneously, and accuracy of electroencephalogram emotion recognition is improved. Meanwhile, a support vector machine and a neural network based on long-term and short-term memory are trained to serve as experimental baselines, and the characteristics of the performance and the gender difference of a new model are highlighted and generally exist in emotion-related electroencephalogram activities.
2) Secondly, the commonly used data sets including Chinese (SEED, SEED-IV and SEED-V) and European (DEAP and DREAMER) data sets are researched by using the international emotional brain-computer interface, so that the experiment is more representative, and the generality and the feasibility of the detection method are enhanced.
3) And finally, analyzing the experimental result and the expression form of common gender difference on the basis of a plurality of data sets and a high-performance model, wherein the expression form comprises a neural mode, a key brain area and a key frequency band. More objective and stable conclusion can be obtained, and the robustness is stronger.
The foregoing embodiments may be modified in many different ways by those skilled in the art without departing from the spirit and scope of the invention, which is defined by the appended claims and all changes that come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein.
Claims (10)
1. A brain electrical characteristic recognition method based on a long-and-short-term memory diagram neural network is characterized in that differential entropy characteristics are extracted from brain electrical data set, then the differential entropy characteristics are converted into characteristic matrixes of representation diagrams, then a long-and-short-term diagram neural network model is trained, brain function connection information and time sequence relation of the characteristic data are collected at the same time, and finally emotion recognition is achieved by the trained network model.
2. The method for electroencephalogram feature recognition based on the long-term and short-term memory map neural network as claimed in claim 1, wherein the training and the use of EEG electroencephalogram signal modal data comprise: china (SEED, SEED-IV, SEED-V), Europe (DEAP, DREAMER) data set.
3. The electroencephalogram feature recognition method based on the long-and-short-term memory map neural network as claimed in claim 1 or 2, characterized in that the training is to pre-process original data, i.e. to extract differential entropy frequency domain features as input data of an experiment, design a universal like-property training strategy and an opposite-property training strategy, train a recognizer for each tested data of each data set by using a leave-one-out cross-validation method to obtain a corresponding like-property model and an opposite-property model, adjust the hyper-parameters of each group of models in iteration, and obtain a final experiment result, and specifically comprises the following steps:
1) firstly, performing information encapsulation on each data set, dividing each data set into male data and female data, normalizing experimental data, and using the normalized experimental data as input of model training;
2) secondly, initializing the adjacency matrix learned by the graph convolution network:
and sets the global connection initial value to: a. theij＝Aij1, global connections of 9 pairs, which span the left and right hemispheres of the brain, maximizing the lateralization of the brain electrical signal and finding functional connectivity between hemispheres;
3) training the same-sex model and the opposite-sex model of each tested data by using a leave-one-out cross-validation method, specifically, taking any tested data as a test set, and taking all other tested electroencephalogram data with the same sex as a training set Xi_sameTraining a same sex model, and simultaneously using all the different sex data as a training set Xi_crossTraining a different sex model;
4) in a long-short time memory diagram neural network, a characteristic matrix sequence is formed
Inputting a long-time and short-time memory module; for the elements in the input sequence, the updating method of the memory module is respectively as follows: an input gate: i.e. it＝σ(Wixxt+bii+Wihht-1+bi) And forget to close the door: f. oft＝σ(Wfxxt+bif+Wfhht-1+bf) And a memory gate: gt＝tanh(Wcxxt+Wchht-1+bc) Memory cell status:
an output gate: ot＝σ(Woxxt+Wohht-1+bo)，ht＝ot*tanh(cf) Wherein: h istHidden layer state at time t, corresponding to ht-1At time t-1 or the initial hidden layer state, σ is a logistic sigmoid function, and is a Hadamard product;
5) for output X of long-and-short-time memory moduleiEach graph convolution module calculates: zi＝SLXiW, output ZiAnd learning the importance of each brain functional connection;
6) then the output of the graph convolution module passes through the pooling layer and the full-connection layer, and the emotion label Y is outputiProbability distribution of (2):
wherein,
the full connection layer takes a softmax function as an activation function, pool (-) is used for global sum pooling, and sigma (Z)i) Is to ZiEach element of (a) is non-linearly transformed: σ (x) max (0, x);
7) in the process of learning the nodes, a domain classifier is trained to learn domain invariant features, and the source domain X is reducedSAnd a target domain XTThe difference between them; the main task of the domain classifier is to minimize the cross-entropy loss function of two binary classification tasks:
the generalization capability of the model is enhanced, and the robustness in the cross-tested experiment is improved;
8) then, a learning process of the gradient inversion layer auxiliary domain classifier is utilized, and a calculation function of a gradient inversion layer factor is as follows:
wherein p is [0, 1 ]]Representing the progress of model training;
9) the emotion perception learner converts a single emotion label into prior probability distribution by using the noise level factor, and replaces the optimization problem of the graph convolution module with the problem of the minimized KL divergence function:
mitigating tag noise issues within the data set;
10) the loss function of the entire model becomes the calculation: phi ″ ═ phi' + phiDAnd finally, outputting a prediction result by using a single-layer full-connection network, respectively calculating the recognition accuracy of the isotropic model and the special-shaped model, and iterating the training process.
4. The electroencephalogram feature recognition method based on the long-and-short-term memory map neural network as claimed in claim 1, wherein the preprocessing refers to: the method comprises the steps of performing baseline correction on original data of electroencephalogram signals in a data set, performing down-sampling on the data to 200Hz, facilitating acceleration of a data analysis process, performing band-pass filtering within a range of 1-75Hz, and filtering noise and artifacts in the data.
5. The electroencephalogram feature identification method based on the long-and-short-term memory map neural network according to claim 1 or 2, wherein the differential entropy feature extraction is as follows: carrying out short-time Fourier transform on the preprocessed data, and extracting differential entropy characteristics of each lead on 5 frequency bands, wherein the characteristics are as follows: δ:1-4Hz, θ:4-8Hz, α: :8-14Hz, beta: 14-31Hz, and γ:31-50 Hz; because the SEED, SEED-IV and SEED-V use 62-lead electroencephalogram caps, the differential entropy characteristics of 310-dimensional electroencephalogram are shared; and because the original EEG data of DEAP and DREAMER filters delta frequency range and 32-lead and 14-lead EEG caps are used, 128-dimensional and 56-dimensional differential entropy characteristics are obtained respectively, and finally, the extracted EEG characteristics are smoothed by using a linear power system, so that quick jitter information irrelevant to emotion is eliminated.
6. The electroencephalogram feature recognition method based on the long-and-short-term memory map neural network as claimed in claim 1, wherein the feature transformation is as follows: using InMemoryDataset base class of PyTorch Geometric tool library, taking each lead of the extracted frequency domain characteristics as a point, and taking the connection among the leads as an edge, converting the characteristic data into
The feature matrix in the form, which is then converted into time series data over a time window T, constitutes the input data for the experiment.
7. The method for recognizing the electroencephalogram characteristics based on the long-short time memory map neural network according to claim 1, wherein the recognizer model is the long-short time memory map neural network which is a graph convolution neural network added with a memory module, and the network comprises: memory module, a plurality of picture convolution module, domain classifier, gradient reversal layer, emotion perception learner, pooling layer, full tie-layer when long and short term, wherein: the long-time and short-time memory module captures time sequence dependence information among the characteristic matrixes, the image volume module extracts brain function connection characteristic information related to emotional experience, the domain classifier is used for solving the efficiency problem of a scene crossing a tested scene, the gradient of the domain classifier is reversed by the gradient reversing layer during back propagation, the emotion perception learner aims at data label noise, the pooling layer pools output characteristics, and finally the full-connection layer is used for decoding the pooled characteristics and predicting an emotion label.
8. The method for recognizing electroencephalogram characteristics based on the long-short-time memory map neural network as claimed in claim 1, wherein the long-short-time memory module comprises one or more self-connected memory cells and three gate units, and for data of each time step, the memory cells of the neural network can extract information from a result of the previous step, so that the memory module can store time sequence dependent information of the data in a long time, and because electroencephalogram signals are also time sequence data, the memory module can capture and utilize the time sequence information therein, and the recognition accuracy is improved.
9. The method for recognizing electroencephalogram characteristics based on the long-short time memory map neural network as claimed in claim 1, wherein the map volume module captures local connection and global connection information among different leads by adopting a sparse adjacency matrix fitting an intracerebral network structure, the matrix is obtained by utilizing reciprocal calculation of physical distances among lead channels, the local connection displays the connectivity of a brain area in anatomy, and the global connection represents the functional connectivity of left and right hemispheres related to emotion.
10. The electroencephalogram feature recognition method based on the long-and-short-term memory map neural network as claimed in claim 1, wherein the domain classifier combines a transfer learning and confrontation training method to reduce the difference between a source domain and a target domain, enhance the generalization capability of a model and solve the problem of poor recognition efficiency in a cross-tested scene.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111335757.2A CN113951883B (en) | 2021-11-12 | 2021-11-12 | Gender difference detection method based on electroencephalogram signal emotion recognition |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111335757.2A CN113951883B (en) | 2021-11-12 | 2021-11-12 | Gender difference detection method based on electroencephalogram signal emotion recognition |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113951883A true CN113951883A (en) | 2022-01-21 |
| CN113951883B CN113951883B (en) | 2022-08-12 |
Family
ID=79470158
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111335757.2A Active CN113951883B (en) | 2021-11-12 | 2021-11-12 | Gender difference detection method based on electroencephalogram signal emotion recognition |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113951883B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115115038A (en) * | 2022-08-30 | 2022-09-27 | 合肥心之声健康科技有限公司 | Model construction method based on single lead electrocardiosignal and gender identification method |
| CN116700206A (en) * | 2023-05-24 | 2023-09-05 | 浙江大学 | Industrial control system anomaly detection method and device based on multi-modal neural network |
| CN118079185A (en) * | 2024-01-03 | 2024-05-28 | 天津大学 | Intelligent sleep regulation and control system based on brain-computer wearable interaction equipment |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190130808A (en) * | 2018-05-15 | 2019-11-25 | 연세대학교 산학협력단 | Emotion Classification Device and Method using Convergence of Features of EEG and Face |
| CN111419221A (en) * | 2020-02-14 | 2020-07-17 | 广东司法警官职业学院 | Electroencephalogram signal analysis method based on graph convolution network |
| CN111949812A (en) * | 2020-07-10 | 2020-11-17 | 上海联影智能医疗科技有限公司 | Brain image classification method and storage medium |
| CN113191225A (en) * | 2021-04-19 | 2021-07-30 | 华南师范大学 | Emotional electroencephalogram recognition method and system based on graph attention network |
| CN113288146A (en) * | 2021-05-26 | 2021-08-24 | 杭州电子科技大学 | Electroencephalogram emotion classification method based on time-space-frequency combined characteristics |
| CN113598774A (en) * | 2021-07-16 | 2021-11-05 | 中国科学院软件研究所 | Active emotion multi-label classification method and device based on multi-channel electroencephalogram data |
-
2021
- 2021-11-12 CN CN202111335757.2A patent/CN113951883B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20190130808A (en) * | 2018-05-15 | 2019-11-25 | 연세대학교 산학협력단 | Emotion Classification Device and Method using Convergence of Features of EEG and Face |
| CN111419221A (en) * | 2020-02-14 | 2020-07-17 | 广东司法警官职业学院 | Electroencephalogram signal analysis method based on graph convolution network |
| CN111949812A (en) * | 2020-07-10 | 2020-11-17 | 上海联影智能医疗科技有限公司 | Brain image classification method and storage medium |
| CN113191225A (en) * | 2021-04-19 | 2021-07-30 | 华南师范大学 | Emotional electroencephalogram recognition method and system based on graph attention network |
| CN113288146A (en) * | 2021-05-26 | 2021-08-24 | 杭州电子科技大学 | Electroencephalogram emotion classification method based on time-space-frequency combined characteristics |
| CN113598774A (en) * | 2021-07-16 | 2021-11-05 | 中国科学院软件研究所 | Active emotion multi-label classification method and device based on multi-channel electroencephalogram data |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115115038A (en) * | 2022-08-30 | 2022-09-27 | 合肥心之声健康科技有限公司 | Model construction method based on single lead electrocardiosignal and gender identification method |
| CN115115038B (en) * | 2022-08-30 | 2022-11-08 | 合肥心之声健康科技有限公司 | Model construction method based on single lead electrocardiosignal and gender identification method |
| CN116700206A (en) * | 2023-05-24 | 2023-09-05 | 浙江大学 | Industrial control system anomaly detection method and device based on multi-modal neural network |
| CN116700206B (en) * | 2023-05-24 | 2023-12-05 | 浙江大学 | Industrial control system anomaly detection method and device based on multi-modal neural network |
| CN118079185A (en) * | 2024-01-03 | 2024-05-28 | 天津大学 | Intelligent sleep regulation and control system based on brain-computer wearable interaction equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113951883B (en) | 2022-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Liu et al. | Multi-channel EEG-based emotion recognition via a multi-level features guided capsule network | |
| CN113951883B (en) | Gender difference detection method based on electroencephalogram signal emotion recognition | |
| Liu et al. | EEG emotion recognition based on the attention mechanism and pre-trained convolution capsule network | |
| CN114052735B (en) | Deep field self-adaption-based electroencephalogram emotion recognition method and system | |
| Mensch et al. | Learning neural representations of human cognition across many fMRI studies | |
| CN110598793B (en) | Brain function network feature classification method | |
| Zheng et al. | Adaptive neural decision tree for EEG based emotion recognition | |
| CN107845390A (en) | A kind of Emotional speech recognition system based on PCNN sound spectrograph Fusion Features | |
| Pusarla et al. | Learning DenseNet features from EEG based spectrograms for subject independent emotion recognition | |
| CN113947127A (en) | Multi-mode emotion recognition method and system for accompanying robot | |
| Prasetio et al. | The facial stress recognition based on multi-histogram features and convolutional neural network | |
| Yang et al. | Mlp with riemannian covariance for motor imagery based eeg analysis | |
| Jinliang et al. | EEG emotion recognition based on granger causality and capsnet neural network | |
| Boloukian et al. | Recognition of words from brain-generated signals of speech-impaired people: Application of autoencoders as a neural Turing machine controller in deep neural networks | |
| Kumari et al. | Automated visual stimuli evoked multi-channel EEG signal classification using EEGCapsNet | |
| Chen et al. | DCTNet: hybrid deep neural network-based EEG signal for detecting depression | |
| Liu et al. | A cross-session motor imagery classification method based on Riemannian geometry and deep domain adaptation | |
| CN114662524A (en) | Plug-and-play type domain adaptation method based on electroencephalogram signals | |
| Kwaśniewska et al. | Real-time facial features detection from low resolution thermal images with deep classification models | |
| Lu et al. | Deep learning solutions for motor imagery classification: A comparison study | |
| CN116821764A (en) | Knowledge distillation-based multi-source domain adaptive EEG emotion state classification method | |
| Saeedi et al. | Schizophrenia diagnosis via FFT and wavelet convolutional neural networks utilizing EEG signals | |
| Peng | Research on Emotion Recognition Based on Deep Learning for Mental Health | |
| Alharthi et al. | Do it the transformer way: a comprehensive review of brain and vision transformers for autism spectrum disorder diagnosis and classification | |
| CN115909438A (en) | Pain expression recognition system based on depth time-space domain convolutional neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220801 Address after: Room 23a, No. 19, Lane 99, Nandan East Road, Xuhui District, Shanghai 200030 Applicant after: Lv Baoliang Address before: 200240 No. 800, Dongchuan Road, Shanghai, Minhang District Applicant before: SHANGHAI JIAO TONG University |
|
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220913 Address after: Room 901, Building A, SOHO Fuxing Plaza, No. 388 Madang Road, Huangpu District, Shanghai, 200025 Patentee after: Shanghai Zero Unique Technology Co.,Ltd. Address before: Room 23a, No. 19, Lane 99, Nandan East Road, Xuhui District, Shanghai 200030 Patentee before: Lv Baoliang |