CN114145745A - Multi-task self-supervision emotion recognition method based on graph - Google Patents
Multi-task self-supervision emotion recognition method based on graph Download PDFInfo
- Publication number
- CN114145745A CN114145745A CN202111532664.9A CN202111532664A CN114145745A CN 114145745 A CN114145745 A CN 114145745A CN 202111532664 A CN202111532664 A CN 202111532664A CN 114145745 A CN114145745 A CN 114145745A
- Authority
- CN
- China
- Prior art keywords
- task
- graph
- emotion
- electroencephalogram
- self
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 62
- 230000008909 emotion recognition Effects 0.000 title claims abstract description 53
- 230000008451 emotion Effects 0.000 claims abstract description 79
- 238000013528 artificial neural network Methods 0.000 claims abstract description 42
- 210000004556 brain Anatomy 0.000 claims abstract description 34
- 238000012549 training Methods 0.000 claims abstract description 23
- 238000012360 testing method Methods 0.000 claims abstract description 21
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 238000000605 extraction Methods 0.000 claims description 16
- 238000010586 diagram Methods 0.000 claims description 10
- 230000006870 function Effects 0.000 claims description 7
- 238000013527 convolutional neural network Methods 0.000 claims description 5
- 238000013461 design Methods 0.000 abstract description 2
- 210000003128 head Anatomy 0.000 description 16
- 230000008569 process Effects 0.000 description 8
- 230000033764 rhythmic process Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 6
- 238000002474 experimental method Methods 0.000 description 6
- 230000002996 emotional effect Effects 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 3
- 208000035475 disorder Diseases 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 230000007935 neutral effect Effects 0.000 description 3
- 238000010276 construction Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000002779 inactivation Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 208000014644 Brain disease Diseases 0.000 description 1
- 206010041349 Somnolence Diseases 0.000 description 1
- 230000005856 abnormality Effects 0.000 description 1
- 230000004913 activation Effects 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 230000036541 health Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 230000003387 muscular Effects 0.000 description 1
- 230000000926 neurological effect Effects 0.000 description 1
- 210000002569 neuron Anatomy 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003334 potential effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 210000004761 scalp Anatomy 0.000 description 1
- 230000003860 sleep quality Effects 0.000 description 1
- 230000008667 sleep stage Effects 0.000 description 1
- 230000002269 spontaneous effect Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Psychiatry (AREA)
- Public Health (AREA)
- Surgery (AREA)
- Veterinary Medicine (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Animal Behavior & Ethology (AREA)
- Molecular Biology (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Psychology (AREA)
- Fuzzy Systems (AREA)
- Signal Processing (AREA)
- Evolutionary Computation (AREA)
- Physiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Mathematical Physics (AREA)
- Child & Adolescent Psychology (AREA)
- Developmental Disabilities (AREA)
- Educational Technology (AREA)
- Hospice & Palliative Care (AREA)
- Social Psychology (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
Abstract
The application belongs to the technical field of information, relates to an electroencephalogram signal classification method, and particularly provides a multi-task self-supervision emotion recognition method based on a graph, which is characterized by comprising the following steps: s1, acquiring electroencephalogram emotion data and preprocessing the electroencephalogram emotion data; s2, constructing an automatic supervision auxiliary task; s3, constructing a graph convolution neural network; s4, training a graph convolution neural network; and S5, testing the graph convolution neural network. The invention firstly considers the design of a self-supervision task for electroencephalogram emotion recognition. The designed space jigsaw task learns the spatial mode related to electroencephalogram emotion by learning the internal spatial relationship among different brain areas; the designed frequency jigsaw task aims at excavating a more key frequency band for emotion recognition; the designed contrast learning task aims to further standardize the feature space and learn the inherent characteristics. The emotion recognition method is high in accuracy rate of emotion recognition.
Description
Technical Field
The application belongs to the technical field of information, relates to a classification method of electroencephalogram signals, and particularly relates to a multi-task self-supervision emotion recognition method based on a graph, which can be used for medical services.
Background
The brain electrical signals are generated by bioelectrical activity of the brain neuron population, and belong to spontaneous electrical potential activity. Generally, electroencephalogram signals are divided into several different rhythms of delta, theta, alpha, beta and gamma according to frequency bands, and the electroencephalogram signals with different rhythms can reflect different physiological and psychological state information of a human body, wherein: the delta rhythm is mainly positioned in a frequency band of 1-3 Hz and reflects a deep sleep state of a human or a special brain disease; the theta rhythm is mainly positioned in a frequency band of 4-7 Hz and reflects the state that a person is in the early sleep stage, meditation, drowsiness or depression; the alpha rhythm is mainly positioned in a frequency band of 8-13 Hz, and the alpha rhythm also comprises mu waves related to body movement besides reflecting the state that a person is clear, quiet and closed; the beta rhythm is mainly positioned in a frequency band of 14-30 Hz, can reflect the state of people in mental stress, emotional excitement or active thinking and concentration, and also contains a part of information related to body movement; the gamma rhythm is mainly located in a frequency band of 31-50 Hz and comprises higher-level thinking activities, such as emotional changes, abstract thinking and other states.
A series of electroencephalogram signal acquisition and processing methods are derived according to different characteristic information of the electroencephalogram signals. At present, the electrode is generally connected with the scalp of a human body through a conductive paste to obtain an electroencephalogram signal, and the method is called a non-invasive electroencephalogram acquisition method. The brain electrical signal acquisition and processing technology can be applied to the research and development of brain-computer interface BCI, medical services and the research of human cognitive activities.
Brain-computer interfaces have been sourced in the seventies of the last century, and early BCI was mainly used for medical services, and was generally designed for serious patients with neurological or muscular disabilities, such as brain-controlled wheelchairs, brain-controlled text input devices, brain-controlled prostheses and robotic arms, and the like. With the advance of research, BCI continuously plays a greater role in medical and rehabilitation applications, and gradually shows wider application potential. In the field of education, student headrings for constantly feeding back teaching quality of teachers are put into the market, and daily life scenes can be embedded, so that attention can be enhanced and attention can be improved; in the health care industry, electroencephalogram-based sleep quality monitoring and improving instruments are available; in the aspect of household entertainment, the brain control interactive system provides a brand-new game form, can be used for enhancing virtual reality and augmented reality and improving user experience; in a special industry, an electroencephalograph is used for monitoring the emotion change and the working state of a user, and timely intervention is performed when the emotion abnormality or fatigue working of the user is found, so that the major loss is avoided; in the military, the united states et al attempted to improve individual combat capability via BCI. For cost and portability reasons, such BCIs typically acquire brain electrical signals using non-invasive methods.
The electroencephalogram emotion recognition method mainly comprises a traditional machine learning method and a deep learning-based method. The traditional machine learning method mainly comprises a method of extracting features by using a linear SVM (support vector machine) by using a multilayer perceptron; the deep learning method mainly comprises the steps of extracting electroencephalogram signal emotion information by a convolutional neural network or a cyclic convolutional neural network and a full-connection layer. With the rise of the atlas neural network, the idea of further researching the electroencephalogram emotion by utilizing the topological structure of the electroencephalogram signal also rises. Therefore, a method for electroencephalogram emotion recognition based on the atlas neural network is also provided. The graph convolution neural network input comprises original electroencephalogram data and a graph structure constructed by modeling an electrode, and the transmission of information between nodes is realized according to the connection relation between the nodes in the graph structure. And after the graph convolution operation finishes feature extraction, sending the features into a full connection layer to realize classification. Representative work of the method is that Song et al proposes a method of dynamic graph convolution in the 'EEG annotation using dynamic graph convolution network', and dynamically learns the internal relationship between different electroencephalogram channels to help emotion recognition. However, the exploration and utilization of electroencephalogram data by the existing traditional machine learning method and the deep learning method based on the convolutional neural network, the cyclic neural network and the graph convolutional neural network are still shallow, and the self attributes of electroencephalogram signals such as complementarity of brain space adjacent information and correlation of brain signal emotion frequency characteristics are not fully utilized, so that the electroencephalogram emotion recognition effect is poor, and the accuracy is low.
Disclosure of Invention
The invention aims to provide a diagram-based multi-task self-monitoring emotion recognition method aiming at the defects in the prior art, and the method is used for solving the problems that in the prior art, the electroencephalogram emotion recognition effect is poor and the accuracy is low due to the fact that the complementarity of brain space adjacent information, the correlation of brain signal emotion frequency characteristics and other attributes of electroencephalograms are not fully utilized.
In order to achieve the purpose, the technical idea of the invention is as follows: the method comprises the steps of designing a plurality of self-supervision auxiliary tasks based on a self-supervision learning theory, and completing the identification of electroencephalogram emotion signals by utilizing a graph convolution neural network based on a multi-task learning theory. Specifically, the following technical scheme is adopted for implementation.
The application provides a multitask self-supervision emotion recognition method based on a graph, which specifically comprises the following steps: s1, acquiring electroencephalogram emotion data and preprocessing the electroencephalogram emotion data; s2, constructing an automatic supervision auxiliary task; s3, constructing a graph convolution neural network; s4, training a graph convolution neural network; and S5, testing the graph convolution neural network.
Further, the preprocessing in step S1 includes data segment selection, baseline removal, power frequency removal, and feature preliminary extraction on the electroencephalogram emotion data.
Further, the self-supervision auxiliary tasks constructed in step S2 include a space puzzle task, a frequency puzzle task, and a contrast learning task.
Furthermore, each electroencephalogram emotion data is divided into 10 blocks according to different brain areas by the space jigsaw task.
Furthermore, each electroencephalogram emotion data is divided into 5 blocks by the frequency jigsaw task according to different frequency band intervals.
Furthermore, the comparison learning task makes the electroencephalogram emotion data into space blocks and frequency band blocks of different brain areas.
Furthermore, the graph convolution neural network constructed in the step S3 includes a graph input module, a feature extraction module and a classification module.
Still further, the feature extraction module includes a chebyshev polynomial based graph convolution operation.
Further, the loss function in step S4 is a cross entropy loss function.
Further, in step S4, the trained standard of the graph convolution neural network is determined such that no overfitting occurs.
Compared with the prior art, the invention has the beneficial effects that:
(1) the invention considers the use of a multi-task framework for electroencephalogram emotion recognition for the first time. Through knowledge sharing of a plurality of emotion related tasks, mutual promotion is achieved, and a more generalized emotion recognition model is obtained. The effect of electroencephalogram emotion recognition is improved, and the recognition accuracy is improved.
(2) The invention firstly considers the design of a self-supervision task for electroencephalogram emotion recognition. The designed space jigsaw task learns the spatial mode related to electroencephalogram emotion by learning the internal spatial relationship among different brain areas; the designed frequency jigsaw task aims at excavating a more key frequency band for emotion recognition; the designed contrast learning task aims to further standardize the feature space and learn the inherent characteristics. The invention considers the property of the brain electrical signals, fully utilizes the complementarity of brain space adjacent information and the correlation of the emotion frequency characteristics of the brain signals, improves the effect of brain electrical emotion recognition, and improves the recognition accuracy.
Drawings
FIG. 1 is a diagram of a diagram-based multitask self-supervised learning framework in a diagram-based multitask self-supervised emotion recognition method provided by the present invention;
FIG. 2 is a schematic diagram illustrating steps of a graph-based method for self-supervised emotion recognition;
fig. 3 is a construction method of a space jigsaw task in step S2 of the multitask self-supervision emotion recognition method based on the graph provided by the present invention;
fig. 4 is a frequency jigsaw task constructing method in step S2 of the diagram-based multitask self-supervised emotion recognition method provided by the present invention;
fig. 5 is a comparative learning task data pair construction method in step S2 of the graph-based multitask self-supervised emotion recognition method provided by the present invention.
Detailed Description
In order to make the implementation of the present invention clearer, the following detailed description is made with reference to the accompanying drawings.
The invention provides a diagram-based multitask self-supervision emotion recognition method, and a diagram-based multitask self-supervision learning frame diagram in the invention is shown in figure 1. As shown in fig. 2, the present invention comprises the steps of:
s1, acquiring electroencephalogram emotion data and preprocessing the electroencephalogram emotion data;
the acquisition process of the electroencephalogram emotion data comprises the following steps: the testee wears the electrode cap to watch the emotion-induced video, and in the process of watching the emotion-induced video, electroencephalogram emotion data of the testee are collected through the electrodes on the electrode cap. The electroencephalogram emotion data acquired by the method can be acquired data of a testee, can also be an existing data set, and can also be a new data set formed by the acquired data of the testee and the existing data set. The invention takes the SEED and SEED-IV electroencephalogram emotion data sets of Shanghai university of traffic and the MPED electroencephalogram emotion data set of the university of southeast as examples, wherein the SEED electroencephalogram emotion data set of the Shanghai university of traffic is a happy, sad and neutral data set, the SEED-IV electroencephalogram emotion data set is a happy, sad, fear and neutral data set, and the MPED electroencephalogram emotion data set of the university of southeast is a happy, interesting, angry, fear, disgust, sadness and neutral data set. And (3) carrying out data segment selection, baseline removal, power frequency removal and primary feature extraction on the acquired electroencephalogram emotion data. The SEED, SEED-IV electroencephalogram emotion data sets of Shanghai university of transportation and the MPED electroencephalogram emotion data sets of southeast university all adopt 62-channel acquisition equipment, the electrode distribution adopts an international 10-20 lead system, 15 and 30 testees participate in data acquisition, and the electroencephalogram emotion data are cut into 1s segments. SEED, SEED-IV electroencephalogram emotion data sets of the Shanghai university of transportation and MPED electroencephalogram emotion data sets of the southeast university are respectively assigned 9: 6. 16: 8. 21: 7, dividing the data into training set data and test set data to carry out the independent test of the testee, so that the division is convenient to compare with other results, and carrying out the independent test of the testee by using a leave-one-out cross validation strategy. The input data required for the emotion recognition task is thus obtained.
S2, constructing an automatic supervision auxiliary task;
and (4) constructing three self-monitoring auxiliary tasks by utilizing the electroencephalogram emotion data acquired in the step (S1), namely a space jigsaw task, a frequency jigsaw task and a contrast learning task. And simultaneously constructing three self-supervision auxiliary tasks of a space jigsaw task, a frequency jigsaw task and a contrast learning task and executing the three tasks on all data in the data set. The method is characterized in that the spatial mode of the electroencephalogram emotion signal is learned by solving a spatial picture splicing task, the frequency band information which is more critical to electroencephalogram emotion recognition is mined by solving a frequency picture splicing task, the characteristic space is further standardized by solving a contrast learning task, and the internal representation of the electroencephalogram signal is learned. Meanwhile, the three related tasks are solved, so that the method can obtain emotional space characteristics and frequency characteristics with generalization, the electroencephalogram emotional signal discrimination capability of the method is strong, and the accuracy of electroencephalogram emotional recognition is high.
The space jigsaw task is to divide the electroencephalogram emotion data into blocks according to different brains and to disorder the blocks to obtain randomly arranged electroencephalogram emotion data, and a label is distributed to the arrangement sequence, wherein the label is a pseudo label. As shown in fig. 3, each electroencephalogram emotion data obtained in step S1 is partitioned into blocks according to different brains, each electroencephalogram emotion data is partitioned into 10 blocks, the space jigsaw task is equivalently converted into an arrangement recognition task, and a new arrangement sequence is obtained after random shuffle. The method provides a basis for the graph convolution neural network to learn the semantic characteristics of the puzzles in each space and the semantic correlation among the puzzles in each space. The method fully utilizes the spatial correlation of different brain areas and fully learns the spatial mode of the electroencephalogram emotion signal, thereby improving the accuracy of electroencephalogram emotion signal identification.
The frequency jigsaw task is to divide the electroencephalogram emotion data into blocks according to different frequency intervals and to disorder the blocks to obtain randomly arranged electroencephalogram emotion data, and a label is allocated to the arrangement sequence, wherein the label is a pseudo label. As shown in fig. 4, each part of electroencephalogram emotion data obtained in step S1 is partitioned into blocks according to different frequency bands, one part of electroencephalogram emotion data is partitioned into 5 blocks, a frequency puzzle task is equivalently converted into an arrangement recognition problem, and random scrambling is performed to obtain a new arrangement sequence. The method provides a basis for the graph convolution neural network to learn the semantic characteristics of each frequency jigsaw and the semantic correlation among each frequency jigsaw. The method fully utilizes the frequency band characteristic of the electroencephalogram emotion signal, fully learns more key frequency information for classifying the electroencephalogram emotion signal, and accordingly improves the accuracy of electroencephalogram emotion signal identification.
The contrast learning task is to disorder electroencephalogram emotion data according to different brain areas and different frequency bands, and as shown in fig. 5, the electroencephalogram emotion data are partitioned into space blocks and frequency bands, so that positive and negative pair data are constructed, and the positive and negative pair similarity is maximized. By maximizing the positive alignment similarity, the extracted positive alignment features are close to the feature space, and the extracted negative alignment features are far away from the feature space. Therefore, the characteristic space of the model can be further normalized, so that the internal representation of the electroencephalogram emotion signal can be conveniently learned, and the accuracy of electroencephalogram emotion recognition can be improved.
The step also comprises the step of carrying out maximum and minimum normalization on electroencephalogram emotion data and original data generated by the space jigsaw task, the frequency jigsaw task and the comparison learning task, so that a result value is mapped between [0 and 1], gradient descent can be accelerated, and the solving speed of the optimal solution is higher.
S3, constructing a graph convolution neural network;
the graph convolution neural network comprises a graph input module, a feature extraction module and a classification module. The graph input module can simultaneously process a plurality of task inputs; the feature extraction module is a Chebyshev polynomial-based graph convolution neural network, is a sharing module and is used for extracting data features; the classification module is formed by a plurality of graph convolution classifications and corresponds to and classifies corresponding input features.
A graph input module: according to the multi-task learning theory, original data, result data of the space jigsaw task, result data of the frequency jigsaw task and positive and negative pair data of the comparison learning task structure are input into a graph convolution neural network at the same time. The classification task of the original data corresponds to a main task, and the result data of the space jigsaw task, the result data of the frequency jigsaw task and the positive and negative pair data input of the contrast learning task structure correspond to three self-supervision auxiliary tasks. The input of the result data of the space jigsaw task, the result data of the frequency jigsaw task and the positive and negative pair data of the structure of the comparison learning task comprises electroencephalogram emotion related space information and frequency information, so that the emotion recognition process of the main task takes the complementarity of brain space adjacent information and the correlation of brain signal emotion frequency characteristics into consideration, and the accuracy of electroencephalogram emotion recognition is improved; meanwhile, each task is mutually promoted in the process of continuously iterative learning, so that the electroencephalogram emotion recognition capability of the method can be further improved.
A feature extraction module: the feature extraction module shares four tasks, namely, the multi-task knowledge sharing is realized. Therefore, the feature extraction module extracts the spatial information and the frequency information corresponding to the three self-supervision auxiliary tasks, and the emotion recognition process of the main task fully considers the complementarity of brain spatial adjacent information and the correlation of brain signal emotion frequency features through knowledge sharing, so that the electroencephalogram emotion recognition accuracy is high. The feature extraction is completed by a graph convolution neural network based on a Chebyshev polynomial, and the expression of the Chebyshev polynomial is as follows:
σ (-) is an activation function, X is the input data, βkIs a parameter learned during network training, Tk(. is) a Chebyshev polynomial of order K, K2, λmaxIs the maximum eigenvalue of the laplacian matrix L. For each task, the input data are provided with manual features, so that the input feature dimension is 62 x 5, and the output feature dimension is 62 x 32, so that the output feature dimension is higher than the input feature dimension, has richer high-dimensional feature information, has higher abstraction degree of the high-dimensional feature information, and contains richer semantic information, and therefore, the electroencephalogram emotion recognition accuracy of the method is higher.
A classification module: the classification function is completed through the full connection layer. The space puzzle task corresponds to a space classification head, the space classification head generates a prediction space classification label, and the output dimension of the space classification head is 128, namely the prediction result of the prediction space classification label is one of 128 types. The number of the predicted space category labels is 128, namely the space information considered in the method is more detailed and comprehensive, so that the method has higher identification accuracy. The frequency jigsaw task corresponds to a frequency classification head, the frequency classification head generates a predicted frequency classification label, the output dimension of the frequency classification head is 120, namely the prediction result of the predicted frequency classification label is one of 120 classes, and the number of the predicted frequency classification labels is 120. And comparing projection heads corresponding to the learning tasks, adopting simple and effective cosine similarity measurement data pairs to measure the similarity, and maximizing the similarity between the pairs, so that similar samples are closer in a feature space, similar samples are gathered together to form a group, the completion of classification tasks is facilitated, and the accuracy of classification results is higher. The emotion recognition task, namely the main task, corresponds to an emotion classification head, and classifies the original electroencephalogram emotion data of the SEED, the SEED-IV and the MPED data sets, and the corresponding output dimensions are 3, 4 and 7 respectively, which are consistent with the number of emotion classes in the step S1 respectively. The label generated by the emotion classification head is a predicted emotion label, the label generated by the space classification head is a predicted space category label, the label generated by the frequency classification head is a predicted frequency category label, and the projection head does not generate a label. Through multi-task knowledge sharing in the feature extraction module, the generation process of the predicted emotion label already comprises complementarity of brain space adjacent information and correlation of brain signal emotion frequency features, therefore, the predicted emotion label is a final prediction result, and the accuracy of electroencephalogram emotion recognition is high due to the fact that the complementarity of the brain space adjacent information and the correlation of the brain signal emotion frequency features are considered in the generation process.
S4, training a graph convolution neural network;
and (4) jointly training the atlas neural network constructed in the step S31 by using the electroencephalogram emotion data of the training set in the step S1 and the self-supervision auxiliary task constructed in the step S2. And (4) using the test set to test the graph convolution neural network in training, if overfitting occurs, adjusting the learning rate to retrain the graph convolution neural network again, and finally obtaining the preliminarily trained graph convolution neural network.
S41, setting the training times as 100-:
S42, 100 samples are taken from the training set each time, corresponding data are generated for each self-supervision task according to the step S2, the data are sent to a graph input module in the graph convolution neural network constructed in the step S3, the characteristics of the electroencephalogram emotion signal are obtained through a characteristic extraction module, and then the characteristics of the electroencephalogram emotion signal are sequentially subjected to nonlinear, downsampling and logarithmic processing and random inactivation and then are sent to a corresponding classification module for classification.
S43, calculating the cross entropy loss for the predicted space category label generated by the space jigsaw task according to the space classification head and the corresponding pseudo label in the step S2; calculating cross entropy loss for the predicted frequency category label generated by the frequency puzzle task according to the frequency classification head and the corresponding pseudo label in the step S2; for the contrast learning task, calculating the normalized cross entropy loss by maximizing the positive similarity; both cross-entropy loss and normalized cross-entropy loss are used for back propagation. For the main task of electroencephalogram emotion recognition, calculating cross entropy loss according to the predicted emotion label and the sample real label; and updating the parameters of the convolution layer and the full connection layer in the graph convolution neural network by the graph convolution neural network optimizer. The cross entropy loss is used for back propagation, and network parameters are updated so as to achieve the purpose of minimizing the cross entropy loss, and the smaller the cross entropy loss is, the higher the emotion recognition accuracy is, thereby improving the emotion recognition accuracy of the method.
And S44, traversing all samples in the training set, completing 1 training, performing a test once every 10 times of iterative training, performing the test by using the test set data, and respectively calculating the accuracy of the graph convolution neural network on the training set and the test set.
S45, comparing the accuracy of the graph convolution neural network on the training set and the testing set. If the accuracy of the graph convolution neural network on the training set is higher than that on the test set by more than 20% along with the increase of the training times, the overfitting occurs, at the moment, the learning rate is reduced, the step S42 is returned, and the training is carried out again. If the accuracy of the training set and the accuracy of the testing set are within 20% with the increase of the training times, the graph convolution neural network after the training is obtained after 1000 times of training.
And S5, testing the graph convolution neural network.
And (4) directly sending the electroencephalogram emotion data in the test set in the step (S1) into the trained atlas neural network for classification, and counting classification results to obtain the identification accuracy of the atlas neural network on the test set. If the accuracy of the test cannot reach the required accuracy, the learning rate needs to be adjusted, and the step S4 is repeated to train the graph convolution neural network again until the required accuracy is met.
On the SEED data set, the accuracy of the test result depending on the experiment result is 96.48 percent, which is higher than 91.70 percent obtained by Li et al in the text of 'A Novel Bi-hemistatistical diagnosis Model for EEG emission Recognition' by using the most advanced BiHDM method under the non-domain self-adaptive experiment protocol; the accuracy of the results of the subject's independent experiment was 86.52%, which is higher than 81.55% disclosed in the above article by plum et al. On the SEED-IV data set, the accuracy of the subject's dependence on the experimental results was 86.37%, which is higher than 72.22% disclosed in the above article by plum et al; the accuracy of the results of the subject's independent experiment was 73.48%, which is higher than 67.47% disclosed in the above article by plum et al. On the MPED data set, the accuracy of the test results is 40.16% depending on the experiment results, which is higher than 38.55% disclosed in the above article by plum et al; the subject's independent experiment results in an accuracy of 28.49%, which is higher than 27.43% disclosed in the above article by plum et al. The invention obtains the result of state-of-the-art under the experimental protocol of the domain-free self-adaptation.
The above is only a preferred embodiment of the present invention, and is not intended to limit the present invention, and various modifications and changes will occur to those skilled in the art. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (10)
1. A diagram-based multitask self-supervised emotion recognition method is characterized by comprising the following steps:
s1, acquiring electroencephalogram emotion data and preprocessing the electroencephalogram emotion data;
s2, constructing an automatic supervision auxiliary task;
s3, constructing a graph convolution neural network;
s4, training the graph convolution neural network;
s5, testing the graph convolution neural network.
2. The graph-based multitask self-supervision emotion recognition method of claim 1, wherein said preprocessing in step S1 includes data segment selection, baselining, power frequency removal, and preliminary feature extraction for electroencephalogram emotion data.
3. The graph-based multi-task self-supervised emotion recognition method of claim 2, wherein the self-supervised auxiliary tasks constructed in step S2 include a space jigsaw task, a frequency jigsaw task and a contrast learning task.
4. The graph-based multi-task self-supervised emotion recognition method of claim 3, wherein the space jigsaw task divides each electroencephalogram emotion data into 10 blocks according to different brain regions.
5. The graph-based multi-task self-supervised emotion recognition method of claim 4, wherein the frequency jigsaw task divides each electroencephalogram emotion data into 5 blocks according to different frequency band intervals.
6. The graph-based multi-task self-supervised emotion recognition method of claim 5, wherein the contrast learning task performs electroencephalogram emotion data as both spatial and frequency band blocks of different brain regions.
7. The graph-based multitask self-supervised emotion recognition method of claim 6, wherein the graph convolution neural network constructed in the step S3 comprises a graph input module, a feature extraction module and a classification module.
8. The graph-based, multi-tasking, self-supervised emotion recognition method of claim 7, wherein the feature extraction module comprises a chebyshev polynomial based graph convolution operation.
9. The graph-based multitask, self-supervised emotion recognition method of claim 8, wherein said loss function in step S4 is a cross entropy loss function.
10. The graph-based multi-task self-supervised emotion recognition method of claim 9, wherein the trained criteria for the graph convolutional neural network are set to the condition that overfitting does not occur in step S4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111532664.9A CN114145745B (en) | 2021-12-15 | 2021-12-15 | Graph-based multitasking self-supervision emotion recognition method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111532664.9A CN114145745B (en) | 2021-12-15 | 2021-12-15 | Graph-based multitasking self-supervision emotion recognition method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114145745A true CN114145745A (en) | 2022-03-08 |
| CN114145745B CN114145745B (en) | 2024-03-29 |
Family
ID=80451213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111532664.9A Active CN114145745B (en) | 2021-12-15 | 2021-12-15 | Graph-based multitasking self-supervision emotion recognition method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114145745B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114626454A (en) * | 2022-03-10 | 2022-06-14 | 华南理工大学 | Visual emotion recognition method integrating self-supervision learning and attention mechanism |
| CN114758298A (en) * | 2022-04-22 | 2022-07-15 | 华南理工大学 | Multitask-based self-supervision target detection method, system, device and medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190042952A1 (en) * | 2017-08-03 | 2019-02-07 | Beijing University Of Technology | Multi-task Semi-Supervised Online Sequential Extreme Learning Method for Emotion Judgment of User |
| CN110399857A (en) * | 2019-08-01 | 2019-11-01 | 西安邮电大学 | A kind of brain electricity emotion identification method based on figure convolutional neural networks |
| CN110897648A (en) * | 2019-12-16 | 2020-03-24 | 南京医科大学 | Emotion recognition classification method based on electroencephalogram signal and LSTM neural network model |
| WO2021006770A2 (en) * | 2019-07-10 | 2021-01-14 | Общество С Ограниченной Ответственностью "Комплект-Ом" | System and method for monitoring and teaching children with autistic spectrum disorders |
| CN113158949A (en) * | 2021-04-30 | 2021-07-23 | 湖北工业大学 | Motor imagery electroencephalogram signal classification method based on self-supervision learning |
| CN113362295A (en) * | 2021-05-28 | 2021-09-07 | 西安医学院第二附属医院 | Liver tumor identification method based on self-supervision dense convolutional neural network |
| CN113469238A (en) * | 2021-06-29 | 2021-10-01 | 中山大学 | Self-supervision learning method for solving puzzle task based on CRNN |
-
2021
- 2021-12-15 CN CN202111532664.9A patent/CN114145745B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190042952A1 (en) * | 2017-08-03 | 2019-02-07 | Beijing University Of Technology | Multi-task Semi-Supervised Online Sequential Extreme Learning Method for Emotion Judgment of User |
| WO2021006770A2 (en) * | 2019-07-10 | 2021-01-14 | Общество С Ограниченной Ответственностью "Комплект-Ом" | System and method for monitoring and teaching children with autistic spectrum disorders |
| CN110399857A (en) * | 2019-08-01 | 2019-11-01 | 西安邮电大学 | A kind of brain electricity emotion identification method based on figure convolutional neural networks |
| CN110897648A (en) * | 2019-12-16 | 2020-03-24 | 南京医科大学 | Emotion recognition classification method based on electroencephalogram signal and LSTM neural network model |
| CN113158949A (en) * | 2021-04-30 | 2021-07-23 | 湖北工业大学 | Motor imagery electroencephalogram signal classification method based on self-supervision learning |
| CN113362295A (en) * | 2021-05-28 | 2021-09-07 | 西安医学院第二附属医院 | Liver tumor identification method based on self-supervision dense convolutional neural network |
| CN113469238A (en) * | 2021-06-29 | 2021-10-01 | 中山大学 | Self-supervision learning method for solving puzzle task based on CRNN |
Non-Patent Citations (2)
| Title |
|---|
| 张俊晓;薄华;: "基于卷积神经网络的脑电情绪识别研究", 现代计算机(专业版), no. 23, pages 14 - 18 * |
| 张娜;唐贤伦;刘庆;: "基于半监督学习的脑电信号特征提取及识别", 工程科学与技术, no. 2, pages 233 - 240 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114626454A (en) * | 2022-03-10 | 2022-06-14 | 华南理工大学 | Visual emotion recognition method integrating self-supervision learning and attention mechanism |
| CN114758298A (en) * | 2022-04-22 | 2022-07-15 | 华南理工大学 | Multitask-based self-supervision target detection method, system, device and medium |
| CN114758298B (en) * | 2022-04-22 | 2024-06-04 | 华南理工大学 | Self-supervision target detection method, system, device and medium based on multitasking |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114145745B (en) | 2024-03-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110765920B (en) | Motor imagery classification method based on convolutional neural network | |
| Ditthapron et al. | Universal joint feature extraction for P300 EEG classification using multi-task autoencoder | |
| CN111728609B (en) | Electroencephalogram signal classification method, classification model training method, device and medium | |
| Li et al. | Multi-modal bioelectrical signal fusion analysis based on different acquisition devices and scene settings: Overview, challenges, and novel orientation | |
| CN109784023B (en) | Steady-state vision-evoked electroencephalogram identity recognition method and system based on deep learning | |
| CN114145745B (en) | Graph-based multitasking self-supervision emotion recognition method | |
| Ma et al. | Capsule network for ERP detection in brain-computer interface | |
| Anwar et al. | EEG signal classification using convolutional neural networks on combined spatial and temporal dimensions for BCI systems | |
| CN113180692A (en) | Electroencephalogram signal classification and identification method based on feature fusion and attention mechanism | |
| Abibullaev et al. | A brute-force CNN model selection for accurate classification of sensorimotor rhythms in BCIs | |
| CN114662547A (en) | MSCRNN emotion recognition method and device based on electroencephalogram signals | |
| Mirzabagherian et al. | Temporal-spatial convolutional residual network for decoding attempted movement related EEG signals of subjects with spinal cord injury | |
| CN114145744A (en) | Cross-device forehead electroencephalogram emotion recognition method and system | |
| Zhu et al. | RAMST-CNN: a residual and multiscale spatio-temporal convolution neural network for personal identification with EEG | |
| CN114052734B (en) | Electroencephalogram emotion recognition method based on progressive graph convolution neural network | |
| CN113057652A (en) | Brain load detection method based on electroencephalogram and deep learning | |
| Mitra et al. | Analyzing clinical 12-lead ECG images using deep learning algorithms for objective detection of cardiac diseases | |
| Si et al. | Temporal aware mixed attention-based convolution and transformer network (MACTN) for EEG emotion recognition | |
| Hasan et al. | Emotion prediction through EEG recordings using computational intelligence | |
| Sun et al. | MEEG-Transformer: transformer Network based on Multi-domain EEG for emotion recognition | |
| Xu et al. | Emotional recognition of EEG signals based on fractal dimension | |
| CN111897428B (en) | Gesture recognition method based on moving brain-computer interface | |
| Tahira et al. | Eeg based mental stress detection using deep learning techniques | |
| Singh et al. | Electroencephalography-based classification of human emotion: a hybrid strategy in machine learning paradigm | |
| Li | A Dual-Modality Emotion Recognition System of EEG and Facial Images and its |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |