CN114004260A

CN114004260A - Emotion electroencephalogram data enhancement model based on generative confrontation network and method for expanding samples

Info

Publication number: CN114004260A
Application number: CN202111321899.3A
Authority: CN
Inventors: 曾颖; 包广城; 童莉; 舒君; 闫镔; 张融恺; 杨凯
Original assignee: Information Engineering University of PLA Strategic Support Force
Current assignee: Information Engineering University of PLA Strategic Support Force
Priority date: 2021-11-09
Filing date: 2021-11-09
Publication date: 2022-02-01

Abstract

The invention discloses an emotion electroencephalogram data enhancement model based on a generative confrontation network and a method for expanding samples, wherein the input of the model is an emotion electroencephalogram topological graph, a VAE maps the emotion electroencephalogram topological graph to a potential space, a potential vector is input into a generator, the spatial distribution of the emotion electroencephalogram topological graph is learned through the potential vector, the generative confrontation network corresponding to the model is provided with two discriminators and one generator, the two discriminators are both formed by a neural network, the structure is the same, but parameters are not shared; one of the discriminators awards a high score to the sample conforming to the true data distribution; another evaluator awarding a high score reward to the generator generated sample; the game between the generator and the two discriminators enables the generator to generate a distribution of synthetic samples that approaches the distribution of the real data infinitely. The emotion electroencephalogram data enhancement model provided by the invention fully exerts respective advantages by combining the variational self-encoder and the generation type confrontation network, and the generated samples have better diversity.

Description

Emotion electroencephalogram data enhancement model based on generative confrontation network and method for expanding samples

Technical Field

The invention belongs to the technical field of electroencephalogram data processing, and particularly relates to an emotion electroencephalogram data enhancement model based on a generating type confrontation network and a sample expansion method.

Background

Emotion calculation, i.e. "calculation related to, resulting from or affecting emotion". The method has wide application scenes in the aspects of computer-aided learning, perception information retrieval, art and entertainment, human health and interaction, wearing equipment and the like. There are studies that have shown that some psychiatric disorders, such as depression and autism, are associated with changes in mood processing. There are many methods for emotion recognition, which are mainly classified into two categories. One is to recognize human emotions using behavioral characteristics of emotions such as facial expressions, body movements, sounds, and the like. The other is to use physiological signals to identify emotion, including electrocardio, respiratory frequency, myoelectricity, eye movement signals, electroencephalogram signals and the like. Physiological signals can produce more reliable recognition results than the former. Among physiological signals, however, the EEG signal has advantages of high time resolution and high recognition accuracy, and is considered as one of the most reliable signals.

In recent years, deep learning research in emotion recognition based on electroencephalogram signals is increasing, but a key problem is ignored: lack of electroencephalogram data. It is well known that deep neural networks require a large amount of data to train in order to obtain a good performance model, such as the classical image processing networks ResNet18, Vgg16 and AlexNet. They all require a large amount of data to train millions or even hundreds of millions of parameters. However, electroencephalogram data acquisition requires expensive acquisition equipment, time, and manpower compared to image data and voice data. These are all the causes of lack of electroencephalogram data. Currently, the number of open image datasets like ImageNet, CIFAR-10, etc. has reached tens or even tens of millions. The common data set SEED, DEAP, mahnobo-HCI for EEG emotions is much smaller compared to the image data. In addition, as is well known, due to the non-stationarity of the electroencephalogram signal, each subject and even each link can have great changes, so that each individual experiment needs to be matched. This process requires consideration of the differences of each individual experiment, making matching difficult, which will further impact the training of the machine learning model. Furthermore, training deep neural network models is a significant challenge because the amount of data per test is too small.

One of the methods for solving the data shortage is to generate new data by transforming the original data, and the newly generated data has a data distribution similar to that of the original data, and this method is called data expansion. Data augmentation methods are generally classified into conventional methods and machine learning-based methods. Conventional data expansion methods include geometric transformations (translation, flipping, rotation, etc.) and noise addition. Compared with image processing, the traditional method is not friendly to electroencephalogram signals, and because the electroencephalogram signals are time sequences, the operations of translation, turnover, rotation and the like on the electroencephalogram signals cannot be performed. If noise is added to the brain electrical signal, the amplitude and data distribution of the original signal will be changed. In the electroencephalogram emotion recognition based research, a researcher extracts characteristics of electroencephalogram signals firstly, then Gaussian noise is added to the characteristics, and a new characteristic sample is generated.

Data enhancement methods based on machine learning have become very popular in recent years, including generative countermeasure networks (GANs) and Variational Automatic Encoders (VAEs). Because GAN can generate artificial data similar to original data, many researchers use GAN to generate artificial images to expand data, improving the recognition rate and stability of the images. The GAN can capture global information of data, but training is unstable, and pattern collapse is easy to occur, so that the generated data is insufficient in diversity. The VAE consists of an encoder and a decoder, the purpose of which is to generate new data for a given data reconstruction. The VAE generates a potential vector by using an encoder and then generates new data by reconstructing the potential vector through a decoder. Since the VAE can establish the relationship between the latent vector and the real data through the decoder, but has limited ability to analyze complex data, the generated image is blurred.

Disclosure of Invention

The invention provides an emotion electroencephalogram data enhancement model based on a generating type confrontation network and a sample expansion method aiming at the problem of lack of electroencephalogram data.

In order to achieve the purpose, the invention adopts the following technical scheme:

the invention provides an emotion electroencephalogram data enhancement model based on a generative confrontation network, wherein the input of the model is an emotion electroencephalogram topological graph, a VAE maps the emotion electroencephalogram topological graph to a potential space, a potential vector is input into a generator of the generative confrontation network, the spatial distribution of the emotion electroencephalogram topological graph is learned through the potential vector, two identifiers and one generator are arranged in the generative confrontation network corresponding to the model, the two identifiers are formed by a neural network, and the two identifiers have the same structure but do not share parameters; one of the discriminators awards a high score to the sample conforming to the true data distribution; another evaluator awarding a high score reward to the generator generated sample; the game between the generator and the two discriminators enables the generator to generate a distribution of synthetic samples that approaches the distribution of the real data infinitely.

In another aspect, the present invention provides a method for enhancing an extended sample based on data, including:

step 1, preprocessing emotion electroencephalogram data;

step 2, emotion electroencephalogram topological graph feature extraction based on differential entropy is carried out on the preprocessed emotion electroencephalogram data;

step 3, learning and training the emotion electroencephalogram data enhancement model through the emotion electroencephalogram topological graph generated in the step 2 to obtain emotion electroencephalogram data enhancement models of various emotions;

and 4, mixing the synthetic samples of corresponding emotion generated by the emotion electroencephalogram data enhancement models of various emotions into real data to complete emotion electroencephalogram data sample expansion.

Further, the step 1 comprises:

extracting emotion electroencephalogram signals presented by corresponding videos, replacing bad leads with average of surrounding lead signals, removing eye charge, filtering by using a band-pass filter, and performing re-reference and baseline correction processing.

Further, the step 2 comprises:

dividing the preprocessed emotion electroencephalogram signals into frequency bands, extracting differential entropy characteristics from each frequency band, and mapping the differential entropy into each electrode by using a Clough-Tocher scheme interpolation method to form an emotion electroencephalogram topological graph.

Further, after the step 4, the method further comprises:

and training a classification model based on the expanded emotion electroencephalogram data sample.

Compared with the prior art, the invention has the following beneficial effects:

the emotion electroencephalogram data enhancement model provided by the invention fully exerts respective advantages by combining a variational self-encoder (VAE) and a generative countermeasure network (GAN). The VAE may learn the spatial distribution of the topology map through the latent vectors. In addition, the dual discriminator of GAN can prevent the generated synthesis sample data from being distributed too intensively to cause mode collapse. Combining these advantages, VAE-D2GAN is easier to train and the generated samples are more diverse.

The invention trains the emotion electroencephalogram data enhancement model according to each type of emotion, so that the characteristics of the same emotion can be well learned. Samples of corresponding emotions generated by the emotion electroencephalogram data enhancement model of each emotion are mixed into real data to fully train the classification model, and classification performance is improved.

In SEED (three positive, neutral and negative classifications) and SEED-IV (four classifications of happy, neutral, angry and sad), classification accuracy rates using the method of the present invention were 92.5% and 82.3%, respectively.

Drawings

FIG. 1 is a schematic structural diagram of an emotion electroencephalogram data enhancement model based on a generative confrontation network according to an embodiment of the present invention;

FIG. 2 is an emotional electroencephalogram topological graph obtained in a method for expanding samples based on data enhancement according to an embodiment of the present invention;

FIG. 3 is a flowchart of a method for enhancing augmented samples based on data according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a Deep Neural Network (DNN);

FIG. 5 is the influence of the emotion electroencephalogram data enhancement model of the present invention on the classification accuracy of different classification models;

FIG. 6 is a two-dimensional visualization of a real sample under test and a composite sample generated from different data enhancement models in the SEED data set.

Detailed Description

The invention is further illustrated by the following examples in conjunction with the accompanying drawings:

as shown in fig. 1, in one aspect, the present invention provides an emotion electroencephalogram data enhancement model (for convenience of description, abbreviated as VAE-D2GAN) based on a generative confrontation network, which can fully utilize respective advantages of a variational self-encoder (VAE) and the generative confrontation network (GAN) to complement respective disadvantages. The encoder of the VAE maps the actual data to a potential space with specific distribution, and inputs potential vectors with specific distribution into the generator, so that the generator can more accurately and effectively learn the distribution of the actual data, and through the interaction of the two discriminators, the synthesized sample has diversity and avoids mode collapse; the input of the model is an emotion electroencephalogram topological graph, the spatial distribution of the emotion electroencephalogram topological graph is learned through potential vectors, two discriminators and a generator are arranged in a generating type confrontation network corresponding to the model, the two discriminators are formed by a neural network, and the discriminators are identical in structure but do not share parameters; one of the discriminators awards a high score to the sample conforming to the true data distribution; another evaluator awarding a high score reward to the generator generated sample; through the game between the generator and the two discriminators, the distribution of the synthetic samples (synthetic emotion electroencephalogram topological graph) generated by the generator is infinitely close to the distribution of real data, so that the mode collapse caused by too concentrated distribution of the generated synthetic samples is effectively avoided.

VAE-D2GAN at encoder E, generator G and two discriminators D₁,D₂And (3) carrying out the optimized game of 4 players, wherein the formula is as follows:

wherein:

x here_realRefers to the actual data, z refers to the potential vectors synthesized by the encoder E, z_pIs a randomly generated obeying Gaussian distribution z_pN (0, I). Alpha and beta are hyper-parameters with a value range of 0<α,β≤1。

step 1, preprocessing emotion electroencephalogram data;

specifically, the step 1 includes:

firstly, emotion electroencephalogram signals corresponding to video presentation are extracted, and channels with data difference are averagely replaced by surrounding channel signals. Then removing ocular artifacts by using a Fastic algorithm, then performing band-pass filtering of 0.1-64hz by using a band-pass filter to filter out high-frequency interference in the electroencephalogram signals, and finally performing re-reference and baseline correction.

specifically, the step 2 includes:

firstly, dividing preprocessed emotion electroencephalogram data electroencephalogram signals into 5 frequency bands, namely theta (1-3Hz), theta (4-7Hz), alpha (8-13Hz), beta (14-30Hz) and gamma (31-50Hz), and then extracting differential entropy characteristics in 5 frequency band ranges. Finally, we map the differential entropy features of 5 bands into each channel using the method of Clough-Tocher scheme interpolation, to generate a 32 by 5 emotional electroencephalogram topology, as shown in FIG. 2, where the length and width of the topology are both 32 pixels and the channel is 5.

Step 3, learning and training the emotion electroencephalogram data enhancement model through the emotion electroencephalogram topological graph generated in the step 2 to obtain emotion electroencephalogram data enhancement models of various emotions, as shown in fig. 3;

Further, after the step 4, the method further comprises:

training of a classification model is carried out based on the expanded emotion electroencephalogram data samples, and classification performance is improved.

To verify the effect of the present invention, the following experiment was performed:

the data enhancement model of the emotion electroencephalogram is experimentally verified on two public data sets (SEED and SEED-IV). The method is based on the division of the training set and the test set of the SEED and SEED-IV data sets by predecessors. In the SEED data set, we use the data of the first 9 sessions as a training set and the data of the last 6 sessions as a test set, wherein the last 6 sessions contain three categories of positive, neutral and negative emotions of two sessions. Similarly, on the SEED-IV dataset, we use the first 16 sessions as training data and the last 8 sessions as test set, where the last 8 sessions contain 2 sessions each for happy, neutral, sad and fear.

First, we generated synthetic samples of various emotions using emotion electroencephalogram data enhancement models of various emotions, the SEED data set had three types of emotions (positive, neutral, and negative), the SEED-IV data set had four types of emotions (happy, neutral, sad, and angry), and 8000 samples were generated for each type of emotion. Thus, on SEED data, 24000 samples are generated for each subject using the generative model; on SEED-IV data, 32000 samples were generated for each subject using the generative model.

Meanwhile, the invention classifies the characteristic topological graph and constructs a classification model DNN based on a neural network, which is composed of two convolutional layers, two maximum pooling layers and two full-connection layers as shown in FIG. 4. Experiments show that the simple image class of the feature topological graph data has better classification performance by using a shallow network.

(1) The analysis of the results of VAE-D2GAN on two data sets proposed by the invention

We compared classical data enhancement models such as VAE, WGAN, DCGAN, etc. The results of the classification of the VAE-D2GAN on the SEED data set and the SEED-IV data set using the DNN classification model compared to other models are shown in table 1 and table 2, respectively. Generating samples according to different models in the SEED data set, and increasing the number of different generated samples in the training set; the average accuracy and standard deviation of classification are obtained by utilizing a DNN classification model; 0 means no generation sample is added to the training set. Generating samples according to different models in a SEED-IV data set, and increasing the number of different generated samples in a training set; the average accuracy and standard deviation of classification are obtained by utilizing a DNN classification model; 0 means no generation sample is added to the training set. The bold is the highest classification accuracy and the smallest variance in each data enhancement model. As can be seen from table 1, in the SEED dataset, the data enhancement performance of both VAE and WGAN models is not improved, but rather is reduced. The recognition rates for DCGAN, D2GAN and VAE-D2GAN were 0.6%, 0.6% and 1.5% higher than the recognition rate without data enhancement, respectively. Wherein, the accuracy of the VAE-D2GAN data enhancement method is the highest and is 92.5 percent. As can be seen from Table 2, in the SEED-IV dataset, the two data-enhanced models VAE and WGAN did not improve the recognition accuracy, while the recognition accuracy of DCGAN, D2GAN and VAE-D2GAN improved by 0.3%, 1.2% and 3.5%, respectively. Wherein, the accuracy of the VAE-D2GAN data enhancement method is the highest and is 82.3 percent. In addition, experimental results on both data sets show that the stability of the identification can be improved by using data enhancement. The precision improvement of the SEED-IV data set is higher than that of the SEED data set. The reason for this is that the number of samples of SEED-IV is much smaller than the SEED data set. Therefore, the effect of data expansion by using small sample data is better.

TABLE 1 results of different data enhancement models in SEED data set Classification Using DNN Classification model

TABLE 2 results of different data enhancement models in SEED-IV dataset Classification Using DNN Classification model

(2) The VAE-D2GAN provided by the invention is used for analyzing results of different classification models

We use data-enhanced models of different classifiers (including various deep networks and traditional machine learning) to analyze the impact of recognition accuracy. In deep networks, we have selected classical VGG16, ResNet18 and AlexNet; in traditional machine learning, we have chosen the classical Support Vector Machine (SVM). Meanwhile, our data was used to enhance the model VAE-D2GAN, and the results are shown in fig. 5. Without data enhancement, the classification results obtained by different classifiers are different. In the SEED dataset, the classification accuracy of Vgg16, ResNet18, AlexNet, SVM and DNN classifiers was 83.64%, 84.85%, 84.89%, 76.24% and 90.97%, respectively. In the SEED-IV dataset, the classification accuracy of Vgg16, ResNet18, AlexNet, SVM and DNN classifiers was 68.67%, 64.23%, 72.62%, 63.13% and 78.83%, respectively. In general, deep web learning is superior to traditional machine learning. The classification accuracy of DNN is highest from the results of SEED dataset (fig. 5 (a)) and SEED-IV dataset (fig. 5 (b)). In addition, in emotion recognition based on electroencephalogram, data enhancement has little influence on traditional machine learning. Furthermore, in deep networks, augmentation with data is more effective, especially on small data sets. Compared to the more complex deep networks Vgg16, ResNet18 and AlexNet, the DNN network is simple in structure but performs best in identifying topological images. Therefore, the more complex the network structure, the better the classification performance is not necessarily. For image classification with simple topological structure, a simple network can obtain good classification performance. However, for more complex classification tasks, such as 4 classes in the SEED-IV dataset, as shown in fig. 5 (b), the classification performance of ResNet18 and AlexNet is very close to DNN. Thus, the more complex the classification task, the more complex the network may have a better classification effect.

(3) Evaluation analysis of the Performance of the VAE-D2GAN proposed by the present invention Using different algorithms

The data enhancement model VAE-D2GAN of the present invention was evaluated using three algorithms currently used as an evaluation algorithm, namely Index Score (IS), Freehet Index Distance (FID) and Maximum Mean Difference (MMD). The higher the IS value, the higher the quality of the samples we consider the model to produce, but the diversity cannot be assessed, i.e. the IS value IS also high when a pattern collapse occurs. The smaller the FID value, the higher the quality and diversity of the samples generated by the model, and the more sensitive the FID IS to pattern collapse compared to IS. The MMD calculates the distance between the two data distributions, and the smaller the MMD value, the more similar the two data distributions are, and the quality of the model generated sample can be reflected. The model proposed by the present invention was evaluated with other models, and the results of the evaluation are shown in table 3, with bolded representation yielding the best results. Since the IS and FID values in WGAN are higher than VAE-D2GAN, WGAN has a mode collapse compared to VAE-D2 GAN. VAE produced poor sample quality with higher MMD and FID values. The samples of VAE-D2GAN were of high quality and good diversity, as FID and MMD were the lowest. To better illustrate the advantages of VAE-D2GAN, we map real samples and samples generated by different models in different iterations to a two-dimensional visualization by t-SNE, as shown in FIG. 6. As can be seen from the figure, VAE-D2GAN is effective in learning different distributed emotional characteristics.

TABLE 3 VAE-D2GAN Performance evaluation results

(4) Analysis of the impact of different numbers of training samples on the performance of the VAE-D2GAN proposed by the present invention

In the above research analysis, we found that the model proposed by the present invention is more friendly to data of small samples. For this reason, we performed classification performance experiments with different sample numbers as training sets on the SEED data set.

Experiment 1: we select the first 3 sessions (1 session each for positive, neutral and negative) of each test as the training set, and the last 6 sessions as the test set.

Experiment 2: we select the first 6 sessions (2 sessions each for positive, neutral, and negative) of each test as the training set, and the last 6 sessions as the test set.

Experiment 3: we chose the first 9 sessions (positive, neutral and negative 3 sessions each) of each test as the training set, and the last 6 sessions as the test set.

Results as shown in table 4, the data for experiment 1, experiment 2 and experiment 3 enhanced the classification results to 79.46%, 83.76% and 92.46%, respectively. Compared with the enhancement without data, the improvement is respectively 11.29%, 8.3% and 1.49%. The number of training samples of experiments 1 to 3 is gradually increased, and the classification performance is gradually improved. The data enhancement model VAE-D2GAN provided by the invention can exert excellent performance in a small sample data set.

TABLE 4 Effect of different numbers of training samples on VAE-D2GAN Performance

In conclusion, the emotion electroencephalogram data enhancement model provided by the invention fully exerts respective advantages by combining a variational self-encoder (VAE) and a generative countermeasure network (GAN). The VAE may learn the spatial distribution of the topology map through the latent vectors. In addition, the dual discriminator of GAN can prevent the generated synthesis sample data from being distributed too intensively to cause mode collapse. Combining these advantages, VAE-D2GAN is easier to train and the generated samples are more diverse.

The above shows only the preferred embodiments of the present invention, and it should be noted that it is obvious to those skilled in the art that various modifications and improvements can be made without departing from the principle of the present invention, and these modifications and improvements should also be considered as the protection scope of the present invention.

Claims

1. The emotion electroencephalogram data enhancement model based on the generative confrontation network is characterized in that the input of the model is an emotion electroencephalogram topological graph, the VAE maps the emotion electroencephalogram topological graph to a potential space, a potential vector is input into a generator of the generative confrontation network, the spatial distribution of the emotion electroencephalogram topological graph is learned through the potential vector, two discriminators and one generator are arranged in the generative confrontation network corresponding to the model, the two discriminators are formed by a neural network, and the two discriminators are identical in structure but do not share parameters; one of the discriminators awards a high score to the sample conforming to the true data distribution; another evaluator awarding a high score reward to the generator generated sample; the game between the generator and the two discriminators enables the generator to generate a distribution of synthetic samples that approaches the distribution of the real data infinitely.

2. The method for augmenting the sample based on data enhancement of the emotion electroencephalogram data enhancement model based on the generative confrontation network as claimed in claim 1, comprising:

step 1, preprocessing emotion electroencephalogram data;

3. The method of claim 2, wherein the step 1 comprises:

4. The method of claim 2, wherein the step 2 comprises:

5. The method for enhancing augmented samples based on data of claim 2, further comprising, after the step 4: