CN112906539B - Object identification method based on EEG data - Google Patents
Object identification method based on EEG data Download PDFInfo
- Publication number
- CN112906539B CN112906539B CN202110172061.6A CN202110172061A CN112906539B CN 112906539 B CN112906539 B CN 112906539B CN 202110172061 A CN202110172061 A CN 202110172061A CN 112906539 B CN112906539 B CN 112906539B
- Authority
- CN
- China
- Prior art keywords
- data
- eeg
- network model
- resblock
- object recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 22
- 230000004913 activation Effects 0.000 claims abstract description 24
- 238000012549 training Methods 0.000 claims abstract description 13
- 230000008569 process Effects 0.000 claims description 7
- 238000005516 engineering process Methods 0.000 claims description 4
- 238000004140 cleaning Methods 0.000 claims description 3
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 230000002349 favourable effect Effects 0.000 claims description 2
- 238000000605 extraction Methods 0.000 abstract description 2
- 238000013527 convolutional neural network Methods 0.000 abstract 1
- 230000006870 function Effects 0.000 description 23
- 210000004556 brain Anatomy 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 4
- 230000007177 brain activity Effects 0.000 description 3
- 238000012360 testing method Methods 0.000 description 3
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000008447 perception Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 241001465754 Metazoa Species 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000013140 knowledge distillation Methods 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 230000000638 stimulation Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000003612 virological effect Effects 0.000 description 1
- 239000011800 void material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
Abstract
The invention discloses an object identification method based on EEG data. The invention firstly uses data enhancement and extraction technology to expand the capacity of the data. Then randomly dividing the data into five equal parts, performing 5-fold training, then using ResBlock as a basic structure to construct a brand new two-dimensional convolutional neural network, removing the first three layers of the network by hole convolution to replace common convolution, using PReLU as an activation function of the network, and using Focalloss as a loss function of a model. Model training was performed using the EEG data object recognition dataset released by the active laboratory in 2017. The data enhancement of the invention can make full use of the data of the small data set. In addition, a deeper network model is adopted, and fewer network parameters are adopted, so that the characteristic which can be used for identification in the EEG data can be learned as much as possible in spite of the fact that the quantity of the EEG data in the data set is small, and the object identification task can be efficiently and accurately realized.
Description
Technical Field
The invention belongs to the technical field of EEG signal recognition, and particularly relates to a scene recognition system based on EEG signals.
Background
Currently, humans are much more capable of classifying scenes than computers. Although the deep learning algorithm in recent years can achieve a very good accuracy of scene recognition, the deep learning algorithm still cannot exceed the accuracy of human recognition. In comparison with the computer vision method, which directly extracts distinguishable features of an image for recognition, the human brain visual recognition process also comprises a perception process and a recognition theory, for example, the color, the shape and the like of an object are stimulated by the human brain, and the human brain cortex reacts to the stimulation. Some studies in the neuroscience field indicate that human brain activity has a specific pattern of brain activity for a specific class of objects. The recognition of the human brain for the same kind of objects is based on the same pattern and the recognition speed is very fast, with a time scale of around a few milliseconds. Therefore, a picture of the object can be presented to the subject and then the EEG data of the subject recorded, and features that can be used for identification can be extracted from the EEG data well with the very high resolution of the time dimension of the EEG data. If the human perception and recognition process of an object can be refined and applied in computer vision, the computer can automatically simulate the human object recognition process.
The decoding of vision-related human brain activities has great significance, not only can provide a brand-new angle for scientific researchers to study the vision-cognition process, but also can enhance the performance of a human-computer interaction system. The object recognition by using EEG data of the human brain can not only change the traditional object recognition method, but also provide a novel method for labeling unlabeled image data. The use of patterns in the human visual system to do computer vision tasks has been a difficult direction, however, the advent of deep learning has enabled researchers to study EEG data and neuroscience more deeply. To the best of our knowledge, there are some studies to directly use EEG signals for object recognition tasks. For example, kaboor a et al propose a method for object recognition using EEG data, which classifies images into three categories (animal, face, inanimate object). Rouby et al use EEG data to do object recognition tasks. They acquired a dataset containing 400 images and corresponding EEG data using a 256-lead EEG data acquisition system. Their average correct rate was 82.70%. Viral et al propose an image annotation system based on EEG data. They utilized 2500 images as the training set and the F1 score reached 0.88. These studies have shown that object recognition tasks using EEG data are feasible. In 2017, the active laboratory has released one of the largest current object recognition EEG datasets. And they constructed a series of EEG recognition models based on recurrent neural networks. Their best recognition accuracy was 82.9%. Although these methods have achieved very good results at present, there is still a great improvement in recognition accuracy, mainly because the network still needs to be improved, and secondly, the data of the image is less, which makes it difficult for the deep learning model to learn the distinguishable features.
Disclosure of Invention
The invention aims to provide an object identification method based on EEG data. Firstly, the data is expanded by utilizing the data enhancement and extraction technology. Then, a brand new two-dimensional convolution neural network is constructed by taking ResBlock as a basic structure, the first three layers of the network replace common convolution by hole convolution, PReLU is used as an activation function of the network, and Focalloss is used as a loss function of the model. Model training was performed using the EEG data object recognition dataset released by the active laboratory in 2017.
The invention provides an object identification method based on EEG. In the data preprocessing stage, the data enhancement technology is utilized to perform random 0 setting processing and random data overturning operation on the original EEG data, so that the purpose of expanding the original data set is achieved. The data was then randomly split into five equal parts and a 5-fold training was performed. Data enhancement may allow the data of a small data set to be fully utilized. In addition, a deeper network model is adopted, and fewer network parameters are adopted, so that the characteristic which can be used for identification in the EEG data can be learned as much as possible in spite of the fact that the quantity of the EEG data in the data set is small, and the object identification task can be efficiently and accurately realized.
In order to achieve the above purpose, the present invention adopts the following technical scheme:
the Python language is preferred as the basis. On the data enhancement, the numpy library is used for carrying out random 0 setting operation and random overturning operation on the original image data, and then a data set after the data enhancement is output. And on this basis, the EEG data was randomly divided into 5 equal parts for 5-fold training. And then, using an open source deep learning framework PyTorch to realize a network model and complete the construction of the model. And 5-fold training is carried out on the data set, each fold is selected to obtain the optimal model on the verification set, and model fusion is carried out on the five selected models to obtain a final result.
An EEG-based object recognition method comprising the steps of:
and step 1, acquiring an EEG object identification data set, and cleaning the acquired original image data set.
The EEG object recognition data set includes EEG data collected from the subjects and a tag for each subject to view the object during the collection of data.
And 2, performing enhancement processing on the cleaned EEG object identification data set by using a data enhancement technology, and increasing the number of samples.
And 3, randomly dividing the EEG object identification data set processed in the step 2 into 5 data sets with the same quantity.
Step 4, constructing an identification network model, utilizing ResBlock as a basic module, adopting hole convolution to the ResBlock layer of the first three layers of the identification network model, adopting a PReLU activation function to the activation function in the network, adopting the following formula of the PReLU activation function,
x i is the input of the activation function in the ith channel, the parameter a can be learned i Is responsible for controlling the slope of the negative half-axis of the activation function.
The recognition network model adopts FocalLoss as a loss function, the FocalLoss loss function has the following formula,
Loss＝-α(1-y′) γ logy′
in the formula, alpha and gamma are two super parameters which are respectively used for solving the problem of data unbalance and improving the classification performance of the difficult-to-separate samples. y' is the output of the recognition network model.
And 5, performing five-fold training on the identification network model according to the 5 data sets obtained in the step 3, and fusing the optimal model obtained by each-fold training.
Further, the step 2 specifically includes the following steps:
step 2.1, randomly setting data to 0: for EEG data in the EEG object identification dataset, 128 channels of each EEG data are randomly taken from 1 to 10 channels, randomly taken from 5 to 15 data points in the time dimension, and these data points are all set to 0.
Step 2.2, random data overturn: and (3) performing a time dimension or channel dimension overturning operation on the EEG data.
The step 4 specifically comprises the following steps:
and 4.1, an EEG-based object recognition network model integrally adopts ResBlock as a basic structure, the ResBlock layers of the first three layers of the recognition network model adopt hole convolution, EEG data are precoded through the hole convolution, and the hole convolution hole ratios of the three ResBlock built-in holes are respectively set to be 1, 10 and 21.
The cavity convolution can enable the object recognition network model to obtain a larger receptive field in the initial stage, and is favorable for extracting information between EEG data cross channels and large time scales.
Step 4.2, a module obtained by superposing MaxPool (maximum pooling) and ResBlock is used as a basic module of the object recognition network model; the three layers adopt the ResBlock layer of the cavity convolution and then are connected with 5 basic modules with the same structure to form an object recognition network model;
the step size of the MaxPool layer in the first 2 basic modules is set to 2 in the EEG data time dimension and 1 in the EEG data channel dimension, so that the feature map is close to the dimension of the channel dimension in the time dimension; the MaxPool operation in the last 3 basic blocks is set to 2 in step size for both channels, i.e. the downsampling operation is performed jointly in both dimensions. All ResBlock in the object recognition network model, and the PReLU activation function is adopted as the activation function. Finally, focalLoss is used as a loss function.
The object recognition network model is firstly downsampled only in the time dimension because EEG data generally has higher resolution in the time dimension, and firstly downsampled in the time dimension enables the channel dimension and the time dimension to be close, and the feature map is close to a square, so that the following convolution can better extract the features. The PReLU activation function is employed so that each layer of the network can learn the best activation function for that layer using back propagation. The adoption of Focalloss enables learning of the network to bias towards learning of hard-to-separate samples.
Compared with the prior art, the invention has the following obvious advantages:
deep learning models typically require a large number of data sets to train to find the best model, whereas EEG data sets are smaller in size than natural images due to the difficulty and cost of acquisition. The model utilizes FocalLoss to strengthen the generalization capability of the model, so that the learning of the model can pay more attention to the learning of difficult-to-separate samples. Meanwhile, the front three layers of the network adopt ResBlock with different void ratios, so that the network can acquire a larger receptive field in the precoding process, and information on a larger time scale is fused. The object recognition network model adopts the steps of firstly downsampling the time scale, so that after the resolution of the feature map on the time scale and the channel scale are close, downsampling operation is carried out on the two scales at the same time.
Drawings
Figure 1 is a flow chart of a method according to the invention;
FIG. 2 is a general block diagram of an EEG object recognition network according to the present invention;
fig. 3 is a network configuration diagram of the present invention.
Fig. 4 is a ResBlock structure of the present invention.
Detailed Description
The present invention will be described in further detail below with reference to the accompanying drawings in conjunction with the detailed description.
The hardware equipment used in the invention is one PC machine and one 1080ti display card;
as shown in fig. 1, the invention provides an EEG object recognition method based on deep learning, which specifically comprises the following steps:
step 1, acquiring an EEG-based object identification data set in the related field, and cleaning (e.g. deleting dirty data) the acquired original image data set.
The object identification data set based on the EEG comprises EEG data and a kind label of the object being watched by the test when the EEG data is acquired;
and 2, enhancing the cleaned EEG data set by using an image enhancement technology, increasing the number of samples and enriching the data content.
And 3, randomly dividing the image data set processed in the step 2 into 5 data sets with the same quantity.
And 4, building an object recognition model, adding hole convolution in a network by using ResBlock as a basic structure, adopting a PReLU activation function as an activation function, and adopting FocalLoss as a loss function.
As shown in fig. 2, is an overall block diagram of the entire EEG-based object recognition model.
As shown in fig. 3, a detailed model structure diagram of the present invention is shown.
Step 4.1, constructing a model using ResBlock as a basic module, as shown in FIG. 4, the model is a 2DResBlock structure of the invention.
Step 4.2, in the first three ResBlock layers of the model, the conventional convolution is replaced by hole convolution.
Step 4.3, changing the step size of the first two largest pooling layers of the model to 2 on the time scale of the EEG data and 1 on the channel scale of the EEG data.
Step 4.4, adding PReLU activation function into the model, wherein the PReLU activation function has the following formula,
x i is the input of the activation function in the ith channel, the parameter a can be learned i Is responsible for controlling the slope of the negative half-axis of the activation function.
And 4.5, using FocalLoss as a loss function of the model. The formula for the FocalLoss function is as follows,
Loss＝-α(1-y′) γ logy′
in the formula, alpha and gamma are two super parameters which are respectively used for solving the problem of data unbalance and improving the classification performance of the difficult-to-separate samples.
And 5, performing five-fold training on the object recognition model according to the 5 data sets obtained in the step 3, and fusing the optimal model obtained by each-fold training. The specific model fusion method is to normalize the output of each model and then calculate the average value.
Through experimental tests, the maximum accuracy of the object identification model on a test set reaches 93.2%, the accuracy of the model is 82.9% raised by a personal laboratory in 2017, the accuracy of the model is improved by 10.3% compared with the model, the accuracy of the knowledge distillation model is 89.6% raised by praay Mukherjee et al in 2019, and the accuracy of the model is improved by 3.6% compared with the model.
Claims (2)
1. An EEG data based object recognition method comprising the steps of:
step 1, acquiring an EEG object identification data set, and cleaning the acquired original image data set;
the EEG object identification data set comprises EEG data acquired from the tested subjects and labels of the objects watched by each tested subject in the process of acquiring the data;
step 2, enhancing the cleaned EEG object identification data set by utilizing a data enhancement technology, and increasing the number of samples;
step 3, randomly dividing the EEG object identification data set processed in the step 2 into 5 data sets with the same quantity;
step 4, constructing an identification network model, utilizing ResBlock as a basic module, adopting hole convolution to the ResBlock layer of the first three layers of the identification network model, adopting a PReLU activation function to the activation function in the network, adopting the following formula of the PReLU activation function,
x i is the input of the activation function in the ith channel, the parameter a can be learned i The slope of the negative half shaft of the activation function is controlled;
the recognition network model adopts FocalLoss as a loss function, the FocalLoss loss function has the following formula,
Loss＝-α(1-y′) γ log y′
in the formula, alpha and gamma are two super parameters which are respectively used for solving the problem of unbalance of data and improving the classification performance of a difficult-to-separate sample; y' is the output of the recognition network model;
step 4.1, an EEG-based object recognition network model integrally adopts ResBlock as a basic structure, a ResBlock layer of the first three layers of the recognition network model adopts hole convolution, EEG data is precoded through the hole convolution, and hole convolution hole ratios of three ResBlock built-in holes are respectively set to be 1, 10 and 21;
the cavity convolution can enable the object recognition network model to obtain a larger receptive field at the initial stage, and is favorable for extracting information between the cross channels and the large time scales of EEG data;
step 4.2, a module obtained by superposing MaxPool and ResBlock is used as a basic module of the object recognition network model; the three layers adopt the ResBlock layer of the cavity convolution and then are connected with 5 basic modules with the same structure to form an object recognition network model;
the step size of the MaxPool layer in the first 2 basic modules is set to 2 in the EEG data time dimension and 1 in the EEG data channel dimension, so that the feature map is close to the dimension of the channel dimension in the time dimension; the MaxPool operation in the last 3 basic modules is set to 2 in the step length of the two channels, namely downsampling operation is carried out in two dimensions together; all ResBlock in the object recognition network model, and the PReLU activation function is adopted as the activation function; finally, focalLoss is adopted as a loss function;
and 5, performing five-fold training on the identification network model according to the 5 data sets obtained in the step 3, and fusing the optimal model obtained by each-fold training.
2. An EEG data based object recognition method according to claim 1, wherein step 2 comprises the steps of:
step 2.1, randomly setting data to 0: for EEG data in the EEG object recognition dataset, randomly taking 1 to 10 channels for 128 channels of each EEG data, randomly taking 5 to 15 data points long in the time dimension, and setting all the data points to 0;
step 2.2, random data overturn: and (3) performing a time dimension or channel dimension overturning operation on the EEG data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172061.6A CN112906539B (en) | 2021-02-08 | 2021-02-08 | Object identification method based on EEG data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110172061.6A CN112906539B (en) | 2021-02-08 | 2021-02-08 | Object identification method based on EEG data |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112906539A CN112906539A (en) | 2021-06-04 |
| CN112906539B true CN112906539B (en) | 2024-04-05 |
Family
ID=76122752
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110172061.6A Active CN112906539B (en) | 2021-02-08 | 2021-02-08 | Object identification method based on EEG data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112906539B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368950A (en) * | 2009-03-16 | 2012-03-07 | 纽罗斯凯公司 | Eeg control of devices using sensory evoked potentials |
| CN109288518A (en) * | 2018-08-16 | 2019-02-01 | 北京薏然阳光电子技术有限公司 | Brain cognition neural Function Appraising system and method based on EEG and ERPs |
| CN111584030A (en) * | 2020-04-30 | 2020-08-25 | 天津大学 | Idea control intelligent rehabilitation system based on deep learning and complex network and application |
| CN111631848A (en) * | 2020-05-31 | 2020-09-08 | 天津大学 | Mind control artificial limb system based on brain-computer hybrid intelligence |
-
2021
- 2021-02-08 CN CN202110172061.6A patent/CN112906539B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102368950A (en) * | 2009-03-16 | 2012-03-07 | 纽罗斯凯公司 | Eeg control of devices using sensory evoked potentials |
| CN109288518A (en) * | 2018-08-16 | 2019-02-01 | 北京薏然阳光电子技术有限公司 | Brain cognition neural Function Appraising system and method based on EEG and ERPs |
| CN111584030A (en) * | 2020-04-30 | 2020-08-25 | 天津大学 | Idea control intelligent rehabilitation system based on deep learning and complex network and application |
| CN111631848A (en) * | 2020-05-31 | 2020-09-08 | 天津大学 | Mind control artificial limb system based on brain-computer hybrid intelligence |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112906539A (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109919031B (en) | Human behavior recognition method based on deep neural network | |
| CN110069958B (en) | Electroencephalogram signal rapid identification method of dense deep convolutional neural network | |
| CN110889672B (en) | Student card punching and class taking state detection system based on deep learning | |
| CN110353673B (en) | Electroencephalogram channel selection method based on standard mutual information | |
| CN110991406B (en) | RSVP electroencephalogram characteristic-based small target detection method and system | |
| CN105160310A (en) | 3D (three-dimensional) convolutional neural network based human body behavior recognition method | |
| Zhang et al. | An novel end-to-end network for automatic student engagement recognition | |
| CN114224342A (en) | Multi-channel electroencephalogram emotion recognition method based on space-time fusion feature network | |
| CN111714118A (en) | Brain cognition model fusion method based on ensemble learning | |
| CN114581823A (en) | Virtual reality video emotion recognition method and system based on time sequence characteristics | |
| CN114332911A (en) | Head posture detection method and device and computer equipment | |
| CN110889335B (en) | Human skeleton double interaction behavior identification method based on multichannel space-time fusion network | |
| CN113486700A (en) | Facial expression analysis method based on attention mechanism in teaching scene | |
| Lyu et al. | Spontaneous facial expression database of learners’ academic emotions in online learning with hand occlusion | |
| Tang et al. | Automatic facial expression analysis of students in teaching environments | |
| CN113705301A (en) | Image processing method and device | |
| US20230290118A1 (en) | Automatic classification method and system of teaching videos based on different presentation forms | |
| CN112906539B (en) | Object identification method based on EEG data | |
| CN111368663A (en) | Method, device, medium and equipment for recognizing static facial expressions in natural scene | |
| Tian et al. | A multitask convolutional neural network for artwork appreciation | |
| CN115719497A (en) | Student concentration degree identification method and system | |
| CN112274154B (en) | Cross-subject fatigue driving classification method based on electroencephalogram sample weight adjustment | |
| Liu et al. | Spatial-temporal convolutional attention for mapping functional brain networks | |
| CN114998252A (en) | Image quality evaluation method based on electroencephalogram signals and memory characteristics | |
| CN114187632A (en) | Facial expression recognition method and device based on graph convolution neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |