CN111012336A

CN111012336A - Parallel convolutional network motor imagery electroencephalogram classification method based on spatio-temporal feature fusion

Info

Publication number: CN111012336A
Application number: CN201911241265.XA
Authority: CN
Inventors: 唐贤伦; 孔德松; 邹密; 刘行谋; 马伟昌; 李伟; 王婷; 彭德光; 李锐
Original assignee: Chongqing University of Post and Telecommunications
Current assignee: Chongqing University of Post and Telecommunications
Priority date: 2019-12-06
Filing date: 2019-12-06
Publication date: 2020-04-17
Anticipated expiration: 2039-12-06
Also published as: CN111012336B

Abstract

The invention requests to protect a parallel convolution neural network motor imagery electroencephalogram identification method with fused space-time characteristics. A new depth network model-parallel convolution neural network is provided to extract the space-time characteristics of the motor imagery electroencephalogram signal by taking the motor imagery electroencephalogram signal as a research object. Different from the traditional electroencephalogram classification algorithm which usually discards electroencephalogram spatial feature information, the 2D electroencephalogram feature map is generated by extracting Theta waves (4-8Hz), alpha waves (8-12Hz) and beta waves (12-36Hz) through fast Fourier transform. And training the electroencephalogram characteristic diagram based on the multiple convolutional neural networks, and extracting spatial characteristics. In addition, a time convolution neural network is used for parallel training, and time sequence characteristics are extracted. And finally, fusing and classifying the spatial features and the time sequence features based on Softmax. Experimental results show that the parallel convolutional neural network has good identification precision and is superior to other latest classification algorithms.

Description

Parallel convolutional network motor imagery electroencephalogram classification method based on spatio-temporal feature fusion

Technical Field

The invention belongs to the field of motor imagery electroencephalogram classification, and particularly relates to a parallel convolutional neural network motor imagery electroencephalogram identification method based on space-time feature fusion.

Background

The brain electricity is a comprehensive reflection of the physiological activity of the brain cells of the scalp, and contains a large amount of physiological and disease information. Brain-computer interaction system (BCI) based on EEG signal communication can replace brain nerve and muscle tissue transmission to be used as a signal transmission channel, and therefore interaction between the brain and bionic machinery is achieved. BCI has been receiving a great deal of attention from researchers and scientists as an extension of human-computer interaction. Based on motor imagery electroencephalogram recognition, the method is a key node for BCI system interaction and interaction with the outside. The motor imagination is the subjective imagination performed by the human brain, such as the imagination of a left-hand handshake, the imagination of a right-hand handshake, the imagination of leg flexion and extension and the like. Through the analysis of the motor imagery brain electric signal, the intention of human brain motor imagery can be identified and output to a bionic system of BCI, thereby realizing brain-computer control. Therefore, the research on the motor imagery electroencephalogram signal processing can promote the exploration on the cerebral nerve cognition, the cerebral disease rehabilitation and the cerebral cortex signal analysis. The potential application prospects push EEG research to a high-speed stage, making it one of the most attractive disciplines.

In the BCI system, there are two important parts of feature extraction and feature classification. Common feature extraction methods include Fast Fourier Transform (FFT), Common Spatial Pattern (CSP), wavelet transform (DT), etc., which not only require a lot of manual data processing, but also are sensitive to noise and easily cause feature confusion. Common feature classification methods include artificial neural networks, support vector machines, and the like. Due to the complex generation mechanism of the EEG, the characteristic classification methods have the problems of shallow iteration level and insufficient characteristic extraction.

In recent years, deep learning has been highly successful in the fields of research such as image recognition, natural language processing, power load prediction, and pattern recognition. It is also applied to electroencephalogram data analysis due to its powerful capability of processing non-linear and high-dimensional data.

The EEG signal contains spatial information represented by the electrode positions as well as intrinsic temporal information. However, in the past, the electroencephalogram acquisition equipment only visualizes time sequence channel data, so that most researchers mainly research how to extract electroencephalogram characteristics under a time sequence. Therefore, a new network model is needed to extract and fuse the temporal features and the spatial features of the motor imagery EEG so as to improve the classification performance of the motor imagery EEG. The invention provides a parallel deep convolution neural network, which fully utilizes space-time information to enhance electroencephalogram feature extraction. The 2D electroencephalogram characteristic map is generated for effectively converting the time-space characteristic information of the electroencephalogram based on the fast Fourier transform. The convolutional neural network is sparse in connection, and the convolutional kernel parameters are shared, so that the storage capacity of the model is reduced, and the spatial characteristics of the pattern can be effectively extracted. The time convolution neural network is based on extended convolution modeling and conforms to the time sequence characteristics of EEG. The method combines the advantages of the two methods, constructs a parallel convolution network, extracts EEG spatial characteristics in convolution, extracts EEG time sequence characteristics in time convolution, and finally fuses EEG space-time characteristics in a characteristic splicing mode.

Disclosure of Invention

The present invention is directed to solving the above problems of the prior art. A parallel convolution network motor imagery electroencephalogram recognition method with fusion of space-time characteristics is provided. The technical scheme of the invention is as follows:

a parallel convolution network motor imagery electroencephalogram identification method based on spatiotemporal feature fusion comprises the following steps:

step 1: acquiring original EEG (electroencephalogram) channel data, and processing the original EEG channel data by adopting the steps including normalization and mean value removal;

step 2: segmenting the original EEG channel data preprocessed in the step 1 based on an overlapping cutting mode;

and step 3: performing wavelet transformation on each EEG channel obtained in the step 2 to obtain three frequency bands of Theta wave, alpha wave and beta wave;

and 4, step 4: solving the sum of squares of values of each frequency band of the Theta wave, the alpha wave and the beta wave obtained in the step 3;

and 5: interpolating the 2D channel distribution map based on an interpolation algorithm by using the sum of squares of each frequency band value obtained in the step 4 to generate a 2D electroencephalogram feature distribution map;

step 6: performing network training on the 2D feature distribution diagram generated in the step 5 by adopting a multiple convolutional neural network;

and 7: and simultaneously carrying out parallel training on the 2D feature maps in the step 5 based on the time convolution neural network.

And 8: and fusing and classifying the spatial features and the time sequence features based on Softmax.

Further, the step 1 adopts steps including normalization and de-averaging to process the raw EEG channel data, which specifically includes:

and (3) mean value removal: subtracting the amplitude from the average value in the data to make the average value of the electroencephalogram signal be 0;

normalization: the original data is linearly transformed such that the result maps between [0,1 ].

Further, the step 2 of segmenting the raw EEG channel data preprocessed in the step 1 based on the overlap-cut method specifically includes:

processing raw time-series channel EEG data based on overlap cutting to enable each frame of EEG data extracted in a motor imagery period to have partial overlap, and defining a formula

x_i＝x_i-1+f-o*f i≠0

x_i＝0 i＝0

Wherein x is a cutting starting point, i is a few samples, f is a frequency size, o x f is an overlapping size, and o is a cutting weight and ranges from 0 to 1;

according to the data matrix [ [ x ]₀,x₀+128],[x₁,x₁+128],[x₀,x₀+128],...,[x_n,x_n+128]]The 14 EEG channels are segmented and the data for each time window is arranged to ensure that the data time series is not corrupted.

Further, the step 3 performs wavelet transformation on each EEG channel obtained in the step 2 to obtain three frequency bands of Theta wave, alpha wave and beta wave, and specifically includes:

for each EEG channel data, after preprocessing, Fourier transform is performed on the data of each frame, and x is set to be equal to C^NIs an EEG signal of length N, the fast fourier transform is:

wherein N is 0,1, N-1 is different frequency, W_N＝e^-j(2π/N)。

The inverse fast fourier transform is:

the real-valued discrete Fourier of length N is obtained by a complex-valued fast Fourier transform of length N/2. Let x be an element of R^NThen the real-valued fast fourier transform:

after fast Fourier transform, we can obtain data matrix x containing each frequency band by belonging frequency bands to Theta wave, alpha wave and beta wave_nExtracting.

Further, in the step 5, a 2D channel distribution map is generated according to the acquired electroencephalogram channel position data, the sum of squares of each frequency band value is obtained, and the calculation formula is:

wherein x is a frequency band value, and i ranges from 1 to n.

The sum of squares of the Theta, alpha and beta values obtained from the previous steps is used as the RGB three-channel value of the image. And interpolating the 2D channel distribution map based on an interpolation algorithm to generate an electroencephalogram characteristic 2D distribution map.

Further, in step 6, the specific network structure of the multiple convolutional neural network is as follows: the input layer is a 28-by-28 2D electroencephalogram characteristic map; after the input layer is a convolution module 1, which is composed of two convolution layers stacked continuously, wherein the convolution layer 1 is based on an edge filling mode, and the convolution 2 is based on an edge reduction mode; the convolution module 1 is followed by a maximum pooling layer, the convolution module 2 is formed by continuously stacking two convolution layers, wherein the convolution layer 3 is based on an edge filling mode, the convolution 4 is based on an edge reduction mode, the convolution module 2 is followed by the maximum pooling layer, and finally a full connection layer is stacked;

initializing multiple convolution network parameters, and carrying out forward propagation training; adjusting network parameters based on the mean square error by back propagation; and when the error meets the precision requirement, storing the weight and the bias, finishing the network training, and otherwise, continuously and iteratively adjusting the weight and the bias until the error precision requirement is met.

Further, in step 7, EEG data time sequence features are extracted based on a time convolution neural network, and the specific network structure of the network is as follows: an input layer, wherein a time sequence convolution layer is stacked behind the input layer, and a full connection layer is stacked behind the time sequence convolution layer;

the original input is 28 × 28, the input sequence is acted by a one-dimensional convolution module to obtain a T × M characteristic sequence, T is the length of a time sequence, and M is the number of one-dimensional convolution kernels; the one-dimensional convolution is input to the sequence by an extended convolution, and the magnitude of the convolution kernel is calculated as:

f_{j_d}＝(d-1)*(f_k-1)+f_k

wherein f is_{k_d}Representing the size of a convolution kernel after adding the expansion convolution, d is the expansion rate, and k is the number of convolution kernels;

the output of the extended convolution is acted on each element by an activation function, renl, which is calculated as:

f(x)＝max(0,x)

wherein f (x) is output and x is input;

and setting a residual error module, adding the extended connection identity mapping based on the residual error, and subtracting the input from the original learning transformation H (x), namely F (x) H (x) -x.

Further, the step 8 of fusing and classifying the spatial features and the time sequence features based on Softmax specifically includes:

extracting a full-connection layer containing a spatial feature multiple convolution neural network and a full-connection layer containing a time sequence feature time convolution network, fusing the spatial and temporal features based on a feature splicing mode, and defining a formula:

the FC is a new full-connection layer, the FC1 is a multiple convolution neural network full-connection layer, the FC2 is a time convolution convolutional neural network full-connection layer, and the new full-connection layer is used as the input of a classifier Softmax to realize classification.

The invention has the following advantages and beneficial effects:

the method is based on fast Fourier transform, combines brain electrode scalp position data, effectively maps the space-time characteristics of EEG data in the 2D characteristic diagram, and improves the problem that only visual time sequence channel data is acquired by brain electricity acquisition equipment. Further, the convolutional neural network is combined with the time convolutional neural network to fully mine the time-space characteristics of the EEG data. Using a convolutional neural network, the spatial features of the EEG can be extracted. Using a time-convolved neural network, the temporal features of the EEG can be extracted. Secondly, based on the mode of feature concatenation, the spatial feature and the time feature of EEG are effectively fused together, have solved in traditional EEG feature classification, often abandon the defect of spatial feature, have improved classification performance.

Drawings

FIG. 1 is a flow chart of a motor imagery electroencephalogram feature extraction and classification method based on space-time feature fusion parallel convolution neural network according to an embodiment of the present invention.

Fig. 2 is a diagram of a parallel convolutional neural network structure.

Detailed Description

The technical solutions in the embodiments of the present invention will be described in detail and clearly with reference to the accompanying drawings. The described embodiments are only some of the embodiments of the present invention.

The technical scheme for solving the technical problems is as follows:

as shown in the figure, the method for extracting and classifying the motor imagery electroencephalogram features based on the space-time feature fusion provided by the embodiment comprises the following steps:

step 1: the raw data is preprocessed. In general, raw EEG channel data obtained from experiments includes noises such as myoelectricity and electrooculogram, and is not suitable for direct network training. Therefore, before feature extraction, BCI researchers perform a series of data processing procedures to improve the signal-to-noise ratio, such as high-pass filtering, normalization, and shadow removal. In this patent we take the following data processing method: and (6) removing the average value. To prevent the effect of large differences on the experiment, the amplitude was subtracted from the mean in the data so that the mean of the brain electrical signals was 0; and (6) normalizing. Normalizing the data can effectively reduce the computational magnitude of the network, quicken the iteration of the network, and perform linear transformation on the original data to enable the result to be mapped between [0,1 ].

Step 2: motor imagery EEG has a strong time-series signature, with the peripheral cranial nerves of the scalp producing a signal response over a period of time. The mainstream non-invasive electroencephalogram acquisition equipment adopts a copper sheet or a colloid sensor to collect signals. Due to hardware devices, human brain response, and the like, certain hysteresis problems can arise when data collection is performed. The original time sequence channel EEG data are processed based on overlap cutting, so that each frame of EEG data extracted in a motion image cycle has partial overlap, the effects of discarding useless data and expanding a data set are achieved as far as possible, and meanwhile, the method is more suitable for the actual scene of signal reaction of a human brain. Defining a formula

x_i＝x_i-1+f-o*f i≠0

x_i＝0 i＝0

Wherein x is the starting point of cutting, i is the number of samples, f is the frequency, o x f is the overlap, and o is the cutting weight and ranges from 0 to 1.

According to the data matrix [ [ x ]₀,x₀+128],[x₁,x₁+128],[x₀,x₀+128],...,[x_n,x_n+128]]The 14 EEG channels are sliced and the data for each time window is arranged to ensure that the time series of data is not corrupted.

And step 3: for each EEG channel data, after preprocessing, Fourier transform is carried out on the data of each frame, and three frequency bands of Theta wave, alpha wave and beta wave are extracted.

And 4, step 4: the sum of squares of the values of each frequency band is obtained by three frequency bands of Theta wave, alpha wave and beta wave, and the calculation formula is as follows:

where x is the band value.

And 5: based on an interpolation algorithm, interpolating the 2D channel distribution map to generate a 2D electroencephalogram feature distribution map, which specifically comprises the following steps: and generating a 2D channel distribution diagram according to the acquired channel position data, and taking the sum of squares of Theta, alpha and beta values obtained from the previous step as three channel values of the image RGB. And interpolating the 2D channel distribution map based on an interpolation algorithm to generate an electroencephalogram characteristic 2D distribution map.

Step 6: adopting a plurality of continuous convolution stacks to improve the traditional neural network, and carrying out network training by utilizing the 2D feature distribution map generated in the step 5, wherein the network training specifically comprises the following steps: the input layer is 28 x 28 2D electroencephalogram. After the input layer is a convolution module 1, which consists of stacking two convolutional layers in series, where convolutional layer 1 is based on the edge-filling approach and convolution 2 is based on the edge-reduction approach. The convolution module 1 is followed by a max pooling layer. And the convolution module 2 is formed by continuously stacking two convolution layers, wherein the convolution layer 3 is based on an edge filling mode, the convolution 4 is based on an edge reduction mode, the convolution module 2 is followed by a maximum pooling layer, and finally a full connection layer is stacked.

The convolutional layer is a feature extraction, and is convolved with M convolution kernels by input, and N feature maps are obtained by nonlinear function mapping. The convolution layer is calculated as:

wherein f is an activation function,

is the index vector of the feature map i in layer l, w is the convolution kernel term, and b is the bias term.

The pooling layer is a feature dimension reduction, and the calculation of the pooling layer is as follows:

where down () is the sampling function, N^lThe window boundary size required for the l-th sub-sampling layer,

and outputting the jth characteristic of the ith layer.

Initializing a network parameter weight { w, b }, and carrying out forward propagation training according to (1) and (2). And based on the mean square error, performing back propagation to adjust the network parameters (w, b). And when the error meets the precision requirement, storing the weight and the bias, finishing the network training, and otherwise, continuously and iteratively adjusting the weight and the bias until the error precision requirement is met.

And 7: and (3) simultaneously carrying out parallel training on the 2D characteristic diagram in the step (5) based on a time convolution neural network, and specifically comprising the following steps: and the input layer is stacked with the time sequence convolution layer, and the full connection layer is stacked after the time sequence convolution layer.

The original input is 28 x 28, the input sequence is acted by a one-dimensional convolution module to obtain a T x M characteristic sequence, T is the length of a time sequence, and M is the number of one-dimensional convolution kernels. The one-dimensional convolution is input to the sequence by an extended convolution, and the magnitude of the convolution kernel is calculated as:

f_{j_d}＝(d-1)*(f_k-1)+f_k

wherein f is_{k_d}Denotes the convolution kernel size after adding the extended convolution, d is the expansion ratio, and k is the number of convolution kernels.

The output of the spreading convolution is acted on by the activation function renl for each element. The calculation formula of the ReUL function is as follows:

f(x)＝max(0,x)

where f (x) is the output and x is the input.

And setting a residual module, adding the extended connection identity mapping based on the residual, and changing the learning transformation H (x) into F (x) H (x) -x.

And 8: fusing and classifying the spatial features and the time sequence features based on Softmax, and specifically comprises the following steps: the extraction contains the spatial feature multiple convolution neural network full link layer and contains the time sequence feature time convolution network full link layer, and two full link layers of the parallel network are spliced based on the feature splicing mode, so that a formula is defined:

the FC is a new full-link layer, the FC1 is a multiple convolution neural network full-link layer, and the FC2 is a time convolution convolutional neural network full-link layer. And taking the new full-connection layer as the input of Softmax, performing feature fusion on the full-connection layer containing the space-time features, and testing the classification performance.

The above examples are to be construed as merely illustrative and not limitative of the remainder of the disclosure. After reading the description of the invention, the skilled person can make various changes or modifications to the invention, and these equivalent changes and modifications also fall into the scope of the invention defined by the claims.

Claims

1. A parallel convolution network motor imagery electroencephalogram classification method based on spatiotemporal feature fusion is characterized by comprising the following steps:

step 1: acquiring original EEG channel data, and processing the original EEG channel data by adopting the steps including normalization and mean value removal;

step 6: performing network training on the 2D feature distribution map generated in the step 5 by adopting a multiple convolutional neural network;

and 7: simultaneously performing parallel training on the 2D characteristic graphs in the step 5 based on a time convolution neural network;

2. The electroencephalogram classification method based on spatiotemporal feature fusion and parallel convolution network motor imagery according to claim 1, wherein the step 1 adopts steps including normalization and de-averaging to process raw EEG channel data, and specifically comprises:

3. The electroencephalogram classification method based on spatiotemporal feature fusion and parallel convolution network motor imagery according to claim 2, wherein the step 2 is to segment the raw EEG channel data preprocessed in the step 1 based on an overlap-cut manner, and specifically comprises:

x_i＝x_i-1+f-o*f i≠0

x_i＝0 i＝0

Wherein x is a cutting starting point, i is a few samples, f is a frequency size, and o x f is an overlapping size, wherein o is a cutting weight and ranges from 0 to 1;

4. The electroencephalogram classification method based on spatiotemporal feature fusion and parallel convolution network motor imagery according to claim 3, wherein the step 3 is to perform wavelet transformation on each EEG channel obtained in the step 2 to obtain three frequency bands of Theta wave, alpha wave and beta wave, and specifically comprises:

for each EEG channel data, after preprocessing, Fourier transform is performed on the data of each frame, and x is set to be equal to C^NIs longDegree N EEG signal, then fast fourier transform:

wherein N is 0,1, N-1 is different frequency, W_N＝e^-j(2π/N)。

The inverse fast fourier transform is:

after fast Fourier transform, we obtain a data matrix x containing each frequency band by belonging to Theta wave, alpha wave and beta wave_nExtracting.

5. The electroencephalogram classification method based on the spatiotemporal feature fusion and the parallel convolutional network motor imagery, according to claim 4, wherein in the step 5, a 2D channel distribution map is generated according to the acquired electroencephalogram channel position data, the sum of squares of each frequency band value is obtained, and the calculation formula is as follows:

wherein x is a frequency band value, and the range of i is 1-n;

6. The electroencephalogram classification method based on the spatiotemporal feature fusion and the parallel convolutional network motor imagery according to claim 5, wherein in the step 6, the specific network structure of the multiple convolutional neural network is as follows: the input layer is a 28-by-28 2D electroencephalogram characteristic map; after the input layer is a convolution module 1, which is composed of two convolution layers stacked continuously, wherein the convolution layer 1 is based on an edge filling mode, and the convolution 2 is based on an edge reduction mode; the convolution module 1 is followed by a maximum pooling layer, the convolution module 2 is formed by continuously stacking two convolution layers, wherein the convolution layer 3 is based on an edge filling mode, the convolution 4 is based on an edge reduction mode, the convolution module 2 is followed by the maximum pooling layer, and finally a full connection layer is stacked;

7. The electroencephalogram classification method based on the spatiotemporal feature fusion and the motor imagery of the parallel convolution network as claimed in claim 6, wherein in the step 7, EEG data time sequence features are extracted based on a time convolution neural network, and the network specific network structure is as follows: an input layer, wherein a time sequence convolution layer is stacked behind the input layer, and a full connection layer is stacked behind the time sequence convolution layer;

the original input is 28 × 28, the input sequence is acted by a one-dimensional convolution module to obtain a T × M characteristic sequence, T is the length of a time sequence, and M is the number of one-dimensional convolution kernels; the one-dimensional convolution is input into the sequence by extended convolution, and the calculation size of the convolution kernel is as follows:

f_{j_d}＝(d-1)*(f_k-1)+f_k

wherein f is_kRepresenting a convolution kernel, f_{k_d}Representing the size of a convolution kernel after adding the expansion convolution, d is the expansion rate, and k is the number of convolution kernels;

f(x)＝max(0,x)

wherein f (x) is output and x is input;

8. The electroencephalogram classification method based on spatiotemporal feature fusion and parallel convolutional network motor imagery according to claim 7, wherein the step 8 of fusing and classifying the spatial features and the time series features based on Softmax specifically comprises the following steps: