CN115105094B - Motor imagery classification method based on attention and 3D dense connection neural network - Google Patents

Abstract

The application is applicable to the technical field of electroencephalogram signal classification, and provides a motor imagery classification method based on attention and a 3D dense connection neural network, which comprises the following steps: acquiring an electroencephalogram signal data set, and constructing three-dimensional characterization data according to the electroencephalogram signal data set; inputting the three-dimensional representation data into a space-frequency spectrum-time attention module, and dynamically capturing characteristics of different electroencephalogram signal channels, frequency bands and time to obtain space-frequency spectrum-time information; inputting space-frequency spectrum-time information into a 3D densely connected neural network, and obtaining two gradient flow characteristics by using a cross-stage structure; and fusing the two gradient flow characteristics by utilizing a characteristic fusion strategy to obtain a characteristic classification model. The human intention can be accurately identified from the brain signals with low noise ratio and non-stability, and the classification effect of the motor imagery classification task is improved.

Description

Motor imagery classification method based on attention and 3D dense connection neural network

Technical Field

The application belongs to the technical field of electroencephalogram signal classification, and particularly relates to a motor imagery classification method based on attention and a 3D dense connection neural network.

Background

The brain handles various high-level neural activities such as consciousness, language, movement, vision, hearing, emotional expression, etc. With the rapid development of computer processing capacity and signal analysis technology, brain-computer interface (Brain-Computer Interface, BCI) provides a new research approach and method for human analysis of Brain thinking model and consciousness formation. As a novel man-machine interaction technique, BCI completes effective communication between the human brain and a computer by analyzing and decoding Electroencephalogram (EEG) data. Sports imagination is a relatively common paradigm in BCI because it is an autonomous mode of interaction. Specifically, the BCI system is capable of processing the extracted scalp EEG signal features in real time, decoding a type of movement imagination, and generating control commands when a subject performs a movement imagination task, thereby controlling external devices (unmanned aerial vehicle, wheelchair, mobile robot, etc.).

However, due to the low signal-to-noise ratio and instability of the electroencephalogram signal, accurately decoding human intent is a challenge. Accordingly, there is still considerable room for improvement in the implementation and application of motor imagery BCI systems, including their accuracy, interpretability, and availability of online systems. Meanwhile, the motor imagery BCI system is based on the fact that: when a subject or patient imagines moving any part, the corresponding brain area responsible for producing the actual movement is activated.

At present, a great deal of research at home and abroad is devoted to research on a characteristic extraction and classification method of an electroencephalogram signal. The Common space mode (Common SPATIAL PATTERN, CSP) is the most typical feature extraction method, which uses the diagonalization of the matrix to find a set of optimal spatial filters for projection, so as to maximize the variance value of the two types of signals, but the method only focuses on the features of the spatial domain and has certain limitation. The existing classification method based on the neural network mostly ignores complementarity between the brain electrical characteristics, which limits the classification capability of the model to a certain extent. At the same time, we have found that the main focus of researchers is to increase network depth to improve classification success rate, but this ignores the complexity of the algorithm.

Disclosure of Invention

The embodiment of the application provides a motor imagery classification method based on attention and a 3D dense connection neural network, which can accurately identify human intention from brain signals with low noise ratio and non-stability and improve the classification effect of motor imagery classification tasks.

In a first aspect, an embodiment of the present application provides a motor imagery classification method based on attention and a 3D densely connected neural network, including:

acquiring an electroencephalogram signal data set, and constructing three-dimensional characterization data according to the electroencephalogram signal data set;

Inputting the three-dimensional characterization data into a space-frequency spectrum-time attention module, and dynamically capturing characteristics of different electroencephalogram signal channels, frequency bands and time to obtain space-frequency spectrum-time information;

inputting the space-frequency spectrum-time information into the 3D densely connected neural network, and obtaining two gradient flow characteristics by using a cross-stage structure;

And fusing the two gradient flow characteristics by utilizing a characteristic fusion strategy to obtain a characteristic classification model.

Optionally, the acquiring an electroencephalogram signal data set and constructing three-dimensional characterization data according to the electroencephalogram signal data set includes:

Dividing the electroencephalogram signal data set into a training data set and a verification data set according to a preset proportion;

Extracting spectral features in a frequency range of 0-30Hz and time sequence features in a motor imagery task execution 20s in the training data set and the verification data set respectively by using short-time Fourier transformation;

and combining the spectrum features and the time sequence features extracted from each electroencephalogram signal channel at the same time to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set.

Optionally, the extracting the spectral features in the 0-30Hz frequency band and the time series features in the execution motor imagery task 20s in the training data set and the verification data set by using short-time fourier transform respectively includes:

and selecting four non-overlapping frequency bands from the 0-30Hz frequency band, and obtaining frequency spectrum characteristics and time sequence characteristics in the training data set and the verification data set according to different time sequences of each frequency band and all electroencephalogram signal channels.

Optionally, the combining the spectral feature and the time series feature extracted from each electroencephalogram signal channel at the same time to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set includes:

According to an electroencephalogram signal channel, the frequency spectrum characteristic and the time sequence characteristic are respectively converted into a 2D map;

performing cubic spline interpolation on the 2D map;

and stacking all the 2D maps by taking the frequency band length as B and the time sequence number as T as the lengths of the three-dimensional feature maps respectively to obtain the three-dimensional representation data of the training data set and the three-dimensional representation data of the verification data set.

Optionally, the inputting the three-dimensional characterization data into a space-spectrum-time attention module dynamically captures characteristics of different electroencephalogram signal channels, frequency bands and time to obtain space-spectrum-time information, which includes:

inputting the space-spectrum characteristics in the three-dimensional characterization data into a first convolution layer to obtain a characteristic diagram M1;

performing channel global pooling operation on the feature map M1;

Constructing a space-frequency spectrum attention module, and respectively importing the feature map M1 subjected to pooling treatment into two pooling layers to obtain a feature map A11 and a feature map A12;

remodeling the characteristic map A11 and the characteristic map A12 to obtain a high-resolution characteristic map R11 and a high-resolution characteristic map R12;

Obtaining a spectrum attention matrix according to the feature map R11 and the softmax function; obtaining a spatial attention matrix according to the feature map R12 and the softmax function;

and obtaining a space-frequency spectrum characteristic diagram according to the frequency spectrum attention matrix and the space attention matrix.

Optionally, the inputting the three-dimensional characterization data into a space-spectrum-time attention module dynamically captures characteristics of different electroencephalogram signal channels, frequency bands and time to obtain space-spectrum-time information, which includes:

Inputting the space-time characteristics in the three-dimensional characterization data into a second convolution layer to obtain a characteristic diagram M2;

carrying out time domain global average pooling operation on the feature map M2 to obtain a feature map A21 and a feature map A22;

Remodeling the characteristic map A21 and the characteristic map A22 to obtain a high-resolution characteristic map R21 and a high-resolution characteristic map R22;

obtaining a time attention matrix according to the feature map R21 and the softmax function; obtaining a spatial attention matrix according to the feature map R22 and the softmax function;

And obtaining a space-time characteristic diagram according to the space attention matrix and the time attention matrix.

Optionally, the inputting the space-spectrum-time information into the 3D densely connected neural network, obtaining two gradient flow features by using a cross-stage structure includes:

Dividing the spatio-spectral feature map and the spatio-temporal feature map into two parts, respectively;

Respectively inputting a part of the space-frequency spectrum characteristic map and a part of the space-time characteristic map into a three-dimensional dense block, and outputting the processed space-frequency spectrum characteristic map and space-time characteristic map to a transition layer by the three-dimensional dense block;

The other part of the spatio-spectral feature map and the other part of the spatio-temporal feature map are input to a transition layer.

Optionally, the fusing the two gradient flow features by using a feature fusion policy to obtain a feature classification model includes:

The transition layer outputs the processed space-frequency spectrum characteristic diagram and space-time characteristic diagram to the fusion layer, and outputs another part of space-frequency spectrum characteristic diagram and another part of space-time characteristic diagram to the fusion layer;

the fusion layer fuses the processed space-frequency spectrum feature map, the space-frequency spectrum feature map of the other part, the processed space-time feature map and the space-frequency spectrum feature map of the other part to obtain a feature classification model.

Optionally, the method further comprises:

inputting the training data set into the feature classification model for training, and after training for preset times, verifying the feature classification model by using the verification data set;

if the verification result is fitted, retraining the feature classification model.

Compared with the prior art, the embodiment of the application has the beneficial effects that:

Aiming at the problem that the classification accuracy is limited due to the fact that the complementary features of the electroencephalogram signals are ignored in the existing classification method, the application provides a motor imagery classification method based on attention and a 3D dense connection neural network under the condition that a complex model structure and complicated parameter training are not needed, and the method realizes motor imagery multi-classification from two angles: firstly, extracting the characteristic with the most discrimination of the electroencephalogram signal through a space-frequency spectrum-time attention mechanism, capturing the complex relation in the data, and thus reserving all channel characteristics related to motor imagery to the maximum extent; secondly, a 3D dense connection neural network is introduced, and the designed cross-stage structure is utilized to divide the obtained electroencephalogram signal characteristic diagram, so that gradient loss is reduced, and the network learning capacity is enhanced. The application realizes four classification of motor imagery electroencephalogram signals and has higher classification precision.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of a motor imagery classification method based on attention and 3D dense connected neural networks according to an embodiment of the present application;

FIG. 2 is an overall frame diagram of a motor imagery classification method based on attention and 3D densely connected neural networks;

FIG. 3 is a schematic diagram of constructing three-dimensional characterization data;

FIG. 4 is a flow chart of a designed spatio-spectral-temporal attention module;

FIG. 5 is a block diagram of a 3D densely connected neural network;

FIG. 6 is a visual representation of the extracted optimal features;

FIG. 7 is a graph of the results of a four-class confusion matrix obtained in accordance with the present application;

Fig. 8 is a comparison of the results of an ablation experiment.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.

It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.

It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.

As used in the present description and the appended claims, the term "if" may be interpreted in context as "when …" or "once" or "in response to a determination" or "in response to detection. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".

Furthermore, the terms "first," "second," "third," and the like in the description of the present specification and in the appended claims, are used for distinguishing between descriptions and not necessarily for indicating or implying a relative importance.

Reference in the specification to "one embodiment" or "some embodiments" or the like means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the application. Thus, appearances of the phrases "in one embodiment," "in some embodiments," "in other embodiments," and the like in the specification are not necessarily all referring to the same embodiment, but mean "one or more but not all embodiments" unless expressly specified otherwise. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless expressly specified otherwise.

As shown in fig. 1 and2, the motor imagery classification method based on the attention and the 3D densely connected neural network includes steps S101 to S104.

Step S101, acquiring an electroencephalogram data set, and constructing three-dimensional characterization data according to the electroencephalogram data set.

Specifically, the electroencephalogram data set can be obtained by inquiring an existing database or creating a test on the internet. For example, the electroencephalogram data set in the present application can use Dataset a of "electroencephalogram data of the german berlin team in the fourth brain-computer interface game", which records EEG (electroencephalogram ) signals of 9 subjects, with a sampling frequency of 250Hz, each subject performing 576 experiments (each type of task 144 experiments, four types of motor imagery tasks altogether).

After the electroencephalogram data set is obtained, three-dimensional representation data are constructed according to the electroencephalogram data set.

Exemplary step S101 includes steps S1011 to S1013.

Step S1011, dividing the EEG signal data set into a training data set and a verification data set according to a preset proportion.

Specifically, the electroencephalogram data set is divided into a training data set and a verification data set according to a preset proportion. For example, the electroencephalogram data set is divided into a training data set and a verification data set at 4:1.

Step S1012, extracting spectral features in the 0-30Hz frequency band and time series features in the execution motor imagery task 20S in the training data set and the verification data set, respectively, using short-time fourier transforms.

Specifically, first, the training data set and the verification data set are subjected to data preprocessing, as shown in fig. 3, and in consideration of frequency band information and time window information closely related to the motion image category, according to the experimental paradigm shown in fig. 3a and 3b, the spectral features within the frequency band range of 0 to 30Hz and the time series features within the time range of 20s for executing the motor imagery task are extracted using short-time fourier transform.

Illustratively, the time series characteristics and the frequency spectrum characteristics of the training data set and the verification data set are obtained from the preprocessed training data set and the verification data set in four non-overlapping frequency bands ([ 1-4 Hz ], [ 4-7 Hz ], [ 8-13 Hz ], [ 14-30 Hz ]) closely related to motor imagery and different time sequences of all brain electrical signal channels.

X ε R ^N×P is defined as the original EEG data set (training data set and validation data set), where N is the number of EEG channels and P is the number of samples per EEG channel. The temporal and spectral features of selected samples of the electroencephalogram data set are denoted by X ^T∈R^N×T and X ^S∈R^N×B, respectively. They are defined as:

Where T represents a time stamp obtained from each sample and B is a frequency band extracted from the electroencephalogram data set.

Step S1013, combining the spectrum feature and the time sequence feature extracted from each electroencephalogram signal channel at the same time to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set.

Specifically, according to the three-dimensional representation of the electroencephalogram signals corresponding to the brain channels in fig. 3c, the spectral features and the time series features of the same time window are extracted from different brain electrical signal channels. The representation method not only maintains the time domain information and the frequency spectrum information of the electroencephalogram data set, but also maintains the space information of the sampled electroencephalogram data set.

Exemplary, according to the position of the EEG signal channel, the spectrum characteristic X ^T∈R^N×T and the time domain characteristic X ^S∈R^N×B of different channels are respectively converted into a 2D mapAnd/>

After the two-dimensional map is obtained, cubic spline interpolation is adopted to ensure the detail unity of the feature map.

These 2D maps are then stacked with the frequency band length B and the time series number T, respectively, as the lengths of the three-dimensional feature maps, respectively, to obtain three-dimensional representations X ^SS∈R^H×W×B and X ^ST∈R^{H ×W×T} of the electroencephalogram data set (training data set and verification data set), which are defined as:

Step S102, three-dimensional representation data are input into a space-frequency spectrum-time attention module, and characteristics of different electroencephalogram signal channels, frequency bands and time are dynamically captured, so that space-frequency spectrum-time information is obtained.

In particular, the spatio-spectral-temporal attention module includes a spatio-spectral attention module and a spatio-temporal attention module.

Based on the variability of the period and frequency band characteristics of the electroencephalogram signals of the individual motor imagery, the space-frequency spectrum characteristics are input into a first convolution layer, and a characteristic diagram M1 is obtained.

And carrying out channel global pooling operation on the feature map M1 to reduce the calculation cost:

Where a ^avg represents a channel-based global pooling distribution, which is formed by puncturing the original signal input X for a number N of channels. F _nAvg (X) represents a channel-based global pooling function.

And constructing a space-spectrum attention module, and respectively importing the feature map M1 subjected to pooling treatment into two pooling layers, wherein the pooling layers comprise a banded global average pooling layer for reducing the frequency band dimension and a space global average pooling layer for reducing the space dimension, so as to obtain a feature map A11 and a feature map A12.

Wherein A _B∈R^H×W represents the spectral feature distribution,Representing the spatial feature distribution of the B-th band, F _bAvg and F _sAvg represent global averaging functions, and B represents the band length.

And (3) remolding the feature map A11 and the feature map A12 to obtain a high-resolution feature map R11 and a high-resolution feature map R12, and correspondingly halving the feature dimension.

Obtaining a spectrum attention matrix S epsilon R according to the characteristic diagram R11 and the softmax function ^H×W×1

S＝softmax(λA_B+γ)

Where λ and γ are learnable parameters.

From the feature map R12 and softmax functions, a spatial attention matrix is obtained. The spatial attention matrix S ₁∈R^1×1×B is implemented by a fully connected layer with an activation function from R12, the result of which is:

Where λ ₁ and γ ₁ are learnable parameters. The softmax activation function is defined as:

thereby obtaining spatial and spectral prediction signals, respectively.

Finally, a spatio-spectral signature X _SS is obtained by the following formula:

Wherein, Representing the operation of element multiplication.

And inputting the space-time characteristics in the three-dimensional characterization data into a second convolution layer to obtain a characteristic diagram M2. And then carrying out time domain global average pooling operation and space domain global average pooling operation on the feature map M2 to obtain a feature map A21 and a feature map A22 respectively.

The matrix a _T of the global average pool in time domain is defined as:

Where A _T∈R^H×W represents the time feature diffusion and F _tAvg represents the time global averaging function.

Matrix of global averaging pools of spatial domainsIs defined as:

Wherein, Representing the spatial feature distribution of the t-th time stamp of all the electroencephalogram channels, F _dAvg represents the spatio-global averaging pooling function.

And (3) remolding the characteristic map A21 and the characteristic map A22 to obtain a high-resolution characteristic map R21 and a high-resolution characteristic map R22.

The feature map R21 is input to a temporal self-attention matrix T ε R ^H×W×1 with a softmax function.

The spatial attention matrix S ₂∈R^1×1×T is implemented by a normalized feature matrix function acting on R22 as follows:

where λ ₂ and γ ₂ are learnable parameters.

Based on such projections, the spatio-temporal feature map is obtained by the following process:

the construction of the spatial-spectral-temporal attention module is thus completed, as shown in fig. 4.

Step S103, inputting the space-frequency spectrum-time information into a 3D densely connected neural network, and obtaining two gradient flow characteristics by using a cross-stage structure.

Specifically, three-dimensional characterization data is constructed based on the electroencephalogram data set in step S101. The space-frequency spectrum characteristic diagram X _SS and the space-time characteristic diagram X _ST are respectively divided into two parts by utilizing a cross-stage structure, so that two gradient flow characteristics are obtained.

And step S104, fusing the two gradient flow characteristics by utilizing a characteristic fusion strategy to obtain a characteristic classification model.

Specifically, a part of the space-frequency spectrum characteristic diagram and a part of the space-time characteristic diagram are respectively input into a three-dimensional dense block, the three-dimensional dense block outputs the processed space-frequency spectrum characteristic diagram and the space-time characteristic diagram to a transition layer, and the other part of the space-frequency spectrum characteristic diagram and the other part of the space-time characteristic diagram are input to the transition layer. The spatio-spectral feature map X _SS and the spatio-temporal feature map X _ST are divided into two parts, respectively, to increase the gradient path.

Exemplary, as shown in fig. 5 and 6, a space-time characteristic diagram X _ST is illustrated. The spatio-temporal profile X _ST is divided into two parts, X "_ST and X" _ST, respectively.

X '_ST is input into the first three-dimensional dense block, while X' _ST is directly connected to the last stage transition layer. The number of three-dimensional dense blocks determines the depth of the network, each three-dimensional dense block comprises k dense layers, and the input of the first+1th dense layer is composed of the connection result of the output of the first dense layer and the input of the first dense layer. Thus, for the first dense layer of a three-dimensional dense block, it receives as input a combined signature generated by the current and previous convolution layers:

X_l＝H_l([X₀,X_l,X₂,…,X_l-1])

Where [ X ₀,X_l,X₂,…,X_l-1 ] represents a concatenation of feature maps generated by all previous three-dimensional convolution layers, H _l is a continuously operating complex function that includes bottleneck layers (consisting of spatial domain convolution layers of size 3X 1 and temporal domain convolution layers of size 1X 3), which reduces the expensive computational cost and memory requirements of conventional three-dimensional convolutions to some extent.

And (3) bringing the result output by the first three-dimensional dense block into a partial transition layer, bringing the result output by the partial transition layer into a new three-dimensional dense block, and repeating the steps to improve the compactness of the model.

And connecting the output result of the partial transition layer corresponding to the last three-dimensional dense block with the transition layer of the final stage, wherein the last transition layer comprises a batch normalization layer and a convolution layer.

The output of the transition layer is brought into an average pooling layer, the number of the output characteristic diagrams is further reduced, the application of a plurality of transition layers improves the compactness of the model, and the learning ability of the network is improved.

And fusing the features of the two gradient flows by utilizing a feature fusion strategy fusion, and compressing a feature map obtained by feature classification by Maxout. Using class cross entropy as a loss function:

Where M represents the number of classes, y _i = (0, 1,2, 3) represents a multi-class index, and p _i represents the probability of prediction. So far, a feature classification model can be obtained:

Y_{classification}＝F(X_SS,X_ST)

Wherein F represents a mapping function, and Y _{classification} represents a classification result of motor imagery.

Steps S101 to S104 propose a motor imagery classification method based on attention and 3D densely connected neural networks, which achieves motor imagery multi-classification from two angles: firstly, extracting the characteristic with the most discrimination of the electroencephalogram signal through a space-frequency spectrum-time attention mechanism, capturing the complex relation in the data, and thus reserving all channel characteristics related to motor imagery to the maximum extent; secondly, a 3D dense connection neural network is introduced, and the designed cross-stage structure is utilized to divide the obtained electroencephalogram signal characteristic diagram, so that gradient loss is reduced, and the network learning capacity is enhanced. The application realizes four classification of motor imagery electroencephalogram signals and has higher classification precision.

To verify the effect of the feature classification model created by the present application, after step S104, it may further include:

The training data set is input into the feature classification model for training, and after a preset number of times (e.g., 10 times) the feature classification model is validated using the validation data set. If the verification result is over-fitted, retraining the feature classification model until the verification is not over-fitted.

And inputting a test set in the electroencephalogram signal data set into a feature classification model for classification, and comparing the test set with baseline methods FBCSP, deep ConvNet, EEGNet and M3D CNN to test the classification precision of the feature classification model.

Illustratively, each subject is first subjected to an experiment, an electroencephalogram data set of the subject is taken, and the electroencephalogram data set is divided into a training set and a test set in a ratio of 4:1. And setting training parameters of the feature classification model, and enabling the self-adaptive time estimation optimizer to be used for minimizing the classification cross entropy loss function. The learning rate was 0.0001. The number of attention modules is set to 4. The super parameters of the Dropout layer and the constants of the batch normalization layer and the weight decay rate were set to 0.5, 10 ^-5 and 0.1, respectively.

Based on the four classification data sets of brain-computer interface large-race motor imagery, the classification results are compared with the classification results of FBCSP, deep ConvNet, EEGNet and M3D CNN methods, particularly shown in the table 1, wherein the thickened numbers are the optimal classification accuracy of each subject. According to the table, the feature classification model of the application has higher average classification accuracy which reaches 84.45%.

TABLE 1

The validity of the features extracted by the proposed method is mined by a confusion matrix, the experimental results of which are given in fig. 7, where the values on the diagonal of the confusion matrix are the correct prediction samples for the classification task for each moving image.

The effectiveness of different modules in the model is further verified through an ablation experiment, which comprises the following steps: an attention mechanism is added in the CNN network, an attention framework is deleted, and only 3D DCSPNet is applied.

Fig. 8 shows the results of the ablation experiment. Firstly, the SST-attention architecture can adaptively extract space-spectrum-time characteristics, realize complementation of different characteristics and improve classification accuracy. The three-dimensional DCSPNet model can strengthen learning of brain electrical characteristics in different layers. Experimental results show that the motor imagery classification method based on the attention and the 3D dense connection neural network provided by the application has good classification performance and strong generalization capability.

The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.

Claims (2)

1. A motor imagery classification method based on attention and a 3D densely connected neural network, comprising:

acquiring an electroencephalogram signal data set, and constructing three-dimensional characterization data according to the electroencephalogram signal data set;

Inputting the three-dimensional characterization data into a space-frequency spectrum-time attention module, and dynamically capturing characteristics of different electroencephalogram signal channels, frequency bands and time to obtain space-frequency spectrum-time information;

inputting the space-frequency spectrum-time information into the 3D densely connected neural network, and obtaining two gradient flow characteristics by using a cross-stage structure;

Fusing the two gradient flow characteristics by utilizing a characteristic fusion strategy to obtain a characteristic classification model;

The acquiring the electroencephalogram signal data set and constructing three-dimensional characterization data according to the electroencephalogram signal data set comprises:

Dividing the electroencephalogram signal data set into a training data set and a verification data set according to a preset proportion;

Extracting spectral features in a frequency range of 0-30Hz and time sequence features in a motor imagery task execution 20s in the training data set and the verification data set respectively by using short-time Fourier transformation;

Combining the spectrum features and the time sequence features extracted from each electroencephalogram signal channel at the same time to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set;

The extracting spectral features in the 0-30Hz frequency band and time series features in the execution motor imagery task 20s in the training data set and the verification data set respectively by using short-time fourier transform comprises:

Selecting four non-overlapping frequency bands from the 0-30Hz frequency band, and obtaining frequency spectrum characteristics and time sequence characteristics in the training data set and the verification data set according to different time sequences of each frequency band and all electroencephalogram signal channels;

Combining the spectral features and the time sequence features extracted from each electroencephalogram signal channel at the same time to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set, wherein the method comprises the following steps:

According to an electroencephalogram signal channel, the frequency spectrum characteristic and the time sequence characteristic are respectively converted into a 2D map;

performing cubic spline interpolation on the 2D map;

taking the frequency band length as B and the time sequence number as T as the lengths of the three-dimensional feature graphs respectively, and stacking all 2D maps to obtain three-dimensional characterization data of the training data set and three-dimensional characterization data of the verification data set;

the three-dimensional characterization data is input to a space-frequency spectrum-time attention module, and characteristics of different electroencephalogram signal channels, frequency bands and time are dynamically captured to obtain space-frequency spectrum-time information, and the method comprises the following steps:

inputting the space-spectrum characteristics in the three-dimensional characterization data into a first convolution layer to obtain a characteristic diagram M1;

performing channel global pooling operation on the feature map M1;

Constructing a space-frequency spectrum attention module, and respectively importing the feature map M1 subjected to pooling treatment into two pooling layers to obtain a feature map A11 and a feature map A12;

remodeling the characteristic map A11 and the characteristic map A12 to obtain a high-resolution characteristic map R11 and a high-resolution characteristic map R12;

Obtaining a spectrum attention matrix according to the feature map R11 and the softmax function; obtaining a spatial attention matrix according to the feature map R12 and the softmax function;

Obtaining a space-frequency spectrum characteristic diagram according to the frequency spectrum attention matrix and the space attention matrix;

the three-dimensional characterization data is input to a space-frequency spectrum-time attention module, and characteristics of different electroencephalogram signal channels, frequency bands and time are dynamically captured to obtain space-frequency spectrum-time information, and the method comprises the following steps:

Inputting the space-time characteristics in the three-dimensional characterization data into a second convolution layer to obtain a characteristic diagram M2;

carrying out time domain global average pooling operation on the feature map M2 to obtain a feature map A21 and a feature map A22;

Remodeling the characteristic map A21 and the characteristic map A22 to obtain a high-resolution characteristic map R21 and a high-resolution characteristic map R22;

obtaining a time attention matrix according to the feature map R21 and the softmax function; obtaining a spatial attention matrix according to the feature map R22 and the softmax function;

Obtaining a space-time characteristic diagram according to the space attention matrix and the time attention matrix;

the step of inputting the space-spectrum-time information into the 3D densely connected neural network, and obtaining two gradient flow characteristics by using a cross-stage structure comprises the following steps:

Dividing the spatio-spectral feature map and the spatio-temporal feature map into two parts, respectively;

Respectively inputting a part of the space-frequency spectrum characteristic map and a part of the space-time characteristic map into a three-dimensional dense block, and outputting the processed space-frequency spectrum characteristic map and space-time characteristic map to a transition layer by the three-dimensional dense block;

Inputting another part of the spatio-spectral feature map and another part of the spatio-temporal feature map into a transition layer;

The method for fusing the two gradient flow features by utilizing a feature fusion strategy to obtain a feature classification model comprises the following steps:

The transition layer outputs the processed space-frequency spectrum characteristic diagram and space-time characteristic diagram to the fusion layer, and outputs another part of space-frequency spectrum characteristic diagram and another part of space-time characteristic diagram to the fusion layer;

the fusion layer fuses the processed space-frequency spectrum feature map, the space-frequency spectrum feature map of the other part, the processed space-time feature map and the space-frequency spectrum feature map of the other part to obtain a feature classification model.

2. The motor imagery classification method based on an attention and 3D dense connectivity neural network of claim 1, wherein the method further comprises:

inputting the training data set into the feature classification model for training, and after training for preset times, verifying the feature classification model by using the verification data set;

if the verification result is fitted, retraining the feature classification model.