CN112057047A - Device for realizing motor imagery classification and hybrid network system construction method thereof - Google Patents
Device for realizing motor imagery classification and hybrid network system construction method thereof Download PDFInfo
- Publication number
- CN112057047A CN112057047A CN202010952863.4A CN202010952863A CN112057047A CN 112057047 A CN112057047 A CN 112057047A CN 202010952863 A CN202010952863 A CN 202010952863A CN 112057047 A CN112057047 A CN 112057047A
- Authority
- CN
- China
- Prior art keywords
- hybrid network
- layer
- motor imagery
- electrodes
- time window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000010276 construction Methods 0.000 title claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 28
- 238000012549 training Methods 0.000 claims description 12
- 238000011176 pooling Methods 0.000 claims description 9
- 230000004913 activation Effects 0.000 claims description 8
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 4
- 230000011218 segmentation Effects 0.000 claims description 4
- 238000002790 cross-validation Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 17
- 238000011161 development Methods 0.000 abstract description 6
- 238000004364 calculation method Methods 0.000 abstract description 5
- 238000012545 processing Methods 0.000 abstract description 5
- 238000013527 convolutional neural network Methods 0.000 description 32
- 230000006870 function Effects 0.000 description 9
- 210000004556 brain Anatomy 0.000 description 8
- 230000018109 developmental process Effects 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000002123 temporal effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 210000003205 muscle Anatomy 0.000 description 2
- 210000000578 peripheral nerve Anatomy 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000000537 electroencephalography Methods 0.000 description 1
- 230000015654 memory Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000006403 short-term memory Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B2576/00—Medical imaging apparatus involving image processing or analysis
- A61B2576/02—Medical imaging apparatus involving image processing or analysis specially adapted for a particular organ or body part
- A61B2576/026—Medical imaging apparatus involving image processing or analysis specially adapted for a particular organ or body part for the brain
Abstract
The invention provides a device for realizing motor imagery classification and a hybrid network system construction method thereof. The hybrid network system construction method for realizing motor imagery classification comprises the following steps: segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals; extracting motor imagery related features of each time window signal; and constructing a mixed network based on the 1D-CNN and the LSTM, and inputting the motor imagery related characteristics into the mixed network to train the mixed network. By adopting the invention, the feature extraction and classification recognition of the EEG signal can be realized by constructing a hybrid network system, the invention has strong advantages in the aspect of processing complex data, combines the advantages of 1D-CNN and LSTM, and can reduce the number of electrodes so as to reduce the calculation cost. The experimental result shows that compared with other existing methods, the method has the highest accuracy and the lowest standard deviation in the development of the data set.
Description
Technical Field
The invention relates to the technical field of information, in particular to a device for realizing motor imagery classification and a hybrid network system construction method thereof.
Background
Brain Computer Interface (BCI) technology has become one of the hot spots of research in recent years. The brain-computer interface is a communication system which is not dependent on the conventional peripheral nerve and muscle system of the brain and directly establishes information communication and control between the brain and electronic equipment such as computers. The BCI system based on EEG (electroencephalogram) signals establishes a direct path between the brain and external things by means of the EEG signals, replaces an output path formed by peripheral nerves and muscles of the brain, and accordingly information exchange and control are achieved. BCI of motor imagery type is the most widely studied class of BCI, and information exchange and control between the brain and the outside world are realized by capturing and recognizing EEGs under different motor imagery tasks. The EEG-based motor imagery classification is mainly divided into three steps: pre-treating; extracting relevant characteristics of motor imagery; and (6) classifying. The pre-processing stage reduces the noise effects in the EEG signal; and the motor imagery related feature extraction stage extracts features related to the motor imagery, and then the classification stage is carried out to realize motor imagery classification.
The correct and quick feature extraction and classification identification aiming at the electroencephalogram signals are key parts based on the motor imagery BCI. At present, there are many feature extraction methods in the aspect of motor imagery, such as Common Spatial Pattern filter (CSP), band power, and the like. However, the CSP algorithm mainly focuses on extracting frequency domain features, neglects time domain features of EEG signals, and results in low accuracy and insufficient generalization capability.
Disclosure of Invention
The invention aims to solve the technical problem that the accuracy of feature extraction and classification identification of electroencephalogram signals in the related technology is low, and provides a device for realizing motor imagery classification and a hybrid network system construction method thereof.
The method for constructing the hybrid network system for realizing the motor imagery classification according to the embodiment of the invention comprises the following steps:
segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
extracting motor imagery related features of each time window signal;
and constructing a mixed network based on the 1D-CNN and the LSTM, and inputting the motor imagery related characteristics into the mixed network to train the mixed network.
According to some embodiments of the invention, there are at least two of the time windows that are different in length.
According to some embodiments of the invention, there is at least two of the time window signals that overlap each other.
According to some embodiments of the invention, the extracting the motor imagery related features of each of the time window signals comprises:
and extracting the CSP characteristics of each time window signal by using a second-order Butterworth filter and an FMS-FBCSP algorithm.
According to some embodiments of the invention, the extracting CSP features of each of the time window signals by an FMS-FBCSP algorithm using a second-order Butterworth filter includes:
setting a plurality of different bandwidths to respectively divide the frequency band signals by using a second-order Butterworth filter so as to obtain a plurality of filters;
extending the FMS-FBCSP algorithm to multi-classification using a one-to-one strategy based on a plurality of said filters to extract CSP features for each of said time window signals.
According to some embodiments of the invention, the constructing the 1D-CNN and LSTM based hybrid network comprises:
setting Layer1 of the hybrid network as a 1D-CNN Layer;
setting Layer2 of the hybrid network as a maximum pooling Layer;
setting Layer3 of the hybrid network as a full connection Layer;
setting layers 4 to 6 of the hybrid network to be LSTM layers;
setting Layer12 of the hybrid network as a Softmax Layer.
According to some embodiments of the invention, the hybrid network comprises:
layer 1: 1D-CNN layer, the size of convolution kernel is 128 x 1, step size is 1, and activation function is relu;
layer 2: maximum pooling layer, pool size 2;
layer 3: a full connection layer, with an activation function relu;
layer 4: an LSTM layer, the LSTM unit is 32;
layer 5: an LSTM layer, the LSTM unit is 32;
layer 6: an LSTM layer, the LSTM unit is 32;
layer 12: and a Softmax layer, which is used for carrying out probability prediction and classification on the result.
According to some embodiments of the invention, the inputting at least part of the motor imagery-related feature into the hybrid network to train the hybrid network comprises:
and training the hybrid network by adopting an Adam algorithm, setting a loss function as MSE and taking at least part of the motor imagery correlation characteristics as input data.
According to some embodiments of the invention, the inputting of at least part of the motor imagery-related features into the hybrid network employs a five-fold cross-validation method to train the hybrid network.
The device for realizing motor imagery classification according to the embodiment of the invention comprises the following components:
an acquisition system comprising: fz electrodes, FCz electrodes, Cz electrodes, CPz electrodes, Pz electrodes, C1 electrodes, C2 electrodes, C3 electrodes, C4 electrodes, C5 electrodes, C6 electrodes, CP1 electrodes, CP2 electrodes, CP3 electrodes, and CP4 electrodes, each for acquiring EEG signals;
a signal segmentation system for segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
the hybrid network system is used for carrying out motor imagery classification on a plurality of time window signals;
the hybrid network system is realized based on the hybrid network system construction method for realizing the motor imagery classification.
By adopting the embodiment of the invention, the feature extraction and classification identification of the EEG signal can be realized by constructing the hybrid network system, the method has strong advantages in the aspect of processing complex data, combines the advantages of 1D-CNN and LSTM, and can reduce the number of electrodes, thereby reducing the calculation cost. The experimental result shows that compared with other existing methods, the method has the highest accuracy and the lowest standard deviation in the development of the data set.
Drawings
Fig. 1 is a flowchart of a hybrid network system construction method for implementing motor imagery classification according to an embodiment of the present invention;
fig. 2 is a flowchart of a hybrid network system construction method for implementing motor imagery classification according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of the motor imagery correlation feature extraction according to an embodiment of the present invention;
fig. 4 is a schematic diagram of a key electrode configuration of the acquisition system according to the embodiment of the present invention.
Detailed Description
To further explain the technical means and effects of the present invention adopted to achieve the intended purpose, the present invention will be described in detail with reference to the accompanying drawings and preferred embodiments.
With the rapid development of the deep neural network, feature extraction and classification methods based on deep learning are gradually increased, and a large amount of tedious manual design work can be greatly saved because the deep neural network does not need to manually design features. Among them, the most representative networks are Convolutional Neural Networks (CNN) and Long-Short Term Memory (LSTM). The convolutional neural network includes a two-dimensional-convolutional neural network (two-dimensional CNN, 2D CNN), a one-dimensional convolutional neural network (one-dimensional CNN, 1D CNN), and a three-dimensional convolutional neural network (three-dimensional CNN, 3D CNN). In contrast, 2D-CNN focuses primarily on spatial information, 3D-CNN captures temporal and spatial information but incurs significant computational costs. Not only can 1D-CNN draw the advantages of 3D-CNN without incurring excessive computational cost, but it can also avoid corrupting the time domain features in the EEG signal. LSTM, on the other hand, can extract temporal features of EEG from the temporal characteristics of EEG signals. The technical problem to be solved in the prior art is urgently needed to provide a hybrid network which can simultaneously utilize the advantages of 1D-CNN and LSTM.
Therefore, the embodiment of the invention provides a device for realizing motor imagery classification and a hybrid network system construction method thereof.
As shown in fig. 1, a hybrid network system construction method for implementing motor imagery classification according to an embodiment of the present invention includes:
s1, segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
s2, extracting the motor imagery related characteristics of each time window signal;
s3, constructing a mixed network based on the 1D-CNN and the LSTM, and inputting the motor imagery related characteristics into the mixed network to train the mixed network.
By adopting the embodiment of the invention, the feature extraction and classification identification of the EEG signal can be realized by constructing the hybrid network system, the method has strong advantages in the aspect of processing complex data, combines the advantages of 1D-CNN and LSTM, and can reduce the number of electrodes, thereby reducing the calculation cost. The experimental result shows that compared with other existing methods, the method has the highest accuracy and the lowest standard deviation in the development of the data set.
According to some embodiments of the invention, there are at least two of the time windows that are different in length. It will be appreciated that there may be a plurality of said time windows in which the lengths of the two time windows differ from each other. For example, the plurality of time windows may include a first time window and a second time window, the first time window may have a length of 1 second to 3 seconds, and the second time window may have a length of 2 seconds to 5 seconds.
Therefore, by adopting the multi-scale time window to extract the relevant characteristics of the motor imagery, the influence of individual difference on classification can be minimized, and the precision of classification identification is improved.
In some embodiments of the invention, the lengths of any two of the time windows may both be different.
According to some embodiments of the invention, any two adjacent time window signals overlap each other. It will be appreciated that any two adjacent time windows have overlapping portions. For example, the plurality of time windows may include a first time window and a second time window, the first time window may have a length of 1 second to 3 seconds, the second time window may have a length of 2 seconds to 5 seconds, the first time window includes 2 seconds to 3 seconds, the second time window also includes 2 seconds to 3 seconds, and 2 seconds to 3 seconds are overlapping portions of the first time window and the second time window.
According to some embodiments of the invention, the extracting the motor imagery related features of each of the time window signals comprises:
and extracting the CSP characteristics of each time window signal by using a second-order Butterworth filter and an FMS-FBCSP algorithm. CSP features may be used as motor imagery related features.
According to some embodiments of the invention, the extracting CSP features of each of the time window signals by an FMS-FBCSP algorithm using a second-order Butterworth filter includes:
utilizing a second-order Butterworth filter, and setting a plurality of different bandwidths to respectively divide the frequency band signals so as to obtain a plurality of filters;
for example, the frequency band signals in the interval of 4hz to 38hz in each time window signal may be extracted, and then the frequency band signals are divided by setting the widths of 2hz, 4hz, 8hz, 16hz, and 32hz respectively, so as to obtain 59 filters, where the 59 filters respectively include: the bandwidth is 2 Hz: a filter of 4hz to 6hz, a filter of 6hz to 8hz, a filter of 8hz to 10hz, a filter of 10hz to 12hz, a filter of 12hz to 14hz, a filter of 14hz to 16hz, a filter of 16hz to 18hz, a filter of 18hz to 20hz, a filter of 20hz to 22hz, a filter of 22hz to 24hz, a filter of 24hz to 26hz, a filter of 26hz to 28hz, a filter of 28hz to 30hz, a filter of 30hz to 32hz, a filter of 32hz to 34hz, a filter of 34hz to 36hz, and a filter of 36hz to 38 hz;
the bandwidth is 4 Hz: a filter of 4hz to 8hz, a filter of 8hz to 12hz, a filter of 12hz to 16hz, a filter of 16hz to 20hz, a filter of 20hz to 24hz, a filter of 24hz to 28hz, a filter of 28hz to 32hz, a filter of 32hz to 36hz, a filter of 6hz to 10hz, a filter of 10hz to 14hz, a filter of 14hz to 18hz, a filter of 18hz to 22hz, a filter of 22hz to 26hz, a filter of 26hz to 30hz, a filter of 30hz to 34hz, and a filter of 34hz to 38 hz;
the bandwidth is 8 Hz: a filter of 4hz to 12hz, a filter of 8hz to 16hz, a filter of 12hz to 20hz, a filter of 16hz to 24hz, a filter of 20hz to 28hz, a filter of 24hz to 32hz, a filter of 28hz to 36hz, a filter of 6hz to 14hz, a filter of 10hz to 18hz, a filter of 14hz to 22hz, a filter of 18hz to 26hz, a filter of 22hz to 30hz, a filter of 26hz to 34hz, and a filter of 30hz to 38 hz;
the bandwidth is 16 Hz: a filter of 4Hz to 20Hz, a filter of 8Hz to 24Hz, a filter of 12Hz to 28Hz, a filter of 16Hz to 32Hz, a filter of 20Hz to 36Hz, a filter of 6Hz to 22Hz, a filter of 10Hz to 26Hz, a filter of 14Hz to 30Hz, a filter of 18Hz to 34Hz, a filter of 22Hz to 38Hz, a bandwidth of 32 Hz: filters of 4hz to 36hz, and filters of 6hz to 38 hz.
Therefore, air filtering can be realized to extract features with the largest difference of EEG signals of different classes, so that motor imagery classification is realized.
Extending the FMS-FBCSP algorithm to multi-classification using a one-to-one strategy based on a plurality of said filters, as can be understood with reference to fig. 3, to extract CSP features for each of said time window signals.
According to some embodiments of the invention, the constructing the 1D-CNN and LSTM based hybrid network comprises:
setting Layer1 of the hybrid network as a 1D-CNN Layer;
setting Layer2 of the hybrid network as a maximum pooling Layer;
setting Layer3 of the hybrid network as a full connection Layer;
setting layers 4 to 6 of the hybrid network to be LSTM layers;
setting Layer12 of the hybrid network as a Softmax Layer.
According to some embodiments of the invention, the hybrid network comprises:
layer 1: 1D-CNN layer, the size of convolution kernel is 128 x 1, step size is 1, and activation function is relu;
layer 2: maximum pooling layer, pool size 2;
layer 3: a full connection layer, with an activation function relu;
layer 4: an LSTM layer, the LSTM unit is 32;
layer 5: an LSTM layer, the LSTM unit is 32;
layer 6: an LSTM layer, the LSTM unit is 32;
layer 12: and a Softmax layer, which is used for carrying out probability prediction and classification on the result.
According to some embodiments of the invention, the inputting at least part of the motor imagery-related feature into the hybrid network to train the hybrid network comprises:
and training the hybrid network by adopting an Adam algorithm, setting a loss function as MSE and taking at least part of the motor imagery correlation characteristics as input data.
According to some embodiments of the invention, the inputting of at least part of the motor imagery-related features into the hybrid network employs a five-fold cross-validation method to train the hybrid network.
The device for realizing motor imagery classification according to the embodiment of the invention comprises the following components:
an acquisition system comprising: fz electrodes, FCz electrodes, Cz electrodes, CPz electrodes, Pz electrodes, C1 electrodes, C2 electrodes, C3 electrodes, C4 electrodes, C5 electrodes, C6 electrodes, CP1 electrodes, CP2 electrodes, CP3 electrodes, and CP4 electrodes, each for acquiring EEG signals, the actual use of 15 acquisition electrodes can be seen with reference to fig. 4;
a signal segmentation system for acquiring the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
the hybrid network system is used for carrying out motor imagery classification on a plurality of time window signals;
the hybrid network system is realized based on the hybrid network system construction method for realizing the motor imagery classification.
By adopting the embodiment of the invention, the feature extraction and classification identification of the EEG signal can be realized by constructing the hybrid network system, the method has strong advantages in the aspect of processing complex data, combines the advantages of 1D-CNN and LSTM, and can reduce the number of electrodes, thereby reducing the calculation cost. The experimental result shows that compared with other existing methods, the method has the highest accuracy and the lowest standard deviation in the development of the data set.
An application example of the present invention is described below with reference to fig. 2 to 4.
The existing hybrid network framework is good at extracting time-space domain features, but the networks mainly focus on forming independent models and neglect the influence of individual difference on model construction. At the same time, training the hybrid network results in high computational costs.
Therefore, the embodiment of the invention provides a hybrid network system construction method for realizing motor imagery classification. A Flexible multi-scale Filter Bank Common Spatial mode (FMS-FBCSP) is used for extracting the motor imagery correlation characteristics, and the multi-scale time window is used for extracting the motor imagery correlation characteristics, so that the influence of individual difference on classification is minimized. And the advantages of the 1D-CNN and the LSTM are combined through a multi-scale Hybrid network (MS-HyCaL) with the combination of one-dimension (1D) Conditional Neural Network (CNN) and Long short-term memory (LSTM), thereby extracting the time-space domain characteristics of the EEG signal and solving the problem of network overfitting. In addition, the problem of high calculation cost of the hybrid network is solved by providing a key electrode selection scheme.
Specifically, the apparatus for implementing motor imagery classification according to the embodiment of the present invention includes:
an acquisition system comprising 15 acquisition electrodes for acquiring EEG signals;
a signal segmentation system for acquiring the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
and the hybrid network system is used for carrying out motor imagery classification on a plurality of time window signals.
The hybrid network system is realized based on an MS-HyCaL network, and the network structure is as follows:
layer 1: 1D-CNN layer, the size of convolution kernel is 128 x 1, step size is 1, and activation function is relu;
layer 2: maximum pooling layer, pool size 2;
layer 3: a full connection layer, with an activation function relu;
layer 4: an LSTM layer, the LSTM unit is 32;
layer 5: an LSTM layer, the LSTM unit is 32;
layer 6: an LSTM layer, the LSTM unit is 32;
layer 12: and a Softmax layer, which is used for carrying out probability prediction and classification on the result.
In the MS-HyCaL hybrid network system, a 1D-CNN layer is used for synchronously extracting airspace characteristics aiming at each time window; secondly, pooling the feature map corresponding to the airspace features through a maximum pooling layer, so as to reduce the size of the feature map; thirdly, inputting the obtained feature graph into a full-connection layer for dimension reduction treatment; fourthly, inputting the feature map after dimension reduction into an LSTM layer to extract time domain features; and finally, inputting the motion imagery result into a Softmax layer, and performing probability prediction and classification on the motion imagery result.
The hybrid network system needs to be trained through training data, and in the actual training process, the embodiment of the invention can be divided into a motor imagery related feature extraction stage, a time-space domain feature extraction stage and a classification stage, as shown in fig. 2, and specifically comprises the following steps: an EEG signal is acquired and the original EEG signal is then segmented into five time window signals of different scales and overlapping each other. And then, performing CSP feature extraction on the five time window signals by adopting an FMS-FBCSP algorithm. The CSP algorithm designs a spatial filter, and different classes of EEG signals with the largest difference can be extracted through spatial filtering, so that motor imagery classification is realized. The frequency band range of the embodiment of the invention adopts 4 hz-38 hz, and the bandwidth is set to be 2hz, 4hz, 8hz, 16hz and 32 hz. The band range is divided into 59 filters to form a filter bank. And then, expanding the CSP algorithm to multi-classification by adopting a one-to-one (OvO) strategy, as shown in FIG. 3, thereby realizing the extraction of relevant features of the motor imagery to form training data. And finally, inputting the training data into the hybrid network system for training. The training method of the hybrid network system is ADAM algorithm, and MSE (mean Square error) is selected as a loss function.
In the aspect of acquisition system setting, the embodiment of the invention provides a key electrode configuration scheme according to the asymmetric characteristics of the brain and the position where the motor imagery phenomenon occurs, and as shown in fig. 4, 15 key electrodes are selected for signal acquisition. Evaluation of three evaluation indexes of accuracy, standard deviation and time consumption shows that the key electrode configuration provided by the embodiment of the invention can not only improve the accuracy and reduce the variance, but also obviously shorten the time cost.
In the embodiment of the invention, the FMS-FBCSP adopts a plurality of time windows to extract the relevant characteristics of the motor imagery according to the difference of individual differences, thereby minimizing the influence of the individual differences on classification. In the MS-HyCaL hybrid network, the 1D-CNN network has strong advantages in the aspect of processing large and complex data, and in addition, the feature learning is automatically carried out layer by layer during feature extraction, so that the features of general data can be better represented without excessively depending on training data compared with the traditional manual feature extraction. And the EEG signal is a typical time series signal, and the time sequence of the EEG signal is captured by using an LSTM network to perform feature extraction. The hybrid network of the present invention combines the advantages of 1D-CNN and LSTM and reduces computational cost by simplifying the electrode setup. The experimental result shows that compared with other existing methods, the method has the highest accuracy and the lowest standard deviation in the development of the data set. Also minimizing the number of critical electrodes based on brain asymmetry characteristics reduces computational costs. The experimental results show that compared with the existing network framework at present, the multi-scale hybrid network framework capable of simplifying the electrode has better performance than the most advanced framework under the same conditions.
While the invention has been described in connection with specific embodiments thereof, it is to be understood that it is intended by the appended drawings and description that the invention may be embodied in other specific forms without departing from the spirit or scope of the invention.
Claims (10)
1. A hybrid network system construction method for realizing motor imagery classification is characterized by comprising the following steps:
segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
extracting motor imagery related features of each time window signal;
and constructing a mixed network based on the 1D-CNN and the LSTM, and inputting the motor imagery related characteristics into the mixed network to train the mixed network.
2. The method according to claim 1, wherein at least two time windows have different lengths.
3. The method as claimed in claim 1, wherein there is at least two overlapping time window signals.
4. The method for constructing a hybrid network system for implementing motor imagery classification according to claim 1, wherein the extracting motor imagery related features of each of the time window signals comprises:
and extracting the CSP characteristics of each time window signal by using a second-order Butterworth filter and an FMS-FBCSP algorithm.
5. The method for constructing a hybrid network system for realizing motor imagery classification according to claim 4, wherein the extracting CSP features of each of the time window signals through FMS-FBCSP algorithm using a second order Butterworth filter comprises:
setting a plurality of different bandwidths to respectively divide the frequency band signals by using a second-order Butterworth filter so as to obtain a plurality of filters;
extending the FMS-FBCSP algorithm to multi-classification using a one-to-one strategy based on a plurality of said filters to extract CSP features for each of said time window signals.
6. The method for constructing a hybrid network system for realizing motor imagery classification according to claim 1, wherein the constructing a 1D-CNN and LSTM based hybrid network comprises:
setting Layer1 of the hybrid network as a 1D-CNN Layer;
setting Layer2 of the hybrid network as a maximum pooling Layer;
setting Layer3 of the hybrid network as a full connection Layer;
setting layers 4 to 6 of the hybrid network to be LSTM layers;
setting Layer12 of the hybrid network as a softmax Layer.
7. The method for constructing a hybrid network system for realizing motor imagery classification according to claim 6, wherein the hybrid network comprises:
layer 1: 1D-CNN layer, the size of convolution kernel is 128 x 1, step size is 1, and activation function is relu;
layer 2: maximum pooling layer, pool size 2;
layer 3: a full connection layer, with an activation function relu;
layer 4: an LSTM layer, the LSTM unit is 32;
layer 5: an LSTM layer, the LSTM unit is 32;
layer 6: an LSTM layer, the LSTM unit is 32;
layer 12: and a Softmax layer, which is used for carrying out probability prediction and classification on the result.
8. The method for constructing a hybrid network system for implementing motor imagery classification according to claim 1, wherein the inputting at least a portion of the motor imagery related features into the hybrid network for training the hybrid network comprises:
and training the hybrid network by adopting an Adam algorithm, setting a loss function as MSE and taking at least part of the motor imagery correlation characteristics as input data.
9. The method for constructing a hybrid network system for implementing motor imagery classification according to claim 1, wherein at least a portion of the motor imagery related features are input into the hybrid network, and a five-fold cross validation method is employed to train the hybrid network.
10. An apparatus for implementing motor imagery classification, comprising:
an acquisition system comprising: fz electrodes, FCz electrodes, Cz electrodes, CPz electrodes, Pz electrodes, C1 electrodes, C2 electrodes, C3 electrodes, C4 electrodes, C5 electrodes, C6 electrodes, CP1 electrodes, CP2 electrodes, CP3 electrodes, and CP4 electrodes, each for acquiring EEG signals;
a signal segmentation system for segmenting the EEG signal using a plurality of time windows to obtain a plurality of time window signals;
the hybrid network system is used for carrying out motor imagery classification on a plurality of time window signals;
the hybrid network system is realized based on the hybrid network system construction method for realizing motor imagery classification according to any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010952863.4A CN112057047A (en) | 2020-09-11 | 2020-09-11 | Device for realizing motor imagery classification and hybrid network system construction method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010952863.4A CN112057047A (en) | 2020-09-11 | 2020-09-11 | Device for realizing motor imagery classification and hybrid network system construction method thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112057047A true CN112057047A (en) | 2020-12-11 |
Family
ID=73696189
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010952863.4A Pending CN112057047A (en) | 2020-09-11 | 2020-09-11 | Device for realizing motor imagery classification and hybrid network system construction method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112057047A (en) |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103429145A (en) * | 2010-03-31 | 2013-12-04 | 新加坡科技研究局 | A method and system for motor rehabilitation |
| CN108776788A (en) * | 2018-06-05 | 2018-11-09 | 电子科技大学 | A kind of recognition methods based on brain wave |
| CN109726751A (en) * | 2018-12-21 | 2019-05-07 | 北京工业大学 | Method based on depth convolutional neural networks identification brain Electrical imaging figure |
| CN109820525A (en) * | 2019-01-23 | 2019-05-31 | 五邑大学 | A kind of driving fatigue recognition methods based on CNN-LSTM deep learning model |
| CN110888526A (en) * | 2019-10-30 | 2020-03-17 | 西安理工大学 | Brain-computer interface technology based on integrated support vector learning |
| KR102096565B1 (en) * | 2018-11-08 | 2020-04-02 | 광운대학교 산학협력단 | Analysis method of convolutional neural network based on Wavelet transform for identifying motor imagery brain waves |
| CN111012336A (en) * | 2019-12-06 | 2020-04-17 | 重庆邮电大学 | Parallel convolutional network motor imagery electroencephalogram classification method based on spatio-temporal feature fusion |
| CN111631908A (en) * | 2020-05-31 | 2020-09-08 | 天津大学 | Active hand training system and method based on brain-computer interaction and deep learning |
-
2020
- 2020-09-11 CN CN202010952863.4A patent/CN112057047A/en active Pending
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103429145A (en) * | 2010-03-31 | 2013-12-04 | 新加坡科技研究局 | A method and system for motor rehabilitation |
| CN108776788A (en) * | 2018-06-05 | 2018-11-09 | 电子科技大学 | A kind of recognition methods based on brain wave |
| KR102096565B1 (en) * | 2018-11-08 | 2020-04-02 | 광운대학교 산학협력단 | Analysis method of convolutional neural network based on Wavelet transform for identifying motor imagery brain waves |
| CN109726751A (en) * | 2018-12-21 | 2019-05-07 | 北京工业大学 | Method based on depth convolutional neural networks identification brain Electrical imaging figure |
| CN109820525A (en) * | 2019-01-23 | 2019-05-31 | 五邑大学 | A kind of driving fatigue recognition methods based on CNN-LSTM deep learning model |
| CN110888526A (en) * | 2019-10-30 | 2020-03-17 | 西安理工大学 | Brain-computer interface technology based on integrated support vector learning |
| CN111012336A (en) * | 2019-12-06 | 2020-04-17 | 重庆邮电大学 | Parallel convolutional network motor imagery electroencephalogram classification method based on spatio-temporal feature fusion |
| CN111631908A (en) * | 2020-05-31 | 2020-09-08 | 天津大学 | Active hand training system and method based on brain-computer interaction and deep learning |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Ferrari et al. | On the personalization of classification models for human activity recognition | |
| CN109614922A (en) | A kind of dynamic static gesture identification method and system | |
| Wang et al. | Event-stream representation for human gaits identification using deep neural networks | |
| CN105069434B (en) | A kind of human action Activity recognition method in video | |
| CN110991406B (en) | RSVP electroencephalogram characteristic-based small target detection method and system | |
| CN113065526B (en) | Electroencephalogram signal classification method based on improved depth residual error grouping convolution network | |
| CN110135244B (en) | Expression recognition method based on brain-computer collaborative intelligence | |
| CN110059593B (en) | Facial expression recognition method based on feedback convolutional neural network | |
| CN111797804A (en) | Channel state information human activity recognition method and system based on deep learning | |
| CN112656431A (en) | Electroencephalogram-based attention recognition method and device, terminal equipment and storage medium | |
| CN110781751A (en) | Emotional electroencephalogram signal classification method based on cross-connection convolutional neural network | |
| CN114266276A (en) | Motor imagery electroencephalogram signal classification method based on channel attention and multi-scale time domain convolution | |
| CN111209873A (en) | High-precision face key point positioning method and system based on deep learning | |
| CN113128353B (en) | Emotion perception method and system oriented to natural man-machine interaction | |
| CN108874137B (en) | General model for gesture action intention detection based on electroencephalogram signals | |
| CN111428601B (en) | P300 signal identification method, device and storage medium based on MS-CNN | |
| CN113506274A (en) | Detection system for human cognitive condition based on visual saliency difference map | |
| CN112308087A (en) | Integrated imaging identification system and method based on dynamic vision sensor | |
| CN112057047A (en) | Device for realizing motor imagery classification and hybrid network system construction method thereof | |
| CN111783719A (en) | Myoelectric control method and device | |
| CN116524380A (en) | Target detection method based on brain-computer signal fusion | |
| CN116115240A (en) | Electroencephalogram emotion recognition method based on multi-branch chart convolution network | |
| CN115374831A (en) | Dynamic and static combination velocity imagery classification method for multi-modal registration and space-time feature attention | |
| CN115631515A (en) | Efficient finger vein image feature extraction method and system | |
| CN110958449A (en) | Three-dimensional video subjective perception quality prediction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |