CN110534180A - The man-machine coadaptation Mental imagery brain machine interface system of deep learning and training method - Google Patents

Abstract

Man-machine coadaptation Mental imagery brain machine interface system and training method disclosed by the invention based on deep learning solves the problems, such as that difference on effect is big between different users, EEG signals stability is poor.System is connected to brain electric data collecting, data prediction, signal transformation differentiation network and visual feedback and module is presented, and further includes by original eeg data, the data set module of signal switch target image and class label composition.Training method includes general pre-training, coadaptation feedback calibration, used for three stages online；Coadaptation training collects data and machine training containing free practice, the task imagination containing man-machine coadaptation training, every wheel training is taken turns more.Training process of the present invention uses the Real-time Feedback in " people is in circuit ", allows the decodable code Mental imagery mode that user's acquistion in trial and error is correct and stable, improves the performance of Mental imagery brain-computer interface, realizes and classifies to the high-accuracy stable of user's Mental imagery.It is mainly used for meeting efficient brain-machine interaction demand.

Description

The man-machine coadaptation Mental imagery brain machine interface system of deep learning and training method

Technical field

The invention belongs to information technology fields, neural with depth further to user is passed through in biological interleaving techniques The coadaptation feedback training of network (Deep Neural Network, DNN) realizes DNN to Mental imagery brain electricity in user The personalized of information is extracted, specifically a kind of man-machine coadaptation Mental imagery brain machine interface system and instruction based on deep learning Practice method, the high-accuracy stable of user's Mental imagery is divided for the performance of lifting motion imagination brain-computer interface and realization Class.

Background technique

Brain-computer interface be established between brain and external equipment be directly connected to access.Brain-computer interface (BCI) technology is biology The product of science and technology and computer technology mixing together is the important means that the mankind explore brain, develop brain, while being also contemporary section Skill big powers must strive field.Moreover brain-computer interface (BCI) technology is many in military combat, rehabilitation, consumer entertainment etc. There is certain application in field.Such as in military field, it can use BCI technology and come to vehicle, weapon etc. carries out auxiliary control System；The specific EEG signals that can be generated in daily life by different thinking activities, to external equipment as being remotely controlled, keyboard, The equipment such as electric light are controlled.Control for robot can also be realized by Mental imagery brain-computer interface, allow robot root Make correlated response according to the will of people, therefore, for BCI research either in the safe level of national science and technology or in day It is all of great significance in the every field often lived.

In BCI technology, Mental imagery brain-computer interface (MI-BCI) has important practical value, is different from stable state vision It induces brain-computer interfaces such as (SSVEP) to need to induce stimulation, MI-BCI is the brain-computer interface based on spontaneous brain electricity, without using person Insufferable induction stimulation, induction type BCI of the comfort used much higher than SSVEP etc..MI-BCI passes through decoding algorithm The detection user sensorimotor rhythm (SMR) (SMR) spontaneous in brain sensorimotor area when imagining limb motion is current to differentiate The limb motion classification that user is imagined, and then realize human-computer interaction and the control to machinery equipment, it is provided more for user For the experience of natural brain-machine interaction.In the EEG signal decoding algorithm of MI-BCI, cospace mode (Common spatial Pattern, CSP) and its innovatory algorithm be most effective feature extracting method.With the development of depth learning technology in recent years, Numerous MI-BCI decoding algorithms based on deep learning are also suggested.Wherein the representative are Shiu Kumar et al. to mention The EEGNet even depth learning method that CSP-DNN and Lawhern Vernon out et al. is proposed.Above-mentioned algorithm is in existing public affairs It opens and achieves good marks sequencing on data set, but since human brain is biosystem living, complexity and time variation are to grind Study carefully the biggest obstacle of brain-computer interface BCI.In the research of MI-BCI technology, the unstable imagination, the mistake of the measured often occurred The problems such as accidentally imagination or even a part of measured (15%~30%) can not detect sensorimotor rhythm (SMR) SMR at all.This makes The development of algorithm does not solve the basic problem of MI-BCI above-mentioned, i.e., the unstable imagination, mistake of measured imagine and " BCI illiteracy " problem, the ITR of MI-BCI are further promoted and have been met with bottleneck.

To solve the above problems, the correctness of cerebration and SMR intensity, researcher mention when promoting user's Mental imagery User of a variety of MI-BCI feedback enhancing training methods to training MI-BCI is gone out.Gao little Rong etc. proposes a kind of feedback instruction Practice normal form, when measured imagines appointed task, the classification results of classifier is presented on screen in real time with arrow form, realize To the feedback training of people.Han-Jeong Hwang etc. provides a kind of more vivid feedback normal form, and EEG Real time visible is turned to The brain mapping of frequency band is supplied to measured and carries out feedback training.Feedback training method effectively improves the fortune of measured The brain signal intensity that the dynamic imagination can differentiate.But since existing feedback method all joined fixed elder generation in feature extraction phases Knowledge is tested, such as defines brain area or frequency band, so that the result of feedback has been limited in the setting range of researcher.

The region of brain signal when in order to cause a machine to learn to arrive user's Mental imagery under priori setting as few as possible And frequency band distribution, it allows people and machine coadaptation, further promotes the performance of MI-BCI.The coadaptation method of MI-BCI by The concern of researcher.Carmen Vidaurre et al. proposes a kind of man-machine based on CSP and linear discriminant analysis (LDA) Coadaptation MI-BCI calibration method, to solve the problems, such as " BCI illiteracy ", and the certain effect obtained.Rafael Cabeza etc. People proposes a kind of complete online MI-BCI realization using autoregression model, trains by the 4 online coadaptations of wheel, measured's Mental imagery classification error rate is decreased obviously.

Existing man-machine coadaptation method further improves the performance of MI-BCI, but machine portion in terms of people and machine two Divide the priori knowledge that still inevitably joined about frequency band and feature extraction mode；Meanwhile machine is equal to the feedback of people For identification feedback, that is, the mode of feedback is the presentation of the classification results of classifier.And brain electricity EEG is a kind of high-dimensional letter Breath, wherein careful about information such as SMR intensity, position distribution and stability containing enriching；Existing man-machine coadaptation method fortune The classification results dimension of the dynamic imagination is low, eeg signal acquisition and classifying quality difference is big, EEG signals between different users Stability is poor, and above-mentioned information abundant can not be presented to user in variation by classifying quality, and then cannot achieve information more Feedback abundant.

The present invention passes through a certain range of retrieval and Cha Xin, still without finding document relevant to present subject matter and report Road.

Summary of the invention

The present invention is directed to the deficiency and demand of existing Mental imagery brain-computer interface (MI-BCI) technology, proposes that a kind of promotion makes The correctness of user's Mental imagery cerebration, feedback result should be readily appreciated that identification, and classification results are more stable accurate, using model Enclose widely man-machine coadaptation Mental imagery brain machine interface system and training method based on deep learning.

The present invention is a kind of man-machine coadaptation Mental imagery brain-computer interface (DeCa-BCI) system based on deep learning first System, which is characterized in that according to information processing sequence, be sequentially connected and include the eeg data of acquisition measured's eeg data Acquisition module, data preprocessing module, signal transformation differentiate that module, vision is presented in network (STD-Net) module and visual feedback Feedback is presented module and provides visual information feedback to measured by feedback display device, by the visual reception of measured, through quilt Brain electric data collecting again is formed after survey person's active accommodation Mental imagery mode；The signal transformation differentiates network (STD- Net) module, wherein be sequentially connected and include signal converting network (ST-Net) submodule and differentiate network layer submodule；System System further includes having data set module, and data set module includes original eeg data, signal switch target image and classification mark Label, data set module and data preprocessing module carry out bidirectional data interaction and take out original eeg data；Data set module and letter Number converting network submodule bidirectional data interaction takes out signal switch target image；Data set module and differentiation network layer submodule Block carries out bidirectional data interaction and takes out class label；Above-mentioned two-way interactive includes the transmission of data and the transmission for controlling signal； During feedback training, ST-Net submodule is presented module for visual feedback and provides signal conversion image；It is migrated in identification In use process, ST-Net submodule converts image to differentiate that network layer submodule provides signal, through differentiating network layer submodule The classification results of EEG signals are obtained after block processing.Each module is described below:

Brain electric data collecting module is made of brain wave acquisition equipment；The brain of brain electric data collecting module reception measured Electric signal completes data acquisition of the user during imagination, the data collected with fixed sample rate and distribution of electrodes For eeg data, eeg data is transferred to data preprocessing module.The design of DeCa-BCI system does not limit EEG signal acquisition Equipment and lead number, i.e. brain wave acquisition equipment are arranged according to demand, or for wired brain wave acquisition equipment or are that wireless brain electricity is adopted Collect equipment.Minimum lead number need to be greater than 1 lead.

Data preprocessing module: original in reception brain electric data collecting module eeg data collected or data set Eeg data successively carries out baseline to the eeg data received, and filtering goes power frequency, down-sampled pretreatment obtains pre- place Eeg data after reason, and send pretreated eeg data to signal transformation and differentiate that the signal in network module converts Network submodular.

Signal transformation differentiates network (STD-Net) module, including signal converting network (ST-Net) submodule and differentiation net Network layers submodule；In the propagated forward use process of ST-Net submodule: ST-Net submodule receives data preprocessing module Processing result, output with discriminant information signal convert image；It is defeated to differentiate that network layer submodule receives ST-Net submodule Signal out converts image, obtains classification results；In the backpropagation training process of ST-Net submodule: eeg data warp ST-Net submodule is sent to after crossing preprocessing module processing, while ST-Net submodule obtains correspondence from data set module The signal switch target image of classification, and as target image execute back-propagation algorithm training ST-Net, eeg data or The eeg data for being the original eeg data in data set module or acquiring in real time.Differentiate that network layer submodule receives The signal of ST-Net submodule output converts image, while class label is obtained from data set module, and hold as label The training of row back-propagation algorithm differentiates network layer.

Module is presented in visual feedback, and the signal for receiving ST-Net submodule output in STD-Net module converts image, will scheme As being presented to measured by feeding back display device；The signal conversion mesh that measured converts image according to signal and data are concentrated The difference of logo image, active accommodation Mental imagery mode develop signal conversion image towards steady and audible direction, realize Feedback regulation to itself EEG signals.

The present invention or a kind of man-machine coadaptation Mental imagery brain-computer interface training method based on signal transformation, in right It is required that being realized on man-machine coadaptation Mental imagery brain machine interface system described in 1-3 based on deep learning, which is characterized in that Include trained and use process:

Successively there are two the general pre-training of stage (Level): Level 1,2 coadaptations of Level to feed back school for training process It is quasi-:

The general pre-training of Level 1: the data training signal converting network (ST-Net) in data set module, ST- are used Net network completes a forward direction and transmits to obtain output of the signal conversion image as ST-Net submodule；It is calculated using backpropagation Method updates ST-Net network parameter, and ST-Net network submodular is made to obtain the initial decoded transform ability to new data.

2 coadaptation feedback calibration of Level: the signal transition diagram that module is exported with ST-Net submodule is presented in visual feedback As being presented to measured by the visual information of feedback by feedback display device as input, by the visual reception of measured, Through measured's active accommodation Mental imagery mode, brain electric data collecting again is formed, iterative cycles are come with this and are carried out, are realized To the feedback training of measured；Signal converting network (ST-Net) receives pretreated eeg data and data set mould simultaneously Corresponding signal switch target image in block updates own net parameter using back-propagation algorithm, realizes and convert net to signal The training of network parameter；Feedback training to user and to the training of signal converting network parameter alternately, finally realize people Machine coadaptation feedback calibration, network parameter at this time are fixed ST-Net parameter；Specific training process be divided into three steps (Step) into Row: Step1 measured's free practice；Step2 measured completes Mental imagery under briefing and records original brain electricity number According to；The training of Step3 machine；When termination machine training is determined that measured is according to the signal transition diagram itself judged by measured The differential effect for the signal switch target image that picture and data are concentrated determines when terminate the coadaptation feedback school of entire Level2 Quasi- process.

Use process has stage (Level): the migration of Level3 identification and use:

The migration of 3 identification of Level and use: it after coadaptation feedback calibration, is added and sentences behind ST-Net submodule Other network layer submodule constitutes signal transformation and differentiates network (STD-Net), and fixed ST-Net parameter, training differentiates network layer, multiple The ST-Net neural network parameter learnt with Level 2 realizes that the high-precision to Mental imagery MI, high stability are classified.

DeCa-BCI of the invention by the study in above three stage, the Synchronous lifting accuracy of people's Mental imagery, Stability；Machine to the mutually coordinated adaptedness of the transformation of Mental imagery EEG signals and discriminating power and brain-computer interface, Finally make DeCa-BCI that Mental imagery EEG signals are obtained with the differentiation performance of accurate stable.

The present invention has the advantage that compared with prior art

Promote the correctness of user's Mental imagery cerebration: effect is poor between traditional Mental imagery brain-computer interface user Different big, unstable result.Since cerebration mechanism of the user to MI-BCI lacks understanding and use experience, the reality of MI-BCI Test that room result difference is huge and whole ITR is lower.There is 15%~30% MI-BCI user that can not become by detection SMR Dissolve the sports category of its imagination of code.This promotes the ITR of MI-BCI further to have met with bottleneck.It is of the invention based on depth The training process of the man-machine coadaptation Mental imagery brain-computer interface of degree study uses the side of the Real-time Feedback in " people is in circuit " Formula, allows the decodable Mental imagery mode that user's acquistion in trial and error is correct and stable.

Final classification result is more stable accurate: machine is to differentiate to the feedback of people in existing man-machine coadaptation method Property feedback, that is, the mode fed back be classifier classification results presentation.And EEG is a kind of high-dimensional information, wherein containing rich It is rich careful about information such as SMR intensity, position distribution and stability；The classification results dimension of Mental imagery is low, can not become Above-mentioned information abundant is presented to user in change, and then cannot achieve the richer feedback of information.DeCa-BCI is by making The coadaptation feedback training of user and deep neural network (Deep Neural Network, DNN) realize DNN to user The personalized of Mental imagery information is extracted in EEG, while being by ST-Net migration under conditions of not losing identification information STD-Net obtains the MI classification results of accurate stable.

Feedback result should be readily appreciated that identification: traditional feedback training method is all made of the feedback form of low dimensional, cannot be to make User provides feedback information abundant.ST-Net completes to be difficult to the EEG signal directly recognized from people to can be with the figure of intuitivism apprehension As the transformation of feedback, allow user that can complete the feedback training to displacement imagination mode according to high-dimensional image information.

Have wide range of applications: this method, as Real-time Feedback, is made without special using the understandable image format of ordinary people The user of industry knowledge can also complete the feedback calibration of MI-BCI under the guidance of layman.This advantage expands significantly The application space for having opened up MI-BCI makes ordinary people operate multiple external equipments with MI-BCI brain-computer interface or communicate with computer It is more convenient.

Detailed description of the invention

Fig. 1 is man-machine coadaptation Mental imagery brain-computer interface (DeCa-BCI) the system knot of the invention based on deep learning Structure block diagram；

Fig. 2 is that signal transformation of the invention differentiates network (STD-Net)；

Fig. 3 is the network structure of signal converting network (ST-Net) of the invention；

Fig. 4 is signal switch target example images of the invention；

Fig. 5 is the establishment process of DeCa-BCI of the invention；

Fig. 6 is to feed back normal form in the Level2 stage of DeCa-BCI of the invention；

Fig. 7 outputs and inputs data mode for free practice step signal converting network (ST-Net) of the invention；

Fig. 8 is the feedback interface that task of the invention imagines step；

Fig. 9 is that interface is presented in the training effect of machine training step of the invention；

Figure 10 is system block diagram of the present invention in identification migration use process.

Specific embodiment

The present invention is described in detail with reference to the accompanying drawing.

Embodiment 1

Brain-computer interface technology is empty in the application that the various fields such as military, medical rehabilitation and Edutainment suffer from the prospect of being rich in Between.MI-BCI has been taken the lead in exploring and be laid out by each military power in terms of military combat collaboration commander, military weapon； Space is explored there is also very big in application scenarios such as medical rehabilitation training, ectoskeleton manipulations；Meanwhile in education and training and toy The consumptions of resident such as game market, is excavated by many big companies and newly established enterprise.

In above-mentioned application field and scene, the accuracy and stability of MI-BCI has to be hoisted.Traditionally to MI- The research of BCI decoding algorithm and normal form all can not fundamentally solve the problems, such as that user is incorrect, the inaccurate imagination, also without Method is completed to be adapted to user and the personalized of algorithm.

The present invention sees the development prospect in this field, by constantly studying and exploring, proposes a kind of based on depth The man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) of study and training method.

The present invention is a kind of man-machine coadaptation Mental imagery brain-computer interface (DeCa-BCI) system based on deep learning first System, according to information processing sequence, is sequentially connected and includes the brain electric data collecting of acquisition measured's eeg data referring to Fig. 1 Module, data preprocessing module, signal transformation differentiate that module, visual feedback is presented in network (STD-Net) module and visual feedback Module is presented by feedback display device, such as screen provides visual information feedback to measured, i.e., by the visual information of feedback It is placed in the horizon range that measured can be high-visible, by the visual reception of measured, through measured's active accommodation Mental imagery Brain electric data collecting again is formed after mode；The signal transformation differentiates network (STD-Net) module, wherein being sequentially connected With include signal converting network (ST-Net) submodule and differentiate network layer submodule.It is man-machine mutually suitable based on deep learning Answering Mental imagery brain-computer interface (DeCa-BCI) system further includes having data set module.Data set module includes original brain electricity Data, signal switch target image and class label, data set module carry out bidirectional data interaction with data preprocessing module and take Original eeg data out.In the present invention, data set module and signal converting network submodule bidirectional data interaction take out signal Switch target image；Data set module and differentiation network layer submodule carry out bidirectional data interaction and take out class label；Above-mentioned Two-way interactive includes the transmission of data and the transmission for controlling signal.During feedback training, ST-Net submodule is that vision is anti- Feedback is presented module and provides signal conversion image.In identification migration and use process, ST-Net submodule is to differentiate network layer Submodule provides signal and converts image, obtains the classification results of EEG signals after being differentiated the processing of network layer submodule.To each mould Block is described below:

Brain electric data collecting module is made of brain wave acquisition equipment；The brain of brain electric data collecting module reception measured Electric signal completes data acquisition of the user during imagination, the data collected with fixed sample rate and distribution of electrodes For eeg data, eeg data is transferred to data preprocessing module；The design of DeCa-BCI system does not limit EEG signal acquisition Equipment and lead number, i.e. brain wave acquisition equipment are arranged according to demand, or for wired brain wave acquisition equipment or are that wireless brain electricity is adopted Collect equipment；Minimum lead number need to be greater than 1 lead.

Data preprocessing module: original in reception brain electric data collecting module eeg data collected or data set Eeg data successively carries out baseline to the eeg data received, and filtering goes power frequency, down-sampled pretreatment obtains pre- place Eeg data after reason, and send pretreated eeg data to signal transformation and differentiate that the signal in network module converts Network submodular.

Signal transformation differentiates network (STD-Net) module, including signal converting network (ST-Net) submodule and differentiation net Network layers submodule.In the propagated forward use process of ST-Net submodule: ST-Net submodule receives data preprocessing module Processing result, output with discriminant information signal convert image；It is defeated to differentiate that network layer submodule receives ST-Net submodule Signal out converts image, obtains classification results；In the backpropagation training process of ST-Net submodule: eeg data warp ST-Net submodule is sent to after crossing preprocessing module processing, while ST-Net submodule obtains correspondence from data set module The signal switch target image of classification, and as target image execute back-propagation algorithm training ST-Net, eeg data or The eeg data for being the original eeg data in data set module or acquiring in real time.Differentiate that network layer submodule receives The signal of ST-Net submodule output converts image, while class label is obtained from data set module, and hold as label The training of row back-propagation algorithm differentiates network layer.

Module is presented in visual feedback, and the signal for receiving ST-Net submodule output in STD-Net module converts image, will scheme As being presented to measured by feedback display device, such as screen；Measured concentrates according to signal conversion image and data The difference of signal switch target image, active accommodation Mental imagery mode make signal convert image towards steady and audible direction The feedback regulation to itself EEG signals is realized in development.

The present invention is for the problems such as difference on effect is big, stability is poor between the generally existing user of Mental imagery brain-computer interface Propose the man-machine coadaptation Mental imagery brain machine interface system based on deep learning and a kind of whole technical side of training method Case.

Specifically the training process of the man-machine coadaptation Mental imagery brain-computer interface DeCa-BCI based on deep learning is adopted With the Real-time Feedback in " people is in circuit ", the decodable Mental imagery side for making user's acquistion in trial and error correct and stable Formula；The DNN of design is gradually allowed further to learn to the movement for being suitble to user using DNN powerful ability in feature extraction simultaneously Imagine feature, is that sorter network realizes personalized high-precision MI-BCI by transfer learning.Experiment shows that DeCa-BCI has It imitates the different types of user of help and improves the decoding efficiency of Mental imagery, pushed the further functionization of MI-BCI, The application space of MI-BCI has been expanded significantly.

DeCa-BCI of the invention is a kind of MI-BCI with coadaptation feedback calibration ability, can be used for MI-BCI's Types of applications scene.In the past due to the Stability and veracity of MI-BCI is bad and can not widespread adoption field, such as army Thing commander, medical rehabilitation etc. will become to be more likely to get large-scale application because of the appearance of DeCa-BCI.DeCa- BCI provides the feedback calibration mode of innovation for MI-BCI, can provide at anyone, any time using before MI-BCI Calibration service, so that machine algorithm can adapt to this time-varying system of brain well, and then it is more extensive to allow MI-BCI to obtain Application.

The correctness of cerebration and SMR intensity when this system improves user's Mental imagery: traditional brain wave acquisition side Formula, since cerebration mechanism shortage understanding and use experience, the laboratory result difference of MI-BCI of the user to MI-BCI are huge Big and whole ITR is lower.There is 15%~30% MI-BCI subject can not be by the fortune of its imagination of detection SMR variation decoding It is dynamic.This promotes the ITR of MI-BCI further to have met with bottleneck.Man-machine coadaptation fortune based on deep learning of the invention The method that the training process of dynamic imagination brain-computer interface uses the Real-time Feedback in " people is in circuit " and coadaptation feedback is presented, Allow the decodable Mental imagery mode that user's acquistion in trial and error is correct and stable.

Embodiment 2

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) based on deep learning is totally constituted and specific Design is with embodiment 1, and the building of STD-Net module is as follows in the present invention:

Referring to Fig. 1, STD-Net module is divided into ST-Net submodule and differentiates network layer submodule, ST-Net submodule packet ST-Net network is contained, has differentiated that network layer submodule contains differentiation network layer；ST-Net submodule, which is realized, believes brain electricity EEG Number be transformed to that a user can intuitively recognize from the dimensional matrix data of time T* lead C with Mental imagery task kind The relevant signal of class converts image, and it is the gray level image with strong identification information which, which converts image, and shape can voluntarily be set It is fixed, for example, the left or right cross that the gray level image is, circle, the shapes such as triangle, so that amateur measured carries out feedback tune Control, at the same can pole easily by ST-Net transfer learning be identification algorithm.Differentiate that network layer submodule receives ST-Net The signal of module output converts image, and propagated forward obtains the classification results of Mental imagery.

Referring to Fig. 3, ST-Net network is made of five layers of convolutional neural networks, and convolution kernel size is N*N, such as N=16, It is respectively provided with activation primitive ReLU；The convolution kernel number of first layer is M, such as M=32, and second layer convolution kernel number is 0.5* M, third layer convolution kernel number are 0.5*M, and the 4th layer of convolution kernel number is 0.25*M, and layer 5 convolution kernel number is 1, are owned Layer convolution step-length is 1；Every layer of output is all the characteristic pattern with input with size, such as time (T) the * lead (C) of input Matrix is T=128, C=64, then exporting result size is also 128*64.Referring to fig. 2, differentiate that network layer is in the defeated of ST-Net A convolution kernel size being added after layer out is the convolutional layer of C*C, and the training method of the convolution kernel is to use the defeated of ST-Net Result uses Mental imagery class label as target value training as the input for differentiating network layer, the fixed part ST-Net out The parameter of the convolution kernel；It completes to convert image to differentiation from signal with the wide individual convolution kernel of result using one As a result conversion balances the stability and flexibility of migration to greatest extent.

Differentiation network layer is that a convolution kernel size being added after the output layer of ST-Net is the convolutional layer of C*C, is used An individual convolution kernel is completed to convert the conversion of image to identification result from signal.

ST-Net network using and training: signal converting network (ST-Net) receives pretreated in use Eeg data and carry out propagated forward obtain signal conversion image；Signal converting network (ST-Net) is in the training process simultaneously It receives pretreated eeg data and data concentrates corresponding signal switch target image, use cross entropy as loss letter The signal conversion image and signal switch target image of number comparison network output, and back transfer algorithm is executed to train ST- Net itself.

Differentiate using and training for network layer: differentiating that network layer receives ST-Net submodule propagated forward in use Obtained signal conversion image, propagated forward obtain the classification results of Mental imagery；Differentiate network layer in the training process simultaneously Corresponding class label in the signal conversion image and data collection module that ST-Net submodule propagated forward obtains is received, is used The comparison of cross entropy loss function differentiates the classification results and class label of network layer output, and uses the training of back transfer algorithm Differentiate network layer.

ST-Net described in this example is completed from people and is difficult to the EEG signal directly recognized to can be anti-with the image of intuitivism apprehension The transformation of feedback, while migrating ST-Net for STD-Net under conditions of not losing identification information, obtain accurate stable MI classification results have pushed the further functionization of MI-BCI, have expanded the application space of MI-BCI significantly.

Embodiment 3

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) based on deep learning is totally constituted and specific With embodiment 1-2, the building of data set module is as follows for design:

Original eeg data and matching signal switch target image and class are contained in data set of the invention Distinguishing label；The data format of original eeg data and the output phase of this system brain electric data collecting module are same；Class label is fortune The classification of the dynamic imagination；Referring to fig. 4, signal switch target image is and the consistent ash containing strong identification information of class label Spend image.Such as defining the signal switch target image of right-hand man's Mental imagery task is the gray level image of left or right cross star, Aspect Ratio is 2:1, black background and the png format-pattern with unilateral white crosses.

Data set module of the invention is training and use and signal converting network (ST-Net) In for differentiating network layer Trained and use process provides data, to realize the adaptation training to ST-Net net machine algorithm.

Embodiment 4

The present invention or a kind of man-machine coadaptation Mental imagery brain-computer interface training method based on signal transformation, above-mentioned The man-machine coadaptation Mental imagery brain machine interface system based on deep learning on realize, it is man-machine mutually suitable based on deep learning Answering Mental imagery brain machine interface system (DeCa-BCI), totally composition and specific design with embodiment 1-3 include referring to Fig. 5 Trained and use process:

Successively there are two stage (Level) for training process: the general pre-training of Level1, Level2 coadaptation feedback calibration:

The general pre-training of Level 1: the data training signal converting network (ST-Net) in data set module, ST- are used Net network completes a forward direction and transmits to obtain output of the signal conversion image as ST-Net submodule；It is calculated using backpropagation Method updates ST-Net network parameter, and ST-Net network submodular is made to obtain the initial decoded transform ability to new data.

2 coadaptation feedback calibration of Level: the signal transition diagram that module is exported with ST-Net submodule is presented in visual feedback As input, by the visual information of feedback by feedback display device, such as screen, being presented to measured can be high-visible Horizon range in, by the visual reception of measured, through measured's active accommodation Mental imagery mode, formed again brain electricity Data acquisition carrys out iterative cycles with this and carries out, realizes the feedback training to measured；Signal converting network (ST-Net) is simultaneously Corresponding signal switch target image in pretreated eeg data and data set module is received, back-propagation algorithm is used Own net parameter is updated, realizes the training to signal converting network parameter；Feedback training to user and signal is converted The training of network parameter alternately, finally realizes man-machine coadaptation feedback calibration, and network parameter at this time is fixed ST-Net Parameter；Referring to Fig. 6, specific training process is divided into three steps (Step) progress: Step1 measured's free practice；Step2 measured Mental imagery is completed under briefing and records original eeg data；The training of Step3 machine；When termination machine training by Measured determines that measured converts the difference of the signal switch target image of image and data concentration according to the signal itself judged Different effect determines when terminate the coadaptation feedback calibration process of entire Level2.

Use process has stage (Level): the migration of Level3 identification and use:

The migration of 3 identification of Level and use: it after coadaptation feedback calibration, is added and sentences behind ST-Net submodule Other network layer submodule constitutes signal transformation and differentiates network (STD-Net), and fixed ST-Net parameter, training differentiates network layer, multiple The ST-Net neural network parameter learnt with Level 2 realizes that the high-precision to Mental imagery MI, high stability are classified.

DeCa-BCI of the invention by the study in above three stage, the Synchronous lifting accuracy of people's Mental imagery, Stability；Machine to the mutually coordinated adaptedness of the transformation of Mental imagery EEG signals and discriminating power and brain-computer interface, Finally make DeCa-BCI that Mental imagery EEG signals are obtained with the differentiation performance of accurate stable.

Embodiment 5

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-4 is applied, the general pre-training in 1 stage of Level of the present invention is specifically:

The general pre-training of Level 1 is using the original eeg data and signal switch target image instruction in data set module Practice the network parameter in signal converting network (ST-Net) submodule, wherein the same data acquisition module of the format of original eeg data The offline Mental imagery eeg data format of the measured of block output is identical, for the eeg data being collected in advance；Class label For the classification of Mental imagery；Signal switch target image be and the matched gray level image of class label；The gray level image is used for The training of ST-Net；What the gray level image of signal switch target image was set as being easy to remember facilitates trained simple image again； Cross star, circular image etc., for facilitating training；Measured is it is observed that and remembered signal turn before carrying out pre-training Target image is changed, converts image comparison with signal for subsequent；Pre-processing image data module obtains original from data set module Beginning eeg data, pre-processes it, sends pretreated eeg data to signal converting network (ST-Net) submodule Block, ST-Net network complete a propagated forward, and signal converting network (ST-Net) submodule converts image as defeated using signal Visual feedback module is passed to out.Signal switch target image is then obtained from data set module, uses cross entropy as damage Lose the difference of signal the conversion image and the signal switch target image in data collection module of function measurement ST-Net submodule output It is different, ST-Net network parameter is updated using back-propagation algorithm.

In the present invention ST-Net submodule using signal conversion image as exporting, the output of the ST-Net submodule using Signal conversion image is placed in the clear visible area of measured by visual feedback module, finally by the visual zone of measured It receives.User converts the effect of image by observing each signal, i.e., converts image according to the signal observed and see before The difference size and degree of stability of signal switch target image examined and remembered, judgement terminate the general of 1 stage of Level manually Pre-training process.

Before general pre-training of the invention is using other user's eeg datas of data format isomorphism or this user Issue makes machine end deep learning according to off-line training signal converting network (Signal Transform Network, ST-Net) Algorithm obtains to a certain degree to the initial decoded transform ability of new data.

Embodiment 6

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-5 is applied, the coadaptation feedback calibration in training method level2 stage of the invention specifically includes:

(exercise wheel number is by user's manual termination comprising taking turns man-machine coadaptation training more for 2 coadaptation training process of Level It trains and determines)；Brain wave acquisition equipment is worn in Level 2 stage user's whole process and visually feedback display device is presented Feedback image, according to the prompt at normal form interface carry out alternately training；Wherein the training of every wheel comprising Step1 free practice, The Step2 task imagination collects data and Step3 machine three steps of training, is described below:

In Step1 free practice step, referring to Fig. 6, user starts to carry out free practice under the prompt at normal form interface, Brain electric data collecting module acquires eeg data in real time simultaneously, and referring to Fig. 7, eeg data becomes after pretreatment as signal The input of switching network submodule is presented module such as screen eventually by feedback and exports corresponding signal conversion image；User is logical The Mental imagery mode for right-hand man of the signal conversion image corresponding adjustment itself presented on viewing screen is crossed, judgement is worked as Front signal conversion image and the difference for being before the signal switch target image remembered in level1, then corresponding adjustment itself The mode for right-hand man's Mental imagery, Mental imagery adjustment mode be strengthen feedback signal be allowed to the smaller side of difference The imagination mode of formula variation, avoids the imagination mode of feedback signal difference；Adjustment result is set to convert mesh as close possible to signal is met Logo image realizes feedback adaptation training of the user to ST-Net neural network algorithm, and training duration is by user's manual termination It trains and determines.

The Step2 task imagination is collected in data step, and user completes Mental imagery according to the prompt information at normal form interface Such as right-hand man's Mental imagery, for Mental imagery feedback interface referring to Fig. 8, the brain electricity generated when the imagination every time all can be by brain electricity number It is recorded according to acquisition module, while recording class label of the prompt information as eeg data, the eeg data and classification mark of record Label are for the training of subsequent machine；When normal form interface display starts machine training, show to complete for ST-Net net machine The collection of the training data of algorithm.

In Step3 machine training step, corresponding letter is concentrated using collected eeg data in step Step2 and data Number switch target image training ST-Net, makes ST-Net study to the signal converting network parameter for being more suitable for current data, together When, live effect is provided referring to Fig. 9 machine training step and is checked, and determines when tie according to real-time training effect by user The training of Shu Benci ST-Net.

Mental imagery brain-computer interface (MI-BCI) can not be solved by developing decoding algorithm merely, and need to put into people BCI system takes in and studies.Man-machine coadaptation Mental imagery brain-computer interface based on deep learning of the invention (DeCa-BCI) training process uses the Real-time Feedback in " people is in circuit ", makes user's acquistion in trial and error correct and steady Fixed decodable Mental imagery mode；Replace training of human and machine in a manner of visual feedback, makes the EEG signals and machine of people The feedback loop of the signal transformation results composition of device is alternately trained under same imagination task, and stable convergence, to realize people Machine coadaptation training.

Embodiment 7

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-6 is applied, the step1 free practice step in 2 coadaptation feedback calibration stage of Level:

User completes the adaptation training to ST-Net machine algorithm in the case where there is feedback condition, and data acquisition module leads in real time The original eeg data that brain wave acquisition equipment obtains user is crossed, 500 milliseconds of periods of newest acquisition are intercepted in a manner of sliding window Acquisition data as original eeg data, original head data feeding data preprocessing module is obtained into pretreated brain Electric data transmit to obtain signal conversion image and send that module, vision is presented to visual feedback by the forward direction of ST-Net network Feedback is presented module and signal conversion image is presented to user by feeding back the screen of display device, and every 50~100 milliseconds complete At primary feedback, user judges that current demand signal conversion image and signal are converted by the feedback image presented on viewing screen The difference and degree of stability of target image, the mode of the Mental imagery of corresponding adjustment itself, Mental imagery adjustment mode is to strengthen Feedback signal is allowed to the imagination mode changed to the smaller mode of difference, avoids the imagination mode of feedback signal difference；Tie adjustment Fruit executes as close possible to signal switch target image, entire feedback procedure circulation is met, and realizes user to ST-Net nerve The feedback adaptation training of network algorithm, training duration are trained by user's manual termination and are determined.

Step1 free practice step of the invention is effectively prevented by allowing user to carry out free practice in advance in MI- In the research of BCI technology, the unstable imagination, the mistake imagination or even a part of measured (15% of the measured often occurred ~30%) SMR can not be detected at all, while introducing feedback, to provide more natural brain-machine interaction body for user It tests.Solving leads to the experiment of MI-BCI since user lacks understanding and use experience to the cerebration mechanism of MI-BCI The problem that room result difference is huge and whole ITR is lower.

Embodiment 8

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-7 is applied, the step2 task imagination in level2 coadaptation feedback calibration stage collects data step:

User completes phase according to the briefing on the feedback display device of normal form interface in the case where there is feedback condition The Mental imagery answered is by taking right-hand man's Mental imagery as an example, referring to Fig. 8, i.e. the user when prompting arrow to appear on the left of screen Left hand Mental imagery is carried out, when arrow is appeared on the right side of screen, carries out right hand Mental imagery；Mental imagery is carried out in user While, brain wave acquisition equipment records Mental imagery eeg data, and the eeg data acquired using briefing as this Label, finally obtain the Mental imagery eeg data with label；A Mental imagery task is completed before the deadline Data acquisition, every motion imagination job order time data acquisition time is 15 seconds, and every type acquires one or many.

The step2 task imagination of the invention collects data step, and user sets in the case where there is feedback condition according to normal form interface Briefing on the feedback display device of meter completes corresponding Mental imagery and finally obtains the Mental imagery brain electricity with label Data, this eeg data effectively prevent grinding in MI-BCI technology compared with the eeg data that traditional acquisition mode acquires In studying carefully, the unstable imagination, the mistake imagination or even a part of measured (15%~30%) of the measured often occurred is at all The problems such as can not detecting sensorimotor rhythm (SMR) SMR.Feedback is introduced simultaneously, to provide more natural brain machine for user Interactive experience.

Embodiment 9

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-8 is applied, the step3 machine training step in level2 coadaptation feedback calibration stage:

Become using the collected tape label data training eeg data of step2 to the signal between signal switch target image Switching network (ST-Net) parameter.In this step, machine complete for several times train after, randomly selected from training set one group of data into To transmitting before row, and consequential signal is converted into image and is presented to user by feeding back display device, task imagines feedback interface Referring to Fig. 9, realize user to the real time inspection of machine training effect；The readability and degree of stability of signal conversion image The performance of ST-Net is reflected, user determines when terminate the ST-Net training of this next round according to training effect.

Machine training of the invention has used the collected tape label data training ST-Net network parameter of step2, due to The data of the collected tape label of step2 are collected under coadaptation feedback condition, the acquisition sides of data and traditional data Formula is compared, and no matter is all greatly improved in accuracy or in terms of stability.Input data is ensured that in this way It is also more preferable to train the network effect come in this way for high quality.

Embodiment 10

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-9 is applied, the migration of network identification and use in the level3 stage:

Identification transition process implementation is to be multiplexed trained ST-Net network parameter, is added differentiates network behind Layer forms STD-Net, and training differentiates network layer parameter, differentiates that network layer is one convolution of addition after the output layer of ST-Net Core size is the convolutional layer such as 64*64 of C*C, is completed using an individual convolution kernel from feedback image to identification result Conversion.The training method of the convolution kernel is the fixed part ST-Net, uses the output result and classification Onehot label of ST-Net The parameter of the training convolution kernel.Changing image is completed to the conversion for differentiating result, most with the wide convolution kernel of result using one Limits balance the stability and flexibility of migration.The ability for making STD-Net have output Mental imagery classification results；Sentence Other network layer receives the output of ST-Net as a result, obtaining corresponding class label from data set simultaneously, is calculated using backpropagation Method completes the training to network layer convolution nuclear parameter is differentiated；

Referring to Figure 10, the use process of network identification migration is data collecting module collected user eeg data, warp Data preprocessing module is crossed to handle to obtain pretreated eeg data and send STD-Net module to and obtain EEG signals Classification results, classification results can be used for the interaction of brain and computer or the control to peripheral equipment.

The network identification in level3 stage of the invention migrates and use carries out after coadaptation feedback calibration, this When can pole easily by ST-Net transfer learning be signal transformation differentiate network (Signal Transformdiscrimination Network, STD-Net).The neural network parameter that multiplexing Level 2 learns, most The high-precision to MI EEG signals, high stability classification are realized eventually.

It is given below one system and method combine together, the more detailed example with concrete operations, to the present invention And its effect further illustrates.

Embodiment 11

Man-machine coadaptation Mental imagery brain machine interface system (DeCa-BCI) and training method based on deep learning are the same as real A 1-10 is applied, referring to Fig. 5, the specific steps are as follows:

Step 1, deployment facility acquisition module

(1a) installs brain wave acquisition device electrode, and sample rate is arranged；64 electrodes are used in this example, and sample rate is set 2048Hz。

(1b) user wears electrode cap, places acquisition electrode according to international standard 10-20 system.

Measured's brain electric data collecting module is formd after being ready to complete.The brain of brain electric data collecting module reception measured Electric signal completes data acquisition of the user during imagination, the data collected with fixed sample rate and distribution of electrodes For eeg data, eeg data is transferred to data preprocessing module；The present invention design do not limit EEG signal acquisition equipment and Lead number, i.e. brain wave acquisition equipment can be arranged according to demand, or for wired brain wave acquisition equipment or be that wireless brain wave acquisition is set It is standby；Minimum lead number need to be greater than 1 lead.

Step 2 establishes data preprocessing module

(2a) goes mean value, subtracts all electrode EEG signals with the EEG signals that electrode each on measured's electrode cap acquires Mean value, the EEG signals after obtaining baseline correction.

(2b) filtering will be removed absolutely by step (2a) treated EEG signals by the bandpass filter of 1-75Hz Most of interference physiological signals, and 50Hz power frequency component is carried out to it and is filtered out.

(2c) is down-sampled, and down-sampled to the lower 200Hz of the result that (2b) is obtained obtains pretreated EEG signals.

Step 3, building signal transformation differentiate network (STD-Net) module

(3a) constructs ST-Net network

ST-Net network is made of five layers of convolutional neural networks, and convolution kernel size is N*N, such as N=16, is respectively provided with Activation primitive ReLU；The convolution kernel number of first layer is M, such as M=32, and second layer convolution kernel number is 0.5*M, third layer Convolution kernel number is 0.5*M, and the 4th layer of convolution kernel number is 0.25*M, and layer 5 convolution kernel number is 1, all layers of convolution step Length is 1；Every layer of output is all the characteristic pattern with input with size, such as time (T) * lead (C) matrix of input is T =128, C=64, then exporting result size is also 128*64.

(3b) building differentiates network layer

Differentiation network layer is that a convolution kernel size being added after the output layer of ST-Net is the convolutional layer of C*C, is used An individual convolution kernel is completed to convert the conversion of image to identification result from signal；The training method of the convolution kernel is to make Use the output result of ST-Net as the input for differentiating network layer, the fixed part ST-Net is made using Mental imagery class label For the parameter of the target value training convolution kernel；Changing image is completed to differentiation result with the wide convolution kernel of result using one Conversion, balance the stability and flexibility of migration to greatest extent.

Step 4 establishes visual feedback presentation module

(4a) writes visual feedback using the tool box Psychtoolbox and interface is presented.The visual feedback interface of DeCa-BCI It is the signal conversion output result of ST-Net.

Visual feedback is presented using liquid crystal display in (4b).ST-Net inputs collected 64 channel * 128 and samples (0.5s) Eeg data matrix, the transformation result image of output is presented in real time by visual feedback display device liquid crystal display Subject, the every 50ms of the process carry out primary.

Step 5, building data set

(5a) original eeg data, the output of the data format and this system brain electric data collecting module of original eeg data It is identical

(5b) signal switch target image, signal switch target image are consistent containing strong identification letter with class label The gray level image of breath, such as defining the signal switch target image of right-hand man's Mental imagery task is the ash of left or right cross star Spend image, Aspect Ratio 2:1, black background and the png format-pattern with unilateral white crosses.

(5c) class label, class label are the classification of Mental imagery.

Step 6 carries out signal converting network (ST-Net) pre-training

(6a) obtains original eeg data and the pretreatment Jing Guo data preprocessing module from data set

(6b) sends pretreated eeg data to signal converting network (ST-Net) submodule, and ST-Net network is complete Cheng Yici propagated forward.

(6c) obtains matched signal switch target image from data set.

(6d) uses cross entropy as the signal conversion image and data collection of loss function measurement ST-Net submodule output In signal switch target image difference.

(6e) updates ST-Net network parameter using back-propagation algorithm.

(6f) user converts the effect of image by observing each signal, i.e., according to the signal conversion image observed with The difference size and degree of stability for the signal switch target image observed and remembered before, judgement terminate the Level1 stage manually General pre-training process.

Step 7, man-machine coadaptation training

(7a) Step1 free practice

(7a1) user starts to carry out free practice under the prompt at normal form interface, while brain electric data collecting module is real When acquire eeg data,

The input as signal converting network submodule after pretreatment of (7a2) eeg data is in eventually by feedback Existing module such as screen exports corresponding signal and converts image；In right-hand man's Mental imagery mode as an example, user passes through film viewing screen The Mental imagery mode for right-hand man of the signal conversion image corresponding adjustment itself presented on curtain, judges that current demand signal turns Change the difference for the signal switch target image remembered in image and i.e. level1 before, then it is corresponding adjust itself for a left side The mode of right hand Mental imagery, Mental imagery adjustment mode strengthen feedback signal and are allowed to change to the smaller mode of difference Imagination mode avoids the imagination mode of feedback signal difference；

(7a3) makes to adjust result as close possible to signal switch target image is met, and realizes user to ST-Net nerve The feedback adaptation training of network algorithm, training duration are trained by user's manual termination and are determined；

(7b) Step2 task imagination collects data

(7b1) user completes Mental imagery such as right-hand man's Mental imagery according to the prompt information at normal form interface

The brain electricity that (7b2) is generated when imagining every time can all be recorded by brain electric data collecting module, while record prompt information As the class label of eeg data, the eeg data and class label of record are for the training of subsequent machine；

(7b3) shows to complete the instruction for ST-Net net machine algorithm when normal form interface display starts machine training Practice the collection of data；

The training of (7c) Step3 machine

(7c1) concentrates corresponding signal switch target image using collected eeg data in step Step2 and data Training ST-Net learns ST-Net to the signal converting network parameter for being more suitable for current data,

(7c2) simultaneously, machine training step provides live effect and checks, and is determined by user according to real-time training effect When the training of this ST-Net is terminated.

Step 8, the migration of the identification of ST-Net

(8a) is multiplexed trained ST-Net network parameter, is added differentiates network layer behind, forms STD-Net, training Differentiate network layer parameter, the ability for making STD-Net have output Mental imagery classification results.

(8b) differentiates that network layer receives the output of ST-Net as a result, obtain corresponding class label from data set simultaneously, The training to network layer convolution nuclear parameter is differentiated is completed using back-propagation algorithm.

The use of step 9, STD-Net network

(9a) removes system service stage unnecessary module.

The classification results that (9b) user is exported by Mental imagery control system, and then realize the control to external equipment And interaction.

Of the invention man-machine coadaptation Mental imagery brain machine interface system and training method based on deep learning solves For the problems such as difference on effect is big, stability is poor between the generally existing user of Mental imagery brain-computer interface, using " people exists The Real-time Feedback in circuit ", while being that sorter network realizes personalized high-precision MI-BCI, clear process by transfer learning It is clear, it is easy to apply under several scenes.And the present invention supports to use trained STD-Net net under offline and presence Network has expanded application scenarios and the space of Mental imagery brain-computer interface.

The training process of DeCa-BCI of the invention uses the Real-time Feedback in " people is in circuit ", allows user in trial and error The correct and stable decodable Mental imagery mode of acquistion；Design is gradually allowed using DNN powerful ability in feature extraction simultaneously DNN further learn to the Mental imagery feature for being suitble to user, be that sorter network realizes personalized by transfer learning High-precision MI-BCI has effectively helped different types of user to improve the decoding efficiency of Mental imagery, has pushed MI-BCI Further functionization, expanded the application space of MI-BCI significantly.For lifting motion the imagination brain-computer interface performance and It realizes and classifies to the high-accuracy stable of user's Mental imagery.

In conclusion the man-machine coadaptation Mental imagery brain machine interface system and instruction disclosed by the invention based on deep learning Practice method, solves that difference on effect between the generally existing different users of Mental imagery brain-computer interface is big, EEG signals stability The technical problems such as difference.System is connected with the brain electric data collecting module for being connected to user, data preprocessing module, letter in turn Number transformation differentiates that module is presented in network (STD-Net) module and visual feedback, and visual feedback is presented module and is set by feeding back to present It is standby to provide visual information feedback to measured such as screen, by the visual reception of measured, through measured's active accommodation Mental imagery Brain electric data collecting again is formed after mode；And data set module provides original eeg data, signal switch target image and Class label.During feedback training, ST-Net submodule is presented module for visual feedback and provides signal conversion image；In In identification migration and use process, ST-Net submodule converts image to differentiate that network layer submodule provides signal, through differentiating The classification results of EEG signals are obtained after the processing of network layer submodule.Method includes that step includes trained and use process: The general pre-training of Level1,2 coadaptation feedback calibration of Level, Level 3 are used online；Wherein Level2 coadaptation training Comprising taking turns man-machine coadaptation training (wheel number is trained by user's manual termination and determined), wherein every wheel training includes process more Step1 free practice, the Step2 task imagination collect data and Step3 machine three steps of training.DeCa-BCI's of the invention Training process uses the Real-time Feedback in " people is in circuit ", allows correct and stable decodable of user's acquistion in trial and error Mental imagery mode；The DNN of design is gradually allowed further to learn to being suitble to make using DNN powerful ability in feature extraction simultaneously The Mental imagery feature of user is that sorter network realizes personalized high-precision MI-BCI by transfer learning, effectively helps Different types of user improves the decoding efficiency of Mental imagery, has pushed the further functionization of MI-BCI, has expanded significantly The application space of MI-BCI.For the performance of lifting motion imagination brain-computer interface and realization to user's Mental imagery High-accuracy stable classification.

Above description is only specific embodiments of the present invention, does not constitute any limitation of the invention.Obviously for this It, all may be without departing substantially from the principle of the invention, structure after having understood the content of present invention and principle for the professional in field In the case where, carry out various modifications and variations in form and details, but these modifications and variations based on inventive concept Still within the scope of the claims of the present invention.

Claims (10)

1. a kind of man-machine coadaptation Mental imagery brain machine interface system based on deep learning, which is characterized in that at information Make sequence in order, be sequentially connected and include brain electric data collecting module, data preprocessing module, signal transformation differentiate network module and Module is presented in visual feedback, and visual feedback is presented module and provides visual information feedback to measured by feedback display device, by The visual reception of measured forms brain electric data collecting again after measured's active accommodation Mental imagery mode；The letter Number transformation differentiate network module, wherein be sequentially connected and include signal converting network submodule and differentiate network layer submodule； System further includes having data set module, and data set module includes original eeg data, signal switch target image and classification mark Label, data set module and data preprocessing module carry out bidirectional data interaction and take out original eeg data；Data set module and letter Number converting network submodule bidirectional data interaction takes out signal switch target image；Data set module and differentiation network layer submodule It carries out bidirectional data interaction and takes out class label；Above-mentioned two-way interactive includes the transmission of data and the transmission for controlling signal；In During feedback training, ST-Net submodule is presented module for visual feedback and provides signal conversion image；Identification migration and In use process, ST-Net submodule converts image to differentiate that network layer submodule provides signal, through differentiating network layer submodule The classification results of EEG signals are obtained after processing；Each module is described below:

Brain electric data collecting module is made of brain wave acquisition equipment；The brain telecommunications of brain electric data collecting module reception measured Number data acquisition of the measured during imagination is completed with fixed sample rate and distribution of electrodes, the data collected are brain electricity Eeg data is transferred to data preprocessing module by data；Man-machine coadaptation Mental imagery brain-computer interface based on deep learning System design does not limit brain electricity EEG signal acquisition equipment and lead number, i.e. brain wave acquisition equipment are arranged according to demand, or to have Line brain wave acquisition equipment is wireless brain wave acquisition equipment；Minimum lead number need to be greater than 1 lead；

Data preprocessing module: the original brain electricity in brain electric data collecting module eeg data collected or data set is received Data successively carry out baseline to the eeg data received, and filtering goes power frequency, down-sampled pretreatment obtains pretreated Eeg data, and send pretreated eeg data to signal converting network submodule in signal transformation differentiation network module Block；

Signal transformation differentiates network module, including signal converting network submodule and differentiation network layer submodule；In ST-Net In the propagated forward use process of module: ST-Net submodule receives the processing result of data preprocessing module, and output, which has, to be sentenced The signal of other information converts image；Differentiate that network layer submodule receives the signal conversion image of ST-Net submodule output, obtains Classification results；In the backpropagation training process of ST-Net submodule: eeg data transmits after preprocessing module is handled ST-Net submodule is given, while ST-Net submodule obtains the signal switch target image of corresponding classification from data set module, And back-propagation algorithm training ST-Net, eeg data or original in data set module are executed as target image Eeg data or the eeg data acquired in real time；Differentiate that network layer submodule receives the signal conversion of ST-Net submodule output Image, while class label is obtained from data set module, and execute back-propagation algorithm training as label and differentiate network Layer；

Module is presented in visual feedback, and the signal for receiving ST-Net submodule output in STD-Net module converts image, and image is led to It crosses feedback display device and is presented to measured；The signal switch target image that measured converts image according to signal and data are concentrated Difference, active accommodation Mental imagery mode, make signal conversion image develop towards steady and audible direction, realize to itself brain The feedback regulation of electric signal.

2. the man-machine coadaptation Mental imagery brain machine interface system according to claim 1 based on deep learning, feature It is, the building of STD-Net module:

STD-Net module is divided into ST-Net submodule and differentiates that network layer submodule, ST-Net submodule contain ST-Net net Network differentiates that network layer submodule contains differentiation network layer；ST-Net submodule is realized brain electricity EEG signal from time T* lead The dimensional matrix data of C is transformed to signal relevant to the Mental imagery task type conversion that a user can intuitively recognize Image, it is the gray level image with strong identification information which, which converts image, so that measured carries out feedback regulation；Differentiate net The signal that network layers submodule receives the output of ST-Net submodule converts image, and propagated forward obtains the classification results of Mental imagery；

ST-Net network is made of five layers of convolutional neural networks, and convolution kernel size is N*N, such as N=16, is respectively provided with activation Function ReLU；The convolution kernel number of first layer is M, such as M=32, and second layer convolution kernel number is 0.5*M, third layer convolution kernel Number is 0.5*M, and the 4th layer of convolution kernel number is 0.25*M, and layer 5 convolution kernel number is 1, and all layers of convolution step-length are 1； Every layer of output is all the feature with input with size.

Differentiation network layer is that a convolution kernel size being added after the output layer of ST-Net is the convolutional layer of C*C, using independent A convolution kernel complete from signal convert image to identification result conversion；

ST-Net network using and training: signal converting network receives pretreated eeg data in use and goes forward side by side Row propagated forward obtains signal conversion image；Signal converting network receives pretreated eeg data simultaneously in the training process Corresponding signal switch target image is concentrated with data, uses cross entropy as the signal conversion of loss function comparison network output Image and signal switch target image, and back transfer algorithm is executed to train ST-Net itself；

Differentiate using and training for network layer: differentiating that network layer receives ST-Net submodule propagated forward in use and obtains Signal convert image, propagated forward obtains the classification results of Mental imagery；Differentiate that network layer receives simultaneously in the training process Corresponding class label in the signal conversion image and data collection module that ST-Net submodule propagated forward obtains, uses cross entropy Loss function comparison differentiates the classification results and class label of network layer output, and differentiates network using the training of back transfer algorithm Layer.

3. the man-machine coadaptation Mental imagery brain machine interface system according to claim 1 based on deep learning, feature It is, the building of data set module:

Original eeg data and matching signal switch target image and class label are contained in data set module；It is former The data format of beginning eeg data and the output phase of this system brain electric data collecting module are same；Class label is the class of Mental imagery Not；Signal switch target image be and the consistent gray level image containing strong identification information of class label.

4. a kind of man-machine coadaptation Mental imagery brain-computer interface training method based on signal transformation, described in claim 1-3 The man-machine coadaptation Mental imagery brain machine interface system based on deep learning on realize, which is characterized in that include to train Journey and use process:

Successively there are two the general pre-training of stage Level:Level 1,2 coadaptation feedback calibration of Level for training process:

The general pre-training of Level 1: using the data training signal converting network in data set module, ST-Net network completes one Secondary forward direction transmits to obtain output of the signal conversion image as ST-Net submodule；ST-Net net is updated using back-propagation algorithm Network parameter makes ST-Net network submodular obtain the initial decoded transform ability to new data；

2 coadaptation feedback calibration of Level: visual feedback is presented the signal conversion image that module is exported with ST-Net submodule and makees It is presented to measured, by the visual reception of measured, through quilt by the visual information of feedback by feedback display device for input Survey person's active accommodation Mental imagery mode, forms brain electric data collecting again, carrys out iterative cycles with this and carries out, realizes to tested The feedback training of person；Signal converting network receives corresponding signal in pretreated eeg data and data set module simultaneously and turns Target image is changed, own net parameter is updated using back-propagation algorithm, realizes the training to signal converting network parameter；To making The feedback training of user and to the training of signal converting network parameter alternately, finally realize man-machine coadaptation feedback calibration, Network parameter at this time is fixed ST-Net parameter；Specific training process is divided into three step Step progress: Step1 measured freely practices It practises；Step2 measured completes Mental imagery under briefing and records original eeg data；The training of Step3 machine；When eventually Only machine training is determined that measured converts mesh according to the signal that the signal conversion image and data that itself judge are concentrated by measured The differential effect of logo image determines when terminate the coadaptation feedback calibration process of entire Level2；

Use process has a stage: the migration of 3 identification of Level and use:

The migration of 3 identification of Level and use: it after coadaptation feedback calibration, is added behind ST-Net submodule and differentiates net Network layers submodule constitutes signal transformation and differentiates network, fixed ST-Net parameter, and training differentiates network layer, and multiplexing Level 2 learns The ST-Net neural network parameter arrived realizes that the high-precision to Mental imagery MI, high stability are classified.

5. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 4 based on signal transformation, It is characterized in that, the general pre-training in 1 stage of Level:

The general pre-training of Level 1 is using the original eeg data and signal switch target image training letter in data set module Network parameter in number converting network submodule, wherein the measured that the format of original eeg data is exported with data acquisition module Offline Mental imagery eeg data format it is identical, for the eeg data being collected in advance；Class label is the class of Mental imagery Not；Signal switch target image be and the matched gray level image of class label；The gray level image is used for the training of ST-Net；Signal What the gray level image of switch target image was set as being easy to remember facilitates trained simple image again；The quilt before carrying out pre-training Survey person is it is observed that and remembered signal switch target image；Pre-processing image data module obtains original from data set module Eeg data pre-processes it, sends pretreated eeg data to signal converting network submodule, ST-Net net Network completes a propagated forward, and signal converting network submodule converts image as output using signal and passes to visual feedback mould Block；Signal switch target image is then obtained from data set module, uses cross entropy as loss function measurement ST-Net The difference of signal the conversion image and the signal switch target image in data collection module of module output, uses back-propagation algorithm Update ST-Net network parameter.

6. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 4 based on signal transformation, It is characterized in that, the coadaptation feedback calibration in 2 stage of level:

2 coadaptation training process of Level includes to take turns man-machine coadaptation training more；Brain is worn in Level 2 stage user's whole process Electricity acquisition equipment and the feedback image that visually feedback display device is presented, carry out alternately training according to the prompt at normal form interface； Wherein every wheel training collects data and Step3 machine three steps of training comprising Step1 free practice, the Step2 task imagination, point It states as follows；

In Step1 free practice step, user starts to carry out free practice, while eeg data under the prompt at normal form interface Acquisition module acquires eeg data in real time, input of the eeg data after pretreatment as signal converting network submodule, most The module such as corresponding signal of screen output is presented by feedback eventually and converts image；User passes through the signal presented on viewing screen Image is converted, then the mode of the Mental imagery of corresponding adjustment itself, Mental imagery adjustment mode is to strengthen feedback signal to be allowed to The imagination mode changed to the smaller mode of difference, avoids the imagination mode of feedback signal difference；Make adjust result as close possible to Meet signal switch target image, realize feedback adaptation training of the user to ST-Net neural network algorithm, training duration by User's manual termination is trained and determines；

The Step2 task imagination is collected in data step, and user completes Mental imagery for example according to the prompt information at normal form interface Right-hand man's Mental imagery, the brain electricity generated when the imagination every time can all be recorded by brain electric data collecting module, while record prompt letter The class label as eeg data is ceased, the eeg data and class label of record are for the training of subsequent machine；When normal form circle When face display starts machine training, show the collection for completing the training data for ST-Net net machine algorithm；

In Step3 machine training step, corresponding signal is concentrated to turn using collected eeg data in step Step2 and data Target image training ST-Net is changed, makes ST-Net study to the signal converting network parameter for being more suitable for current data, meanwhile, machine Training step provides live effect and checks, and determines according to real-time training effect by user when to terminate the instruction of this ST-Net Practice.

7. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 6 based on signal transformation, It is characterized in that, the step1 free practice step in level2 coadaptation feedback calibration stage:

User completes the adaptation training to ST-Net machine algorithm in the case where there is feedback condition, and data acquisition module passes through brain in real time Electricity acquisition equipment obtains the original eeg data of user, and adopting for 500 milliseconds of periods of newest acquisition is intercepted in a manner of sliding window Collect data as original eeg data, original head data feeding data preprocessing module is obtained into pretreated brain electricity number According to transmitting to obtain signal conversion image by the forward direction of ST-Net network and send module, visual feedback is presented to visual feedback Module is presented, signal conversion image is presented to user by feeding back the screen of display device, every 50~100 milliseconds are completed one Secondary feedback, user judge current demand signal conversion image and signal switch target by the feedback image presented on viewing screen The difference and degree of stability of image, the mode of the Mental imagery of corresponding adjustment itself, Mental imagery adjustment mode is to strengthen feedback Signal is allowed to the imagination mode changed to the smaller mode of difference, avoids the imagination mode of feedback signal difference；Keep adjustment result most It may be close to and meet signal switch target image, entire feedback procedure circulation executes, and realizes user to ST-Net neural network The feedback adaptation training of algorithm, training duration are trained by user's manual termination and are determined.

8. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 6 based on deep learning, It is characterized in that, the step2 task imagination in level2 coadaptation feedback calibration stage collects data:

User completes in the case where there is feedback condition according to the briefing on the feedback display device of normal form interface corresponding Mental imagery is by taking right-hand man's Mental imagery as an example, i.e., when prompting arrow to appear on the left of screen, user's progress left hand movement is thought As carrying out the right hand Mental imagery when arrow is appeared on the right side of screen；While user carries out Mental imagery, brain wave acquisition Equipment records Mental imagery eeg data, and using briefing as the label of this eeg data acquired, finally obtains Mental imagery eeg data with label；A Mental imagery task data acquisition, every type are completed before the deadline Mental imagery job order time data acquisition time is 15 seconds, and every type acquires one or many.

9. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 6 based on deep learning, It is characterized in that, the step3 machine training step in level2 coadaptation feedback calibration stage:

Using the collected tape label data training signal converting network parameter of step2, in this step, machine completion is trained for several times Afterwards, it is randomly selected from training set before one group of data carries out to transmitting, and obtained signal conversion image is presented by feedback Equipment is presented to user, realizes user to the real time inspection of machine training effect；Signal convert image readability and Degree of stability reflects the performance of ST-Net, and user determines when terminate the ST-Net training of this next round according to training effect.

10. the man-machine coadaptation Mental imagery brain-computer interface training method according to claim 4 based on deep learning, It is characterized in that, the migration of network identification and use in the level3 stage:

Identification transition process implementation is to be multiplexed trained ST-Net network parameter, is added differentiates network layer behind, STD-Net is formed, training differentiates network layer parameter, the ability for making STD-Net have output Mental imagery classification results；Differentiate net Network layers receive the output of ST-Net as a result, obtaining corresponding class label from data set module simultaneously, are calculated using backpropagation Method completes the training to network layer convolution nuclear parameter is differentiated；

Use process is data collecting module collected user eeg data, handles and is pre-processed by data preprocessing module Rear eeg data simultaneously sends STD-Net module to and obtains the classification results of EEG signals, classification results can be used for brain with The interaction of computer or control to peripheral equipment.