CN103699216A - Email communication system and method based on motor imagery and visual attention mixed brain-computer interface - Google Patents

Abstract

The invention discloses an email communication system and method based on a motor imagery and visual attention mixed brain-computer interface. The system comprises a visual stimulator, an electroencephalogram acquisition platform, a system control unit, a data processing module, a cursor control module and a character input module, wherein the visual stimulator comprises an LCD (Liquid Crystal Display)-based visual stimulator and an LED (Light-Emitting Diode)-based visual stimulator; the LCD-based visual stimulator is provided for a user in an interface display way, and the LED-based visual stimulator is implemented through a hardware circuit. According to the method, the motion control and target selection of a two-dimensional cursor are realized by simulating a computer mouse through the mixed brain-computer interface based on motor imagery and LCD visual attention, and input of email addresses and contents is realized by simulating a computer keyboard through the brain-computer interface based on LED visual attention; the working modes of the mouse and the keyboard are switched through a stimulation key in an LCD simulator. The system has the advantages of high cursor motion control accuracy, high robustness, high character input accuracy and the like.

Description

A kind of E-mail communication system and method based on the motion imagination and vision attention mixing brain-computer interface

Technical field

The present invention relates to brain-computer interface technology and the communication technology, more particularly, the present invention relates to a kind of E-mail communication system and method based on the motion imagination and vision attention mixing brain-computer interface.

Background technology

Human brain is communicated by letter or controls external environment condition with extraneous by neuromuscular channel, and numerous disease may destroy these neuromuscular channels.Neural channel or the infringement muscle itself of for example, muscular dystrophy lateral sclerosis, brain stem apoplexy, spinal cord injury, brain paralysis, op parkinson's, multiple hardening illness and other numerous diseases meeting damage control muscle.Those people that are subject to these sickness influences may lose autonomous muscle and control, thereby produce dyskinesia, to such an extent as to can not communicate or control with the external world external unit.

The cerebration of brain-computer interface (Brain-Computer Interface, BCI) monitor user ', understands user's intention, and user's intention is converted to external command.As a kind of communication port new, non-muscle, BCI can make people directly by brain, express thoughts or maneuvering device, and need to be by language or limb action.For serious motion disabled patient, BCI can be sent to external device (ED) by their intention, such as computing machine, household electrical appliance, care appliances and nerve prosthesis etc., thereby improves their quality of life.

Different brain electricity (Electroencephalography, EEG) component of signal, for example slow cortical potential, the Mu/Beta rhythm and pace of moving things, the relevant P300 current potential of event and VEP (Visual Evoked Potential, VEP), can be as the characteristic signal of BCI.

The motion imagination is a kind of important BCI implementation pattern.Electrophysiologic studies shows, when a people carries out or imagines certain motion, at the power of the Mu/Beta of its brain specific region rhythm and pace of moving things signal, can decline, and part relevant desynchronize (Event-Related Desynchronization, ERD) while being called; When Motor execution or the motion imagination finish, at the power of the Mu/Beta in its specific region of brain rhythm and pace of moving things signal, can rise, be called event related synchronization (Event-Related Synchronization, ERS).The Motor execution of different limbs or the motion imagination cause that the power of the Mu/Beta rhythm and pace of moving things of zones of different changes.BCI can differentiate the different imagination task of user according to this variation, thereby determines user's intention, and this intention is converted into the control command of external unit.

Vision attention is another kind of important BCI implementation pattern.Vision attention can produce in brain specific region VEP (Visual Evoked Potential, VEP).VEP has reflected the Vision information processing mechanism of brain, is a kind of response of human eye to flash stimulation.Different according to the frequency of repetitive stimulation, VEP can be divided into transient state VEP and stable state VEP.When the repetition frequency of visual stimulus is during higher than 6Hz, the response meeting superposition of continuous Induced by Stimulation several times together, Cortical Neurons granting is synchronizeed with frequency of stimulation, the rhythm and pace of moving things in brain occipitalia region electrical activity of brain obviously strengthens, form a kind of stable response, be called stable state VEP(Stedy-State VEP, SSVEP).SSVEP has the fundamental frequency identical with visual stimulus frequency, and its fundamental frequency can detect with specific signal processing algorithm, thereby determines user's intention.

Communication is the important component part that people live, and phone and Email are two kinds of main communication modes.E-mail communication comprises collects mail and two aspects of writing letter.The collection of letters needs user to move and control and target is selected two dimensional cursor; Write letter and not only need user to carry out cursor control and target selection, and need user can carry out character input.For the normal user of upper limbs, these operations are easy to realize by the mouse of computing machine and keyboard.Yet, for the user of upper limb disability, under the condition of not using mouse and keyboard, how to realize cursor movement control, target selection and character input, thereby realize E-mail communication, be a major issue urgently to be resolved hurrily.

Two dimensional cursor motion control need two independently control variable respectively, independent, control cursor motion in the horizontal and vertical directions simultaneously, and the motion of this two dimensional cursor must be continuous, cursor could be moved to any one target location from any one initial position.Mixing BCI based on the motion imagination and vision attention can provide two independently control variable simultaneously, and imagines that based on motion the output variable of BCI is continuous or simulating signal, thereby has guaranteed the continuity of two dimensional cursor motion.In addition, visual stimulus signal can be produced by liquid crystal display (Liquid Crystal Display, LCD), also can be produced by light emitting diode (Light Emitting Diode, LED) display.Owing to being subject to the restriction of screen refresh rate, the visual stimulator based on LCD can only provide a few frequency of stimulation, can be used for controlling cursor motion in the vertical direction; LED-based visual stimulator can provide much tens frequency of stimulation, can be used for character input.

Also do not have at present to find do not using under the condition of computer mouse or keyboard, use EEG signals to realize the technology of E-mail communication.

Summary of the invention

The object of the invention is to propose a kind of E-mail communication system and method based on the motion imagination and vision attention mixing brain-computer interface.

The technical problem to be solved in the present invention is that E-mail communication comprises collects mail and two aspects of writing letter.The collection of letters requires user can two dimensional cursor motion be controlled and target or interested icon are selected; Write letter and not only require user can carry out cursor control and target selection, and require user can carry out character input.For the user of two upper limb disabilities, under the condition of not using mouse and keyboard, how to realize cursor movement control, target selection and character input, thereby realize E-mail communication.

The present invention is achieved by the following technical solutions.

One aspect of the present invention has been to provide a kind of E-mail communication system based on the motion imagination and vision attention mixing brain-computer interface, and this system comprises visual stimulator, brain wave acquisition platform, system control unit, data processing module, cursor control module and character load module.Wherein system control unit, data processing module, cursor control module and character load module are realized by software in computing machine.Wherein visual stimulator is used for bringing out Steady State Visual Evoked Potential (SSVEP) signal, comprise based on liquid crystal display (Liquid Crystal Display, LCD) visual stimulator with based on light emitting diode (Light Emitting Diode, LED) display.LCD visual stimulator offers user in interface display mode, and LED visual stimulator is realized by hardware circuit; Brain wave acquisition platform comprises electrode cap and electroencephalogramdata data collector.Electrode cap is used for gathering EEG signals, electroencephalogramdata data collector for the EEG signals gathering is amplified, filtering and analog to digital conversion, and digitized EEG signals is inputted to computing machine by data line; System control unit receives, preserves eeg data, and system works interface is provided, and controls the stimulus key of LCD visual stimulator with the frequency flash of light of setting; Data processing module carries out respectively real-time pre-service, feature extraction and Classification and Identification to motion imagination eeg data and vision attention eeg data; Cursor control module is calculated cursor in the displacement of horizontal and vertical direction according to the result of Classification and Identification, controls cursor and carries out continuous two dimensional motion, and interested target is selected; Character load module starts LED visual stimulator, determines the character of input according to the result of data processing module Classification and Identification, and the character of input is presented to the position at cursor place.System has " mouse " and " keyboard " two kinds of mode of operations, can switch by " Mouse/Keyboard " key in LCD stimulator.

Described LCD visual stimulator comprises 8 stimulus keys, is distributed in screen surrounding.Wherein each " up(upwards) " stimulus key of upper side frame middle left and right, with the flash of light of 10Hz frequency, moves upward in the vertical direction for controlling cursor; Each " down(is downward) " stimulus key of lower frame middle left and right, with the frequency flash of light of 12Hz, moves downward in the vertical direction for controlling cursor; In the middle of upper and lower frame, each " mouse/keyboard(mouse/keyboard) " stimulus key, with the frequency flash of light of 15Hz, switches between " mouse " and " keyboard " pattern for system; In the middle of left and right frame, each " stop(stops) " stimulus key is with the frequency flash of light of 20Hz, for interested target is selected.

Described LED visual stimulator comprises 32 stimulus keys, and they are with different frequency flashes of light, and frequency range is between 6Hz～28Hz, and frequency interval is 0.5Hz, for character and numeral input.

Another aspect of the present invention has been to provide a kind of E-mail telecommunicating method based on the motion imagination and vision attention mixing brain-computer interface.First, system control unit starts LCD visual stimulator, control the frequency flash of light of 8 stimulus keys to set, wherein two " up " keys are used for controlling cursor and move upward in the vertical direction, two " down " moves downward in the vertical direction for controlling cursor, two " stop " keys are for interested icon is selected, and two " mouse/keyboard " keys switch between " mouse " and " keyboard " mode of operation for system; User is according to the current position of cursor and want the target location that arrives, watches specific stimulus key attentively and carries out specific motion imagination task simultaneously; Brain wave acquisition platform Real-time Collection scalp EEG signals, after amplification, filtering and analog to digital conversion, by data line input system control module; System control unit receives eeg data, after motion being imagined to the eeg data of generation and the eeg data of vision attention generation are separately according to electrode position, is kept in the internal memory of appointment; Data processing module carries out different pre-service, feature extraction and Classification and Identification to these two classes data in real time; Cursor control module, according to the classification results of two class data, is calculated cursor in the displacement of horizontal and vertical direction, controls cursor and carries out continuous two dimensional motion.When cursor movement arrives interested picture mark position, user selects icon by " stop " key of watching attentively on LCD stimulator.When system starts, in " mouse " mode of operation, by continuous cursor movement control and icon, select, user can realize collecting of Email.When user need to write Email, can successively cursor movement be arrived to address field and text area, by " Mouse/Keyboard " key, system is switched to " keyboard " mode of operation, then by watching corresponding stimulus key on LED stimulator attentively, inputs respectively e-mail address and write mail.After addresses of items of mail and mail write, user can be switched to system " mouse " pattern by " Mouse/Keyboard " key.

The method comprises following concrete steps:

1) system initialization: user dresses electrode cap, is sitting in computing machine dead ahead, and eyes and screen keep the distance of about 0.6 meter.Electrode is placed according to " international 10/20 standard lead system ", and the electrode of record motion imagination data is positioned at brain elementary motion-sensing region and synkinesia region, and the electrode that records vision attention data is positioned at brain occipitalia region.Inject conducting resinl to recording electrode passage, and guarantee that it contacts well with scalp.Open system works interface, start visual stimulator work.

2) eeg signal acquisition: user is according to the current position of cursor and want the target location that arrives, when watching specific stimulus key attentively, carry out specific motion imagination task, the scalp EEG signals producing gathers by electrode cap, after electroencephalogramdata data collector amplification, filtering and analog to digital conversion, by data line, digitized eeg data is inputted to computing machine.System control unit receives eeg data, after according to the position of electrode, two class eeg datas being separated, is kept in the internal memory of appointment.

3) eeg data is processed: the EEG signals that the EEG signals that data processing module produces the motion imagination respectively and vision attention produce is carried out pre-service, feature extraction and Classification and Identification successively, then the result of Classification and Identification is transferred to cursor control module.

4) cursor movement is controlled with icon and is selected: cursor control module, according to the classification results of two class data, is calculated respectively cursor in the displacement of horizontal and vertical direction, controls cursor and in workspace, carries out continuous two dimensional motion.In cursor movement process, user judges whether cursor arrives target location.If cursor arrives target location, user selects icon by " stop " key, and this cursor movement control task finishes; If cursor does not arrive target location, user continues to control cursor movement.

5) Email is collected: by continuous two dimensional cursor motion control, user can be by cursor movement to " collection of letters " icon or " inbox " icon; By " collection of letters " icon or " inbox " icon are selected, user can open inbox; By cursor movement, to letter column, by letter column is selected, user can collect an envelope mail.

6) email composition: first user arrives address field by cursor movement, system is switched to " keyboard " pattern by " Mouse/Keyboard " key, starts the work of LED stimulator, by watching corresponding stimulus key input addresses of items of mail attentively; Then by " Mouse/Keyboard " key, system is switched to " mouse " pattern, cursor movement is arrived to text area; By " Mouse/Keyboard " key, system is switched to " keyboard " pattern again, by watching corresponding stimulus key on LED stimulator attentively, writes Mail Contents.

Described step 2) in, specific stimulus key refers to, when wanting to control cursor and move upward in the vertical direction, user need to watch one of left and right two " up " keys in LCD stimulator upper side frame attentively.These two stimulus keys are with the frequency flash of light of 10Hz, and SSVEP signal corresponding to this frequency is designated as control cursor and moves upward; When wanting to control cursor and move downward in the vertical direction, user need to watch one of two " down " keys in LCD stimulator lower frame attentively.These two stimulus keys are with the frequency flash of light of 12Hz, and SSVEP signal corresponding to this frequency is designated as control cursor and moves downward; When wanting interested icon to select, user need to watch one of two " stop " keys in the middle of the left and right frame of working interface attentively.These two stimulus keys are with the frequency flash of light of 15Hz, and SSVEP signal corresponding to this frequency is designated as to be selected interested icon; When wanting system to switch between " mouse " and " keyboard " mode of operation, user need to watch one of two " mouse/keyboard " keys in the middle of the upper and lower frame of LCD stimulator attentively, these two stimulus keys are with the frequency flash of light of 20Hz, and SSVEP signal corresponding to this frequency is designated as system is switched between " mouse " and " keyboard " two kinds of mode of operations.

Described step 2) in, specific motion imagination task refers to, when user wants to control cursor in the horizontal direction to left movement, user need to carry out left hand motion imagination task, and this task is designated as controls cursor to left movement; When user wants to control cursor and moves right in the horizontal direction, user need to carry out right hand motion imagination task, and this task is designated as to be controlled cursor and move right.

Described step 2) in, according to the position of electrode, two class eeg datas are separately referred to, the data that are positioned at the electrode record in brain elementary motion-sensing region and synkinesia region are the eeg data that the motion imagination produces, and the data that are positioned at the electrode record in brain occipitalia region are the eeg data that vision attention produces.

The EEG signals in described step 3), the motion imagination being produced is carried out pre-service and is comprised reduce sampling frequency, uses average reference (Common Average Reference altogether, CAR) bandpass filtering that the data of reduce sampling frequency is reset reference point and carry out 8～30Hz to resetting the data of reference point, relevant the desynchronizing of event (Event-Related Desynchronization, the ERD) signal that extraction comprises the Mu rhythm and pace of moving things and the Beta rhythm and pace of moving things.

The EEG signals in described step 3), the motion imagination being produced is carried out feature extraction and is referred to, use spatial domain pattern (Common Spatial Pattern altogether, CSP) data of algorithm after to bandpass filtering are carried out airspace filter, by two class data projections, to the direction that has most differentiation power, after extraction projection, the variance of data is characteristic of division.The concrete steps of CSP algorithm are as follows:

1. calculate respectively the average covariance matrix of normalization of two class data

${\mathrm{<figure-callout\; id="1"\; label="R"\; filenames="131116221336.png"\; state="\{\{state\}\}">R</figure-callout>}}_1=\frac{1}{{\mathrm{<figure-callout\; id="1"\; label="N"\; filenames="131116221336.png"\; state="\{\{state\}\}">N</figure-callout>}}_1}\underset{i=1}{\overset{N_1}{\mathrm{\&Sigma;}}}\frac{X_{1i}{X}_{1i}^T}{\mathrm{trace}\left(X_{1i}{X}_{1i}^T\right)},R_2=\frac{1}{N_2}\underset{i=1}{\overset{N_2}{\mathrm{\&Sigma;}}}\frac{X_{2i}{X}_{2i}^T}{\mathrm{trace}\left(X_{2i}{X}_{2i}^T\right)}---\left(1\right)$

X in formula _1iwith X _2ibe respectively the class 1(left hand motion imagination) with the class 2(right hand motion imagination) the multichannel brain electric data of testing for the i time, N ₁with N ₂be respectively class 1 and class 2 number of training, T is transpose operator, and trace (M) represents to ask the element sum on matrix M diagonal line.

2. to mixing covariance matrix R _c=R ₁+ R ₂carry out Eigenvalues Decomposition

$R_c=U_c{\mathrm{\&Sigma;}}_C{U}_c^T---\left(2\right)$

U in formula _cfor feature matrix, Σ _cfor eigenwert diagonal matrix.

3. calculate whitening transformation matrix

$P={\mathrm{\&Sigma;}}_c^{-1/2}{U}_c^T---\left(3\right)$

4. to R ₁and R ₂carry out whitening transformation

R _1t＝PR ₁P ^T,R _2t＝PR ₂P ^T (4)

5. to R _1tand R _2tcarry out feature decomposition

R _1t＝UΣ ₁U ^T,R _2t＝UΣ ₂U ^T (5)

R _1tand R _2thave identical feature matrix U, their eigenvalue matrix sum is unit matrix, i.e. Σ ₁+ Σ ₂=I.Therefore, when the eigenwert of class data is got maximal value, the eigenwert of another kind of data will be got minimum value, thereby two class data farthest can be separated.Eigenwert is arranged by the order declining, and proper vector is arranged by same order, and CSP projection matrix is defined as W=U ^tp.The row of projection matrix W is called spatial filter, and its row are called spatial domain pattern.Capable and the capable airspace filter matrix F that forms of rear m by the front m of W.

6. the test data of single experiment is carried out to airspace filter

Z _i＝FX _i (6)

Z _ifor the source signal of the i time experiment EEG signals after airspace filter, the variance of source signal can be used as characteristic of division signal.

The EEG signals in described step 3), the motion imagination being produced is classified and is referred to, uses support vector machine (Support Vector Machine, SVM) sorter to classify to the brain electrical feature signal extracting based on CSP algorithm.

The EEG signals in described step 3), vision attention being produced is carried out pre-service and is referred to, then the EEG signals reduce sampling frequency that vision attention is produced carries out the bandpass filtering of 4～35Hz, extracts and comprises the band signal that may can be used as frequency of stimulation.

Described step 3) EEG signals in, vision attention being produced is carried out feature extraction and is referred to, use canonical correlation analysis (Canonical Correlation Analysis, CCA) to calculate reference signal that each frequency of stimulation is corresponding and the maximum correlation coefficient between the EEG signals of record.Although CCA algorithm can produce a plurality of related coefficients, for actual application problem such as electroencephalogramsignal signal analyzings, generally use maximum correlation coefficient.CCA algorithm comprises following two steps:

1. determine reference signal: supposition exists frequency of stimulation to be respectively f ₁, f ₂..., f _kk stimulation target.X and Y _fthe stochastic variable that represents two multidimensional, wherein X is N _tthe multichannel brain electric signal that second is long; Y _frepresent the reference signal identical with X length.This reference signal is the column vector that a sine by frequency of stimulation f and harmonic wave thereof and cosine form

Y _f＝(sin(2πft),cos(2πft),…,sin(2πN _hft),cos(2πN _hft)) ^T (7)

N in formula _hbe the number of harmonic wave, T is transpose operator.

2. each frequency of stimulation is calculated to maximum correlation coefficient: multichannel brain electric signal X and each reference signal

as the input of CCA algorithm, the frequency computation part maximum CCA coefficient ρ corresponding to each stimulus key in visual stimulator _k.Consider that a pair of linearity is in conjunction with x=X ^tw _xwith y=Y ^tw _y.The effect of CCA algorithm is to find weight vector W _xwith W _y, make the relevant maximization between x and y.In other words, constrained optimization problem below can solve multichannel brain electric signal X and each reference signal

maximum CCA coefficient

$\begin{array}{c}\underset{W_x,W_y}{\max }\mathrm{\&rho;}(x,y)=\frac{E\left[x^{T}y\right]}{\sqrt{E\left[x^{T}x\right]E\left[y^{T}y\right]}}=\frac{E\left[W_x^T{\mathrm{XY}}^T{W}_y\right]}{\sqrt{E\left[W_x^T{\mathrm{XX}}^T{W}_x\right]E\left[W_y^T{\mathrm{YY}}^T{W}_y\right]}}\\ \mathrm{subjecttoE}\left[{\mathrm{xx}}^T\right]=E\left[W_x^T{\mathrm{XX}}^T{W}_x\right]=1,E\left[{\mathrm{yy}}^T\right]=E\left[W_y^T{\mathrm{XX}}^T{W}_y\right]=1\end{array}---\left(8\right)$

The EEG signals in described step 3), vision attention being produced is carried out Classification and Identification and is referred to, maximum CCA coefficient ρ corresponding to each reference frequency obtaining according to characteristic extraction step _k, glow frequency and the corresponding user command thereof of identification user fixation object.At K K the maximum CCA coefficient ρ that frequency of stimulation is corresponding _kin, thering is peaked coefficient and be judged as CCA coefficient corresponding to target frequency that user watches attentively, order corresponding to this target, for user wants the order C expressing, can be formulated as follows

$C=\underset{k}{\max }{\mathrm{\&rho;}}_k,k=\mathrm{1,2},...,K---\left(5\right)$

ρ in formula _kthat EEG signals is at frequency of stimulation f _kcCA coefficient, K is the number of stimulation target.

In described step 6), corresponding stimulus key refers to the stimulus key on LED stimulator, comprises 10 numerals of 26 English alphabets, special symbol ﹫, 5 conventional punctuation marks and 5 function keys.

Tool of the present invention has the following advantages and beneficial effect:

1) the present invention's motion control for two dimensional cursor by the mixing BCI based on the motion imagination and vision attention, has realized difference, independence and the simultaneously control of cursor in horizontal and vertical directions, user's easy operating.

2) the present invention makes user can control cursor any one initial position from computer screen and moves to any one target location, and the movement of cursor on screen be continuously smooth, has avoided factitious Z-shaped jumping.

3) the present invention carries out combination in conjunction with BCI and two kinds of BCI based on vision attention based on the motion imagination, realizes collecting and writing of Email, has that cursor movement control accuracy is high, robustness good and character input accuracy advantages of higher.

Accompanying drawing explanation

Fig. 1 is the E-mail communication system that the present invention is based on the motion imagination and vision attention mixing brain-computer interface.

Fig. 2 is the mixing BCI system composition structure that the present invention realizes E-mail communication.

Fig. 3 is the working interface of two dimensional cursor motion control of the present invention and target selection.

Fig. 4 is the signal processing algorithm process flow diagram that the present invention realizes two dimensional cursor motion control.

Fig. 5 is that the present invention realizes the signal processing algorithm process flow diagram that icon is selected.

Fig. 6 is the sorting technique that the present invention is based on the motion imagination data of CSP algorithm.

Fig. 7 is the frequency identification method that the present invention is based on the SSVEP signal of CCA algorithm.

Fig. 8 is the BCI keyboard layout that the present invention is based on LED visual stimulator.

Embodiment

Below in conjunction with drawings and Examples, the present invention is described in further details, but embodiments of the present invention are not limited to this.

As shown in Figure 1, the present invention proposes a kind of E-mail communication system based on the motion imagination and vision attention mixing brain-computer interface, this system comprises visual stimulator, brain wave acquisition platform, system control unit, data processing module, cursor control module and character load module.Wherein system control unit, data processing module, cursor control module and character load module are realized by programming in computing machine.

Wherein, visual stimulator comprises visual stimulator and the visual stimulator based on light-emitting diode display based on LCD display.Stimulator based on LCD comprises 8 stimulus keys that produce SSVEP visual stimulus, is distributed in the four edges frame of screen.Each stimulus key of upper side frame middle left and right is luminous with the frequency of 10Hz, each stimulus key of lower frame middle left and right is luminous with the frequency of 12Hz, in the middle of left and right two frames, each stimulus key is luminous with the frequency of 15Hz, and in the middle of upper and lower frame, each stimulus key is luminous with the frequency of 20Hz.It is for fear of user's mobile sight line too much that each frequency of stimulation arranges two stimulus keys, person's operation easy to use; LED-based stimulator comprises 32 stimulus keys, and frequency of stimulation is in the scope of 6Hz～28Hz, between adjacent two frequency of stimulation, is spaced apart 0.5Hz.

Wherein, brain wave acquisition platform comprises electrode cap and electroencephalogramdata data collector.Electrode cap is used for gathering multichannel brain electric signal, and electrode position is placed by " international 10/20 standard lead system ", the electrode that comprises the electrode of record motion imagination signal and record vision attention signal for recording the electrode of EEG signals.The electrode of record motion imagination signal is positioned at brain elementary motion-sensing region and synkinesia region; The electrode that records vision attention signal is positioned at brain occipitalia region; Electroencephalogramdata data collector for the EEG signals gathering is amplified, filtering and analog to digital conversion, by data line by digitizing eeg data input computing machine.

Wherein, system control unit receives eeg data, after the two class eeg datas that the motion imagination and vision attention produced according to electrode position separate, is kept in the internal memory of appointment; Start system works interface, and control the frequency flash of light of LCD stimulator moderate stimulation key to set.

Wherein, the eeg data that the eeg data that data processing module produces the motion imagination respectively and vision attention produce carries out real-time pre-service, feature extraction and Classification and Identification.

Wherein, cursor control module, according to the result of data processing module Classification and Identification, is calculated cursor in the displacement of horizontal and vertical direction, controls cursor and carries out continuous two dimensional motion, or interested target is selected.

Wherein, character load module starts LED visual stimulator, determines the character of input according to the result of data processing module Classification and Identification, and the character of input is presented to the position at cursor place.

The present invention proposes a kind of E-mail telecommunicating method based on the motion imagination and vision attention mixing brain-computer interface.Below in conjunction with Fig. 2 to Fig. 8, be elaborated.

Fig. 2 is that the mixing BCI system that realizes E-mail communication forms structure.Realize E-mail communication, mix the necessary analog computer mouse of BCI system and two functions of keyboard, according to user, need between mouse and two kinds of patterns of keyboard, switch.

Mouse performance pattern requires two BCI to work simultaneously, use the ERD signal of motion imagination generation and the SSVEP signal of vision attention generation to control respectively cursor motion in the horizontal and vertical directions, and in conjunction with these two characteristic signals, target (being icon) is selected.Due to movement in vertical direction, only have upwards, downwards and stop three kinds of selections, computing machine LCD display can produce needed visual stimulus signal.As shown in Figure 2, two dimensional cursor motion control and target selection can be realized by ERD BCI and LCD SSVEP BCI.

Keyboard mode of operation realizes character input, needs the character of input to comprise 26 English characters, 10 numerals, conventional punctuation mark and a small amount of special character.Character input also requires BCI keyboard to have the functions such as error recovery, backspace and carriage return.LCD display cannot produce so many stimulus signals, need to use light-emitting diode display design hardware stimulator.As shown in Figure 2, character is inputted by the BCI based on LED vision attention, and LED SSVEP BCI realizes separately.The switching of system from mouse mode to keyboard mode or from keyboard mode to mouse, is realized by the stimulus key " Mouse/Keyboard " of LCD stimulator.

Fig. 3 is the working interface of two dimensional cursor motion control.As shown in Figure 3, the main interface of working is client region, corresponding e-mail system homepage, and user starts to carry out mail from mailing system homepage and collects and the work of writing.8 stimulus keys that produce SSVEP visual stimulus have distributed in working interface four edges frame.These 8 stimulus keys have formed the LCD visual stimulator that brings out SSVEP signal.Wherein, each " up " key control cursor of upper side frame middle left and right moves upward; Each " down " key control cursor of lower frame middle left and right moves downward, and in the middle of left and right two frames, each " stop " key is target selection key, when cursor arrives picture mark position, icon is selected; In the middle of upper and lower two frames, each " Mouse/Keyboard " key is system works pattern shift key, for system is switched between " mouse " and " keyboard " pattern.

Fig. 4 is the signal processing algorithm process flow diagram of realizing two dimensional cursor motion control.As shown in Figure 4, data processing module carries out different pre-service, the feature extraction identification of presorting to the two class data of separately depositing in internal memory.

For motion imagination data, data processing module is total to average reference to its use successively and resets reference point, carries out 8～30Hz bandpass filtering extraction ERD/ERS signal, uses CSP algorithm to extract and move and imagine that characteristic signal and use SVM classify to characteristic signal.Cursor control module is calculated cursor displacement in the horizontal direction according to svm classifier result, and the horizontal coordinate after definite cursor displacement.

For vision attention data, the bandpass filtering that data processing module carries out 4～35Hz to it successively extracts and comprises that the band signal, the use CCA algorithm that can be used as SSVEP frequency of stimulation extract SSVEP feature and SSVEP frequency is identified.Cursor control module is calculated cursor displacement in the vertical direction according to frequency identification result, and the vertical coordinate after the displacement of definite cursor.

Fig. 5 is the signal processing algorithm process flow diagram that realize target or icon are selected.Data processing module is used the method identical with two dimensional cursor motion control to extract respectively motion imagination data characteristics and vision attention data characteristics, then two category feature signals are linked together and form composite character vector, input svm classifier device is classified, and finally according to classification results, makes the judgement that icon is selected or refused.

Fig. 6 is the motion imagination data classification method process flow diagram based on CSP algorithm.CSP is a kind of supervised recognition algorithm, when it is applied to left hand and the classification of right hand motion imagination eeg data, need to gather training data for specific user, according to the training data of known class, estimates two spatial filters.Use CSP algorithm as follows to the concrete steps of motion imagination Data classification:

1) training data of Real-time Collection is carried out to common average reference and process and 8～30Hz bandpass filtering, use the training data after bandpass filtering to estimate left hand and two spatial filters of the right hand motion imagination;

2) use left hand and right hand motion imagination spatial filter respectively the training data of single experiment to be carried out to airspace filter;

3) variance of single experiment training data after calculating left hand and right hand airspace filter, definition left hand and right hand filtering variance are characteristic of division with the logarithm of the ratio of left hand and right hand filtering variance sum, and these two characteristic of divisions are connected to a proper vector;

4) use left hand and the right hand motion imagination two category feature vector training svm classifier devices, determine sorter model parameter;

5) one of Real-time Collection section of test data is carried out to common average reference and process and 8～30Hz bandpass filtering, the test data of two spatial filters that use step 1) estimation after to bandpass filtering carried out airspace filter;

6) variance of test data after calculating left hand and right hand airspace filter, definition left hand and right hand filtering variance are characteristic of division with the logarithm of the ratio of left hand and right hand filtering variance sum, and these two characteristic of divisions are connected to a proper vector;

7) use the svm classifier device of step 4) training to classify to the proper vector of test data.Classification results is inputted to cursor control module, for the calculating of cursor horizontal shift and horizontal coordinate.

As shown in Figure 7, use CCA algorithm to process in real time eeg data, the frequency of identification SSVEP signal.Concrete steps are as follows:

1) determine reference signal: supposition exists frequency of stimulation to be respectively f ₁, f ₂..., f _kk target.X and Y _fthe stochastic variable that represents two multidimensional, wherein X is N _tthe multichannel brain electric signal that second is long; Y _frepresent the reference signal identical with X length.This reference signal is the column vector that a sine by frequency of stimulation f and harmonic wave thereof and cosine form

Y _f＝(sin(2πft),cos(2πft),…,sin(2πN _hft),cos(2πN _hft)) ^T (6)

N in formula _hthe number of harmonic wave, in the present embodiment N _h=3.

2) all frequency of stimulation are calculated to CCA coefficient: one of multichannel brain electric signal X and reference signal

as the input of CCA algorithm, each frequency of stimulation of this two dimensional cursor kinetic control system is calculated to CCA coefficient.

Consider that a pair of linearity is in conjunction with x=X ^tw _xwith y=Y ^tw _y.The effect of CCA is to find weight vector W _xwith W _y, make the relevant maximization between x and y.Constrained optimization problem below can solve multichannel brain electric signal X and each reference signal

maximum CCA coefficient

$\begin{array}{c}\underset{W_x,W_y}{\max }\mathrm{\&rho;}(x,y)=\frac{E\left[x^{T}y\right]}{\sqrt{E\left[x^{T}x\right]E\left[y^{T}y\right]}}=\frac{E\left[W_x^T{\mathrm{XY}}^T{W}_y\right]}{\sqrt{E\left[W_x^T{\mathrm{XX}}^T{W}_x\right]E\left[W_y^T{\mathrm{YY}}^T{W}_y\right]}}\\ \mathrm{subjecttoE}\left[{\mathrm{xx}}^T\right]=E\left[W_x^T{\mathrm{XX}}^T{W}_x\right]=1,E\left[{\mathrm{yy}}^T\right]=E\left[W_y^T{\mathrm{XX}}^T{W}_y\right]=1\end{array}---\left(7\right)$

3) determine user command: with W _xand W _ycorresponding maximal value ρ _kfor maximum typical related coefficient.X and Y are respectively at W _xand W _yon projection, x and y, be called as canonical variable.The canonical correlation ρ of output _kcan be used for SSVEP frequency identification.At K K the maximum CCA coefficient ρ that frequency of stimulation is corresponding _kin, thering is peaked coefficient and be judged as CCA coefficient corresponding to target frequency that user watches attentively, order corresponding to this target, for user wants the order C expressing, can be formulated as follows

$C=\underset{k}{\max }{\mathrm{\&rho;}}_k,k=\mathrm{1,2},...,K---\left(8\right)$

ρ in formula _kthat EEG signals is at frequency of stimulation f _kcCA coefficient, K is the number of stimulation target.

Fig. 8 is the BCI keyboard layout based on LED visual stimulator.As shown in Figure 8, the stimulator of LED comprises 32 stimulus keys, can be used for inputting 26 English alphabets (a～z), special character, 10 numerals (0～9), 5 punctuation marks:, (comma).(fullstop); (branch): (colon) and (question mark) and 5 function keys: ← (backspace), ━ (space),

(carriage return), L/D(letter/number) conversion, L/C(letters/symbols) conversion.Wherein, 10 numerals and 5 punctuation marks and monogram are used, and switch respectively by letter/number (L/D) shift key and letters/symbols (L/C) shift key.32 stimulus keys are luminous with different frequencies, and frequency range is 6Hz～28Hz, between frequency, are spaced apart 0.5Hz.

Claims (2)

1. the E-mail communication system based on the motion imagination and vision attention mixing brain-computer interface, it is characterized in that comprising visual stimulator, brain wave acquisition platform, system control unit, data processing module, cursor control module and character load module, wherein system control unit, data processing module, cursor control module and character load module are realized by software in computing machine; Visual stimulator comprises LCD visual stimulator and LED visual stimulator, and for bringing out Steady State Visual Evoked Potential signal, LCD visual stimulator offers user in interface display mode; Brain wave acquisition platform comprises electrode cap and electroencephalogramdata data collector, and electrode cap gathers EEG signals, electroencephalogramdata data collector to the EEG signals gathering amplify, filtering and analog to digital conversion, and digitized EEG signals is inputted to computing machine by data line; System control unit receives, preserves eeg data, and system works interface is provided, and controls the stimulus key of LCD visual stimulator with the frequency flash of light of setting; Data processing module carries out respectively real-time pre-service, feature extraction and Classification and Identification to motion imagination eeg data and vision attention eeg data; Cursor control module is calculated cursor in the displacement of horizontal and vertical direction according to the result of Classification and Identification, controls cursor and carries out continuous two dimensional motion, and interested target is selected; Character load module starts LED visual stimulator, determines the character of input according to the result of data processing module Classification and Identification, and the character of input is presented to the position at cursor place;

Described LCD visual stimulator comprises 8 stimulus keys, is distributed in screen surrounding; Wherein each " up " stimulus key of upper side frame middle left and right, with the flash of light of 10Hz frequency, moves upward in the vertical direction for controlling cursor; Each " down " stimulus key of lower frame middle left and right, with the frequency flash of light of 12Hz, moves downward in the vertical direction for controlling cursor; In the middle of upper and lower frame, each " mouse/keyboard " stimulus key, with the frequency flash of light of 15Hz, switches between " mouse " and " keyboard " pattern for system; In the middle of left and right frame, each " stop " stimulus key is with the frequency flash of light of 20Hz, for interested target is selected;

Described LED visual stimulator comprises 32 stimulus keys, and they are with different frequency flashes of light, and frequency range is between 6Hz～28Hz, and frequency interval is 0.5Hz, for character and numeral input.

2. the control method of the E-mail communication system based on the motion imagination and vision attention mixing brain-computer interface claimed in claim 1, is characterized in that:

1) system initialization: user dresses electrode cap, is sitting in computing machine dead ahead, and eyes and screen keep the distance of about 0.6 meter; Electrode is placed according to " international 10/20 standard lead system ", and the electrode of record motion imagination data is positioned at brain elementary motion-sensing region and synkinesia region, and the electrode that records vision attention data is positioned at brain occipitalia region; Inject conducting resinl to recording electrode passage, and guarantee that it contacts with scalp well; Open system works interface, start visual stimulator work;

2) eeg signal acquisition: user is according to the current position of cursor and want the target location that arrives, when watching particular stimulation key attentively, carry out specific motion imagination task, the scalp EEG signals producing gathers by electrode cap, after electroencephalogramdata data collector amplification, filtering and analog to digital conversion, by data line, digitized eeg data is inputted to computing machine; System control unit receives eeg data, after according to the position of electrode, two class eeg datas being separated, is kept in the internal memory of appointment;

3) eeg data is processed: the EEG signals that the EEG signals that data processing module produces the motion imagination respectively and vision attention produce is carried out pre-service, feature extraction and Classification and Identification successively, then the result of Classification and Identification is transferred to cursor control module;

4) cursor movement is controlled with icon and is selected: cursor control module, according to the classification results of two class data, is calculated respectively cursor in the displacement of horizontal and vertical direction, controls cursor and in workspace, carries out continuous two dimensional motion; In cursor movement process, user judges whether cursor arrives target location; If cursor arrives target location, user selects icon by " stop " key, and this cursor movement control task finishes; If cursor does not arrive target location, user continues to control cursor movement;

5) Email is collected: by continuous two dimensional cursor motion control, user can be by cursor movement to " collection of letters " icon or " inbox " icon; By " collection of letters " icon or " inbox " icon are selected, user can open inbox; By cursor movement, to letter column, by letter column is selected, user can collect an envelope mail;

6) email composition: first user arrives address field by cursor movement, system is switched to " keyboard " pattern by " Mouse/Keyboard " key, starts the work of LED stimulator, by watching corresponding stimulus key input addresses of items of mail attentively; Then by " Mouse/Keyboard " key, system is switched to " mouse " pattern, cursor movement is arrived to text area; Passing through again Mouse/Keyboard " key is switched to system " keyboard " pattern, by watching corresponding stimulus key on LED stimulator attentively, writes Mail Contents.

Images