US11379039B2 - Brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement - Google Patents

Brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement Download PDF

Info

Publication number
US11379039B2
US11379039B2 US16/977,751 US201916977751A US11379039B2 US 11379039 B2 US11379039 B2 US 11379039B2 US 201916977751 A US201916977751 A US 201916977751A US 11379039 B2 US11379039 B2 US 11379039B2
Authority
US
United States
Prior art keywords
electroencephalogram signal
real
digital
phase
time
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US16/977,751
Other versions
US20200401226A1 (en
Inventor
Aiguo SONG
Wenbin Zhang
Hong Zeng
Baoguo XU
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Assigned to SOUTHEAST UNIVERSITY reassignment SOUTHEAST UNIVERSITY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ZENG, HONG, SONG, Aiguo, XU, BAOGUO, ZHANG, WENBIN
Publication of US20200401226A1 publication Critical patent/US20200401226A1/en
Application granted granted Critical
Publication of US11379039B2 publication Critical patent/US11379039B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/375—Electroencephalography [EEG] using biofeedback
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/372—Analysis of electroencephalograms
    • A61B5/374—Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/369—Electroencephalography [EEG]
    • A61B5/377—Electroencephalography [EEG] using evoked responses
    • A61B5/378—Visual stimuli
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7253—Details of waveform analysis characterised by using transforms
    • A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/74—Details of notification to user or communication with user or patient; User input means
    • A61B5/7455—Details of notification to user or communication with user or patient; User input means characterised by tactile indication, e.g. vibration or electrical stimulation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/14—Fourier, Walsh or analogous domain transformations, e.g. Laplace, Hilbert, Karhunen-Loeve, transforms
    • G06F17/141—Discrete Fourier transforms
    • G06F17/142—Fast Fourier transforms, e.g. using a Cooley-Tukey type algorithm
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/016—Input arrangements with force or tactile feedback as computer generated output to the user
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2203/00—Indexing scheme relating to G06F3/00 - G06F3/048
    • G06F2203/01—Indexing scheme relating to G06F3/01
    • G06F2203/011—Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns

Definitions

  • the present invention relates to a brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement, and belongs to the technical field of a brain-computer interface.
  • the first International Conference on Brain-Computer Interface held in 1999 gave the definition of a brain-computer interface, that is, the brain-computer interface is a communication system which does not depend on a normal output pathway consisting of peripheral nerves and muscles.
  • a biological principle of the brain-computer interface is that when the brain performs thinking activities, generates motion consciousness or is stimulated by the outside, nerve cells will generate micro-electrical activities of dozens of millivolts, and the electrical activities of a large number of nerve cells are transmitted to the surface layer of the scalp to form brain waves.
  • the brain-computer interface (BCI) is an advanced technology for converting brain activity features into predefined commands to thus realize communication with the outside or the control on other external devices on the basis of electroencephalogram signals or other related technologies.
  • Electroencephalogram signals extracted by a method of placing electrodes on the scalp are called as scalp electroencephalogram (EEG) signals.
  • EEG scalp electroencephalogram
  • a signal-to-noise ratio of the EEG signals is very low, and it is difficult to extract stable and reliable signals. This is a serious problem in the control of external devices, and seriously influences the application range of the brain-computer interface.
  • BCI users can initiate brain control by simply executing imagery of left-hand or right-hand motions.
  • the kinesthetic imagery of the hand motion generates event-related desynchronization/event-related synchronization (ERD/ERS) in the sensory motor cortex of the subject.
  • the event-related synchronization is defined as a decrease or increase of power in a specific frequency band (such as ⁇ wave band 8-13 Hz).
  • motor imagery-induced ERD activity focuses the contralateral hemisphere, that is, the motor imagery induces the ERD in the contralateral sensory motor cortex, and these lateral cortex activities constitute the neurophysiological basis of the motor imagery-based brain-computer interface.
  • the human tactile receptors are mainly Meissner corpuscles, distributed in the superficial skin and sensing the light pressing stimulation of the skin; Meissner's corpuscles, mainly responsible for sensing the tactile sensations; Pacinian corpuscles in a deeper layer, mainly sensing pressing sensations; and Pinkusi corpuscles, many nerve endings, tactile menisci, etc.
  • the Meissner corpuscles are positioned at a tip adjacent to the main ridge and closest to the skin surface. They are very effective in signal conduction of low-frequency vibration (1-40 Hz), and thus achieve an important effect in sensory vibration detection.
  • non-invasive electrical stimulation can more effectively modulate the neural activities while locking potential brain rhythms.
  • Vibratory stimulation may potentially be in phase with natural oscillations.
  • due to complexity and temporal variability of the EEG signals, matching with spontaneous oscillations of the human body is a challenging problem.
  • the present invention aims at solving the technical problems to overcome the defects in the prior art and to provide a brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement by aiming at the characteristic problems of nonlinear, non-stable, non-gaussian processes and the like of an electroencephalogram signal per se, thus improving a signal-to-noise ratio of a brain-computer interface system and enhancing a brain source signal.
  • a brain-computer interface method based on real-time closed loop vibration stimulation enhancement includes the following steps:
  • the motor imagery task in the method includes a left-hand or right-hand motor imagery action.
  • the intercepted digital electroencephalogram signal in the preset time period is subjected to a waveband band-pass filtering.
  • y(t) is a digital electroencephalogram signal after Hilbert transform is performed on x(t); x(t) is the digital electroencephalogram signal of the period after the band-pass filtering; p and v are integrals in the Cauchy principal value sense; and ⁇ x (t) is an instantaneous phase at a moment t.
  • the predicted sine wave f s generated in the method is specifically:
  • f main is a main frequency of the intercepted digital electroencephalogram signal of the preset time period; t f is a length of a predicted waveform, and ⁇ x (t) is an instantaneous phase at the moment t.
  • the brain-computer interface system based on real-time closed loop vibration stimulation enhancement provided by the present invention includes:
  • a human-computer interaction module configured to display and provide a motor imagery task to a subject
  • an electroencephalogram signal collection module configured to collect a digital electroencephalogram signal generated during motor imagery of the subject
  • a real-time phase prediction module configured to read the collected digital electroencephalogram signal, judge whether the preset time period is exceeded or not, intercept the digital electroencephalogram signal in the preset time period if YES, and continuously read the collected digital electroencephalogram signal if NO; and configured to obtain time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform after performing band-pass filtering on the intercepted digital electroencephalogram signal in the preset time period, extract a frequency value with highest frequency energy in the time-frequency characteristics as its main frequency, obtain an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform, generate predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predict and obtain real-time phase information at the current moment according to the predicted sine waves;
  • an electroencephalogram signal analysis module configured to judge whether the real-time phase is in a vibration stimulation application phase interval or not according to the predicted and obtained real-time phase information at the current moment, and generate and output a control instruction according to the judging result;
  • a vibration stimulation feedback module configured to control a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction output by the electroencephalogram signal analysis module.
  • the electroencephalogram signal collection module includes an electroencephalogram cap, an electroencephalogram signal amplifier, a low-pass and band-stop filter, an analog-to-digital conversion module and a communication module in sequential connection.
  • the real-time phase prediction module uses a band-pass filter to perform band-pass filtering on the digital electroencephalogram signal in the preset time period.
  • the band-pass filter adopts a ten-order oval infinite impulse response filter.
  • the vibration stimulation feedback module sets different vibration frequencies to act on the left hand and the right hand according to the control instruction.
  • the present invention can achieve the following technical effects:
  • the method and system of the present invention achieve a real-time closed loop vibration stimulation effect so as to enhance a signal-to-noise ratio of the electroencephalogram signal and improve a decoding rate of the motor imagery task by guiding a subject to execute a motor imagery task via display, generating predicted sine waves by using the data period time-frequency characteristics of the collected electroencephalogram signals, then predicting the real-time electroencephalogram signal phase information on the basis of the predicted sine waves, and controlling the vibration motor to apply vibration stimulation on the fingertip of a person by using the predicted instantaneous phase information.
  • the present invention can feedback, regulate and control the electroencephalogram rhythm through vibration stimulation.
  • a signal-to-noise ratio of the brain-computer interface system is improved.
  • a new scheme is provided for enhancing a brain source signal.
  • the recognition rate of the motor imagery signal is enhanced.
  • a “BCI blind” phenomenon is reduced. Therefore, the communication between the user and the outside is more effective, more convenient and faster.
  • the present invention can improve real-time performance of vibration stimulation.
  • the stimulation effect is maximized by concentrating the stimulation on the optimum phase for enhancing the electroencephalogram signal and ensuring the repeating in a plurality of cycles.
  • the decoding rate of the motor imagery task is improved. Differences of the brain-computer interface between individuals are reduced. Compared with a traditional open-loop continuous stimulation method, such a system can realize vibration control at lower power demand and higher specificity, and can reduce tolerance and rebounding risks.
  • FIG. 1 is a schematic structure diagram of a brain-computer interface system based on real-time closed-loop vibration stimulation enhancement.
  • FIG. 2 is a real-time phase prediction flow diagram in a method of the present invention.
  • FIG. 3 is a schematic diagram of a motor imagery experimental paradigm based on real-time closed-loop stimulation feedback of the present invention.
  • the present invention designs a brain-computer interface system based on real-time closed-loop vibration stimulation enhancement.
  • the system mainly includes: a human-computer interaction module, an electroencephalogram signal collection module, a real-time phase prediction module, an electroencephalogram signal analysis module and a vibration stimulation feedback module.
  • An output end of the electroencephalogram signal collection module is connected with an input end of the real-time phase prediction module.
  • An output end of the real-time phase prediction module is connected with an input end of the electroencephalogram signal analysis module.
  • An output end of the electroencephalogram signal analysis module is connected with the vibration stimulation feedback module.
  • the vibration stimulation feedback module directly acts on the body of a subject, and acts on the left and right hands of the subject in the present embodiment.
  • the human-computer interaction module consists of a display screen displaying a motor imagery task on a screen, and is configured to display and provide the motor imagery task to the subject and guide the subject to complete the left-hand or right-hand motor imagery task through a visual channel.
  • the electroencephalogram signal collection module is configured to collect a digital electroencephalogram signal of the subject generated during motor imagery, and mainly includes an electroencephalogram cap, an electroencephalogram signal amplifier, a low-pass and band-stop filter, an analog-to-digital conversion module and a communication module in sequential connection.
  • the electroencephalogram cap is worn on the head of the subject, and collects the electroencephalogram signal of the subject generated during motor imagery.
  • the digital electroencephalogram signal is obtained sequentially through amplification by the electroencephalogram signal amplifier, low-pass filtering by the low-pass and band-stop filter and the analog-to-digital conversion by the analog-to-digital conversion module.
  • the digital electroencephalogram signal is transmitted to the real-time phase prediction module through the communication module.
  • the real-time phase prediction module is configured to receive the digital electroencephalogram signal transmitted by the electroencephalogram signal collection module through a USB interface, read the collected digital electroencephalogram signal, judge whether a preset time period is exceeded or not, intercept the digital electroencephalogram signal in the preset time period if YES, and continuously read the collected digital electroencephalogram signal if NO; and is configured to obtain time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform after performing band-pass filtering on the intercepted digital electroencephalogram signal in the preset time period, extract a frequency value with highest frequency energy in the time-frequency characteristics as its main frequency, obtain an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform, generate predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predict and obtain real-time phase information at the current moment according to the predicted sine waves to realize the instantane
  • the electroencephalogram signal analysis module is configured to judge whether the real-time phase is in a vibration stimulation application phase interval or not according to the real-time phase information predicted and obtained by the real-time phase prediction module at the current moment, and generate and output a control instruction according to the judging result.
  • the vibration stimulation feedback module is configured to control a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction output by the electroencephalogram signal analysis module, stimulate the fingertip of the subject to further influence the sensory channel of the subject, and regulate the electroencephalogram signal rhythm.
  • the digital electroencephalogram signal includes an imagery fisting motion signal.
  • the real-time phase prediction module of the system of the present invention adopts the band-pass filter to perform band-pass filtering on the digital electroencephalogram signal in the preset time period.
  • the frequency band of the band-pass filtering is a brain activity ⁇ frequency band.
  • the band-pass filter adopts a ten-order oval infinite impulse response filter. That is, 500 ms is used as a data segment for real-time analysis.
  • the electroencephalogram signal data segment is subjected to band-pass filtering by the ten-order oval infinite impulse response filter to obtain the required digital electroencephalogram signal at an ⁇ frequency band and in a range of 8-12 hz.
  • the passband ripple is set to be 0.5 dB.
  • the stopband attenuation is set to be 40 dB.
  • the selected data fragment is short, so the order number will be limited if a finite impulse response (FIR) filter is adopted.
  • An infinite impulse response (IIR) filter is adopted.
  • the vibration stimulation feedback module in the system of the present invention can set different vibration frequencies according to the control instruction to act on the left hand and the right hand.
  • a vibration stimulation signal set in the vibration stimulation module and acting on the left hand is sine waves at a vibration frequency of 22 hz
  • a vibration stimulation signal acting on the right hand is sine waves at a vibration frequency of 26 hz.
  • the human body sensory system has different sensitive degrees on the left and the right, so the optimum stimulation effect can be generated in an auxiliary way by setting different vibration frequencies.
  • the present invention further provides a brain-computer interface method based on real-time closed-loop vibration stimulation enhancement.
  • the method is used for signal processing of the above system, and specifically includes the following steps:
  • Step 1 The human-computer interaction module displays and provides the motor imagery task to a subject.
  • the motor imagery task may include a left-hand or right-hand motor imagery action.
  • Stimulation displayed in a display in the human-computer interaction module includes three modes, specifically including:
  • the vibration stimulation module is mainly configured to generate stimulation feedback according to the electroencephalogram signal obtained in real time, thus achieving an effect of enhancing the electroencephalogram signal.
  • the subject sits on a comfortable chair according to requirements, and wears a 64-lead electroencephalogram cap on the head.
  • a distance between the two eyes and the display is about one meter.
  • the left-hand or right-hand fisting action is imaged according to a pointing direction of the arrow displayed on the display.
  • the real-time electroencephalogram signal data of the user is obtained through the 64-lead electroencephalogram collection cap in the electroencephalogram signal collection module. Sequentially through processing and collection by each component, the digital electroencephalogram signal data generated during motor imagery is obtained.
  • Step 2 The real-time phase prediction module executes a real-time electroencephalogram signal phase prediction algorithm while the motor imagery task is started. As shown in the figure, a specific process is as follows:
  • the real-time phase prediction module reads the collected digital electroencephalogram signal and judges whether the preset time period is exceeded or not. For example, whether 500 ms is exceeded or not. If 500 ms is exceeded, the digital electroencephalogram signal in the preset time period 500 ms is intercepted for analysis and prediction. If 500 ms is not exceeded, the collected digital electroencephalogram signal is continuously read.
  • the real-time phase prediction module performs an analysis and prediction process:
  • the intercepted digital electroencephalogram signal in the preset time period 500 ms is subjected to band-pass filtering to obtain a specific occurrence frequency band of an ERD phenomenon under the motor imagery task, such as the ⁇ wave band.
  • the range is 8-12 hz.
  • the time-frequency characteristics of the digital electroencephalogram signal of the period is obtained through calculation by fast Fourier transform, and a frequency value with highest frequency energy is extracted to be used as the main frequency of the digital electroencephalogram signal of the period.
  • the frequency value with the highest frequency energy in the signal period is extracted to be used as the main frequency of the signal of the period.
  • an instantaneous phase of the digital electroencephalogram signal x(t) of the period is obtained through calculation by Hilbert transform.
  • the digital electroencephalogram signal y(t) after Hilbert transform is performed on x(t) is obtained.
  • the following formula is used:
  • y ( t ) p ⁇ 1 ⁇ ⁇ v ⁇ ⁇ - ⁇ + ⁇ ⁇ x ( t ) t - ⁇ ⁇ d ⁇ ⁇ ⁇ ⁇ .
  • the instantaneous phase at the moment t may be calculated through the following formula:
  • ⁇ x ⁇ ( t ) tan - 1 ⁇ y ⁇ ( t ) x ⁇ ( t ) ,
  • ⁇ x (t) is an instantaneous phase at the moment t
  • x(t) is the digital electroencephalogram signal of the period after the band-pass filtering
  • p and v are integrals in the Cauchy principal value sense.
  • the predicted sine waves are generated by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves.
  • the prediction on the real-time phase is realized according to the real-time phase information predicted and obtained by the predicted sine waves.
  • the sine wave function formula is as follows:
  • f s is predicted sine waves
  • f main is the main frequency of the intercepted digital electroencephalogram signal of the preset time period
  • t f is a length of a predicted waveform, is set to be 50 ms herein, and may be determined according to the time difference of the selected current time point and the time point of obtaining the instantaneous phase of the electroencephalogram signal
  • ⁇ x (t) is an instantaneous phase at the moment t.
  • the main frequency f main of the data fragment is obtained through calculation by fast Fourier transform.
  • a sine wave functional expression can be listed to obtain the precise predicted sine waves.
  • the length t f of the predicted waveform is set to be one-tenth of a length of a reference signal, i.e., 50 ms.
  • a principle is that the electroencephalogram signal is a nonlinear and instable signal, and therefore, 500 ms of data is extracted in per 50 ms fragment for prediction, so as to improve the prediction accuracy.
  • Step 3 The electroencephalogram signal analysis module judges whether the real-time phase is in a vibration stimulation application phase interval or not according to the real-time phase information predicted and obtained by the real-time phase prediction module at the current moment t. If the judging result is that the real-time phase is in the phase interval, a control instruction is generated and output.
  • the vibration stimulation feedback module controls a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction. Otherwise, if the judging result is that the real-time phase is not in the phase interval, the control instruction is not output, and the vibration stimulation does not need to be generated.
  • the present invention uses the above method. Its test result is as shown in FIG. 3 .
  • About 3 ms is needed from the generation of the electroencephalogram signal, to the recording by the electroencephalogram cap, and then to the entering of the real-time phase prediction module through an amplifier and USB transmission.
  • About 5 ms is needed for a computer to execute a prediction algorithm for data processing.
  • About 2 ms is needed for obtaining the instantaneous phase value and converting the instantaneous phase value into an instruction to control the vibration motor to vibrate. Therefore, the 30 th ms position of the predicted sine waves may be approximately regarded as the electroencephalogram signal of the current time point.
  • the instantaneous phase at a 30 ms position is taken to be used as a point for the vibration motor to control a triggering signal, thus achieving the goal of stimulation based on the real-time phase.
  • the present invention designs the brain-computer interface method and system capable of performing fingertip vibration stimulation so as to enhance the decoding rate of the motor imagery task on the basis of the real-time phase of the electroencephalogram signal.
  • a method for enhancing the electroencephalogram signal is designed.
  • the present invention achieves the maximum stimulation effect through cumulative effect by concentrating the stimulation on the optimum phase for enhancing the electroencephalogram signal and ensuring the repeating in a plurality of cycles. Therefore, the vibration control can be realized at lower power demand and higher specificity. Additionally, the tolerance and rebounding risks can be reduced.
  • the present invention improves a signal-to-noise ratio of the brain-computer interface system, and enhances the recognition rate of a motor imagery signal.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • Surgery (AREA)
  • Animal Behavior & Ethology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • Pathology (AREA)
  • Biophysics (AREA)
  • Psychiatry (AREA)
  • Human Computer Interaction (AREA)
  • Psychology (AREA)
  • Mathematical Optimization (AREA)
  • Mathematical Analysis (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Dermatology (AREA)
  • Neurosurgery (AREA)
  • Neurology (AREA)
  • Algebra (AREA)
  • Software Systems (AREA)
  • Databases & Information Systems (AREA)
  • Discrete Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Physiology (AREA)
  • Signal Processing (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

Brain-computer interface method and system include displaying and providing a motor imagery task to a subject, and collecting a generated digital electroencephalogram signal; reading the digital electroencephalogram signal, performing interception if a preset time period is exceeded, and performing continuous reading if not; performing band-pass filtering, obtaining time-frequency characteristics of the digital electroencephalogram signal, and extracting a frequency value with highest frequency energy as a main frequency; obtaining an instantaneous phase of the digital electroencephalogram signal; generating predicted sine waves by respectively using the main frequency and the instantaneous phase as a frequency and an initial phase of sine waves, and predicting and obtaining real-time phase information; and judging whether the real-time phase is in a vibration stimulation application phase interval, generating and outputting a control instruction, and controlling a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction.

Description

BACKGROUND Technical Field
The present invention relates to a brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement, and belongs to the technical field of a brain-computer interface.
Related Art
The first International Conference on Brain-Computer Interface held in 1999 gave the definition of a brain-computer interface, that is, the brain-computer interface is a communication system which does not depend on a normal output pathway consisting of peripheral nerves and muscles. A biological principle of the brain-computer interface is that when the brain performs thinking activities, generates motion consciousness or is stimulated by the outside, nerve cells will generate micro-electrical activities of dozens of millivolts, and the electrical activities of a large number of nerve cells are transmitted to the surface layer of the scalp to form brain waves. The brain-computer interface (BCI) is an advanced technology for converting brain activity features into predefined commands to thus realize communication with the outside or the control on other external devices on the basis of electroencephalogram signals or other related technologies.
Electroencephalogram signals extracted by a method of placing electrodes on the scalp are called as scalp electroencephalogram (EEG) signals. Because of isolation by meninges, skulls and multilayer tissues from the brain to the scalp, a signal-to-noise ratio of the EEG signals is very low, and it is difficult to extract stable and reliable signals. This is a serious problem in the control of external devices, and seriously influences the application range of the brain-computer interface. At the same time, there are differences between the EEG signals of different individuals. Studies show that a decoding rate is low in a process of using the brain-computer interface by about 30% of people, so that it is difficult to realize communication and control between the brain and the external environment. We call this kind of people as “BCI blind”. Therefore, how to improve the practical performance of the decoding rate of BCI and the like is a key problem of the brain-computer interface.
BCI users can initiate brain control by simply executing imagery of left-hand or right-hand motions. The kinesthetic imagery of the hand motion generates event-related desynchronization/event-related synchronization (ERD/ERS) in the sensory motor cortex of the subject. The event-related synchronization is defined as a decrease or increase of power in a specific frequency band (such as α wave band 8-13 Hz). Generally, motor imagery-induced ERD activity focuses the contralateral hemisphere, that is, the motor imagery induces the ERD in the contralateral sensory motor cortex, and these lateral cortex activities constitute the neurophysiological basis of the motor imagery-based brain-computer interface.
A potential method to the problems faced by BCI development is to use the tactile sense. Tactile sensations produce less visual fatigue than visual stimulation, so that the user cannot generate excessive fatigue. The human tactile receptors are mainly Meissner corpuscles, distributed in the superficial skin and sensing the light pressing stimulation of the skin; Meissner's corpuscles, mainly responsible for sensing the tactile sensations; Pacinian corpuscles in a deeper layer, mainly sensing pressing sensations; and Pinkusi corpuscles, many nerve endings, tactile menisci, etc. The Meissner corpuscles are positioned at a tip adjacent to the main ridge and closest to the skin surface. They are very effective in signal conduction of low-frequency vibration (1-40 Hz), and thus achieve an important effect in sensory vibration detection.
More and more studies have shown that non-invasive electrical stimulation can more effectively modulate the neural activities while locking potential brain rhythms. Vibratory stimulation may potentially be in phase with natural oscillations. However, due to complexity and temporal variability of the EEG signals, matching with spontaneous oscillations of the human body is a challenging problem.
SUMMARY
The present invention aims at solving the technical problems to overcome the defects in the prior art and to provide a brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement by aiming at the characteristic problems of nonlinear, non-stable, non-gaussian processes and the like of an electroencephalogram signal per se, thus improving a signal-to-noise ratio of a brain-computer interface system and enhancing a brain source signal.
The present invention solves the above technical problems specifically by the following technical solution:
A brain-computer interface method based on real-time closed loop vibration stimulation enhancement includes the following steps:
displaying and providing a motor imagery task to a subject, and collecting a digital electroencephalogram signal of the subject generated during motor imagery;
reading the collected digital electroencephalogram signal, judging whether a preset time period is exceeded or not, intercepting the digital electroencephalogram signal in the preset time period if YES, and continuously reading the collected digital electroencephalogram signal if NO;
after band-pass filtering is performed on the intercepted digital electroencephalogram signal in the preset time period, obtaining time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform, and extracting a frequency value with highest frequency energy as its main frequency; obtaining an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform on the digital electroencephalogram signal subjected to band-pass filtering; generating predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predicting and obtaining real-time phase information at the current moment according to the predicted sine waves; and judging whether the real-time phase is in a vibration stimulation application phase interval or not according to the predicted and obtained real-time phase information at the current moment, generating and outputting a control instruction according to the judging result, and controlling a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction.
Further, as a preferable technical solution of the present invention, the motor imagery task in the method includes a left-hand or right-hand motor imagery action.
Further, as a preferable technical solution of the present invention, in the method, the intercepted digital electroencephalogram signal in the preset time period is subjected to a waveband band-pass filtering.
Further, as a preferable technical solution of the present invention, for obtaining the instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform in the method, the following formulas are used:
y ( t ) = p · 1 π · v · ∫ - ∞ + ∞ ⁢ x ( t ) t - τ ⁢ d ⁢ ⁢ τ ; and θ x ⁡ ( t ) = tan - 1 ⁢ y ⁡ ( t ) x ⁡ ( t ) ,
wherein y(t) is a digital electroencephalogram signal after Hilbert transform is performed on x(t); x(t) is the digital electroencephalogram signal of the period after the band-pass filtering; p and v are integrals in the Cauchy principal value sense; and θx(t) is an instantaneous phase at a moment t.
Further, as a preferable technical solution of the present invention, the predicted sine wave fs generated in the method is specifically:
f s = sin ⁡ ( 2 · π · f main · t f + θ ⁢ ⁢ x ⁡ ( t ) + π 2 ) ,
wherein fmain is a main frequency of the intercepted digital electroencephalogram signal of the preset time period; tf is a length of a predicted waveform, and θx(t) is an instantaneous phase at the moment t.
The brain-computer interface system based on real-time closed loop vibration stimulation enhancement provided by the present invention includes:
a human-computer interaction module, configured to display and provide a motor imagery task to a subject;
an electroencephalogram signal collection module, configured to collect a digital electroencephalogram signal generated during motor imagery of the subject;
a real-time phase prediction module, configured to read the collected digital electroencephalogram signal, judge whether the preset time period is exceeded or not, intercept the digital electroencephalogram signal in the preset time period if YES, and continuously read the collected digital electroencephalogram signal if NO; and configured to obtain time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform after performing band-pass filtering on the intercepted digital electroencephalogram signal in the preset time period, extract a frequency value with highest frequency energy in the time-frequency characteristics as its main frequency, obtain an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform, generate predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predict and obtain real-time phase information at the current moment according to the predicted sine waves;
an electroencephalogram signal analysis module, configured to judge whether the real-time phase is in a vibration stimulation application phase interval or not according to the predicted and obtained real-time phase information at the current moment, and generate and output a control instruction according to the judging result; and
a vibration stimulation feedback module, configured to control a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction output by the electroencephalogram signal analysis module.
Further, as a preferable technical solution of the present invention, the electroencephalogram signal collection module includes an electroencephalogram cap, an electroencephalogram signal amplifier, a low-pass and band-stop filter, an analog-to-digital conversion module and a communication module in sequential connection.
Further, as a preferable technical solution of the present invention, the real-time phase prediction module uses a band-pass filter to perform band-pass filtering on the digital electroencephalogram signal in the preset time period.
Further, as a preferable technical solution of the present invention, the band-pass filter adopts a ten-order oval infinite impulse response filter.
Further, as a preferable technical solution of the present invention, the vibration stimulation feedback module sets different vibration frequencies to act on the left hand and the right hand according to the control instruction.
By using the above technical solution, the present invention can achieve the following technical effects:
The method and system of the present invention achieve a real-time closed loop vibration stimulation effect so as to enhance a signal-to-noise ratio of the electroencephalogram signal and improve a decoding rate of the motor imagery task by guiding a subject to execute a motor imagery task via display, generating predicted sine waves by using the data period time-frequency characteristics of the collected electroencephalogram signals, then predicting the real-time electroencephalogram signal phase information on the basis of the predicted sine waves, and controlling the vibration motor to apply vibration stimulation on the fingertip of a person by using the predicted instantaneous phase information. The present invention can feedback, regulate and control the electroencephalogram rhythm through vibration stimulation. A signal-to-noise ratio of the brain-computer interface system is improved. A new scheme is provided for enhancing a brain source signal. The recognition rate of the motor imagery signal is enhanced. A “BCI blind” phenomenon is reduced. Therefore, the communication between the user and the outside is more effective, more convenient and faster.
Therefore, the present invention can improve real-time performance of vibration stimulation. The stimulation effect is maximized by concentrating the stimulation on the optimum phase for enhancing the electroencephalogram signal and ensuring the repeating in a plurality of cycles. The decoding rate of the motor imagery task is improved. Differences of the brain-computer interface between individuals are reduced. Compared with a traditional open-loop continuous stimulation method, such a system can realize vibration control at lower power demand and higher specificity, and can reduce tolerance and rebounding risks.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a schematic structure diagram of a brain-computer interface system based on real-time closed-loop vibration stimulation enhancement.
FIG. 2 is a real-time phase prediction flow diagram in a method of the present invention.
FIG. 3 is a schematic diagram of a motor imagery experimental paradigm based on real-time closed-loop stimulation feedback of the present invention.
 DETAILED DESCRIPTION
Implementations of the present invention are described below with reference to the accompanying drawings of this specification.
As shown in FIG. 1, the present invention designs a brain-computer interface system based on real-time closed-loop vibration stimulation enhancement. The system mainly includes: a human-computer interaction module, an electroencephalogram signal collection module, a real-time phase prediction module, an electroencephalogram signal analysis module and a vibration stimulation feedback module. An output end of the electroencephalogram signal collection module is connected with an input end of the real-time phase prediction module. An output end of the real-time phase prediction module is connected with an input end of the electroencephalogram signal analysis module. An output end of the electroencephalogram signal analysis module is connected with the vibration stimulation feedback module. The vibration stimulation feedback module directly acts on the body of a subject, and acts on the left and right hands of the subject in the present embodiment.
The human-computer interaction module consists of a display screen displaying a motor imagery task on a screen, and is configured to display and provide the motor imagery task to the subject and guide the subject to complete the left-hand or right-hand motor imagery task through a visual channel.
The electroencephalogram signal collection module is configured to collect a digital electroencephalogram signal of the subject generated during motor imagery, and mainly includes an electroencephalogram cap, an electroencephalogram signal amplifier, a low-pass and band-stop filter, an analog-to-digital conversion module and a communication module in sequential connection. The electroencephalogram cap is worn on the head of the subject, and collects the electroencephalogram signal of the subject generated during motor imagery. The digital electroencephalogram signal is obtained sequentially through amplification by the electroencephalogram signal amplifier, low-pass filtering by the low-pass and band-stop filter and the analog-to-digital conversion by the analog-to-digital conversion module. The digital electroencephalogram signal is transmitted to the real-time phase prediction module through the communication module.
The real-time phase prediction module is configured to receive the digital electroencephalogram signal transmitted by the electroencephalogram signal collection module through a USB interface, read the collected digital electroencephalogram signal, judge whether a preset time period is exceeded or not, intercept the digital electroencephalogram signal in the preset time period if YES, and continuously read the collected digital electroencephalogram signal if NO; and is configured to obtain time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform after performing band-pass filtering on the intercepted digital electroencephalogram signal in the preset time period, extract a frequency value with highest frequency energy in the time-frequency characteristics as its main frequency, obtain an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform, generate predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predict and obtain real-time phase information at the current moment according to the predicted sine waves to realize the instantaneous phase prediction on the real-time electroencephalogram signal.
The electroencephalogram signal analysis module is configured to judge whether the real-time phase is in a vibration stimulation application phase interval or not according to the real-time phase information predicted and obtained by the real-time phase prediction module at the current moment, and generate and output a control instruction according to the judging result.
The vibration stimulation feedback module is configured to control a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction output by the electroencephalogram signal analysis module, stimulate the fingertip of the subject to further influence the sensory channel of the subject, and regulate the electroencephalogram signal rhythm.
Preferably, the digital electroencephalogram signal includes an imagery fisting motion signal. The real-time phase prediction module of the system of the present invention adopts the band-pass filter to perform band-pass filtering on the digital electroencephalogram signal in the preset time period. The frequency band of the band-pass filtering is a brain activity α frequency band. Further, the band-pass filter adopts a ten-order oval infinite impulse response filter. That is, 500 ms is used as a data segment for real-time analysis. Firstly, the electroencephalogram signal data segment is subjected to band-pass filtering by the ten-order oval infinite impulse response filter to obtain the required digital electroencephalogram signal at an α frequency band and in a range of 8-12 hz. The passband ripple is set to be 0.5 dB. The stopband attenuation is set to be 40 dB. The selected data fragment is short, so the order number will be limited if a finite impulse response (FIR) filter is adopted. An infinite impulse response (IIR) filter is adopted.
Additionally, the vibration stimulation feedback module in the system of the present invention can set different vibration frequencies according to the control instruction to act on the left hand and the right hand. For example, a vibration stimulation signal set in the vibration stimulation module and acting on the left hand is sine waves at a vibration frequency of 22 hz, and a vibration stimulation signal acting on the right hand is sine waves at a vibration frequency of 26 hz. The human body sensory system has different sensitive degrees on the left and the right, so the optimum stimulation effect can be generated in an auxiliary way by setting different vibration frequencies.
The present invention further provides a brain-computer interface method based on real-time closed-loop vibration stimulation enhancement. The method is used for signal processing of the above system, and specifically includes the following steps:
Step 1: The human-computer interaction module displays and provides the motor imagery task to a subject. The motor imagery task may include a left-hand or right-hand motor imagery action. Stimulation displayed in a display in the human-computer interaction module includes three modes, specifically including:
(1) a white cross mark occurs in the right center of a screen, and the subject is relaxed and does not perform imagery;
(2) a red arrow occurs at a left end of the white cross mark, and the subject executes a left-hand motor imagery task; and
(3) a red arrow occurs at a right end of the white cross mark, and the subject executes a right-hand motor imagery task.
While the arrow occurs, the real-time vibration stimulation module is activated. The vibration stimulation module is mainly configured to generate stimulation feedback according to the electroencephalogram signal obtained in real time, thus achieving an effect of enhancing the electroencephalogram signal.
Then, the subject sits on a comfortable chair according to requirements, and wears a 64-lead electroencephalogram cap on the head. A distance between the two eyes and the display is about one meter. The left-hand or right-hand fisting action is imaged according to a pointing direction of the arrow displayed on the display. The real-time electroencephalogram signal data of the user is obtained through the 64-lead electroencephalogram collection cap in the electroencephalogram signal collection module. Sequentially through processing and collection by each component, the digital electroencephalogram signal data generated during motor imagery is obtained.
Step 2: The real-time phase prediction module executes a real-time electroencephalogram signal phase prediction algorithm while the motor imagery task is started. As shown in the figure, a specific process is as follows:
Firstly, the real-time phase prediction module reads the collected digital electroencephalogram signal and judges whether the preset time period is exceeded or not. For example, whether 500 ms is exceeded or not. If 500 ms is exceeded, the digital electroencephalogram signal in the preset time period 500 ms is intercepted for analysis and prediction. If 500 ms is not exceeded, the collected digital electroencephalogram signal is continuously read.
Then, the real-time phase prediction module performs an analysis and prediction process:
The intercepted digital electroencephalogram signal in the preset time period 500 ms is subjected to band-pass filtering to obtain a specific occurrence frequency band of an ERD phenomenon under the motor imagery task, such as the α wave band. The range is 8-12 hz.
For the digital electroencephalogram signal data subjected to band-pass filtering, the time-frequency characteristics of the digital electroencephalogram signal of the period is obtained through calculation by fast Fourier transform, and a frequency value with highest frequency energy is extracted to be used as the main frequency of the digital electroencephalogram signal of the period. In the present embodiment, the frequency value with the highest frequency energy in the signal period is extracted to be used as the main frequency of the signal of the period.
For the digital electroencephalogram signal subjected to band-pass filtering, an instantaneous phase of the digital electroencephalogram signal x(t) of the period is obtained through calculation by Hilbert transform. For the instantaneous phase extraction, through calculation by Hilbert transform, the digital electroencephalogram signal y(t) after Hilbert transform is performed on x(t) is obtained. The following formula is used:
y ( t ) = p · 1 π · v · ∫ - ∞ + ∞ ⁢ x ( t ) t - τ ⁢ d ⁢ ⁢ τ .
The instantaneous phase at the moment t may be calculated through the following formula:
θ x ⁡ ( t ) = tan - 1 ⁢ y ⁡ ( t ) x ⁡ ( t ) ,
wherein θx(t) is an instantaneous phase at the moment t; x(t) is the digital electroencephalogram signal of the period after the band-pass filtering; and p and v are integrals in the Cauchy principal value sense.
Finally, the predicted sine waves are generated by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves. The prediction on the real-time phase is realized according to the real-time phase information predicted and obtained by the predicted sine waves. The sine wave function formula is as follows:
f s = sin ⁡ ( 2 · π · f main · t f + θ ⁢ ⁢ x ⁡ ( t ) + π 2 ) ,
wherein fs is predicted sine waves; fmain is the main frequency of the intercepted digital electroencephalogram signal of the preset time period; tf is a length of a predicted waveform, is set to be 50 ms herein, and may be determined according to the time difference of the selected current time point and the time point of obtaining the instantaneous phase of the electroencephalogram signal; and θx(t) is an instantaneous phase at the moment t.
In the above, the instantaneous phase at a 480 ms position in the 500 ms electroencephalogram signal data is obtained through Hilbert transform, i.e., t=480 ms. The main frequency fmain of the data fragment is obtained through calculation by fast Fourier transform. Through these parameters, a sine wave functional expression can be listed to obtain the precise predicted sine waves. The length tf of the predicted waveform is set to be one-tenth of a length of a reference signal, i.e., 50 ms. A principle is that the electroencephalogram signal is a nonlinear and instable signal, and therefore, 500 ms of data is extracted in per 50 ms fragment for prediction, so as to improve the prediction accuracy.
Step 3: The electroencephalogram signal analysis module judges whether the real-time phase is in a vibration stimulation application phase interval or not according to the real-time phase information predicted and obtained by the real-time phase prediction module at the current moment t. If the judging result is that the real-time phase is in the phase interval, a control instruction is generated and output. The vibration stimulation feedback module controls a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction. Otherwise, if the judging result is that the real-time phase is not in the phase interval, the control instruction is not output, and the vibration stimulation does not need to be generated.
The present invention uses the above method. Its test result is as shown in FIG. 3. About 3 ms is needed from the generation of the electroencephalogram signal, to the recording by the electroencephalogram cap, and then to the entering of the real-time phase prediction module through an amplifier and USB transmission. About 5 ms is needed for a computer to execute a prediction algorithm for data processing. About 2 ms is needed for obtaining the instantaneous phase value and converting the instantaneous phase value into an instruction to control the vibration motor to vibrate. Therefore, the 30th ms position of the predicted sine waves may be approximately regarded as the electroencephalogram signal of the current time point. The instantaneous phase at a 30 ms position is taken to be used as a point for the vibration motor to control a triggering signal, thus achieving the goal of stimulation based on the real-time phase.
Based on the above, the present invention designs the brain-computer interface method and system capable of performing fingertip vibration stimulation so as to enhance the decoding rate of the motor imagery task on the basis of the real-time phase of the electroencephalogram signal. By aiming at the problems of low decoding rate, too great individual differences, “BCI blind” phenomenon and the like of the existing motor imagery brain-computer interface, a method for enhancing the electroencephalogram signal is designed. Compared with the past continuous closed-loop vibration stimulation, the present invention achieves the maximum stimulation effect through cumulative effect by concentrating the stimulation on the optimum phase for enhancing the electroencephalogram signal and ensuring the repeating in a plurality of cycles. Therefore, the vibration control can be realized at lower power demand and higher specificity. Additionally, the tolerance and rebounding risks can be reduced. By feeding back, regulating and controlling the electroencephalogram rhythm through vibration stimulation, the present invention improves a signal-to-noise ratio of the brain-computer interface system, and enhances the recognition rate of a motor imagery signal.
The implementations of the present invention are described above in detail with reference to the accompanying drawings, but the present invention is not limited to the implementations described above. A person of ordinary skill in the art may make various variations without departing from the spirit of the present invention.

Claims (10)

What is claimed is:
1. A brain-computer interface method based on real-time closed loop vibration stimulation enhancement, comprising the following steps:
displaying and providing a motor imagery task to a subject, and collecting a digital electroencephalogram signal of the subject generated during motor imagery;
reading the collected digital electroencephalogram signal, judging whether a preset time period is exceeded or not, intercepting the digital electroencephalogram signal in the preset time period if YES, and continuously reading the collected digital electroencephalogram signal if NO;
after band-pass filtering is performed on the intercepted digital electroencephalogram signal in the preset time period, obtaining time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform, and extracting a frequency value with highest frequency energy as a main frequency; obtaining an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform on the digital electroencephalogram signal subjected to band-pass filtering; generating predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predicting and obtaining real-time phase information at the current moment according to the predicted sine waves; and
judging whether the real-time phase is in a vibration stimulation application phase interval or not according to the predicted and obtained real-time phase information at the current moment, generating and outputting a control instruction according to the judging result, and controlling a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction.
2. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 1, wherein the motor imagery task in the method comprises a left-hand or right-hand motor imagery action.
3. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 1, wherein in the method, the intercepted digital electroencephalogram signal in the preset time period is subjected to α waveband band-pass filtering.
4. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 1, wherein for obtaining the instantaneous phase of the digital electroencephalogram signal through calculation by Hilbert transform in the method, the following formulas are used:
y ( t ) = p · 1 π · v · ∫ - ∞ + ∞ ⁢ x ( t ) t - τ ⁢ d ⁢ ⁢ τ ; and θ x ⁡ ( t ) = tan - 1 ⁢ y ⁡ ( t ) x ⁡ ( t ) ,
wherein y(t) is a digital electroencephalogram signal after Hilbert transform is performed on x(t); x(t) is the digital electroencephalogram signal after the band-pass filtering; p and v are integrals in the Cauchy principal value sense; and θx(t) is an instantaneous phase at a moment t.
5. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 1, wherein the predicted sine wave fs generated in the method is specifically:
f s = sin ⁡ ( 2 · π · f main · t f + θ ⁢ ⁢ x ⁡ ( t ) + π 2 ) ,
wherein fmain is a main frequency of the intercepted digital electroencephalogram signal of the preset time period; tf is a length of a predicted waveform, and θx(t) is an instantaneous phase at the moment t.
6. A brain-computer interface system based on real-time closed loop vibration stimulation enhancement, comprising:
a human-computer interaction module, configured to display and provide a motor imagery task to a subject;
an electroencephalogram signal collection module, configured to collect a digital electroencephalogram signal generated during motor imagery of the subject;
a real-time phase prediction module, configured to read the collected digital electroencephalogram signal, judge whether the preset time period is exceeded or not, intercept the digital electroencephalogram signal in the preset time period if YES, and continuously read the collected digital electroencephalogram signal if NO; and configured to obtain time-frequency characteristics of the digital electroencephalogram signal of the period through calculation by fast Fourier transform after performing band-pass filtering on the intercepted digital electroencephalogram signal in the preset time period, extract a frequency value with highest frequency energy in the time-frequency characteristics as its main frequency, obtain an instantaneous phase of the digital electroencephalogram signal of the period through calculation by Hilbert transform, generate predicted sine waves by respectively using the main frequency and the instantaneous phase of the digital electroencephalogram signal of the period as a frequency and an initial phase of sine waves, and predict and obtain real-time phase information at the current moment according to the predicted sine waves; and
a controller that is configured to function as:
an electroencephalogram signal analysis module, configured to judge whether the real-time phase is in a vibration stimulation application phase interval or not according to the predicted and obtained real-time phase information at the current moment, and generate and output a control instruction according to the judging result; and
a vibration stimulation feedback module, configured to control a vibration motor to vibrate and to stimulate a sensory channel of the subject according to the control instruction output by the electroencephalogram signal analysis module.
7. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 6, wherein the real-time phase prediction module uses a band-pass filter to perform band-pass filtering on the digital electroencephalogram signal in the preset time period.
8. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 7, wherein the band-pass filter adopts a ten-order oval infinite impulse response filter.
9. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 6, wherein the electroencephalogram signal collection module comprises an electroencephalogram cap, an electroencephalogram signal amplifier, a low-pass and band-stop filter, an analog-to-digital conversion module and a communication module in sequential connection.
10. The brain-computer interface method based on real-time closed loop vibration stimulation enhancement according to claim 6, wherein the vibration stimulation feedback module sets different vibration frequencies to act on the left hand and the right hand according to the control instruction.
US16/977,751 2018-12-19 2019-03-21 Brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement Active 2039-06-27 US11379039B2 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
CN201811553136.XA CN109656365B (en) 2018-12-19 2018-12-19 A brain-computer interface method and system based on real-time closed-loop vibration stimulation enhancement
CN201811553136.X 2018-12-19
PCT/CN2019/079096 WO2020124838A1 (en) 2018-12-19 2019-03-21 Brain-computer interface method and system based on real-time closed-loop vibration stimulation enhancement

Publications (2)

Publication Number Publication Date
US20200401226A1 US20200401226A1 (en) 2020-12-24
US11379039B2 true US11379039B2 (en) 2022-07-05

Family

ID=66114660

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/977,751 Active 2039-06-27 US11379039B2 (en) 2018-12-19 2019-03-21 Brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement

Country Status (3)

Country Link
US (1) US11379039B2 (en)
CN (1) CN109656365B (en)
WO (1) WO2020124838A1 (en)

Families Citing this family (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110321691B (en) * 2019-07-30 2022-03-11 东南大学 A user authentication device suitable for a brain-computer interface and an authentication method thereof
CN113208610B (en) * 2020-01-17 2022-06-28 广西医科大学 A phase point estimation method, device and closed-loop neural stimulation system
CN111728822B (en) * 2020-07-24 2021-11-26 清华大学 Brain-computer interactive closed-loop rehabilitation robot control method used after brain injury
CN113180698B (en) * 2021-04-30 2024-03-01 西安臻泰智能科技有限公司 Wireless automatic deviation compensation method of electroencephalogram device and electroencephalogram device
CN113476060B (en) * 2021-06-16 2024-03-08 南京曦光信息科技研究院有限公司 Closed-loop feedback type photoacoustic and electromagnetic integrated brain work rhythm adjusting device
US11689878B2 (en) * 2021-09-07 2023-06-27 Qualcomm Incorporated Audio adjustment based on user electrical signals
CN114259651B (en) * 2022-01-17 2024-11-26 天津大学 An active real-time closed-loop electrical stimulation method and system for Parkinson's disease
CN114533086B (en) * 2022-02-21 2023-10-24 昆明理工大学 A motor imagery EEG decoding method based on spatial feature time-frequency transformation
CN114870249B (en) * 2022-04-18 2023-06-13 北京理工大学 A closed-loop adaptive AC electric stimulation neural network control method and system
CN114783395B (en) * 2022-04-19 2025-08-08 何明宗 An audio production method and terminal based on EEG nonlinear dynamics analysis
CN115089197A (en) * 2022-06-08 2022-09-23 上海暖禾脑科学技术有限公司 A closed-loop feedback optical stimulation system based on gamma EEG signals
KR102799750B1 (en) * 2022-06-14 2025-04-25 울산과학기술원 Apparatus and method for neurofeedback using slow electro encephalography
US20250281750A1 (en) * 2022-06-20 2025-09-11 The Regents Of The University Of California Implantable electrocorticogram brain-computer interface systems for movement and sensation restoration
CN115509347A (en) * 2022-09-05 2022-12-23 同济大学 Walking intention inducing method and device based on emotional brain machine and storage medium
CN115640827B (en) * 2022-09-27 2023-06-27 首都师范大学 Intelligent closed-loop feedback network method and system for processing electrical stimulation data
CN116269447B (en) * 2023-05-17 2023-08-29 之江实验室 A Speech Cognitive Assessment System Based on Speech Modulation and EEG Signals
CN117873330B (en) * 2024-03-11 2024-05-17 河海大学 Electroencephalogram-eye movement hybrid teleoperation robot control method, system and device
CN117899363B (en) * 2024-03-20 2024-05-24 江西朴拙医疗设备有限公司 Closed loop electric stimulation therapeutic instrument combined with ERP
CN118410374B (en) * 2024-03-25 2025-02-11 中国科学院自动化研究所 Method, device and equipment for issuing continuous peripheral control instructions based on electroencephalogram signals

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100042011A1 (en) * 2005-05-16 2010-02-18 Doidge Mark S Three-dimensional localization, display, recording, and analysis of electrical activity in the cerebral cortex
CN103268149A (en) 2013-04-19 2013-08-28 杭州电子科技大学 Real-time active system control method based on brain-computer interface
US20140223462A1 (en) * 2012-12-04 2014-08-07 Christopher Allen Aimone System and method for enhancing content using brain-state data
CN104951082A (en) 2015-07-09 2015-09-30 浙江大学 Brain-computer interface method for intensifying EEG (electroencephalogram) signals through stochastic resonance
US20190082990A1 (en) * 2017-09-19 2019-03-21 Neuroenhancement Lab, LLC Method and apparatus for neuroenhancement
US10898706B2 (en) * 2017-10-31 2021-01-26 Stimscience Inc. Systems, methods, and devices for brain stimulation and monitoring

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101810479B (en) * 2009-12-16 2011-09-14 天津大学 Phase characteristic extraction method for brain waves of compound imaginary movements of lower limbs
CN103869979A (en) * 2014-03-03 2014-06-18 上海交通大学 BCI (Brain-Computer Interface) system based on selective feeling
CN104048677A (en) * 2014-07-03 2014-09-17 哈尔滨工业大学 Gyroscope fault diagnosis method based on K-S (Kolmogorov-Smirnov) distribution check and HHT (Hilbert-Huang Transform)
KR101617005B1 (en) * 2015-06-01 2016-05-11 고려대학교 산학협력단 Apparatus and method for brain-brain interface using brainwave magnetic resonance and transcranial laser
CN105411580B (en) * 2015-10-27 2018-05-11 中国航天员科研训练中心 A kind of brain control wheelchair system based on tactile auditory evoked potential
CN106774851A (en) * 2016-11-25 2017-05-31 华东理工大学 Tactile finger movement rehabilitation system and method based on brain-computer interface
CN106909226B (en) * 2017-03-06 2019-09-24 广东工业大学 A kind of polymorphic brain machine interface system
CN108363493B (en) * 2018-03-20 2021-08-27 山东建筑大学 User characteristic model establishing method and system based on brain-computer interface and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100042011A1 (en) * 2005-05-16 2010-02-18 Doidge Mark S Three-dimensional localization, display, recording, and analysis of electrical activity in the cerebral cortex
US20140223462A1 (en) * 2012-12-04 2014-08-07 Christopher Allen Aimone System and method for enhancing content using brain-state data
CN103268149A (en) 2013-04-19 2013-08-28 杭州电子科技大学 Real-time active system control method based on brain-computer interface
CN104951082A (en) 2015-07-09 2015-09-30 浙江大学 Brain-computer interface method for intensifying EEG (electroencephalogram) signals through stochastic resonance
US20190082990A1 (en) * 2017-09-19 2019-03-21 Neuroenhancement Lab, LLC Method and apparatus for neuroenhancement
US10898706B2 (en) * 2017-10-31 2021-01-26 Stimscience Inc. Systems, methods, and devices for brain stimulation and monitoring

Also Published As

Publication number Publication date
US20200401226A1 (en) 2020-12-24
CN109656365A (en) 2019-04-19
WO2020124838A1 (en) 2020-06-25
CN109656365B (en) 2021-03-30

Similar Documents

Publication Publication Date Title
US11379039B2 (en) Brain-computer interface method and system based on real-time closed loop vibration stimulation enhancement
CN110537929B (en) SSVEP-based attention assessment method, training method and brain-computer interface
CN103699226B (en) A kind of three mode serial brain-computer interface methods based on Multi-information acquisition
CN101464728B (en) A Human-Computer Interaction Method Based on Visual-Motion-Related Neural Signals
CN101339455B (en) Brain-computer interface system based on the N170 component of the specific wave for face recognition
CN104951082B (en) A kind of brain-machine interface method for strengthening EEG signals using accidental resonance
CN101391129B (en) Brain-machine interface intelligentized upper-limb recovery training device based on P300 signal and signal processing method
CN102799267B (en) Multi-brain-computer interface method for three characteristics of SSVEP (Steady State Visual Evoked Potential), blocking and P300
CN106362287A (en) Novel MI-SSSEP mixed brain-computer interface method and system thereof
CN109471530A (en) Brain-controlled input method based on steady-state visual evoked potentials and motor imagery
CN102063180A (en) HHT-based high-frequency combined coding steady state visual evoked potential brain-computer interface method
CN113101021A (en) Mechanical arm control method based on MI-SSVEP hybrid brain-computer interface
WO2025030833A1 (en) Low-training-cost electroencephalogram feature decoding method for steady-state visual evoked potential
CN107957780B (en) A brain-computer interface system based on the physiological properties of steady-state visual evoked potentials
CN111481799A (en) Brain wave closed-loop control equipment
Cotrina et al. Towards an architecture of a hybrid BCI based on SSVEP-BCI and passive-BCI
CN101339413B (en) Switching control method based on brain electric activity human face recognition specific wave
CN102799274A (en) Method of asynchronous brain switch based on steady state visual evoked potentials
Zhang et al. A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode
CN110688013A (en) A kind of English keyboard spelling system and spelling method based on SSVEP
Chumerin et al. Processing and decoding steady-state visual evoked potentials for brain-computer interfaces
CN103246348B (en) Quantitative analysis system and method for electric touches
Manyakov et al. Decoding SSVEP responses using time domain classification
Park et al. Application of EEG for multimodal human-machine interface
CN101571747A (en) Method for realizing multi-mode EEG-control intelligent typewriting

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO SMALL (ORIGINAL EVENT CODE: SMAL); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

AS Assignment

Owner name: SOUTHEAST UNIVERSITY, CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:SONG, AIGUO;ZHANG, WENBIN;ZENG, HONG;AND OTHERS;SIGNING DATES FROM 20200825 TO 20200827;REEL/FRAME:053719/0790

STPP Information on status: patent application and granting procedure in general

Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YR, SMALL ENTITY (ORIGINAL EVENT CODE: M2551); ENTITY STATUS OF PATENT OWNER: SMALL ENTITY

Year of fee payment: 4