CN113918008A - Brain-computer interface system based on source space brain magnetic signal decoding and application method - Google Patents
Brain-computer interface system based on source space brain magnetic signal decoding and application method Download PDFInfo
- Publication number
- CN113918008A CN113918008A CN202111004557.9A CN202111004557A CN113918008A CN 113918008 A CN113918008 A CN 113918008A CN 202111004557 A CN202111004557 A CN 202111004557A CN 113918008 A CN113918008 A CN 113918008A
- Authority
- CN
- China
- Prior art keywords
- brain
- signals
- real
- time
- decoding
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 210000004556 brain Anatomy 0.000 title claims abstract description 88
- 238000000034 method Methods 0.000 title claims abstract description 21
- 230000008035 nerve activity Effects 0.000 claims abstract description 27
- 238000010223 real-time analysis Methods 0.000 claims abstract description 13
- 230000000638 stimulation Effects 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 210000005036 nerve Anatomy 0.000 claims abstract description 5
- 230000001939 inductive effect Effects 0.000 claims abstract description 3
- 238000012545 processing Methods 0.000 claims abstract description 3
- 230000001537 neural effect Effects 0.000 claims description 22
- 210000003128 head Anatomy 0.000 claims description 16
- 239000000523 sample Substances 0.000 claims description 14
- 238000013528 artificial neural network Methods 0.000 claims description 13
- 210000003792 cranial nerve Anatomy 0.000 claims description 12
- 230000008569 process Effects 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- 238000012549 training Methods 0.000 claims description 8
- 230000004886 head movement Effects 0.000 claims description 7
- 238000002582 magnetoencephalography Methods 0.000 claims description 7
- 230000000306 recurrent effect Effects 0.000 claims description 7
- 210000004761 scalp Anatomy 0.000 claims description 7
- 230000033001 locomotion Effects 0.000 claims description 6
- 210000003625 skull Anatomy 0.000 claims description 6
- 230000033228 biological regulation Effects 0.000 claims description 4
- 230000001054 cortical effect Effects 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 210000004884 grey matter Anatomy 0.000 claims description 4
- 230000006698 induction Effects 0.000 claims description 4
- 210000000976 primary motor cortex Anatomy 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 125000004122 cyclic group Chemical group 0.000 claims description 3
- 230000007613 environmental effect Effects 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000000926 separation method Methods 0.000 claims description 3
- 210000000467 autonomic pathway Anatomy 0.000 claims description 2
- 230000004424 eye movement Effects 0.000 claims description 2
- 238000012880 independent component analysis Methods 0.000 claims description 2
- 238000000605 extraction Methods 0.000 abstract description 4
- 238000004458 analytical method Methods 0.000 abstract description 3
- 230000001149 cognitive effect Effects 0.000 description 6
- 241000238366 Cephalopoda Species 0.000 description 5
- 230000003993 interaction Effects 0.000 description 4
- 238000002610 neuroimaging Methods 0.000 description 3
- 230000002238 attenuated effect Effects 0.000 description 2
- 230000003925 brain function Effects 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 230000001771 impaired effect Effects 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000005057 refrigeration Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 208000000094 Chronic Pain Diseases 0.000 description 1
- 208000021384 Obsessive-Compulsive disease Diseases 0.000 description 1
- 208000002193 Pain Diseases 0.000 description 1
- 206010033799 Paralysis Diseases 0.000 description 1
- 208000006011 Stroke Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 150000001340 alkali metals Chemical group 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 210000003710 cerebral cortex Anatomy 0.000 description 1
- 230000037410 cognitive enhancement Effects 0.000 description 1
- 230000006998 cognitive state Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 230000001037 epileptic effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 229910052734 helium Inorganic materials 0.000 description 1
- 239000001307 helium Substances 0.000 description 1
- SWQJXJOGLNCZEY-UHFFFAOYSA-N helium atom Chemical compound [He] SWQJXJOGLNCZEY-UHFFFAOYSA-N 0.000 description 1
- 238000002513 implantation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 230000003340 mental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003183 myoelectrical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 210000004092 somatosensory cortex Anatomy 0.000 description 1
- 208000020431 spinal cord injury Diseases 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/242—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents
- A61B5/245—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents specially adapted for magnetoencephalographic [MEG] signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
Abstract
The invention discloses a brain-computer interface system based on source space brain magnetic signal decoding and an application method. The system comprises a brain magnetic signal acquisition device, a data acquisition workstation and a brain magnetic signal acquisition module, wherein the brain magnetic signal acquisition device is worn on the head of a subject, acquires brain magnetic signals of the subject and sends the brain magnetic signals to the data acquisition workstation; the data acquisition workstation is used for synchronously receiving the multi-channel brain magnetic signals acquired by the brain magnetic signal acquisition device and sending the multi-channel brain magnetic signals to the real-time analysis workstation; the real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the received brain magnetic signals and sending decoding information to the multi-modal stimulation presentation device and the external controlled equipment; the multi-modal stimulation presenting device is used for presenting stimulation information for inducing the brain nerve activity of the subject or nerve feedback signals decoded by the real-time analysis workstation; and the external controlled equipment is used for carrying out corresponding processing according to the received decoding signal. The invention can realize the real-time extraction, pretreatment and traceability analysis of the multi-channel high-flux whole brain nerve activity magnetic signal.
Description
Technical Field
The invention relates to the field of brain-computer interface research, in particular to a brain-computer interface system for realizing direct interaction with the outside and closed-loop neural regulation by decoding a traceable brain magnetic signal and a using method thereof.
Background
By detecting and decoding Brain nerve activity, a Brain-Computer Interface (BCI) can directly establish a connection channel between a human Brain and external equipment, and a Brain-Computer interaction nerve signal closed loop is constructed. With the rapid development of wearable brain imaging devices towards miniaturization and portability, brain-computer interfaces have become hot spots of international science and industrial layout, and are an important competitive field of the core strength of various science and technology republic of China.
Early brain-computer interface applications were primarily one-way connections from the human brain to external devices, such as those with impaired or severely impaired motor skills (e.g., stroke, spinal cord injury, paralyzed patients) to reestablish connections to the outside world, typically by decoding the neural activity of the brain motor sensory cortex to manipulate the external device to effect interaction with the outside world. In recent years, the application scenes of brain-computer interface technology are greatly expanded by the rapid development of brain imaging and artificial intelligence technology, so that the brain-computer interface application is raised from the aspects of feeling and perception to the aspect of advanced cognitive activities. By constructing real-time bidirectional connection between the human brain and the computer, the human brain can directly control external equipment, and conversely, the brain activity can be monitored and controlled. By automatically monitoring the human brain for higher level cognitive activities and states (including internal speech, emotional tendencies, cognitive load, and attentional state, etc.). Furthermore, brain-computer interface technology can also be used for neurorehabilitation, improvement of mental symptoms (obsessive-compulsive disorder, chronic pain), and cognitive enhancement by closed-loop neurofeedback (closed-loop neurofeedback). Implementing these advanced brain-computer interface applications requires accurate localization, extraction, and decoding of neural activity signals of the relevant brain regions or brain networks in real time throughout the brain.
The brain function imaging technology is the basis of a brain-computer interface, and the accurate detection of the cranial nerve activity with high complexity and dynamic characteristics is the premise of realizing efficient human-computer interaction. Clinically, the cerebral cortex placement/implantation electrode recording nerve activity under the skull has the advantages of high time and space resolution, but the electrode recording nerve activity can only be used for special patient groups (such as epileptics) as an invasive measurement means. In addition, limited by clinical requirements and safety considerations, implanted electrodes (arrays) can only cover local brain regions, cannot achieve synchronous recording of global brain neural activity, and are difficult to use in brain-computer interface applications involving multi-brain region advanced cognitive activities. The electroencephalogram can record the electrical signals generated by the cranial nerve activity directly outside the scalp in a non-invasive manner, and has the characteristics of high time resolution, good wearability and good portability. However, due to the characteristics of poor conductivity and anisotropy of brain media, the propagation of electrical signals of neural activity in the brain is greatly distorted and lost, so that the electroencephalogram has low signal-to-noise ratio, limited effective signal bandwidth (signals above 50Hz are seriously attenuated after passing through the skull), poor spatial resolution, difficulty in accurately tracing the brain region where the neural activity occurs and providing spatial information, and limitation of the application scene of the electroencephalogram as a brain-computer interface. In contrast, the conduction of the brain neural activity magnetic signal is hardly influenced by the brain medium, so that the specific position of the neural activity can be traced accurately, and the neural activity decoding which takes time and spatial resolution into account can be completed in the signal source space. The non-invasiveness, the ultrahigh time resolution and the excellent space resolution after tracing of the magnetoencephalogram make the magnetoencephalogram an ideal brain function imaging technology for advanced brain-computer interface applications.
Traditional magnetoencephalograms based on Superconducting Quantum interferometers (SQUIDs) require maintaining low temperature Superconducting operating conditions. Therefore, because of the fixed position of the SQUID magnetoencephalogram detector, the SQUID magnetoencephalogram detector cannot be flexibly adjusted to be close to the scalp, and the magnetic signal of the detected nerve activity is seriously attenuated in the propagation process, so that the magnetoencephalogram signal from weak nerve activity and deep brain area cannot be effectively detected. In addition, the bulky volume of the liquid helium refrigeration equipment cannot meet the requirement of a brain-computer interface on wearable portability. In recent years, miniaturization of ultra-high sensitivity Atomic Magnetometers (AM) has made portable wearable magnetoencephalograms possible. The AM magnetoencephalogram can detect a weak magnetic field under the room temperature environment based on the action of alkali metal atoms and laser, and has higher sensitivity. In addition, because the AM magnetoencephalogram detector array is not limited by huge refrigeration equipment any more, the position of the probe on the AM magnetoencephalogram detector array can be flexibly adjusted to be close to the scalp, so that the attenuation of magnetoencephalogram signals in the propagation process is reduced to the maximum extent, and the weak nerve activity and deep brain region nerve activity can be effectively detected. Meanwhile, the wearable AM magnetoencephalogram system also allows a user to move freely, and is more suitable for real life scenes compared with SQUID magnetograms.
In summary, as a non-invasive brain imaging technique covering the entire brain, the detection of cranial nerve activity by magnetoencephalogram can be done with both temporal and spatial resolution. The brain-computer interface based on the wearable AM magnetoencephalogram can be applied to clinical and cognitive neuroscience under complex scenes, but at present, a complete and practical brain-computer interface system scheme and a use method based on AM magnetoencephalogram decoding in a source space are not provided.
Disclosure of Invention
In view of the above, the present invention provides a brain-computer interface system based on source space brain magnetic signals and an application method thereof, which can realize real-time extraction, preprocessing and traceability analysis of multichannel high-flux whole brain neural activity magnetic signals, and realize various brain-computer interface applications by using source space brain magnetic signals with both spatial specificity and temporal resolution.
The technical scheme adopted by the invention is as follows:
a brain-computer interface system for decoding brain magnetic signals based on a source space comprises the following modules:
the brain magnetic signal acquisition device comprises a detector and matched devices (such as a digital signal processing acquisition card, a front/back end amplifier and the like) for acquiring magnetic signals generated by brain nerve activity; the brain magnetic signal acquisition device can be realized by a conventional superconducting quantum interferometer magnetometer or a new generation atomic magnetometer; the surface of the brain magnetic signal acquisition device is provided with a plurality of detectors, each detector provides a signal channel, and signals acquired by the detectors form a multi-channel brain magnetic signal and are sent to the data acquisition workstation;
magnetic shielding means for shielding an environmental magnetic field (e.g. earth magnetic field, interfering magnetic signals of electronic devices) not related to cranial nerve activity;
the data acquisition workstation is used for synchronously receiving the multi-channel brain magnetic signals acquired by the brain magnetic signal acquisition device, cutting and packaging continuous data streams and sending the continuous data streams to the real-time analysis workstation;
the real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the brain magnetic signals and sending decoding information to the multi-modal stimulation presentation device;
a multi-modal stimulation presentation device for presenting stimulation information for inducing brain neural activity to a subject or presenting neural feedback signals (such as neural activity intensity of a specific brain region, control intention of the subject, etc.) decoded by a real-time analysis workstation, wherein the stimulation information or feedback signals can be presented through a plurality of modalities (such as vision, hearing and touch);
an external controlled device, an external device (e.g., a prosthetic, a speech synthesizer, etc.) controlled by the decoded neural signal.
The relationship between the modules of the hardware system is shown in FIG. 1, and the signal transmission adopts TCP/IP protocol. Based on the system, the control of external equipment can be realized, and a nerve feedback closed loop for nerve regulation/rehabilitation can also be realized. In actual use, the implementation of each module can be flexibly adjusted according to actual conditions and requirements.
Preferably, the brain magnetic signal acquisition device uses a wearable magnetoencephalogram based on an atomic magnetometer without Spin Exchange Relaxation Free (SERF) principle, but the present system is also applicable to a conventional SQUID magnetogram.
Preferably, the magnetic shielding device can be a multi-layer magnetic shielding barrel with better shielding effect or a magnetic shielding room providing free movement space. For wearable AM magnetoencephalograms, an active compensation coil can also be added to each AM probe to cancel the ambient residual magnetic interference signal due to motion.
Further, the data acquisition workstation synchronously integrates the received multichannel brain magnetic signals (the sampling rate is more than 1000Hz, and the channel number is more than 96) and the stimulation time sequence information in a cache; the real-time analysis workstation adopts GPU acceleration optimization, and can realize neural decoding based on a deep neural network complex model.
Preferably, the multi-modal stimulation presentation device can simulate multi-sensory stimulation signals in a real environment in a virtual reality manner and perform neural feedback in the same manner.
Further, the real-time analysis workstation pre-processes the brain magnetic signals in real time and decodes the brain magnetic signals in the source space as follows (as shown in fig. 2):
1) acquiring a magnetic resonance head structure image of a user, and constructing a head model comprising a scalp, the inner and outer surfaces of a skull and cortical gray matter;
2) constructing a source space of cranial nerve activity on the basis of a head model, and calculating a transfer matrix from the source space to a probe array space by combining the relative positions of the head of a user and a magnetoencephalogram probe array;
3) performing Signal-space separation (Signal-space separation) on multi-channel brain magnetic signals (with a time window not exceeding 50 ms) received from a data acquisition workstation, and removing environmental interference signals except for brain nerve activity;
4) further preprocessing the multichannel brain magnetic signals, including band-stop filtering for removing power frequency (50Hz) in a time domain, band-pass filtering for extracting required frequency components, independent component analysis for removing eye movement, electrocardio and myoelectric noise near a probe, and compensating the influence of head movement of a user by utilizing head position information recorded in real time;
5) calculating a noise covariance matrix of a probe by calculating the blank acquired magnetoencephalogram data under the condition of no user, and performing real-time magnetoencephalography signal tracing by combining a transfer matrix and adopting Minimum Norm Estimate (MNE);
6) decoding the neural activity signal of the source space, extracting the intention of the user for controlling external equipment, or directly feeding the neural activity signal back to the user for autonomic nerve regulation and control.
The brain-computer interface system based on source space brain magnetic signal decoding has the following advantages:
1. the system can realize real-time extraction, pretreatment and traceability analysis of the whole brain nerve activity magnetic signal, and the magnetoencephalogram is used as a non-invasive measurement means and is suitable for almost all healthy people and clinical patients;
2. the traced brain magnetic signals have both space specificity and high time resolution, and the neural activity signals of specific brain regions/networks or specific frequency components can be used in a targeted manner by developing neural decoding in the source space, so that the decoding accuracy is further improved, and the application range of the brain-computer interface is expanded.
3. By combining the wearable magnetoencephalogram and the virtual reality equipment, the system can restore the ecological validity (ecological validity) of the use scene to the maximum extent, and can meet various brain-computer interface application requirements of cognitive neuroscience research, auxiliary clinical neurorehabilitation and the like in a real scene.
Drawings
FIG. 1 is a schematic diagram of the hardware system of the present invention;
FIG. 2 is a flow chart of real-time tracing and decoding of brain magnetic signals;
fig. 3 is a schematic diagram of the application of the brain-computer interface for the source space brain magnetic signal real-time decoding language.
Detailed Description
In the following description, the brain-computer interface system of the present invention is further described in terms of specific embodiments to facilitate a more thorough understanding of the features and advantages of the invention by those skilled in the art. It should be noted that the following description is merely representative of one exemplary application. It is to be understood that the invention is not limited to any specific structure, function, device or method described herein as it may have other embodiments or combinations of embodiments. The software/hardware modules described in this disclosure or shown in the drawings may also be flexibly adjusted as desired.
According to one embodiment of the invention, the brain-computer interface application of the language is decoded in real time through the brain magnetic signals in the source space, and the application comprises a preparation phase, a training phase and an application phase.
Preparation phase
And customizing the magnetoencephalo-cap carrying the AM probe according to the shape of the head of the user. Collecting user high-definition magnetic resonance head structure diagramImage (spatial resolution at least 1x 1x 1 cm)3) A head model including the scalp, the inner and outer surfaces of the skull, and cortical gray matter was constructed by the Boundary Element Method (BEM). The signal (not less than 2 minutes) of the magnetoencephalogram at no-load is collected and used for calculating the noise covariance matrix of the probe array.
2000 sentences (a single sentence does not exceed 20 characters) used in daily conversations are extracted from a corpus, corresponding pinyin is transcribed for each sentence, and a training expectation set corresponding to the pinyin is constructed (for example, "which is you going in the morning.
Training phase
The user wears the magnetoencephalo-cap (no less than 306 channels) carrying the AM probe in the magnetic shielding environment, and the head movement of the user is recorded in real time through the position coil on the magnetoencephalo-cap, so that the influence of the correction compensation head movement on the magnetoencephalo-signal is compensated.
The corpus is divided into 10 trials, each trial comprising 200 sentences and their corresponding pinyins. The method comprises the steps of displaying a sentence and a corresponding pinyin form on a display in a visual input mode, watching a screen by a user, imagining that pinyin letters of each character are written in sequence, enabling all parts of a body to be free of movement, and recording the cranial nerve activity magnetic signals (namely the cranial magnetic signals) in the process in real time by a brain magnetic map in the whole process. Because the length of each sentence in the corpus is different, each sentence has no fixed imagination writing time, and a user can enter the next sentence through key feedback after finishing the operation.
In addition to the corpus, the user also needs to imagine writing 26 pinyin letters and four symbols (comma "," ", period" - ", space" _ "and question mark". Each character is repeated imaginative writing 50 times for a total of 1500 characters. The sequence was randomly scrambled and divided into 10 trials, each trial comprising 150 characters. The magnetic signal of cranial nerve activity like writing characters is recorded by the full-time real-time magnetoencephalogram.
After all trials are completed, magnetic signals of cranial nerve activity will be obtained, which imagine writing 2000 sentences and 1450 characters. Tracing the brain magnetic signals and extracting the time sequence of the brain magnetic signals of each voxel in the primary motor cortex, the primary body induction cortex and the motor auxiliary area (N voxels in total, as shown in figure 3) related to the imagination writing of the left and right hemispheres of the brain. Using the time series of the magnetoencephalo-signal for each voxel in these brain regions (i.e. the time series of the magnetoencephalo-signal for each voxel in the left and right hemispheres of the brain associated with imagination writing, primary somatosensory cortex, and motor-assisted region), a Recurrent Neural Network (RNN) can be trained, which inputs the time series of the magnetoencephalo-signal in 50ms time windows and outputs the probabilities p (X) for each of the 26 pinyin characters and the 4 punctuation marks, where X is any character. The training of this recurrent neural network is performed on-line.
Stage of actual use
In actual use, since the wearing position of the magnetoencephalophone cap cannot be strictly ensured to be the same every time, the system calibration needs to be performed first. Specifically, a user imagines and writes 26 phonetic letters and 4 punctuations one by one according to screen prompts after wearing a magnetoencephalo-cap, and magnetoencephalo signals acquired in the process are used for calibrating a pre-trained recurrent neural network after tracing.
After the calibration is completed, for any sentence desired to be expressed, the user can imagine writing the pinyin corresponding to the sentence. The brain-computer interface system can record the brain magnetic signals of a user during imagination writing in real time, trace the source by taking a time window of 50 mm as a unit, and extract the signals of each voxel in a primary motor cortex, a primary body induction cortex and a motion auxiliary area which are related to imagination writing in a source space. These traced signals are immediately input into the trained recurrent neural network, so as to obtain the probability of writing a certain character in the current time window, and the character with the highest probability is output in all the characters exceeding the probability threshold (for example, 0.4) (the system judges whether the current input is a new character according to the probability-time curve of each character). If no characters exceed the probability threshold in the time window, no output is made.
The original signal output by the recurrent neural network may be decoded with errors in the individual pinyin characters, such as "nj _ hao" shown in fig. 3. Therefore, the system further adopts a deep neural network language model based on big data pre-training to carry out real-time correction on pinyin obtained by decoding the cyclic neural network, corrects individual characters with decoding errors, supplements pinyin tone by using context information, and finally outputs characters.
The foregoing embodiments are merely illustrative of the principles and utilities of the present invention and are not intended to limit the invention. Modifications and variations can be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the present invention. Therefore, the scope of the invention should be determined from the following claims.
Claims (9)
1. A brain-computer interface system based on source space brain magnetic signal decoding is characterized by comprising
The brain magnetic signal acquisition device is worn on the head of the subject, acquires brain magnetic signals generated by the brain nerve activity of the subject and sends the brain magnetic signals to the data acquisition workstation;
the data acquisition workstation is used for synchronously receiving the multi-channel brain magnetic signals acquired by the brain magnetic signal acquisition device and sending the multi-channel brain magnetic signals to the real-time analysis workstation;
the real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the received brain magnetic signals and sending decoding information to the multi-modal stimulation presentation device and the external controlled equipment;
the multi-modal stimulation presenting device is used for presenting stimulation information inducing the brain nerve activity of the subject or presenting nerve feedback signals decoded by the real-time analysis workstation;
and the external controlled equipment is used for carrying out corresponding processing according to the received decoding signal.
2. The brain-computer interface system according to claim 1, further comprising a magnetic shielding device for providing shielding of the subject from ambient magnetic fields not associated with cranial nerve activity.
3. The brain-computer interface system of claim 2, wherein said magnetic shielding means is a multi-layered magnetic shielding bucket or a magnetic shielding room providing free movement space.
4. The brain-computer interface system according to claim 1, 2 or 3, wherein the real-time analysis workstation performs real-time preprocessing, tracing and decoding on the brain magnetic signals by:
1) acquiring a magnetic resonance head structure image of a subject, and constructing a head model including a scalp, the inner and outer surfaces of a skull and cortical gray matter;
2) constructing a source space of cranial nerve activity on the basis of a head model, and calculating a transfer matrix from the source space to a magnetoencephalogram probe array space by combining the head of a subject and the relative position of the magnetoencephalogram probe array of a magnetoencephalogram signal acquisition device;
3) performing signal space separation on the multi-channel brain magnetic signals to remove environmental interference signals except brain nerve activity;
4) preprocessing a multi-channel magnetoencephalography signal, including band-stop filtering for removing power frequency in a time domain, band-pass filtering for extracting required frequency components, independent component analysis for removing eye movement, electrocardio and electromyographic noise near a probe, and compensating the influence of head movement of a testee by utilizing head position information recorded in real time;
5) calculating a noise covariance matrix of a magnetoencephalogram probe of a magnetoencephalogram signal acquisition device according to magnetoencephalogram data acquired in an empty mode under the condition of no subject, and performing real-time magnetoencephalogram signal tracing by adopting minimum norm estimation in combination with the transfer matrix;
6) decoding the neural activity signal of the source space, extracting decoding information for controlling external controlled equipment, or feeding back the neural activity signal obtained by decoding to the subject for autonomic nerve regulation and control.
5. A brain-computer interface system according to claim 1, 2 or 3 wherein the data acquisition workstation slit-packs the received continuous data stream and transmits it to the real-time analysis workstation.
6. The brain-computer interface system of claim 1, wherein said externally controlled device comprises a prosthetic, a voice synthesizer.
7. The brain-computer interface system according to claim 1, wherein said brain magnetic signal acquisition means employs an atomic magnetometer based on the principle of spin-free exchange relaxation.
8. A brain-computer interface system application method based on source space brain magnetic signal decoding comprises the following steps:
1) a preparation stage: acquiring a magnetic resonance head structure image of a user by using a brain magnetic cap, and constructing a head model including a scalp, the inner and outer surfaces of a skull and cortical gray matter; calculating a probe array noise covariance matrix according to signals acquired when the magnetoencephalogram is in no-load; constructing a training expected set corresponding to characters and pinyin;
2) a training stage: the head movement of the user is recorded in real time by utilizing a position coil on the magnetoencephalography cap worn by the user, and the head movement is used for correcting and compensating the influence of the head movement on signals; dividing the training corpus into M trial times, wherein each trial time comprises a plurality of sentences and corresponding pinyin; then, a sentence and a corresponding pinyin form are presented on a display, and magnetic signals of cranial nerve activity in the process that a user watches a screen to imagine that pinyin letters of each character are written in sequence are collected; collecting magnetic signals of cranial nerve activity in the process that a user imagines to write all phonetic letters and set punctuation marks; then, extracting the time sequence of the magnetoencephalo signals of each voxel in the primary motor cortex, the primary body induction cortex and the motor auxiliary area, which are related to imagination writing, of the left and right hemispheres of the brain according to the tracing of the acquired magnetoencephalo signals; then using the extracted time sequence of the magnetoencephalography signal to train a cyclic neural network, inputting the magnetoencephalography signal in a time window with a set length, and outputting the probabilities of all the pinyin letters and set punctuation marks;
3) the use stage is as follows: tracing according to the acquired magnetoencephalography signals of the process of imagining and writing each phonetic alphabet and each set punctuation mark one by one according to screen prompts after a user wears a magnetoencephalography cap, and calibrating the cyclic neural network; then for a sentence i which is expressed unexpectedly, brain magnetic signals of a Chinese pinyin process corresponding to the sentence i which is imaginarily written by a user are collected and traced to obtain signals of each voxel in a primary motor cortex, a primary body induction cortex and a motion auxiliary area which are related to imagination writing in a source space; and then inputting the tracing signals into the calibrated recurrent neural network to obtain the probability of the characters written in the current time window, and outputting the characters with the maximum probability.
9. The system of claim 8, wherein the pinyin decoded by the recurrent neural networks is corrected in real time using a deep neural network language model pre-trained on the basis of big data, the individual characters decoded incorrectly are corrected, the pinyin tones are supplemented with contextual information, and words are output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111004557.9A CN113918008A (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111004557.9A CN113918008A (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113918008A true CN113918008A (en) | 2022-01-11 |
Family
ID=79233477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111004557.9A Pending CN113918008A (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113918008A (en) |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052668A (en) * | 2022-01-17 | 2022-02-18 | 北京航空航天大学杭州创新研究院 | Brain function analysis method based on magnetoencephalogram data |
| CN115018018A (en) * | 2022-08-05 | 2022-09-06 | 北京航空航天大学杭州创新研究院 | Double spatial filtering method for inhibiting background noise |
| CN115344122A (en) * | 2022-08-15 | 2022-11-15 | 中国科学院深圳先进技术研究院 | Sound wave non-invasive brain-computer interface and control method |
| CN115359236A (en) * | 2022-10-19 | 2022-11-18 | 之江实验室 | Work memory task magnetoencephalogram classification system based on machine learning |
| CN116088686A (en) * | 2023-01-17 | 2023-05-09 | 北京融域智胜科技有限公司 | Electroencephalogram tracing motor imagery brain-computer interface training method and system |
| CN116369950A (en) * | 2023-05-25 | 2023-07-04 | 武汉理工大学 | Target detection method based on electroencephalogram tracing and multi-feature extraction |
| CN116491960A (en) * | 2023-06-28 | 2023-07-28 | 南昌大学第一附属医院 | Brain transient monitoring device, electronic device, and storage medium |
| CN117100277A (en) * | 2022-07-25 | 2023-11-24 | 北京大学 | Multi-mode synchronous brain imaging method based on magnetoencephalography and near infrared brain imaging |
| WO2023246226A1 (en) * | 2022-06-24 | 2023-12-28 | 清华大学 | Computing system based on brain-computer interface, and electronic device |
| CN117100277B (en) * | 2022-07-25 | 2024-04-30 | 北京大学 | Multi-mode synchronous brain imaging method based on magnetoencephalography and near infrared brain imaging |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120022391A1 (en) * | 2010-07-22 | 2012-01-26 | Washington University In St. Louis | Multimodal Brain Computer Interface |
| US20140058528A1 (en) * | 2010-05-05 | 2014-02-27 | University Of Maryland, College Park | Time Domain-Based Methods for Noninvasive Brain-Machine Interfaces |
| US20160174862A1 (en) * | 2013-09-11 | 2016-06-23 | Korea Research Institute Of Standards And Science | Magnetoencephalography measuring apparatus and magnetoencephalography measuring method |
| CN109199376A (en) * | 2018-08-21 | 2019-01-15 | 北京工业大学 | The coding/decoding method of Mental imagery EEG signals based on the imaging of OA-WMNE brain source |
| CN109567785A (en) * | 2018-12-05 | 2019-04-05 | 北京昆迈生物医学研究院有限公司 | A kind of quantum magneticencephalogram system and method based on magnetic shielding cylinder |
| CN111543949A (en) * | 2020-05-13 | 2020-08-18 | 北京航空航天大学 | Child ASD diagnosis device based on magnetoencephalogram and electroencephalogram |
| JP2020124445A (en) * | 2019-02-06 | 2020-08-20 | 株式会社リコー | Information display device, information display method, and information display system |
| CN111973180A (en) * | 2020-09-03 | 2020-11-24 | 北京航空航天大学 | Brain structure imaging system and method based on MEG and EEG fusion |
| CN113143293A (en) * | 2021-04-12 | 2021-07-23 | 天津大学 | Continuous speech envelope nerve entrainment extraction method based on electroencephalogram source imaging |
| US20210244330A1 (en) * | 2020-02-12 | 2021-08-12 | Hi Llc | Estimating the magnetic field at distances from direct measurements to enable fine sensors to measure the magnetic field from the brain using a neural detection system |
-
2021
- 2021-08-30 CN CN202111004557.9A patent/CN113918008A/en active Pending
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140058528A1 (en) * | 2010-05-05 | 2014-02-27 | University Of Maryland, College Park | Time Domain-Based Methods for Noninvasive Brain-Machine Interfaces |
| US20120022391A1 (en) * | 2010-07-22 | 2012-01-26 | Washington University In St. Louis | Multimodal Brain Computer Interface |
| US20160174862A1 (en) * | 2013-09-11 | 2016-06-23 | Korea Research Institute Of Standards And Science | Magnetoencephalography measuring apparatus and magnetoencephalography measuring method |
| CN109199376A (en) * | 2018-08-21 | 2019-01-15 | 北京工业大学 | The coding/decoding method of Mental imagery EEG signals based on the imaging of OA-WMNE brain source |
| CN109567785A (en) * | 2018-12-05 | 2019-04-05 | 北京昆迈生物医学研究院有限公司 | A kind of quantum magneticencephalogram system and method based on magnetic shielding cylinder |
| JP2020124445A (en) * | 2019-02-06 | 2020-08-20 | 株式会社リコー | Information display device, information display method, and information display system |
| US20210244330A1 (en) * | 2020-02-12 | 2021-08-12 | Hi Llc | Estimating the magnetic field at distances from direct measurements to enable fine sensors to measure the magnetic field from the brain using a neural detection system |
| CN111543949A (en) * | 2020-05-13 | 2020-08-18 | 北京航空航天大学 | Child ASD diagnosis device based on magnetoencephalogram and electroencephalogram |
| CN111973180A (en) * | 2020-09-03 | 2020-11-24 | 北京航空航天大学 | Brain structure imaging system and method based on MEG and EEG fusion |
| CN113143293A (en) * | 2021-04-12 | 2021-07-23 | 天津大学 | Continuous speech envelope nerve entrainment extraction method based on electroencephalogram source imaging |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052668A (en) * | 2022-01-17 | 2022-02-18 | 北京航空航天大学杭州创新研究院 | Brain function analysis method based on magnetoencephalogram data |
| WO2023246226A1 (en) * | 2022-06-24 | 2023-12-28 | 清华大学 | Computing system based on brain-computer interface, and electronic device |
| CN117100277A (en) * | 2022-07-25 | 2023-11-24 | 北京大学 | Multi-mode synchronous brain imaging method based on magnetoencephalography and near infrared brain imaging |
| CN117100277B (en) * | 2022-07-25 | 2024-04-30 | 北京大学 | Multi-mode synchronous brain imaging method based on magnetoencephalography and near infrared brain imaging |
| CN115018018A (en) * | 2022-08-05 | 2022-09-06 | 北京航空航天大学杭州创新研究院 | Double spatial filtering method for inhibiting background noise |
| CN115018018B (en) * | 2022-08-05 | 2022-11-11 | 北京航空航天大学杭州创新研究院 | Double spatial filtering method for inhibiting background noise |
| CN115344122A (en) * | 2022-08-15 | 2022-11-15 | 中国科学院深圳先进技术研究院 | Sound wave non-invasive brain-computer interface and control method |
| CN115359236A (en) * | 2022-10-19 | 2022-11-18 | 之江实验室 | Work memory task magnetoencephalogram classification system based on machine learning |
| CN115359236B (en) * | 2022-10-19 | 2023-03-28 | 之江实验室 | Work memory task magnetoencephalogram classification system based on machine learning |
| CN116088686A (en) * | 2023-01-17 | 2023-05-09 | 北京融域智胜科技有限公司 | Electroencephalogram tracing motor imagery brain-computer interface training method and system |
| CN116088686B (en) * | 2023-01-17 | 2023-07-25 | 北京融域智胜科技有限公司 | Electroencephalogram tracing motor imagery brain-computer interface training method and system |
| CN116369950B (en) * | 2023-05-25 | 2024-01-26 | 武汉理工大学 | Target detection method based on electroencephalogram tracing and multi-feature extraction |
| CN116369950A (en) * | 2023-05-25 | 2023-07-04 | 武汉理工大学 | Target detection method based on electroencephalogram tracing and multi-feature extraction |
| CN116491960B (en) * | 2023-06-28 | 2023-09-19 | 南昌大学第一附属医院 | Brain transient monitoring device, electronic device, and storage medium |
| CN116491960A (en) * | 2023-06-28 | 2023-07-28 | 南昌大学第一附属医院 | Brain transient monitoring device, electronic device, and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113918008A (en) | Brain-computer interface system based on source space brain magnetic signal decoding and application method | |
| Ramadan et al. | Basics of brain computer interface | |
| Zink et al. | Mobile EEG on the bike: disentangling attentional and physical contributions to auditory attention tasks | |
| Pfurtscheller et al. | 15 years of BCI research at Graz University of Technology: current projects | |
| CN109875501B (en) | Physiological parameter measurement and feedback system | |
| Norani et al. | A review of signal processing in brain computer interface system | |
| CN111930238B (en) | Brain-computer interface system implementation method and device based on dynamic SSVEP (secure Shell-and-Play) paradigm | |
| Sadeghi et al. | Recent advances in hybrid brain-computer interface systems: a technological and quantitative review | |
| Dovgialo et al. | Assessment of statistically significant command-following in pediatric patients with disorders of consciousness, based on visual, auditory and tactile event-related potentials | |
| Xu et al. | EEG decoding method based on multi-feature information fusion for spinal cord injury | |
| Sun et al. | Motor imagery EEG-based online control system for upper artificial limb | |
| Rajyalakshmi et al. | Exploration of recent advances in the field of brain computer interfaces | |
| Kaur et al. | Analyzing eeg based neurological phenomenon in bci systems | |
| Van Kokswijk et al. | Self adaptive BCI as service-oriented information system for patients with communication disabilities | |
| Olaniyan et al. | Neural signaling and communication using machine learning | |
| Gangwar et al. | Statistical Learning for Brain–Computer Interface | |
| Kambhamettu et al. | Brain–Computer Interface‐Assisted Automated Wheelchair Control Management–Cerebro: A BCI Application | |
| CN111616704A (en) | Research method for brain dynamic function network interaction mode of sentence listening task | |
| Saeed et al. | Investigation of Feature Extraction Method for EEG Signal Processing | |
| Wang et al. | A review of research on brain-computer interface based on imagined speech | |
| Adsawinnawanawa et al. | Decode Brain Signal Into Thai Word Using EEG and L-SVM | |
| CN111897428B (en) | Gesture recognition method based on moving brain-computer interface | |
| Ghasemi et al. | A new method for features normalization in motor imagery few-shot learning using resting-state | |
| Wang et al. | A Method of Expanding EEG Data Based on Transfer Learning Theory | |
| Kazi et al. | Image Retrieval and Interested Image Detection for thought processing using EEG Signals |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |