CN112998725A - Rehabilitation method and system of brain-computer interface technology based on motion observation - Google Patents
Rehabilitation method and system of brain-computer interface technology based on motion observation Download PDFInfo
- Publication number
- CN112998725A CN112998725A CN202110181467.0A CN202110181467A CN112998725A CN 112998725 A CN112998725 A CN 112998725A CN 202110181467 A CN202110181467 A CN 202110181467A CN 112998725 A CN112998725 A CN 112998725A
- Authority
- CN
- China
- Prior art keywords
- rehabilitation
- brain
- patient
- data
- electroencephalogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
- A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
- A61B5/6802—Sensor mounted on worn items
- A61B5/6803—Head-worn items, e.g. helmets, masks, headphones or goggles
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
- A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Veterinary Medicine (AREA)
- Signal Processing (AREA)
- Public Health (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Physiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Mathematical Physics (AREA)
- Power Engineering (AREA)
- Evolutionary Computation (AREA)
- Fuzzy Systems (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
The invention discloses a rehabilitation method and a rehabilitation system based on a brain-computer interface technology of motion observation, wherein the method comprises the following steps: assembling an electroencephalogram data acquisition device for a patient; playing a sport rehabilitation action multimedia file for the patient; collecting brain wave signals of the patient watching the multimedia files; preprocessing the brain wave signals and then extracting features; inputting the extracted features into a rehabilitation evaluation model; and adjusting a rehabilitation strategy based on the output result of the rehabilitation evaluation model. The brain-computer interface based on the movement observation therapy can feed back the change condition of brain waves of a patient during movement observation in a certain mode through the brain-computer interface technology, so that the attention and the enthusiasm of the patient during rehabilitation training are greatly improved, and effective nerve remodeling and motor function recovery can be carried out.
Description
Technical Field
The invention belongs to the field of biomedicine, and particularly relates to a rehabilitation method and a rehabilitation system based on a brain-computer interface technology for motion observation.
Background
Cerebral apoplexy comprises cerebrovascular diseases such as cerebral hemorrhage and cerebral infarction, is mainly caused by cerebral nerve dysfunction caused by cerebral vessel occlusion, and is a main disease causing limb movement disorder, wherein sequelae are mostly left after the cerebral apoplexy is treated, and the sequelae are manifested by limb movement dysfunction with different degrees such as impaired motor function, muscular atrophy and the like. If the brain nerves can be reactivated, the motor nerve pathways can be reestablished, and the limbs can be trained to improve the muscle strength, the patient can recover the motor function.
Domestic and foreign researches prove that the disability rate of the cerebral apoplexy patient can be effectively reduced by intervening rehabilitation nursing measures on the cerebral apoplexy patient. At present, the conventional treatment means for the stroke motor dysfunction in China mainly comprises rehabilitation nursing, including traditional Chinese medicine rehabilitation nursing measures, modern rehabilitation nursing measures and traditional Chinese and western medicine combined rehabilitation nursing, and physical therapy and operation therapy which are used for assisting in carrying out function training on upper and lower limbs of a hemiplegia by a therapist or a rehabilitation robot to promote motor function recovery.
However, the conventional treatment means has many defects and shortcomings, the rehabilitation nursing training needs to be performed under the guidance of professional doctors, a large amount of manpower and material resources are consumed, the price is high, and the like, so that the popularization and the use of families and communities are greatly limited. The rehabilitation nursing training method is monotonous and repeated, a patient is often in a passive state when rehabilitation training is carried out, timely and intuitive feedback on the rehabilitation effect is lacked, the participation degree and the initiative of the patient are not high, and the physical and psychological pain of the patient is easily increased. In addition, most of the existing various new rehabilitation technologies for improving the motor function of the patient need the patient to have certain motor ability, and no effective rehabilitation training method exists for the patient with serious motor dysfunction.
Motor imagery therapy refers to a process of continuously simulating and reproducing human body movement in the thinking process without obvious body movement output. Both domestic and foreign researches prove that the motor imagery therapy can improve the limb motor function of the stroke patient and improve the daily living and activity ability of the patient. It can activate the damaged neural network, and can build or recreate a new neural pathway, thereby restoring motor function to the patient. Motor imagery therapy is a rehabilitation therapy closely related to active movement of a patient, but the motor imagery therapy has high requirements on stroke patients and requires the patients to have high matching degree and imagination, and brain-computer interface technology based on the method can obtain certain curative effect by using external rehabilitation equipment in a matching way, so that the patients are required to have certain movement capacity, otherwise, certain pain is caused to the patients.
The motion observation therapy is a physical and mental rehabilitation method based on the advantages of the mirror image therapy and the motor imagery therapy, a stroke patient can activate the brain sensory movement related area similar to the active or passive motion of the patient to exercise the brain nerves by watching videos intensively or observing healthy people face to perform rehabilitation tasks without performing imagination and additional output of specific actions, and the similarity can be found in the healthy people and the stroke patient and is suitable for any stage of the stroke. The condition of the motor observation therapy for the activation of the nerves of the brain motor area has certain relevance with motor imagery, and the requirements on the coordination degree and the imagination of the patient are relatively low, so long as the attention is focused as much as possible.
The brain-computer interface is a method for establishing a direct communication or control channel between the human brain and a computer or other electronic equipment by collecting and analyzing the bioelectricity signals of the human brain, namely expressing will or operating external equipment to communicate with the outside directly by controlling brain electricity without depending on the normal output channel of peripheral nerves and muscles of the brain.
Disclosure of Invention
Aiming at the defects in the prior art, the invention provides a rehabilitation method of a brain-computer interface technology based on motion observation, which comprises the following steps:
assembling an electroencephalogram data acquisition device for a patient;
playing a sport rehabilitation action multimedia file for the patient;
collecting brain wave signals of the patient watching the multimedia files;
preprocessing the brain wave signals and then extracting features;
inputting the extracted features into a rehabilitation evaluation model;
and adjusting a rehabilitation strategy based on the output result of the rehabilitation evaluation model.
Furthermore, the electroencephalogram data acquisition equipment uses a portable electroencephalogram signal amplifier, the electrode position adopts international standard 10-20 electrode lead positioning, the electroencephalogram cap is provided with 35 channels, the reference electrode is arranged at the mastoid position, and the sampling frequency is set to be 256 Hz.
Further, preprocessing the brain wave signals, including:
the brain wave signals are amplified by an amplifier and converted into discrete digital signals through A/D conversion pretreatment;
c3, Cz and C4 lead signals are extracted according to a region related to the brain motor nerves, a wave trap is used for filtering 50Hz power frequency interference, a 4-order Butterworth filter is used for extracting electroencephalogram signals within a frequency band range of 8-30Hz, and electroencephalogram data of 2s before and 4s after each motion observation task are intercepted according to different event labels;
the preprocessed brain wave signals are three-dimensional arrays (3, 6 x fs, N), the first dimension is extracted lead numbers, the second dimension is sampling data points of intercepted task events, fs is sampling frequency points, the third dimension N is the number of samples of one experimental event, and the number of times (trail number) of moving observation tasks in one experiment.
Further, the event label refers to an event label of the influence of the currently played rehabilitation action on the brain of the patient.
Further, the rehabilitation evaluation model is a neural network model and comprises a convolutional layer, a pooling layer, an LSTM and a full-link layer.
Further, the neural network model is trained using several patient data prior to application.
Further, the plurality of patient data is acquired using the steps of:
extracting features and labels from the acquired plurality of patient data: intercepting 6s data according to the event label observed by movement, and intercepting two-dimensional data consisting of each lead and sampling points;
forming a three-dimensional array by each intercepted two-dimensional data;
converting the event label into a one-hot code;
and processing all three-dimensional arrays into an NxDxS format, wherein N is the number of times of a motion observation task in one experiment, D is a lead number, and S is the number of sampling points.
All three-dimensional arrays are divided into a 60% training set, a 20% testing set, and a 20% validation set.
Further, Fourier transform is carried out on the extracted features to obtain a time-frequency map;
respectively overlapping according to different tasks, and searching for an obvious characteristic frequency band of ERD/ERS;
and analyzing the power spectral density according to the ERD/ERS characteristic frequency band positioned by the time-frequency map, obtaining a curve graph of the change of the power of the electroencephalogram in the characteristic lead characteristic frequency band during the task period along with time, analyzing the corresponding relation between task actions and leads and the characteristic frequencies of different actions, and judging the influence of different rehabilitation actions on brain movement parts according to the curve graph.
Further, the adjusting the rehabilitation strategy based on the output result of the rehabilitation evaluation model comprises:
and increasing the playing time of the multimedia file with the evaluation result larger than the preset threshold value, and driving the simulated rehabilitation appliance to assist the patient in the action.
The invention also provides a rehabilitation system of the brain-computer interface technology based on motion observation, which comprises:
the brain wave number acquisition equipment is used for acquiring brain wave signals of the patient;
a playing device for playing a motor rehabilitation action multimedia file for the patient;
the feature extraction equipment is used for carrying out feature extraction after preprocessing the brain wave signals;
a feature input device for inputting the extracted features into the rehabilitation evaluation model;
an adjustment device for adjusting a rehabilitation strategy based on an output of the rehabilitation assessment model.
Compared with the prior art, the brain-computer interface based on the movement observation therapy can feed back the change condition of the brain waves of the patient during movement observation in a certain mode through the brain-computer interface technology, so that the attention and the enthusiasm of the patient during rehabilitation training are greatly improved, and effective nerve remodeling and motor function recovery can be carried out.
The invention adopts a method for activating a brain motor nerve functional area based on a brain-computer interface observed by movement, collects and designs a motor rehabilitation video library related to a brain-computer interface paradigm, carries out preprocessing, short-time Fourier transform and power spectrum energy transform on collected brain electrical signals, analyzes lead signals of the movement related area on time frequency, and visually displays the activation effect of the movement observation on the brain movement area by qualitatively and quantitatively analyzing the ERD/ERS phenomenon and value of the related frequency band. A large amount of electroencephalogram data acquired offline are subjected to interception and division of a training set and a data set, the acquired electroencephalogram data are sent into a well-built CNN + LSTM neural network for model training, characteristics are automatically extracted and classified and predicted, a model with small loss error and good classification effect is trained and stored, the model is good in adaptability and strong in generalization capability, the data of different people can be well classified, and cross-person identification and classification can be achieved. Data during online experiments are sent into a neural network, result prediction is carried out through a training model, predicted classification results are output to control a desktop to simulate a rehabilitation appliance to complete set actions, real-time feedback is given to a tested person, and the tested person is assisted to participate in training more actively. The simulated rehabilitation appliance comprises a mechanical arm which is used for assisting an exoskeleton motion assistor or a prosthetic to help a patient to perform actual motion recovery training instead.
Drawings
The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar or corresponding parts and in which:
fig. 1 is a flowchart illustrating a rehabilitation method based on brain-computer interface technology of motion observation according to an embodiment of the invention;
FIG. 2 is a diagram illustrating a neural network architecture according to an embodiment of the present invention;
FIG. 3 is a time-frequency diagram illustrating the C3 leads during a motion observation task according to an embodiment of the present invention;
FIG. 4 is a PSD curve at C3 lead illustrating different triggering tasks according to an embodiment of the present invention;
fig. 5 is a block diagram illustrating a brain-computer interface technology rehabilitation system based on motion observation according to an embodiment of the invention;
fig. 6 is a functional block diagram illustrating a brain-computer interface technology rehabilitation system based on motion observation according to another embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise, and "a plurality" typically includes at least two.
It should be understood that although the terms first, second, third, etc. may be used to describe … … in embodiments of the present invention, these … … should not be limited to these terms. These terms are used only to distinguish … …. For example, the first … … can also be referred to as the second … … and similarly the second … … can also be referred to as the first … … without departing from the scope of embodiments of the present invention.
It should be understood that the term "and/or" as used herein is merely one type of association that describes an associated object, meaning that three relationships may exist, e.g., a and/or B may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
The words "if", as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.
It is also noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that an article or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such article or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in the article or device in which the element is included.
Alternative embodiments of the present invention are described in detail below with reference to the accompanying drawings.
Deep learning has higher accuracy in the fields of image processing, voice recognition and the like, and the current BCI direction is also a field of deep learning and artificial neural network research. The electroencephalogram signal has the characteristics of non-stability and randomness, the traditional electroencephalogram signal processing needs preprocessing, feature extraction and classification recognition, for example, feature parameters are extracted by using methods such as a common space mode (CSP) and a power spectral density, and then classification is carried out by using a classification algorithm. The Convolutional Neural Networks (CNN) have better effectiveness in processing time sequences, the CNN is used for automatically extracting the characteristics of electroencephalogram data, and then a Long Short-Term Memory (LSTM) method is used for classifying the extracted characteristics, so that errors are reduced.
The first embodiment,
As shown in fig. 1, the invention discloses a rehabilitation method of brain-computer interface technology based on motion observation, comprising the following steps:
assembling an electroencephalogram data acquisition device for a patient;
playing a sport rehabilitation action multimedia file for the patient;
collecting brain wave signals of the patient watching the multimedia files;
preprocessing the brain wave signals and then extracting features;
inputting the extracted features into a rehabilitation evaluation model;
and adjusting a rehabilitation strategy based on the output result of the rehabilitation evaluation model.
Example II,
The invention provides a brain-computer interface technology rehabilitation method based on motion observation, which comprises the following steps:
assembling an electroencephalogram data acquisition device for a patient;
playing a sport rehabilitation action multimedia file for the patient;
collecting brain wave signals of the patient watching the multimedia files;
preprocessing the brain wave signals and then extracting features;
inputting the extracted features into a rehabilitation evaluation model;
and adjusting a rehabilitation strategy based on the output result of the rehabilitation evaluation model.
Furthermore, the electroencephalogram data acquisition equipment uses a portable electroencephalogram signal amplifier, the electrode position adopts international standard 10-20 electrode lead positioning, the electroencephalogram cap is provided with 35 channels, the reference electrode is arranged at the mastoid position, and the sampling frequency is set to be 256 Hz.
Further, preprocessing the brain wave signals, including:
the brain wave signals are amplified by an amplifier and converted into discrete digital signals through A/D conversion pretreatment;
c3, Cz and C4 lead signals are extracted according to a region related to the brain motor nerves, a wave trap is used for filtering 50Hz power frequency interference, a 4-order Butterworth filter is used for extracting electroencephalogram signals within a frequency band range of 8-30Hz, and electroencephalogram data of 2s before and 4s after each motion observation task are intercepted according to different event labels;
the preprocessed brain wave signals are three-dimensional arrays (3, 6 x fs, N), the first dimension is extracted lead numbers, the second dimension is sampled data points of intercepted task events, fs is sampled frequency points, and the third dimension is feature numbers N of extracted features.
Further, the event label refers to an event label of the influence of the currently played rehabilitation action on the brain of the patient.
Further, the rehabilitation evaluation model is a neural network model and comprises a convolutional layer, a pooling layer, an LSTM and a full-link layer.
Further, the neural network model is trained using several patient data prior to application.
Further, the plurality of patient data is acquired using the steps of:
extracting features and labels from the acquired plurality of patient data: intercepting 6s data according to the event label observed by movement, and intercepting two-dimensional data consisting of each lead and sampling points;
forming a three-dimensional array by each intercepted two-dimensional data;
converting the event label into a one-hot code;
and processing all three-dimensional arrays into an NxDxS format, wherein N is the number of times of a motion observation task in one experiment, D is a lead number, and S is the number of sampling points.
All three-dimensional arrays are divided into a 60% training set, a 20% testing set, and a 20% validation set.
Further, Fourier transform is carried out on the extracted features to obtain a time-frequency map;
respectively overlapping according to different tasks, and searching for an obvious characteristic frequency band of ERD/ERS;
and analyzing the power spectral density according to the ERD/ERS characteristic frequency band positioned by the time-frequency map, obtaining a curve graph of the change of the power of the electroencephalogram in the characteristic lead characteristic frequency band during the task period along with time, analyzing the corresponding relation between task actions and leads and the characteristic frequencies of different actions, and judging the influence of different rehabilitation actions on brain movement parts according to the curve graph.
Further, the adjusting the rehabilitation strategy based on the output result of the rehabilitation evaluation model comprises:
and increasing the playing time of the multimedia file with the evaluation result larger than the preset threshold value, and driving the simulated rehabilitation appliance to assist the patient in the action.
Example III,
In one embodiment of the invention, the following method is disclosed:
the method comprises the following steps: and (3) establishing a rehabilitation video library required for motion observation, and determining video types with different rehabilitation functions.
Step two: and selecting the rehabilitation task, setting the experiment times, and collecting the electroencephalogram signals.
Step three: and processing the acquired electroencephalogram data, and filtering myoelectricity and electrooculogram to finish preprocessing. Data segments before and after the occurrence of a stimulation event of leads (C3, CZ, C4) associated with the motion region are intercepted.
Step four: and performing time-frequency analysis on the preprocessed electroencephalogram data, qualitatively and quantitatively analyzing the brain activity condition, and converting into an index capable of visually displaying the brain motion activity.
Step five: and (3) building a CNN + LSTM neural network, wherein the neural network comprises a convolutional layer, a pooling layer, an LSTM and a full connection layer.
Step six: and training the preprocessed offline data by using the built neural network, and establishing and storing a model with higher accuracy. The electroencephalogram primitive signals can be read in real time in the next experiment, electroencephalogram data of sampling points are input into a neural network, prediction is carried out by using a model after calling, the predicted probability value is returned finally, and the category with high probability is taken as the classification result finally.
Step seven: the neural network classification result is applied to the feedback of the tested person, and the classification result outputs different instructions to control the desktop simulation rehabilitation device to complete corresponding movement, so that whether the training of the tested person is concentrated and effective or not is intuitively prompted. The mechanical arm is used as an execution peripheral, is used for simulating an exoskeleton motion assistive device or artificial limb and can be used as a feedback prompt, and can also help a patient to perform actual motion recovery training
In the first step, the exercise rehabilitation action video library is a rehabilitation exercise guide video recorded by combining standard rehabilitation actions used in hospital rehabilitation, and comprises training actions aiming at muscles of single joints and each affected side limb, such as internal and external rotation, buckling and stretching actions of each joint of the upper and lower limbs; fine movements of each limb part, such as middle finger training, opposite finger training, thumb pinching, alternate pointing of both feet, front and back pointing of the tiptoe, straight stepping and the like; the necessary actions in daily life, such as book turning, toothpaste squeezing, desk wiping, clothes wearing and taking off, stair climbing and the like, including joint movement, balance function and walking training of a lying position, a sitting position and a standing position, can more comprehensively recover patients in various recovery periods with various limb movement functions. The rehabilitation exercise training of the video library mainly adopts single joint movement and single muscle coordination training, prevents muscle spasm, enhances the muscle strength of the trunk and gradually leads to the training of the complex actions which are commonly used in daily life. The simple actions are used for decomposing complex actions one by one, training of the next action is carried out after training of one action is skilled, sensory impression and motion programs are formed, and after the training of the decomposed actions can be completed skillfully, the training can be combined together for training, the training of the complex actions is completed, and the balance of limbs is trained. The principle of the rehabilitation training is determined by the muscle and limb specificity in the recovery period of stroke limb movement, the training is from joint movement to muscle strengthening, the training is from single muscle training to multi-block muscle training, and the training method is from simple limb movement to compound limb movement. The limb function training action more conforming to the patient is selected according to different recovery periods, the control capability of the brain of the patient on the limb can be gradually improved, the capability of mutual integration and cooperative work of all functional areas of the brain is trained, the coordination of the nerve path and the body muscle of the brain is recovered, the complication is effectively prevented, and the daily life capability of the patient is gradually improved.
In the second step, a visual interaction platform built by a python programming language is used, a task video required to be trained for an experiment is selected, the task duration is automatically read, an experiment time sequence and a cycle number MI _ T are set, the sample number MI _ T and the stimulation event time T of data processing are determined, and the occurrence number of different tasks is the same in the experiment process but the sequence is random.
In the third step, the preprocessing generally includes filtering and segmentation. The frequency bands related to limb movement are an alpha frequency band and a beta frequency band, a 4-order Butterworth filter is used, the cut-off frequency and the stop band frequency are set to be 8Hz and 30Hz respectively, and electroencephalogram signals in the frequency band range of 8-30Hz are extracted. And because the band-pass filter can not filter all signals except the cut-off frequency and the stop band frequency, the power of the power frequency interference signal of 50Hz is larger, and the power frequency interference of 50Hz can be accurately removed by using the IIR wave trap with a steep transition band. According to MARK shot before the beginning of a stimulation event in an experiment, effective data segments from MARK front 1s to MARK back 4s of C3, Cz and C4 leads related to a brain motion region are intercepted, electroencephalogram data segments induced by different stimulation are extracted, and samples are classified according to task types.
In the fourth step of the method, the first step of the method,
in the formula: ω is angular frequency, which represents the conjugate of the complex number, w (t) is the observation window, a hanning window is selected as the window function, and equation (2) is as follows:
Fig. 3 shows a time-frequency diagram of C3 leads during a sports observation task, in which a Power Spectral Density (PSD) curve analysis method is introduced to quantitatively analyze brain activity. The Power Spectral Density (PSD) is also based on the short-time fourier transform (STFT), an expression for the Power Spectral Density (PSD) of the signal:
P(ω,t)＝|STFTx(ω,t)|2/2π (3)
as shown in fig. 4, PSD curves of two tasks in C3 leads are shown, wherein PSD curves of C3, CZ, and C4 in different frequency bands are respectively drawn, so we can perform more detailed analysis on data according to the ERD/ERS characteristic time period and frequency band located by the time-frequency map. The time-frequency map and the PSD curve can qualitatively analyze the brain activation condition, and ERD/ERS coefficients are introduced to obtain quantitative calculation quantitative parameters on the basis. There are many defining and calculating methods for the ERD/ERS coefficient, and the band power spectrum defining method is adopted herein:
wherein REP is t after the event occurs1Power spectral density over time, REF being t before occurrence of an event2Power spectral density over time. Three leads C3, Cz and C4 are still selected, ERD/ERS coefficients of the three leads at 8-29 Hz are respectively calculated, and t is taken1＝4s,t2＝2s。
In the fifth step, the constructed convolutional neural network comprises a convolutional layer (CNN), a pooling layer, a long-term memory network (LSTM) and a full connection layer. The network structure parameter characteristics are as follows:
(1) the convolutional layer uses one-dimensional convolution with a convolution kernel number of 40, a kernel length of 1x20, a step size of 4, an input format defined as (3, 1024), a feature derivation number of 3, and a sampling point number of each truncated trail sample of 1024, determined by fs x t. The activation function uses a Relu activation function, and the operation formula is as follows:
f(x)＝max(0，x) (5)
(2) a Dropout layer is added behind the convolutional layer to prevent data overfitting, and the Dropout rate is set to be 0.5.
(3) And standardizing the extracted data characteristics, wherein batch standardization is to calculate the mean value of the sample, then calculate the variance and finally carry out data standardization treatment. Calculating the formula:
wherein xi(i ═ 1,2 … m) as input, μβIs the average value of the average of the values,
is the variance, and γ and β are parameters learned by the network.
(4) The kernel length of the pooling layer is 1x4, the step length is 4, in order to reduce the number of training parameters, the dimensionality of the feature vector output by the convolutional layer is reduced, and meanwhile, the over-fitting phenomenon is reduced.
(5) The long-short time memory (LSTM) model principle simultaneously has not only an original data set when inputting, but also adds an output result of the previous data, namely h (t-1). The units output of the LSTM layer is 30.
(6) The Flatten flattening layer has the function of outputting multidimensional data in a one-dimensional mode, is used for transition from the convolution layer to the full connection, and does not influence the size of the batch.
(7) And the full connection layer applies a softmax function, and carries out motor imagery two-classification through softmax function classification.
In the sixth step, the neural network is used for model training of the acquired electroencephalogram data, and the method is characterized by comprising the following steps:
(1) intercepting time period data according to the label of the motor imagery, intercepting two-dimensional data formed by each lead and sampling points, and forming a three-dimensional array by each intercepted two-dimensional data.
(2) The data tags are extracted and converted to one-hot codes using the mne library.
(3) The data is then divided into a training set, a test set, and a validation set. And changing the data into NxDxS format, wherein N is trail number, D is lead number, and S is sampling point number.
(4) Expanding the data by a factor of n of 10, respectively, prevents the data features from being too insignificant to result in overfitting.
(5) Then the data is sent into a neural network for training, the CNN training comprises two stages, firstly, signal transmission is carried out through some neural network layers, then, output forward propagation is obtained on an output layer, finally, the error between a real value and an actual value is calculated by utilizing backward propagation, then, the gradient is continuously updated, and the error is trained towards the direction with small gradient. The error adopts cross entropy as a loss function, and the cross entropy loss error is the difference condition of probability distribution and real value distribution obtained by evaluating the current training. It characterizes the distance of the actual output (probability) from the desired output (probability), i.e. the smaller the value of the cross entropy, the closer the two probability distributions are. Wherein the formula of the cross entropy loss function:
(6) and finally calling a tentorflow function to store the offline data-trained model locally for each trail of the next experiment to be sent to a neural network for online prediction of classification results.
Example four,
As shown in fig. 5, in one embodiment of the present invention, the following method is disclosed:
step 1: the electroencephalogram data acquisition equipment uses a Neuroscan Grael portable electroencephalogram signal amplifier, the electrode position adopts international standard 10-20 electrode lead positioning, the electroencephalogram cap has 35 channels in total, the reference electrode is arranged at the mastoid position, and the sampling frequency is set to be 256 Hz.
Step 2: the invention adopts a video-induced experimental paradigm, induces the patient to carry out exercise observation by watching standard exercise rehabilitation action videos and voice prompts by the patient, the experiment needs to be carried out in a quiet environment, and the patient sits on a chair which is about 1m away from a computer display, thereby avoiding eye movement and body movement as much as possible. In the experimental process, the original brain wave signals collected by the brain wave collecting equipment are converted into discrete digital signals through amplification of an amplifier and A/D conversion processing, and the discrete digital signals are sent to a computer and are stored locally by collecting software.
And step 3: and preprocessing off-line feature extraction and analysis are carried out according to the acquired off-line data.
Step 3-1, extracting C3, Cz and C4 lead signals according to a brain motor nerve related area, filtering 50Hz power frequency interference by using a wave trap, extracting electroencephalogram signals within a frequency band range of 8-30Hz by using a 4-order Butterworth filter, intercepting electroencephalogram data of the front 2s and the rear 4s of each motion observation task according to different event labels, wherein the preprocessed electroencephalogram data are three-dimensional arrays (3, 6 x fs, N), the first dimension is the extracted lead number, the second dimension is the sampling data point of the intercepted task event, and the third dimension is the extracted sample number N (experimental trail number).
Step 3-2: and analyzing the time-frequency spectrum by using short-time Fourier transform, overlapping the time-frequency spectrums of 20 trail in each group of experiments according to different tasks, and searching for the apparent characteristic frequency band of the ERD/ERS.
Step 3-3: the power spectral density is analyzed according to the ERD/ERS characteristic time period and frequency band positioned by the time-frequency map, the curve graph of the change of the electroencephalogram power along with time in the task period of the characteristic lead frequency band is obtained, the corresponding relation between task actions and leads (mainly C3, Cz and C4) and the characteristic frequencies of different actions are analyzed, and accordingly the activation condition of different rehabilitation actions on brain motion parts can be judged.
And 4, step 4: extracting data and labels from the collected multi-person data, intercepting 6s of data according to the labels observed by movement, intercepting two-dimensional data consisting of each lead and each sampling point, and forming a three-dimensional array by each intercepted two-dimensional data; the tag is converted to one-hot encoding.
And 5: the extracted data was divided into a 60% training set, a 20% testing set, and a 20% validation set. And changing the data into NxDxS format, wherein N is trail number, D is lead number, and S is sampling point number.
And 7: and sending the data into a neural network model, wherein the neural network comprises a convolutional layer, a pooling layer, an LSTM and a full connection layer. The overall structure is shown in fig. 2, and the specific implementation steps are as follows:
step 7-1: the data is first fed into a one-dimensional convolutional layer, the number of convolutional kernels is 40, the kernel length is 1x20, and the step size is 4. Each convolution kernel has a weight parameter, b is a bias parameter, the data dimension of the one-dimensional convolution input is 1024 x 3, and the dimension of the filter is 256 x 1. A relu activation function is used.
Step 7-2: and a Dropout layer is connected behind the convolution layer to prevent data overfitting in the training process, and the Dropout rate is set to be 0.5.
And 7-3: the maximum pooling layer uses a kernel length of 1x4, step size 4.
And 7-4: by using the LSTM layer, not only the original data set is input, but also the output result of the previous data, namely h (t-1), is added, so as to solve the problems of gradient explosion and gradient disappearance in long sequence training.
And 7-5: and the full connection layer applies a softmax function, and performs secondary classification of the motion observation task through multi-classification of the softmax function.
And 7-6: the trained model is saved locally using the tenserflow module, format h5 file.
And 8: and when the same task experiment is carried out next time, data of each trail is extracted in real time and sent to the neural network, and a model stored in the local before is called to carry out real-time prediction.
And step 9: according to the predicted result, the side with the larger probability value is taken as the final classification result, and different classification results control the external simulation rehabilitation appliance (but not limited to the tabletop mechanical arm) to complete corresponding activities.
Example V,
As shown in fig. 6, the present invention further provides a rehabilitation system based on brain-computer interface technology of motion observation, which includes:
the electroencephalogram data acquisition equipment is used for acquiring electroencephalogram signals of a patient;
a playing device for playing a motor rehabilitation action multimedia file for the patient;
the feature extraction equipment is used for carrying out feature extraction after preprocessing the brain wave signals;
a feature input device for inputting the extracted features into the rehabilitation evaluation model;
an adjustment device for adjusting a rehabilitation strategy based on an output of the rehabilitation assessment model.
Example six,
The disclosed embodiments provide a non-volatile computer storage medium having stored thereon computer-executable instructions that may perform the method steps as described in the embodiments above.
The foregoing describes preferred embodiments of the present invention, and is intended to provide a clear and concise description of the spirit and scope of the invention, and not to limit the same, but to include all modifications, substitutions, and alterations falling within the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A rehabilitation method of brain-computer interface technology based on motion observation comprises the following steps:
assembling an electroencephalogram data acquisition device for a patient;
playing a sport rehabilitation action multimedia file for the patient;
collecting brain wave signals of the patient watching the multimedia files;
preprocessing the brain wave signals and then extracting features;
inputting the extracted features into a rehabilitation evaluation model;
the rehabilitation evaluation model analyzes the characteristics;
and adjusting a rehabilitation strategy based on the output result of the rehabilitation evaluation model.
2. The method of claim 1, wherein the electroencephalogram data acquisition device uses a portable electroencephalogram signal amplifier, the electrode location is positioned using international standard 10-20 electrode leads, the electroencephalogram cap has 35 channels, the reference electrode is positioned at the mastoid, and the sampling frequency is set to 256 Hz.
3. The method as claimed in claim 1, wherein the brain wave signal is pre-processed, including:
the brain wave signals are amplified by an amplifier and converted into discrete digital signals through A/D conversion pretreatment;
c3, Cz and C4 lead signals are extracted according to a region related to the brain motor nerves, a wave trap is used for filtering 50Hz power frequency interference, a 4-order Butterworth filter is used for extracting electroencephalogram signals within a frequency band range of 8-30Hz, and electroencephalogram data of 2s before and 4s after each motion observation task are intercepted according to different event labels;
the preprocessed brain wave signals are three-dimensional arrays (3, 6 x fs, N), the first dimension is extracted lead number, the second dimension is the number of sampling data points of intercepted task events, fs is sampling frequency points, and the third dimension N is the frequency of motion observation tasks in one experiment.
4. The method of claim 3, wherein the event label refers to an event label of an effect of a currently played rehabilitation action on the brain of the patient.
5. The method of claim 1, wherein the rehabilitation assessment model is a neural network model comprising convolutional layers, pooling layers, LSTM, and fully-connected layers.
6. The method of claim 5, wherein the neural network model is trained using a number of patient data prior to application.
7. The method of claim 6, wherein the plurality of patient data is acquired using the steps of:
intercepting 6s data according to the event label observed by movement, and intercepting two-dimensional data consisting of each lead and sampling points;
forming a three-dimensional array by each intercepted two-dimensional data;
converting the event label into a one-hot code;
and processing all three-dimensional arrays into an NxDxS format, wherein N is the number of times of a motion observation task in one experiment, D is a lead number, and S is the number of sampling points.
All three-dimensional arrays are divided into a 60% training set, a 20% testing set, and a 20% validation set.
8. The method of claim 1, wherein the rehabilitation assessment model analyzes the features, comprising:
carrying out Fourier transform on the extracted features to obtain a time-frequency map;
respectively overlapping according to different tasks, and searching for an obvious characteristic frequency band of ERD/ERS;
and analyzing the power spectral density according to the ERD/ERS characteristic frequency band positioned by the time-frequency map, obtaining a curve graph of the change of the electroencephalogram power of the characteristic leads along with time in the characteristic frequency band task period, analyzing the corresponding relation between task actions and leads and the characteristic frequencies of different actions, and judging the influence of different rehabilitation actions on the cerebral motor nerve functional area according to the curve graph.
9. The method of claim 1, wherein said adjusting a rehabilitation strategy based on the output of said rehabilitation assessment model comprises:
and increasing the playing time of the multimedia file with the evaluation result larger than the preset threshold value, and driving the simulated rehabilitation appliance to assist the patient in the action.
10. A rehabilitation system based on brain-computer interface technology for motion observation, comprising:
the brain wave number acquisition equipment is used for acquiring brain wave signals of the patient;
a playing device for playing a motor rehabilitation action multimedia file for the patient;
the feature extraction equipment is used for carrying out feature extraction after preprocessing the brain wave signals;
a feature input device for inputting the extracted features into the rehabilitation evaluation model;
an adjustment device for adjusting a rehabilitation strategy based on an output of the rehabilitation assessment model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110181467.0A CN112998725A (en) | 2021-02-10 | 2021-02-10 | Rehabilitation method and system of brain-computer interface technology based on motion observation |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110181467.0A CN112998725A (en) | 2021-02-10 | 2021-02-10 | Rehabilitation method and system of brain-computer interface technology based on motion observation |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112998725A true CN112998725A (en) | 2021-06-22 |
Family
ID=76402022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110181467.0A Pending CN112998725A (en) | 2021-02-10 | 2021-02-10 | Rehabilitation method and system of brain-computer interface technology based on motion observation |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112998725A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113792588A (en) * | 2021-08-05 | 2021-12-14 | 深兰科技（上海）有限公司 | Brain wave processing device and method, computer equipment and storage medium |
| CN116636817A (en) * | 2023-07-26 | 2023-08-25 | 四川新源生物电子科技有限公司 | Anesthesia depth evaluation method, anesthesia depth evaluation system, anesthesia depth evaluation device and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102138860A (en) * | 2011-01-10 | 2011-08-03 | 西安交通大学 | Intelligentized rehabilitation training equipment for hand functions of patients suffering from cerebral injury |
| CN204813892U (en) * | 2015-07-28 | 2015-12-02 | 苏州明思特医疗科技有限公司 | A device for rehabilitation training of mirror image neuron |
| CN107961007A (en) * | 2018-01-05 | 2018-04-27 | 重庆邮电大学 | A kind of electroencephalogramrecognition recognition method of combination convolutional neural networks and long memory network in short-term |
| CN109166612A (en) * | 2018-08-14 | 2019-01-08 | 龚映清 | A kind of big game scene rehabilitation system and method based on eye movement and brain electric information |
| CN110993056A (en) * | 2019-10-29 | 2020-04-10 | 浙江迈联医疗科技有限公司 | Hybrid active rehabilitation method and device based on mirror image neurons and brain-computer interface |
-
2021
- 2021-02-10 CN CN202110181467.0A patent/CN112998725A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102138860A (en) * | 2011-01-10 | 2011-08-03 | 西安交通大学 | Intelligentized rehabilitation training equipment for hand functions of patients suffering from cerebral injury |
| CN204813892U (en) * | 2015-07-28 | 2015-12-02 | 苏州明思特医疗科技有限公司 | A device for rehabilitation training of mirror image neuron |
| CN107961007A (en) * | 2018-01-05 | 2018-04-27 | 重庆邮电大学 | A kind of electroencephalogramrecognition recognition method of combination convolutional neural networks and long memory network in short-term |
| CN109166612A (en) * | 2018-08-14 | 2019-01-08 | 龚映清 | A kind of big game scene rehabilitation system and method based on eye movement and brain electric information |
| CN110993056A (en) * | 2019-10-29 | 2020-04-10 | 浙江迈联医疗科技有限公司 | Hybrid active rehabilitation method and device based on mirror image neurons and brain-computer interface |
Non-Patent Citations (9)
| Title |
|---|
| LIYU WANG ET AL.: "Comparison of EEG Changes during Action Observation, Action Observation", 《2017 INTERNATIONAL CONFERENCE ON COMPUTER SYSTEMS, ELECTRONICS AND CONTROL》, 27 August 2018 (2018-08-27), pages 1 - 10 * |
| NAOTSUGU KANEKO ET AL.: "Phase dependent modulation of cortical activity during action observation", 《NEUROIMAGE》 * |
| NAOTSUGU KANEKO ET AL.: "Phase dependent modulation of cortical activity during action observation", 《NEUROIMAGE》, vol. 225, 18 October 2020 (2020-10-18), pages 1075 - 1080, XP086410382, DOI: 10.1016/j.neuroimage.2020.117486 * |
| 张路霞等: "《健康医疗大数据的管理与应用》", 30 April 2020, 上海交通大学出版社, pages: 172 * |
| 王保增等: "结合运动想象和动作观察的伸-握动作识别方法", 《西安交通大学学报》 * |
| 王保增等: "结合运动想象和动作观察的伸-握动作识别方法", 《西安交通大学学报》, vol. 53, no. 10, 1 August 2019 (2019-08-01), pages 1 * |
| 阮敬: "python数据分析基础 第2版", 中国统计出版社, pages: 419 - 438 * |
| 陈志华等: "视觉刺激情感变化β波同步分析", 《大连交通大学学报》 * |
| 陈志华等: "视觉刺激情感变化β波同步分析", 《大连交通大学学报》, vol. 36, no. 4, 31 August 2015 (2015-08-31), pages 1 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113792588A (en) * | 2021-08-05 | 2021-12-14 | 深兰科技（上海）有限公司 | Brain wave processing device and method, computer equipment and storage medium |
| CN113792588B (en) * | 2021-08-05 | 2024-04-09 | 深兰科技（上海）有限公司 | Brain wave processing device, brain wave processing method, computer equipment and storage medium |
| CN116636817A (en) * | 2023-07-26 | 2023-08-25 | 四川新源生物电子科技有限公司 | Anesthesia depth evaluation method, anesthesia depth evaluation system, anesthesia depth evaluation device and storage medium |
| CN116636817B (en) * | 2023-07-26 | 2023-11-03 | 四川新源生物电子科技有限公司 | Anesthesia depth evaluation method, anesthesia depth evaluation system, anesthesia depth evaluation device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108304917B (en) | P300 signal detection method based on LSTM network | |
| WO2021143353A1 (en) | Gesture information processing method and apparatus, electronic device, and storage medium | |
| RU2415642C1 (en) | Method of classification of electroencephalographic signals in interface brain-computer | |
| CN110555468A (en) | Electroencephalogram signal identification method and system combining recursion graph and CNN | |
| Alwasiti et al. | Motor imagery classification for brain computer interface using deep metric learning | |
| WO2022027822A1 (en) | Electromyographic signal-based intelligent gesture action generation method | |
| Sun et al. | Golden subject is everyone: A subject transfer neural network for motor imagery-based brain computer interfaces | |
| Fang et al. | Improve inter-day hand gesture recognition via convolutional neural network-based feature fusion | |
| CN112998725A (en) | Rehabilitation method and system of brain-computer interface technology based on motion observation | |
| Chen et al. | Surface-electromyography-based gesture recognition using a multistream fusion strategy | |
| Ma et al. | A novel and efficient feature extraction method for deep learning based continuous estimation | |
| Tyacke et al. | Hand gesture recognition via transient sEMG using transfer learning of dilated efficient CapsNet: towards generalization for neurorobotics | |
| Soroushmojdehi et al. | Transfer learning in hand movement intention detection based on surface electromyography signals | |
| MP | Idendifying eye movements using neural networks for human computer interaction | |
| Li et al. | Multi-modal emotion recognition based on deep learning of EEG and audio signals | |
| Xiong et al. | A Global and Local Feature fused CNN architecture for the sEMG-based hand gesture recognition | |
| Kumar et al. | A critical review on hand gesture recognition using semg: Challenges, application, process and techniques | |
| Massa et al. | Graph neural networks for hd emg-based movement intention recognition: An initial investigation | |
| Wang et al. | Multi-source integration based transfer learning method for cross-user semg gesture recognition | |
| CN115024735B (en) | Cerebral apoplexy patient rehabilitation method and system based on movement intention recognition model | |
| Ho et al. | Brain-wave bio potentials based mobile robot control: wavelet-neural network pattern recognition approach | |
| Qi et al. | Recognition of composite motions based on sEMG via deep learning | |
| Li et al. | MyoTac: Real‐Time Recognition of Tactical Sign Language Based on Lightweight Deep Neural Network | |
| Naik et al. | Real-time hand gesture identification for human computer interaction based on ICA of surface electromyogram | |
| Dang et al. | An interactive care system based on a depth image and EEG for aged patients with dementia |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |