CN113705464A - Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface - Google Patents

Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface Download PDF

Info

Publication number
CN113705464A
CN113705464A CN202111003357.1A CN202111003357A CN113705464A CN 113705464 A CN113705464 A CN 113705464A CN 202111003357 A CN202111003357 A CN 202111003357A CN 113705464 A CN113705464 A CN 113705464A
Authority
CN
China
Prior art keywords
data
test
value
motor imagery
experiment
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111003357.1A
Other languages
Chinese (zh)
Other versions
CN113705464B (en
Inventor
李卓明
李华清
张羽
李永健
陈幸
董衡
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Harbin Institute of Technology
Original Assignee
Harbin Institute of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Harbin Institute of Technology filed Critical Harbin Institute of Technology
Priority to CN202111003357.1A priority Critical patent/CN113705464B/en
Publication of CN113705464A publication Critical patent/CN113705464A/en
Application granted granted Critical
Publication of CN113705464B publication Critical patent/CN113705464B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/02—Preprocessing
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
    • Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A domain adaptation method for solving the problem of feature migration in a motor imagery brain-computer interface belongs to the technical field of migration learning in the motor imagery brain-computer interface. The invention solves the problem that the classification accuracy of MI-BCI is low due to the deviation caused by electroencephalogram signal characteristic migration in a motor imagery brain-computer interface. The method obtains the required mapping according to the sample distribution of the source domain and the target domain, the sample distribution of the target domain is mapped into a new sample distribution, the difference between the new sample distribution and the sample distribution of the source domain is minimum, the sample distribution of the target domain is changed through the mapping, so that the consistency of the sample distribution of the target domain and the training domain in the brain-computer interface is maintained, the problem of deviation caused by electroencephalogram signal characteristic migration is solved, and the classification accuracy of the new sample distribution is effectively improved. The invention can be applied to feature migration in a motor imagery brain-computer interface.

Description

Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface
Technical Field
The invention belongs to the technical field of transfer learning in a motor imagery brain-computer interface, and particularly relates to a domain adaptation method for solving the problem of feature transfer in the motor imagery brain-computer interface.
Background
The Brain-Computer Interface (BCI) provides a direct connection mode between the human Brain and the electronic device, and realizes information exchange between the human Brain and the external device. The brain-computer interface can be divided into different brain-computer interfaces according to different types of electroencephalogram signals. Among them, a brain-computer interface based on a Motor imaging Electroencephalogram (MI-EEG) is widely used because it is active and motion-related. However, the problem of electroencephalogram feature migration exists in a Motor image Brain-Computer Interface (MI-BCI), the deviation caused by the feature migration can reduce the classification accuracy of the MI-BCI, and the machine learning algorithm in the traditional Brain-Computer Interface cannot solve the problem, so that the problem becomes a main reason for limiting the development of the MI-BCI.
Disclosure of Invention
The invention aims to solve the problem that the classification accuracy of MI-BCI is low due to deviation caused by electroencephalogram signal feature migration in a motor imagery brain-computer interface, and provides a domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface.
The technical scheme adopted by the invention for solving the technical problems is as follows: a domain adaptation method for solving a feature migration problem in a motor imagery brain-computer interface, the method comprising in particular the steps of:
taking a person using a brain-computer interface system as a tested subject, carrying out M motor imagery experiments on the tested subject, and collecting electroencephalogram signal data of the tested subject in the whole process from the beginning of the motor imagery experiments to the end of the motor imagery experiments in the process of each motor imagery experiment;
dividing the acquired electroencephalogram data into a training set and a testing set, and numbering the data in the training set and the testing set according to the corresponding experiment sequence;
step two, intercepting the electroencephalogram signal data of the motor imagery time interval in the training set and the test set;
thirdly, performing band-pass filtering processing on the intercepted electroencephalogram signal data to obtain electroencephalogram signal data subjected to band-pass filtering processing;
processing the training set data and the test set data by adopting a CSP (compact strip service) feature extraction algorithm to obtain a spatial filter;
performing spatial filtering on the electroencephalogram signal data after the band-pass filtering processing corresponding to each experiment in the training set and the testing set by using a spatial filter, respectively obtaining a spatial filtering matrix corresponding to the acquired data of each experiment in the training set and the testing set, respectively processing the spatial filtering matrix, then obtaining a characteristic matrix of the electroencephalogram signal data acquired by the corresponding experiment, and performing normalization processing on the characteristic matrix;
fifthly, training the SVM function by using the normalized feature matrix corresponding to the training set data to obtain a trained model;
calculating a threshold value threshold by using a normalized feature matrix of the training set data, selecting a plurality of groups of test data from the test set data according to rules, and calculating an average feature value of each group of test data by using the normalized feature matrix corresponding to the test set data;
selecting a plurality of groups of test data from the test set data according to the rule, and calculating the average characteristic value of each group of test data by using the normalized characteristic matrix corresponding to the test set data; the specific process comprises the following steps:
number l is 1 to M1Between them in turn, serial number j0In the range of 1 to M2In turn, M2Representing the total experiment times corresponding to the test set data;
the sequence number l starts with 1, and when l is 1, the sequence number j0Take 1,2, …, M in sequence2When the serial number j0When the value of (1) is less than that of the serial number l, the 1 st to the j th times in the test set are carried out0-1 experimental data was selected as a set of data; when j is0When the test is more than or equal to l, the jth in the test set0L times to j times0-1 experimental data was selected as a set of data; up to sequence number j0Traverse 1,2, …, M in turn2Then, together obtain M2Respectively calculating the average characteristic value of each group of data after grouping the data;
similarly, the sequence number l is taken over 1 to M in sequence1For each value of sequence number l, M is obtained correspondingly2Group data;
respectively comparing the average characteristic value of each group of test data with the average characteristic value of all the data in the training set
Figure BDA0003236301610000021
Performing difference, and executing a step seven on the test data group with the 2-norm value of the difference result being greater than the threshold value threshold;
step seven, after the mapping F corresponding to each group of test data is calculated respectively, calculating new test data of the corresponding experiment by using the calculated mapping F;
for a certain value of the serial number l, updating corresponding data in the test set by using the new test data calculated under the value to obtain updated test set data under the value;
in the same way, updated test set data under each value of the serial number l are respectively obtained;
step eight, respectively processing the new test data of each experiment obtained in the step seven by utilizing a CSP feature extraction algorithm to respectively obtain a feature matrix of the new test data; and the spatial filter adopted during the processing is the same as the spatial filter obtained in the fourth step;
classifying the feature matrix of the updated test set data under each value of the serial number l by using the model trained in the fifth step, comparing the classification result with the label of the test set data, and respectively obtaining the classification accuracy of the updated test set data under each value of the serial number l;
selecting a serial number l value corresponding to the highest classification accuracy, and taking the selected value l as the step length of the tested test data;
step nine, classifying the collected actual electroencephalogram signal data to be tested according to the step length of the test data obtained in the step eight;
the concrete process of the ninth step is as follows:
ninthly, firstly, acquiring actual electroencephalogram data to be tested to construct a training set A, wherein the training set A contains acquired M3The time length of each data acquisition is the same;
ninthly, processing the data in the training set A in the third step and the fourth step to obtain a normalized feature matrix of the data in the training set A;
training the SVM function by using the normalized feature matrix of the data in the training set A to obtain a trained model;
step nine, calculating a new threshold value threshold' by using the normalized feature matrix obtained in the step nine two;
step nine and four, collecting the actual electroencephalogram signal data of the tested object in real time;
after calculating the characteristic value of the acquired data, subtracting the characteristic value of the acquired data from the average characteristic value of the data in the training set A, and if the 2-norm value of the subtraction result is less than or equal to the threshold value threshold', classifying the acquired data by using the model trained in the step nine two to obtain a classification result;
otherwise, if the 2-norm value of the difference result is greater than the threshold value threshold', calculating mapping by adopting the method in the seventh step, obtaining mapped new data corresponding to the acquired data according to the mapping, and classifying the mapped new data by using the model trained in the ninth step and the second step to obtain a classification result;
step nine five, real-time collecting the actual electroencephalogram signal data of the tested once again, and calculating the average characteristic value of the data collected at the current time and the data collected in the step nine four; the average characteristic value is differed from the average characteristic value of the data in the training set A, and if the 2-norm value of the difference result is less than or equal to the new threshold value threshold', the data collected at the current time are classified by using the model trained in the step nine two, so that a classification result is obtained; otherwise, if the 2-norm value of the difference result is greater than the new threshold value threshold', calculating mapping by adopting the method in the step seven, obtaining mapped new data corresponding to the current acquired data according to the mapping, and classifying the mapped new data by using the model trained in the step nine two to obtain a classification result;
step nine six, repeatedly executing the process of the step nine five, and when the data acquisition times exceed the step length l of the test data corresponding to the tested object0Previously, the average characteristic values of the data collected from the nine fourth steps are calculated; at the time of exceeding test data step length l0Thereafter, it is calculated that l was most recently acquired0Average eigenvalues of the secondary data;
until the motor imagery brain-computer interface is stopped.
The invention has the beneficial effects that: the method obtains the required mapping according to the sample distribution of the source domain (training set) and the target domain (testing set), the sample distribution of the target domain is mapped into a new sample distribution, the difference between the new sample distribution and the sample distribution of the source domain is minimum, the sample distribution of the target domain is changed through the mapping, so that the consistency of the sample distribution of the target domain and the training domain in the brain-computer interface is maintained, the problem of deviation caused by electroencephalogram signal characteristic migration is solved, and the classification accuracy of the new sample distribution is effectively improved.
Drawings
FIG. 1 is a flow chart of an experimental paradigm for acquiring electroencephalogram signals;
fig. 2 is a flowchart of applying the KL divergence migration learning method to the motor imagery brain-computer interface.
Detailed Description
First embodiment this embodiment will be described with reference to fig. 2. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface described in this embodiment specifically includes the following steps:
taking a person using a brain-computer interface system as a tested subject, carrying out M motor imagery experiments on the tested subject, and collecting electroencephalogram signal data of the tested subject in the whole process from the beginning of the motor imagery experiments to the end of the motor imagery experiments in the process of each motor imagery experiment;
dividing the acquired electroencephalogram data into a training set and a testing set, and numbering the data in the training set and the testing set according to the corresponding experiment sequence, wherein the data in the training set and the data in the testing set are numbered sequentially from 1, and each number corresponds to the electroencephalogram data acquired in the process of one experiment;
step two, intercepting the electroencephalogram signal data of the motor imagery time interval in the training set and the test set;
thirdly, performing band-pass filtering processing on the intercepted electroencephalogram signal data to obtain electroencephalogram signal data subjected to band-pass filtering processing;
the data are subjected to band-pass filtering to filter out high-frequency electro-oculogram and myoelectricity interference, the tested optimal filtering frequency bands are different, in the experiment of the invention, the tested passband cut-off frequency of A01 and A09 is 7-18Hz, and the stopband cut-off frequency is 5-20 Hz; the cut-off frequency of the pass band of A02 and A05 tested is 18-28Hz, and the cut-off frequency of the stop band is 15-33 Hz; the cut-off frequency of the other tested pass bands is 7-30Hz, and the cut-off frequency of the stop band is 4-35 Hz;
processing the training set data and the test set data by adopting a CSP (compact strip service) feature extraction algorithm to obtain a spatial filter;
performing spatial filtering on the electroencephalogram signal data after the band-pass filtering processing corresponding to each experiment in the training set and the testing set by using a spatial filter, respectively obtaining a spatial filtering matrix corresponding to the acquired data of each experiment in the training set and the testing set, respectively processing the spatial filtering matrix, then obtaining a characteristic matrix of the electroencephalogram signal data acquired by the corresponding experiment, and performing normalization processing on the characteristic matrix;
fifthly, training the SVM function by using the normalized feature matrix corresponding to the training set data to obtain a trained model (model);
an LIBSVM tool box in the MATLAB has an integrated SVM function;
calculating a threshold value threshold by using a normalized feature matrix of the training set data, selecting a plurality of groups of test data from the test set data according to rules, and calculating an average feature value of each group of test data by using the normalized feature matrix corresponding to the test set data;
selecting a plurality of groups of test data from the test set data according to the rule, and calculating the average characteristic value of each group of test data by using the normalized characteristic matrix corresponding to the test set data; the specific process comprises the following steps:
number l is 1 to M1Between them in turn, serial number j0In the range of 1 to M2In turn, M2Representing the total experiment times corresponding to the test set data;
the sequence number l starts with 1, and when l is 1, the sequence number j0Take 1,2, …, M in sequence2When the serial number j0When the value of (1) is less than that of the serial number l, the 1 st to the j th times in the test set are carried out0-1 experimental data was selected as a set of data; when j is0When the test is more than or equal to l, the jth in the test set0L times to j times0-1 experimental data was selected as a set of data; up to sequence number j0Traverse 1,2, …, M in turn2Then, together obtain M2Respectively calculating the average characteristic value of each group of data after grouping the data;
similarly, the sequence number l is taken over 1 to M in sequence1For each value of sequence number l, M is obtained correspondingly2Group data;
respectively comparing the average characteristic value of each group of test data with the average characteristic value of all the data in the training set
Figure BDA0003236301610000051
Performing difference, and executing a step seven on the test data group with the 2-norm value of the difference result being greater than the threshold value threshold;
computing
Figure BDA0003236301610000052
Wherein the content of the first and second substances,
Figure BDA0003236301610000053
is the average characteristic value of a certain set of test data if
Figure BDA0003236301610000054
Considering that the characteristic distribution difference of the group of test data and the training set data is large, executing a seventh step by using a KL divergence domain adaptive algorithm, and otherwise, executing an eighth step;
step seven, after the mapping F corresponding to each group of test data is calculated respectively, calculating new test data of the corresponding experiment by using the calculated mapping F;
for a certain value of the serial number l, updating corresponding data in the test set by using the new test data calculated under the value to obtain updated test set data under the value;
in the same way, updated test set data under each value of the serial number l are respectively obtained;
step eight, respectively processing the new test data of each experiment obtained in the step seven by utilizing a CSP feature extraction algorithm to respectively obtain a feature matrix of the new test data; and the spatial filter adopted during the processing is the same as the spatial filter obtained in the fourth step;
classifying the feature matrix of the updated test set data under each value of the serial number l by using the model trained in the fifth step, comparing the classification result with the label of the test set data, and respectively obtaining the classification accuracy of the updated test set data under each value of the serial number l;
selecting a serial number l value corresponding to the highest classification accuracy, and taking the selected value l as the step length of the tested test data;
step nine, classifying the collected actual electroencephalogram signal data to be tested according to the step length of the test data obtained in the step eight.
The problem of electroencephalogram characteristic migration can be effectively solved by migrating existing knowledge to solve the problem of the target field by means of migration learning, and the field adaptation method is more suitable for a motor imagery brain-computer interface due to the characteristic of high adaptability. The basic idea of the domain adaptation method is to establish a mapping, the sample distribution of the target domain is mapped into a new sample distribution, and the difference between the new sample distribution and the sample distribution of the source domain is the minimum.
The present invention measures the magnitude of the difference in sample distribution between the source domain and the target domain using the KL divergence. The smaller the KL divergence value, the smaller the difference in the distribution of the two samples. The required mapping is obtained by minimizing the KL divergence value of the sample distribution of the source domain and the target domain, and the sample distribution of the target domain is changed through the mapping, so that the consistency of the sample distribution of the target domain and the source domain in the brain-computer interface is maintained, and the problem of feature migration is solved.
The second embodiment is as follows: this embodiment will be described with reference to fig. 1. The difference between the present embodiment and the first embodiment is that, in the first step, the specific process of the motor imagery experiment is as follows:
the tested object is aligned with the computer screen, t is 0 at the beginning of the experiment, a fixed cross symbol appears on the screen until t is t1An arrow prompt appears on the screen at the second moment, the arrow prompt points to the left or the right, wherein the arrow prompt points to the left corresponding to the imagination left-hand movement, the arrow prompt points to the right corresponding to the imagination right-hand movement, and the arrow prompt lasts for t0Second until t equals t2The cross symbol disappears at second, at t1～t2Meanwhile, the subject is always subjected to motor imagery;
at t ═ t2After second, the system enters a rest state, and the rest time lasts for t0After second, the current motor imagery experiment is finished; acquiring the current motor imagery experiment from t-0 to t-t2+t0' second of electroencephalogram signal data;
the motor imagery experiment process is repeated for M times for the tested subject.
The method comprises the steps of correspondingly acquiring tested electroencephalogram signals each time a motor imagery experiment is carried out, and acquiring 144 groups of experimental data for the tested person, wherein in the tested person, the frequency of imagining right-hand movement is the same as the frequency of imagining left-hand movement, and the sampling rate of a data set is 250 Hz. The front 34 groups of experimental data are used as training sets to train the model, the rear 110 groups of experimental data are used as test sets, and the experimental degree of freedom is 2, namely the experimental data only contain the left-hand and right-hand motor imagery. And the acquired data is subjected to 0.5-100 Hz band-pass filtering and 50Hz power frequency notch filtering.
Other steps and parameters are the same as those in the first embodiment.
The third concrete implementation mode: in this embodiment, the difference from the first or second embodiment is that t1Has a value of 2, t0Is taken to be 1.25, t2Has a value of 6, t0' takes a value of 1.5.
Other steps and parameters are the same as those in the first or second embodiment.
The fourth concrete implementation mode: the difference between this embodiment and one of the first to third embodiments is that the specific process of the second step is:
and intercepting the electroencephalogram data in the motor imagery time period from the acquired electroencephalogram data by utilizing a time window with the bias of 0.5 second and the window length of 3 seconds.
The collected experimental data are labeled differently according to the experimental contents in different time periods, wherein the 768 label represents the start of the experiment, the 769 label represents the left-hand motor imagery prompt, the 770 label represents the right-hand motor imagery prompt, and the 32766 label represents the start of a new group of experiments. A time window with a bias of 0.5s and a window length of 3s is set to retrieve data corresponding to labels 769 and 770.
In the embodiment, the time window with the window length of 3 seconds is adopted for intercepting, so that data corresponding to the tested reaction time can be removed, and the electroencephalogram data in the motor imagery time interval can be efficiently extracted while the calculated amount is reduced.
Other steps and parameters are the same as those in one of the first to third embodiments.
The fifth concrete implementation mode: the difference between the first embodiment and the fourth embodiment is that in the fourth step, a spatial filter is used to perform spatial filtering on the electroencephalogram signal data after the bandpass filtering processing corresponding to each experiment in the training set and the testing set, so as to obtain a spatial filtering matrix corresponding to each experiment acquisition data in the training set and the testing set, and then the spatial filtering matrix is processed, so as to obtain a feature matrix of the electroencephalogram signal data acquired by the corresponding experiment; the specific process comprises the following steps:
for the electroencephalogram data after the band-pass filtering processing corresponding to a certain experiment, multiplying the data by a spatial filter to obtain a multiplication result which is still a matrix, and solving the variance of the multiplication result to obtain a spatial filtering matrix, namely solving the variance of the data in the matrix obtained by multiplication;
extracting a first row, a second row and the last two rows of the spatial filter matrix to be used as a characteristic matrix of electroencephalogram data acquired in the experiment;
similarly, the feature matrix of the electroencephalogram signal data collected during each experiment in the training set and the testing set is obtained respectively.
Other steps and parameters are the same as in one of the first to fourth embodiments.
The sixth specific implementation mode: the difference between this embodiment and one of the first to fifth embodiments is that, in the sixth step, the threshold is calculated by using the normalized feature matrix of the training set data, and the calculation method is as follows:
selecting electroencephalogram data of experiments from the kth time to the kth + l-1 time in a training set, and taking the selected data as a group of data, wherein the serial number l is 1-M1Sequentially takes values, and the serial number k is between 1 and M1Values are sequentially selected from-l +1, which specifically comprises:
the sequence number l is taken from 1, and when l is 1, k is taken through 1,2, … and M in sequence1L +1, each time k takes a value, a corresponding set of data is obtained, i.e. when l is 1, M is obtained altogether1Group of data l +1, M1Representing the total experiment times corresponding to the training set data; taking l for 2-M times in sequence1Obtaining each group of data;
for any group of obtained data, calculating the average characteristic value of the group of data by using the normalized characteristic matrix corresponding to the group of data
Figure BDA0003236301610000081
Figure BDA0003236301610000082
Wherein, ClkFor a data set in which a left-hand movement is imagined in the set of data, CrkFor a data set in the group of data that imagines right-hand movement,
Figure BDA0003236301610000083
representation set ClkThe total number of experiments corresponding to the data in (1),
Figure BDA0003236301610000084
representation set CrkThe total number of experiments corresponding to the data in (1),
Figure BDA0003236301610000085
representation set ClkThe eigenvalues of the normalized feature matrix of the ith experimental data,
Figure BDA0003236301610000086
representation set CrkThe characteristic value of the normalized characteristic matrix of the j-th experimental data;
after traversing each group of data, respectively obtaining the average characteristic value of each group of data;
calculating average characteristic value of all experimental data in training set
Figure BDA0003236301610000087
Figure BDA0003236301610000088
Wherein, ClData set representing the imaginary left-hand movement in the training set, CrData set, L, representing the imaginary right hand movement in the training setDLRepresentation set ClTotal number of experiments, L, corresponding to data in (1)DRRepresentation set CrThe total number of experiments corresponding to the data in (1),
Figure BDA0003236301610000089
representation set ClThe eigenvalues of the normalized feature matrix of the i' th experimental data,
Figure BDA00032363016100000810
representation set CrThe characteristic value of the normalized characteristic matrix of the j' th experimental data;
using average eigenvalue sums for each set of data
Figure BDA0003236301610000091
Calculating a threshold value threshold:
Figure BDA0003236301610000092
wherein the content of the first and second substances,
Figure BDA0003236301610000093
to represent
Figure BDA0003236301610000094
The 2-norm of (a) of (b),
Figure BDA0003236301610000095
mean eigenvalue of each group of data is represented
Figure BDA00032363016100000914
And after difference is made, respectively calculating 2-norm of difference making results corresponding to each group of data, and taking the calculated maximum 2-norm result as a threshold value threshold.
Other steps and parameters are the same as those in one of the first to fifth embodiments.
The seventh embodiment: the seventh step is to calculate a mapping F corresponding to each set of test data, and then calculate new test data of the corresponding experiment by using the calculated mapping F; the specific process comprises the following steps:
computing ith in training set0Covariance of sub-experimental data
Figure BDA0003236301610000096
Wherein the content of the first and second substances,
Figure BDA0003236301610000097
for the ith in the training set0The secondary experimental data, tr (-) is the trace of the matrix, the mean covariance of the training set data
Figure BDA0003236301610000098
Covariance of nth experimental data in a set of test data
Figure BDA0003236301610000099
Wherein, TnMean covariance of the set of test data for the nth experiment data in the set of test data
Figure BDA00032363016100000910
Comprises the following steps:
Figure BDA00032363016100000911
wherein, Cl1Is shown when j0When < l, the number j0Previous j0Set of left-hand motor imagery data corresponding to SVM classification probability greater than 65% in 1 experimental data, Cr1Is shown when j0When < l, the number j0Previous j0-a set of right-handed motor imagery data with a corresponding SVM classification probability of greater than 65% in 1 experimental data;
Figure BDA00032363016100000912
the probability that the n-th experimental data is classified by the SVM and the result is the left-hand motor imagery is shown,
Figure BDA00032363016100000913
representing the probability that the N-th experimental data is classified into right-hand motor imagery through the SVM, NLAnd NRRespectively represent a set Cl1And Cr1The total experiment times corresponding to the data;
set Cl1The experimental data in (1) is classified as left-hand motor imagery by SVMThe probability of data is greater than 65%; set Cr1The probability of the experimental data in (1) is more than 65% when the experimental data is classified into right-hand motor imagery data by the SVM;
Cl2is shown when j0J is greater than or equal to l in the set of test data0L times to j times01 set of left hand motor imagery data with SVM classification probability greater than 65%, Cr2Is shown when j0J is greater than or equal to l in the set of test data0L times to j times01 set of right-handed motor imagery data with SVM classification probability greater than 65% for experimental data;
by using
Figure BDA0003236301610000101
And
Figure BDA0003236301610000102
computing mappings
Figure BDA0003236301610000103
Calculating the serial number j corresponding to the group of test data according to the mapping F0Test data of
Figure BDA0003236301610000104
Corresponding new test data
Figure BDA0003236301610000105
Figure BDA0003236301610000106
The above processing is performed for each set of test data to be processed.
Other steps and parameters are the same as those in one of the first to sixth embodiments.
The specific implementation mode is eight: the difference between this embodiment and one of the first to seventh embodiments is that the specific process of step nine is:
ninthly, firstly, acquiring actual electroencephalogram data of a tested object to construct a training set A, wherein the training set A isIn the middle, the training set A contains the collected M3The time length of each data acquisition is the same;
ninthly, processing the data in the training set A in the third step and the fourth step to obtain a normalized feature matrix of the data in the training set A;
training the SVM function by using the normalized feature matrix of the data in the training set A to obtain a trained model;
step nine, calculating a new threshold value threshold' by using the normalized feature matrix obtained in the step nine two; the method for calculating the new threshold' is the same as the sixth step;
step nine and four, collecting the actual electroencephalogram signal data of the tested object in real time;
after calculating the characteristic value of the acquired data, subtracting the characteristic value of the acquired data from the average characteristic value of the data in the training set A, and if the 2-norm value of the subtraction result is less than or equal to the threshold value threshold', classifying the acquired data by using the model trained in the step nine two to obtain a classification result;
otherwise, if the 2-norm value of the difference result is greater than the threshold value threshold', calculating mapping by adopting the method in the seventh step, obtaining mapped new data corresponding to the acquired data according to the mapping, and classifying the mapped new data by using the model trained in the ninth step and the second step to obtain a classification result;
step nine five, real-time collecting the actual electroencephalogram signal data of the tested once again, and calculating the average characteristic value of the data collected at the current time and the data collected in the step nine four; the average characteristic value is differed from the average characteristic value of the data in the training set A, and if the 2-norm value of the difference result is less than or equal to the new threshold value threshold', the data collected at the current time are classified by using the model trained in the step nine two, so that a classification result is obtained; otherwise, if the 2-norm value of the difference result is greater than the new threshold value threshold', calculating mapping by adopting the method in the step seven, obtaining mapped new data corresponding to the current acquired data according to the mapping, and classifying the mapped new data by using the model trained in the step nine two to obtain a classification result;
step nine six, repeatedly executing the process of the step nine five, and when the data acquisition times exceed the step length l of the test data corresponding to the tested object0Before (i.e. from step nine to step four, the number of data acquisitions does not exceed the test data step length l)0) Calculating the average characteristic values of the data collected from the nine fourth steps; at the time of exceeding test data step length l0Thereafter, it is calculated that l was most recently acquired0Average eigenvalues of the secondary data;
until the motor imagery brain-computer interface is stopped.
The method of the present embodiment can be applied to any one of the subjects. When the mapping is calculated in the step nine, the data acquisition times do not exceed the step length l of the test data from the step nine to the step four0Mean covariance of time
Figure BDA0003236301610000111
Is the average covariance of all data collected from the ninth four steps, and when the data collection times exceed the step length l of the test data from the ninth four steps0Mean covariance of time
Figure BDA0003236301610000112
Is the most recently acquired l0Mean covariance of secondary data.
The above-described calculation examples of the present invention are merely to explain the calculation model and the calculation flow of the present invention in detail, and are not intended to limit the embodiments of the present invention. It will be apparent to those skilled in the art that other variations and modifications of the present invention can be made based on the above description, and it is not intended to be exhaustive or to limit the invention to the precise form disclosed, and all such modifications and variations are possible and contemplated as falling within the scope of the invention.

Claims (8)

1. A domain adaptation method for solving a feature migration problem in a motor imagery brain-computer interface, the method comprising the steps of:
taking a person using a brain-computer interface system as a tested subject, carrying out M motor imagery experiments on the tested subject, and collecting electroencephalogram signal data of the tested subject in the whole process from the beginning of the motor imagery experiments to the end of the motor imagery experiments in the process of each motor imagery experiment;
dividing the acquired electroencephalogram data into a training set and a testing set, and numbering the data in the training set and the testing set according to the corresponding experiment sequence;
step two, intercepting the electroencephalogram signal data of the motor imagery time interval in the training set and the test set;
thirdly, performing band-pass filtering processing on the intercepted electroencephalogram signal data to obtain electroencephalogram signal data subjected to band-pass filtering processing;
processing the training set data and the test set data by adopting a CSP (compact strip service) feature extraction algorithm to obtain a spatial filter;
performing spatial filtering on the electroencephalogram signal data after the band-pass filtering processing corresponding to each experiment in the training set and the testing set by using a spatial filter, respectively obtaining a spatial filtering matrix corresponding to the acquired data of each experiment in the training set and the testing set, respectively processing the spatial filtering matrix, then obtaining a characteristic matrix of the electroencephalogram signal data acquired by the corresponding experiment, and performing normalization processing on the characteristic matrix;
fifthly, training the SVM function by using the normalized feature matrix corresponding to the training set data to obtain a trained model;
calculating a threshold value threshold by using a normalized feature matrix of the training set data, selecting a plurality of groups of test data from the test set data according to rules, and calculating an average feature value of each group of test data by using the normalized feature matrix corresponding to the test set data;
selecting a plurality of groups of test data from the test set data according to the rule, and calculating the average characteristic value of each group of test data by using the normalized characteristic matrix corresponding to the test set data; the specific process comprises the following steps:
number l is 1 to M1Between them in turn, serial number j0In the range of 1 to M2Wei Yi (an instrument for regulating middle energizer)Value of less, M2Representing the total experiment times corresponding to the test set data;
the sequence number l starts with 1, and when l is 1, the sequence number j0Take 1,2, …, M in sequence2When the serial number j0When the value of (1) is less than that of the serial number l, the 1 st to the j th times in the test set are carried out0-1 experimental data was selected as a set of data; when j is0When the test is more than or equal to l, the jth in the test set0L times to j times0-1 experimental data was selected as a set of data; up to sequence number j0Traverse 1,2, …, M in turn2Then, together obtain M2Respectively calculating the average characteristic value of each group of data after grouping the data;
similarly, the sequence number l is taken over 1 to M in sequence1For each value of sequence number l, M is obtained correspondingly2Group data;
respectively comparing the average characteristic value of each group of test data with the average characteristic value of all the data in the training set
Figure FDA0003236301600000021
Performing difference, and executing a step seven on the test data group with the 2-norm value of the difference result being greater than the threshold value threshold;
step seven, after the mapping F corresponding to each group of test data is calculated respectively, calculating new test data of the corresponding experiment by using the calculated mapping F;
for a certain value of the serial number l, updating corresponding data in the test set by using the new test data calculated under the value to obtain updated test set data under the value;
in the same way, updated test set data under each value of the serial number l are respectively obtained;
step eight, respectively processing the new test data of each experiment obtained in the step seven by utilizing a CSP feature extraction algorithm to respectively obtain a feature matrix of the new test data; and the spatial filter adopted during the processing is the same as the spatial filter obtained in the fourth step;
classifying the feature matrix of the updated test set data under each value of the serial number l by using the model trained in the fifth step, comparing the classification result with the label of the test set data, and respectively obtaining the classification accuracy of the updated test set data under each value of the serial number l;
selecting a serial number l value corresponding to the highest classification accuracy, and taking the selected value l as the step length of the tested test data;
step nine, classifying the collected actual electroencephalogram signal data to be tested according to the step length of the test data obtained in the step eight.
2. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface according to claim 1, wherein in the first step, the specific process of the motor imagery experiment is:
the tested object is aligned with the computer screen, t is 0 at the beginning of the experiment, a fixed cross symbol appears on the screen until t is t1An arrow prompt appears on the screen at the second moment, the arrow prompt points to the left or the right, wherein the arrow prompt points to the left corresponding to the imagination left-hand movement, the arrow prompt points to the right corresponding to the imagination right-hand movement, and the arrow prompt lasts for t0Second until t equals t2The cross symbol disappears at second, at t1～t2Meanwhile, the subject is always subjected to motor imagery;
at t ═ t2After second, the system enters a rest state, and the rest time lasts for t0After second, the current motor imagery experiment is finished; acquiring the current motor imagery experiment from t-0 to t-t2+t0' second of electroencephalogram signal data;
the motor imagery experiment process is repeated for M times for the tested subject.
3. The method of claim 2, wherein the t is a domain adaptation method for solving a feature migration problem in a motor imagery brain-computer interface (MOV-MAI)1Has a value of 2, t0Is taken to be 1.25, t2Has a value of 6, t0' takes a value of 1.5.
4. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface according to claim 3, wherein the specific process of the second step is:
and intercepting the electroencephalogram data in the motor imagery time period from the acquired electroencephalogram data by utilizing a time window with the bias of 0.5 second and the window length of 3 seconds.
5. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface of claim 4, wherein in the fourth step, spatial filtering is performed on the electroencephalogram signal data after the band-pass filtering processing corresponding to each experiment in the training set and the testing set by using a spatial filter, so as to obtain a spatial filtering matrix corresponding to each experiment acquisition data in the training set and the testing set, and then the spatial filtering matrix is processed, so as to obtain a feature matrix of the electroencephalogram signal data acquired by the corresponding experiment; the specific process comprises the following steps:
for the electroencephalogram data after the band-pass filtering processing corresponding to a certain experiment, multiplying the data by a spatial filter, and then solving the variance of the multiplication result to obtain a spatial filtering matrix;
extracting a first row, a second row and the last two rows of the spatial filter matrix to be used as a characteristic matrix of electroencephalogram data acquired in the experiment;
similarly, the feature matrix of the electroencephalogram signal data collected during each experiment in the training set and the testing set is obtained respectively.
6. The domain adaptation method for solving the feature migration problem in the motor imagery brain-computer interface as claimed in claim 5, wherein in the sixth step, the threshold is calculated by using a normalized feature matrix of training set data, and the calculation method is as follows:
selecting electroencephalogram data of experiments from the kth time to the kth + l-1 time in a training set, and taking the selected data as a group of data, wherein the serial number l is 1-M1Sequentially takes values, and the serial number k is between 1 and M1-l +1 in sequenceValues, which are specifically:
the sequence number l is taken from 1, and when l is 1, k is taken through 1,2, … and M in sequence1L +1, each time k takes a value, a corresponding set of data is obtained, i.e. when l is 1, M is obtained altogether1Group of data l +1, M1Representing the total experiment times corresponding to the training set data; taking l for 2-M times in sequence1Obtaining each group of data;
for any group of obtained data, calculating the average characteristic value of the group of data by using the normalized characteristic matrix corresponding to the group of data
Figure FDA0003236301600000031
Figure FDA0003236301600000032
Wherein, ClkFor a data set in which a left-hand movement is imagined in the set of data, CrkFor a data set in the group of data that imagines right-hand movement,
Figure FDA0003236301600000041
representation set ClkThe total number of experiments corresponding to the data in (1),
Figure FDA0003236301600000042
representation set CrkThe total number of experiments corresponding to the data in (1),
Figure FDA0003236301600000043
representation set ClkThe eigenvalues of the normalized feature matrix of the ith experimental data,
Figure FDA0003236301600000044
representation set CrkThe characteristic value of the normalized characteristic matrix of the j-th experimental data;
after traversing each group of data, respectively obtaining the average characteristic value of each group of data;
calculating average characteristic value of all experimental data in training set
Figure FDA0003236301600000045
Figure FDA0003236301600000046
Wherein, ClData set representing the imaginary left-hand movement in the training set, CrData set, L, representing the imaginary right hand movement in the training setDLRepresentation set ClTotal number of experiments, L, corresponding to data in (1)DRRepresentation set CrThe total number of experiments corresponding to the data in (1),
Figure FDA0003236301600000047
representation set ClThe eigenvalues of the normalized feature matrix of the i' th experimental data,
Figure FDA0003236301600000048
representation set CrThe characteristic value of the normalized characteristic matrix of the j' th experimental data;
using average eigenvalue sums for each set of data
Figure FDA0003236301600000049
Calculating a threshold value threshold:
Figure FDA00032363016000000410
wherein the content of the first and second substances,
Figure FDA00032363016000000411
to represent
Figure FDA00032363016000000412
The 2-norm of (a) of (b),
Figure FDA00032363016000000413
mean eigenvalue of each group of data is represented
Figure FDA00032363016000000414
And after difference is made, respectively calculating 2-norm of difference making results corresponding to each group of data, and taking the calculated maximum 2-norm result as a threshold value threshold.
7. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface according to claim 6, wherein in step seven, after a mapping F corresponding to each set of test data is calculated, new test data corresponding to the experiment is calculated using the calculated mapping F; the specific process comprises the following steps:
computing ith in training set0Covariance of sub-experimental data
Figure FDA00032363016000000415
Wherein the content of the first and second substances,
Figure FDA00032363016000000416
for the ith in the training set0The secondary experimental data, tr (-) is the trace of the matrix, the mean covariance of the training set data
Figure FDA00032363016000000417
Covariance of nth experimental data in a set of test data
Figure FDA00032363016000000418
Wherein, TnMean covariance of the set of test data for the nth experiment data in the set of test data
Figure FDA00032363016000000419
Comprises the following steps:
Figure FDA0003236301600000051
wherein, Cl1Is shown when j0When < l, the number j0Previous j0Set of left-hand motor imagery data corresponding to SVM classification probability greater than 65% in 1 experimental data, Cr1Is shown when j0When < l, the number j0Previous j0-a set of right-handed motor imagery data with a corresponding SVM classification probability of greater than 65% in 1 experimental data;
Figure FDA0003236301600000052
the probability that the n-th experimental data is classified by the SVM and the result is the left-hand motor imagery is shown,
Figure FDA0003236301600000053
representing the probability that the N-th experimental data is classified into right-hand motor imagery through the SVM, NLAnd NRRespectively represent a set Cl1And Cr1The total experiment times corresponding to the data;
Cl2is shown when j0J is greater than or equal to l in the set of test data0L times to j times01 set of left hand motor imagery data with SVM classification probability greater than 65%, Cr2Is shown when j0J is greater than or equal to l in the set of test data0L times to j times01 set of right-handed motor imagery data with SVM classification probability greater than 65% for experimental data;
by using
Figure FDA0003236301600000054
And
Figure FDA0003236301600000055
computing mappings
Figure FDA0003236301600000056
Calculating the set of test numbers according to the mapping FAccording to the corresponding serial number j0Test data of
Figure FDA0003236301600000057
Corresponding new test data
Figure FDA0003236301600000058
Figure FDA0003236301600000059
The above processing is performed for each set of test data to be processed.
8. The domain adaptation method for solving the problem of feature migration in the motor imagery brain-computer interface according to claim 7, wherein the specific process of the ninth step is:
ninthly, firstly, acquiring actual electroencephalogram data to be tested to construct a training set A, wherein the training set A contains acquired M3The time length of each data acquisition is the same;
ninthly, processing the data in the training set A in the third step and the fourth step to obtain a normalized feature matrix of the data in the training set A;
training the SVM function by using the normalized feature matrix of the data in the training set A to obtain a trained model;
step nine, calculating a new threshold value threshold' by using the normalized feature matrix obtained in the step nine two;
step nine and four, collecting the actual electroencephalogram signal data of the tested object in real time;
after calculating the characteristic value of the acquired data, subtracting the characteristic value of the acquired data from the average characteristic value of the data in the training set A, and if the 2-norm value of the subtraction result is less than or equal to the threshold value threshold', classifying the acquired data by using the model trained in the step nine two to obtain a classification result;
otherwise, if the 2-norm value of the difference result is greater than the threshold value threshold', calculating mapping by adopting the method in the seventh step, obtaining mapped new data corresponding to the acquired data according to the mapping, and classifying the mapped new data by using the model trained in the ninth step and the second step to obtain a classification result;
step nine five, real-time collecting the actual electroencephalogram signal data of the tested once again, and calculating the average characteristic value of the data collected at the current time and the data collected in the step nine four; the average characteristic value is differed from the average characteristic value of the data in the training set A, and if the 2-norm value of the difference result is less than or equal to the new threshold value threshold', the data collected at the current time are classified by using the model trained in the step nine two, so that a classification result is obtained; otherwise, if the 2-norm value of the difference result is greater than the new threshold value threshold', calculating mapping by adopting the method in the step seven, obtaining mapped new data corresponding to the current acquired data according to the mapping, and classifying the mapped new data by using the model trained in the step nine two to obtain a classification result;
step nine six, repeatedly executing the process of the step nine five, and when the data acquisition times exceed the step length l of the test data corresponding to the tested object0Previously, the average characteristic values of the data collected from the nine fourth steps are calculated; at the time of exceeding test data step length l0Thereafter, it is calculated that l was most recently acquired0Average eigenvalues of the secondary data;
until the motor imagery brain-computer interface is stopped.
CN202111003357.1A 2021-08-30 2021-08-30 Domain adaptation method for solving characteristic migration problem in motor imagery brain-computer interface Active CN113705464B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111003357.1A CN113705464B (en) 2021-08-30 2021-08-30 Domain adaptation method for solving characteristic migration problem in motor imagery brain-computer interface

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111003357.1A CN113705464B (en) 2021-08-30 2021-08-30 Domain adaptation method for solving characteristic migration problem in motor imagery brain-computer interface

Publications (2)

Publication Number Publication Date
CN113705464A true CN113705464A (en) 2021-11-26
CN113705464B CN113705464B (en) 2024-01-09

Family

ID=78656722

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111003357.1A Active CN113705464B (en) 2021-08-30 2021-08-30 Domain adaptation method for solving characteristic migration problem in motor imagery brain-computer interface

Country Status (1)

Country Link
CN (1) CN113705464B (en)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111931656A (en) * 2020-08-11 2020-11-13 西安交通大学 User independent motor imagery classification model training method based on transfer learning
CN112684891A (en) * 2020-12-30 2021-04-20 杭州电子科技大学 Electroencephalogram signal classification method based on multi-source manifold embedding migration
CN113010013A (en) * 2021-03-11 2021-06-22 华南理工大学 Wasserstein distance-based motor imagery electroencephalogram migration learning method

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111931656A (en) * 2020-08-11 2020-11-13 西安交通大学 User independent motor imagery classification model training method based on transfer learning
CN112684891A (en) * 2020-12-30 2021-04-20 杭州电子科技大学 Electroencephalogram signal classification method based on multi-source manifold embedding migration
CN113010013A (en) * 2021-03-11 2021-06-22 华南理工大学 Wasserstein distance-based motor imagery electroencephalogram migration learning method

Also Published As

Publication number Publication date
CN113705464B (en) 2024-01-09

Similar Documents

Publication Publication Date Title
CN110840402B (en) Atrial fibrillation signal identification method and system based on machine learning
CN110070105B (en) Electroencephalogram emotion recognition method and system based on meta-learning example rapid screening
CN110991406B (en) RSVP electroencephalogram characteristic-based small target detection method and system
CN107157477A (en) EEG signals Feature Recognition System and method
CN101515200B (en) Target selecting method based on transient visual evoked electroencephalogram
Xu et al. High accuracy classification of EEG signal
CN102835955A (en) Method of automatically removing ocular artifacts from electroencephalogram signal without setting threshold value
CN106236080B (en) The removing method of myoelectricity noise in EEG signals based on multichannel
CN113065526B (en) Electroencephalogram signal classification method based on improved depth residual error grouping convolution network
CN109124620B (en) Atrial fibrillation detection method, device and equipment
Thomas et al. Adaptive tracking of discriminative frequency components in electroencephalograms for a robust brain–computer interface
Kang et al. A method of denoising multi-channel EEG signals fast based on PCA and DEBSS algorithm
CN113128552A (en) Electroencephalogram emotion recognition method based on depth separable causal graph convolution network
CN113536882B (en) Multi-class motor imagery electroencephalogram signal feature extraction and classification method
CN111820876A (en) Dynamic construction method of electroencephalogram spatial filter
CN103720471B (en) A kind of electricity of the eye based on factorial analysis artefact minimizing technology
CN111832452A (en) Feature optimization and recognition method of special gesture instruction based on electroencephalogram
CN111428601B (en) P300 signal identification method, device and storage medium based on MS-CNN
Liu et al. Motor imagery tasks EEG signals classification using ResNet with multi-time-frequency representation
CN106859641B (en) A method of based on nuclear-magnetism artefact in automatic ICA removal EEG signal
CN117520891A (en) Motor imagery electroencephalogram signal classification method and system
CN113705464A (en) Domain adaptation method for solving feature migration problem in motor imagery brain-computer interface
CN109924974B (en) EcoG-based epileptic neural signal decoding method and cloud decoding system
CN114795247A (en) Electroencephalogram signal analysis method and device, electronic equipment and storage medium
CN110516711B (en) Training set quality evaluation method of MI-BCI system and optimization method of single training sample

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant