CN107292329B - Event idea classification method based on CI-CSP algorithm - Google Patents
Event idea classification method based on CI-CSP algorithm Download PDFInfo
- Publication number
- CN107292329B CN107292329B CN201710256389.XA CN201710256389A CN107292329B CN 107292329 B CN107292329 B CN 107292329B CN 201710256389 A CN201710256389 A CN 201710256389A CN 107292329 B CN107292329 B CN 107292329B
- Authority
- CN
- China
- Prior art keywords
- matrix
- characteristic
- csp
- head
- tail
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides an event imagination classification method based on a complete information common space mode, which comprises the following steps: (1) preprocessing the original data, whitening the two types of imagination signal covariance matrixes by using a whitening matrix, and decomposing eigenvalues to obtain an eigenvalue vector B; (2) arranging the characteristic value vectors B in descending order according to the sizes of the characteristic values, and selecting the head characteristic vector B and the tail characteristic vector B1And intermediate feature vector B2(ii) a (3) Calculating head and tail characteristic vector projection matrix Q1And a projection matrix Q of intermediate eigenvectors2(ii) a (4) Respectively passing the two types of preprocessed data through Q1，Q2Projection to ZiThen calculate Z againiAnd obtaining a head-tail characteristic matrix and a middle high-dimensional characteristic matrix. (5) And reducing the dimension of the middle high-dimensional feature matrix to the optimal dimension, and combining the middle high-dimensional feature matrix and the head-tail feature matrix to form a complete feature vector. According to the method, the complete characteristic information extracted by the method of combining the common space mode with the high-dimensional characteristic dimension reduction can effectively improve the accuracy of event imagination signal classification, reduce the training complexity of imagination events, shorten the training time, improve the universality of a BCI system and optimize the classification performance of other CSP improved algorithm families.
Description
Technical Field
The invention belongs to the field of Brain-Computer Interface (BCI) research, relates to a signal processing method of Event Related Potentials (ERP), and particularly relates to superiority of a Complete Information Common Spatial Pattern algorithm (CI-CSP) in Event imagination classification. The complete information-based Common space mode algorithm is characterized in that a Common spatial mode algorithm (CSP) and a dimension reduction algorithm are combined, a large amount of useful characteristic information can be extracted by the traditional CSP algorithm in the process of processing event imagination related potential signals, and the useful characteristic information can be further extracted from the information abandoned in the steps of the traditional CSP algorithm by utilizing the dimension reduction algorithm, so that the classification accuracy is greatly improved.
Background
The brain-machine interface is a man-machine interaction system which does not depend on peripheral nerves and muscle tissues such as limbs of a human being and communicates or controls with the outside. The method establishes connection between a human body and a computer, and utilizes electroencephalogram signals collected from the human brain to control the computer or other external electronic equipment so as to realize human-computer interaction. The research of the brain-computer interface system enables the brain to directly carry out information interaction with the outside, greatly expands the action range of brain functions, reduces the brain will realize the dependence on limbs, and has wide application prospect in the fields of medicine, intelligent control, military and the like. Brain-machine interfaces are often used to assist disabled persons in rehabilitation training or to control wheelchairs; the brain-controlled robot is generally used for brain-controlled special robots in military affairs to work in dangerous or unsuited manual operation environments.
The main task of the brain-computer interface system is to interpret the brain intent. The brain is the central part of human mental activities, and is the command part for receiving external signals to further form consciousness, sending instructions and generating corresponding behaviors. Electroencephalograms (EEG) are the distribution of electrical potentials generated by brain activity on the scalp that reflect brain activity and one's wishes. The electroencephalogram signals of a specific mode can be induced by setting a specific experimental paradigm, and the electroencephalogram data collected on the scalp are analyzed to decode the subjective will of a person.
A large number of experimental studies show that the actual limb movement or the imaginary limb movement of the human body can cause the power spectrum energy of some frequency band signals in a specific area of the cerebral cortex to change, the phenomenon of the power spectrum energy reduction is called an Event-related desynchronization (ERD), and conversely, the phenomenon of the power spectrum energy increase is called an Event-related desynchronization (ERS). When one-sided limb movement is imagined, the contralateral primary sensorimotor cortical region is active and the rhythmic activity at mu and beta frequencies is manifested as a decrease in amplitude, termed the ERD phenomenon, whereas the ipsilateral cortical region is inhibited and the rhythmic activity at mu and beta frequencies is manifested as an increase in amplitude, termed the EDS phenomenon, and similarly, tongue and foot movement is also imagined. Therefore, ERD/ERS waves induced by the motor events of different parts are applied to the brain-computer interface system as classification features, and the limb motor events of different parts are mapped to different control instructions, so that the aim of interpreting the intention of the brain to control external equipment is fulfilled.
At present, a relatively mature CSP algorithm is applied to a motor imagery event electroencephalogram signal feature extraction algorithm, a common spatial filter is constructed through different types of motor imagery, an optimal set of spatial projection directions is found, the projection energy of one type of electroencephalogram signal in the spatial direction is enabled to be maximum, and the projection energy of the other type of electroencephalogram signal in the spatial direction is enabled to be minimum. The invention embeds the idea of dimension reduction algorithm into the CSP algorithm process, thus fundamentally optimizing the CSP algorithm. In the dimension reduction process, errors caused by redundant information and noise information in a high-dimensional space can be reduced, and therefore identification precision is improved. At present, a visualization dimension reduction technology is provided, and the data visualization is realized by reducing the dimensionality of high-dimensional data to two-dimensional or three-dimensional data. The invention utilizes the dimension reduction algorithm to supplement the feature information of the CSP algorithm, and provides an event idea classification method based on the CI-CSP algorithm.
Disclosure of Invention
The invention provides an event imagination classification method of a CI-CSP algorithm combining a common space mode and a high-dimensional feature dimension reduction method, which improves a classic CSP algorithm, and adopts a mapping method to map multi-lead information in a abandoned high-dimensional space in the CSP algorithm to a low-dimensional space on the basis of the CSP algorithm so as to be supplemented to feature information extracted by the traditional CSP to event imagination signals to form complete feature information. The key steps of the CI-CSP algorithm are as follows:
1. preprocessing original four-class event imagination EEG data (respectively imagining left-hand movement, imagining right-hand movement, imagining double-foot movement and imagining tongue movement, wherein the data is data sets IIIa of BCI 2005), wherein the preprocessing comprises the steps of segmenting data, performing spatial filtering by using a Common Average Reference (CAR) method, and performing frequency domain filtering on the data, and the step aims to improve the signal-to-noise ratio of signals;
2. in the training stage, four types of event imagination data are divided into two types, wherein the left hand and the right hand are imagined as one type of data, and the tongue and the foot are imagined as the other type of data;
3. respectively calculating normalized covariance matrixes of two types of sample data, and calculating the sum R of the two types of covariance matrixesc；
4. Summing R of two kinds of covariance matricescCarrying out eigenvalue decomposition and calculating a whitening matrix W;
5. using whitening matrix to respectively correspond to two kinds of covariance matrixes R1，R2Whitening, decomposing the eigenvalue, and calculating the eigenvalue vector B and the eigenvalue lambda1，λ2The sum of their eigenvalues can be proven to be an identity matrix;
6. the characteristic value vectors B are arranged according to the magnitude sequence of the characteristic values, and a certain number of rows before and after the arranged characteristic value vectors B are selected to form a new head characteristic vector B and a new tail characteristic vector B1The rest is composed as an intermediate feature vector B2(classical CSP algorithms would discard the remaining intermediate feature vectors directly);
7. respectively calculating projection matrix Q of head and tail eigenvectors and intermediate eigenvectors1，Q2；
8. Then the preprocessed EEG two kinds of data are respectively passed through Q1，Q2Projecting to obtain a new matrix ZiThen calculate Z againiObtaining a head-to-tail feature matrix and a middle high-dimensional feature matrix lambda'11，λ'12，λ'21，λ'22；
9. Dividing a middle high-dimensional feature matrix lambda'21，λ'22Reducing to the optimal dimension by using a suitable dimension reduction algorithm to obtain omega'21，ω'22；
10. Will feature vector λ'11，λ'12，ω'21，ω'22Complete feature vectors F are assembled.
Drawings
FIG. 1 is a flow chart of an event imagination classification algorithm based on a CI-CSP algorithm
FIG. 2 is a diagram of feature information extracted from intermediate feature vectors using a dimension reduction algorithm
FIG. 3 is a comparison graph of complete feature information and feature information extracted by traditional CSP algorithm
FIG. 4 k3b comparison graph of classification accuracy results of tested event imagination experiment CI-CSP and CSP algorithm
FIG. 5 l1b comparison graph of classification accuracy results of tested event imagination experiment CI-CSP and CSP algorithm
FIG. 6CI-CSP optimized RB-CSP results are plotted
Detailed Description
The specific algorithm steps of the present invention are further described in detail below with reference to the accompanying drawings.
The flow of the CI-CSP algorithm provided by the invention is shown in figure 1:
1. preprocessing an original EEG, and calculating the sum of normalized covariance of two types of EEG data;
let XdD ∈ {1,2} represents EEG signals under two types of imagination events, and after sample data is de-centered, the covariance matrix is:
the sum of the two types of covariance is:
wherein U iscIs RcOf the eigenvector matrix, λcIs RcThe eigenvector matrix corresponds to a diagonal matrix formed by eigenvalues.
2. Computing a whitening matrix W and respectively aligning two types of covariance matrixes R1，R2Whitening is carried out;
the whitening transformation aims at making variance uniform, and the calculation formula is as follows:
two types of covariance matrices may be whitened by the whitening matrix W:
S1and S2There is a common eigenvector, and the sum of the eigenvalue matrix is the identity matrix:
3. arranging the eigenvector matrix B according to the size of the eigenvalue in a descending order, and selecting a head eigenvector matrix B and a tail eigenvector matrix B1And an intermediate eigenvector matrix B2Respectively calculating the head and tail projection matrix Q1Intermediate projection matrix Q2；
Q1，Q2The calculation formula is as follows:
4. passing two classes of EEG data through a projection matrix Q1，Q2To obtain a matrix ZijAnd calculating a covariance matrix lambda' of the data;
in the formula ZijRepresenting class j data via QiThe resulting matrix after projection. Prepared from lambda'21And λ'22Merge into feature f2。
5. The middle high-dimensional feature matrix f2Reducing to the optimal dimension by using a proper dimension reduction algorithm to obtain an intermediate feature matrix f ', and separating f' into omega 'according to two types of sample numbers'21,ω'22That is to say that,
f2＝[ω'21；ω'22](8)
as can be seen from fig. 2, the intermediate feature vector contains a large amount of feature information, and the feature information extracted from the intermediate feature vector can greatly improve the accuracy of classification.
6. Will feature vector λ'11，λ'12，ω'21，ω'22Complete feature vectors F are combined;
fig. 3 shows that the complete feature information F extracted by the CI-CSP is 148 × 13-dimensional feature vectors, the sample number of the two types of data is 148 (the sample number of each type is 74), the feature information extracted by the head-to-tail feature vector is the front 10-dimensional feature vector, the feature information extracted by the middle feature vector is the rear 3-bit feature vector, i.e., the place enclosed by the small purple rectangular box on the right in fig. 3, it can be seen from fig. 3 that the feature information extracted by the middle feature vector is very rich, the two types of data can be clearly distinguished only by naked eyes, the feature value of the first type of data is significantly higher than that of the second type of data (red indicates a large value, and blue indicates a small value), while the head-to-tail feature information extracted by the conventional CSP is shown as a purple rectangular box on the left in fig. 3, and the feature information of the CSP is not as obvious as the complete feature information extracted by the CI-. It can be seen from fig. 2 and 3 that the CI-CSP extracted complete feature information is proved to be superior to the feature information extracted by the conventional CSP both in theory and experiment.
FIG. 4 is a comparison graph of mean classification accuracy of K3b tested four classes of imagination event 60 lead EEG data by using a CI-CSP algorithm and a CSP algorithm, wherein a red straight line on a left graph represents that the mean classification accuracy of 10 times of the CSP algorithm is 91.2%, a blue broken line represents that a middle high-dimensional feature matrix in the CI-CSP algorithm is respectively reduced to 10 times of mean classification accuracy of 2-30 dimensions for comparison, and it can be seen that the classification results of 2-30 taken by dimension reduction parameters of the CI-CSP algorithm are all higher than the CSP algorithm results, and when the dimension reduction parameters are set to be 9, the highest classification accuracy is 93.58%, a right graph in FIG. 4 represents that K3b tested four classes of imagination event 60 lead EEG data are optimized and compared by using CI-CSP to CSP, a blue bar represents a CI-CSP optimization result, and red represents an unoptimiz; the abscissa is the total accuracy of the three trainers (the generic trainer, the left and right hand trainer, the tongue and foot trainer) and the ordinate is the average accuracy, and it can be seen from the figure that the CI-CSP algorithm improves the CSP on each trainer, and the improvement is most obvious particularly on the left and right hand trainers.
Fig. 5 is a comparison graph of the classification accuracy results of the l1b tested event imagery experiment CI-CSP and CSP algorithm, the horizontal and vertical coordinates of the left and right graphs have the same meaning as fig. 4, but the tested experimental imagery data is analyzed from the power spectrum, the ERD/ERS wave energy signature induced by its locomotor events is not significant, and is related to the physical constitution of the individual being tested, the classification accuracy of the tested event imagination experiment data by the CSP algorithm is lower than that of the tested k3b, the average accuracy of 10 times of the method is 86.75%, while the CI-CSP provided by the invention can effectively optimize the robustness and universality of the CSP, and as can be seen from the blue broken line of the left graph in FIG. 5, the average accuracy of the CI-CSP algorithm in dimension reduction parameter selection 2-30 is greatly higher than that of the CSP, even most results are better than that of k3b tested, wherein when the dimension reduction parameter selection 4 is adopted, the accuracy reaches 95.55% at most. As can also be seen in the right panel of FIG. 5, CI-CSP optimizes the CSP most effectively for left and right hand trainers.
The CI-CSP method can also optimize other improved CSP improved algorithms, a comparison graph of the optimization results of the CI-CSP to RB-CSP algorithms (one CSP improved algorithm) is shown in figure 6, the average classification correct rate is higher than that of the RB-CSP when dimension reduction parameters are selected from 2 to 30 as can be seen from the left graph, the correct rate reaches the highest 94.44% when dimension reduction parameters are selected from 3, and the CI-CSP algorithm is also the most obvious RB-CSP optimization effect on a left-hand trainer and a right-hand trainer as can be seen from the right graph.
The complete characteristic information provided by the CI-CSP algorithm provided by the invention can greatly improve the classification accuracy of EEG data of the imagination event, and the algorithm is superior to the CSP algorithm in robustness and universality.
Claims (1)
1. An event idea classification method based on a CI-CSP algorithm comprises the following steps:
(1) calculating covariance matrixes of the two types of EEG signals, and performing eigenvalue decomposition on the covariance matrixes to obtain eigenvalue vectors B;
(2) the characteristic value vectors B are arranged according to the magnitude sequence of the characteristic values, and a certain number of rows before and after the arranged characteristic value vectors B are selected to form a new head characteristic vector B and a new tail characteristic vector B1The middle part is composed into a middle characteristic vector B2；
(3) Respectively calculating projection matrix Q of head and tail eigenvectors and intermediate eigenvectors1，Q2；
(4) Respectively passing the two kinds of data through Q1，Q2Projecting to obtain a new matrix ZiThen calculate Z againiObtaining a head-to-tail feature matrix and a middle high-dimensional feature matrix lambda'11，λ′12，λ′21，λ′22；
(5) Dividing a middle high-dimensional feature matrix lambda'21，λ′22Down to the optimal dimension to get ω'21，ω′22；
(6) The head-to-tail feature matrix and the dimensionality-reduced middle high-dimensional matrix lambda'11，λ′12，ω′21，ω′22Complete feature vectors F are assembled.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710256389.XA CN107292329B (en) | 2017-04-19 | 2017-04-19 | Event idea classification method based on CI-CSP algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710256389.XA CN107292329B (en) | 2017-04-19 | 2017-04-19 | Event idea classification method based on CI-CSP algorithm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107292329A CN107292329A (en) | 2017-10-24 |
| CN107292329B true CN107292329B (en) | 2020-10-16 |
Family
ID=60093861
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710256389.XA Active CN107292329B (en) | 2017-04-19 | 2017-04-19 | Event idea classification method based on CI-CSP algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107292329B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108108763B (en) * | 2017-12-25 | 2021-07-23 | 中国科学院深圳先进技术研究院 | Electroencephalogram classification model generation method and device and electronic equipment |
| CN108294748A (en) * | 2018-01-23 | 2018-07-20 | 南京航空航天大学 | A kind of eeg signal acquisition and sorting technique based on stable state vision inducting |
| CN108415565A (en) * | 2018-02-25 | 2018-08-17 | 西北工业大学 | The machine integrated intelligent control method of unmanned plane brain and technology |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012153965A2 (en) * | 2011-05-09 | 2012-11-15 | 광주과학기술원 | Brain-computer interface device and classification method therefor |
| CN103340624A (en) * | 2013-07-22 | 2013-10-09 | 上海交通大学 | Method for extracting motor imagery electroencephalogram characteristics on condition of few channels |
| CN103425249A (en) * | 2013-09-06 | 2013-12-04 | 西安电子科技大学 | Electroencephalogram signal classifying and recognizing method based on regularized CSP and regularized SRC and electroencephalogram signal remote control system |
| CN104408713A (en) * | 2014-11-10 | 2015-03-11 | 中国科学院深圳先进技术研究院 | Method and system for extracting image characteristics of diffusion tensor |
| CN105956624A (en) * | 2016-05-06 | 2016-09-21 | 东南大学 | Motor imagery electroencephalogram classification method based on space-time-frequency optimization feature sparse representation |
-
2017
- 2017-04-19 CN CN201710256389.XA patent/CN107292329B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2012153965A2 (en) * | 2011-05-09 | 2012-11-15 | 광주과학기술원 | Brain-computer interface device and classification method therefor |
| CN103340624A (en) * | 2013-07-22 | 2013-10-09 | 上海交通大学 | Method for extracting motor imagery electroencephalogram characteristics on condition of few channels |
| CN103425249A (en) * | 2013-09-06 | 2013-12-04 | 西安电子科技大学 | Electroencephalogram signal classifying and recognizing method based on regularized CSP and regularized SRC and electroencephalogram signal remote control system |
| CN104408713A (en) * | 2014-11-10 | 2015-03-11 | 中国科学院深圳先进技术研究院 | Method and system for extracting image characteristics of diffusion tensor |
| CN105956624A (en) * | 2016-05-06 | 2016-09-21 | 东南大学 | Motor imagery electroencephalogram classification method based on space-time-frequency optimization feature sparse representation |
Non-Patent Citations (4)
| Title |
|---|
| Brain computer interface system based on indoor semi-autonomous navigation and motor imagery for unmanned aerial vehicle control;Shi, T et al.;《Expert Systems with Applications》;20151231;第42卷(第9期);第4196-4206页 * |
| Filter Bank Common Spatial Pattern (FBCSP) in brain-computer interface;Ang K K et al.;《IEEE International Joint Conference on Neural Networks》;20081231;第2391-2398页 * |
| 基于DIVA模型的汉语语音脑机接口系统的研究;曾又;《中国优秀硕士学位论文全文数据库 信息科技辑》;20160515;第4.5.2节，第4.6.3节 * |
| 基于正则化CSP和SRC的运动想象脑电信号特征提取与分类算法研究;方贺琪;《中国优秀硕士学位论文全文数据库 信息科技辑》;20141015;正文第19页第7-15行 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107292329A (en) | 2017-10-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Lv et al. | Advanced machine-learning methods for brain-computer interfacing | |
| Qiu et al. | Improved SFFS method for channel selection in motor imagery based BCI | |
| Park et al. | Filter bank regularized common spatial pattern ensemble for small sample motor imagery classification | |
| Bascil et al. | Spectral feature extraction of EEG signals and pattern recognition during mental tasks of 2-D cursor movements for BCI using SVM and ANN | |
| Rathee et al. | Current source density estimation enhances the performance of motor-imagery-related brain–computer interface | |
| Sun et al. | On-line EEG classification for brain-computer interface based on CSP and SVM | |
| CN107292329B (en) | Event idea classification method based on CI-CSP algorithm | |
| Kirar et al. | Composite kernel support vector machine based performance enhancement of brain computer interface in conjunction with spatial filter | |
| Liu et al. | Uncorrelated multiway discriminant analysis for motor imagery EEG classification | |
| Ji et al. | EEG classification for hybrid brain-computer interface using a tensor based multiclass multimodal analysis scheme | |
| Feng et al. | Feature extraction algorithm based on csp and wavelet packet for motor imagery eeg signals | |
| Yin et al. | Optimal channel-based sparse time-frequency blocks common spatial pattern feature extraction method for motor imagery classification | |
| Sun | Extreme energy difference for feature extraction of EEG signals | |
| CN113967022A (en) | Motor imagery electroencephalogram characteristic characterization method based on individual self-adaption | |
| Li et al. | Subject-based dipole selection for decoding motor imagery tasks | |
| Zhao et al. | Temporal and spatial features of single-trial EEG for brain-computer interface | |
| Shan et al. | EEG-based motor imagery classification accuracy improves with gradually increased channel number | |
| Li et al. | Classification of imaginary movements in ECoG | |
| Koldovsky et al. | A treatment of EEG data by underdetermined blind source separation for motor imagery classification | |
| Mohanchandra et al. | Twofold classification of motor imagery using common spatial pattern | |
| Dong et al. | Improved common spatial pattern for brain-computer interfacing | |
| Gupta et al. | Improved classification of motor imagery datasets for BCI by using approximate entropy and WOSF features | |
| He et al. | Channel and trials selection for reducing covariate shift in EEG-based brain-computer interfaces | |
| Liu et al. | Single-trial discrimination of EEG signals for stroke patients: a general multi-way analysis | |
| Li et al. | Enhancing feature extraction with sparse component analysis for brain-computer interface |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |