CN110781926B - Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction - Google Patents
Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction Download PDFInfo
- Publication number
- CN110781926B CN110781926B CN201910935589.7A CN201910935589A CN110781926B CN 110781926 B CN110781926 B CN 110781926B CN 201910935589 A CN201910935589 A CN 201910935589A CN 110781926 B CN110781926 B CN 110781926B
- Authority
- CN
- China
- Prior art keywords
- image data
- auxiliary
- robust
- matrix
- objective function
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Abstract
The invention provides a robust auxiliary information reconstruction-based multi-spectral image analysis method of a support vector machine, which comprises the following steps: selecting training samples from an original multispectral image data set, respectively calculating a kernel matrix and a robust constraint matrix of all input multispectral image samples and auxiliary image data, defining intermediate variables, and optimizing and solving a quadratic programming problem corresponding to an objective function to obtain optimal variables; based on the current optimal variable, optimizing and solving a reconstruction coefficient matrix; repeating until the model converges, and outputting a robust classifier model; and verifying the classification effect, predicting the label of the test sample possibly with noise, and comparing the label with the real label. The invention considers the difficult situation that the original multispectral data sample and the auxiliary image data possibly have noise, and makes up the defect that the existing method lacks effective solving strategies.
Description
Technical Field
The invention belongs to the technical field of multi-spectral image analysis processing, and particularly relates to a support vector machine multi-spectral image analysis method based on robust auxiliary information reconstruction.
Background
With the rapid development of machine learning, more and more approaches are being made to multi-spectral image analysis processing technology. Image classification, one of the important analysis methods, has been continuously paid attention. Traditional multi-spectral image classification methods often employ a supervised approach to obtain better classification results, i.e., based on a set of samples and their corresponding labels, the traditional supervised approach can train a classification model and then use it to classify unknown test samples. However, the multispectral image sometimes has the problems of insufficient image resolution, inferior spectrum information as compared with hyperspectral image information, and the like, and especially in the case of a small number of marked samples, if an additional image with higher resolution and more spectrum dimensions is obtained as auxiliary information, the image classification and analysis results will be more accurate.
The Support Vector Machine (SVM) is one of the most popular classifiers at present, and has better classification effect on multi-dimensional data with less marked data. However, in reality, the success of the existing methods of SVM is usually achieved when the training and test data are considered to be clean, noise-free. In practice, the training set may be designed according to specific requirements, but it is difficult or even impossible to determine what the test data is, with or without noise. Existing methods lack explicit strategies to address potential noise in the data, and therefore the practical performance of these methods will be compromised. In addition, the existing method often uses auxiliary data as new feature expression to be directly overlapped to the original data feature expression in the training and testing process, and the feature dimension of the multi-spectrum image data is increased so as to improve the classification accuracy. However, in practical situations, it is difficult to collect a large amount of test data into corresponding auxiliary image data, and when the test data cannot collect the auxiliary image data, many existing methods cannot use the auxiliary image data any more, and the auxiliary image data which can be obtained by the training sample is wasted. At the same time, we cannot always ensure that the acquired auxiliary image data are always accurate and noiseless, and if noise exists in the auxiliary image data, their addition may not ensure that a correct judgment is made, thereby affecting the performance of the model. In addition, when only a portion of the training samples has auxiliary information, the existing method can only utilize the portion of the existing auxiliary information, and does not consider possible auxiliary information expressions that can learn the remaining training samples with the existing auxiliary information.
Therefore, there is a need for a multi-spectral image classification analysis method that can generate auxiliary information while being robust to noise present in the data samples and auxiliary image information, and a model of the method can make best use of the existing auxiliary information as possible while tolerating greater data noise, thereby achieving better classification results. The research team of the applicant develops a robust support vector machine multi-spectral image analysis method by utilizing auxiliary information in the day before, and the robustness of the model is enhanced by adding a robust regularization function to an improved support vector machine classification method in consideration of the disturbance of noise to the original image sample data and the auxiliary image data. However, the practice proves that the auxiliary information is not necessarily complete, and further improvement schemes are required.
Disclosure of Invention
In order to solve the technical problems, the invention provides a multi-spectral band image analysis method of a support vector machine based on reconstruction of robust auxiliary information, which can reconstruct unknown image auxiliary information by using the existing auxiliary image information, considers disturbance of noise on original image sample data and auxiliary image data, and enhances the robustness of a model by adding a robust regularization function into an improved support vector machine classification method and considering the mutual expression characteristic among the auxiliary information.
The technical scheme adopted by the invention provides a support vector machine multispectral image analysis method based on robust auxiliary information reconstruction, which comprises the following steps:
step 1, selecting n image samples and labels from an original multispectral image data set as training samples to obtain auxiliary image datal is the number of auxiliary image data, where l < n, corresponding auxiliary image data, reconstructing unknown training auxiliary image data +.>Then, the reconstruction coefficient matrix V= [ V ] is recorded 1 ,…,v n ]，v 1 ,…,v n Reconstructing coefficient vectors for the corresponding auxiliary data;
step 2, computing the kernel matrices K and K of all the input multispectral image samples and auxiliary image data respectively * ；
Step 3, calculating a robust constraint matrix H of each image sample and auxiliary image data respectively s ；
Step 4, calculating a matrixAnd->Wherein ρ and λ are respectively preset hyper-parameters defining an intermediate variable comprising μ= [ α ] T ,β T ] T Alpha and beta are Lagrangian multipliers, and a quadratic programming problem corresponding to an objective function is optimized and solved to obtain an optimal variable mu;
step 5, optimizing and solving a reconstruction coefficient matrix V based on the current mu;
step 6, repeating the steps 3 to 5 until the model converges, and outputting a robust classifier model;
and 7, verifying the classification effect, predicting the label of the test sample possibly with noise, and comparing the label with the real label.
In step 4, moreover, the objective function is as follows,
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the product of vector element correspondence, alpha and beta are Lagrangian multipliers, y= [ y ] 1 ,…,y n ]Is the label of the training sample, 1 represents the vector with all elements of 1, the upper right corner mark T represents the transposition of the matrix, and C 1 And C 3 Is a pre-set super parameter.
In step 4, an intermediate variable μ= [ α ] is defined T ,β T ] T And the simplification yields the following objective function,
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
and optimizing and solving a quadratic programming problem corresponding to the objective function by using a quadratic programming tool kit to obtain an optimal variable mu.
In step 5, moreover, the implementation of the optimization solution to the reconstruction coefficient matrix V is as follows,
let the intermediate variable matrixSolving the following objective function using the alternate direction multiplier method
Wherein C is 1 And C 2 Is a pre-set super-parameter which is used for the control of the power supply,is a row vector with length l and elements 1, respectively,>the elements of length n-l are row vectors of 1.
In step 6, the difference between the two iterations before and after the objective function value is less than 10 -3 The model is considered to converge.
The beneficial effects of the invention are as follows:
(1) Aiming at the problem of multi-spectral image classification analysis, auxiliary image data such as high-spectral image or high-image resolution is considered to be used simultaneously in the training process, and the discrimination capability of the image data representation and analysis model is enhanced.
(2) The unknown training aid image data is reconstructed taking into account training aid image data that can be acquired using existing parts.
(3) The difficult condition that noise possibly exists in the original multispectral data sample and the auxiliary image data is considered, and the defect that the existing method lacks an effective solution strategy is overcome;
(4) The robustness of the model is considered in theory, so that the final classifier has better stability and robustness for classifying the image data with noise;
(5) The invention has the characteristics of high adaptability and easy solving of the optimization problem, and can effectively solve the problem of classifying images with noise and expanding the images to non-image data.
Drawings
FIG. 1 is a flow chart of an embodiment of the present invention.
Detailed Description
In order to facilitate the understanding and practice of the invention by those of ordinary skill in the art, reference will now be made in detail to the drawings and examples, it being understood that the examples described herein are for illustration and explanation only and are not intended to limit the invention thereto.
Referring to fig. 1, the method for analyzing the multi-spectral band image of the support vector machine based on the reconstruction of the robust auxiliary information provided by the embodiment of the invention comprises the following steps:
step 1: selecting a proportion of image samples { x } from an image dataset 1 ,…,x n N is the number of image samples, provided that auxiliary image data of different spectral bands or higher resolution can be obtainedl is the number of auxiliary image data, where l < n, and the label y= [ y ] 1 ,…,y n ]As training samples, as input data for training models. For reconstructing unknown training aid image data +.>Learning to reconstruct unknown auxiliary data using existing data Is a dictionary composed of the existing privilege information, wherein ψ (·) is a function of which auxiliary image data is +.>Mapping from original feature space into regenerated kernel Hilbert space, v j Is the j-th auxiliary data->Is used to reconstruct the coefficient vector. The reconstruction coefficient vectors of all the auxiliary data are written in a matrix form, which can be expressed as a reconstruction coefficient matrix v= [ V ] 1 ,…,v n ]Wherein the coefficient vector v of any ith auxiliary sample among the first l auxiliary samples i The coefficients are vectors with the i-th element being 1 and the remainder being 0.
The training data are clean and noise free. The test data is subject to random noise. For example, the image net2012 match single target classification data set can be used, and the target in-frame area where the target is located is taken as auxiliary image data. 40% of the samples were selected as training image samples, 30% as validation set, and the remainder as test set.
Step 2: the kernel matrices K and K of all training image samples and auxiliary image data are calculated separately * 。
In an embodiment, each image sample x is calculated using a gaussian kernel function i And x j Nuclear function value betweene is natural logarithm, x i Represents the firsti training image samples, x j Represents the jth training image sample, phi (·) is a function, x will be i Mapping from the original feature space into the regenerated nuclear hilbert space,<,>representing the inner product, wherein gamma is a preset super parameter, and the value range is preferably 10 -3 ,…,10 3 ]Adjusting the middle part; calculate each auxiliary data vector +.>And->Nuclear function value between->Wherein the method comprises the steps ofRepresenting the ith auxiliary image sample, +.>Representing the jth auxiliary image sample. Building a kernel matrix K and K of image samples and auxiliary image data * Wherein K is ij ＝k(x i ,x j ) Represents the elements of the ith row and jth column of matrix K,>representative matrix K * The i-th row and j-th column of (c).
Step 3: robust constraint matrix H for separately computing each image sample and auxiliary image data s ，H s The ith row and the jth column of (a) are the gradient of the ith training sample and the ith training sample kernel function, and the gradient of the jth training sample and the jth training sample kernel function, i.e. the kernel function is respectively applied to x j And x s The bias is derived and then the inner product between the two gradients is calculated, where i, j and s are taken from 1 to n.
Step 4: computing a matrixMatrix->Wherein ρ and λ are respectively preset super parameters, and ρ ranges from [10 ] -2 ,…,10 2 ]Is regulated in [10 ] -5 ,…,10 1 ]In-range adjustment, the upper right corner mark T represents the transpose operation, resulting in the objective function as follows:
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
wherein, the liquid crystal display device comprises a liquid crystal display device,representing the product of vector element correspondence, alpha and beta are Lagrangian multipliers, y= [ y ] 1 ,…,y n ]Is the label of the training sample, 1 represents the vector with all elements of 1, the upper right corner mark T represents the transposition of the matrix, and C 1 And C 3 Is a preset super parameter, preferably in the range of [10 ] -2 ,…,10 2 ]And (3) adjusting.
Definition of the intermediate variable μ= [ α ] T ,β T ] T And to simplify the expression of the subsequent objective function, the following objective function is obtained:
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
optimizing and solving a quadratic programming problem corresponding to the objective function by using the existing quadratic programming tool kit to obtain an optimal variable mu; .
Step 5: based on the current optimal mu, let the intermediate variable matrixTo solve the reconstruction coefficient matrix V, the following objective function is solved, in particular using the Alternate Direction Multiplier Method (ADMM)
Wherein C is 1 And C 2 Is a preset super parameter, preferably in the range of [10 ] -2 ,…,10 2 ]And (3) adjusting.Is a row vector with length l and elements 1, respectively,>a row vector of elements of length n-l are all 1,
step 6: repeating steps 4 and 5 iteratively, when the optimization changes of alpha, beta and V are smaller, for example, the difference value between the two iterations before and after the corresponding objective function value is smaller than 10 -3 The model is considered to be converged, and a robust classifier model is output;
step 7: and verifying classification effect, predicting labels of test samples possibly with noise, comparing the labels with real labels, and calculating the overall classification accuracy (the ratio of the number of the predicted correct samples to the number of all samples).
The implementation platform of the embodiment of the invention is MATLAB software, and the basis of data reading and writing, basic mathematical operation, optimization solution and the like is a well-known technology in the technical field, and is not described herein.
In the specific implementation, the automatic operation of the flow can be realized in a software mode. The means of operating the process should also be within the scope of the present invention.
It should be understood that the parts not specifically described in the present specification belong to the prior art, and the above description of the preferred embodiment is not to be construed as limiting the scope of the present invention, and those skilled in the art can make substitutions or modifications without departing from the scope of the present invention as defined by the appended claims, and the scope of the present invention shall be defined by the appended claims.
Claims (4)
1. The multi-spectral band image analysis method of the support vector machine based on the robust auxiliary information reconstruction is characterized by comprising the following steps of:
step 1, selecting n image samples and labels from an original multispectral image data set as training samples to obtain auxiliary image datal is the number of auxiliary image data, where l<n, corresponding auxiliary image data, reconstructing unknown training auxiliary image data +.>Then, the reconstruction coefficient matrix V= [ V ] is recorded 1 ,…,v n ]，v 1 ,…,v n Reconstructing coefficient vectors for the corresponding auxiliary data;
step 2, computing the kernel matrices K and K of all the input multispectral image samples and auxiliary image data respectively * ；
Step 3, calculating a robust constraint matrix H of each image sample and auxiliary image data respectively s ；
Step 4, calculating a matrixAnd->Wherein ρ and λ are respectively preset hyper-parameters defining an intermediate variable comprising μ= [ α ] T ,β T ] T Alpha and beta are Lagrangian multipliers, and a quadratic programming problem corresponding to an objective function is optimized and solved to obtain an optimal variable mu;
step 5, optimizing and solving a reconstruction coefficient matrix V based on the current mu;
step 6, repeating the steps 4 to 5 until the model converges, and outputting a robust classifier model;
step 7, verifying the classification effect, predicting the label of the test sample possibly having noise, and comparing the label with the real label;
in step 4, the objective function is as follows,
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
wherein α Σy represents the product of vector elements, α and β are lagrangian multipliers, y= [ y ] 1 ,…,y n ]Is the label of the training sample, 1 represents the vector with all elements of 1, the upper right corner mark T represents the transposition of the matrix, and C 1 And C 3 Is a pre-set super parameter.
2. The robust side information reconstruction-based support vector machine multi-spectral band image analysis method according to claim 1, wherein: in step 4, the intermediate variable μ= [ α ] is defined T ,β T ] T And θ=The simplification yields the following objective function,
1 T (α+β-C 1 )＝0
y T α＝0
0≤α≤C 3
β≥0
and optimizing and solving a quadratic programming problem corresponding to the objective function by using a quadratic programming tool kit to obtain an optimal variable mu.
3. The robust side information reconstruction-based support vector machine multi-spectral band image analysis method according to claim 2, wherein: in step 5, the implementation of the optimization solution of the reconstruction coefficient matrix V is as follows,
let the intermediate variable matrixSolving the following objective function using the alternate direction multiplier method
Wherein C is 1 And C 2 Is a pre-set super-parameter which is used for the control of the power supply,is a row vector with length l and elements 1, respectively,>length ofRow vectors with elements of 1, +.>The expression objective function requires reconstruction of coefficient matrix V to be minimum.
4. A support vector machine multi-spectral band image analysis method based on robust side information reconstruction according to claim 1 or 2 or 3, characterized in that: in step 6, the difference between the two iterations before and after the objective function value is less than 10 -3 The model is considered to converge.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910935589.7A CN110781926B (en) | 2019-09-29 | 2019-09-29 | Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910935589.7A CN110781926B (en) | 2019-09-29 | 2019-09-29 | Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110781926A CN110781926A (en) | 2020-02-11 |
| CN110781926B true CN110781926B (en) | 2023-09-19 |
Family
ID=69384986
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910935589.7A Active CN110781926B (en) | 2019-09-29 | 2019-09-29 | Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110781926B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335756A (en) * | 2015-10-30 | 2016-02-17 | 苏州大学 | Robust learning model and image classification system |
| CN105608471A (en) * | 2015-12-28 | 2016-05-25 | 苏州大学 | Robust transductive label estimation and data classification method and system |
| CN106250856A (en) * | 2016-08-03 | 2016-12-21 | 武汉大学 | A kind of high-definition picture scene classification method based on non-supervisory feature learning |
| CN106991382A (en) * | 2017-03-13 | 2017-07-28 | 南京信息工程大学 | A kind of remote sensing scene classification method |
| CN107392128A (en) * | 2017-07-13 | 2017-11-24 | 南京邮电大学 | The robust image recognition methods returned based on double low-rank representations and local constraint matrix |
| CN107451545A (en) * | 2017-07-15 | 2017-12-08 | 西安电子科技大学 | The face identification method of Non-negative Matrix Factorization is differentiated based on multichannel under soft label |
| CN108171261A (en) * | 2017-12-21 | 2018-06-15 | 苏州大学 | Adaptive semi-supervision image classification method, device, equipment and the medium of robust |
| CN109753973A (en) * | 2018-12-21 | 2019-05-14 | 西北工业大学 | High spectrum image change detecting method based on Weighted Support Vector |
| CN109770932A (en) * | 2019-02-21 | 2019-05-21 | 河北工业大学 | The processing method of multi-modal brain neuroblastoma image feature |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8380647B2 (en) * | 2009-08-14 | 2013-02-19 | Xerox Corporation | Training a classifier by dimension-wise embedding of training data |
| US8429102B2 (en) * | 2011-03-31 | 2013-04-23 | Mitsubishi Electric Research Laboratories, Inc. | Data driven frequency mapping for kernels used in support vector machines |
-
2019
- 2019-09-29 CN CN201910935589.7A patent/CN110781926B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105335756A (en) * | 2015-10-30 | 2016-02-17 | 苏州大学 | Robust learning model and image classification system |
| CN105608471A (en) * | 2015-12-28 | 2016-05-25 | 苏州大学 | Robust transductive label estimation and data classification method and system |
| CN106250856A (en) * | 2016-08-03 | 2016-12-21 | 武汉大学 | A kind of high-definition picture scene classification method based on non-supervisory feature learning |
| CN106991382A (en) * | 2017-03-13 | 2017-07-28 | 南京信息工程大学 | A kind of remote sensing scene classification method |
| CN107392128A (en) * | 2017-07-13 | 2017-11-24 | 南京邮电大学 | The robust image recognition methods returned based on double low-rank representations and local constraint matrix |
| CN107451545A (en) * | 2017-07-15 | 2017-12-08 | 西安电子科技大学 | The face identification method of Non-negative Matrix Factorization is differentiated based on multichannel under soft label |
| CN108171261A (en) * | 2017-12-21 | 2018-06-15 | 苏州大学 | Adaptive semi-supervision image classification method, device, equipment and the medium of robust |
| CN109753973A (en) * | 2018-12-21 | 2019-05-14 | 西北工业大学 | High spectrum image change detecting method based on Weighted Support Vector |
| CN109770932A (en) * | 2019-02-21 | 2019-05-21 | 河北工业大学 | The processing method of multi-modal brain neuroblastoma image feature |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110781926A (en) | 2020-02-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112836610B (en) | Land use change and carbon reserve quantitative estimation method based on remote sensing data | |
| Mamalakis et al. | Neural network attribution methods for problems in geoscience: A novel synthetic benchmark dataset | |
| Feilhauer et al. | Multi-method ensemble selection of spectral bands related to leaf biochemistry | |
| Zhang et al. | Domain adaptation under target and conditional shift | |
| US20200134469A1 (en) | Method and apparatus for determining a base model for transfer learning | |
| CN111191514A (en) | Hyperspectral image band selection method based on deep learning | |
| CN105574548A (en) | Hyperspectral data dimensionality-reduction method based on sparse and low-rank representation graph | |
| WO2022095645A1 (en) | Image anomaly detection method for latent space auto-regression based on memory enhancement | |
| US11574198B2 (en) | Apparatus and method with neural network implementation of domain adaptation | |
| CN110852446A (en) | Machine learning model training method, device and computer readable storage medium | |
| CN113887661B (en) | Image set classification method and system based on representation learning reconstruction residual analysis | |
| CN111476272B (en) | Dimension reduction method based on structural constraint symmetric low-rank retention projection | |
| CN110598636B (en) | Ship target identification method based on feature migration | |
| CN115457311B (en) | Hyperspectral remote sensing image band selection method based on self-expression transfer learning | |
| CN108121962B (en) | Face recognition method, device and equipment based on nonnegative adaptive feature extraction | |
| CN113744136A (en) | Image super-resolution reconstruction method and system based on channel constraint multi-feature fusion | |
| Wu et al. | Compiler-aware neural architecture search for on-mobile real-time super-resolution | |
| Zhou et al. | MSAR‐DefogNet: Lightweight cloud removal network for high resolution remote sensing images based on multi scale convolution | |
| CN114694039A (en) | Remote sensing hyperspectral and laser radar image fusion classification method and device | |
| CN108509840B (en) | Hyperspectral remote sensing image waveband selection method based on quantum memory optimization mechanism | |
| CN115601182A (en) | Data analysis method, pricing method and related equipment based on improved XGboost method | |
| CN110781926B (en) | Multi-spectral band image analysis method of support vector machine based on robust auxiliary information reconstruction | |
| Yang | Random distributional response model based on spline method | |
| CN116343032A (en) | Classification method combining Gaussian regression mixed model and MRF hyperspectral function data | |
| Hogland et al. | Improved analyses using function datasets and statistical modeling |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |