CN108596154A - Classifying Method in Remote Sensing Image based on high dimensional feature selection and multi-level fusion - Google Patents
Classifying Method in Remote Sensing Image based on high dimensional feature selection and multi-level fusion Download PDFInfo
- Publication number
- CN108596154A CN108596154A CN201810455398.6A CN201810455398A CN108596154A CN 108596154 A CN108596154 A CN 108596154A CN 201810455398 A CN201810455398 A CN 201810455398A CN 108596154 A CN108596154 A CN 108596154A
- Authority
- CN
- China
- Prior art keywords
- feature
- dimensional
- features
- remote sensing
- sensing image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 53
- 238000000034 method Methods 0.000 title claims abstract description 37
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 38
- 238000000605 extraction Methods 0.000 claims abstract description 8
- 238000010219 correlation analysis Methods 0.000 claims abstract description 7
- 238000003384 imaging method Methods 0.000 claims abstract description 4
- 230000007246 mechanism Effects 0.000 claims abstract description 4
- 238000001228 spectrum Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 24
- 230000008569 process Effects 0.000 claims description 19
- 238000012549 training Methods 0.000 claims description 18
- 238000002790 cross-validation Methods 0.000 claims description 16
- 239000013598 vector Substances 0.000 claims description 16
- 230000009466 transformation Effects 0.000 claims description 13
- 238000012360 testing method Methods 0.000 claims description 9
- 238000005457 optimization Methods 0.000 claims description 7
- 230000003595 spectral effect Effects 0.000 claims description 7
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 238000001914 filtration Methods 0.000 claims description 3
- 238000012706 support-vector machine Methods 0.000 claims 10
- 238000004458 analytical method Methods 0.000 abstract description 3
- 230000009467 reduction Effects 0.000 description 5
- 230000007547 defect Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000007500 overflow downdraw method Methods 0.000 description 2
- 238000003909 pattern recognition Methods 0.000 description 2
- 238000000513 principal component analysis Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/10—Terrestrial scenes
- G06V20/13—Satellite images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/211—Selection of the most significant subset of features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
- G06V10/464—Salient features, e.g. scale invariant feature transforms [SIFT] using a plurality of salient features, e.g. bag-of-words [BoW] representations
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Astronomy & Astrophysics (AREA)
- Remote Sensing (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a kind of Classifying Method in Remote Sensing Image based on high dimensional feature selection and multi-level fusion.First, on the basis of analysis remote sensing images special imaging mechanism, a variety of heterogeneous characteristics such as extraction remote sensing images shape, spectrum, texture.Secondly, on the one hand the higher-dimension heterogeneous characteristic obtained for extraction uses category information subset feature selection algorithm to carry out feature selecting to each high dimensional feature, obtains corresponding one group of optimal low-dimensional character subset；On the other hand every two classes higher-dimension heterogeneous characteristic is merged using discriminating correlation analysis algorithm, obtains corresponding low-dimensional fusion feature.Then, each optimal low-dimensional character subset and low-dimensional fusion feature are separately input to SVM classifier, preliminary classification is carried out to remote sensing images.Finally, the decision level fusion grader weighted based on SVM is designed, each SVM classifier classification results are merged, final remote sensing image classification result is obtained.
Description
Technical Field
The invention relates to a remote sensing image classification method based on high-dimensional feature selection and multilevel fusion, and belongs to the technical field of image processing and pattern recognition.
Background
Remote sensing image classification is a research hotspot in the field of remote sensing image processing and analysis, and people tend to extract increasingly rich feature information in order to obtain increasingly improved classification results, so that the image feature dimension extracted in many remote sensing image classification researches is higher and higher. However, as the feature dimension grows, redundant and even negative correlation information is highly likely to be generated between feature attributes, which not only increases the calculation amount of the classifier sharply, but also affects the performance of classification.
In order to solve the above problems, the conventional solution is to use Principal Component Analysis (PCA) algorithm to perform dimensionality reduction on high-dimensional features, or use Relief algorithm to design 'correlation statistics' to measure the importance of features so as to implement dimensionality reduction. However, these conventional feature selection or dimension reduction methods often only select a group of solutions that are considered to be optimal, and therefore, it is likely that the dimension is reduced while the information of the features is weakened.
In addition, in order to improve the accuracy of image classification, different fusion strategies are receiving more and more attention. For example, the pixel-level fusion can enhance information carried by an image, the feature-level fusion can enhance cross-correlation information among different types of features so as to improve the classification capability of the features, and the decision-level fusion can comprehensively judge a plurality of classification results without affecting the classification capability of each feature so as to obtain a better result. The document (Haghighat M, Abdel-Mottalbeb M, Alhalabi W. characterization Analysis: Real-Time FeatureLevel Fusion for Multimodal biometrical registration [ M ]. IEEE Press,2016.) shows that feature level Fusion strategies generate feature data that, when used for classification, achieve better classification results than pixel level Fusion or decision level Fusion, and therefore feature level Fusion is essential. According to the literature (Kuncheva L I, Bezdek J C, DuinR P W. precision templates for multiple classifier fusion: an experimental composition [ J ]. Pattern Recognition,2001,34(2):299-314.), the effect of classifying by fusing the results using a plurality of classifiers is often better than that of classifying by using a single classifier.
In summary, how to comprehensively extract various heterogeneous features of an image in remote sensing image classification, how to effectively reduce dimensions of high-dimensional heterogeneous features, how to mine the relationship among the heterogeneous features to enable the heterogeneous features to be subjected to feature level fusion, and design a decision level classifier still remain challenging problems at present.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the problems in the prior art, the invention provides a remote sensing image classification method based on high-dimensional feature selection and multilevel fusion, which firstly provides various heterogeneous features for extracting a remote sensing image; then, aiming at the defects that the Feature dimension extracted by the Feature extraction method in the classification of the remote sensing image is generally too high and important Feature information is easily lost only by adopting a common dimension reduction algorithm, an algorithm (SISFS) based on class information subset Feature Selection is provided for carrying out effective Feature Selection on the high-dimensional Feature; meanwhile, in order to solve the problem that the existing fusion algorithm is difficult to acquire the characteristics with identification information when the multi-class heterogeneous characteristics are fused, a heterogeneous characteristic fusion method based on a Differential Correlation Analysis (DCA) is provided; and finally, a satisfactory classification performance is achieved by combining a decision-level fusion strategy.
The technical scheme is as follows: a remote sensing image classification method based on high-dimensional feature selection and multilevel fusion comprises the following steps:
step 1: on the basis of analyzing the special imaging mechanism of the remote sensing image, various heterogeneous characteristics such as the shape, the spectrum, the texture and the like of the remote sensing image are extracted. The specific process is as follows:
firstly, extracting shape features of the remote sensing image. Firstly, extracting dense SURF (Speeded Up Robustoffees) Features and sparse SURF Features of an image, wherein the dense SURF Features are extracted by adopting a grid to divide the image, the central point of the grid is taken as a dense SURF feature point, and the sparse SURF feature points are detected by a traditional SURF feature extraction algorithm. Then, a SURF algorithm is used to extract features in each feature point region. And then, clustering the extracted dense SURF features according to a K-Means algorithm to obtain a class center as an encoding dictionary. Finally, the extracted features are subjected to local Constrained Linear Coding (LLC).
Wherein a set of d-dimensional features obtained by describing one image is denoted by X, where X ═ X1,x2,...,xn]∈Rd ×nAnd a feature dictionary B obtained by K-Means algorithm from the bottom features of all the training images, wherein the dictionary B is [ B ]1,b2,...,bm]∈Rd×mThe objective function encoded with the LLC algorithm is as follows:
wherein the lines indicate multiplication between corresponding pixels, dian adapter representing the ith position, which assigns a weight to each atom based on its similarity to the input descriptor, λ being a parameter greater than 0, α ═ α1,α2,...,αn]，αiCorresponding feature xithe result after LLC encoding, α is the set of encoding results for all features X.
And performing LLC feature coding on the dense SURF to obtain dense SURF-LLC features, then layering according to a Spatial Pyramid Matching model (SPM), performing LLC coding on the sparse SURF features to extract sparse SURF-LLC features, and connecting the dense SURF-LLC features and the sparse SURF-LLC features in series to obtain final shape features which are recorded as DS-SURF-LLC.
Secondly, extracting the spectral characteristics of the remote sensing image. First, the remote sensing image on the original RGB space is transformed into HSV and XYZ spaces. Then, the remote sensing image is divided on the 9 subspaces according to grids, and the mean value and the variance of each divided subregion are obtained. And finally, coding the extracted Mean value and variance information by adopting LLC, layering by adopting an SPM model to obtain final spectral characteristics, and marking as Mean-Std-LLC.
And thirdly, extracting texture features of the remote sensing image. Firstly, Gabor filtering and scale transformation are carried out on the remote sensing image. Then, for the transformed image at each scale, a Complete Local Binary Pattern (CLBP) feature is extracted. And finally, connecting the CLBP features under multiple scales in series to obtain the final multi-scale CLBP texture feature which is recorded as MS-CLBP.
Step 2: and (3) aiming at the extracted high-dimensional heterogeneous features (the shape feature of the remote sensing image, the spectral feature of the remote sensing image and the texture feature of the remote sensing image), performing feature Selection on each high-dimensional feature by adopting a similar information subset feature Selection algorithm (SISFS) to obtain a corresponding group of optimal low-dimensional feature subsets. The specific process is as follows:
firstly, for each extracted high-dimensional heterogeneous feature, in order to correctly evaluate the classification capability of the optimal low-dimensional feature subset to be selected, the high-dimensional feature needs to be sent to an SVM classifier, and then the optimal parameter of the SVM classifier corresponding to the feature is obtained through grid search and cross validation.
Second, four objective functions of the SISFS algorithm are defined, and an optimal set of low-dimensional feature subsets is obtained by optimizing the four objective functions.
Let S be a subset selected from the high-dimensional features X, Y be a class label set corresponding to S, and we define the following four objective functions for S:
(1) correlation:
wherein x isiThe ith column attribute representing S. I (x)i,y)＝H(xi)+H(y)-H(xi,y)。H(xi) Is xiEntropy of (2). H (y) is the entropy of y. H (x)iY) is xiAnd the joint entropy of y.
(2) Redundancy:
(3) feature subset dimension:
f3(S)＝dim(S)
where dim (·) represents the dimensioning function.
(4) Cross validation average accuracy:
wherein, trNumber of samples correctly predicted for testing, tallIn order to test the total number of samples, the training samples are sent to an SVM classifier, and the average accuracy is calculated through 5-fold cross validation.
Third, a subset of class information for the SISFS algorithm is defined. Is provided withRepresents the average classification accuracy of S, SiAnd SjTwo different subsets of information are selected, ifThen call SiIs SjIs used to determine the class information subset.
Fourth, feature subsets are selected. The process mainly comprises two steps:
(1) firstly, a subset is selected as an initial subset S arbitrarily, and the initial subset is optimized by adopting a heuristic global optimization algorithm based on an ethnic group. Namely: first, four objective function values of the initial subset are calculated, and the four objective function values are optimized (i.e., the correlation f is optimized)1(S) and Cross-validation average accuracy f4(S) is maximized while making the redundancy f2(S) and feature subset dimension f3(S) minimize) to continually update the subset until an optimization termination condition is reached. It should be noted that after the process is completed, a set of optimized subsets is generated.
(2) Searching the sub-set with the optimal cross validation average accuracy in the optimized sub-sets1Then selecting sub based on the preset threshold value delta1Subsets sub with differences in average accuracy within a threshold δ2,...,subnAnd finally, all optimized subsets are sub ═ sub1,sub2,...,subn}。
And step 3: and (3) aiming at the high-dimensional heterogeneous features extracted in the step (1), fusing every two types of high-dimensional heterogeneous features by using a Discrimination Correlation Analysis (DCA) algorithm to obtain corresponding low-dimensional fusion features. The specific process is as follows:
first, for any set of high-dimensional heterogeneous feature sets X (p × n for p dimensions), all training classes for each target class in the feature set are first trainedTraining the features of the sample, calculating the average feature vectorThen calculate the average vector of the whole feature set X
Wherein x isjie.X denotes the feature vector of the ith sample of the jth class object. Class j target samples have njAnd (4) respectively. n represents the number of all training samples. J denotes the number of object classes.
Second, for X, find its interspecies scatter matrix Sbx，SbxIs defined as follows:
wherein,
third, to SbxCarrying out diagonalization:
where P is a matrix of orthogonal eigenvectors.A diagonal matrix composed of non-negative eigenvalues.
Fourthly, solving the low-dimensional transformation result of X. Firstly, selecting the eigenvectors corresponding to the first r nonzero eigenvalues from the matrix P,form a matrix Q, can be obtainedThen, an inter-class scatter matrix S is usedbxThe feature vectors of the first r are mapped by: q → phibxQ, to give (phi)bxQ)TSbx(ΦbxQ)＝Λr×r. Then let Wbx＝ΦbxQΛ-1/2Then there isFinally, byThat is, the dimension of the high-dimensional heterogeneous feature set X can be reduced from p × n to r × n, and the specific operations are as follows:where X' is the result of a low-dimensional transformation of X (with dimensions r n).
Fifthly, for any two groups of different types of high-dimensional feature sets X1 and X2, according to the first step to the fourth step, low-dimensional transformation results of the feature sets X1 and X2 can be obtained:
sixth, to make the correlation between different types of features of the same target class stronger, first, their covariance matrices are calculated for the low-dimensional transformed results X1' and X2Then, toUsing singular value decomposition to diagonalize, one can obtain:u and V have no specific physical meaning here. With U and V being singular value decompositionAs a result, the singular vectors representing the non-square matrix are determined by the magnitude of the corresponding singular values. Then, let Where Σ is a diagonal matrix with main diagonal elements not zero, thenFinally, X1 'and X2' are further transformed:
seventh, theAndcarrying out fusion:
Fus12namely the final DCA fusion feature vector.
And 4, step 4: the sub-set of the low-dimensional features of the class information and all the low-dimensional fusion features Fus are combined12Respectively inputting the data into an SVM classifier, initially classifying the remote sensing image, and obtaining the probability P that each sample belongs to different classes under each different characteristicij，PijIndicating the probability that the sample belongs to the ith class based on the feature j.
And 5: and designing a decision-level fusion classifier based on SVM weighting, fusing classification results of each SVM classifier, and realizing remote sensing image classification. The specific process is as follows:
first, let α be given for each sample imagejWeight (f) representing the jth featureJ represents the number of features), given a set αjThen, the probability that the sample belongs to the i-th class target is calculated
Second, for each sample image, P is calculatedm＝max{P1,P2,...,PJAt this time, the sample is determined as the mth class.
Thirdly, the prediction categories and the real categories of all the samples are compared, so that the total accuracy of the whole training sample is obtainedtrNumber of samples correctly predicted for the entire training sample, tallIs the total number of samples.
fourthly, the first to the third steps are repeated, and all possible alpha is traversedjselecting the weight combination with the highest accuracy as the optimal weight β, and constructing a decision-level fusion classifier:
class＝max{Q1,Q2,...,QJ}
wherein,representing the probability that the test sample belongs to the class i object.
Has the advantages that: the technical scheme provided by the invention is characterized by comprising the following related actions: extracting various heterogeneous characteristics of the remote sensing image; aiming at the defects that the feature dimension extracted by the feature extraction method in the classification of the remote sensing image is generally too high and important feature information is easily lost only by adopting a common dimension reduction algorithm, the algorithm based on class information subset feature selection is provided for carrying out effective feature selection on high-dimensional features; in order to solve the problem that the existing fusion algorithm is difficult to obtain the characteristics with identification information when the various heterogeneous characteristics are fused, a heterogeneous characteristic fusion method based on identification correlation analysis is provided; and a decision-level fusion strategy and the like are provided, so that the classification performance is improved in various aspects, and a satisfactory classification result is finally obtained.
Drawings
FIG. 1 is a block diagram of the method of the present invention.
Detailed Description
The present invention is further illustrated by the following examples, which are intended to be purely exemplary and are not intended to limit the scope of the invention, as various equivalent modifications of the invention will occur to those skilled in the art upon reading the present disclosure and fall within the scope of the appended claims.
A remote sensing image classification method based on high-dimensional feature selection and multilevel fusion is shown in a block diagram of a method in figure 1 and mainly comprises the following steps.
Step 1: on the basis of analyzing the special imaging mechanism of the remote sensing image, various heterogeneous characteristics such as the shape, the spectrum, the texture and the like of the remote sensing image are extracted. The specific process is as follows:
firstly, extracting shape features of the remote sensing image. Firstly, extracting dense SURF (Speeded Up Robustoffees) Features and sparse SURF Features of an image, wherein the dense SURF Features are extracted by adopting a grid to divide the image, the central point of the grid is taken as a dense SURF feature point, and the sparse SURF feature points are detected by a traditional SURF feature extraction algorithm. Then, a SURF algorithm is used to extract features in each feature point region. And then, clustering the extracted dense SURF features according to a K-Means algorithm to obtain a class center as an encoding dictionary. Finally, the extracted features are subjected to local Constrained Linear Coding (LLC).
Wherein a set of d-dimensional features obtained by describing one image is denoted by X, where X ═ X1,x2,...,xn]∈Rd ×nAnd a feature dictionary B obtained by K-Means algorithm from the bottom features of all the training images, wherein the dictionary B is [ B ]1,b2,...,bm]∈Rd×mThe objective function encoded with the LLC algorithm is as follows:
wherein the lines indicate multiplication between corresponding pixels, dian adapter representing the ith position, which assigns a weight to each atom based on its similarity to the input descriptor, λ being a parameter greater than 0, α ═ α1,α2,...,αn]，αiCorresponding feature xithe result after LLC encoding, α is the set of encoding results for all features X.
And after the dense SURF extraction features are coded, the dense SURF-LLC features are obtained, then layering is carried out according to a Spatial Pyramid Matching model (SPM), the sparse SURF-LLC features are extracted in the same way, and the dense SURF-LLC features and the sparse SURF-LLC features are connected in series to obtain final shape features which are recorded as DS-SURF-LLC.
Secondly, extracting the spectral characteristics of the remote sensing image. First, the remote sensing image on the original RGB space is transformed into HSV and XYZ spaces. Then, the remote sensing image is divided on the 9 subspaces according to grids, and the mean value and the variance of each divided subregion are obtained. And finally, coding the extracted Mean value and variance information by adopting LLC, layering by adopting an SPM model to obtain final spectral characteristics, and marking as Mean-Std-LLC.
And thirdly, extracting texture features of the remote sensing image. Firstly, Gabor filtering and scale transformation are carried out on the remote sensing image. Then, for the transformed image at each scale, a Complete Local Binary Pattern (CLBP) feature is extracted. And finally, connecting the CLBP features under multiple scales in series to obtain the final multi-scale CLBP texture feature which is recorded as MS-CLBP.
Step 2: and (3) for the extracted high-dimensional heterogeneous features, performing Feature Selection on each high-dimensional Feature by adopting a similar information Subset Feature Selection algorithm (SISFS) to obtain a corresponding group of optimal low-dimensional Feature subsets. The specific process is as follows:
firstly, for each extracted high-dimensional heterogeneous feature, in order to correctly evaluate the classification capability of the optimal low-dimensional feature subset to be selected, the high-dimensional feature needs to be sent to an SVM classifier, and then the optimal parameter of the SVM classifier corresponding to the feature is obtained through grid search and cross validation.
Second, four objective functions of the SISFS algorithm are defined, and an optimal set of low-dimensional feature subsets is obtained by optimizing the four objective functions.
Let S be a subset selected from the high-dimensional features X, Y be a class label set corresponding to S, and we define the following four objective functions for S:
(1) correlation:
wherein x isiThe ith column attribute representing S. I (x)i,y)＝H(xi)+H(y)-H(xi,y)。H(xi) Is xiEntropy of (2). H (y) is the entropy of y. H (x)iY) is xiAnd the joint entropy of y.
(2) Redundancy:
(3) feature subset dimension:
f3(S)＝dim(S)
where dim (·) represents the dimensioning function.
(4) Cross validation average accuracy:
wherein, trNumber of samples correctly predicted for testing, tallIn order to test the total number of samples, the training samples are sent to an SVM classifier, and the average accuracy is calculated through 5-fold cross validation.
Third, a subset of class information for the SISFS algorithm is defined. Is provided withRepresents the average classification accuracy of S, SiAnd SjTwo different subsets of information are selected, ifThen call SiIs SjIs used to determine the class information subset.
Fourth, feature subsets are selected. The process mainly comprises two steps:
(1) firstly, a subset is selected as an initial subset S arbitrarily, and the initial subset is optimized by adopting a heuristic global optimization algorithm based on an ethnic group. Namely: first, four objective function values of the initial subset are calculated, and the four objective function values are optimized (i.e., the correlation f is optimized)1(S) and Cross-validation average accuracy f4(S) is maximized while making the redundancy f2(S) and feature subset dimension f3(S) minimization) ofThe subset is continually updated until an optimization termination condition is reached. It should be noted that after the process is completed, a set of optimized subsets is generated.
(2) Searching the sub-set with the optimal cross validation average accuracy in the optimized sub-sets1Then selecting sub based on the preset threshold value delta1Subsets sub with similar average accuracy2,...,subnFinally all optimized subsets, i.e. sub ═ sub }1,sub2,...,subn}。
And 3, fusing every two types of high-dimensional heterogeneous features by using a Discriminative Correlation Analysis (DCA) aiming at the high-dimensional heterogeneous features extracted in the step 1 to obtain corresponding low-dimensional fusion features. The specific process is as follows:
first, for any set of high-dimensional heterogeneous feature sets X (with dimension p × n), first, an average feature vector is calculated for the features of all training samples of each target class in the feature setThen calculate the average vector of the whole feature set X
Wherein x isjie.X denotes the feature vector of the ith sample of the jth class object. Class j target samples have njAnd (4) respectively. n represents the number of all training samples. J denotes the number of object classes.
Second, for X, find its interspecies scatter matrix Sbx，SbxIs defined as follows:
wherein,
third, to SbxCarrying out diagonalization:
where P is a matrix of orthogonal eigenvectors.A diagonal matrix composed of non-negative eigenvalues.
Fourthly, solving the low-dimensional transformation result of X. Firstly, selecting the eigenvectors corresponding to the first r nonzero eigenvalues from the matrix P to form a matrix Q, and obtaining the matrix QThen, an inter-class scatter matrix S is usedbxThe first r most significant feature vectors of (a) are mapped by: q → phibxQ, to give (phi)bxQ)TSbx(ΦbxQ)＝Λr×r. Then let Wbx＝ΦbxQΛ-1/2Then there isFinally, byThat is, the dimension of the high-dimensional heterogeneous feature set X can be reduced from p × n to r × n, and the specific operations are as follows:where X' is the result of a low-dimensional transformation of X (with dimensions r n).
Fifthly, for any two groups of different types of high-dimensional feature sets X1 and X2, according to the first step to the fourth step, low-dimensional transformation results of the feature sets X1 and X2 can be obtained:
sixth, to make the correlation between different types of features of the same target class stronger, first, their covariance matrices are calculated for the low-dimensional transformed results X1' and X2Then, toUsing singular value decomposition to diagonalize, one can obtain:then, letWhere Σ is a diagonal matrix with main diagonal elements not zero, thenFinally, X1 'and X2' are further transformed:
seventh, theAndcarrying out fusion:
Fus12namely the final DCA fusion feature vector.
And 4, step 4: the sub-set of the low-dimensional features of the class information and all the low-dimensional fusion features Fus are combined12Respectively input into an SVM classifier to carry out initial classification on the remote sensing images, and the probability P that each sample belongs to different classes under each different characteristic is obtainedij，PijIndicating the probability that the sample belongs to the ith class based on the feature j.
And 5: and designing a decision-level fusion classifier based on SVM weighting, fusing classification results of each SVM classifier, and realizing remote sensing image classification. The specific process is as follows:
first, let α be given for each sample imagejWeight (f) representing the jth featureJ represents the number of features), given a set αjThen, the probability that the sample belongs to the i-th class target is calculated
Second, for each sample image, P is calculatedm＝max{P1,P2,...,PJAt this time, the sample is determined as the mth class.
Thirdly, the prediction categories and the real categories of all the samples are compared, so that the total accuracy of the whole training sample is obtainedtrNumber of samples correctly predicted for the entire training sample, tallIs the total number of samples.
fourthly, the first to the third steps are repeated, and all possible alpha is traversedjselecting the weight combination with the highest accuracy as the optimal weight β, and constructing a decision-level fusion classifier:
class＝max{Q1,Q2,...,QJ}
wherein,representing the probability that the test sample belongs to the class i object.
Claims (7)
1. A remote sensing image classification method based on high-dimensional feature selection and multilevel fusion is characterized by comprising the following steps:
step 1: on the basis of analyzing a special imaging mechanism of the remote sensing image, extracting various heterogeneous characteristics of the remote sensing image, including the shape, spectrum and texture characteristics of the remote sensing image;
step 2: and (3) for the extracted high-dimensional heterogeneous features, performing Feature Selection on each high-dimensional Feature by adopting a similar information Subset Feature Selection algorithm (SISFS) to obtain a corresponding group of optimal low-dimensional Feature subsets.
And step 3: and aiming at the extracted high-dimensional heterogeneous features, fusing every two types of high-dimensional heterogeneous features by using a Discriminative Correlation Analysis (DCA) to obtain corresponding low-dimensional fusion features.
And 4, step 4: and respectively inputting each optimal low-dimensional feature subset and low-dimensional fusion feature into an SVM (Support vector machine) classifier, and initially classifying the remote sensing image.
And 5: and designing a decision-level fusion classifier based on SVM weighting, fusing classification results of each SVM classifier, and realizing remote sensing image classification.
2. The remote sensing image classification method based on high-dimensional feature selection and multilevel fusion as claimed in claim 1, characterized in that the specific process of step 1 is as follows:
firstly, extracting shape features of a remote sensing image; firstly, extracting dense SURF (Speeded Up robust Features-Bag of Features) Features and sparse SURF Features of an image, wherein the dense SURF Features are extracted by adopting a grid to divide the image, the central point of the grid is taken as a dense SURF feature point, and the sparse SURF feature points are detected by a traditional SURF feature extraction algorithm; then, extracting features in each feature point region by using an SURF algorithm; secondly, clustering the extracted dense SURF characteristics according to a K-Means algorithm to obtain a class center as a coding dictionary; finally, the extracted features are subjected to local Constrained Linear Coding (LLC); after extracting the feature codes, layering the dense SURF-LLC features according to a Spatial Pyramid Matching model (SPM), and connecting the dense SURF-LLC features in series to obtain final shape features which are recorded as DS-SURF-LLC;
secondly, extracting spectral features of the remote sensing image; firstly, converting a remote sensing image on an original RGB space into HSV and XYZ spaces; then, dividing the remote sensing image on the 9 subspaces according to grids respectively, and solving the mean value and the variance of each divided subregion; finally, similar to the method in the first step, the LLC is adopted to encode the extracted Mean value and variance information, and an SPM model is adopted for layering to obtain the final spectral characteristics which are marked as Mean-Std-LLC;
thirdly, extracting texture features of the remote sensing image; firstly, carrying out Gabor filtering and scale transformation on a remote sensing image; then, extracting Complete Local Binary Pattern (CLBP) characteristics of the transformed image under each scale; and finally, connecting the CLBP features under multiple scales in series to obtain the final multi-scale CLBP texture feature which is recorded as MS-CLBP.
3. The remote sensing image classification method based on high-dimensional Feature Selection and multi-level fusion as claimed in claim 1, characterized in that, for the extracted high-dimensional heterogeneous features, a class information Subset Feature Selection algorithm (SISFS) is used to perform Feature Selection on each high-dimensional Feature to obtain a corresponding set of optimal low-dimensional Feature subsets; the specific process is as follows:
firstly, aiming at each high-dimensional heterogeneous feature, in order to correctly evaluate the classification capability of an optimal low-dimensional feature subset to be selected, the high-dimensional feature needs to be sent into an SVM classifier, and then the optimal parameter of the SVM classifier corresponding to the feature is obtained through grid search and cross validation;
secondly, defining four objective functions of a SISFS algorithm, and obtaining an optimal low-dimensional feature subset by optimizing the four objective functions;
setting S as a subset selected from the high-dimensional features X, setting Y as a class label set corresponding to S, and defining the following four objective functions aiming at S: correlation objective function f1(S), redundancy objective function f2(S), feature subset dimension objective function f3(S), cross-validation of average accuracy target function f4(S)；
Thirdly, defining a class information subset of the SISFS algorithm; is provided withRepresents the average classification accuracy of S, SiAnd SjIs to selectTwo different subsets of information are output ifThen call SiIs SjA subset of class information of;
fourthly, selecting a feature subset; the process mainly comprises two steps:
(1) firstly, randomly selecting a subset as an initial subset S, and optimizing the initial subset by adopting a heuristic global optimization algorithm based on an ethnic group, wherein the specific process is as follows: firstly, four objective function values of an initial subset are calculated, and the subset is continuously updated by optimizing the four objective function values until an optimization termination condition is reached; after the process is finished, a group of optimized subsets is generated; four objective function value optimization means to obtain the correlation f1(S) and Cross-validation average accuracy f4(S) is maximized while making the redundancy f2(S) and feature subset dimension f3(S) minimization;
(2) searching the sub-set with the optimal cross validation average accuracy in the optimized sub-sets1Then selecting sub based on the preset threshold value delta1Several subsets with similar average accuracy.
4. The remote sensing image classification method based on high-dimensional feature selection and multilevel fusion as claimed in claim 3, characterized in that the four objective functions are specifically:
(1) correlation:
wherein x isiThe ith column attribute, I (x), representing Si,y)＝H(xi)+H(y)-H(xi,y)。H(xi) Is xiH (y) is the entropy of y, H (x)iY) is xiAnd the joint entropy of y;
(2) redundancy:
(3) feature subset dimension:
f3(S)＝dim(S)
wherein dim (·) represents the dimensionality function;
(4) cross validation average accuracy:
wherein, trNumber of samples correctly predicted for testing, tallIn order to test the total number of samples, the training samples are sent to an SVM classifier, and the average accuracy is calculated through 5-fold cross validation.
5. The remote sensing image classification method based on high-dimensional feature selection and multilevel fusion of claim 1, characterized in that, aiming at the extracted high-dimensional heterogeneous features, each two types of high-dimensional heterogeneous features are fused by using an identification correlation analysis algorithm to obtain corresponding low-dimensional fusion features; the specific process is as follows:
firstly, for any group of high-dimensional heterogeneous feature sets X, the dimension is set as p multiplied by n, and firstly, the average feature vector of all the training samples of each target class in the feature sets is calculated according to the features of all the training samplesThen calculate the average vector of the whole feature set X
Wherein x isjiE X represents the feature vector of the ith sample of the jth class target, and the jth class target sample has njN, representing the number of all training samples,j represents the number of object categories;
second, for X, find its interspecies scatter matrix Sbx，SbxIs defined as follows:
wherein,
third, to SbxCarrying out diagonalization:
wherein P is a matrix of orthogonal eigenvectors,a diagonal matrix composed of non-negative eigenvalues;
fourthly, solving a low-dimensional transformation result of X; firstly, selecting the eigenvectors corresponding to the first r nonzero eigenvalues from the matrix P to form a matrix Q, and obtaining the matrix QThen, an inter-class scatter matrix S is usedbxThe first r most significant feature vectors of (a) are mapped by: q → phibxQ, to give (phi)bxQ)TSbx(ΦbxQ)＝Λr×r(ii) a Then let Wbx＝ΦbxQΛ-1/2Then there isFinally, byThat is, the dimension of the high-dimensional heterogeneous feature set X can be reduced from p × n to r × n, and the specific operations are as follows:where X' is the result of the low dimensional transformation of X, with dimensions r n.
Fifthly, for any two groups of different types of high-dimensional feature sets X1 and X2, according to the first step to the fourth step, low-dimensional transformation results of the feature sets X1 and X2 can be obtained:
sixth, to make stronger correlation between different types of features of the same target class, first, their covariance matrices are calculated for the results X1 'and X2' after low-dimensional transformation, and then diagonalized using singular value decomposition, one can obtain: then, let us say where Σ is a diagonal matrix with a main diagonal element not zero, then finally, X1 'and X2' are further transformed:
seventh, theAndcarrying out fusion:
Fus12namely the final DCA fusion feature vector.
6. The remote sensing image classification method based on high-dimensional feature selection and multilevel fusion of claim 1, characterized in that each optimal low-dimensional feature subset and low-dimensional fusion feature are respectively input into an SVM classifier, the remote sensing image is initially classified, and the probability P that each sample belongs to different classes under each different feature is obtainedij，PijIndicating the probability that the sample belongs to the ith class based on the feature j.
7. The remote sensing image classification method based on high-dimensional feature selection and multi-level fusion as claimed in claim 1, wherein decision-level fusion classifiers based on SVM weighting are designed, and classification results of each SVM classifier are fused to realize remote sensing image classification; the specific process is as follows:
first, let α be given for each sample imagejThe weight of the jth feature is represented,j represents the number of features, given a set of αjThen, the probability that the sample belongs to the i-th class target is calculated
Second, for each sample image, P is calculatedm＝max{P1,P2,...,PJAt this time, the sample is determined as the mth type;
thirdly, the prediction categories and the real categories of all the samples are compared, so that the total accuracy of the whole training sample is obtainedtrNumber of samples correctly predicted for the entire training sample, tallThe total number of samples is obtained;
fourthly, the first to the third steps are repeated, and all possible alpha is traversedjselecting the weight combination with the highest accuracy as the optimal weight β, and constructing a decision-level fusion classifier:
class＝max{Q1,Q2,...,QJ}
wherein,indicating that the test specimen belongs to the i-th classThe probability of the object.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810455398.6A CN108596154B (en) | 2018-05-14 | 2018-05-14 | Remote sensing image classification method based on high-dimensional feature selection and multilevel fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810455398.6A CN108596154B (en) | 2018-05-14 | 2018-05-14 | Remote sensing image classification method based on high-dimensional feature selection and multilevel fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108596154A true CN108596154A (en) | 2018-09-28 |
| CN108596154B CN108596154B (en) | 2021-09-24 |
Family
ID=63637000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810455398.6A Active CN108596154B (en) | 2018-05-14 | 2018-05-14 | Remote sensing image classification method based on high-dimensional feature selection and multilevel fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108596154B (en) |
Cited By (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492700A (en) * | 2018-11-21 | 2019-03-19 | 西安中科光电精密工程有限公司 | A kind of Target under Complicated Background recognition methods based on multidimensional information fusion |
| CN109670418A (en) * | 2018-12-04 | 2019-04-23 | 厦门理工学院 | In conjunction with the unsupervised object identification method of multi-source feature learning and group sparse constraint |
| CN109800812A (en) * | 2019-01-24 | 2019-05-24 | 山东大学第二医院 | CT image classification feature selection approach and system based on counterfeit filter |
| CN110008925A (en) * | 2019-04-15 | 2019-07-12 | 中国医学科学院皮肤病医院 | A kind of skin automatic testing method based on integrated study |
| CN110210534A (en) * | 2019-05-21 | 2019-09-06 | 河海大学 | High score remote sensing images scene multi-tag classification method based on more packet fusions |
| CN111340061A (en) * | 2020-01-18 | 2020-06-26 | 中国人民解放军国防科技大学 | Multi-mode data fusion and classification method based on SVM model parameter optimization |
| CN111598131A (en) * | 2020-04-17 | 2020-08-28 | 北京百度网讯科技有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| CN112364859A (en) * | 2020-11-05 | 2021-02-12 | 哈尔滨工业大学 | Multi-spectral-band infrared target identification method based on feature optimization and fusion |
| CN112464977A (en) * | 2020-10-15 | 2021-03-09 | 深圳先进技术研究院 | Object classification method, computer equipment and storage medium |
| CN113011465A (en) * | 2021-02-25 | 2021-06-22 | 浙江净禾智慧科技有限公司 | Household garbage throwing intelligent supervision method based on grouping multi-stage fusion |
| CN113378941A (en) * | 2021-06-16 | 2021-09-10 | 中国石油大学(华东) | Multi-decision fusion small sample image classification method |
| CN113673441A (en) * | 2021-08-23 | 2021-11-19 | 王彬 | Quantitative variation texture-driven high-resolution remote sensing image classification method |
| CN113673563A (en) * | 2021-07-15 | 2021-11-19 | 浙江大华技术股份有限公司 | Image classification method and device, electronic equipment and computer-readable storage medium |
| CN114461804A (en) * | 2022-02-10 | 2022-05-10 | 电子科技大学 | Text classification method, classifier and system based on key information and dynamic routing |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120207359A1 (en) * | 2011-02-11 | 2012-08-16 | Microsoft Corporation | Image Registration |
| CN102799856A (en) * | 2012-06-15 | 2012-11-28 | 天津大学 | Human action recognition method based on two-channel infrared information fusion |
| CN104036293A (en) * | 2014-06-13 | 2014-09-10 | 武汉大学 | Rapid binary encoding based high resolution remote sensing image scene classification method |
| CN104751477A (en) * | 2015-04-17 | 2015-07-01 | 薛笑荣 | Space domain and frequency domain characteristic based parallel SAR (synthetic aperture radar) image classification method |
| CN106127225A (en) * | 2016-06-13 | 2016-11-16 | 西安电子科技大学 | Semi-supervised hyperspectral image classification method based on rarefaction representation |
| CN106203487A (en) * | 2016-06-30 | 2016-12-07 | 北京航空航天大学 | A kind of image classification method based on Multiple Kernel Learning Multiple Classifier Fusion and device |
| CN107480620A (en) * | 2017-08-04 | 2017-12-15 | 河海大学 | Remote sensing images automatic target recognition method based on heterogeneous characteristic fusion |
-
2018
- 2018-05-14 CN CN201810455398.6A patent/CN108596154B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120207359A1 (en) * | 2011-02-11 | 2012-08-16 | Microsoft Corporation | Image Registration |
| CN102799856A (en) * | 2012-06-15 | 2012-11-28 | 天津大学 | Human action recognition method based on two-channel infrared information fusion |
| CN104036293A (en) * | 2014-06-13 | 2014-09-10 | 武汉大学 | Rapid binary encoding based high resolution remote sensing image scene classification method |
| CN104751477A (en) * | 2015-04-17 | 2015-07-01 | 薛笑荣 | Space domain and frequency domain characteristic based parallel SAR (synthetic aperture radar) image classification method |
| CN106127225A (en) * | 2016-06-13 | 2016-11-16 | 西安电子科技大学 | Semi-supervised hyperspectral image classification method based on rarefaction representation |
| CN106203487A (en) * | 2016-06-30 | 2016-12-07 | 北京航空航天大学 | A kind of image classification method based on Multiple Kernel Learning Multiple Classifier Fusion and device |
| CN107480620A (en) * | 2017-08-04 | 2017-12-15 | 河海大学 | Remote sensing images automatic target recognition method based on heterogeneous characteristic fusion |
Cited By (21)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109492700A (en) * | 2018-11-21 | 2019-03-19 | 西安中科光电精密工程有限公司 | A kind of Target under Complicated Background recognition methods based on multidimensional information fusion |
| CN109492700B (en) * | 2018-11-21 | 2020-09-08 | 西安中科光电精密工程有限公司 | Complex background target identification method based on multi-dimensional information fusion |
| CN109670418A (en) * | 2018-12-04 | 2019-04-23 | 厦门理工学院 | In conjunction with the unsupervised object identification method of multi-source feature learning and group sparse constraint |
| CN109670418B (en) * | 2018-12-04 | 2021-10-15 | 厦门理工学院 | Unsupervised object identification method combining multi-source feature learning and group sparsity constraint |
| CN109800812A (en) * | 2019-01-24 | 2019-05-24 | 山东大学第二医院 | CT image classification feature selection approach and system based on counterfeit filter |
| CN110008925A (en) * | 2019-04-15 | 2019-07-12 | 中国医学科学院皮肤病医院 | A kind of skin automatic testing method based on integrated study |
| CN110210534A (en) * | 2019-05-21 | 2019-09-06 | 河海大学 | High score remote sensing images scene multi-tag classification method based on more packet fusions |
| CN110210534B (en) * | 2019-05-21 | 2022-09-02 | 河海大学 | Multi-packet fusion-based high-resolution remote sensing image scene multi-label classification method |
| CN111340061A (en) * | 2020-01-18 | 2020-06-26 | 中国人民解放军国防科技大学 | Multi-mode data fusion and classification method based on SVM model parameter optimization |
| CN111598131A (en) * | 2020-04-17 | 2020-08-28 | 北京百度网讯科技有限公司 | Image processing method, image processing device, electronic equipment and storage medium |
| US20210232856A1 (en) * | 2020-04-17 | 2021-07-29 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Image processing method, apparatus, electronic device and storage medium |
| US11727676B2 (en) * | 2020-04-17 | 2023-08-15 | Beijing Baidu Netcom Science And Technology Co., Ltd. | Feature fusion method and apparatus for image processing, electronic device and storage medium |
| CN111598131B (en) * | 2020-04-17 | 2023-08-25 | 北京百度网讯科技有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN112464977A (en) * | 2020-10-15 | 2021-03-09 | 深圳先进技术研究院 | Object classification method, computer equipment and storage medium |
| CN112364859A (en) * | 2020-11-05 | 2021-02-12 | 哈尔滨工业大学 | Multi-spectral-band infrared target identification method based on feature optimization and fusion |
| CN112364859B (en) * | 2020-11-05 | 2024-03-19 | 哈尔滨工业大学 | Multi-spectrum infrared target identification method based on feature optimization and fusion |
| CN113011465A (en) * | 2021-02-25 | 2021-06-22 | 浙江净禾智慧科技有限公司 | Household garbage throwing intelligent supervision method based on grouping multi-stage fusion |
| CN113378941A (en) * | 2021-06-16 | 2021-09-10 | 中国石油大学(华东) | Multi-decision fusion small sample image classification method |
| CN113673563A (en) * | 2021-07-15 | 2021-11-19 | 浙江大华技术股份有限公司 | Image classification method and device, electronic equipment and computer-readable storage medium |
| CN113673441A (en) * | 2021-08-23 | 2021-11-19 | 王彬 | Quantitative variation texture-driven high-resolution remote sensing image classification method |
| CN114461804A (en) * | 2022-02-10 | 2022-05-10 | 电子科技大学 | Text classification method, classifier and system based on key information and dynamic routing |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108596154B (en) | 2021-09-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108596154B (en) | Remote sensing image classification method based on high-dimensional feature selection and multilevel fusion | |
| CN110443143B (en) | Multi-branch convolutional neural network fused remote sensing image scene classification method | |
| Boureau et al. | A theoretical analysis of feature pooling in visual recognition | |
| Sikka et al. | Exploring bag of words architectures in the facial expression domain | |
| CN113657425B (en) | Multi-label image classification method based on multi-scale and cross-modal attention mechanism | |
| US8374442B2 (en) | Linear spatial pyramid matching using sparse coding | |
| CN110321830B (en) | Chinese character string picture OCR recognition method based on neural network | |
| Schwartz et al. | Visual material traits: Recognizing per-pixel material context | |
| CN102902979B (en) | A kind of method of synthetic-aperture radar automatic target detection | |
| KR101777601B1 (en) | Distinction method and system for characters written in caoshu characters or cursive characters | |
| Lin et al. | Biomedical image classification based on a cascade of an SVM with a reject option and subspace analysis | |
| CN107480620A (en) | Remote sensing images automatic target recognition method based on heterogeneous characteristic fusion | |
| CN103942562A (en) | Hyperspectral image classifying method based on multi-classifier combining | |
| Zagoris et al. | Image retrieval systems based on compact shape descriptor and relevance feedback information | |
| CN110569860A (en) | Image interesting binary classification prediction method combining discriminant analysis and multi-kernel learning | |
| CN103679187A (en) | Image identifying method and system | |
| Wei et al. | Visual descriptors for scene categorization: experimental evaluation | |
| Simon et al. | Fine-grained classification of identity document types with only one example | |
| Jiang et al. | Joint kernel-based supervised hashing for scalable histopathological image analysis | |
| CN111414958B (en) | Multi-feature image classification method and system for visual word bag pyramid | |
| CN105389588A (en) | Multi-semantic-codebook-based image feature representation method | |
| Abdullah et al. | Ensembles of novel visual keywords descriptors for image categorization | |
| Yuan et al. | Holistic learning-based high-order feature descriptor for smoke recognition | |
| Kumar et al. | Siamese based Neural Network for Offline Writer Identification on word level data | |
| CN104361354B (en) | A kind of large nuber of images sorting technique based on sparse coding K arest neighbors histograms |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |