CN112308111B - Rail surface state identification method based on multi-feature fusion - Google Patents
Rail surface state identification method based on multi-feature fusion Download PDFInfo
- Publication number
- CN112308111B CN112308111B CN202011008823.0A CN202011008823A CN112308111B CN 112308111 B CN112308111 B CN 112308111B CN 202011008823 A CN202011008823 A CN 202011008823A CN 112308111 B CN112308111 B CN 112308111B
- Authority
- CN
- China
- Prior art keywords
- rail surface
- feature
- image
- fusion
- gray
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000004927 fusion Effects 0.000 title claims abstract description 36
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000007500 overflow downdraw method Methods 0.000 claims abstract description 4
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 239000011159 matrix material Substances 0.000 claims description 13
- 238000000605 extraction Methods 0.000 claims description 9
- 238000012360 testing method Methods 0.000 claims description 8
- 238000007619 statistical method Methods 0.000 claims description 7
- 238000004088 simulation Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 238000006243 chemical reaction Methods 0.000 claims description 3
- 238000012549 training Methods 0.000 claims description 3
- 238000012795 verification Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000001808 coupling effect Effects 0.000 description 1
- 230000032798 delamination Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2411—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on the proximity to a decision surface, e.g. support vector machines
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- General Engineering & Computer Science (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a rail surface state identification method based on multi-feature fusion, which comprises the following steps of: s1, acquiring rail surface images under different conditions, and preprocessing the images; s2, extracting the features of the preprocessed image; extracting color features and texture features; s3, fusing color features and texture features in the rail surface image by adopting a serial fusion method to obtain a fusion feature vector of the rail surface image; s4, designing a multi-feature fusion SVM model; and S5, inputting the fusion feature vector obtained in the S3 into a multi-feature fusion SVM model, and outputting a classification result. Compared with a single feature identification method, the method has the advantages that the accuracy is higher, and the precision is more accurate; the performance is good, and the identification effect is good; the method can be applied to machines and solves the problem that the existing machine vision identification method is difficult to be applied to rail surface state identification.
Description
Technical Field
The invention relates to a rail surface identification method, in particular to a rail surface state identification method based on multi-feature fusion.
Background
The effective performance of the traction/braking performance of the rail transit vehicle depends on the adhesion utilization condition when the wheel set and the rail are in mutual contact, the wheel rail contact behavior is complicated and changeable in transient phenomenon due to the fact that the wheel rail adhesion characteristics are caused by the complexity and the strong nonlinear coupling effect, the influence factors are numerous, especially the rail surface state change is an important factor for changing the wheel rail adhesion, the wheel rail adhesion characteristic difference under different rail surface states is large, if the adhesion coefficient of the accumulated snow rail surface is far smaller than that of a dry rail surface, and the rail surface state mutation in the operation process occurs at all times due to the complexity of the rail transit vehicle operation area in China, the effective identification of the rail surface state can be realized, the wheel rail adhesion utilization rate can be improved, and the rail transit vehicle operation efficiency is improved to provide support.
Students at home and abroad have done certain research work on identifying the rail surface state, and there are documents which construct a rail surface state identification model based on a fuzzy rule by observing and measuring the wheel rail adhesion coefficient and utilizing the estimated creep speed. Some problem documents provide a rail surface state identification method based on a BP-Adaboost algorithm based on the adhesion state obtained in real time. However, most of the conventional rail surface state identification methods are realized by constructing a data drive mode of the rail surface state based on state data such as an adhesion coefficient and a creep speed when the vehicle is running, and are actually a posterior state detection method, and it is difficult to perform effective adhesion control in advance for problems such as control and coasting caused by abrupt changes in the rail surface state. In fact, when the 'third media' on the rail surfaces such as water, oil, ice and snow are different, the visual information presented by the rail surfaces has larger difference, such as the roughness of the dry rail surface is larger, the ice covered rail surface is grey white or transparent, and the snow covered rail surface is highly reflective white.
Disclosure of Invention
The invention provides a rail surface state identification method based on multi-feature fusion, aiming at the problems that the rail surface image in the prior art is weak in texture feature and large in visual feature quantity, and the existing machine vision identification method is difficult to be applied to rail surface state identification.
In order to achieve the purpose, the invention adopts the following technical scheme:
a rail surface state identification method based on multi-feature fusion comprises the following steps:
s1, acquiring rail surface images under different conditions, and preprocessing the rail surface images;
s2, extracting the characteristics of the preprocessed rail surface image; extracting color features and texture features; the color feature extraction selects gray features, and the gray feature calculation formula is as follows:
in the above formula
Represents mean value,. Sup.>
Representing variance, σ f Represents the standard deviation; f (x, y) represents the grayscale value of the rail surface image (x, y) point.
S3, fusing the color features and the texture features in the preprocessed rail surface image by adopting a serial fusion method to obtain a fusion feature vector of the rail surface image, wherein the fusion feature vector is as follows:
Features＝[A 1 A 2 ]；
wherein A1 is a color feature, and A2 is a texture feature;
s4, designing a multi-feature fusion SVM model;
s5, inputting the fusion feature vector obtained in the S3 into a multi-feature fusion SVM model, and outputting a classification result;
and S6, verifying the effectiveness and feasibility of the method through simulation.
Further, rail surface images under 4 different conditions were acquired in step S1.
Further, the rail surface images under 4 different conditions include a dry rail surface, a wet rail surface, an oil stain rail surface and an accumulated snow rail surface.
Further, in step S2, a gray scale feature is selected from the color feature extraction, the image is converted into a gray image, and then the gray scale feature is extracted by using a gray scale probability statistical method.
Further, the conversion algorithm for converting the image into the gray image is as follows:
wherein, R represents a red channel in the image, G represents a green channel in the image, and B represents a blue channel in the image.
Furthermore, a statistical analysis method is adopted for extracting the texture characteristics, and a gray level co-occurrence matrix is selected for describing the texture characteristics.
Further, the formula for calculating the gray level co-occurrence matrix is as follows:
in the above formula, the gray level of the image is 0-N, and the co-occurrence matrix is an NxN matrix; where the value of the force of the element p (i, j) located at (i, j) represents the probability of the occurrence of two pixel pairs separated by Δ δ = (Δ x, Δ y), one gray level being i and the other gray level being j.
Further, in step S7, the simulation verification uses MATLAB software to verify the effectiveness and feasibility of the method.
Further, in step S7, 200 rail surface image samples are used in the simulation verification, and the test sample and the training sample are divided according to the following ratio 1.
The invention has the beneficial effects that: compared with a single feature identification method, the identification method is higher in accuracy and more accurate in precision; the performance is good, and the identification effect is good; the method can be applied to machines and solves the problem that the existing machine vision identification method is difficult to be applied to rail surface state identification.
Drawings
FIG. 1 is a flow chart of SVM model design;
FIG. 2 is a gray-level mean scatter plot of four rail planes;
FIG. 3 is a gray scale standard deviation scatter plot of four rail planes;
FIG. 4 is a gray scale variance scattergram for four rail planes;
FIG. 5 is an inertial scatter plot of four rail surfaces;
FIG. 6 is an energy scatter plot of four rail planes;
fig. 7 is a classification result diagram.
Detailed Description
The present invention will be further described with reference to the following embodiments.
A rail surface state identification method based on multi-feature fusion comprises the following steps:
s1, acquiring rail surface images in different conditions, and preprocessing the images;
in step S1, rail surface images of 4 different conditions are acquired.
The 4 rail surface images in different conditions comprise a dry rail surface, a wet rail surface, an oil stain rail surface and an accumulated snow rail surface.
S2, extracting the features of the preprocessed image; the method comprises color feature extraction and texture feature extraction.
In step S2, the grayscale feature is selected from the color feature extraction, the image is first converted into a gray image, and then the grayscale feature is extracted by using a grayscale probability statistical method.
The conversion algorithm for converting the image into the gray image is as follows:
the gray scale feature calculation formula is as follows:
in the above formula
Means for mean value +>
Representing variance, σ f Represents the standard deviation; f (x, y) represents the grayscale value of the rail surface image (x, y) point.
The extraction of the texture features adopts a statistical analysis method, and the description of the texture features adopts a gray level co-occurrence matrix.
The calculation formula of the gray level co-occurrence matrix is as follows:
in the above formula, the gray level of the image is 0 to N, and the co-occurrence matrix is an nxn matrix; where the value of the force of the element p (i, j) located at (i, j) represents the probability of the occurrence of two pixel pairs separated by Δ δ = (Δ x, Δ y), one gray level being i and the other gray level being j.
S3, fusing color features and texture features in the rail surface image by adopting a serial fusion method to obtain a fusion feature vector of the rail surface image;
Features＝[A 1 A 2 ]；
wherein, A1 is a color feature, and A2 is a texture feature.
S4, designing a multi-feature fusion SVM model;
given an input pattern x, set m classification functions as
Ideally, there should be some k e {1, 2.., m }, such that
And satisfies the following conditions: f. of J (x)<0,j =1, · k-1, k +1,. Page, m, the input pattern should belong to class k
f k (x)>τf J (x)<-τ
J＝1,...,k-1,k+1,...,m
In the formula: if τ is greater than 0, then it is determined that the input pattern should belong to the kth class.
S5, inputting the fusion feature vector obtained in the S3 into a multi-feature fusion SVM model, and outputting a classification result;
and S6, verifying the effectiveness and feasibility of the method through simulation.
Simulation experiment
Selecting 50 dry, wet, greasy dirt and snow rail surface pictures respectively, extracting color and texture characteristics of the four rail surface images by adopting a probability statistical method, and obtaining experimental results shown in figures 2-6
From fig. 2-4, it can be seen that the gray feature statistical graphs of the four rail surfaces are layered obviously, and the snow rail surface has smaller gray fluctuation due to one expanse of rail surface, so the gray feature is smaller. And secondly, oil stain rail surfaces, dry rail surfaces and wet rail surfaces.
Fig. 2 is a gray-level mean scatter diagram of four rail surfaces, and it can be found that the four rail surfaces have a layering phenomenon. And when the gray brightness of the snow rail surface is minimum, the average value is minimum. Secondly, greasy dirt rail surface and dry rail surface. While the wet rail surface has the greatest brightness and the average gray scale value.
Fig. 3 is a gray scale standard deviation scatter diagram of four rail surfaces, and a layering phenomenon of the four rail surfaces can also be found. The accumulated snow rail surface has the smallest gray standard deviation due to the single and uniform color. And then the rail surface is oil-polluted and dried. And the distribution of the gray level of the wet rail surface is dispersed, and the standard deviation of the gray level is the largest.
Fig. 4 is a gray variance scattergram of four rail surfaces, and the phenomenon of delamination of the four rail surfaces can also be found. The snow cover rail surface covers an expanse of white due to snow, and the gray level fluctuation is small, so that the variance is minimum. And secondly, oil stain rail surfaces, dry rail surfaces and wet rail surfaces.
Fig. 5 and 6 are scatter diagrams of gray level co-occurrence matrix features of four rail surfaces, respectively. It can be seen from the figure that the four rail surfaces have obvious layering, and the snow rail surface has the smallest CON and the largest ASM due to uniform distribution and complex texture change. And the texture of the oil stain rail surface is the most fine, so CON is the largest and ASM is the smallest.
FIG. 5 is a contrast scattergram of four rail surfaces, where it can be seen that the snow rail surface CON is the smallest and the texture is the most complex; the oil stain rail surface CON is the largest, and the texture is the finest; and the wet rail surface CON approaches the dry rail surface, but the dry rail surface CON is larger, so that the texture of the wet rail surface is finer than that of the dry rail surface. Fig. 6 is a scatter diagram of two angular moments of four rail surfaces, and it can be seen that the states of the four rail surfaces are layered obviously, the accumulated snow rail surfaces are distributed uniformly, the ASM is maximum, and the accumulated snow rail surfaces are wet rail surfaces, dry rail surfaces and greasy rail surfaces.
The features of serial fusion of color features and texture features are used as input, and the output corresponds to different rail surface slippery condition results. 200 rail surface image samples are adopted, and a test sample and a training sample are divided according to the following steps of 1. The test set classification results are shown in fig. 6.
As can be seen from fig. 6: the classification accuracy of the accumulated snow rail surface and the dry rail surface reaches 100 percent, which is greatly related to the obvious characteristics of the accumulated snow rail surface and the dry rail surface. And the texture of the wet rail surface is similar to that of the greasy dirt rail surface, so that the wet rail surface and the greasy dirt rail surface are difficult to distinguish.
As can be seen from fig. 6: only 2 test pictures of 40 test pictures are not identified correctly, and the identification accuracy reaches 95%. Therefore, the SVM based on multi-feature fusion can effectively recognize the rail surface state and has high recognition accuracy.
In order to verify the effectiveness of the method, the same image samples (200) are input into SVM classifiers of three different feature models, and the table 1 shows the identification and comparison results of the rail surface slippery states of the three feature models.
TABLE 1 comparison of recognition results for three feature models
As can be seen from Table 1, the recognition accuracy of the model based on the fusion characteristics is higher than that of the model based on the single characteristics, and the accuracy can reach 95%. The model based on the color features is better than the model based on the texture features in the single feature model, and the reason may be that the rail surface is worn by the wheel rail for a long time and the texture information is weak. The comparison test results of the three models show that the SVM model based on multi-feature fusion has good performance and better recognition effect.
For a running rail transit vehicle, the change of the rail surface state is an important factor causing the change of the wheel rail adhesion characteristics, and the effective identification of the rail surface state is a precondition for detecting the wheel rail adhesion characteristics. The rail surface slippery state image recognition method based on the multi-feature fusion SVM is characterized in that the color feature and the texture feature of the acquired rail surface image are fused in series, a rail surface slippery state recognition model is established by the aid of the SVM, and recognition accuracy of the rail surface image is improved. Finally, through comparison tests of three different characteristic models, comparison results show that the identification precision of the multi-characteristic fusion model is higher than that of a single characteristic model.
The above-mentioned embodiments are merely preferred embodiments for fully illustrating the present invention, and the scope of the present invention is not limited thereto. The equivalent substitution or change made by the technical personnel in the technical field on the basis of the invention is all within the protection scope of the invention.
Claims (5)
1. A rail surface state identification method based on multi-feature fusion is characterized by comprising the following steps:
s1, acquiring rail surface images under different conditions, and preprocessing the rail surface images;
s2, extracting the characteristics of the preprocessed rail surface image; extracting color features and texture features; the color feature extraction selects gray features, and the gray feature calculation formula is as follows:
in the above formula
Represents mean value,. Sup.>
Represents the variance, σ f Represents the standard deviation; f (x, y) represents the gray value of the rail surface image (x, y) point;
s3, fusing the color features and the texture features in the S2 by adopting a serial fusion method to obtain a fusion feature vector of the rail surface image, wherein the fusion feature vector is as follows:
Features＝[A 1 A 2 ]；
wherein A1 is a color feature, and A2 is a texture feature;
s4, designing a multi-feature fusion SVM model;
s5, inputting the fusion feature vector obtained in the S3 into a multi-feature fusion SVM model, and outputting a classification result;
the rail surface images under different conditions in the step S1 comprise a dry rail surface, a wet rail surface, an oil stain rail surface and an accumulated snow rail surface;
in step S2, the extraction of the texture features adopts a statistical analysis method, the description of the texture features adopts a gray level co-occurrence matrix, and the calculation formula of the gray level co-occurrence matrix is as follows:
in the above formula, f 1 Is the moment of inertia, f 2 Is energy; the gray level of the image is 0-N, and the co-occurrence matrix is an NxN matrix; the value of the force of the element p (i, j) located at (i, j) represents the probability that two pixel pairs having a gray level of i and a gray level of j separated by Δ δ = (Δ x, Δ y) will appear.
2. The rail surface state identification method based on multi-feature fusion as claimed in claim 1, wherein in step S2, a gray feature is selected for color feature extraction, the image is converted into a gray image, and then a gray statistical method is used to extract the gray feature.
3. The rail surface state identification method based on multi-feature fusion as claimed in claim 2, wherein the conversion algorithm for converting the image into the gray image is as follows:
wherein, R represents a red channel in the image, G represents a green channel in the image, and B represents a blue channel in the image.
4. The rail surface state identification method based on multi-feature fusion as claimed in claim 1, wherein MATLAB software is adopted to simulate and verify the effectiveness and feasibility of the method.
5. The rail surface state identification method based on multi-feature fusion is characterized in that 200 rail surface image samples are adopted in simulation verification, and a test sample and a training sample are divided according to the ratio of 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011008823.0A CN112308111B (en) | 2020-09-23 | 2020-09-23 | Rail surface state identification method based on multi-feature fusion |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011008823.0A CN112308111B (en) | 2020-09-23 | 2020-09-23 | Rail surface state identification method based on multi-feature fusion |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112308111A CN112308111A (en) | 2021-02-02 |
| CN112308111B true CN112308111B (en) | 2023-04-18 |
Family
ID=74488087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011008823.0A Active CN112308111B (en) | 2020-09-23 | 2020-09-23 | Rail surface state identification method based on multi-feature fusion |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112308111B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113095412B (en) * | 2021-04-14 | 2024-02-09 | 中北大学 | Mixed fine aggregate classification and identification method based on multi-feature fusion and support vector machine |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005786A (en) * | 2015-06-19 | 2015-10-28 | 南京航空航天大学 | Texture image classification method based on BoF and multi-feature fusion |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10706294B2 (en) * | 2018-05-03 | 2020-07-07 | Volvo Car Corporation | Methods and systems for generating and using a road friction estimate based on camera image signal processing |
| CN110782443B (en) * | 2019-10-23 | 2023-04-07 | 四川大学 | Railway track defect detection method and system |
| CN111611956B (en) * | 2020-05-28 | 2023-09-26 | 中国科学院自动化研究所 | Rail detection method and system for subway visual image |
-
2020
- 2020-09-23 CN CN202011008823.0A patent/CN112308111B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105005786A (en) * | 2015-06-19 | 2015-10-28 | 南京航空航天大学 | Texture image classification method based on BoF and multi-feature fusion |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112308111A (en) | 2021-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108830188B (en) | Vehicle detection method based on deep learning | |
| CN111738315B (en) | Image classification method based on countermeasure fusion multi-source transfer learning | |
| CN111368886B (en) | Sample screening-based label-free vehicle picture classification method | |
| CN109118479B (en) | Capsule network-based insulator defect identification and positioning device and method | |
| CN107945153A (en) | A kind of road surface crack detection method based on deep learning | |
| CN109800736A (en) | A kind of method for extracting roads based on remote sensing image and deep learning | |
| CN109376792A (en) | Photovoltaic cell open defect classification method based on multichannel residual error neural network | |
| CN106408015A (en) | Road fork identification and depth estimation method based on convolutional neural network | |
| CN110598693A (en) | Ship plate identification method based on fast-RCNN | |
| CN103745454B (en) | Method for evaluating image processing algorithms or systems based on rank learning | |
| CN109993201A (en) | A kind of image processing method, device and readable storage medium storing program for executing | |
| CN109284779A (en) | Object detection method based on deep full convolution network | |
| CN103544499A (en) | Method for reducing dimensions of texture features for surface defect detection on basis of machine vision | |
| CN101251896B (en) | Object detecting system and method based on multiple classifiers | |
| CN111860459A (en) | Gramineous plant leaf stomata index measuring method based on microscopic image | |
| CN111223087B (en) | Automatic bridge crack detection method based on generation countermeasure network | |
| CN107818321A (en) | A kind of watermark date recognition method for vehicle annual test | |
| CN103824092A (en) | Image classification method for monitoring state of electric transmission and transformation equipment on line | |
| CN112308111B (en) | Rail surface state identification method based on multi-feature fusion | |
| CN113239720A (en) | Subway vehicle running gear fault diagnosis method based on deep migration learning | |
| CN111931805A (en) | Knowledge-guided CNN-based small sample similar abrasive particle identification method | |
| CN104751175A (en) | Multi-label scene classification method of SAR (Synthetic Aperture Radar) image based on incremental support vector machine | |
| CN105913090A (en) | SAR image object classification method based on SDAE-SVM | |
| CN109523514A (en) | To the batch imaging quality assessment method of Inverse Synthetic Aperture Radar ISAR | |
| WO2020119624A1 (en) | Class-sensitive edge detection method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |