CN115908897B - Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images - Google Patents
Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images Download PDFInfo
- Publication number
- CN115908897B CN115908897B CN202211339322.XA CN202211339322A CN115908897B CN 115908897 B CN115908897 B CN 115908897B CN 202211339322 A CN202211339322 A CN 202211339322A CN 115908897 B CN115908897 B CN 115908897B
- Authority
- CN
- China
- Prior art keywords
- power supply
- remote sensing
- model
- station
- speed railway
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Image Analysis (AREA)
Abstract
The invention relates to the technical field of remote sensing image processing, in particular to an intelligent recognition method for a high-speed railway power supply facility based on a high-resolution remote sensing image, which comprises the following steps: step S1: selecting a high-resolution remote sensing image, and judging and drawing samples in the surrounding area of a high-speed train station; step S2: setting initial parameters, calculating a loss function by using samples with a boundary box of the true position of a power supply network rod, and participating in iteration and updating of a parameter weight W and an offset value b in the back propagation process, so that training of a model is completed; step S3: calculating by adopting a road network searching mode; step S4: and randomly selecting a plurality of station images, putting the station images into a trained model to identify the power supply network rods, and checking the identification result. The invention can realize automatic identification and extraction of high-speed railway power supply facilities, greatly improve identification efficiency and ensure the stability of target detection.
Description
Technical Field
The invention relates to the technical field of remote sensing image processing, in particular to an intelligent recognition method for a high-speed railway power supply facility based on high-resolution remote sensing images.
Background
The power source of the high-speed train is electric energy, and the bow net system is a power supply mode of the core of the high-speed train, so that the power supply facility of the high-speed railway mainly refers to a bow net contact net rod in the bow net system. The railway line of the high-speed railway is often erected in a natural environment without smoke, the railway line enters the mountain and goes with water, and the situation that Lin Jinlin goes out is more normal, so that in the detection of the state of railway power supply facilities, especially in the problem of checking whether natural environment and artificial environment near the bow net are dangerous to the bow net or not and even damaging the bow net system, people are organized to enter deep mountain and old forests along the high-speed railway line, the high-speed railway line is involved with water, the underground manual check of the large-scale railway line is performed, the real-time detection is not performed, and on the premise that the real-time detection is not performed, the method based on manual interpretation is not ideal for continuously using the processing of large-scale area identification and extraction targets, because the method is also not effective and real-time for checking and preventing.
The conventional identification classification relies on manual experience to conduct interpretation, so that the efficiency is low, timeliness is not achieved, the efficiency is only achieved when the target detection in a small-range area is solved, once the demand scale of the user is expanded to a provincial area, the efficiency is low, manpower is wasted, the efficiency is natural, the timeliness is not achieved, and the research area cannot be monitored from time to detect the events which threaten the high-speed bow net power supply facility and are about to occur.
Disclosure of Invention
The invention provides an intelligent recognition method for high-speed railway power supply facilities based on high-resolution remote sensing images, which can efficiently and accurately recognize the high-speed railway power supply facilities through a small number of samples and limited calculation support.
The invention relates to a high-speed railway power supply facility intelligent identification method based on high-resolution remote sensing images, which comprises the following steps of:
step S1: selecting a high-resolution remote sensing image, and judging and drawing samples in the surrounding area of a high-speed train station;
step S2: setting initial parameters, calculating a loss function by using samples with a boundary box of the true position of a power supply network rod, and participating in iteration and updating of a parameter weight W and an offset value b in the back propagation process, so that training of a model is completed;
step S3: calculating by adopting a road network searching mode;
step S4: and randomly selecting a plurality of station images, putting the station images into a trained model to identify the power supply network rods, and checking the identification result.
Preferably, the specific steps of the step S1 are as follows:
step S11: drawing the position of a power supply network rod according to the high-resolution satellite remote sensing image;
step S12: generating a minimum enveloping rectangle for the position of the power supply network trunk by adopting a GIS tool;
step S13: and cutting the original image according to the sketching area, and finally respectively storing the shape file of the face element, the KML file of the enveloping rectangle and the cut image TIFF file.
Preferably, the specific steps of the step S2 are as follows:
step S21: the original image is put into a convolution network FastRCNN to extract characteristic information, wherein the loss function of the FastRCNN is as follows:
L(p,u,t u ,v)＝L cls (p,u)+λ[u≥1]L loc (t u ,v) (1)
wherein p is the probability distribution predicted by the classifier, u corresponds to the target real class label, t u Regression parameters corresponding to the category u predicted by the boundary box regressor, v corresponding to the boundary box regression parameters of the real target;
step S22: the method comprises the steps of putting a feature image serving as a template into an RPN (remote procedure network) to screen out a boundary frame, further carrying out regression on the boundary frame to obtain a more accurate position, and mapping coordinate information of the boundary frame onto the feature image, wherein the RPN adopts the following loss function, takes individual discrimination as a proxy task to construct a self-supervision feature learning model, and pretrains parameters of the model:
wherein i denotes an index of an anchor in a mini-batch, p i An estimated probability indicating whether the anchor point i is an object; if the anchor point is a positive sample, the true mark1, negative sample 0; t is t i Vectors representing 4 parameterized coordinates of the prediction bounding box,/->Representing a real bounding box; classification loss L cls Is a logarithmic penalty covering both categories; for regression loss, use +.>Wherein R represents robust smoothl 1 loss; />Representing regression loss for sample anchor points only; the outputs of the cls and reg branches are respectively represented by { p } i Sum { t } i Composition;
step S23, putting the feature map with the boundary frame into the ROI Pooling layer to unify the dimensions of the boundary frame and outputting feature vectors with fixed sizes;
and step S24, calculating the category by using a Softmax classifier in the full connection layer, and carrying out regression of the bounding box again to obtain more accurate positioning.
Preferably, the specific steps of the step S3 are as follows:
step S31: performing initialization prediction by adopting a model obtained by training in S22 according to the initial range of the step S11;
step S32: when a station appears in the prediction result, a root node is established; and further, searching an area grid near the station, establishing sub-nodes when the railway exists nearby, further searching through the area grid to traverse all railway lines, returning to the sub-nodes when the power supply facility searching is completed, and continuously predicting along the railway advancing direction to sequentially complete the identification and extraction of the power supply facilities in the whole area.
Preferably, the specific steps of the step S4 are as follows:
step S41: randomly selecting a plurality of station images as test data, putting the test data into a model to identify the power supply network rods, and checking the confidence coefficient of the identification result;
step S42: considering the influence of different landforms on the power supply facility identification monitoring, the high-speed railway images under different landforms are input into the model, and the robustness of the model under different influence is checked.
In summary, due to the adoption of the technical scheme, the beneficial effects of the invention are as follows:
1. according to the invention, a satellite image is adopted to acquire a sample to train the fast R-CNN deep learning model, so that the automatic identification and extraction of a high-speed railway power supply facility can be realized, the identification efficiency is greatly improved, and the stability of target detection is ensured;
2. according to the method, a high-speed railway power supply facility identification scheme based on area search is constructed, and the mode of traversing the area of the model is changed into the mode of area search by establishing the root node-child node, so that the calculation power loss is effectively reduced, and the prediction speed is greatly accelerated;
3. the identification results under different landforms show that even if a certain landform influence exists, the effect of the model is very good and has little difference.
Drawings
FIG. 1 is a flow chart of a high-speed railway power supply facility intelligent identification method based on high-resolution remote sensing images in an embodiment;
FIG. 2 is an example of generating a minimum envelope rectangle containing power grid bar positions in an embodiment;
fig. 3 is a grid view of a station and a power grid pole sketched in an embodiment.
Detailed Description
For a further understanding of the present invention, the present invention will be described in detail with reference to the drawings and examples. It is to be understood that the examples are illustrative of the present invention and are not intended to be limiting.
Examples
As shown in fig. 1, the embodiment provides a high-speed railway power supply facility intelligent identification method based on high-resolution remote sensing images, which comprises the following steps:
step S1: and selecting a high-resolution remote sensing image, and judging and drawing samples in an area around a high-speed train station by about 1000 meters.
Step S2: setting initial parameters, calculating a loss function by using samples with a boundary box of the true position of a power supply network rod, and participating in iteration and updating of a parameter weight W and an offset value b in the back propagation process, thereby completing training of a model.
Step S3: considering that only a small part of the high-resolution remote sensing images contain high-speed railway power supply facilities, the rest parts can be regarded as the background, and therefore, a road network searching mode is adopted to replace grid traversal for calculation.
Step S4: and randomly selecting a plurality of station images, putting the station images into a trained model to identify the power supply network rods, and checking the identification result.
Further, the specific steps of the step S1 are as follows:
step S11: drawing a proper position of a power supply network rod according to the high-resolution satellite remote sensing image;
step S12: generating a minimum enveloping rectangle for the position of the power supply network trunk by adopting a GIS tool;
step S13: considering the problem of file size, the original image needs to be cut according to the sketching area, and finally the shapefile of the face element, the KML file of the enveloping rectangle and the cut image TIFF file are respectively stored.
Further, the specific steps of the step S2 are as follows:
step S21: the original image is put into a convolution network to extract characteristic information, wherein the loss function of FastRCNN is as follows:
L(p,u,t u ,v)＝L cls (p,u)+λ[u≥1]L loc (t u ,v) (1)
where p is the probability distribution of classifier predictions p= (p) 0 ,…,p k ) U corresponds to the real class label of the target, t u Regression parameters of corresponding category u predicted by corresponding bounding box regressorv corresponds to the bounding box regression parameters (v x ，v y ，v w ，v h )
Step S22: the method comprises the steps of putting a feature image serving as a template into an RPN (remote procedure network) to screen out a boundary frame, further carrying out regression on the boundary frame to obtain a more accurate position, and mapping coordinate information of the boundary frame onto the feature image, wherein the RPN adopts the following loss function, takes individual discrimination as a proxy task to construct a self-supervision feature learning model, and pretrains parameters of the model:
wherein i denotes an index of an anchor in a mini-batch, p i Indicating an estimated probability of whether the anchor point i is an object. If the anchor point is a positive sample, the true mark1, the negative sample is 0.t is t i Vectors representing 4 parameterized coordinates of the prediction bounding box,/->Representing a real bounding box. Classification loss L cls Is a logarithmic penalty covering both categories. NeedleFor regression loss we use +.>Wherein R represents a robust smoothl 1 loss. />Representing regression loss for sample anchor only, others are ignored. The outputs of the cls and reg branches are respectively represented by { p } i Sum { t } i Composition.
And S23, placing the feature map with the bounding box into the ROI Pooling layer to unify the dimensions of the bounding box and outputting feature vectors with fixed sizes.
And step S24, calculating the category by using a Softmax classifier in the full connection layer, and carrying out regression of the bounding box again to obtain more accurate positioning.
Further, the specific steps of the step S3 are as follows:
step S31: performing initialization prediction by adopting a model obtained by training in S22 according to the initial range of the step S11;
step S32: and when the station appears in the prediction result, establishing a root node. Further, regional grid searching is carried out near the station, sub-nodes are built when railways exist near the station, all railway lines are further traversed through regional grid searching, when the power supply facility searching is completed, the sub-nodes are returned, prediction is carried out continuously along the railway advancing direction, and the whole regional power supply facility identification and extraction are sequentially completed;
further, the specific steps of the step S4 are as follows:
step S41: and randomly selecting a plurality of station images as test data, putting the test data into a model to identify the power supply network rod, and checking the confidence coefficient of the identification result.
Step S42: considering the influence of different landforms on the power supply facility identification monitoring, the high-speed railway images under different landforms are input into the model, and the robustness of the model under different influence is checked.
Fig. 1 shows the overall architecture design of the present invention, which is divided into four parts S1, S2, S3, S4 according to steps, sequentially performed in sequence. Each section has a separate yield and is the input to the next section.
Fig. 2 shows an example of a minimum envelope rectangle containing the position of the supply grid pole, where the large rectangle is the envelope rectangle generated for the station and the small rectangle is the envelope rectangle generated for the supply grid pole.
Fig. 3 shows a grid view of a station and a power supply network pole of the present invention, wherein the left side is a waiting hall of the station, the long bar is a track, and the middle is a platform. The original image is cut according to the sketching area, and the surface elements are converted into grid files to be used as a cutting range.
In order to achieve high efficiency and stability of target detection and to be able to reach the need to free human resources, this embodiment uses related theory based on target detection in computer vision and builds a fast R-CNN deep machine learning model to solve the problems in the background art.
Compared with the traditional method based on manual interpretation and recognition, the method for automatically recognizing the power supply facilities based on the target detection and the deep learning model establishment based on the computer vision has the advantages that the efficiency is greatly improved, and meanwhile, the requirement on human resources is reduced, time and labor are saved, so that the method for acquiring samples by using 19-level resolution satellite images to train the fast R-CNN model and realize automatic recognition and extraction of other areas is necessary, and the method has great practical significance.
The invention and its embodiments have been described above by way of illustration and not limitation, and the invention is illustrated in the accompanying drawings and described in the drawings in which the actual structure is not limited thereto. Therefore, if one of ordinary skill in the art is informed by this disclosure, the structural mode and the embodiments similar to the technical scheme are not creatively designed without departing from the gist of the present invention.
Claims (4)
1. The intelligent recognition method for the high-speed railway power supply facilities based on the high-resolution remote sensing images is characterized by comprising the following steps of: the method comprises the following steps:
step S1: selecting a high-resolution remote sensing image, and judging and drawing samples in the surrounding area of a high-speed train station;
step S2: setting initial parameters, calculating a loss function by using samples with a boundary box of the true position of a power supply network rod, and participating in iteration and updating of a parameter weight W and an offset value b in the back propagation process, so that training of a model is completed;
the specific steps of the step S2 are as follows:
step S21: the original image is put into a convolutional network Fast RCNN to extract characteristic information, wherein the loss function of the Fast RCNN is as follows:
L(p,u,t u ,v)＝L cls (p,u)+λ[u≥1]L loc (t u ,v) (1)
wherein p is the probability distribution predicted by the classifier, u corresponds to the target real class label, t u Regression parameters corresponding to the category u predicted by the boundary box regressor, v corresponding to the boundary box regression parameters of the real target;
step S22: the method comprises the steps of putting a feature image serving as a template into an RPN (remote procedure network) to screen out a boundary frame, further carrying out regression on the boundary frame to obtain a more accurate position, and mapping coordinate information of the boundary frame onto the feature image, wherein the RPN adopts the following loss function, takes individual discrimination as a proxy task to construct a self-supervision feature learning model, and pretrains parameters of the model:
wherein i denotes an index of an anchor in a mini-batch, p i An estimated probability indicating whether the anchor point i is an object; if the anchor point is a positive sample, the true mark1, negative sample 0; t is t i Vectors representing 4 parameterized coordinates of the prediction bounding box,/->Representing a real bounding box; classification loss L cls Is a logarithmic penalty covering both categories; for regression loss, useWherein R represents robust smoothl 1 loss; />Representing regression loss for sample anchor points only; the outputs of the cls and reg branches are respectively represented by { p } i Sum { t } i Composition;
s23, placing the feature map with the boundary frame into a ROIPooling layer to unify the dimensions of the boundary frame and outputting feature vectors with fixed sizes;
step S24, calculating categories by using a Softmax classifier in the full connection layer, and carrying out regression of the bounding box again to obtain more accurate positioning;
step S3: calculating by adopting a road network searching mode;
step S4: and randomly selecting a plurality of station images, putting the station images into a trained model to identify the power supply network rods, and checking the identification result.
2. The intelligent recognition method for the high-speed railway power supply facilities based on the high-resolution remote sensing image according to claim 1, wherein the intelligent recognition method is characterized by comprising the following steps of: the specific steps of the step S1 are as follows:
step S11: drawing the position of a power supply network rod according to the high-resolution satellite remote sensing image;
step S12: generating a minimum enveloping rectangle for the position of the power supply network trunk by adopting a GIS tool;
step S13: and cutting the original image according to the sketching area, and finally respectively storing the shape file of the face element, the KML file of the enveloping rectangle and the cut image TIFF file.
3. The intelligent recognition method for the high-speed railway power supply facilities based on the high-resolution remote sensing image according to claim 2, which is characterized by comprising the following steps of: the specific steps of the step S3 are as follows:
step S31: performing initialization prediction by adopting a model obtained by training in S22 according to the initial range of the step S11;
step S32: when a station appears in the prediction result, a root node is established; and further, searching an area grid near the station, establishing sub-nodes when the railway exists nearby, further searching through the area grid to traverse all railway lines, returning to the sub-nodes when the power supply facility searching is completed, and continuously predicting along the railway advancing direction to sequentially complete the identification and extraction of the power supply facilities in the whole area.
4. The intelligent recognition method for the high-speed railway power supply facilities based on the high-resolution remote sensing image according to claim 3, wherein the intelligent recognition method is characterized by comprising the following steps of: the specific steps of the step S4 are as follows:
step S41: randomly selecting a plurality of station images as test data, putting the test data into a model to identify the power supply network rods, and checking the confidence coefficient of the identification result;
step S42: considering the influence of different landforms on the power supply facility identification monitoring, the high-speed railway images under different landforms are input into the model, and the robustness of the model under different influence is checked.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211339322.XA CN115908897B (en) | 2022-10-28 | 2022-10-28 | Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211339322.XA CN115908897B (en) | 2022-10-28 | 2022-10-28 | Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115908897A CN115908897A (en) | 2023-04-04 |
| CN115908897B true CN115908897B (en) | 2023-09-12 |
Family
ID=86485289
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211339322.XA Active CN115908897B (en) | 2022-10-28 | 2022-10-28 | Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115908897B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110414795A (en) * | 2019-07-02 | 2019-11-05 | 华侨大学 | Method is influenced based on the newly-increased high-speed rail hinge accessibility for improving two moved further search methods |
| CN111091105A (en) * | 2019-12-23 | 2020-05-01 | 郑州轻工业大学 | Remote sensing image target detection method based on new frame regression loss function |
| CN111241994A (en) * | 2020-01-09 | 2020-06-05 | 中国交通通信信息中心 | Method for extracting remote sensing image rural highway desertification road section for deep learning |
| CN112287896A (en) * | 2020-11-26 | 2021-01-29 | 山东捷讯通信技术有限公司 | Unmanned aerial vehicle aerial image target detection method and system based on deep learning |
| CN112395975A (en) * | 2020-11-17 | 2021-02-23 | 南京泓图人工智能技术研究院有限公司 | Remote sensing image target detection method based on rotating area generation network |
| CN114694038A (en) * | 2022-04-11 | 2022-07-01 | 中国矿业大学 | High-resolution remote sensing image classification method and system based on deep learning |
-
2022
- 2022-10-28 CN CN202211339322.XA patent/CN115908897B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110414795A (en) * | 2019-07-02 | 2019-11-05 | 华侨大学 | Method is influenced based on the newly-increased high-speed rail hinge accessibility for improving two moved further search methods |
| CN111091105A (en) * | 2019-12-23 | 2020-05-01 | 郑州轻工业大学 | Remote sensing image target detection method based on new frame regression loss function |
| CN111241994A (en) * | 2020-01-09 | 2020-06-05 | 中国交通通信信息中心 | Method for extracting remote sensing image rural highway desertification road section for deep learning |
| CN112395975A (en) * | 2020-11-17 | 2021-02-23 | 南京泓图人工智能技术研究院有限公司 | Remote sensing image target detection method based on rotating area generation network |
| CN112287896A (en) * | 2020-11-26 | 2021-01-29 | 山东捷讯通信技术有限公司 | Unmanned aerial vehicle aerial image target detection method and system based on deep learning |
| CN114694038A (en) * | 2022-04-11 | 2022-07-01 | 中国矿业大学 | High-resolution remote sensing image classification method and system based on deep learning |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115908897A (en) | 2023-04-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20220043182A1 (en) | Spatial autocorrelation machine learning-based downscaling method and system of satellite precipitation data | |
| CN108109437B (en) | Unmanned aerial vehicle autonomous route extraction and generation method based on map features | |
| CN109493320B (en) | Remote sensing image road extraction method and system based on deep learning, storage medium and electronic equipment | |
| CN109376605B (en) | Electric power inspection image bird-stab-prevention fault detection method | |
| CN112287807B (en) | Remote sensing image road extraction method based on multi-branch pyramid neural network | |
| CN109376606B (en) | Power inspection image tower foundation fault detection method | |
| Zou et al. | CNN-based statistics and location estimation of missing components in routine inspection of historic buildings | |
| CN111553236A (en) | Road foreground image-based pavement disease target detection and example segmentation method | |
| KR102346676B1 (en) | Method for creating damage figure using the deep learning-based damage image classification of facility | |
| CN115409069A (en) | Village and town building identification method, classification method, device, electronic equipment and medium | |
| CN110910440B (en) | Power transmission line length determination method and system based on power image data | |
| CN112990086A (en) | Remote sensing image building detection method and device and computer readable storage medium | |
| CN112329751A (en) | Deep learning-based multi-scale remote sensing image target identification system and method | |
| CN111652835A (en) | Method for detecting insulator loss of power transmission line based on deep learning and clustering | |
| CN112819066A (en) | Res-UNet single tree species classification technology | |
| CN114283285A (en) | Cross consistency self-training remote sensing image semantic segmentation network training method and device | |
| CN106500594A (en) | Fusion reflected intensity and the railroad track method for semi-automatically detecting of geometric properties | |
| Xu et al. | Building height calculation for an urban area based on street view images and deep learning | |
| CN111241905A (en) | Power transmission line nest detection method based on improved SSD algorithm | |
| CN113902792A (en) | Building height detection method and system based on improved RetinaNet network and electronic equipment | |
| CN114037836A (en) | Method for applying artificial intelligence recognition technology to three-dimensional power transmission and transformation engineering measurement and calculation | |
| CN113420670A (en) | Environment-friendly supervision method for changing power transmission and transformation line migration based on high-resolution remote sensing | |
| CN115908897B (en) | Intelligent identification method for high-speed railway power supply facilities based on high-resolution remote sensing images | |
| CN116503750A (en) | Large-range remote sensing image rural block type residential area extraction method and system integrating target detection and visual attention mechanisms | |
| CN115937492A (en) | Transformer equipment infrared image identification method based on feature identification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |