CN113392721B - Remote sensing satellite video target tracking method - Google Patents
Remote sensing satellite video target tracking method Download PDFInfo
- Publication number
- CN113392721B CN113392721B CN202110566623.5A CN202110566623A CN113392721B CN 113392721 B CN113392721 B CN 113392721B CN 202110566623 A CN202110566623 A CN 202110566623A CN 113392721 B CN113392721 B CN 113392721B
- Authority
- CN
- China
- Prior art keywords
- frame
- tracking
- target
- current frame
- template
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 29
- 238000000605 extraction Methods 0.000 claims abstract description 25
- 238000012549 training Methods 0.000 claims abstract description 11
- 238000004364 calculation method Methods 0.000 claims abstract description 7
- 230000004044 response Effects 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 5
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000012360 testing method Methods 0.000 description 5
- 230000003044 adaptive effect Effects 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 230000010354 integration Effects 0.000 description 2
- 230000002708 enhancing effect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
Landscapes
- Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Artificial Intelligence (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Image Analysis (AREA)
Abstract
The invention relates to a remote sensing satellite video target tracking method, which comprises the following steps: inputting the current frame into a tracking algorithm, cutting a search area tracked by the current frame, extracting the features of the search area of the current frame by adopting a feature extraction algorithm, and acquiring a search area feature map F of the current frame C (ii) a F is to be C Tracking model M with previous frame C‑1 Carrying out similarity calculation to determine the position of a tracking target; judging whether the tracking is finished or not; if yes, ending the tracking; otherwise, the feature extraction algorithm is adopted to extract the features of the target position tracked by the current frame to obtain a current frame target feature map A C (ii) a According to the formula a = α a 1 +βA C +(1‑α‑β)A h Obtaining a current frame updating template A, wherein A in the formula h Updating the template for the previous frame, wherein alpha and beta are weights; performing model training on the current frame updating template A by adopting the tracking algorithm to obtain a current frame tracking templateType M C (ii) a And acquiring the next frame of the remote sensing satellite video, taking the next frame as a current frame, and repeating the above method until the tracking is finished.
Description
Technical Field
The invention relates to a video target tracking method, in particular to a remote sensing satellite video target tracking method.
Background
When the target is tracked in the satellite video, the problem of updating the template is involved. Solutions to this problem can be divided into two categories: 1. the first frame is always used as a template without updating; 2. the result of the tracking is used as an updated template.
Matthews L et al mention in The literature "The Template Update protocol" in IEEE Transactions on Pattern Analysis and Machine Analysis, vol.26, no.6, 2004, 810-815: although the first frame is used as the template, the first frame is a correct sample, in the satellite video, the appearance of the target is affected by the change of the light of the shielding object and the surrounding environment, and further the shape and the size of the target are changed, so that the first frame is only used as the template and cannot adapt to the changed tracking environment.
The tracking result is used as an updated template, and can adapt to dynamically changing targets and environments to a certain extent. Kenan Dai et al in the literature "Visual Tracking via Adaptive mapping-regulated Correlation Filters" in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Registration (CVPR), 2019,4670-4679 use the results of the Tracking as an updated template. However, if the tracking result is biased or even wrong, the error will affect the tracking of the next frame, so that the tracking of the next frame is also wrong. According to the template updating method, tracking errors are accumulated, and if the tracking result is not very accurate, the subsequent tracking errors are larger and larger, so that once error deviation is introduced into the tracking result by using the tracking result as the template updating method, the irreversible tracking failure problem can be caused.
Disclosure of Invention
The invention aims to solve the problem that a first frame is used as a template and cannot adapt to a changing tracking environment, and the problem that error deviation introduced in the tracking process is larger and larger in the tracking of subsequent frames due to the fact that a tracking result is used as the template, and provides a remote sensing satellite video target tracking method.
The technical scheme of the invention is as follows:
the invention provides a remote sensing satellite video target tracking method, which comprises the following steps:
When the remote sensing satellite video is tracked, the first frame can completely determine the tracked target, so that the target area can be directly cut when the first frame is processed;
step 2, obtaining a first frame target characteristic diagram A 1 Performing model training as a template to obtain a tracking model M of a first frame 1 ；
Step 3, inputting the second frame of the remote sensing satellite video into the tracking algorithm, cutting out the target search area of the second frame by referring to the target position in the first frame, adopting the feature extraction algorithm to extract the features of the target search area of the second frame, and obtaining the feature map F of the search area of the second frame 2 ；
Step 4, searching the second frame for a region feature map F 2 And tracking model M 1 Performing similarity calculation to obtain a response graph of the similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of a tracking target;
step 5, adopting the characteristic extraction algorithm to extract the characteristics of the target position tracked by the second frame in the step 4 to obtain a second frame target characteristic image A 2 According to the formula A = α A 1 +βA C +(1-α-β)A h Obtaining a current frame updating template A, wherein A C Is a target feature map of the current frame, A h Updating the template for the previous frame, wherein alpha and beta are weights；
Step 6, carrying out model training by using the updated template A obtained in the step 5 to obtain a tracking model M 2 ；
Step 7, acquiring a next frame of the remote sensing satellite video, and taking the next frame as a current frame;
inputting the current frame into the tracking algorithm, and cutting out a target search area of the current frame by referring to the target position tracked by the previous frame; extracting the characteristics of the target search area of the current frame by adopting the characteristic extraction algorithm to obtain a search area characteristic diagram F of the current frame C ；
Step 8, searching the regional characteristic graph F in the step 7 C And the previous frame tracking model M C-1 Performing similarity calculation to obtain a response graph of the similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of a tracking target;
step 9, judging whether the tracking is finished or not; if the tracking is judged to be finished, the tracking is finished; if the tracking is not finished, entering step 10;
step 10, adopting the characteristic extraction algorithm to extract the characteristics of the target position tracked by the current frame in the step 8 to obtain a current frame target characteristic diagram A C (ii) a The current frame update template a is obtained according to the following formula,
A＝αA 1 +βA C +(1-α-β)A h
wherein, A C For the current frame object feature map, A h Updating the template for the previous frame; in the formula, alpha and beta are weights;
when the remote sensing satellite video tracks, a first frame can completely determine a tracked target, and a previous frame updating template has target information adaptive to dynamic change, so that the target information in a first frame target feature map, the target information in a current frame target feature map and the target information in a previous frame updating template are integrated when the template is updated, so that the updated template is ensured to contain accurate target information and recently updated target information, and a target tracking result is more accurate and the precision is higher;
step 11, carrying out model training on the current frame updating template A to obtain a current frame tracking model M C (ii) a And returning to the step 7.
Further, the feature extraction algorithm is HOG or Color-Name or ImageNet or ResNet or Sift or Cannon or Gabor or VGG or LSTM or GRU or fast RCNN.
Further, the tracking algorithm is a correlation filtering algorithm.
Further, the related filtering algorithm is ASRCF or KCF or SRDCF or DSST or C-COT or ECO or SAMF or ARCF.
Compared with the prior art, the invention has the beneficial effects that:
1. the tracking error is reduced. The invention integrates the first frame target characteristic image, the current frame target characteristic image and the previous frame updating template into a new template, so that the new template has complete and correct information of the target in the first frame, the problem that when the tracking result is wrong in the target tracking process, the error can be accumulated and amplified in the tracking of the subsequent frame is avoided, and meanwhile, the invention is also beneficial to tracking the target after the target is shielded.
2. And enhancing the model generalization. The shape of the video target can be changed in the tracking process, and the environment where the target is located can also be changed, so that the target information in the first frame is only contained, which easily causes model overfitting, causes poor model generalization capability, and finally causes tracking failure. The updated template provided by the invention not only contains the target information of the first frame, but also contains the target information contained in the updated template of the previous frame, so that the new template can adapt to the changed target and environment, thereby improving the tracking precision.
Drawings
FIG. 1 is a flow chart of a remote sensing satellite video target tracking method according to the invention;
FIG. 2 is a schematic diagram of template updating of the remote sensing satellite video target tracking method of the invention.
Detailed Description
The method for tracking the remote sensing satellite video target is specifically described below with reference to fig. 1 and 2.
The remote sensing satellite video target tracking method provided by the embodiment comprises the following steps:
s1, inputting a first frame of a remote sensing satellite video into a tracking algorithm KCF, performing data preprocessing on the first frame, namely cutting out a target area of the first frame, performing feature extraction on the target area in the first frame by adopting a feature extraction algorithm HOG, and acquiring a first frame target feature map A 1 。
Because the size of the target position in the first frame is known, the target feature map extracted from the first frame contains the least noise and contains accurate target information, and therefore, the first frame target feature map is added for template updating, the noise quantity can be effectively controlled, and model drift is avoided.
S2, obtaining a target characteristic diagram A of the first frame 1 Performing model training as a template to obtain a tracking model M of a first frame 1 。
S3, inputting a second frame of the remote sensing satellite video into a tracking algorithm KCF, cutting out a target search area of the second frame by referring to the target position in the first frame, extracting the features of the target search area of the second frame by adopting a feature extraction algorithm HOG, and acquiring a search area feature map F of the second frame 2 。
S4, searching a region feature map F for a second frame 2 And tracking model M 1 And performing similarity calculation to obtain a response graph of the similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of the tracking target.
S5, performing feature extraction on the target position tracked by the second frame in the step 4 by adopting a feature extraction algorithm HOG to obtain a second frame target feature map A 2 According to the formula A = α A 1 +βA C +(1-α-β)A h Obtaining a current frame updating template A, wherein A C Is a target feature map of the current frame, A h The template is updated for the previous frame, where α and β are weights.
S6, carrying out model training by using the updated template A obtained in the step 5 to obtain a tracking model M 2 。
And S7, acquiring the next frame of the remote sensing satellite video, and taking the next frame as the current frame.
Inputting the current frame into a tracking algorithm KCF, and cutting out a target search area of the current frame by referring to a target position tracked by the previous frame; adopting a feature extraction algorithm HOG to extract features of a target search area of the current frame to obtain a search area feature map F of the current frame C 。
S8, searching the regional characteristic graph F in the step 7 C And the previous frame tracking model M C-1 And performing similarity calculation to obtain a response graph of the similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of the tracking target.
S9, judging whether the tracking is finished or not; if the tracking is judged to be finished, the tracking is finished; if the tracking is not finished, step 10 is entered.
S10, extracting the features of the target position tracked by the current frame in the step 8 by adopting a feature extraction algorithm HOG to obtain a current frame target feature map A C (ii) a The current frame update template a is obtained according to the following formula,
A＝αA 1 +βA C +(1-α-β)A h
wherein, AC is a current frame target feature map, ah is a previous frame updating template; in the formula, alpha and beta are weights.
S11, performing model training on the current frame updating template A to obtain a current frame tracking model MC; and returning to the step 7.
After the target tracking of the current frame is finished, namely the target position is determined, the current frame target characteristic image is extracted and integrated with the first frame target characteristic image and the updating template of the previous frame to obtain the current frame updating template, and the target tracking of the searching area characteristic image of the next frame is carried out by adopting the current frame tracking model obtained by training the current frame updating template. The template updating method provided by the invention comprises the target information which is updated recently and the completely accurate target information, and is more adaptive to the target tracking under the conditions of appearance change, environment change, illumination change and the like, and the tracking precision is higher.
In other embodiments, the feature extraction algorithm may be Color-Name, imageNet, resNet, sift, cannon, gabor, VGG, LSTM, GRU, faster RCNN.
In other embodiments, the correlation filtering algorithm may be ASRCF, DSST, SRDCF, C-COT, ECO, SAMF, ARCF.
The remote sensing satellite video target tracking method provided by the invention can realize target tracking on the remote sensing satellite video in the storage device and can also realize target tracking on the remote sensing satellite real-time video.
The inventor adopts the remote sensing satellite video target tracking method provided by the invention and further verifies the effect of the invention through experiments.
The test conditions are as follows: the test is carried out in a central processing unit
And MATLAB software is applied to an i 5-3470.2GHz CPU and a memory 4G, ubuntu operating system.
The test contents are as follows: the ASRCF algorithm is selected in the test and divided into two groups, wherein the group 1 utilizes the template updating method of the original algorithm, and the group 2 utilizes the template updating method provided by the invention. And respectively carrying out tracking experiments on the two groups of video sequences on the same video sequence, and comparing tracking effects.
The total number of Video sequences in the test is 7, and 7 videos belong to the same data set and are all from a remote sensing Satellite Video target Tracking data set (OTSVD). The tracking results are shown in table 1.
TABLE 1 comparison of remote sensing satellite video target tracking results
| Algorithm | Number of integration under line | |
| Group | ||
| 1 | 74.1 | 85.9 |
| Group 2 | 74.6 | 91.8 |
As can be seen from Table 1, compared with the template updating method in the original algorithm, the integration number under the line of the tracking result of the template updating method provided by the invention is improved by 0.5, the precision fraction is improved by 5.9, and the precision and the area under the line are improved, thereby illustrating the effectiveness of the template updating method provided by the invention.
Claims (4)
1. A remote sensing satellite video target tracking method is characterized by comprising the following steps:
step 1, inputting a first frame of a remote sensing satellite video into a tracking algorithm, performing data preprocessing on the first frame, namely cutting out a target area of the first frame, performing feature extraction on the target area in the first frame by adopting a feature extraction algorithm, and acquiring a target feature map A of the first frame 1 ；
Step 2, obtaining a first frame target characteristic diagram A 1 Performing model training as a template to obtain a tracking model M of a first frame 1 ；
Step 3, inputting the second frame of the remote sensing satellite video into the tracking algorithm, cutting out the target search area of the second frame by referring to the target position in the first frame, adopting the feature extraction algorithm to extract the features of the target search area of the second frame, and acquiring the feature map F of the search area of the second frame 2 ；
Step 4, searching the second frame for a region feature map F 2 And tracking model M 1 Performing similarity calculation to obtain a response graph of the similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of a tracking target;
step 5, adopting the characteristic extraction algorithm to carry out characteristic extraction on the target position tracked by the second frame in the step 4 to obtain a second frame target characteristic diagram A 2 According to the formula A = α A 1 +βA C +(1-α-β)A h Obtaining a current frame updating template A, wherein A in the formula C Is a target feature map of the current frame, A h Updating the template for the previous frame, wherein alpha and beta are weights;
step 6, carrying out model training by using the updated template A obtained in the step 5 to obtain a tracking model M 2 ；
Step 7, acquiring a next frame of the remote sensing satellite video, and taking the next frame as a current frame;
inputting the current frame into the tracking algorithm, and cutting out a target search area of the current frame by referring to the target position tracked by the previous frame; extracting the characteristics of the target search area of the current frame by adopting the characteristic extraction algorithm to obtain a search area characteristic diagram F of the current frame C ；
Step 8, searching the regional characteristic graph F in the step 7 C And the previous frame tracking model M C-1 Performing similarity calculation to obtain a response graph of similarity, and selecting a position with the maximum similarity in the response graph, wherein the position with the maximum similarity is the position of a tracking target;
step 9, judging whether the tracking is finished or not; if the tracking is judged to be finished, the tracking is finished; if the tracking is not finished, entering step 10;
step 10, adopting the characteristic extraction algorithm to extract the characteristics of the target position tracked by the current frame in the step 8 to obtain a current frame target characteristic diagram A C (ii) a The current frame update template a is obtained according to the following formula,
A＝αA 1 +βA C +(1-α-β)A h
wherein, A in the formula C For the current frame object feature map, A h Updating the template for the previous frame; alpha and beta are weights;
step 11, carrying out model training on the current frame updating template A to obtain a current frame tracking model M C (ii) a And returning to the step 7.
2. The remote sensing satellite video target tracking method according to claim 1, characterized in that: the feature extraction algorithm is HOG, color-Name, imageNet, resNet, sift, cannon, gabor, VGG, LSTM, GRU, or fast RCNN.
3. The remote sensing satellite video target tracking method according to claim 1, characterized in that: the tracking algorithm is a correlation filtering algorithm.
4. The remote sensing satellite video target tracking method according to claim 3, characterized in that: the related filtering algorithm is ASRCF or KCF or SRDCF or DSST or C-COT or ECO or SAMF or ARCF.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110566623.5A CN113392721B (en) | 2021-05-24 | 2021-05-24 | Remote sensing satellite video target tracking method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110566623.5A CN113392721B (en) | 2021-05-24 | 2021-05-24 | Remote sensing satellite video target tracking method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113392721A CN113392721A (en) | 2021-09-14 |
| CN113392721B true CN113392721B (en) | 2023-02-10 |
Family
ID=77618881
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110566623.5A Active CN113392721B (en) | 2021-05-24 | 2021-05-24 | Remote sensing satellite video target tracking method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113392721B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109815790A (en) * | 2018-12-12 | 2019-05-28 | 中国科学院西安光学精密机械研究所 | The poly- detection network of gate axis and method for remote sensing target detection |
| CN110706252A (en) * | 2019-09-09 | 2020-01-17 | 西安理工大学 | Robot nuclear correlation filtering tracking algorithm under guidance of motion model |
| CN111639551A (en) * | 2020-05-12 | 2020-09-08 | 华中科技大学 | Online multi-target tracking method and system based on twin network and long-short term clues |
| CN111931685A (en) * | 2020-08-26 | 2020-11-13 | 北京建筑大学 | Video satellite moving target detection method based on bidirectional tracking strategy |
| CN111931686A (en) * | 2020-08-26 | 2020-11-13 | 北京建筑大学 | Video satellite target tracking method based on background knowledge enhancement |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110033478A (en) * | 2019-04-12 | 2019-07-19 | 北京影谱科技股份有限公司 | Visual target tracking method and device based on depth dual training |
-
2021
- 2021-05-24 CN CN202110566623.5A patent/CN113392721B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109815790A (en) * | 2018-12-12 | 2019-05-28 | 中国科学院西安光学精密机械研究所 | The poly- detection network of gate axis and method for remote sensing target detection |
| CN110706252A (en) * | 2019-09-09 | 2020-01-17 | 西安理工大学 | Robot nuclear correlation filtering tracking algorithm under guidance of motion model |
| CN111639551A (en) * | 2020-05-12 | 2020-09-08 | 华中科技大学 | Online multi-target tracking method and system based on twin network and long-short term clues |
| CN111931685A (en) * | 2020-08-26 | 2020-11-13 | 北京建筑大学 | Video satellite moving target detection method based on bidirectional tracking strategy |
| CN111931686A (en) * | 2020-08-26 | 2020-11-13 | 北京建筑大学 | Video satellite target tracking method based on background knowledge enhancement |
Non-Patent Citations (4)
| Title |
|---|
| Object Tracking on Satellite Videos: A Correlation Filter-Based Tracking Method With Trajectory Correction by Kalman Filter;Yujia Guo et.al;《IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing》;20190821;第12卷(第9期);第3538-3551页 * |
| 基于KCF和SIFT特征的抗遮挡目标跟踪算法;包晓安等;《计算机测量与控制》;20180525(第05期);第154-158页 * |
| 基于视频流的实时人脸检测与跟踪;高俊艳;《中国优秀硕士学位论文全文数据库 信息科技辑》;20200215;第I138-2032页 * |
| 高光谱影像质量改善及降维研究;郑向涛;《中国博士学位论文全文数据库 工程科技Ⅱ辑》;20190615;第C028-3页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113392721A (en) | 2021-09-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2022089236A1 (en) | Image processing method apparatus based on artificial intelligence, and computer device and storage medium | |
| CN107424171B (en) | Block-based anti-occlusion target tracking method | |
| CN107688808B (en) | Rapid natural scene text detection method | |
| CN110008867A (en) | A kind of method for early warning based on personage's abnormal behaviour, device and storage medium | |
| CN111476826A (en) | Multi-target vehicle tracking method based on SSD target detection | |
| Meng et al. | Object instance search in videos via spatio-temporal trajectory discovery | |
| WO2022142417A1 (en) | Target tracking method and apparatus, electronic device, and storage medium | |
| Tang et al. | Robust tracking with discriminative ranking lists | |
| CN111508006A (en) | Moving target synchronous detection, identification and tracking method based on deep learning | |
| CN115457277A (en) | Intelligent pavement disease identification and detection method and system | |
| CN112926592A (en) | Trademark retrieval method and device based on improved Fast algorithm | |
| CN114998628A (en) | Template matching-based twin network long-term target tracking method | |
| Papapetros et al. | Visual loop-closure detection via prominent feature tracking | |
| Wang et al. | Action recognition based on object tracking and dense trajectories | |
| CN113392721B (en) | Remote sensing satellite video target tracking method | |
| Gao et al. | Occluded person re-identification based on feature fusion and sparse reconstruction | |
| CN108845999A (en) | A kind of trademark image retrieval method compared based on multiple dimensioned provincial characteristics | |
| CN115018886B (en) | Motion trajectory identification method, device, equipment and medium | |
| Li et al. | Collaborative convolution operators for real-time coarse-to-fine tracking | |
| Wu et al. | A clustering-optimized segmentation algorithm and application on food quality detection | |
| CN112989911B (en) | Pedestrian re-identification method and system | |
| CN114821482A (en) | Vector topology integrated passenger flow calculation method and system based on fisheye probe | |
| CN112232289A (en) | Ship retrieval method and device, electronic equipment and storage medium | |
| Chu et al. | Regressive scale estimation for visual tracking | |
| CN109448020B (en) | Target tracking method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |