CN106023099B - Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC - Google Patents
Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC Download PDFInfo
- Publication number
- CN106023099B CN106023099B CN201610313400.7A CN201610313400A CN106023099B CN 106023099 B CN106023099 B CN 106023099B CN 201610313400 A CN201610313400 A CN 201610313400A CN 106023099 B CN106023099 B CN 106023099B
- Authority
- CN
- China
- Prior art keywords
- correction
- radiant
- radiant correction
- image
- slice
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012937 correction Methods 0.000 title claims abstract description 111
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000004364 calculation method Methods 0.000 claims abstract description 9
- 238000012952 Resampling Methods 0.000 claims abstract description 7
- 239000000284 extract Substances 0.000 claims abstract description 4
- 238000007667 floating Methods 0.000 claims description 10
- 230000005855 radiation Effects 0.000 claims description 5
- 238000003384 imaging method Methods 0.000 claims description 4
- 238000012546 transfer Methods 0.000 claims description 4
- 235000013399 edible fruits Nutrition 0.000 claims description 3
- 238000000605 extraction Methods 0.000 abstract description 3
- 238000013461 design Methods 0.000 description 7
- 229910052754 neon Inorganic materials 0.000 description 4
- GKAOGPIIYCISHV-UHFFFAOYSA-N neon atom Chemical compound [Ne] GKAOGPIIYCISHV-UHFFFAOYSA-N 0.000 description 4
- 235000013350 formula milk Nutrition 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- RTZKZFJDLAIYFH-UHFFFAOYSA-N Diethyl ether Chemical compound CCOCC RTZKZFJDLAIYFH-UHFFFAOYSA-N 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000001934 delay Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000009977 dual effect Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Classifications
-
- G06T5/80—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2200/00—Indexing scheme for image data processing or generation, in general
- G06T2200/28—Indexing scheme for image data processing or generation, in general involving image processing hardware
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10041—Panchromatic image
Abstract
The present invention provides a kind of remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC, specific step is as follows: Step 1: original remote sensing CCD image data is after slice extracts, slice image data is transferred to the caching FIFO in logical gate by FMC interface；Auxiliary data is transmitted to ARM by serial ports and then caches to DDR；Radiant correction coefficient is stored in SD card in the form of a file；It exports Step 2: being cached in FIFO data, is completed using radiant correction coefficient to its radiant correction, the result after radiant correction is stored in BRAM, is finally stored in chip external memory DDR；Step 3: double-core ARM carries out positioning calculation to four vertex in the extraction region of the sectioning image after radiant correction, then complete by projection, resampling without control geometric correction；Step 4: returning to correction result.This method can realize CCD initial data radiant correction and geometric correction on SoPC, have fine real-time and stability.
Description
Technical field
The invention belongs to remote sensing image preconditioning technique fields, and in particular to a kind of remote sensing full-colour picture based on SoPC
As slice radiant correction and geometric correction implementation method.
Background technique
The pretreatment main purpose of remotely-sensed data is geometry and the radiation deformation corrected in original image, i.e., by image
The correction of the deformation, distortion, the radiated noise that are generated in acquisition process etc., thus obtain one it is as upper anti-in geometry and radiation as possible
Reflect the image of scenery real information.
Zhang Chunhong etc. is mentioned in article " the spaceborne in-orbit intelligence pretreatment of mass remote sensing data and transmission control technology research "
Radiant correction and geometric correction are realized using the framework of DSP+FPGA out, but its design complexities is compared with high, development difficulty is big.
SoPC system has the advantage that integrated level is high, flexibility is big, low in energy consumption, the development cycle is short, including software design and hardware design
Two parts.The Zynq-7000 series of programmable logical device of new generation that Xilinx company in 2013 releases by arm processor and
28nm programmable logic resource is integrated in same chip, is mentioned in the form of scalable processors platform for system developer
New mentality of designing and design concept are supplied.The pre- place that the mode of the Hardware/Software Collaborative Design of system level is sliced remote sensing images
Reason hardware realization has the characteristics that can to cut, is extendible, configurable, easy upgrading.
Due to data volume is big, computation complexity is high etc., the pretreatment of remotely-sensed data at present is mainly on CPU/GPU
It completes, requires in real-time and the high application demand of portability in emergency, disaster relief etc. there are limitation, and Zynq-7000 platform
Be suitble to data volume it is big, can flowing water processing task, while also having preferable performance in calculated performance.
Summary of the invention
The present invention is in order to realize remote sensing full-colour image slice radiant correction and geometric correction system compact, low-power consumption, just
Formula design is taken, a kind of remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC, the party are proposed
Method can realize CCD initial data radiant correction and geometric correction on FPGA+ double-core ARM (SoPC), have fine real-time
And stability.
Realize that technical scheme is as follows:
A kind of remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC, specific steps are such as
Under:
Step 1: Remote sensing data reception and radiant correction coefficient are read
For original remote sensing CCD image data after slice extracts, slice image data is transferred to logic section by FMC interface
Divide the caching FIFO in (PL)；Auxiliary data is transmitted to ARM by serial ports and then caches to DDR；Radiant correction coefficient is with file
Form is stored in SD card, and ARM reads the radiant correction coefficient being stored in SD card by DMA transfer to correction after system starting
In coefficient cache module DP_BRAM_A and DP_BRAM_B；
Step 2: radiant correction
It is cached in FIFO data to export, be completed using radiant correction coefficient to its radiant correction, the knot after radiant correction
Fruit is stored in BRAM, is finally stored in chip external memory DDR；
Step 3: geometric correction
Double-core ARM carries out positioning calculation to four vertex in the extraction region of the sectioning image after radiant correction, then passes through
Cross projection, resampling is completed without control geometric correction；
Step 4: returning to correction result
Via radiation correct and geometric correction after slice image data be stored in DDR in the form of a file, then through gigabit with
Too network interface is uploaded to PC machine.
Further, the detailed process of step 2 of the present invention are as follows:
, 201, caching FIFO each period exports a slice image data, and output data turns floating-point by a fixed point
IP kernel is converted to floating number；
202, the floating number is corresponding to the radiant correction coefficient progress in DP_BRAM_A and DP_BRAM_B to be calculated, thus
Obtain the output result of radiant correction；
203, the output result of radiant correction turns fixed point IP kernel by a floating-point and is converted to 8 fixed-point numbers and by DP_
BRAM_D caching.
204, step 201-203 is repeated, until each of sectioning image pixel is disposed, then by ARM
DMA is carried in DDR and is stored.
205, step 201-204 is repeated, until the correction of each slice image data finishes.
Further, the detailed process of step 3 of the present invention are as follows:
301, it is carried out by 9 angle points of the ARM double-core to the sectioning image after radiant correction using tight geometry imaging model
Positioning calculation；
302, gauss projection is carried out by 9 angle points of the ARM double-core to the sectioning image after radiant correction；
303, by 9 corner locations and its gauss projection, using least square method by ARM CPU0 evaluator system
Number a and b, image is shown below with original image corresponding relationship after geometric correction:
X=a0+a1X+a2Y+a3X2+a4XY+a5Y2
Y=b0+b1X+b2Y+b3X2+b4XY+b5Y2
Wherein, a=a0~a5And b=b0~b5Representative polynomial coefficient, image picture point is sat after (x, y) indicates geometric correction
Mark, (X, Y) indicate original image picpointed coordinate；
304, Q0And Q1Register is loaded into [1 X Y], Q4And Q5It is loaded into coefficient a and b, Q respectively1In X and Y and Q0Element
Multiplied result is stored in Q respectively2And Q3, by Q2One or two element is assigned to Q1, third element is assigned a value of 1；Q1In element and Q4
In the result of element multiplication be stored in Q0In, Q3In element and Q5In the result of element multiplication be stored in Q2In, by Q0With
Q2Middle corresponding element is added；
305, brightness resampling is carried out using the result that bilinear interpolation is obtained to 304, after final output geometric correction
Result.
Beneficial effect
First, the present invention is based on SoPC system completion remote sensing images slice radiant correction and without control geometric correction, with lower
Design difficulty complete single-chip remote sensing images slice pretreatment, reduce preprocessing of remote sensing images system bulk, have
Preferable real-time and portability.
Second, the present invention is based on the designs of ARM dual core processor, since remote sensing images slice size is small, then by positioning
Method after nine angle points using fitting of a polynomial is directly to image rectification.It can parallel feature reasonable distribution calculating according to the algorithm
Task is executed the positioning calculation of nine angle points of sectioning image by double-core ARM parallel, is obtained compared to monokaryon processing mode real-time
It effectively improves.
Third, the present invention made full use of when indirect scheme of digital rectification NEON calculating speed it is fast, can parallelization the characteristics of, use
NEON coprocessor is corrected transformation to general polynomial and is accelerated, and real-time effectively improves.
Detailed description of the invention
Fig. 1 is that SoPC remote sensing images are sliced radiant correction and geometric correction system architecture diagram；
Fig. 2 is that double-core ARM realizes that remote sensing images are sliced radiant correction and geometric correction flow chart；
Fig. 3 is NEON coprocessor second order general polynomial calculation process.
Specific embodiment
It elaborates with reference to the accompanying drawing to the embodiment of the method for the present invention.
A kind of remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC, was embodied
Journey is as follows:
Step 1: Remote sensing data reception and radiant correction coefficient are read
For original remote sensing CCD image data after slice extracts, slice image data is transferred to logic section by FMC interface
Divide the caching FIFO in (PL)；Auxiliary data is transmitted to ARM by serial ports and then caches to DDR；Radiant correction coefficient is with file
Form is stored in SD card, and ARM reads the radiant correction coefficient being stored in SD card by DMA transfer to school after system starting
In positive coefficient cache module DP_BRAM_A and DP_BRAM_B；
Detailed process are as follows:
(1) radiant correction uses Y-factor method Y, updating formula are as follows:
Y=Ax+B
Wherein, y indicates correction as a result, x indicates that first response is visited in input, and A, B are radiant correction coefficient.
(2) after system starting, ARM reads the radiant correction coefficient being stored in SD card and is stored in DDR, radiant correction
Coefficient is that A and B is 12288*32 floating numbers.
(3) CCD visits the position of member where ARM receives the sectioning image that size is 8192*8192*8 by serial ports
Information, then ARM delays the corresponding position the 8192*8192*32 radiant correction coefficient A and B for visiting member by DMA transfer to correction coefficient
Storing module DP_BRAM_A and DP_BRAM_B, final system enter reception data mode.
(4) system passes through FMC interface slice of data and is stored in cache module FIFO_8K, and the depth of FIFO_8K is
4096, setting input is that the output of 32 bit wides is set as 8.
Step 2: radiant correction
It is cached in FIFO data to export, be completed using radiant correction coefficient to its radiant correction, the knot after radiant correction
Fruit is stored in BRAM, is finally stored in chip external memory DDR；Detailed process are as follows:
(1) module FIFO_8K each period exports a data, and output data turns Floating Point IP conversion by a fixed point
For floating number.
(2) the correction system in result and DP_BRAM_B after the floating number is multiplied with the correction coefficient A in DP_BRAM_A
Number B is added, to obtain the output result of radiant correction.
(3) radiant correction output result is converted to 8 fixed-point numbers and by DP_BRAM_D by a floating-point turn fixed point IP kernel
Caching.
(4) 1 in the step is repeated) 2) 3), until 8192 pixels of a line are disposed, then by ARM DMA general
It is carried in DDR and stores.
(5) 1 in the step is repeated) 2) 3) 4), until 8192 row Data corrections finish.
Step 3: geometric correction
Double-core ARM carries out positioning calculation to four vertex in the extraction region of sectioning image, then by projection, resampling
It completes without control geometric correction；
Wherein, step 1 is by auxiliary data corresponding to the slice of the 8192*8192 size by serial ports (Configuration of baud rate
115200) to be transmitted to ARM, then cache to DDR.
(1) positioning calculation, ARM are carried out using tight geometry imaging model by 9 angle points of the ARM double-core to sectioning image
CPU0 resolves point 0,3,6,1, ARM CPU1 and resolves point 2,5,8,4,7.
(2) gauss projection is carried out by 9 angle points of the ARM double-core to sectioning image.ARM CPU0 calculates point 0,3,6,1, ARM
CPU1 calculates point 2,5,8,4,7.
(3) Geometric Correction of Remote Sensing Image based on second order general polynomial is used, image is corresponding with original image after geometric correction
Relationship is stated by the second order general polynomial.It is counted using least square method by ARM CPU0 by 9 angle points and its gauss projection
Calculate multinomial coefficient a and b.General polynomial corrects transformation for mula are as follows:
X=a0+a1X+a2Y+a3X2+a4XY+a5Y2
Y=b0+b1X+b2Y+b3X2+b4XY+b5Y2
(4) indirect scheme of digital rectification scheme is used, referring to Fig. 2, the left side is corrected indirectly by ARM CPU0, by ARM CPU1
The right is corrected indirectly；
(5) second order general polynomial transformation calculations (x, y) are completed by NEON, referring to Fig. 3.QDF respectively indicates 128 in figure
The position (4*32), 64,32 floating point vector registers, Q0Q1Register is loaded into [1 X Y], Q4Q5It is right respectively to be loaded into coefficient a or b
Should corresponding position in figure, Q1In X and Y and Q0Element difference multiplied result be stored in Q2Q3, by Q2One or two element is assigned to Q1,
Third element is assigned a value of 1, Q1Q3Respectively with Q4Q5Multiplication obtains multinomial all elements, all elements is added to obtain final
As a result.
(6) brightness resampling is carried out using bilinear interpolation.Repeat the 5-7 in step 3 obtain it is defeated after geometric correction
Result out.
Step 4: returning to correction result
It corrects via radiation and is stored in DDR in the form of dat file with the slice image data after geometric correction, through gigabit ether
Network interface is uploaded to PC machine.
Claims (3)
1. a kind of remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC, which is characterized in that tool
Steps are as follows for body:
Step 1: Remote sensing data reception and radiant correction coefficient are read
After slice extracts, slice image data is transferred in logical gate original remote sensing CCD image data by FMC interface
Caching FIFO；Auxiliary data is transmitted to ARM by serial ports and then caches to chip external memory DDR；Radiant correction coefficient is with file
Form be stored in SD card, ARM reads the radiant correction coefficient being stored in SD card by DMA transfer to school after system starting
In positive coefficient cache module DP_BRAM_A and DP_BRAM_B；
Step 2: radiant correction
It is cached in FIFO data to export, be completed using radiant correction coefficient to its radiant correction, the result after radiant correction is deposited
Enter BRAM, is finally stored in chip external memory DDR；
Step 3: geometric correction
Double-core ARM carries out positioning calculation using tight geometry imaging model to 9 angle points of the sectioning image after radiant correction, so
It completes afterwards by projection, resampling without control geometric correction；
Step 4: returning to correction result
It corrects via radiation and is stored in chip external memory DDR in the form of a file with the slice image data after geometric correction, then pass through
Gigabit ethernet interface is uploaded to PC machine.
2. remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC according to claim 1,
It is characterized in that, the detailed process of the step 2 are as follows:
201, caching FIFO each period export a slice image data, and output data turns Floating Point IP by a fixed point and turns
It is changed to floating number；
202, the floating number is corrected calculating with the radiant correction coefficient in DP_BRAM_A and DP_BRAM_B, to obtain
The output result of radiant correction；
203, the output result of radiant correction turns fixed point IP kernel by a floating-point and is converted to 8 fixed-point numbers and by DP_BRAM_D
Caching；
204, step 201-203 is repeated, until each of sectioning image pixel is disposed, then by ARM DMA general
It is carried in DDR and stores；
205, step 201-204 is repeated, until the correction of each slice image data finishes.
3. remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC according to claim 1,
It is characterized in that, the detailed process of the step 3 are as follows:
301, it is positioned by 9 angle points of the ARM double-core to the sectioning image after radiant correction using tight geometry imaging model
It resolves；
302, gauss projection is carried out by 9 angle points of the ARM double-core to the sectioning image after radiant correction；
303, by 9 corner locations and its gauss projection, using least square method by ARM CPU0 evaluator coefficient a and
B, image is shown below with original image corresponding relationship after geometric correction:
X=a0+a1X+a2Y+a3X2+a4XY+a5Y2
Y=b0+b1X+b2Y+b3X2+b4XY+b5Y2
Wherein, a=a0~a5And b=b0~b5Representative polynomial coefficient, (x, y) indicate image picpointed coordinate after geometric correction, (X,
Y original image picpointed coordinate) is indicated；
304, Q0And Q1Register is loaded into [1 X Y], Q4And Q5Register is loaded into multinomial coefficient a and b, Q respectively1In X and Y with
Q0Element difference multiplied result be stored in register Q2And Q3, by Q2First and two elements are assigned to Q1, third element assignment
It is 1；Q1In element and Q4In the result of element multiplication be stored in Q0In, Q3In element and Q5In element multiplication result
It is stored in Q2In, by Q0With Q2Middle corresponding element is added；
305, brightness resampling is carried out using the result that bilinear interpolation is obtained to 304, the knot after final output geometric correction
Fruit.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610313400.7A CN106023099B (en) | 2016-05-12 | 2016-05-12 | Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610313400.7A CN106023099B (en) | 2016-05-12 | 2016-05-12 | Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106023099A CN106023099A (en) | 2016-10-12 |
| CN106023099B true CN106023099B (en) | 2019-03-12 |
Family
ID=57100646
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610313400.7A Active CN106023099B (en) | 2016-05-12 | 2016-05-12 | Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106023099B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107197217B (en) * | 2017-04-19 | 2020-08-14 | 西安空间无线电技术研究所 | Satellite high-frame-frequency image data real-time slicing transmission method |
| CN110866452B (en) * | 2019-10-22 | 2024-04-02 | 陕西航天技术应用研究院有限公司 | Method for acquiring real-time target detection slice of original data of optical imaging satellite |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104361563A (en) * | 2014-11-06 | 2015-02-18 | 西北工业大学 | GPS-based (global positioning system based) geometric precision correction method of hyperspectral remote sensing images |
| CN104851106A (en) * | 2015-06-03 | 2015-08-19 | 西北大学 | CCD original image processing method |
-
2016
- 2016-05-12 CN CN201610313400.7A patent/CN106023099B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104361563A (en) * | 2014-11-06 | 2015-02-18 | 西北工业大学 | GPS-based (global positioning system based) geometric precision correction method of hyperspectral remote sensing images |
| CN104851106A (en) * | 2015-06-03 | 2015-08-19 | 西北大学 | CCD original image processing method |
Non-Patent Citations (3)
| Title |
|---|
| THE SOPC BASED DESIGN FOR REAL-TIME RADAR SEEKER SIGNAL PROCESSING;Pang Long 等;《Radar Conference,2009 IET International》;20090331;正文第1-5节 |
| 基于SOPC的图像采集和处理系统的设计与研究;张建荣;《中国优秀硕士学位论文全文数据库 信息科技辑》;20111215;第1-4页 |
| 基于SOPC的实时图像处理系统设计;许峥 等;《光电技术应用》;20110831;第26卷(第4期);第4-7页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106023099A (en) | 2016-10-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105809712B (en) | A kind of efficient big displacement light stream method of estimation | |
| CN109117947A (en) | Profile testing method and Related product | |
| CN111937009A (en) | Systolic convolutional neural network | |
| CN101916429B (en) | Geometric correction and disparity extraction device of binocular camera | |
| CN111414994B (en) | FPGA-based Yolov3 network computing acceleration system and acceleration method thereof | |
| CN116050495A (en) | System and method for training neural networks with sparse data | |
| CN101841730A (en) | Real-time stereoscopic vision implementation method based on FPGA | |
| CN109859178B (en) | FPGA-based infrared remote sensing image real-time target detection method | |
| Tomasi et al. | High-performance optical-flow architecture based on a multi-scale, multi-orientation phase-based model | |
| CN106023099B (en) | Remote sensing full-colour image slice radiant correction and geometric correction implementation method based on SoPC | |
| Seong et al. | A novel hardware architecture of the Lucas–Kanade optical flow for reduced frame memory access | |
| CN206177294U (en) | Binocular stereoscopic vision system | |
| Li et al. | HSurf-Net: Normal estimation for 3D point clouds by learning hyper surfaces | |
| CN105387846A (en) | Normal incidence correcting method and system for satellite images | |
| Licciardo et al. | Frame buffer-less stream processor for accurate real-time interest point detection | |
| Zheng et al. | Volumetric parameterization with truncated hierarchical B-splines for isogeometric analysis | |
| Bellas et al. | Real-time fisheye lens distortion correction using automatically generated streaming accelerators | |
| CN117095132B (en) | Three-dimensional reconstruction method and system based on implicit function | |
| Ramesh et al. | Optimization and evaluation of image-and signal-processing kernels on the TI C6678 multi-core DSP | |
| CN108198125A (en) | A kind of image processing method and device | |
| Chalimbaud et al. | Design of an imaging system based on FPGA technology and CMOS imager | |
| Gálvez et al. | Firefly algorithm for Bézier curve approximation | |
| Ryoo et al. | Real-time implementation of an LUT-based image warping system | |
| Belyakov et al. | An FPGA-optimized architecture of variational optical flow | |
| Qamar et al. | Design space exploration of a stereo vision system using high-level synthesis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |