CN114663304A - Sparse prior-based optical synthetic aperture system imaging enhancement method - Google Patents
Sparse prior-based optical synthetic aperture system imaging enhancement method Download PDFInfo
- Publication number
- CN114663304A CN114663304A CN202210257107.9A CN202210257107A CN114663304A CN 114663304 A CN114663304 A CN 114663304A CN 202210257107 A CN202210257107 A CN 202210257107A CN 114663304 A CN114663304 A CN 114663304A
- Authority
- CN
- China
- Prior art keywords
- image
- synthetic aperture
- aperture system
- prior
- optical synthetic
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 47
- 230000003287 optical effect Effects 0.000 title claims abstract description 36
- 238000003384 imaging method Methods 0.000 title claims abstract description 31
- 108091006146 Channels Proteins 0.000 claims description 32
- 239000011159 matrix material Substances 0.000 claims description 15
- 230000008569 process Effects 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 6
- OAICVXFJPJFONN-UHFFFAOYSA-N Phosphorus Chemical compound [P] OAICVXFJPJFONN-UHFFFAOYSA-N 0.000 claims 1
- 230000015556 catabolic process Effects 0.000 abstract description 4
- 238000006731 degradation reaction Methods 0.000 abstract description 4
- 230000000694 effects Effects 0.000 abstract description 4
- 238000004088 simulation Methods 0.000 description 5
- 238000001914 filtration Methods 0.000 description 4
- 238000002474 experimental method Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 239000006185 dispersion Substances 0.000 description 1
- 210000000887 face Anatomy 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
Abstract
The invention discloses an optical synthetic aperture system imaging enhancement method based on sparse prior. The method is used for solving the problem of imaging degradation of the optical synthetic aperture system caused by the influence of array structure and common phase error. The method is characterized in that a dark channel and image gradient prior are used as sparse prior to design of an imaging enhancement model, the method flow comprises the steps of firstly calculating a point spread function under a synthetic aperture system array structure to be used as initial blur kernel input, then solving the sparse prior through a half-quadratic splitting method, and solving the model to estimate a final blur kernel and an image after imaging enhancement. The invention utilizes the inherent prior theory of the synthetic aperture system and has the advantages of wide application range, simple realization, good enhancement effect and the like.
Description
Technical Field
The invention relates to the technical field of image processing and optical engineering, in particular to the field of image restoration and enhancement of an optical synthetic aperture system, and particularly relates to an imaging enhancement method of the optical synthetic aperture system based on sparse prior.
Background
The optical synthetic aperture system uses smaller apertures, which are easier to manufacture, in a spatial arrangement to achieve resolution equivalent to a single larger aperture. The method faces serious problems of degraded blurring of images, and the caused reasons are mainly light-passing area reduction caused by an array structure and PSF dispersion caused by a common phase error. And therefore needs to be addressed from the perspective of imaging enhancement.
At present, a wiener filtering method is generally adopted in an imaging enhancement method aiming at a synthetic aperture system. The wiener filtering is based on the least square principle, the digital image is regarded as a two-dimensional stable continuous signal, and the central idea is to minimize the mean square error and maximize the similarity between the restored image and the image before degradation. Wiener filtering has a good effect of restoring a known PSF and a noise type, but when a noise power spectrum cannot be well estimated, the restoration effect is poor, phase errors cannot be solved, and a ringing phenomenon can be generated in a restored image.
Disclosure of Invention
In view of the problems in the prior art, the present invention provides a sparse prior-based optical synthetic aperture system imaging enhancement method, as shown in fig. 2, by using a histogram of dark channel values of a sharp image and a blur image of 445 optical synthetic aperture system imaging simulations, it can be found that the dark channel value of the sharp image is close to 0 value, and the dark channel value of the blur image increases. Therefore, the dark channel can be used as the inherent prior of the image enhancement of the synthetic aperture system, and the imaging effect of the system can be enhanced only by considering the point spread function under the fixed array structure aiming at the specific optical synthetic aperture system in combination with the image gradient prior.
The method adopts the technical scheme that: an optical synthetic aperture system imaging enhancement method based on sparse prior comprises the following steps:
the method comprises the following steps: the system PSF function of the optical synthetic aperture under a fixed array structure is calculated.
Step two: a model for optical synthetic aperture imaging enhancement was proposed:
wherein the first item
As a fidelity term, where B denotes the input blurred image matrix, with m x n x 3 size, k denotes the blur kernel matrix, with p x p size, I denotes the estimated sharp image matrix, with m x n x 3 size,
representing the convolution process, which uses the L2 norm constraint to constrain the value of the sharp image after convolution with the blur kernel to be the minimum with the blur image loss; second item
The solution used to regularize the blur kernel, this term is constrained using the L2 norm; item III
In order to be a gradient constraint term,
a gradient matrix representing the image, the term constrained using the L0 norm; item four
D (I) represents a matrix of dark channel values for image I, which is constrained using the norm L0, for the dark channel constraint term, with α, β, and λ as weighting parameters.
Step three: the image dark channels and gradients are sparsely constrained as regularization terms in the model for optical synthetic aperture imaging enhancement in the form of the L0 norm. The dark channel is expressed asTaking the minimum value of three channels in the range of 3 x 3 of each pixel point field of the RGB image, and calculating according to the following mode
Where x and y represent the position of the image pixel, respectively, and P (x) is a neighborhood centered at x; c is the color channel of the set r, g, b, D (I) is (x) represents the dark channel value of image I at x pixel,
any pixel representing image I is set to the minimum value in the RGB three channels in its neighborhood p (x). The gradient represents the change rate of the pixel point in the horizontal and vertical directions, and the calculation mode is as follows:
gx and gy represent the change rate in the horizontal and vertical directions respectively, and the calculation interval is 2 pixel points.
Step four: and inputting the blurred image acquired by the synthetic aperture system and the PSF function calculated in the step one as an initial blur kernel.
Step five: the image is down sampled in the form of a pyramid. In the downsampling process, firstly, the number num _ scales of layers of a pyramid is calculated according to an acquired synthetic aperture system image and the input fuzzy kernel size KemelSize:
and in the first iteration process, the image is reduced to the minimum value according to the layer number proportion, and the later iteration is carried out according to the result of the previous layer and the image pyramid layer number proportion up-sampling.
Step six: and solving the objective function of the model by adopting a semi-quadratic splitting method. The size of the blur kernel is calculated according to the initial blur kernel size KernelSize and the number i of layers of the image pyramid, num _ scales, as follows:
each estimate of the blur kernel is the result of the previous layer of calculations.
Compared with the prior art, the invention has the following advantages and innovativeness:
1. with an inherent prior: the dark channel and the gradient prior have the characteristics of simplicity and high efficiency, and the method only needs to input a point spread function under the array structure as an initial value of fuzzy kernel estimation and can obtain a final clear image through multiple iterative estimation.
2. Compared with the prior image enhancement method which needs strict optical system prior, the method has better generalization capability, and can initialize the blur kernel to the zero matrix without the parameters of the optical system, and carry out blind restoration on the blur image imaged by the system.
3. The invention can effectively solve the problem of image degradation caused by noise and common phase error in system imaging.
Drawings
FIG. 1 is a flowchart of an image enhancement method for an optical synthetic aperture system based on sparse prior in the present invention.
FIG. 2 is a histogram of the dark channel values of pixels of a sharp image and a blurred image according to the verification of the dark channel prior theory of the present invention.
FIG. 3 shows the simulation results of the present invention.
FIG. 4 shows the average PSNR and SSIM index results of the simulation experiments of the present invention.
Fig. 5 shows the blind restoration result of the practical application of the method.
Detailed Description
The invention is further described with reference to the following figures and detailed description.
The invention provides an imaging enhancement method for a synthetic aperture optical system. The method can solve the problem of image blurring caused by image noise, common phase errors and other factors and restore the detailed part of the image.
As shown in FIG. 1, the invention provides an optical synthetic aperture system imaging enhancement method based on L0 sparse prior, comprising the following steps:
step 1: the system PSF function of the optical synthetic aperture under a fixed array structure is calculated. The calculation method is as follows:
wherein PSF (x, y) is a PSF function of the system; x and y represent the position of the coordinate plane; n represents the number of subapertures of the synthetic aperture system; n and m represent any two sub-apertures; λ is the center wavelength; z is the distance between the exit pupil plane and the image plane of the system, namely the focal length of the synthetic aperture system; a ismAnd bmRespectively as the position of the center of a sub-aperture on a coordinate axis; PSFsub(x, y) is the amplitude spread function for each subaperture of the system, expressed as:
wherein r is0Field stop diameter of subaperture, J1Is a first order Bessel function;
step 2: a model for optical synthetic aperture imaging enhancement is proposed:
first item
As a fidelity term, where B denotes the input blurred image matrix, with m x n x 3 size, k denotes the blur kernel matrix, with p x p size, I denotes the estimated sharp image matrix, with m x n x 3 size,
presentation volumeA product process, which uses an L2 norm constraint to constrain the value of the sharp image after convolution with the blur kernel to a minimum with the blur image loss; second item
The solution to regularize the blur kernel, which is solved by fast fourier transform using the L2 norm constraint; item III
In order to be a gradient constraint term,
representing the gradient of the image, which is used for eliminating the smaller gradient and retaining the detail with the larger gradient, and the L0 norm constraint is used; item four
D (I) represents a matrix of dark channel values (of size m × n) for image I, such that the estimated degree of dark channel sparsity for sharp images is as high as possible, for the dark channel constraint term, which is initialized to 0.001, 0.004 and 0.004, respectively, using the L0 norm constraint with α, β and λ as weighting parameters.
And step 3: image dark channels and gradients are sparsely constrained in the model as regularization terms in the form of an L0 norm. The dark channel is expressed as the minimum value of three channels within 3 x 3 of each pixel point field of the obtained RGB image of the optical synthetic aperture system, and is calculated according to the following mode
Where x and y represent the position of the image pixel, respectively, and p (x) is a neighborhood centered at x, set to 3 x 3 pixels in size; c is the color channel of the set r, g, b, D (l) and (x) represents the dark channel value of the image I at x pixel points,
any pixel representing image I is set to the minimum value in the RGB three channels in its neighborhood p (x). Gradient representation pixel points are in horizontal and vertical directionsThe change rate in the direction is calculated in the following way:
and gX and gy respectively represent the change rate in the horizontal direction and the vertical direction, and the calculation interval is 2 pixel points.
And 4, step 4: and inputting the blurred image acquired by the synthetic aperture system and the PSF function calculated in the step one as an initial blur kernel.
And 5: the image is down sampled in the form of a pyramid. The down-sampling process firstly calculates the number num _ scales of pyramid layers according to the input fuzzy kernel size KemelSize 35 × 35:
and in the first iteration process, the image is reduced to the minimum value according to the layer number proportion, and the subsequent iteration process is up-sampled according to the result of the previous layer and the layer number proportion of the image pyramid.
And 6: and solving the objective function of the model by adopting a semi-quadratic splitting method. The size of the blur kernel is calculated according to the initial blur kernel size KemelSize and the number i of layers of the image pyramid num _ scales as follows:
each estimation of the blur kernel is the result of the previous layer calculation.
By carrying out image simulation degradation on the optical synthetic aperture system, carrying out a comparison experiment on the method and a wiener filtering method, carrying out simulation imaging on the synthetic aperture system under a Golay-3 array structure with the sub-aperture diameter D of 200mm and the circumscribed circle diameter D of 500mm, wherein the imaging enhancement result is shown in figure 3, and the two objective index results of peak signal-to-noise ratio (PSNR) and Structural Similarity (SSIM) are shown in figure 4. Meanwhile, the method of the present invention was subjected to practical engineering experiments, and the results are shown in fig. 5. The above experimental results can prove that the present invention is effective for enhancement of an optical synthetic aperture imaging system.
It should be understood that the above description of the preferred embodiments is given for clarity and not for any purpose of limitation, and that various changes, substitutions and alterations can be made herein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (6)
1. An optical synthetic aperture system imaging enhancement method based on sparse prior is characterized in that: comprises the following steps:
the method comprises the following steps: calculating a system PSF function of the optical synthetic aperture under a fixed array structure;
step two: providing a model for optical synthetic aperture imaging enhancement;
step three: sparsely constraining dark channels and gradients of the image as regularization terms in a model for optical synthetic aperture imaging enhancement in the form of an L0 norm;
step four: inputting the blur image acquired by the synthetic aperture system and the PSF function calculated in the first step as an initial blur kernel;
step five: downsampling the image in a pyramid form;
step six: and solving the objective function of the model by adopting a semi-quadratic splitting method.
2. The sparse-prior-based optical synthetic aperture system imaging enhancement method of claim 1, wherein: in step two and step three, the model for optical synthetic aperture imaging enhancement is as follows:
wherein the first item
For fidelity, where B represents the input blurred image matrix with size m n 3, kRepresenting a fuzzy kernel matrix, with p × p size; i denotes the estimated sharp image matrix, with m x n x 3 size;
representing the convolution process, which uses the L2 norm to constrain the value of the sharp image after convolution with the blur kernel to be the minimum with the blur image loss; second item
The solution used to regularize the blur kernel, this term is constrained using the L2 norm; item III
In order to be a gradient constraint term,
a gradient matrix representing the image, the term constrained using the L0 norm; the fourth term | | D (I) | non-woven phosphor0D (I) represents a matrix of dark channel values for image I, which is constrained using the norm L0, for the dark channel constraint term, with α, β, and λ as weighting parameters.
3. The sparse-prior-based optical synthetic aperture system imaging enhancement method of claim 1 or 2, wherein: the dark channel is expressed as the minimum value of three channels within 3 x 3 of each pixel point field of the obtained RGB image of the optical synthetic aperture system, and is calculated according to the following mode
Where x and y represent the position of the image pixel, respectively; p (x) is a neighborhood centered at x; c is the color channel of the set r, g, b, D (I) is (x) represents the dark channel value of image I at x pixel,
any pixel x representing image I is set to the minimum value in the RGB three channels in its neighborhood p (x).
4. The sparse-prior-based optical synthetic aperture system imaging enhancement method of claim 1 or 3, wherein: the gradient represents the change rate of the pixel point in the horizontal and vertical directions, and the calculation mode is as follows:
gx and gy represent the change rate in the horizontal and vertical directions respectively, and the calculation interval is 2 pixel points.
5. The sparse-prior-based optical synthetic aperture system imaging enhancement method of claim 1, wherein: in the fifth step, in the downsampling process, firstly, the number of layers num _ scales of the pyramid is calculated according to the input blur kernel size KernelSize of the acquired synthetic aperture system image:
and in the first iteration process, the image is reduced to the minimum value according to the layer number proportion, and the subsequent iteration process is up-sampled according to the result of the previous layer and the layer number proportion of the image pyramid.
6. The sparse-prior-based optical synthetic aperture system imaging enhancement method of claim 1 or 6, wherein: the size of the blur kernel is calculated according to the initial blur kernel size KernelSize and the number i of layers of the image pyramid as follows:
each blur kernel estimate is the result of the previous layer of computations.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210257107.9A CN114663304B (en) | 2022-03-16 | 2022-03-16 | Sparse prior-based optical synthetic aperture system imaging enhancement method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210257107.9A CN114663304B (en) | 2022-03-16 | 2022-03-16 | Sparse prior-based optical synthetic aperture system imaging enhancement method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114663304A true CN114663304A (en) | 2022-06-24 |
| CN114663304B CN114663304B (en) | 2023-09-19 |
Family
ID=82028792
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210257107.9A Active CN114663304B (en) | 2022-03-16 | 2022-03-16 | Sparse prior-based optical synthetic aperture system imaging enhancement method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114663304B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150254810A1 (en) * | 2014-03-07 | 2015-09-10 | The University Of British Columbia | System and method for solving inverse imaging problems |
| CN105023245A (en) * | 2015-05-05 | 2015-11-04 | 苏州大学 | Image smoothing method under strength and gradient sparsity constraint |
| CN105824030A (en) * | 2016-03-10 | 2016-08-03 | 中国科学院光电技术研究所 | Sparse optical synthetic aperture imaging method based on sub-aperture shutter modulation phase difference method |
| CN109633646A (en) * | 2019-01-21 | 2019-04-16 | 中国人民解放军陆军工程大学 | Bistatic ISAR imaging method based on weighted L1 norm constraint |
| CN109767404A (en) * | 2019-01-25 | 2019-05-17 | 重庆电子工程职业学院 | Infrared image deblurring method under a kind of salt-pepper noise |
| CN109919871A (en) * | 2019-03-05 | 2019-06-21 | 重庆大学 | Fuzzy core estimation method based on image and fuzzy core mixed constraints |
| US20190365355A1 (en) * | 2017-01-18 | 2019-12-05 | Technion Research & Development Foundation Ltd. | Sparsity-based ultrasound super-resolution imaging |
| CN113538374A (en) * | 2021-07-15 | 2021-10-22 | 中国科学院上海技术物理研究所 | Infrared image blur correction method for high-speed moving object |
| CN113689336A (en) * | 2021-08-25 | 2021-11-23 | 华北电力大学（保定） | Power equipment infrared image non-blind super-resolution method |
| CN113963349A (en) * | 2021-08-17 | 2022-01-21 | 大连理工大学 | Method for extracting individual space-time feature vector and tested classification |
-
2022
- 2022-03-16 CN CN202210257107.9A patent/CN114663304B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150254810A1 (en) * | 2014-03-07 | 2015-09-10 | The University Of British Columbia | System and method for solving inverse imaging problems |
| CN105023245A (en) * | 2015-05-05 | 2015-11-04 | 苏州大学 | Image smoothing method under strength and gradient sparsity constraint |
| CN105824030A (en) * | 2016-03-10 | 2016-08-03 | 中国科学院光电技术研究所 | Sparse optical synthetic aperture imaging method based on sub-aperture shutter modulation phase difference method |
| US20190365355A1 (en) * | 2017-01-18 | 2019-12-05 | Technion Research & Development Foundation Ltd. | Sparsity-based ultrasound super-resolution imaging |
| CN109633646A (en) * | 2019-01-21 | 2019-04-16 | 中国人民解放军陆军工程大学 | Bistatic ISAR imaging method based on weighted L1 norm constraint |
| CN109767404A (en) * | 2019-01-25 | 2019-05-17 | 重庆电子工程职业学院 | Infrared image deblurring method under a kind of salt-pepper noise |
| CN109919871A (en) * | 2019-03-05 | 2019-06-21 | 重庆大学 | Fuzzy core estimation method based on image and fuzzy core mixed constraints |
| CN113538374A (en) * | 2021-07-15 | 2021-10-22 | 中国科学院上海技术物理研究所 | Infrared image blur correction method for high-speed moving object |
| CN113963349A (en) * | 2021-08-17 | 2022-01-21 | 大连理工大学 | Method for extracting individual space-time feature vector and tested classification |
| CN113689336A (en) * | 2021-08-25 | 2021-11-23 | 华北电力大学（保定） | Power equipment infrared image non-blind super-resolution method |
Non-Patent Citations (6)
| Title |
|---|
| BIN FAN等: "Effective Spectral Unmixing via Robust Representation and Learning-based Sparsity", pages 1 - 13 * |
| ZONGYONG CUI等: "Target recognition in synthetic aperture radar images via non-negative matrix factorisation", vol. 9, no. 9, pages 1376 - 1385, XP006054594, DOI: 10.1049/iet-rsn.2014.0407 * |
| 廖斌等: "基于多分辨率正则化LR的图像盲复原", no. 9, pages 16 - 17 * |
| 李霄: "基于稀疏约束的图像盲复原方法研究", no. 3, pages 138 - 4611 * |
| 郑天宇等: "基于增强拉东变换及稀疏先验的模糊星斑复原", no. 6, pages 189 - 198 * |
| 高敬坤: "阵列三维成像及雷达增强成像技术研究", no. 1, pages 136 - 295 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114663304B (en) | 2023-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110675347B (en) | Image blind restoration method based on group sparse representation | |
| CN111709895A (en) | Image blind deblurring method and system based on attention mechanism | |
| US6240219B1 (en) | Apparatus and method for providing optical sensors with super resolution | |
| CN114647079B (en) | Monolithic broadband diffraction calculation imaging method | |
| Oktem et al. | High-resolution multi-spectral imaging with diffractive lenses and learned reconstruction | |
| Hui et al. | Image restoration of optical sparse aperture systems based on a dual target network | |
| Yang et al. | MRDN: A lightweight Multi-stage residual distillation network for image Super-Resolution | |
| El-Henawy et al. | A comparative study on image deblurring techniques | |
| CN112488920B (en) | Image regularization super-resolution reconstruction method based on Gaussian-like fuzzy core | |
| CN114663304B (en) | Sparse prior-based optical synthetic aperture system imaging enhancement method | |
| CN111179171A (en) | Image super-resolution reconstruction method based on residual module and attention mechanism | |
| Tom et al. | Reconstruction of a high resolution image from multiple low resolution images | |
| Wang et al. | Optical satellite image MTF compensation for remote-sensing data production | |
| Deepak et al. | Analysis of Single Image Super-Resolution Techniques: An Evolutionary Study | |
| Su et al. | Atmospheric turbulence degraded image restoration using a modified dilated convolutional network | |
| CN108681988B (en) | Robust image resolution enhancement method based on multiple images | |
| CN108665412B (en) | Method for performing multi-frame image super-resolution reconstruction by using natural image priori knowledge | |
| Chopade et al. | Lucy Richardson and Mean Modified Wiener Filter for Construction of Super-Resolution Image | |
| CN112330550A (en) | Image restoration method and system based on image blind deconvolution technology | |
| Gunturk et al. | Fundamentals of image restoration | |
| Van Vliet et al. | Improving spatial resolution in exchange of temporal resolution in aliased image sequences | |
| Wu et al. | An image restoration method for the sparse aperture optical system based on deep learning | |
| CN114119400B (en) | Single photon counting image combined denoising and super-resolution convolution network method | |
| Zhong et al. | RPIR: A Semi-Blind Unsupervised Learning Image Restoration Method for Optical Synthetic Aperture Imaging Systems with Co-Phase Errors | |
| CN112308781B (en) | Single image three-dimensional super-resolution reconstruction method based on deep learning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |