CN106097342B - A kind of self-calibrating method of robot astronaut binocular vision system - Google Patents
A kind of self-calibrating method of robot astronaut binocular vision system Download PDFInfo
- Publication number
- CN106097342B CN106097342B CN201610414437.9A CN201610414437A CN106097342B CN 106097342 B CN106097342 B CN 106097342B CN 201610414437 A CN201610414437 A CN 201610414437A CN 106097342 B CN106097342 B CN 106097342B
- Authority
- CN
- China
- Prior art keywords
- parallel lines
- vision system
- self
- image
- binocular vision
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Abstract
The invention discloses a kind of self-calibrating methods of robot astronaut binocular vision system, by establishing information bank, solve the problems, such as the self-calibrating method based on parallel lines to light sensitive using Asift&Ransac, image segmentation, hough transformation and Ransac fitting；The position for introducing the method control vision system of distance threshold, solves the restricted problem of the self-calibrating method imaging plane based on parallel lines.When robot astronaut is transported heaven and the something unexpected happened in operation process, the environment that can use space station carries out self-calibration to its binocular vision system, and according to the environment of space station during self-calibration, adaptive adjustment algorithm, obtain accurate binocular vision system parameter, it realizes accurate three-dimensional localization, carries out the tasks such as operating terminal for robot astronaut and support is provided.
Description
Technical field
The present invention relates to a kind of self-calibrating methods of robot astronaut binocular vision system, solve complex space station environment
The real-time Auto-calibration problem of inner machine people's astronaut's binocular vision system.
Background technique
Binocular vision system can be generally installed in paleocinetic robot astronaut system, to assist its realization to lead
The multifarious tasks such as boat, positioning, operating terminal.In order to use binocular vision system to carry out accurate three-dimensional localization, it is necessary to
To the parameter of binocular vision system, that is, demarcate.Binocular vision system during transporting by rocket, hold very much by parameter
It easily changes, needs to re-scale it；And after coming into operation in space station, if camera breaks down and replaces, or
Collide with Deng fortuitous events are encountered in operation process, are equally also required to re-scale its parameter.Traditional vision system mark
The method of determining, which needs to have, understands that the people of relevant professional knowledge assists to demarcate using target.Task is various after spacefarer steps on day, religion
The method that spacefarer grasps calibration technique is worthless, and by the method for remote control, ground expert progress proving operation is cumbersome,
And can not real-time calibration, therefore traditional scaling method is restricted in this case.
So the present invention discloses a kind of robot astronaut binocular vision system self-calibrating method.Existing self-calibrating method
It include mainly method, hierarchical reconfiguration method, the method based on geometrical characteristic etc. based on Kruppa equation.Based on Kruppa equation
Method be not necessarily to given parameters initial value, it is easy to use, but to noise-sensitive, do not solve unique Solve problems.Hierarchical reconfiguration
Method precision is relative to Kruppa higher, but poor robustness, initial value are chosen sensitive.Method based on geometrical characteristic mainly limits
System is that application is needed to have corresponding material, and the method for relatively mainstream is to carry out self-calibration based on parallel lines feature now
Method.But this algorithm is handled based on parallel lines, it is more sensitive to parallel lines identification situation, therefore this algorithm at present
Still in the self-calibration application stage under laboratory and simple background.
In order to solve the problems, such as that traditional calibration algorithm brings the calibration of Space-Station-Robots astronaut's binocular vision system,
The technology used in the present invention means are: self-calibrating method being introduced into space station, using common parallel in space station
Line, select the self-calibrating method based on parallel lines vanishing point geometrical characteristic, with realize in space station binocular vision system from
Calibration.
The reason of this algorithm is applied in space station, calibration result precision is influenced mainly has: complicated linear relation, multiple
The angle of miscellaneous space station light environment, parallel lines place plane and imaging plane.
Summary of the invention
In order to solve the above technical problems, the technical solution adopted by the present invention is that providing a kind of robot astronaut binocular vision
The self-calibrating method of system carries out matched method solution twice using Asift&Ransac combination algorithm by establishing information bank
Determined the self-calibrating method based on parallel lines complex background application in parallel lines identification the problem of.By utilizing Asift&
Ransac combination algorithm realizes object matching, realizes the extraction of area-of-interest, is converted using Tu Xiangfenge &hough and realizes sense
The diminution in interest region realizes the parallel lines fit under high-noise environment using Ransac algorithm fitting a straight line.
To sum up, the self-calibrating method based on parallel lines is solved the problems, such as to light sensitive.
The position for introducing the method control vision system of distance threshold, solves the self-calibrating method imaging based on parallel lines
The restricted problem of plane.
The self-calibrating method of the robot astronaut binocular vision system, includes the following steps:
(1) information bank is established, includes the information with the object of parallel lines feature in the information bank, to be used for self-calibration
The matching of middle characteristic point；
(2) the binocular vision system random acquisition image, carries out parallel line drawing；
(3) self-calibration is carried out using the data of extracted parallel lines.
The method further includes the verifying of step (4) distance threshold, to control the position of the vision system.
The step (1) includes: by acquiring image, before the transmitting of space station, to special with parallel lines in space station again
The object of sign carries out sampling typing information bank；Also, the region where manual extraction each object parallel lines, by region extend out with
For more accurately Feature Points Matching, the region where parallel lines after corresponding objects are extended out is recorded into information bank.
The object includes: luggage carrier, notebook, station and/or bulkhead.
The step (2) includes:<2.1>random acquisition image again, is carried out using the object in Asift algorithm and information bank
Feature Points Matching, after carrying out Mismatching point rejecting using Ransac algorithm, the object for selecting match point most, as this acquisition
Parallel line drawing object, i.e. parallel lines corresponding region.
The Feature Points Matching includes: to address the corresponding objects of deposit information bank in advance according to path in information bank again
This region and acquisition image are carried out the matching of further feature point using Asift algorithm and Ransac algorithm by parallel lines region.
The step (2) includes:<2.2>after obtaining rough parallel lines corresponding region again, using image segmentation,
The diminution of area-of-interest is realized in hough transformation.
The mode of described image segmentation includes Threshold segmentation.
The step (2) includes:<2.3>after extracting parallel lines using hough transformation again, the parallel lines extracted herein
On extend out, taken at random a little in the region after this is extended out, obtain original point set, using aberration correction algorithm to original point set carry out
Correction obtains correction point set.
The step (2) includes: that<2.4>are fitted correction point set using Ransac algorithm again, obtains parallel lines solution
Analysis solution, to complete the extraction of parallel lines data.
The step (4) includes:<4.1>before step (2) again, and vanishing point coordinate is arranged relative to image coordinate system origin
Distance threshold d1；Setting acquisition image frequency n=1；
<4.2>after step (2), by acquired parallel lines analytic solutions, vanishing point coordinate is obtained, and then be calculated
Distance d of the vanishing point coordinate relative to image coordinate system origin；
<4.3>judge whether d meets less than distance threshold d1；
Such as be unsatisfactory for, then adjust the movement of the vision system according to the difference of the two, then return step (2) again into
Row Image Acquisition；
If met, then the data are stored, and judge whether n >=8,
If it is not, setting n=n+1, return step (2) re-start Image Acquisition；
If so, stopping carrying out Image Acquisition, and view is calculated using the parallel lines data of 8 groups of obtained parallel lines analytic solutions
Feel system parameter and exports.
The movement for adjusting the vision system includes adjusting the shooting orientation of camera in the vision system.
The adjusting in the shooting orientation is realized by the adjusting of the rotational angle of robot head.
When robot astronaut is transported heaven and the something unexpected happened in operation process, space can use
The environment stood carries out self-calibration to its binocular vision system, and according to the environment of space station during self-calibration, adaptively
Adjustment algorithm, obtain accurate binocular vision system parameter, realize accurate three-dimensional localization, grasp for robot astronaut
Make the tasks such as terminal and support is provided.
The present invention is quasi- using Asift&Ransac, image segmentation, hough transformation and Ransac by establishing information bank
Conjunction solves the problems, such as the self-calibrating method based on parallel lines to light sensitive；Introduce the method control vision system of distance threshold
Position solves the restricted problem of the self-calibrating method imaging plane based on parallel lines.
The invention has the advantages that can have when robot astronaut's binocular vision system needs calibrating parameters
When target, illumination power, automatic Calibration is realized in any operating area in space station, especially in operation, burst
Situation parameter change can carry out real-time automatic Calibration, assist without ground staff and astronaut.
Detailed description of the invention
Fig. 1 is geometrical model of the two groups of orthogonal parallel lines Jing Guo preferred view.
Relational graph of the Fig. 2 between coordinate system.
Fig. 3 is parallel line drawing flow chart.
Fig. 4 is self-calibration overall flow figure.
Specific embodiment
Detailed description of the present invention embodiment with reference to the accompanying drawing.
Self-calibration model is established, two vanishing points obtained under image coordinate system by template image are as follows: A (uA,vA), B (uB,
vB), line midpoint is E (uE,vE), uE=(uA+uB)/2, vE=(vA+vB)/2, with the company of vanishing point A, B under camera coordinate system
Line is the spherosome equation of diameter are as follows:
By optical center coordinateIt is located on the ball, obtains:
In formula (1), f is lens focus, dx、dyRespectively represent physical size of the pixel in X-axis, Y direction；
Formula (2) is about camera intrinsic parameter cx,cy,fx,fyEquation, wherein cx、cyRespectively the central point of image is sat in image
X-axis, y-axis coordinate value under mark system, fx=f/dx, fy=f/dy, shoot four width or images above can be obtained above-mentioned 4 it is unknown
Several unique solutions.
As shown in Fig. 2, world coordinate system is chosen are as follows: with L1,L3Intersection point OwFor the center of circle, L1,L3Respectively x-axis and y-axis side
To determining z-axis by right-handed system rule.Under camera coordinate system, the vector that vanishing point A, B and optical center are constituted is respectivelyIfVector after being normalized is respectively a, b, c, vanishing point coordinate system
A under OABC, b, c are respectively the unit vector in 3 reference axis, then are the spin matrix R' satisfaction of OABC and camera coordinate system:
R'=[a b c].The direction relations of the spin matrix R and R' of world coordinate system and camera coordinate system can utilize projection vectorWith x', the syntactics of y' judge, wherein Ow' it is world coordinates origin OwProjection.
Coordinate (u of the origin of the known world coordinate system in the plane of delineationw,vw), then the origin of world coordinate system is in video camera
Coordinate under coordinate system is λ [(uw-cx)·dx(vw-cy)·dyF], only mono- unknown number of λ, the plane where known parallel lines
The coordinate under coordinate and world coordinate system under the camera coordinate system at upper any point, can be obtained translation matrix.
As shown in figure 3, the self-calibrating method of robot astronaut's binocular vision system, by establishing information bank, in determination
After starting self-calibration program, binocular vision system random acquisition image implements parallel line drawing in turn, and then utilizes extracted
The data of parallel lines carry out self-calibration.Wherein, the parallel line drawing, and include the following steps:
Obtain the image of random acquisition after the starting of self-calibration program, using the object in Asift algorithm and information bank into
Row Feature Points Matching, after carrying out Mismatching point rejecting using Ransac algorithm, the object for selecting match point most is adopted as this
The parallel line drawing object of collection, i.e. parallel lines corresponding region.In this step: deposit in advance being addressed according to path in information bank
The parallel lines region of the corresponding objects of information bank is carried out this region and acquisition image using Asift algorithm and Ransac algorithm
The matching of further feature point obtains the parallel lines corresponding region in acquisition image.
After obtaining this rough parallel lines corresponding region, area-of-interest is realized using image segmentation, hough transformation
It reduces.Wherein, the mode of described image segmentation includes Threshold segmentation.
Area-of-interest after above-mentioned diminution carries out scatterplot extraction.Since there are the abnormal of certain level for camera lens
Become.On distortion imaging surface, the projection of straight line will bend.As directly mentioned using hough transformation in conjunction with least square method
It makes even line, it will ignore a critically important internal reference of video camera: lens distortion, so that calibration result be made to generate certain mistake
Difference.Thus, after going out parallel lines using hough change detection, certain region is extended out on the parallel lines that extract herein, in this area
It is taken at random a little in domain, obtains original point set, original point is corrected using aberration correction algorithm afterwards, obtain correction point set.
About aberration correction algorithm,
Establish the target function of verticality
In formulaThe link vector of two vanishing points and optical center O that are determined for the i-th width image,
(u'Ai,v'Ai) and (u'Bi,v'Bi) it is to the i-th width image projection straight line by two blankings obtained after formula (5) amendment
The image coordinate of point, establishes optimization problem shown in formula (6)
X=x'/(1+k (x'2+y'2)
Y=y'/(1+k (x'2+y'2) (5)
For the R=[r acquired1' r2' r3'], due to optimizing, obtained intrinsic parameter is different surely to make r1',r2',r3'
Meet rotating orthogonal property, the optimum solution of R is sought using minimum distance criterion.Even carrying out singular value decomposition to R, i.e. USV divides
Solution, R=USVT, work as R=UVTWhen obtain maximum value, to obtain outer parametric optimal solution.
Correction point set is fitted using Ransac algorithm, parallel lines analytic solutions are obtained, to complete parallel lines data
Extraction.
The process for establishing information bank includes: by acquiring image, before the transmitting of space station, to having in space station again
The object of parallel lines feature, such as luggage carrier, notebook, station, bulkhead carry out sampling typing information bank；And
Region where manual extraction each object parallel lines extends out in region slightly, in order to subsequent more accurate spy
Sign point matching, the region where parallel lines after corresponding objects are extended out is recorded into information bank.
In addition, as shown in figure 4, the self-calibrating method further includes the verifying of distance threshold, to control the position of vision system
It sets, to solve the restricted problem of the self-calibration algorithm imaging plane based on parallel lines.
In the verifying of the distance threshold, firstly, distance of the setting vanishing point coordinate relative to image coordinate system origin
Threshold value d1；Setting acquisition image frequency n=1；Carry out Image Acquisition；Carry out parallel line drawing；Pass through acquired parallel lines solution
Analysis solution, obtains vanishing point coordinate, and then distance d of the vanishing point coordinate relative to image coordinate system origin is calculated, judges that d is
It is no to meet less than distance threshold d1。
It is such as unsatisfactory for, then adjusts the movement of the vision system according to the difference of the two, then return and re-start image
Acquisition；Wherein, the movement for adjusting the vision system includes adjusting the shooting orientation of camera in the vision system；The bat
The adjusting for taking the photograph orientation is realized by the adjusting of the rotational angle of robot head.
If met, then data are stored, and judge whether n >=8, if it is not, setting n=n+1, re-starts Image Acquisition；If
It is then to stop carrying out Image Acquisition, and the parallel lines data of obtain 8 groups of parallel lines analytic solutions are inputted in self-calibration model,
Computation vision system parameter (f, k, cx,cy, R, T) and export, to complete self-calibration.Wherein, f is lens focus, and k is camera lens
Distortion, cx、cyRespectively x-axis, y-axis coordinate value of the central point of image under image coordinate system, R, T are respectively right mesh camera phase
For spin matrix, the translation vector of left mesh camera.
A specific example illustrates the principle and implementation of the invention for use above, the explanations of above embodiments
It is merely used to help understand method and its core concept of the invention；At the same time, for those skilled in the art, according to this
The thought of invention, there will be changes in the specific implementation manner and application range, in conclusion the content of the present specification is not answered
It is interpreted as limitation of the present invention.
Claims (9)
1. a kind of self-calibrating method of robot astronaut binocular vision system, which comprises the steps of:
(1) information bank is established, includes the information with the object of parallel lines feature in the information bank, for special in self-calibration
Levy the matching of point；
(2) the binocular vision system random acquisition image, carries out parallel line drawing；
(3) self-calibration is carried out using the data of extracted parallel lines；
Wherein, the step (2) includes: again
<2.1>random acquisition image carries out Feature Points Matching using the object in Asift algorithm and information bank, utilizes Ransac
After algorithm carries out Mismatching point rejecting, the object for selecting match point most is as the parallel line drawing object of this acquisition, i.e., flat
Line corresponding region；
<2.2>after obtaining rough parallel lines corresponding region, area-of-interest is realized using image segmentation, hough transformation
It reduces；
<2.3>it after extracting parallel lines using hough transformation, is extended out on the parallel lines that extract herein, the area after this is extended out
It is taken at random a little in domain, obtains original point set, original point set is corrected using aberration correction algorithm, obtain correction point set；
<2.4>correction point set is fitted using Ransac algorithm, parallel lines analytic solutions is obtained, to complete parallel lines data
Extraction.
2. the method as claimed in claim 1, which is characterized in that further include the verifying of step (4) distance threshold, described in control
The position of binocular vision system.
3. such as the method for claims 1 or 2, which is characterized in that the step (1) includes: by acquiring image, in sky again
Between station transmitting before, in space station with parallel lines feature object carry out sampling typing information bank；Also, manual extraction is each
Region where object parallel lines extends out region to be used for more accurately Feature Points Matching, flat after corresponding objects are extended out
It records into information bank in region where line.
4. the method as claimed in claim 1, which is characterized in that the object include: luggage carrier, notebook, station and/or
Bulkhead.
5. the method as claimed in claim 1, which is characterized in that the Feature Points Matching includes: in information bank according to road again
The parallel lines region of diameter addressing corresponding objects of deposit information bank in advance, using Asift algorithm and Ransac algorithm by this region
The matching of further feature point is carried out with acquisition image.
6. the method as claimed in claim 1, which is characterized in that the mode of described image segmentation includes Threshold segmentation.
7. the method as claimed in claim 2, which is characterized in that the step (4) includes: again
<4.1>before step (2), distance threshold d of the vanishing point coordinate relative to image coordinate system origin is set1；Setting acquisition figure
As frequency n=1；
<4.2>after step (2), by acquired parallel lines analytic solutions, vanishing point coordinate is obtained, and then blanking is calculated
Distance d of the point coordinate relative to image coordinate system origin；
<4.3>judge whether d meets less than distance threshold d1；
Such as be unsatisfactory for, then adjust the movement of the binocular vision system according to the difference of the two, then return step (2) again into
Row Image Acquisition；
If met, then the data are stored, and judge whether n >=8,
If it is not, setting n=n+1, return step (2) re-start Image Acquisition；
If so, stopping carrying out Image Acquisition, and utilize the parallel lines data computation vision system of 8 groups of obtained parallel lines analytic solutions
System parameter simultaneously exports.
8. the method as claimed in claim 7, which is characterized in that the movement for adjusting the binocular vision system includes described in adjusting
The shooting orientation of camera in binocular vision system.
9. the method as claimed in claim 8, which is characterized in that the rotation that the adjusting in the shooting orientation passes through robot head
The adjusting of angle is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610414437.9A CN106097342B (en) | 2016-06-13 | 2016-06-13 | A kind of self-calibrating method of robot astronaut binocular vision system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610414437.9A CN106097342B (en) | 2016-06-13 | 2016-06-13 | A kind of self-calibrating method of robot astronaut binocular vision system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN106097342A CN106097342A (en) | 2016-11-09 |
| CN106097342B true CN106097342B (en) | 2019-04-30 |
Family
ID=57846430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610414437.9A Active CN106097342B (en) | 2016-06-13 | 2016-06-13 | A kind of self-calibrating method of robot astronaut binocular vision system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN106097342B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111968074A (en) * | 2020-07-14 | 2020-11-20 | 北京理工大学 | Method for detecting and harvesting lodging crops of harvester by combining binocular camera and IMU |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105222760A (en) * | 2015-10-22 | 2016-01-06 | 一飞智控(天津)科技有限公司 | The autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method |
-
2016
- 2016-06-13 CN CN201610414437.9A patent/CN106097342B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105222760A (en) * | 2015-10-22 | 2016-01-06 | 一飞智控(天津)科技有限公司 | The autonomous obstacle detection system of a kind of unmanned plane based on binocular vision and method |
Non-Patent Citations (8)
| Title |
|---|
| Design of a Space Robot System to Simulate Climbing of Astronaut Based on Binocular Vision System;Que Dong et al.;《 Intelligent Autonomous Systems 12》;20131231;第231-243页 |
| Dynamic Stability Control for a Bio-robot with Primates-Inspired Active Tail;Li Xiaoyun et al.;《IEEE International Conference on Mechatronics & Automation》;20151231;第2035-2040页 |
| Robust Camera Calibration with Vanishing Points;Xiaoquan Xu et al.;《2012 5th International Congress on Image and Signal Processing》;20121231;第931-935页 |
| Target-tools recognition method based on an image feature library for space station cabin service robots;Lingbo Cheng et al.;《Robotica》;20141231;第1-17页 |
| 井下移动机器人双目视觉摄像机的标定方法;陈学惠 等;《中国煤炭》;20111231;第37卷(第11期);第59-63页 |
| 双目立体视觉测量系统的标定;杨景豪 等;《光学精密工程》;20160229;第24卷(第2期);第300-308页 |
| 基于平行直线的摄像机标定方法;马长正 等;《北京航空航天大学学报》;20091031;第35卷(第10期);第1210-1213、1219页 |
| 机器人结构光视觉系统标定研究;刘艳;《中国博士学位论文全文数据库》;20160415;第2016年卷(第4期);第I138-46页 |
Also Published As
| Publication number | Publication date |
|---|---|
| CN106097342A (en) | 2016-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109308693B (en) | Single-binocular vision system for target detection and pose measurement constructed by one PTZ camera | |
| CN105678689B (en) | High-precision map data registration relation determining method and device | |
| CN111640157B (en) | Checkerboard corner detection method based on neural network and application thereof | |
| CN109409292A (en) | The heterologous image matching method extracted based on fining characteristic optimization | |
| CN111721259B (en) | Underwater robot recovery positioning method based on binocular vision | |
| CN109211198B (en) | Intelligent target detection and measurement system and method based on trinocular vision | |
| CN103942796A (en) | High-precision projector and camera calibration system and method | |
| CN112819903A (en) | Camera and laser radar combined calibration method based on L-shaped calibration plate | |
| CN106340044A (en) | Camera external parameter automatic calibration method and calibration device | |
| CN108647580B (en) | Improved SIFT-based ISAR image feature point extraction and matching method | |
| CN112862881B (en) | Road map construction and fusion method based on crowd-sourced multi-vehicle camera data | |
| CN111738320B (en) | Shielded workpiece identification method based on template matching | |
| CN109448059B (en) | Rapid X-corner sub-pixel detection method | |
| CN107862319B (en) | Heterogeneous high-light optical image matching error eliminating method based on neighborhood voting | |
| CN113313659B (en) | High-precision image stitching method under multi-machine cooperative constraint | |
| CN111967337A (en) | Pipeline line change detection method based on deep learning and unmanned aerial vehicle images | |
| CN111784655A (en) | Underwater robot recovery positioning method | |
| CN114331995A (en) | Multi-template matching real-time positioning method based on improved 2D-ICP | |
| CN113793266A (en) | Multi-view machine vision image splicing method, system and storage medium | |
| CN115239820A (en) | Split type flying vehicle aerial view real-time splicing and parking space detection method | |
| CN114998773A (en) | Characteristic mismatching elimination method and system suitable for aerial image of unmanned aerial vehicle system | |
| CN106097342B (en) | A kind of self-calibrating method of robot astronaut binocular vision system | |
| US20220114387A1 (en) | Microscopy System and Method for Generating Training Data | |
| CN112416000B (en) | Unmanned equation motorcycle race environment sensing and navigation method and steering control method | |
| CN104484647B (en) | A kind of high-resolution remote sensing image cloud height detection method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |