CN107218886A - A kind of optical alignment tracing system and method based on stealthy combination road sign - Google Patents
A kind of optical alignment tracing system and method based on stealthy combination road sign Download PDFInfo
- Publication number
- CN107218886A CN107218886A CN201610165554.6A CN201610165554A CN107218886A CN 107218886 A CN107218886 A CN 107218886A CN 201610165554 A CN201610165554 A CN 201610165554A CN 107218886 A CN107218886 A CN 107218886A
- Authority
- CN
- China
- Prior art keywords
- road sign
- infrared
- combination road
- infrared markers
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003287 optical effect Effects 0.000 title claims abstract description 22
- 238000000034 method Methods 0.000 title claims abstract description 18
- 239000000463 material Substances 0.000 claims abstract description 8
- 239000002131 composite material Substances 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 230000008447 perception Effects 0.000 claims description 2
- 230000003796 beauty Effects 0.000 abstract description 3
- 230000007613 environmental effect Effects 0.000 abstract description 3
- 230000003111 delayed effect Effects 0.000 abstract description 2
- 230000003190 augmentative effect Effects 0.000 description 5
- 239000000203 mixture Substances 0.000 description 2
- 208000012886 Vertigo Diseases 0.000 description 1
- 239000011324 bead Substances 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000005094 computer simulation Methods 0.000 description 1
- 239000007788 liquid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/002—Measuring arrangements characterised by the use of optical techniques for measuring two or more coordinates
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The present invention proposes a kind of optical alignment tracing system and method based on stealthy combination road sign, combination road sign is made up of infrared markers and visible ray mark, infrared markers use retroreflecting material, it can be seen that signal uses multistable material, the color that road sign is combined by making is changed into the color of its accompanying object to realize hidden function.Described image capture module can obtain infrared image and visible images.Image processing module is used to carry out image corresponding decoding, and calculates according to the location tracking method attitude information of image capture module six degree of freedom in space.Optical alignment tracing system and method that the present invention is provided, have the advantages that tracking range is big, the low, price that is delayed is low, deployment is easy and does not influence environmental beauty.
Description
Technical field
The present invention relates to optical locating techniques field, optical alignment tracing system and method in virtual reality and augmented reality are related specifically to.
Background technology
From the aspect of the main scalability three from the accuracy of road sign, real-time and in large complicated environment of design of artificial landmark, two-dimensional bar code or circular code mostly using black and white in conventional road sign scheme are used as road sign pattern.Although these schemes all have certain autgmentability, influenceed larger by noise and shooting angle, decoding speed is slow, and the aesthetic property of influence environment.Therefore, consider the design requirement of artificial landmark and a variety of road sign schemes having pointed out, devise it is a kind of based on visible signal and infrared markers can stealth combine road sign.
Augmented reality is a kind of by real-world information and the new technology of virtual world information " perfection is connected ".Virtual reality technology is a kind of can to create the computer simulation technique with the experiencing virtual world.In augmented reality, dummy model is registering with real world inaccurate.In virtual reality, it is impossible to realize that user associates in real world with the position and attitude in virtual world, and easily produce spinning sensation.And stable location tracking system is the key technology for solving these problems, current virtual reality also has the problem of capture range is small with the tracing system in augmented reality equipment, and on a large scale more than tracing system by the way of Outside-in, it is expensive, it is difficult to dispose.
The content of the invention
In view of the shortcomings of the prior art, this patent proposes a kind of optical alignment tracing system and method based on stealthy combination road sign, is particularly suitable for use in virtual reality and augmented reality.
To achieve the above object, the technical solution adopted by the present invention is as follows：
A kind of optical alignment tracing system based on stealthy combination road sign.It includes the combination road sign disposed in the environment and the image capture module and image processing module that are deployed on object to be followed the trail of.
Combination road sign by can be stealthy visible signal and can be stealthy infrared markers constitute.It can be seen that signal is used for presentation code information, infrared markers are used to represent location information.It can be seen that signal is made up of multistable material, it can change between multiple color, and only need energy when changing color, maintain not needing energy during color state or need a small amount of energy, when tracing system works, visible signal is made to be shown as the encoded colors specified, when the tracking decorum is stopped, not influence environmental beauty, make visible signal and its accompanying object color same or like to realize its hidden function.Infrared markers use retroreflecting material, can postback infrared light reflection at light, so as to be captured by image capture module.Infrared markers are consistent with the color keep of accompanying environment all the time to realize its hidden function.
The biocular systems that the colour imagery shot of the image capture module infrared camera infrared by perceiving and perception visible ray is constituted, sensitive chip cross-distribution in the monocular system being either made up of a camera, monocular system inside camera the pixel that can be perceived infrared pixel and perceive visible light wave range.
Image processing module, the image for being captured to image capture module is decoded, and calculates the attitude information of image capture module six degree of freedom in space.
It will be seen that signal is combined coding with infrared markers, specific coding scheme is as follows：Each combination road sign includes the infrared markers of certain amount, by infrared markers with specific, the topology easily recognized is disposed, and the visible signal of certain color is laid around each infrared markers, each color can all be mapped as a concrete numerical value, using the numerical value corresponding to the color around infrared markers as the infrared markers characteristic value, combining all infrared markers in road sign, order is arranged according to the rules, and be combined according to the eigenvalue cluster that the arrangement will organize interior all infrared markers, form a numerical value, this numerical value is the coding corresponding to the combination road sign.
To achieve the above object, invention further provides a kind of optical positioning method based on stealthy combination road sign, comprise the following steps：
Combination road sign is demarcated, the three-dimensional coordinate of all combination road signs is calculated, and is stored in database；
Image capture module obtains infrared image and visible images；
Image processing module is decoded to the combination road sign in image, draws the corresponding ID of each combination road sign；
Using the ID of each combination road sign as index, the true three-dimension coordinate corresponding to each infrared markers in the combination road sign is searched in database；
According to the internal reference of coordinate of each infrared markers in infrared image, true three-dimension coordinate and image collecting device, six-degree-of-freedom posture of the image collecting device in three dimensions is calculated.
The step of wherein image processing module is decoded to the combination road sign in image is as follows：
Each infrared markers are identified from infrared image；
Infrared markers are divided into different groups, every group of infrared markers belong to same combination road sign, remove noise；
Determine putting in order for every group of infrared markers；
Coordinate of each infrared markers in infrared image is mapped in visible images, finds position of the infrared markers in visible images；
The color of infrared markers ambient visible light mark is determined according to this position；
Color is mapped as numerical value, the characteristic value of the infrared markers is used as；
According to the order of every group of infrared markers, it is the decoded result of the composite marking that the characteristic value of every group of infrared markers is sequentially combined into numeral an ID, the digital ID.
Compared with prior art, tracing system involved in the present invention has invented visible ray and infrared combination road sign that can be stealthy, location tracking is carried out by the way of Inside-out, has the advantages that tracking range is big, the low, price that is delayed is low, deployment is easy and does not influence environmental beauty.
Brief description of the drawings
Fig. 1 shows the structural representation of the optical alignment tracing system according to the preferred embodiment of the invention based on stealthy combination road sign；
Fig. 2 shows the composition structure chart of the combination road sign of the optical alignment tracing system according to the preferred embodiment of the invention based on stealthy combination road sign；
Fig. 3 is a kind of flow chart based on the stealthy optical alignment method for tracing for combining road sign according to the preferred embodiment of the present invention.
Embodiment
In order that the narration of the present invention it is more detailed with it is complete, propose illustrative description below for embodiments of the present invention and specific embodiment：But this is not implemented or with the sole mode of the specific embodiment of the invention.In the following description, many specific details be will be described in detail so that reader can fully understand following embodiment.However, embodiments of the invention can be put into practice in the case of without these specific details.
Fig. 1 is a kind of composition frame chart of the optical alignment tracing system based on stealthy combination road sign.It includes the combination road sign module 1 and the image capture module 2 being deployed on object to be followed the trail of and the image processing module 3 being connected with image capture module 2 disposed in the environment.
As shown in Fig. 2 combination road sign module 1 includes visible signal 11,13,15,17 and infrared markers 12,14,16,18.Specific to adopt with the following method to design combination road sign module, wherein infrared markers are made of the retroreflecting material containing bead or crystallite lattice, it is seen that signal is made up of multistable material, such as electric ink or multistable state liquid srystal.On four summits that the center of four infrared markers 12,14,16,18 is arranged in square at certain intervals in the direction of the clock, and the visible signal of certain color is laid around infrared markers, each color can all be mapped as a concrete numerical value.Using the numerical value corresponding to the color around infrared markers as the infrared markers characteristic value, all infrared markers are arranged in the direction of the clock in combination road sign, and be combined according to the eigenvalue cluster that the arrangement will organize interior all infrared markers, a numerical value is formed, this numerical value is the coding corresponding to the combination road sign.For example, setting red, green, blue, purple four color, red is mapped as 1, green is mapped as 2, and blueness is mapped as 3, and purple is mapped as 4.Red visible signal 11 is laid around infrared markers 12, the visible signal 13 of green is laid around infrared markers 14, the visible signal 15 of blueness is laid around infrared markers 16, the visible signal 17 of purple is laid around infrared markers 18, then the result after the combination road sign is decoded clockwise is 1234.
Image capture module 2 is arranged on virtual reality device, the infrared camera 21 infrared by that can perceive of image capture module 2 is constituted with the colour imagery shot 22 that can perceive visible ray.Infrared camera 21 is used to capture the infrared markers in combination road sign module 1, and colour imagery shot module is used to capture the visible signal in combination road sign module 1.Image processing module 3 is used to decode the image that image capture module 2 is captured, and calculates the attitude information of the six degree of freedom in space of image capture module 2.
A kind of flow chart of the optical alignment method for tracing based on stealthy combination road sign is illustrated in figure 3, with reference to Fig. 1, Fig. 2 and Fig. 3, the flow to this method is illustrated：
Step 1：Combination road sign is demarcated, the three-dimensional coordinate of all combination road signs is calculated, and is stored in database；
Step 2：Image collecting device obtains infrared image and visible images；
Step 3：The infrared markers identified are grouped, every group of infrared markers belong to same combination road sign；
Step 4：Denoising is carried out to packet obtained by step 3, putting in order for every group of infrared markers is determined；
Step 5：Coordinate of each infrared markers in infrared image is mapped in visible images, finds position of the infrared markers in visible images；
Step 6：The color of infrared markers ambient visible light mark is determined according to gained position in step 5, will
Color is mapped as numerical value, is used as the characteristic value of the infrared markers；
Step 7：According to the order of every group of infrared markers, the characteristic value of every group of infrared markers is sequentially combined into one
Individual digital ID, the digital ID are the decoded result of the composite marking；
Step 8：The corresponding ID of each combination road sign is calculated, using the ID of each combination road sign as index,
Collect the true three-dimension coordinate searched in the combination road sign corresponding to each infrared markers in database；
Step 9：According to the internal reference of coordinate of each infrared markers in infrared image, true three-dimension coordinate and image collecting device, six-degree-of-freedom posture information of the image collecting device in three dimensions is calculated.
Claims (8)
1. a kind of optical alignment tracing system based on combination road sign, it is characterised in that include combination road sign, image capture module, image processing module, wherein combination road sign is made up of visible signal and infrared markers, image capture module can obtain infrared image and visible images.
2. a kind of optical alignment tracing system based on combination road sign according to claim 1, it is characterised in that combination road sign can be stealthy, by making the color for combining the colour switching of road sign for its accompanying object realize its hidden function.
3. a kind of optical alignment tracing system based on combination road sign according to claim 1, it is characterized in that, visible signal in combination road sign is made up of multistable material, it can change between multiple color, and energy only is needed when changing color, maintain not needing energy during color state or need few energy.
4. a kind of optical alignment tracing system based on combination road sign according to claim 1, it is characterised in that the infrared markers in combination road sign use retroreflecting material, can postback infrared light reflection at light, so as to be captured by image capture module.
5. a kind of optical alignment tracing system based on combination road sign according to claim 1, it is characterized in that, the biocular systems that the colour imagery shot of the image capture module infrared camera infrared by perceiving and perception visible ray is constituted, sensitive chip cross-distribution in the monocular system being either made up of a camera, monocular system inside camera the pixel for perceiving infrared band and the pixel for perceiving visible light wave range.
6. a kind of optical alignment tracing system based on combination road sign according to claim 1, it is characterised in that the coding method of combination road sign is as follows：Each combination road sign includes the infrared markers of certain amount, by infrared markers with specific, the topology easily recognized is disposed, and the visible signal of certain color is laid around each infrared markers, each color can all be mapped as a concrete numerical value, using the numerical value corresponding to the color around infrared markers as the infrared markers characteristic value, combining all infrared markers in road sign, order is arranged according to the rules, and be combined according to the eigenvalue cluster that the arrangement will organize interior all infrared markers, form a numerical value, this numerical value is the corresponding coding of the combination road sign.
7. a kind of optical positioning method based on combination road sign, it is characterised in that be divided into following steps：
Demarcated, calculate the three-dimensional coordinate of all combination road signs, and be stored in database；
Obtain infrared image and visible images；
Combination road sign in image is decoded, the corresponding ID of each combination road sign is drawn；
Using the ID of each combination road sign as index, the true three-dimension coordinate corresponding to each infrared markers in the combination road sign is searched in database；
According to the internal reference of coordinate of each infrared markers in infrared image, true three-dimension coordinate and image collecting device, six-degree-of-freedom posture of the image collecting device in three dimensions is calculated.
8. a kind of optical positioning method based on combination road sign according to claim 7, it is characterised in that be to the process that the combination road sign in image is decoded：
Each infrared markers are identified from infrared image；
Infrared markers are divided into different groups, every group of infrared markers belong to same combination road sign, remove noise；
Determine putting in order for every group of infrared markers；
Coordinate of each infrared markers in infrared image is mapped in visible images, finds position of the infrared markers in visible images；
The color of infrared markers ambient visible light mark is determined according to this position；
Color is mapped as numerical value, the characteristic value of the infrared markers is used as；
According to the order of every group of infrared markers, it is the decoded result of the composite marking that the characteristic value of every group of infrared markers is sequentially combined into numeral an ID, the digital ID.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610165554.6A CN107218886B (en) | 2016-03-22 | 2016-03-22 | Optical positioning tracking system and method based on invisible combined road sign |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201610165554.6A CN107218886B (en) | 2016-03-22 | 2016-03-22 | Optical positioning tracking system and method based on invisible combined road sign |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107218886A true CN107218886A (en) | 2017-09-29 |
| CN107218886B CN107218886B (en) | 2020-07-03 |
Family
ID=59927223
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201610165554.6A Active CN107218886B (en) | 2016-03-22 | 2016-03-22 | Optical positioning tracking system and method based on invisible combined road sign |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107218886B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108828521A (en) * | 2018-06-21 | 2018-11-16 | 重庆科技学院 | Visible light localization method and positioning system based on image combination |
| WO2019109242A1 (en) * | 2017-12-05 | 2019-06-13 | Xu Hongzhen | Systems, apparatus, and methods for identifying and tracking object based on light coding |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH03275389A (en) * | 1990-03-26 | 1991-12-06 | Agency Of Ind Science & Technol | Printed matter having discrimination mark |
| WO2008007076A1 (en) * | 2006-07-10 | 2008-01-17 | Retainagroup Limited | Vehicle identification system |
| CN201095727Y (en) * | 2007-09-26 | 2008-08-06 | 陈伟 | Digitization wireless control solar energy protrusion road mark |
| CN101550678A (en) * | 2009-04-28 | 2009-10-07 | 周良勇 | Road sign convenient to locate positions |
| CN101811481A (en) * | 2010-04-28 | 2010-08-25 | 燕山大学 | Anti-counterfeiting license plate containing optical hidden code and production and identification method thereof |
| EP2448251A2 (en) * | 2010-10-31 | 2012-05-02 | Mobileye Technologies Limited | Bundling night vision and other driver assistance systems (DAS) using near infra red (NIR) illumination and a rolling shutter |
| CN102773862A (en) * | 2012-07-31 | 2012-11-14 | 山东大学 | Quick and accurate locating system used for indoor mobile robot and working method thereof |
| CN202702247U (en) * | 2012-07-31 | 2013-01-30 | 山东大学 | Rapid and accurate positioning system used for indoor mobile robot |
| US20140099005A1 (en) * | 2012-10-09 | 2014-04-10 | Sony Corporation | Authentication apparatus, authentication method, and program |
| CN103837147A (en) * | 2014-03-13 | 2014-06-04 | 北京理工大学 | Active infrared dot-matrix type artificial road sign, intelligent body locating system and intelligent body locating method |
| CN104584035A (en) * | 2012-03-30 | 2015-04-29 | 3M创新有限公司 | Retroreflective articles having a machine-readable code |
| DE102013021101A1 (en) * | 2013-12-18 | 2015-06-18 | Abb Ag | Access control system for buildings |
| US20150317923A1 (en) * | 2012-12-11 | 2015-11-05 | 3M Innovative Properties Company | Inconspicuous optical tags and methods therefor |
| CN205068424U (en) * | 2015-08-19 | 2016-03-02 | 郑晓萌 | Motion tracking system based on stealthy code road sign |
-
2016
- 2016-03-22 CN CN201610165554.6A patent/CN107218886B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JPH03275389A (en) * | 1990-03-26 | 1991-12-06 | Agency Of Ind Science & Technol | Printed matter having discrimination mark |
| WO2008007076A1 (en) * | 2006-07-10 | 2008-01-17 | Retainagroup Limited | Vehicle identification system |
| CN201095727Y (en) * | 2007-09-26 | 2008-08-06 | 陈伟 | Digitization wireless control solar energy protrusion road mark |
| CN101550678A (en) * | 2009-04-28 | 2009-10-07 | 周良勇 | Road sign convenient to locate positions |
| CN101811481A (en) * | 2010-04-28 | 2010-08-25 | 燕山大学 | Anti-counterfeiting license plate containing optical hidden code and production and identification method thereof |
| EP2448251A2 (en) * | 2010-10-31 | 2012-05-02 | Mobileye Technologies Limited | Bundling night vision and other driver assistance systems (DAS) using near infra red (NIR) illumination and a rolling shutter |
| CN104584035A (en) * | 2012-03-30 | 2015-04-29 | 3M创新有限公司 | Retroreflective articles having a machine-readable code |
| CN102773862A (en) * | 2012-07-31 | 2012-11-14 | 山东大学 | Quick and accurate locating system used for indoor mobile robot and working method thereof |
| CN202702247U (en) * | 2012-07-31 | 2013-01-30 | 山东大学 | Rapid and accurate positioning system used for indoor mobile robot |
| US20140099005A1 (en) * | 2012-10-09 | 2014-04-10 | Sony Corporation | Authentication apparatus, authentication method, and program |
| US20150317923A1 (en) * | 2012-12-11 | 2015-11-05 | 3M Innovative Properties Company | Inconspicuous optical tags and methods therefor |
| DE102013021101A1 (en) * | 2013-12-18 | 2015-06-18 | Abb Ag | Access control system for buildings |
| CN103837147A (en) * | 2014-03-13 | 2014-06-04 | 北京理工大学 | Active infrared dot-matrix type artificial road sign, intelligent body locating system and intelligent body locating method |
| CN205068424U (en) * | 2015-08-19 | 2016-03-02 | 郑晓萌 | Motion tracking system based on stealthy code road sign |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2019109242A1 (en) * | 2017-12-05 | 2019-06-13 | Xu Hongzhen | Systems, apparatus, and methods for identifying and tracking object based on light coding |
| CN108828521A (en) * | 2018-06-21 | 2018-11-16 | 重庆科技学院 | Visible light localization method and positioning system based on image combination |
| CN108828521B (en) * | 2018-06-21 | 2022-06-07 | 重庆科技学院 | Visible light positioning method and positioning system based on image combination |
Also Published As
| Publication number | Publication date |
|---|---|
| CN107218886B (en) | 2020-07-03 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108510592B (en) | Augmented reality display method of real physical model | |
| CN106372702B (en) | Positioning identifier and positioning method thereof | |
| CN106101689B (en) | The method that using mobile phone monocular cam virtual reality glasses are carried out with augmented reality | |
| US8422777B2 (en) | Target and method of detecting, identifying, and determining 3-D pose of the target | |
| CN107016704A (en) | A kind of virtual reality implementation method based on augmented reality | |
| CN110017841A (en) | Vision positioning method and its air navigation aid | |
| CN108540542B (en) | Mobile augmented reality system and display method | |
| CN110036392A (en) | System and method for being drawn based on multiple-pass data | |
| US20160019722A1 (en) | System and Method for Defining an Augmented Reality View in a Specific Location | |
| CN103837147B (en) | Active infrared dot-matrix type artificial road sign, intelligent body locating system and intelligent body locating method | |
| CN104748746A (en) | Attitude determination and virtual reality roaming method of intelligent machine | |
| US11232636B2 (en) | Methods, devices, and systems for producing augmented reality | |
| CN109920000B (en) | Multi-camera cooperation-based dead-corner-free augmented reality method | |
| CN108564662A (en) | The method and device that augmented reality digital culture content is shown is carried out under a kind of remote scene | |
| CN111833458A (en) | Image display method and device, equipment and computer readable storage medium | |
| CN106355592A (en) | Educational toy suite and its circuit elements and electric wires identifying method thereof | |
| CN106611441A (en) | Processing method and device for three-dimensional map | |
| CN110336973A (en) | Information processing method and its device, electronic equipment and medium | |
| CN105095314A (en) | Point of interest (POI) marking method, terminal, navigation server and navigation system | |
| CN110060230A (en) | Three-dimensional scenic analysis method, device, medium and equipment | |
| CN103647960A (en) | Three-dimensional image compositing method | |
| CN107218886A (en) | A kind of optical alignment tracing system and method based on stealthy combination road sign | |
| CN115731370A (en) | Large-scene element universe space superposition method and device | |
| CN110060296A (en) | Estimate method, electronic equipment and the method and apparatus for showing virtual objects of posture | |
| EP3534338B1 (en) | Creation of a simulation scene from a specified view point |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| CB03 | Change of inventor or designer information |
Inventor after: Zheng Xiaomeng Inventor before: Zhou Kaidi Inventor before: Zheng Xiaomeng |
|
| CB03 | Change of inventor or designer information | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20180913 Address after: 100190 No. two south of Zhongguancun, Haidian District, Beijing 1 Applicant after: NATIONAL SPACE SCIENCE CENTER, CAS Address before: 100084 dormitory fourteen, Tsinghua University, Tsinghua Yuan, Haidian District, Beijing Applicant before: Zhou Kaidi |
|
| TA01 | Transfer of patent application right | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |