CN1176351C - Method and device of 3D digital imaging with dynamic multiple resolution ratio - Google Patents
Method and device of 3D digital imaging with dynamic multiple resolution ratio Download PDFInfo
- Publication number
- CN1176351C CN1176351C CNB021310963A CN02131096A CN1176351C CN 1176351 C CN1176351 C CN 1176351C CN B021310963 A CNB021310963 A CN B021310963A CN 02131096 A CN02131096 A CN 02131096A CN 1176351 C CN1176351 C CN 1176351C
- Authority
- CN
- China
- Prior art keywords
- phase
- scenery
- frequency
- space
- alliteration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Landscapes
- Length Measuring Devices By Optical Means (AREA)
Abstract
The present invention discloses a 3D digital imaging method and a device with dynamic multiresolution. The device comprises a double acoustic-optic illumination emitter, an image sensing receiver and an image processor. The illumination emitter is used for generating structured light sequences with different spatial periods or frequency and different phase movement, and the structured light sequences are projected on a digitized scene according to time sequences. The depth information of the scene is coded according to space-time sequences, and the image sensing receiver obtains digital images of structured light coded by the space-time sequences. The image processor obtains multiresolution 3D depth information through the decoding of the images of the structured light coded by the space-time sequences, namely the 3D digital images. The present invention has the advantages that the device has no mechanical motion, and the stability and the repeatability are good; the double acoustic-optic illumination emitter adopts a double light path or equal optical path symmetrical structure, and the external interference resistant performance is strong; the frequency and the phase of the spatial structured light are respectively changed and regulated in a real-time way, the space-time phase is rebuild in a hierarchical way, and the precision and the flexibility of the 3D digital imaging are high.
Description
Technical field
The present invention relates to a kind of method and device of 3 D digital imaging of dynamic multiple resolution ratio, belong to the 3 D digital imaging technology.
Background technology
Can obtain the high density depth data of scenery non-contactly based on the three-dimensional imaging technology of active laser triangulation, comprise the three-dimensional imaging technology of laser spots scanning triangulation and laser sheet scanning triangulation, for example CYBERWARE (
Www.cyberware.com) and DIGIBOT (
Www.digibotics.com) 3D digitizer of company, said method and apparatus all needs machinery or optical scanning mechanism, so the speed of three-dimensional imaging is very limited.Three-dimensional imaging technology based on fringe projection and phase measurement need not any machinery or optical scanning mechanism, and not only have the characteristics of noncontact, high density depth data but also possess the feature of whole audience Depth Information Acquistion, thereby by extensive studies and development.
The 3 D digital imaging based on fringe projection and phase measurement that proposes roughly comprises two classes at present: laser interferencefringes projection and white light optical grating projection.Striped or grating project on the scenery as the spatial carrier of single-frequency, scenery is coded in depth information among the spatial carrier of modulation, imaging device and photodetector obtain single width or several space encoder carrier wave images, by said coding carrier wave image being carried out the depth data of optics or digital decoding acquisition scenery.One of effective ways of the spatial carrier image of decode modulated are by " phase shift algorithm ", and this need produce the above phase shifts space encoder carrier wave figure of three width of cloth.Said method can only produce the striped or the grating of fixing and single spatial frequency (or cycle) mostly, being difficult to fast, (as video rate) produces phase shifts, can not be applied to the occasion of dynamic 3 D sensing, three-dimensional imaging to arbitrary shape scenery object does not have universality, and for example said method all lost efficacy when the very big variation of gradient appears in scenery.
The technical literature that can contrast has down with four pieces:
[1] patent of invention: publication number CN1205453A
[2] patent of invention: Granted publication CN1093935C.
[3]M.Rioux,“Three?dimensional?imaging?method?and?device”.US?Patent?4,627,734
[4]M.Halioua?and?V.Srinivasan,“Method?and?apparatus?for?surface?profilometry”.US?Patent4,641,972
Summary of the invention
The object of the present invention is to provide a kind of method and device of 3 D digital imaging of dynamic multiple resolution ratio.This technology is applied to dynamic 3 D sensing occasion with the illumination of the sine space carrier wave sequence of real-time generation spatial variable frequency and phase shift, and the three-dimensional imaging of the scenery object of arbitrary shape is all had universality.
For achieving the above object, the present invention is realized by following technical proposals.Employing comprises the device that alliteration optical illumination transmitter, image sensing receiver and image processor constitute, and implements the three-dimension digital imaging method to the scenery dynamic multiple resolution ratio.This method is placed on fixed angle of formation in the same plane, space with alliteration optical modulation illumination transmitter and image sensing receiver, they and illuminated scenery constitute a triangle, the line of relative position is a baseline, it is characterized in that: the illumination transmitter is with the spatial variable frequency of real-time generation and the sine space carrier wave sequence illumination scenery of phase shifts, and scenery carried out the room and time sequential coding of the degree of depth, promptly be respectively with ξ
1, ξ
2=x ξ
1... ξ
M=x ξ
M-1Different frequency and same frequency are with δ
1, δ
2δ
NOut of phase is moved the striated structure light scenery that throws light on successively, obtains the fringe intensity distribution plan of M * N coding respectively, utilizes " phase shift algorithm " to calculate the phase diagram Φ of each fringe intensity distribution plan
1, Φ
2, Φ
3Φ
MIn each phase diagram, eliminate 2 π uncertainties, thereby scenery is obtained the degree of depth picture that resolution is gone forward one by one successively, obtain the structured light of the phase shift sinusoidal carrier of room and time sequential coding, obtain multiresolution three-dimensional depth information to scenery, i.e. 3-dimensional digital picture through decoding again.
According to above-mentioned method, realize the device of the 3 D digital imaging of dynamic multiple resolution ratio, comprise alliteration optical modulation illumination transmitter, image sensing receiver and image processor, said image sensing receiver comprises that optical imaging lens and photodetector constitute, optical imaging lens can be imaging len or the lens combination that focuses distance or varifocal, the binary optical imaging system, diffraction element imaging system, micro imaging system; Said photoelectric detector can be a charge-coupled image sensor, liquid crystal device, spatial light modulation device or by digital camera; Described image processor is the combination of digital signal processor and programmable asic, also can be that general image transaction card and computer combined constitute, it is characterized in that, alliteration optical modulation illumination transmitter, it is to be made of coherent source, beam splitter, alliteration photomodulator and radio-frequency (RF) driver circuit, directional trim device, spatial filter, beam separation transducer (wedge), condenser lens, microcobjective; Its driving radiofrequency signal is that frequency is identical and have the drive signal of 0~2 π controllable phase difference and a real-time change frequency.
The invention has the advantages that: (1) total system adopts structure of whole solid state, does not have any mechanical motion, and therefore stability and repeated performance are fine; (2) " light path altogether " or " aplanatism " symmetrical structure are adopted in the design of alliteration optical modulation illumination transmitter, influence that therefore can the make-up air disturbance; (3) the striated structure light sequence of different azimuth be can obtain by plural alliteration optical modulation illumination transmitter and the various combination of image sensing receiver on the locus, the precision and the dirigibility of 3 D digital imaging increased thus greatly.
Description of drawings:
Fig. 1 is the principle of work synoptic diagram of the inventive method
Fig. 2 is the structured flowchart of alliteration light (AOM) modulation illumination transmitter
Fig. 3 is that synoptic diagram is put in the position when adopting an alliteration optical modulation illumination transmitter and two image sensing receivers to scenery irradiation imaging.
Fig. 4 is the vertical view of Fig. 3 device when same plane and straight line is put.
Fig. 5 is Fig. 3 device at the vertical view that same plane is triangular in shape when putting.
Fig. 6 adopts an image sensing receiver and two alliteration optical modulation illumination transmitters that scenery irradiation image space is put synoptic diagram.
Fig. 7 is the vertical view of Fig. 6 device when same plane and straight line is put.
Fig. 8 is Fig. 6 device at the vertical view that same plane is triangular in shape when putting.
101 is alliteration optical modulation illumination transmitters among the figure, the 102nd, image sensing receiver, the 103rd, image processor, the 104th, scenery, the 201st, coherent source, the 202nd, beam splitter, 203 and 204 is acousto-optic modulators, 205 and 206 is directional trim device (wedges), the 207th, the double space wave filter, the 208th, bundling device, the 209th, condenser lens, the 210th, microcobjective.
Embodiment
Below in conjunction with accompanying drawing embodiments of the present invention are done and to be explained.
As shown in Figure 1: same plane forms a fixing angle in the space for alliteration optical modulation illumination transmitter and image sensing receiver, and the relative position of the optical axis of luminaire and imaging optical axis and illumination transmitter and image sensing receiver constitutes a triangle.The frequency of the radio-frequency (RF) driving signal of alliteration optical modulation illumination transmitter and phase place are by computing machine or the digital signal processor and the D/A conversion and control of image processor.Line between illumination transmitter and the image sensing receiver relative position is defined as baseline, and alliteration optical modulation illumination transmitter and image sensing receiver together are placed on the place ahead of scenery to be measured.Alliteration optical modulation illumination transmitter produces the Space Time striated structure light sequence of spatial variable frequency and different phase-shift phases, and project on the scenery, scene depth information carries out phase encode and the result is sent to image processor the Space Time striated structure light sequence of spatial variable frequency and different phase-shift phases.Following steps are adopted in concrete operations: (1) uses space low-limit frequency ξ
1(ξ for example
1Be the inverse in a spatial fringe cycle) and out of phase move (δ
1, δ
2..., δ
N, N 〉=3) the striated structure light tested scenery that throws light on successively, obtain the fringe intensity distribution plan of the space encoding of each width of cloth phase shifts by imaging system and image sensing receiver sequential, utilize " phase shift algorithm " to calculate the phase diagram φ that fringe intensity distributes
1(not having 2 π uncertainties) can calculate definite non-integer fringe order by said phase diagram according to above-mentioned said triangle relation, further obtains the rough profile of scenery thus; (2) spatial frequency is adjusted to ξ
2=X ξ
1, wherein X is a positive integer, uses frequency ξ
2The tested scenery of striated structure optical illumination with out of phase is moved calculates second phase diagram φ according to above-mentioned said method
2, can eliminate 2 π uncertainties with the phase retrieval algorithm, the phase diagram that obtains will make φ
2Resolution improve X doubly than the resolution of φ 1 because the preceding ξ that once uses
1The uncertain result of 2 π who has told can inherit and use ξ
2In the process of measuring; (3) in case use ξ
2It is uncertain to tell 2 π, then can proceed to the 3rd measurement of differentiating level, so repeats, and can improve constantly the resolution of scenery profile three-dimensional imaging.
Fig. 2 is a specific embodiment of alliteration light (AOM) modulation illumination transmitter 101.The light beam that coherent source 201 produces forms two bundle directional lights through Amici prism 202 backs, incide the input end of two acousto-optic modulators 203 and 204 respectively with the Bragg angle, output terminal at two acousto-optic modulators (AOM) takes out positive first-order diffraction light respectively, readjusting through two wedges 205 and 206 is two bundle directional lights, enter bundling device 208 through spatial filter again and change its spacing, obtain the microstructure of interference field then through condenser lens 209, the sine streak that obtains amplifying through micro-amplifying unit 210 distributes again.The frequency of the radio-frequency (RF) driving signal by two acousto-optic modulators of computer control (AOM) can two AOM of sequence change first-order diffraction diffraction of light angle, control the phase shifts that first-order diffraction light that the phase differential of the radio-frequency (RF) driving signal of two acousto-optic modulators (AOM) can two AOM of sequence change produces each other, thereby produce the Space Time striated structure light sequence of spatial variable frequency and different phase-shift phases.Fringe spatial frequency and phase shift momentum all can change by the bandwidth that changes radiofrequency signal, the delay that changes sonic propagation can realize that therefore the Space Time frequency conversion bar graph illumination that produces with said method can be much higher than video rate (about 40 milliseconds of every frame) in the millisecond magnitude.
Fig. 3 is another embodiment of the present invention, and it is to adopt an alliteration optical modulation photograph transmitter and two image sensing receivers that scenery is shone imaging device location arrangements synoptic diagram, and Fig. 4 is the vertical view of two kinds of different layouts in space of Fig. 3 with Fig. 5.Use an alliteration light (AOM) modulation illumination transmitter 101 and two image sensing receivers 102 in this embodiment, produce the Space Time striated structure light sequence of also projecting space frequency conversion and different phase-shift phases by said alliteration light (AOM) modulation illumination transmitter, fringe spatial frequency and phase shift momentum all can change by the bandwidth that changes radiofrequency signal, obtain the light intensity distributions of the Space Time sequence bar graph after the process scene depth information is encoded respectively successively or simultaneously from both sides by said two image sensing receivers, and the result is sent to image processor the light intensity distributions of said coding Space Time sequence bar graph is carried out phase decoding, computing machine with two groups of phase decoding data fusion of said two receivers of correspondence together result to the end.Use two image sensing receivers can avoid since the scenery degree of depth produce block phenomenon, increased the field range of three-dimensional sensing.
Fig. 6 is another embodiment of the present invention, and it is to adopt an image sensing receiver and two alliteration rayed transmitters that scenery is shone imaging device location arrangements synoptic diagram, and Fig. 7 and 8 is their vertical views two kinds of different layouts in space.Use two alliteration light (AOM) modulation illumination transmitter 101 and an image sensing receiver 102 in this embodiment, produce and throw the identical or different spatial variable frequency of two kinds of dimensional orientations and spatial frequency and the Space Time striated structure light sequence of phase shift from both sides by said two alliteration light (AOM) modulation illumination transmitter, fringe spatial frequency and phase shift momentum all can change by the bandwidth that changes radiofrequency signal, obtain successively or simultaneously through behind the scene depth information coding from a side by a said image sensing receiver, the light intensity distributions of the Space Time sequence bar graph of corresponding said two alliteration light (AOM) modulation illumination transmitter, and the result is sent to image processor the light intensity distributions of said coding Space Time sequence bar graph is carried out phase decoding successively or simultaneously, computing machine with two groups of phase decoding data fusion of corresponding two transmitters together result to the end.Use two alliteration light (AOM) modulation illumination transmitter can produce dimensional orientation and the identical or different sequence Space-Time Structured Illumination of frequency, can increase the sensitivity and the universality of 3 D digital imaging greatly.
Claims (2)
1, a kind of method of 3 D digital imaging of dynamic multiple resolution ratio, this method is placed on fixed angle of formation in the same plane, space with alliteration optical modulation illumination transmitter and image sensing receiver, they and illuminated scenery constitute a triangle, the line of relative position is a baseline, it is characterized in that: the illumination transmitter is with the spatial variable frequency of real-time generation and the sine space carrier wave sequence illumination scenery of phase shifts, and scenery carried out the room and time sequential coding of the degree of depth, promptly be respectively with ξ
1, ξ
2=x ξ
1... ξ
M=x ξ
M-1Different frequency and same frequency are with δ
1, δ
2δ
NOut of phase is moved the striated structure light scenery that throws light on successively, obtains the fringe intensity distribution plan of M * N coding respectively, utilizes " phase shift algorithm " to calculate the phase diagram Φ of each fringe intensity distribution plan
1, Φ
2, Φ
3Φ
MIn each phase diagram, eliminate 2 π uncertainties, thereby scenery is obtained the degree of depth picture that resolution is gone forward one by one successively, obtain the structured light of the phase shift sinusoidal carrier of room and time sequential coding, obtain multiresolution three-dimensional depth information to scenery, i.e. 3-dimensional digital picture through decoding again.
2, in accordance with the method for claim 1 a kind of, realize the device of the 3 D digital imaging of dynamic multiple resolution ratio, this device comprises that alliteration optical modulation illumination transmitter, image sensing receiver and image processor constitute, said image sensing receiver comprises that optical imaging lens and photodetector constitute, optical imaging lens can be imaging len or the lens combination that focuses distance or varifocal, the binary optical imaging system, diffraction element imaging system, micro imaging system; Said photoelectric detector can be a charge-coupled image sensor, liquid crystal device, spatial light modulation device or digital camera; Described image processor is the combination of digital signal processor and programmable asic, also can be that general image transaction card and computer combined constitute, it is characterized in that: alliteration optical modulation illumination transmitter, it is to be made of coherent source, beam splitter, alliteration photomodulator and radio-frequency (RF) driver circuit, directional trim device, spatial filter, beam separation transducer, condenser lens, microcobjective; Its driving radiofrequency signal is that frequency is identical and have the drive signal of 0~2 π controllable phase difference and a real-time change frequency.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CNB021310963A CN1176351C (en) | 2002-10-09 | 2002-10-09 | Method and device of 3D digital imaging with dynamic multiple resolution ratio |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CNB021310963A CN1176351C (en) | 2002-10-09 | 2002-10-09 | Method and device of 3D digital imaging with dynamic multiple resolution ratio |
Publications (2)
Publication Number | Publication Date |
---|---|
CN1414420A CN1414420A (en) | 2003-04-30 |
CN1176351C true CN1176351C (en) | 2004-11-17 |
Family
ID=4746566
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CNB021310963A Expired - Fee Related CN1176351C (en) | 2002-10-09 | 2002-10-09 | Method and device of 3D digital imaging with dynamic multiple resolution ratio |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN1176351C (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101512599B (en) * | 2006-09-21 | 2012-07-18 | 汤姆森特许公司 | Method and system for obtaining three-dimensional model |
CN107014810A (en) * | 2017-04-26 | 2017-08-04 | 苏州迪美格智能科技有限公司 | A kind of multi-frequency feature structure optical arrays scaling down processing system and method |
Families Citing this family (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE102004028806B3 (en) * | 2004-06-15 | 2005-12-29 | Infineon Technologies Ag | Receiver for a wireless communication system |
KR101058032B1 (en) * | 2005-04-01 | 2011-08-19 | 삼성전자주식회사 | Stereoscopic Imager |
JP4986679B2 (en) * | 2007-03-29 | 2012-07-25 | 学校法人福岡工業大学 | Non-stationary object three-dimensional image measurement apparatus, three-dimensional image measurement method, and three-dimensional image measurement program |
CN101349551B (en) * | 2007-07-20 | 2012-08-29 | 通用电气公司 | Configuration measuring apparatus and operation method thereof |
CN101256672B (en) * | 2008-03-21 | 2011-10-12 | 北京中星微电子有限公司 | Object image depth restruction apparatus based on video camera apparatus as well as projecting apparatus thereof |
US9870068B2 (en) | 2010-09-19 | 2018-01-16 | Facebook, Inc. | Depth mapping with a head mounted display using stereo cameras and structured light |
TWI403784B (en) * | 2010-12-31 | 2013-08-01 | Pai Chi Li | Photoacoustic imaging system, coded laser emitting apparatus and photoacoustic signal receiving apparatus |
US9229580B2 (en) * | 2012-08-03 | 2016-01-05 | Technokey Company Limited | System and method for detecting object in three-dimensional space using infrared sensors |
US20140307055A1 (en) | 2013-04-15 | 2014-10-16 | Microsoft Corporation | Intensity-modulated light pattern for active stereo |
US9443310B2 (en) * | 2013-10-09 | 2016-09-13 | Microsoft Technology Licensing, Llc | Illumination modules that emit structured light |
BR112016009202A8 (en) | 2013-10-23 | 2020-03-24 | Oculus Vr Llc | apparatus and method for generating a structured light pattern |
CN104482877B (en) * | 2014-12-03 | 2017-02-01 | 深圳大学 | Motion compensation method and system in three-dimensional imaging of dynamic object |
WO2016154218A1 (en) | 2015-03-22 | 2016-09-29 | Oculus Vr, Llc | Depth mapping with a head mounted display using stereo cameras and structured light |
CN105180828A (en) * | 2015-08-21 | 2015-12-23 | 西安交通大学 | Multi-beam laser intersection scanning contourgraph |
CN105571496B (en) * | 2016-01-13 | 2018-01-23 | 南京卓实电气有限责任公司 | A kind of live detection electrical apparatus external insulation configuration parameter measuring method |
TWI647661B (en) * | 2017-08-10 | 2019-01-11 | 緯創資通股份有限公司 | Image depth sensing method and image depth sensing device |
US10551614B2 (en) * | 2017-08-14 | 2020-02-04 | Facebook Technologies, Llc | Camera assembly with programmable diffractive optical element for depth sensing |
US10586342B2 (en) | 2017-08-31 | 2020-03-10 | Facebook Technologies, Llc | Shifting diffractive optical element for adjustable depth sensing resolution |
CN107734283A (en) * | 2017-09-11 | 2018-02-23 | 广东欧珀移动通信有限公司 | Picture processing method, device and the storage medium of Video chat |
CN107707839A (en) * | 2017-09-11 | 2018-02-16 | 广东欧珀移动通信有限公司 | Image processing method and device |
CN107505753B (en) | 2017-09-15 | 2020-06-05 | 京东方科技集团股份有限公司 | Array substrate and display device |
CN111758013A (en) * | 2017-10-27 | 2020-10-09 | 精密自动化股份有限公司 | System and method for collecting data and information of surface properties of an object |
WO2020056059A1 (en) * | 2018-09-11 | 2020-03-19 | Tetravue, Inc. | Electro-optic modulator and methods of using and manufacturing same for three-dimensional imaging |
-
2002
- 2002-10-09 CN CNB021310963A patent/CN1176351C/en not_active Expired - Fee Related
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101512599B (en) * | 2006-09-21 | 2012-07-18 | 汤姆森特许公司 | Method and system for obtaining three-dimensional model |
CN107014810A (en) * | 2017-04-26 | 2017-08-04 | 苏州迪美格智能科技有限公司 | A kind of multi-frequency feature structure optical arrays scaling down processing system and method |
Also Published As
Publication number | Publication date |
---|---|
CN1414420A (en) | 2003-04-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN1176351C (en) | Method and device of 3D digital imaging with dynamic multiple resolution ratio | |
CA2538162C (en) | High speed multiple line three-dimensional digitization | |
US9170098B2 (en) | Device and method using a spatial light modulator to find 3D coordinates of an object | |
US6084712A (en) | Three dimensional imaging using a refractive optic design | |
US6438272B1 (en) | Method and apparatus for three dimensional surface contouring using a digital video projection system | |
EP3500820A1 (en) | Structured light projector | |
US10706570B2 (en) | System and method to acquire the three-dimensional shape of an object using a moving patterned substrate | |
CA2319898C (en) | Position encoding optical device and method | |
US20060209193A1 (en) | Aperture coded camera for three dimensional imaging | |
CN109798845A (en) | A kind of method and apparatus that the reconstruction accuracy based on laser raster scan is promoted | |
CN108680142A (en) | A kind of dimensional visual measurement system projecting principle based on high speed triangular wave striped | |
CA2397095A1 (en) | Apparatus and methods for surface contour measurement | |
CN113465545A (en) | Three-dimensional measurement system based on high-speed LED array and measurement method thereof | |
CN1595057A (en) | Multiple differentiation three-dimensional digital imaging method based on space orthogonal striped projection | |
KR20190107090A (en) | Hologram Light Detection and Measurement | |
Zhang et al. | Hybrid structured light for scalable depth sensing | |
CN113701662B (en) | Vibrating mirror-based structured light coding fringe grating pattern generation method | |
Sansoni et al. | 3d imager for dimensional gauging of industrial workpieces: state of the art of the development of a robust and versatile system | |
JPH10232204A (en) | Device for measuring refractive index | |
Harding | Phase-grating use for slope discrimination in moire contouring | |
CN1206513C (en) | Configurable dynamic 3D digit sensing device for sequential lighting with frequency variable fringes as well as its procedue | |
Harding et al. | High-speed moire contouring methods analysis | |
CN1112567C (en) | Sinusoidal light field generating method with binary coding templet | |
Vuylsteke et al. | Image Sensors for Real-Time 3D Acquisition: Part-1-Three Dimensional Image Acquisition | |
Deng et al. | Fast 3-D Measurement Based on Spatial-temporal Speckle Projection |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
C06 | Publication | ||
PB01 | Publication | ||
C10 | Entry into substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
C14 | Grant of patent or utility model | ||
GR01 | Patent grant | ||
C19 | Lapse of patent right due to non-payment of the annual fee | ||
CF01 | Termination of patent right due to non-payment of annual fee |