CN106023170A - Binocular 3D distortion rectification method based on GPU - Google Patents

Binocular 3D distortion rectification method based on GPU Download PDF

Info

Publication number
CN106023170A
CN106023170A CN201610318480.5A CN201610318480A CN106023170A CN 106023170 A CN106023170 A CN 106023170A CN 201610318480 A CN201610318480 A CN 201610318480A CN 106023170 A CN106023170 A CN 106023170A
Authority
CN
China
Prior art keywords
gpu
left eyes
picture
robj
lobj
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201610318480.5A
Other languages
Chinese (zh)
Other versions
CN106023170B (en
Inventor
余刚
王高飞
李广群
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Chengdu Sobey Digital Technology Co Ltd
Original Assignee
Chengdu Sobey Digital Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Chengdu Sobey Digital Technology Co Ltd filed Critical Chengdu Sobey Digital Technology Co Ltd
Priority to CN201610318480.5A priority Critical patent/CN106023170B/en
Publication of CN106023170A publication Critical patent/CN106023170A/en
Application granted granted Critical
Publication of CN106023170B publication Critical patent/CN106023170B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration
    • G06T5/50Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00Indexing scheme for image analysis or image enhancement
    • G06T2207/10Image acquisition modality
    • G06T2207/10016Video; Image sequence
    • G06T2207/10021Stereoscopic video; Stereoscopic image sequence

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Image Analysis (AREA)

Abstract

The invention discloses a binocular 3D distortion rectification method based on a GPU. The method comprises steps of segmenting left and right eye frames by using the GPU, acquiring a main foreground object, and placing objects near a central point on a zero plane; screening and registering the characteristic points of the foreground object; rotating and scaling the left and right eye frames; regulating a parallax error according to translation quantity. Compared with original left and right eye original frames, the processed left and right eye frames are decreased in image edge distortion. A problem is solved that the left and right eye frames are not in the same plane previously. The letter box of the image edge does not exist fundamentally. The parallax error of the left and right eye frames is reduced. The target frames are significantly improved in watching comfortableness than the original frames.

Description

A kind of binocular 3D distortion correction method based on GPU processor
Technical field
The present invention relates to a kind of binocular 3D distortion correction method based on GPU processor.
Background technology
In recent years, the micro display technology with liquid crystal on silicon, silica-based OLED as Typical Representative is developed rapidly, meanwhile, 3D imaging and Display Technique are developed rapidly the most in the near future, and corresponding product is at aspects such as military affairs, field detection, medical science The most universal, particularly benefit from large screen flat plate display (display panels, PDP display floater) and popularize rapidly, 3D imaging device in civil area is the most of common occurrence.Research according to the 3D image disparity principle to human eye draws, binocular 3D The most real 3D imaging means of imaging technique.
Summary of the invention
It is an object of the invention to overcome the deficiencies in the prior art, it is provided that a kind of binocular 3D distortion correction side based on GPU processor Method, uses GPU to split right and left eyes picture, obtains main foreground object, central point adnexa object is placed in zero plane; Foreground object is carried out characteristic point screening registration;Right and left eyes picture is rotated and scales correction;Parallax is regulated according to translational movement.
It is an object of the invention to be achieved through the following technical solutions: a kind of binocular 3D distortion correction side based on GPU processor Method, it comprises the steps:
S1: right and left eyes picture LImg1 and RImg1 of video camera shooting is carried out camera calibration, calculates the inside of video camera Parameter, obtains radial distortion and tangential distortion information, and computing formula is as follows:
x u = x d ( 1 + k 1 r d 2 ) + k 3 ( 3 x d 2 + y d 2 ) + 2 k 4 x d y d
y u = y d ( 1 + k 2 r d 2 ) + k 4 ( x d 2 + 3 y d 2 ) + 2 k 3 x d y d
Wherein,(xd,yd) it is preferred view coordinate, (xu,yu) it is actual projection coordinate, (k1,k2) for imaging Machine radial distortion, (k3,k4) it is tangential distortion;
S2: be up in GPU by right and left eyes picture LImg1 and RImg1, uses the formula that S1 obtains, pixel-by-pixel to original left Right eye picture carries out the distortion correction of inner parameter, obtains LImg2 and RImg2;
S3: right and left eyes picture LImg2 and RImg2 after distortion correction is carried out in GPU edge and carries contours extract and image divides Cut, obtain main foreground object sequence LObj and the RObj of right and left eyes picture respectively;
S4: main foreground object sequence LObj and RObj to right and left eyes picture carry out feature extraction in GPU, respectively obtain The rotation of LObj and RObj, Pan and Zoom information;
S5: calculate LObj and RObj average translation amount, and then calculate shooting 3D video pictures video camera between relative away from From;
S6: average according to LObj and RObj rotates and scalability information, carries out right and left eyes picture LImg2 and RImg2 in opposite directions Rotation and scaling correction, thus by right and left eyes picture rotate to same angle and ensure the main foreground object of right and left eyes picture big Little unanimously, obtain right and left eyes correction coupling after picture LImg3 and RImg3;
S7: according to correction result screen LImg3 and RImg3, calculate the main front scenery that distance central point in right and left eyes is nearest Body is as video camera zero plane, and foundation right and left eyes optimal parallax empirical value between 2% to 3%, during in conjunction with 3D video capture The distance of right and left eyes video camera, with reference to the average translation amount of LObj and RObj, is adjusted the parallax of right and left eyes picture, After obtain right and left eyes picture LImgD and RImgD that be disposed.
Camera calibration described in step S1 uses the image containing chessboard or grid lines carry out automatic Calibration or use manually side Formula is selected a plurality of parallel lines and is demarcated.
In step S3, image segmentation uses the image segmentation algorithm that the applicable GPU of limited Clustering calculates, and this algorithm includes as follows Process step:
A. pending image is up in GPU be denoted as ImgA, to each pixel in ImgA from the upper left corner to the lower right corner according to Ranks order starts assignment successively from 0, will be denoted as InfoA in this information upstream to GPU;
B. ImgA image is divided into the subimage block of 8*8, utilize the concurrency of GPU simultaneously to subimage block according to color and Range information carries out sub-cluster, and by cluster result record in InfoA;
The subimage block of the ImgA c. previous step obtained up and down four be one group, regard new subimage block as, according to InfoA is merged by subimage block boundary pixel value information;
The new subimage block of the ImgA obtained after being d. combined carries out sub-cluster according to color and range information, and by cluster result Record is in InfoA;
E. it return back to step c order perform, until whole image ImgA only one of which subimage block, then stop cluster, perform Step f;
F. information search 8 neighborhood to InfoA record, carries out merger, then final InfoA note by isolated point or fritter Record is exactly the image segmentation result of ImgA.
The SURF algorithm using GPU to optimize in step S4, extracts the characteristic point of Lobj and Robj, uses Euclidean distance to calculate Obtain the similarity of characteristic point, be ranked up according to similarity, take front 1/2nd characteristic points and registrate, calculated by RANSAC The characteristic point of pairing is tested screening by method, calculates the rotation of Lobj and Robj, Pan and Zoom information.
The object that during described calculating scalability information, selected distance picture central point is nearest calculates as standard, accelerates scaling letter The calculating of breath.
The invention has the beneficial effects as follows: the invention provides a kind of binocular 3D distortion correction method based on GPU processor, use Right and left eyes picture is split by GPU, obtains main foreground object, and central point adnexa object is placed in zero plane;To front scenery Body carries out characteristic point screening registration;Right and left eyes picture is rotated and scales correction;Parallax is regulated according to translational movement.
Accompanying drawing explanation
Fig. 1 is the flow chart of the inventive method;
Fig. 2 is binocular 3D camera imaging model schematic diagram.
Detailed description of the invention
Technical scheme is described in further detail below in conjunction with the accompanying drawings, but protection scope of the present invention is not limited to following institute State.
Being illustrated in figure 2 binocular 3D camera imaging model schematic diagram, in Fig. 1, C1 and C2 is the angle point of right and left eyes video camera Position, (x1, y1, z1) and (x2, y2, z2) be respectively right and left eyes video camera camera coordinates system, (x1, OI 1, y1) and (x2, OI2, y2) is respectively the image plane of right and left eyes video camera, and the point (Xw, Yw, Zw) in world coordinate system is at right and left eyes video camera The projection of image plane is respectively (X1, Y1) and (X2, Y2).
Technical scheme, based on Fig. 2 institute representation model, is illustrated.Technical scheme realizes, by following Several steps complete.
As it is shown in figure 1, a kind of binocular 3D distortion correction method based on GPU processor, it comprises the steps:
S1: input color format be left eye raw frames LImg1 of BGRA and right eye raw frames RImg1 to internal memory, to shooting Right and left eyes picture LImg1 and RImg1 of machine shooting carries out camera calibration, calculates the inner parameter of video camera, obtains radially Distortion and tangential distortion information, camera calibration uses the image containing chessboard or grid lines carry out automatic Calibration or use manually side Formula is selected a plurality of parallel lines and is demarcated, and computing formula is as follows:
x u = x d ( 1 + k 1 r d 2 ) + k 3 ( 3 x d 2 + y d 2 ) + 2 k 4 x d y d
y u = y d ( 1 + k 2 r d 2 ) + k 4 ( x d 2 + 3 y d 2 ) + 2 k 3 x d y d
Wherein,(xd,yd) it is preferred view coordinate, (xu,yu) it is actual projection coordinate, (k1,k2) for imaging Machine radial distortion, (k3,k4) it is tangential distortion;
S2: be up in GPU by right and left eyes picture LImg1 and RImg1, uses the formula that S1 obtains, pixel-by-pixel to original left Right eye picture LImg1 and RImg1 carries out the distortion correction of inner parameter, the right and left eyes picture LImg2 after being corrected and RImg2;
S3: use the GPU image segmentation algorithm of limited clustering method to right and left eyes picture LImg2 and RImg2 after distortion correction GPU is carried out edge carry contours extract and image segmentation, respectively obtain right and left eyes picture main foreground object sequence LObj and RObj;
S4: right and left eyes foreground object sequence LObj and RObj that step 3 is obtained by use SURF algorithm carry out feature extraction, To right and left eyes foreground object characteristic of correspondence point, re-use Euclidean distance and be calculated characteristic point similarity, and arrange according to similarity Sequence, before taking similarity afterwards, the characteristic point of 1/2nd registrates.It is then used by RANSAC algorithm the characteristic point of pairing is carried out Inspection screening, finally calculates right and left eyes foreground object sequence LObj and the rotation of RObj, Pan and Zoom information;Calculate contracting The object that when putting information, selected distance picture central point is nearest calculates as standard, accelerates the calculating of scalability information;
S5: according to right and left eyes foreground object sequence LObj and the rotation of RObj, Pan and Zoom information, is calculated average rotation Turn, average translation and average scalability information, calculate the binocular camera of shooting right and left eyes raw frames according to average translation information Between relative distance;
S6: reversely rotate according to right and left eyes picture LImg2 and RImg2 after distortion correction and scale, thus by left and right Eye picture rotates to identical angle position and ensure that the main foreground object of right and left eyes picture is in the same size, obtains right and left eyes school Picture LImg3 and RImg3 after just mating;
S7: according to correction result screen LImg3 and RImg3, calculate the main front scenery that distance central point in right and left eyes is nearest Body as video camera zero plane, then according to right and left eyes disparity range 2% to 3% empirical value, integrating step 5 calculates Distance between the binocular camera of shooting right and left eyes video, with reference to right and left eyes foreground object sequence LObj and the average translation of RObj Amount, is adjusted right and left eyes picture parallax, finally obtains right and left eyes picture LImgD and RImgD being disposed.
By final right and left eyes picture LImgD and RImgD from GPU up-downgoing to right and left eyes internal storage data, and internal storage data is made For the purpose of output return.
In step S3, image segmentation uses the image segmentation algorithm that the applicable GPU of limited Clustering calculates.Describe below this Pending image is up in GPU be denoted as ImgA by the process step of algorithm: a., to each pixel in ImgA from the upper left corner Start successively assignment according to ranks order from 0 to the lower right corner, this information upstream to GPU will be denoted as InfoA;B. by ImgA Image is divided into the subimage block of 8*8, utilizes the concurrency of GPU according to color and range information, subimage block to be carried out son simultaneously Cluster, and by cluster result record in InfoA;The subimage block of the ImgA c. previous step obtained four up and down It is one group, regards new subimage block as, according to subimage block boundary pixel value information (color and distance), InfoA is closed And;The new subimage block of the ImgA obtained after being d. combined carries out sub-cluster according to color and range information, and by cluster result Record is in InfoA;E. it return back to step c order perform, until whole image ImgA only one of which subimage block, then stop Only cluster, performs step f;F. information search 8 neighborhood to InfoA record, carries out merger by isolated point or fritter, Then what final InfoA recorded is exactly the image segmentation result of ImgA.
Right and left eyes picture image marginal distortion after the binocular 3D distortion correction method of the present invention is corrected reduces a lot, the most original Right and left eyes image not have also been obtained solution in conplane problem, then image border problem of black borders is also substantially not present, Finally contrasting right and left eyes picture parallax again, image parallactic also reduces, and purpose picture is compared raw frames and has been obviously improved viewing Comfort level.

Claims (5)

1. a binocular 3D distortion correction method based on GPU processor, it is characterised in that: it comprises the steps:
S1: right and left eyes picture LImg1 and RImg1 of video camera shooting is carried out camera calibration, calculates the inside of video camera Parameter, obtains radial distortion and tangential distortion information, and computing formula is as follows:
x u = x d ( 1 + k 1 r d 2 ) + k 3 ( 3 x d 2 + y d 2 ) + 2 k 4 x d y d
y u = y d ( 1 + k 2 r d 2 ) + k 4 ( x d 2 + 3 y d 2 ) + 2 k 3 x d y d
Wherein, For preferred view coordinate, (xu,yu) it is actual projection coordinate, (k1,k2) for imaging Machine radial distortion, (k3,k4) it is tangential distortion;
S2: be up in GPU by right and left eyes picture LImg1 and RImg1, uses the formula that S1 obtains, pixel-by-pixel to original left Right eye picture carries out the distortion correction of inner parameter, obtains LImg2 and RImg2;
S3: right and left eyes picture LImg2 and RImg2 after distortion correction is carried out in GPU edge and carries contours extract and image divides Cut, obtain main foreground object sequence LObj and the RObj of right and left eyes picture respectively;
S4: main foreground object sequence LObj and RObj to right and left eyes picture carry out feature extraction in GPU, respectively obtain The rotation of LObj and RObj, Pan and Zoom information;
S5: calculate LObj and RObj average translation amount, and then calculate shooting 3D video pictures video camera between relative away from From;
S6: average according to LObj and RObj rotates and scalability information, carries out right and left eyes picture LImg2 and RImg2 in opposite directions Rotation and scaling correction, thus by right and left eyes picture rotate to same angle and ensure the main foreground object of right and left eyes picture big Little unanimously, obtain right and left eyes correction coupling after picture LImg3 and RImg3;
S7: according to correction result screen LImg3 and RImg3, calculate the main front scenery that distance central point in right and left eyes is nearest Body is as video camera zero plane, and foundation right and left eyes optimal parallax empirical value between 2% to 3%, during in conjunction with 3D video capture The distance of right and left eyes video camera, with reference to the average translation amount of LObj and RObj, is adjusted the parallax of right and left eyes picture, After obtain right and left eyes picture LImgD and RImgD that be disposed.
A kind of binocular 3D distortion correction method based on GPU processor the most according to claim 1, it is characterised in that: Camera calibration described in step S1 uses the image containing chessboard or grid lines carry out automatic Calibration or use manual mode choosing Go out a plurality of parallel lines to demarcate.
A kind of binocular 3D distortion correction method based on GPU processor the most according to claim 1, it is characterised in that: In step S3, image segmentation uses the image segmentation algorithm that the applicable GPU of limited Clustering calculates, and this algorithm includes following flow process Step:
A. pending image is up in GPU be denoted as ImgA, to each pixel in ImgA from the upper left corner to the lower right corner according to Ranks order starts assignment successively from 0, will be denoted as InfoA in this information upstream to GPU;
B. ImgA image is divided into the subimage block of 8*8, utilize the concurrency of GPU simultaneously to subimage block according to color and Range information carries out sub-cluster, and by cluster result record in InfoA;
The subimage block of the ImgA c. previous step obtained up and down four be one group, regard new subimage block as, according to InfoA is merged by subimage block boundary pixel value information;
The new subimage block of the ImgA obtained after being d. combined carries out sub-cluster according to color and range information, and by cluster result Record is in InfoA;
E. it return back to step c order perform, until whole image ImgA only one of which subimage block, then stop cluster, perform Step f;
F. information search 8 neighborhood to InfoA record, carries out merger, then final InfoA note by isolated point or fritter Record is exactly the image segmentation result of ImgA.
A kind of binocular 3D distortion correction method based on GPU processor the most according to claim 1, it is characterised in that: The SURF algorithm using GPU to optimize in step S4, extracts the characteristic point of Lobj and Robj, uses Euclidean distance to be calculated The similarity of characteristic point, is ranked up according to similarity, takes front 1/2nd characteristic points and registrate, by RANSAC algorithm pair The characteristic point of pairing is tested screening, calculates the rotation of Lobj and Robj, Pan and Zoom information.
A kind of binocular 3D distortion correction method based on GPU processor the most according to claim 4, it is characterised in that: The object that during described calculating scalability information, selected distance picture central point is nearest calculates as standard, accelerates scalability information Calculate.
CN201610318480.5A 2016-05-13 2016-05-13 A kind of binocular 3D distortion correction methods based on GPU processors Active CN106023170B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610318480.5A CN106023170B (en) 2016-05-13 2016-05-13 A kind of binocular 3D distortion correction methods based on GPU processors

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610318480.5A CN106023170B (en) 2016-05-13 2016-05-13 A kind of binocular 3D distortion correction methods based on GPU processors

Publications (2)

Publication Number Publication Date
CN106023170A true CN106023170A (en) 2016-10-12
CN106023170B CN106023170B (en) 2018-10-23

Family

ID=57100077

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610318480.5A Active CN106023170B (en) 2016-05-13 2016-05-13 A kind of binocular 3D distortion correction methods based on GPU processors

Country Status (1)

Country Link
CN (1) CN106023170B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107154027A (en) * 2017-04-17 2017-09-12 深圳大学 Compensation method and device that a kind of fault image restores
CN107481267A (en) * 2017-08-14 2017-12-15 华南理工大学 A kind of shooting projection interactive system and method based on binocular vision
CN107492080A (en) * 2017-09-25 2017-12-19 天津智慧视通科技有限公司 Exempt from calibration easily monocular lens image radial distortion antidote
CN109842756A (en) * 2017-11-28 2019-06-04 东莞市普灵思智能电子有限公司 A kind of method and system of lens distortion correction and feature extraction

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104010178A (en) * 2014-06-06 2014-08-27 深圳市墨克瑞光电子研究院 Binocular image parallax adjusting method and device and binocular camera
US20160105662A1 (en) * 2014-10-08 2016-04-14 Samsung Display Co., Ltd. Three-dimensional glasses and method of driving the same

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104010178A (en) * 2014-06-06 2014-08-27 深圳市墨克瑞光电子研究院 Binocular image parallax adjusting method and device and binocular camera
US20160105662A1 (en) * 2014-10-08 2016-04-14 Samsung Display Co., Ltd. Three-dimensional glasses and method of driving the same

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
XIE RUOBING 等: "The Research on Binocular Stereo Video Imaging and Display System Based on Low-light CMOS", 《APPLIED OPTICS AND PHOTONICS CHINA》 *
刘春群 等: "基于特征点图像配准方法的应用研究", 《科技与企业》 *
咋了啦: "相机标定原理-标定原理介绍", 《CSDN》 *
张开玉: "基于改进SURF的双目立体视觉三维重建", 《中国优秀硕士学位论文全文数据库 信息科技辑(月刊)》 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107154027A (en) * 2017-04-17 2017-09-12 深圳大学 Compensation method and device that a kind of fault image restores
CN107154027B (en) * 2017-04-17 2020-07-07 深圳大学 Compensation method and device for restoration of distorted image
CN107481267A (en) * 2017-08-14 2017-12-15 华南理工大学 A kind of shooting projection interactive system and method based on binocular vision
CN107492080A (en) * 2017-09-25 2017-12-19 天津智慧视通科技有限公司 Exempt from calibration easily monocular lens image radial distortion antidote
CN107492080B (en) * 2017-09-25 2021-03-23 杭州云栖智慧视通科技有限公司 Calibration-free convenient monocular head image radial distortion correction method
CN109842756A (en) * 2017-11-28 2019-06-04 东莞市普灵思智能电子有限公司 A kind of method and system of lens distortion correction and feature extraction

Also Published As

Publication number Publication date
CN106023170B (en) 2018-10-23

Similar Documents

Publication Publication Date Title
EP3163535B1 (en) Wide-area image acquisition method and device
CN109314753B (en) Method and computer-readable storage medium for generating intermediate views using optical flow
CN107358577B (en) Rapid splicing method of cubic panoramic image
CN110211043A (en) A kind of method for registering based on grid optimization for Panorama Mosaic
CN111047510A (en) Large-field-angle image real-time splicing method based on calibration
CN103337094A (en) Method for realizing three-dimensional reconstruction of movement by using binocular camera
CN110956661B (en) Method for calculating dynamic pose of visible light and infrared camera based on bidirectional homography matrix
CN110910431B (en) Multi-view three-dimensional point set recovery method based on monocular camera
CN102982524B (en) Splicing method for corn ear order images
CN106023170A (en) Binocular 3D distortion rectification method based on GPU
CN105631859B (en) Three-degree-of-freedom bionic stereo visual system
CN110099268B (en) Blind area perspective display method with natural color matching and natural display area fusion
CN103313081A (en) Image processing apparatus and method
CN108181319A (en) A kind of laying dust detecting device and method based on stereoscopic vision
CN107016707B (en) A kind of integration imaging super large three-dimensional scenic shooting method for correcting image
CN109493282A (en) A kind of stereo-picture joining method for eliminating movement ghost image
WO2020151268A1 (en) Generation method for 3d asteroid dynamic map and portable terminal
CN105606123A (en) Method for automatic correction of digital ground elevation model for low-altitude aerial photogrammetry
CN107330856B (en) Panoramic imaging method based on projective transformation and thin plate spline
Gao et al. Design of signal processing pipeline for stereoscopic cameras
CN113096016A (en) Low-altitude aerial image splicing method and system
CN110430400A (en) A kind of ground level method for detecting area of the movable video camera of binocular
EP3229106A1 (en) Efficient determination of optical flow between images
CN108830804A (en) Virtual reality fusion Fuzzy Consistent processing method based on line spread function standard deviation
CN111629194B (en) Method and system for converting panoramic video into 6DOF video based on neural network

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant