CN110580686A - Polarization image restoration device and method based on binocular vision in scattering environment - Google Patents

Polarization image restoration device and method based on binocular vision in scattering environment Download PDF

Info

Publication number
CN110580686A
CN110580686A CN201910713565.7A CN201910713565A CN110580686A CN 110580686 A CN110580686 A CN 110580686A CN 201910713565 A CN201910713565 A CN 201910713565A CN 110580686 A CN110580686 A CN 110580686A
Authority
CN
China
Prior art keywords
image
light
polarization
light intensity
binocular
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201910713565.7A
Other languages
Chinese (zh)
Other versions
CN110580686B (en
Inventor
胡浩丰
张燕彬
刘铁根
王辉
李校博
程振洲
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tianjin University
Original Assignee
Tianjin University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tianjin University filed Critical Tianjin University
Priority to CN201910713565.7A priority Critical patent/CN110580686B/en
Publication of CN110580686A publication Critical patent/CN110580686A/en
Application granted granted Critical
Publication of CN110580686B publication Critical patent/CN110580686B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00Image enhancement or restoration

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention discloses a polarization image restoration device and method based on binocular vision in a scattering environment, wherein the device comprises a light source (1), a first horizontal polarizer (2), a binocular imaging system with an orthogonal polarizer and a computer processing unit; step 1, acquiring two polarized images by using a binocular imaging system with orthogonal polarizers, wherein the two polarized images are respectively parallel to and perpendicular to illumination polarized light; step 2, obtaining a depth image of the target object, and mapping the gray value of each pixel point of the right image to the left image to obtain a matched right image; step 3, obtaining the transmittance t (x, y) of different positions (x, y); step 4, obtaining a restoration image L (x, y); and 5, obtaining the degree of polarization DOP of the image. Compared with the prior art, the method can recover a clear image and identify objects with different polarization degrees in a scattering medium environment with higher concentration, and particularly can distinguish the objects according to the difference of the polarization degrees when the light intensities of the objects are similar.

Description

polarization image restoration device and method based on binocular vision in scattering environment
Technical Field
the invention relates to the technical field of polarization imaging detection, in particular to a binocular vision depth and polarization imaging restoration method and an object polarization degree calculation method in a scattering environment.
Background
the polarization imaging technology has a very important position in the fields of national defense and military, biomedicine, industrial production and the like. Compared with the traditional optical imaging, the polarization imaging technology is a novel optical detection technology. The traditional optical technology can only obtain the light intensity and wavelength information of the image, and the polarization imaging technology can also obtain the polarization information, thereby obtaining more multidimensional information. In addition, even under the condition of scattering environment, a detection image with relatively high contrast can be obtained by utilizing the polarization imaging technology, and objects with similar light intensity are identified according to the difference of polarization information. The traditional polarization image restoration method can only take the polarization degree of an object as 0, thereby estimating the value of the transmittance t and restoring the image. The method is only suitable for the object with low polarization degree to obtain a better restored image, but has a poorer image restoration effect for the object with higher polarization degree. In addition, the existing polarization image restoration technology can only restore the light intensity image of the object, and cannot obtain the information of the polarization degree of the object, so that the object with different polarization degrees cannot be identified.
The binocular vision technology can obtain the depth information of an object, and the depth information and the polarization information are combined, so that the definition of the restored image is improved, and meanwhile, the polarization information of the object is obtained.
disclosure of Invention
in order to overcome the defects of the prior art, the invention provides a polarization image restoration and image identification method based on binocular vision in a scattering environment.
The invention relates to a polarization image restoration device based on binocular vision in a scattering environment, which comprises a light source 1, a first horizontal polarizer 2, a binocular imaging system with orthogonal polarizers and a computer processing unit, wherein the first horizontal polarizer is arranged on the front surface of the light source; the binocular imaging system comprises a first and a second binocular camera 6, 7; whereinThe light emitted by the light source 1 is changed into horizontally polarized light through the horizontal polarizing film 2 and then irradiated on a target object, and then passes through the orthogonal polarizing film, namely the second horizontal polarizing film 4 and the vertical polarizing film 5 and then respectively enters the first binocular camera 6 and the second binocular camera 7, and simultaneously two polarized images including a light intensity image I in the horizontal direction are obtainedand intensity profile I in the vertical directionand the computer processing unit processes the two polarized images to realize the restoration of the polarized images.
The invention relates to a polarization image restoration method based on binocular vision in a scattering environment, which comprises the following concrete implementation steps:
Step 1, acquiring two polarized images by using a binocular imaging system with orthogonal polarizing plates in an active linearly polarized light illumination mode, wherein the two polarized images are respectively parallel to and perpendicular to illumination polarized light;
step 2, using the left picture IMatching the right graph I by adopting a dense matching mode as a reference graphobtaining a parallax map, further obtaining a depth image rho of the target object, and mapping the gray value of each pixel point of the right image to the left image to obtain a matched right image;
The depth information expression of each point in the depth map rho is as follows:
Wherein f is the focal length of the camera, b is the camera line distance of the binocular camera, d (x, y) is the number of pixels of the phase difference between corresponding points of the reference image and the matching image, and ps is the size of a single pixel;
And 3, obtaining the transmittance t (x, y) of different positions (x, y), wherein the calculation formula is as follows:
t(x,y)=e-β(x,y)ρ(x,y)
Wherein β (x, y) is a scattering coefficient, and ρ (x, y) is the depth information of the target object obtained in step 2;
and 4, obtaining the restored image L, which comprises the following specific steps:
The light intensity I (x, y) received by the CCD can be divided into two parts, namely the object reflected light L (x, y)) (light intensity value of each point of the restored image) attenuated light intensity and background light intensity AScattered backscattered light, the formula is as follows:
I(x,y)=L(x,y)t(x,y)+A[1-t(x,y)]
Where t (x, y) is the transmittance obtained in step 3, and an expression of the light reflected by the object, i.e., a restored image, can be derived from the above formula, where the expression is as follows:
Wherein, L represents the restored image, and L (x, y) represents the light intensity value of the specific point in the restored image;
And step 5, obtaining the polarization degree of the object, namely a polarization degree diagram DOP, wherein the formula is as follows:
wherein L (x, y) is the light intensity value of each point in the restoration image obtained in step 4, Δ L (x, y) is the difference between the horizontal polarized light intensity and the vertical polarized light intensity of the object reflected light, and the calculation formula is as follows:
Wherein t (x, y) is the transmittance obtained in step 3, and Δ D (x, y) is the difference between the light intensity after the attenuation of the horizontally polarized light and the light intensity after the attenuation of the vertically polarized light of the reflected light of the object, and can be obtained by the following formula:
Wherein, Δ I (x, y) is the difference between the left and right images received by CCD, Pscatis the background light polarization at the background, Ais the light intensity at the background.
compared with the prior art, the method can recover a clear image and identify objects with different polarization degrees in a scattering medium environment with higher concentration, and particularly can distinguish the objects according to the difference of the polarization degrees when the light intensities of the objects are similar.
Drawings
FIG. 1 is a schematic overall flow chart of a binocular vision-based polarization image restoration and image identification method in a scattering environment according to the present invention; the images 1 and 2 are directly shot images, and the rest images are obtained after the processing of each step.
FIG. 2 is a schematic structural diagram of an apparatus for image restoration and identification in a scattering environment using binocular vision and polarization imaging technology according to the present invention;
Reference numerals:
1. a light source; 2. the system comprises a first horizontal polarizer, 3, a target object, 4, a second horizontal polarizer, 5, a vertical polarizer, 6, 7, a first binocular camera and a second binocular camera.
Detailed Description
The technical solution of the present invention is described in detail below with reference to the accompanying drawings and examples.
Fig. 1 is a schematic structural diagram of an image restoration apparatus using binocular vision and polarization in a scattering environment according to the present invention. The entire apparatus includes a light source 1, a horizontal polarizer 2, first to second polarizers 2, 4, 5, and binocular imaging systems 6, 7. Light emitted by the light source 1 passes through the horizontal polarizing film 2 to become horizontal linear polarized light and irradiates on a target object, and then passes through the mutually orthogonal polarizing films 4 and 5 and is collected by the binocular cameras 6 and 7. Wherein, a horizontal polarizer 4 is arranged in front of the left camera 6, and a vertical polarizer 5 is arranged in front of the right camera 7, so as to obtain two polarized images. By processing the two polarized images, a restored clear image and an image displaying the polarization degree of the object can be obtained.
as shown in fig. 1, the following will describe the embodiment of the present invention in further detail according to the overall flow diagram of the method, and the detailed steps are as follows:
Step 1, acquiring two images by using a binocular camera with a polaroid in an active illumination mode;
Irradiating the target object with linearly polarized light in the horizontal direction, and taking a picture with a binocular camerashooting images, wherein a horizontal polarizing film and a vertical polarizing film are respectively arranged in front of a binocular camera; according to the conventional polarization imaging theory, the polarization images acquired by the imaging system are respectively: left picture I(image 1), right image I(image 2). Wherein, Ifor the image acquired by the left camera plus the horizontal linear polarizer, Ithe image collected by the right camera and the vertical linear polaroid.
Step 2, obtaining a left image and a right image through the double cameras to obtain a disparity map so as to obtain depth information of the object;
and matching the right graph by taking the left graph as a reference graph in a dense matching mode (namely, each point is matched). Firstly, image segmentation is carried out on a left image, a matching window is designed according to the segmented shape, a matching cost function can select a proper similarity judgment standard according to different scenes, and filtering processing is carried out on a matching result to obtain a disparity map (image 3). On the basis of the completion of matching, mapping the gray value of each pixel point of the right image to the corresponding left image to obtain a matched right image I'(image 4). The number d of pixels of the phase difference between the matching points can be known through the disparity map, and the depth information of the object, namely the depth map (image 5) can be obtained according to the formula (1).
Wherein rho (x, y) is depth information of each point, f is a camera focal length, b is a machine line distance (distance between central points of two cameras), d (x, y) is the number of pixels with phase difference between corresponding points, and ps is the size of a single pixel;
And 3, obtaining the transmittance t (x, y) of different positions by using the depth information:
t(x,y)=e-β(x,y)ρ(x,y) (2)
wherein, beta (x, y) is a scattering coefficient, and rho (x, y) is the depth information of each point of the image obtained in the step 2;
Step 4, restoring an image by using a polarization imaging model in a scattering environment;
I(x,y)=D(x,y)+B(x,y) (3)
wherein, I (x, y) is the light intensity value received by the CCD, D (x, y) is the light intensity after the attenuation of the reflected light of the object, and B (x, y) is the background scattered light;
D(x,y)=L(x,y)t(x,y) (4)
Wherein, L (x, y) is the target object reflected light;
B(x,y)=A[1-t(x,y)] (5)
Wherein A isThe light intensity at infinity (i.e., at background).
derived from formula (6):
and a restored image (image 6) is obtained.
Step 5, calculating the polarization degree by using the transmittance t and the restoration image L;
in the polarization imaging model, the background scattered light of the whole scene is approximately considered to be equal according to the formula
wherein, Δ I (x, y) is the light intensity difference between the left and right images received by the CCD, as long as the background light polarization P at the background is obtainedscatand the intensity of light A at the backgroundthe difference Δ D (x, y) between the horizontally polarized light and the vertically polarized light of the object can be obtained, where Δ B (x, y) is the difference between the horizontally polarized light and the vertically polarized light of the background scattered light. Since D (x, y) is L (x, y) t (x, y), that is, Δ D (x, y) is Δ L (x, y) t (x, y), the formula of the degree of polarization is further derived:
By combining the transmittance t obtained in step 3 and the restoration image L obtained in step 4, a polarization degree map (image 7) can be obtained, and the image can be recognized.
the method is suitable for fog-containing image restoration and object polarization degree calculation in a scattering medium environment, utilizes a binocular camera with an orthogonal polaroid to perform imaging, calculates the depth by a binocular vision related method, and combines a polarization imaging model to further realize image restoration and image identification. According to the invention, the image is restored directly through the depth information, so that a more accurate transmittance t value can be obtained, and more accurate restoration is further realized; the depth information is combined with the polarization defogging model, the polarization degree of an object in a scene can be obtained, the information can not be obtained in the traditional polarization recovery model, the polarization recovery model can be used for identifying the objects with different polarization degrees, and particularly, the objects can be distinguished according to the difference of the polarization degrees when the light intensity of the objects is similar.

Claims (2)

1. A binocular vision based polarization image restoration device in a scattering environment, characterized in that the device comprises a light source (1), a first horizontal polarizer (2), a binocular imaging system with orthogonal polarizers, and a computer processing unit; the binocular imaging system comprises a first and a second binocular camera (6) (7); wherein, the light emitted by the light source (1) is changed into horizontal polarized light through the horizontal polarizing film (2) and then is irradiated on a target object, then passes through the orthogonal polarizing film, namely a second horizontal polarizing film (4) and a vertical polarizing film (5) and then respectively enters a first binocular camera (6) and a second binocular camera (7), and simultaneously two polarized images including a light intensity chart I in the horizontal direction are obtainedand intensity profile I in the vertical directionAnd the computer processing unit processes the two polarized images to realize the restoration of the polarized images.
2. a polarization image restoration method based on binocular vision in a scattering environment is characterized by comprising the following specific implementation steps:
Step 1, acquiring two polarized images by using a binocular imaging system with orthogonal polarizing plates in an active linearly polarized light illumination mode, wherein the two polarized images are respectively parallel to and perpendicular to illumination polarized light;
step 2, using the left picture Imatching the right graph I by adopting a dense matching mode as a reference graphObtaining a parallax map, further obtaining a depth image rho of the target object, and taking each pixel of the right imageMapping the gray value of the point to the left image to obtain a matched right image;
the depth information expression of each point in the depth map rho is as follows:
Wherein f is the focal length of the camera, b is the camera line distance of the binocular camera, d (x, y) is the number of pixels of the phase difference between corresponding points of the reference image and the matching image, and ps is the size of a single pixel;
And 3, obtaining the transmittance t (x, y) of different positions (x, y), wherein the calculation formula is as follows:
t(x,y)=e-β(x,y)ρ(x,y)
Wherein β (x, y) is a scattering coefficient, and ρ (x, y) is the depth information of the target object obtained in step 2;
and 4, obtaining the restored image L, which comprises the following specific steps:
The light intensity I (x, y) received by the CCD can be divided into two parts, namely the light intensity after the object reflected light L (x, y) (the light intensity value of each point of the restored image) is attenuated and the light intensity A at the backgroundScattered backscattered light, the formula is as follows:
I(x,y)=L(x,y)t(x,y)+A[1-t(x,y)]
Where t (x, y) is the transmittance obtained in step 3, and an expression of the light reflected by the object, i.e., a restored image, can be derived from the above formula, where the expression is as follows:
Wherein, L represents the restored image, and L (x, y) represents the light intensity value of the specific point in the restored image;
and step 5, obtaining the polarization degree of the object, namely a polarization degree diagram DOP, wherein the formula is as follows:
wherein L (x, y) is the light intensity value of each point in the restoration image obtained in step 4, Δ L (x, y) is the difference between the horizontal polarized light intensity and the vertical polarized light intensity of the object reflected light, and the calculation formula is as follows:
wherein t (x, y) is the transmittance obtained in step 3, and Δ D (x, y) is the difference between the light intensity after the attenuation of the horizontally polarized light and the light intensity after the attenuation of the vertically polarized light of the reflected light of the object, and can be obtained by the following formula:
Wherein, Δ I (x, y) is the difference between the left and right images received by CCD, PscatIs the background light polarization at the background, AIs the light intensity at the background.
CN201910713565.7A 2019-08-02 2019-08-02 Polarization image restoration device and method based on binocular vision in scattering environment Active CN110580686B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910713565.7A CN110580686B (en) 2019-08-02 2019-08-02 Polarization image restoration device and method based on binocular vision in scattering environment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910713565.7A CN110580686B (en) 2019-08-02 2019-08-02 Polarization image restoration device and method based on binocular vision in scattering environment

Publications (2)

Publication Number Publication Date
CN110580686A true CN110580686A (en) 2019-12-17
CN110580686B CN110580686B (en) 2023-01-20

Family

ID=68810847

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910713565.7A Active CN110580686B (en) 2019-08-02 2019-08-02 Polarization image restoration device and method based on binocular vision in scattering environment

Country Status (1)

Country Link
CN (1) CN110580686B (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111343368A (en) * 2020-02-18 2020-06-26 清华大学 Method and device for recovering depth of scattering medium based on polarization
CN111899290A (en) * 2020-07-09 2020-11-06 武汉大学 Three-dimensional reconstruction method combining polarization and binocular vision
CN113554575A (en) * 2020-04-23 2021-10-26 华东交通大学 High-reflection object surface highlight removing method based on polarization principle
WO2024055445A1 (en) * 2022-09-13 2024-03-21 深圳先进技术研究院 Underwater high-throughput imaging monitoring apparatus

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN2033173U (en) * 1988-04-01 1989-02-22 雷民 Tv adding device for gaining stereoscopic sense
US20120069320A1 (en) * 2009-01-09 2012-03-22 Asmr Holding B.V. Optical rangefinder and imaging apparatus with chiral optical arrangement
CN104052967A (en) * 2014-06-04 2014-09-17 河海大学 Intelligent system for acquiring underwater polarization target depth map and method thereof
CN104777624A (en) * 2015-03-25 2015-07-15 清华大学深圳研究生院 Polarization imaging device and method
US20160270665A1 (en) * 2009-02-05 2016-09-22 The Provost, Fellows & Scholars of the College of the Holy & Undivided Trinity of Queen Elizabeth Ne Method and apparatus for imaging tissue topography
CN107966412A (en) * 2017-12-05 2018-04-27 天津大学 Based on cross-polarization image stretch and the associated image defogging method of degree of polarization
CN108335324A (en) * 2018-01-29 2018-07-27 清华大学 Scattering scene depth method for reconstructing and equipment based on polarization transient state imaging
US10331207B1 (en) * 2013-03-15 2019-06-25 John Castle Simmons Light management for image and data control
CN110044300A (en) * 2019-01-22 2019-07-23 中国海洋大学 Amphibious 3D vision detection device and detection method based on laser

Patent Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN2033173U (en) * 1988-04-01 1989-02-22 雷民 Tv adding device for gaining stereoscopic sense
US20120069320A1 (en) * 2009-01-09 2012-03-22 Asmr Holding B.V. Optical rangefinder and imaging apparatus with chiral optical arrangement
US20160270665A1 (en) * 2009-02-05 2016-09-22 The Provost, Fellows & Scholars of the College of the Holy & Undivided Trinity of Queen Elizabeth Ne Method and apparatus for imaging tissue topography
US10331207B1 (en) * 2013-03-15 2019-06-25 John Castle Simmons Light management for image and data control
CN104052967A (en) * 2014-06-04 2014-09-17 河海大学 Intelligent system for acquiring underwater polarization target depth map and method thereof
CN104777624A (en) * 2015-03-25 2015-07-15 清华大学深圳研究生院 Polarization imaging device and method
CN107966412A (en) * 2017-12-05 2018-04-27 天津大学 Based on cross-polarization image stretch and the associated image defogging method of degree of polarization
CN108335324A (en) * 2018-01-29 2018-07-27 清华大学 Scattering scene depth method for reconstructing and equipment based on polarization transient state imaging
CN110044300A (en) * 2019-01-22 2019-07-23 中国海洋大学 Amphibious 3D vision detection device and detection method based on laser

Non-Patent Citations (7)

* Cited by examiner, † Cited by third party
Title
GLENN H SCHNEIDER 等: ""Detection and characterization of circumstellar material with a WFIRST or EXO-C coronagraphic instrument: simulations and observational methods"", 《SPIE》 *
HAOFENG HU 等: ""Enhancing visibility of polarimetric underwater image by transmittance correction"", 《IEEE》 *
HAOFENG HU 等: ""Underwater image recovery under the nonuniform optical field based on polarimetric imaging"", 《IEEE》 *
YONGQING ZHAO 等: ""Bio-inspired multi-band polarization imaging"", 《MULTI-BAND POLARIZATION IMAGING AND APPLICATIONS》 *
胡浩丰 等: ""基于偏振成像的水下图像复原技术研究最新进展"", 《红外与激光工程》 *
范新南等: "结构相似性的水下偏振图像复原", 《中国图象图形学报》 *
赵琳: ""基于偏振信息的水下复杂环境中灰度图像复原方法的研究"", 《中国优秀硕士学位论文全文数据库信息科技辑》 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111343368A (en) * 2020-02-18 2020-06-26 清华大学 Method and device for recovering depth of scattering medium based on polarization
CN111343368B (en) * 2020-02-18 2021-08-20 清华大学 Method and device for recovering depth of scattering medium based on polarization
CN113554575A (en) * 2020-04-23 2021-10-26 华东交通大学 High-reflection object surface highlight removing method based on polarization principle
CN111899290A (en) * 2020-07-09 2020-11-06 武汉大学 Three-dimensional reconstruction method combining polarization and binocular vision
WO2024055445A1 (en) * 2022-09-13 2024-03-21 深圳先进技术研究院 Underwater high-throughput imaging monitoring apparatus

Also Published As

Publication number Publication date
CN110580686B (en) 2023-01-20

Similar Documents

Publication Publication Date Title
CN110580686B (en) Polarization image restoration device and method based on binocular vision in scattering environment
Swirski et al. 3Deflicker from motion
US8334893B2 (en) Method and apparatus for combining range information with an optical image
CN107635129B (en) Three-dimensional trinocular camera device and depth fusion method
CN110956661B (en) Method for calculating dynamic pose of visible light and infrared camera based on bidirectional homography matrix
WO2014044126A1 (en) Coordinate acquisition device, system and method for real-time 3d reconstruction, and stereoscopic interactive device
TW201100760A (en) Distance measuring apparatus having dual stereo cameras
CN103458261B (en) Video scene variation detection method based on stereoscopic vision
Punnappurath et al. Modeling defocus-disparity in dual-pixel sensors
CN106570899B (en) Target object detection method and device
EP2509324A1 (en) Method and apparatus for analyzing stereoscopic or multi-view images
CN110969667A (en) Multi-spectrum camera external parameter self-correction algorithm based on edge features
CN108257165B (en) Image stereo matching method and binocular vision equipment
CN111080709A (en) Multispectral stereo camera self-calibration algorithm based on track feature registration
CN111738941A (en) Underwater image optimization method fusing light field and polarization information
Liu et al. High quality depth map estimation of object surface from light-field images
Ni et al. Reflection removal based on single light field capture
CN112470189B (en) Occlusion cancellation for light field systems
CN111951339A (en) Image processing method for performing parallax calculation by using heterogeneous binocular cameras
Anderson et al. Augmenting depth camera output using photometric stereo.
CN105530419B (en) Image acquisition system, image acquisition processing system and image acquisition processing method
CN108805937B (en) Single-camera polarization information prediction method
CN113706693B (en) Polarization three-dimensional reconstruction method under low-light condition
CN115471537A (en) Monocular camera-based moving target distance and height measuring method
CN107610170B (en) Multi-view image refocusing depth acquisition method and system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant