CN113034572B - Epipolar extraction method based on eight-parameter epipolar model - Google Patents

Epipolar extraction method based on eight-parameter epipolar model Download PDF

Info

Publication number
CN113034572B
CN113034572B CN202110460890.4A CN202110460890A CN113034572B CN 113034572 B CN113034572 B CN 113034572B CN 202110460890 A CN202110460890 A CN 202110460890A CN 113034572 B CN113034572 B CN 113034572B
Authority
CN
China
Prior art keywords
epipolar
parameter
parameters
epipolar line
equation
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202110460890.4A
Other languages
Chinese (zh)
Other versions
CN113034572A (en
Inventor
余俊鹏
沈刚
李洁明
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guangdong University of Technology
Original Assignee
Guangdong University of Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guangdong University of Technology filed Critical Guangdong University of Technology
Priority to CN202110460890.4A priority Critical patent/CN113034572B/en
Publication of CN113034572A publication Critical patent/CN113034572A/en
Application granted granted Critical
Publication of CN113034572B publication Critical patent/CN113034572B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Image Processing (AREA)

Abstract

An epipolar extraction method based on an eight-parameter epipolar model belongs to the field of photogrammetry and computer vision. The invention solves the problems of low precision of the epipolar line result extracted by the existing approximate epipolar line method and complex epipolar line extraction process. The method comprises the following steps: acquiring stereo image data; matching homonymy points in the stereo image data, and recording image point coordinates of the homonymy points; establishing an eight-parameter epipolar line equation according to the coordinates of the image points of the same-name points; determining initial values of the eight parameters according to an eight-parameter epipolar equation; linearizing the eight-parameter epipolar equation to obtain an error equation; establishing a normal equation by using a least square method based on an error equation, and calculating correction quantity of eight parameters according to the normal equation; judging whether the correction quantity meets the convergence requirement, if so, correcting the eight parameters according to the correction quantity to obtain corrected eight parameters; and calculating and generating an epipolar line of the three-dimensional image of the target area through the corrected eight parameters. The invention is used for extracting the stereoscopic image epipolar line.

Description

Epipolar line extraction method based on eight-parameter epipolar line model
Technical Field
The invention relates to an epipolar line extraction method based on an eight-parameter epipolar line model. Belonging to the field of photogrammetry and computer vision.
Background
In photogrammetry, a single image cannot determine the spatial position of a point on an object, only the spatial direction of the photographic light of the object point can be determined, two images which are overlapped with each other are generally used for forming a stereopair which is a basic unit of stereophotogrammetry, and a stereopair model formed by the stereopair is the basis of photogrammetry three-dimensional positioning.
Epipolar lines are a basic concept of photogrammetry. After Helava et al put forward the theory related to epipolar lines in the early 70 s of the 20 th century, the role of epipolar lines was quickly gaining attention in the field of photogrammetry. In photogrammetry, a plane formed by a shooting baseline and any point on the ground is a nuclear surface, an intersection line of one nuclear surface and a left image and a right image with stereo overlapping degree is a homonymous nuclear line, and homonymous image points are strictly positioned on the homonymous nuclear line on a stereo image projected by a single center. In image matching, the most important geometric constraint is the epipolar constraint. The search range of the homonymous points can be reduced from two dimensions to one dimension by utilizing epipolar line constraint in image matching, so that the search space of the homonymous points is greatly reduced, and the matching efficiency and accuracy of the homonymous points of the images are improved. In multi-view image matching and computer stereo vision, epipolar constraint plays an irreplaceable role in ensuring the reliability of dense matching processing.
The satellite photogrammetry technique is a space pairThe earth observation technology mainly adopts a linear array push-broom type sensor to obtain geometric information of earth surface, and is widely applied to topographic mapping, resource investigation, digital city and national defense construction. The method for generating the epipolar line of the linear array satellite image is always a difficult point in the fields of photogrammetry and remote sensing. Because each line of the linear array image corresponds to different exterior orientation elements, a unique photographic baseline does not exist among the three-dimensional images, and the classical epipolar line theory is not applicable. Research has shown that the epipolar line of the linear array stereo image is a curve similar to a hyperbola and can be regarded as a straight line only in a small range. The existing method for extracting the approximate epipolar line from the satellite linear array image is mainly a projection trajectory method and various derivative methods thereof. The projection trajectory method is based on the known left and right image orientation parameters, and the projection trajectory line of the left image target point a on the right image, namely the kernel curve l ', the homonymous point a' of the a must be on l ', and the point-line corresponding relation of the a and the l' is obtained. Further research shows that the adjacent points b ' and c ' on the right image kernel curve l ' correspond to the left image kernel curve l 1 ,l 2 Closely, the same curve l can be used instead, so that the line-line correspondence of l and l' in the local range can be established. The derivation method based on the projection trajectory method includes a linear array push-broom type satellite stereopair approximate epipolar image generation method based on a projection reference surface, an epipolar image generation method along a horizontal image surface and the like.
Although the approximate epipolar line extracted by the above method can play a role of epipolar line constraint in epipolar line image matching, the following problems exist:
(1) The coordinate transformation relationship between the epipolar line image and the original image is complex, and the epipolar line image depends on a strict geometric model or a rational polynomial function model of the image, and the model parameters are not available at times, so that the model parameter solution has certain difficulty.
(2) Different from the strict epipolar line of a single-center projection image, the approximate epipolar line has certain vertical parallax, and the accuracy of the epipolar line generated by different satellite linear array images is different.
In view of the shortcomings of the existing approximate epipolar line method, an epipolar line model and an epipolar line extraction method which are higher in precision, convenient and easy to use and good in universality are urgently needed to provide technical support for photogrammetry and computer stereoscopic vision data processing.
Disclosure of Invention
The method aims to solve the problems that the epipolar line extraction result extracted by the existing approximate epipolar line method is low in precision and the epipolar line extraction process is complex. An epipolar extraction method based on an eight-parameter epipolar model is now provided.
The epipolar line extraction method based on the eight-parameter epipolar line model comprises the following steps:
step one, acquiring stereo image data covering the same target area;
matching a plurality of uniformly distributed homonymy points on the stereo image, and recording image point coordinates of the homonymy points;
establishing an eight-parameter epipolar equation of the target at the homonymous points of the left image and the right image according to the homonymous point image point coordinates;
determining initial values of the eight parameters according to an eight-parameter epipolar equation;
step five, carrying out linearization processing on the eight-parameter epipolar equation to obtain an error equation;
step six, establishing a normal equation by using a least square method based on an error equation, and solving correction quantity of eight parameters according to the normal equation;
step seven, judging whether the correction quantity meets the convergence requirement, if so, correcting the eight parameters according to the correction quantity to obtain corrected eight parameters, if not, replacing the initial values of the eight parameters with the corrected eight parameters, and executing the steps three to six until convergence;
and step eight, calculating and generating the epipolar line of the three-dimensional image of the target area through the corrected eight parameters.
Advantageous effects
(1) And the nuclear line precision is high. The method is provided based on the inclination angle change rule of the linear array image epipolar line, and tests show that the error in the upper and lower parallax of the epipolar line extracted by the eight-parameter epipolar line model can be less than 0.5 pixel, the precision can be improved by 30 to 70 percent compared with the common epipolar line model based on the projection trajectory method, and the geometric constraint requirement of the same-name point of the stereoscopic image can be fully met.
(2) Is simple and easy to use. The parameter solution of the eight-parameter epipolar line model only needs a few homonymous connecting points, and any image imaging model parameter is not needed to be known. The coordinate conversion from the original image to the epipolar line image is simple in calculation, and the method is very beneficial to the resampling treatment of the subsequent epipolar line image.
(3) The application range is wide. The eight-parameter epipolar line model can be simultaneously suitable for the epipolar line extraction of single-center projection images and multi-center projection images, and can be applied to aerospace, aviation, close-range photogrammetry and computer stereoscopic vision processing.
Drawings
FIG. 1 is a flow chart of the method of the present invention;
FIG. 2 is a graph of the image epipolar line accuracy comparison.
Detailed Description
The first embodiment is as follows: specifically, the present embodiment is described with reference to fig. 1, and the epipolar line image generation method based on the eight-parameter epipolar line model according to the present embodiment includes:
step one, acquiring three-dimensional image data covering the same target area;
matching a plurality of (not less than 4) uniformly distributed homonymy points in the stereo image data, and recording image point coordinates of the homonymy points;
establishing an eight-parameter epipolar equation of the target at the homonymous points of the left image and the right image according to the homonymous point image point coordinates;
determining initial values of the eight parameters according to an eight-parameter epipolar equation;
step five, carrying out linearization processing on the eight-parameter epipolar equation to obtain an error equation;
establishing a normal equation by using a least square method based on an error equation, and calculating correction quantity of eight parameters according to the normal equation;
and step seven, judging whether the correction quantity meets the convergence requirement, if so, correcting the eight parameters according to the correction quantity to obtain the corrected eight parameters, if not, replacing the initial values of the eight parameters with the corrected eight parameters, and executing the steps three to six until convergence.
And step eight, calculating and generating the epipolar line of the three-dimensional image of the target area through the corrected eight parameters.
The second embodiment is as follows: the difference between this embodiment and the first embodiment is that the eight-parameter epipolar equation in the third step is expressed by the following formula:
0＝x L ′-x R ' (1) wherein,
Figure BDA0003042189980000031
Figure BDA0003042189980000032
wherein (x) L ,y L ) Representing column coordinates and row coordinates of the target on the left stereoscopic image; (x) R ,y R ) Column coordinates and row coordinates representing the target on the right stereoscopic image; (x) L ′,y L ') column coordinates and row coordinates of the object on the left epipolar line image; (x) R ′,y R ') column coordinates and row coordinates of the object on the right epipolar line image; a is 1 、b 1 、c 1 、a 2 、b 2 、c 2 Affine transformation parameters of the right epipolar image relative to the right original image; alpha is alpha 0 Indicating the inclination, alpha, of the left epipolar line passing through a reference point on the left stereoscopic image v Representing the rate of change of the left epipolar line dip;
wherein alpha is 0 、α v 、a 1 、b 1 、c 1 、a 2 、b 2 、c 2 Eight parameters corrected for the left and right homonymy epipolar lines are parameters to be solved for the epipolar line model; the initial values of the eight parameters are represented by vectors as:
Figure BDA0003042189980000041
in the present embodiment, when the epipolar line direction is in the column direction, the column coordinate and the row coordinate of the target on the left epipolar line image are (x) L ′,y L ') to a test; if the epipolar line direction is in the row direction, the row coordinate on the left epipolar line imageAnd row coordinate of (x) L ″,y L ") expressed as:
Figure BDA0003042189980000042
other steps and parameters are the same as those in the first embodiment.
The third concrete implementation mode: the difference between this embodiment and the first or second embodiment is that the error equation in the fourth step is:
V＝Bx-l,P (5)
wherein, x = [ Δ α = 0 Δα v Δa 1 Δb 1 Δc 1 Δa 2 Δb 2 Δc 2 ] T Correction amount is eight parameters; b is a partial derivative coefficient matrix of the formula (1) to eight parameters; l is a vector with an irregular value, i.e. an initial value X of eight parameters 0 A value substituted for formula (1); v is a correction vector; p is a weight matrix.
Other steps and parameters are the same as those in the first or second embodiment.
The fourth concrete implementation mode: the difference between the first embodiment and the third embodiment is that the fifth step is based on an error equation and establishes a normal equation by using a least square method; solving the correction quantity of eight parameters according to a normal equation; the specific process is as follows:
establishing a normal equation according to a least square principle:
B T PBx-B T Pl＝0 (6)
wherein T represents transpose;
order:
N BB ＝B T PB,W＝B T Pl
can be abbreviated as
N BB x-W＝0(7)
In the formula, N BB Is a full rank matrix, where x has a unique solution and is also the optimal solution,
Figure BDA0003042189980000043
or x = (B) T PB) -1 B T Pl。
Other steps and parameters are the same as those in one of the first to third embodiments.
The fifth concrete implementation mode: the present embodiment is different from the first to the fourth embodiments in that the sixth step corrects the eight parameters according to the correction amount of the eight parameters; the specific process is as follows:
X＝X 0 +x (8)
wherein X represents the corrected eight parameters.
Other steps and parameters are the same as in one of the first to fourth embodiments.
The sixth specific implementation mode is as follows: the difference between this embodiment and one of the first to fifth embodiments is that the convergence requirement is:
setting a threshold value 10 -8 If the absolute value of the correction number of each parameter of X is smaller than the threshold value, calculating eight parameters according to the numerical value of X at the moment;
if the absolute value of the correction number of each parameter in X is greater than or equal to the threshold value, let X = X 0 And repeating the third step to the sixth step.
Other steps and parameters are the same as in one of the first to fifth embodiments.
Examples
In order to verify the effectiveness of the method, simulation data is adopted to carry out a nuclear model parameter estimation experiment. And (3) simulating and generating homonymous point pairs uniformly distributed in 1000 × 1000, 2000 × 2000, 3000 × 3000, 4000 × 4000 and 5000 × 5000 pixel image ranges by using RPC parameters attached to the Pleiades, SPOT-5, quickbird and ZY3 satellite images respectively, and obtaining the vertical parallax medium errors of different satellite images in different image ranges by using eight-parameter epipolar lines.
Table 1 test satellite linear array stereo image parameter table
Figure BDA0003042189980000051
The results of the eight-parameter epipolar model are shown in table 2, and the epipolar precision ratio for each image is shown in fig. 2.
TABLE 2 errors in top and bottom parallax obtained by eight parameter epipolar line model
Figure BDA0003042189980000052
As can be seen from table 2, the error in the up-down parallax obtained in the eight-parameter model for different satellite stereo images increases as the image range is enlarged. The comparison result of the errors in the epipolar vertical parallax of the four images is: ZY3< SPOT-5 straw cloth and pleiades straw cloth and Quickbird. When the image range of the ZY3, the Pleiades, the SPOT-5 and the Quickbird satellite image data is 5000 multiplied by 5000, the errors in the upper and lower parallaxes are respectively 0.0015 pixel, 0.1216 pixel, 0.0255 pixel and 0.7883 pixel.
It should be noted that, for a person skilled in the art, there may be variations to the embodiments and applications according to the idea of the invention, and all such modifications and changes are intended to fall within the scope of the appended claims.

Claims (8)

1. The epipolar line extraction method based on the eight-parameter epipolar line model is characterized by comprising the following steps of:
step one, acquiring three-dimensional image data covering the same target area;
matching a plurality of uniformly distributed homonymy points on the stereo image, and recording image point coordinates of the homonymy points;
step three, establishing an eight-parameter epipolar line equation of the target at the homonymous points of the left image and the right image according to the homonymous point image point coordinates, specifically:
0＝x L ′-x R ′ (1)
wherein:
Figure FDA0003758125940000011
Figure FDA0003758125940000012
wherein (x) L ，y L ) Representing column coordinates and row coordinates of the target on the left stereoscopic image; (x) R ，y R ) The column coordinate and the row coordinate of the target on the right original image are represented; (x) L ′，y L ') column coordinates and row coordinates of the object on the left epipolar line image; (x) R ′，y R ') column coordinates and row coordinates of the object on the right epipolar line image; a is 1 、b 1 、c 1 、a 2 、b 2 、c 2 Affine transformation parameters of the right epipolar line image relative to the right original image; alpha is alpha 0 Indicating the inclination, alpha, of the left epipolar line passing through a reference point on the left stereoscopic image v Representing the rate of change of the left epipolar line dip;
determining initial values of the eight parameters according to an eight-parameter epipolar equation;
step five, carrying out linearization processing on the eight-parameter epipolar equation to obtain an error equation;
step six, establishing a normal equation by using a least square method based on an error equation, and solving correction quantity of eight parameters according to the normal equation;
step seven, judging whether the correction quantity meets the convergence requirement, if so, correcting the eight parameters according to the correction quantity to obtain corrected eight parameters, if not, replacing the initial values of the eight parameters with the corrected eight parameters, and executing the steps three to six until convergence;
and step eight, calculating and generating an epipolar line of the three-dimensional image of the target area through the corrected eight parameters.
2. The epipolar line extraction method based on an eight-parameter epipolar line model according to claim 1, wherein the homonymy points in the second step are not less than four.
3. The epipolar extraction method based on the eight-parameter epipolar model according to claim 2, wherein the initial values of the eight parameters are specifically:
α 0 、α v 、a 1 、b 1 、c 1 、a 2 、b 2 、c 2 eight parameters corrected for left and right homonymic epipolar lines the initial values of the eight parameters are expressed in vector form as:
Figure FDA0003758125940000013
4. the method for extracting epipolar lines based on an eight-parameter epipolar line model according to claim 3, wherein the error equation in the fourth step is expressed as:
V＝Bx-l，P (4)
wherein, the vector is a correction number vector; x is correction quantity with eight parameters; b is a partial derivative coefficient matrix of eight parameters by an eight-parameter epipolar equation; l is an inconsistency vector, namely substituting the initial value X0 of the eight parameters into the value of an eight-parameter epipolar equation; p is a weight matrix.
5. The epipolar line extraction method based on an eight-parameter epipolar line model according to claim 4, wherein the eight-parameter correction quantity x is expressed as:
x＝[Δα 0 Δα v Δa 1 Δb 1 Δc 1 Δaa 2 Δb 2 Δc 2 ]T。
6. the epipolar line extraction method based on the eight-parameter epipolar line model according to claim 5, wherein the fifth step is based on an error equation, a normal equation is established by using a least square method, and an eight-parameter correction quantity is obtained according to the normal equation; the specific process is as follows:
establishing a normal equation:
B T PBx-B T Pl＝0 (5)
wherein T represents transpose;
let N BB ＝B T PB，W＝B T Pl, then the equation of the law is abbreviated as:
N BB x-W＝0 (6)
wherein N is BB Representing a full rank matrix, in this case eight parametersHas a unique solution to the correction quantity x of (c),
Figure FDA0003758125940000021
or x = (B) T PB) -1 B T Pl。
7. The epipolar line extraction method based on an eight-parameter epipolar line model according to claim 6, wherein step six corrects the eight parameters according to correction amounts of the eight parameters; the specific process is as follows:
X＝X 0 +x (7)
wherein X represents the eight parameters after correction.
8. The epipolar line extraction method based on an eight-parameter epipolar line model according to claim 7, wherein the convergence requirement is:
setting a threshold value 10 -8 If the absolute value of the correction number of each parameter of X is smaller than the threshold value, calculating eight parameters according to the numerical value of X at the moment;
if the absolute value of the correction number of each parameter in X is greater than or equal to the threshold value, let X = X 0 And repeating the third step to the sixth step.
CN202110460890.4A 2021-04-27 2021-04-27 Epipolar extraction method based on eight-parameter epipolar model Active CN113034572B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110460890.4A CN113034572B (en) 2021-04-27 2021-04-27 Epipolar extraction method based on eight-parameter epipolar model

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110460890.4A CN113034572B (en) 2021-04-27 2021-04-27 Epipolar extraction method based on eight-parameter epipolar model

Publications (2)

Publication Number Publication Date
CN113034572A CN113034572A (en) 2021-06-25
CN113034572B true CN113034572B (en) 2022-12-23

Family

ID=76454718

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110460890.4A Active CN113034572B (en) 2021-04-27 2021-04-27 Epipolar extraction method based on eight-parameter epipolar model

Country Status (1)

Country Link
CN (1) CN113034572B (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023147A (en) * 2016-05-06 2016-10-12 西安航天天绘数据技术有限公司 GPU-based linear array remote sensing image DSM rapid extraction method

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101598550B (en) * 2009-07-03 2011-06-15 武汉大学 Rapid rearrangement method of satellite stereoscopic image approximate epipolar line based on projection trajectory
CN101915913B (en) * 2010-07-30 2012-09-19 中交第二公路勘察设计研究院有限公司 Steady automatic matching method for high-resolution satellite image connecting points
CN103630120B (en) * 2013-07-16 2016-09-14 中国人民解放军信息工程大学 Martian surface linear array image core line method for resampling based on tight geometric model
CN106991643B (en) * 2017-03-13 2020-08-25 广东南方数码科技股份有限公司 Real-time line checking method and real-time line checking system with low resource consumption

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106023147A (en) * 2016-05-06 2016-10-12 西安航天天绘数据技术有限公司 GPU-based linear array remote sensing image DSM rapid extraction method

Also Published As

Publication number Publication date
CN113034572A (en) 2021-06-25

Similar Documents

Publication Publication Date Title
CN112102458B (en) Single-lens three-dimensional image reconstruction method based on laser radar point cloud data assistance
CN111126148B (en) DSM (digital communication system) generation method based on video satellite images
CN110853075B (en) Visual tracking positioning method based on dense point cloud and synthetic view
CN112001926B (en) RGBD multi-camera calibration method, system and application based on multi-dimensional semantic mapping
CN111724465B (en) Satellite image adjustment method and device based on plane constraint optimization virtual control point
CN108038902A (en) A kind of high-precision three-dimensional method for reconstructing and system towards depth camera
CN103198524A (en) Three-dimensional reconstruction method for large-scale outdoor scene
CN107014399A (en) A kind of spaceborne optical camera laser range finder combined system joint calibration method
CN102778224B (en) Method for aerophotogrammetric bundle adjustment based on parameterization of polar coordinates
CN109215118B (en) Incremental motion structure recovery optimization method based on image sequence
CN110660099B (en) Rational function model fitting method for remote sensing image processing based on neural network
CN104048649B (en) A kind of multi-view images and the rapid registering method of threedimensional model
CN115719320B (en) Tilt correction dense matching method based on remote sensing image
CN112070891B (en) Image area network adjustment method and system using digital ground model as three-dimensional control
CN103985154A (en) Three-dimensional model reestablishment method based on global linear method
CN115326025B (en) Binocular image measurement and prediction method for sea waves
CN115439571A (en) Method and device suitable for generating linear array push-broom satellite image epipolar image
Li et al. Research on multiview stereo mapping based on satellite video images
Liu et al. Dense stereo matching strategy for oblique images that considers the plane directions in urban areas
CN113034572B (en) Epipolar extraction method based on eight-parameter epipolar model
CN109029379A (en) A kind of high-precision stereo mapping with low base-height ratio method
CN107240133A (en) Stereoscopic vision mapping model building method
CN106875374B (en) Weak connection image splicing method based on line features
CN110555880A (en) focal length unknown P6P camera pose estimation method
CN111862332A (en) Method and system for correcting fitting error of satellite image general imaging model

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant