CN112697112B - Method and device for measuring horizontal plane inclination angle of camera - Google Patents

Method and device for measuring horizontal plane inclination angle of camera Download PDF

Info

Publication number
CN112697112B
CN112697112B CN202011588254.1A CN202011588254A CN112697112B CN 112697112 B CN112697112 B CN 112697112B CN 202011588254 A CN202011588254 A CN 202011588254A CN 112697112 B CN112697112 B CN 112697112B
Authority
CN
China
Prior art keywords
camera
calibration plate
image
motor
circle
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202011588254.1A
Other languages
Chinese (zh)
Other versions
CN112697112A (en
Inventor
史金飞
孟超
戴敏
郝飞
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202011588254.1A priority Critical patent/CN112697112B/en
Publication of CN112697112A publication Critical patent/CN112697112A/en
Application granted granted Critical
Publication of CN112697112B publication Critical patent/CN112697112B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Length Measuring Devices By Optical Means (AREA)

Abstract

A method and a device for measuring the horizontal plane inclination angle of a camera belong to the field of precision measurement and image processing. The invention controls the movement of the working platform through the motor, so that the calibration board graph enters the visual field of the camera, simultaneously adjusts the proper position of the calibration board graph in the visual field of the camera, shoots an image of one calibration board graph, then keeps the Y axis of the motor still, moves a certain step length along the X axis of the motor, shoots an image of the other calibration board graph, and because the camera is arranged on the horizontal plane, the coordinate values of the common area points in the two calibration board graph images are different, and the two shot images are processed, so that the inclination angle of the camera on the horizontal plane can be obtained. The method has the advantages that the pattern manufacturing precision of the calibration plate is high, the control precision of the motion of the working platform is also high, the calibration precision of the horizontal plane inclination angle of the camera is high, the step size can be changed aiming at the calibration of different camera views and different motion ranges, and the method has strong adaptability.

Description

Method and device for measuring horizontal plane inclination angle of camera
Technical Field
The invention belongs to the field of precision measurement and image processing, and relates to a method and a device for measuring a horizontal plane inclination angle of a camera.
Background
In industrial production application, assembly accuracy is an important index for measuring an automatic assembly technology, and calibration of a visual unit in an automatic assembly system has very important significance on the assembly accuracy and is of great importance on accurate positioning of products and reduction of assembly errors.
At present, the calibration of a visual unit in an automatic assembly system mainly comprises monocular and monocular calibration of a camera and pose calibration of the camera and a manipulator. The monocular and monocular calibration of the camera is mainly used for acquiring internal and external parameters of the camera, so that the parameters are used for size measurement, three-dimensional reconstruction and the like of an object, and the pose calibration of the camera and the manipulator is mainly used for establishing a relation between a terminal coordinate system of a camera motion mechanism and a terminal coordinate system of the manipulator, so that the pose calibration is used for identification, positioning and the like of the object.
However, after the camera is installed, the installation process of the camera may cause the camera to incline in a vertical plane and a horizontal plane, and when the camera has an inclined angle in the vertical plane, due to the limited depth of field of the camera, the acquired image may have a situation of being partly clear and partly blurred, which may greatly affect the accuracy of the result. The method comprises the steps of manufacturing a high-precision camera calibration plate, moving a camera driving mechanism to enable a calibration graph on the calibration plate to enter a camera visual field, moving a camera along an axis of camera motion in a certain step length, and shooting the calibration graph on the calibration plate, wherein due to the fact that an inclination angle exists in the vertical plane of the camera, the definition of graphs at the upper left corner, the upper right corner, the lower right corner and the lower left corner in the camera visual field is different, and the shot images are processed, so that the vertical plane inclination angle of the camera [ a method for measuring the inclination angle of the camera, 2019, application number 201910004492.4 ] can be obtained.
When the camera has an inclination angle on the horizontal plane, especially for positioning and measuring a large-size object, image stitching or coordinate transformation is generally required, and the inclination angle on the horizontal plane of the camera causes stitching or transformation errors, so that an accurate result cannot be obtained. There is currently no effective method of measuring and calibrating the tilt angle of the camera horizontal plane.
Disclosure of Invention
In order to solve the existing problems, the invention provides a method for measuring the horizontal plane inclination angle of a camera, which realizes the system angle measurement of two different planes of different planes, and obtains the inclination angle of the camera on the horizontal plane by calculating the coordinate difference of the common area point of the calibration plate in the images of the two calibration plates.
The technical scheme of the invention is as follows:
a device for measuring the horizontal plane inclination angle of a camera comprises a working platform, a motor Y axis, a calibration plate, a motor X axis, a calibration plate carrying platform, a light source, a lens, a camera, a support base and a camera support;
the camera comprises a light source, a camera support, a support base and a camera, wherein the light source is arranged on the camera, the camera is fixedly arranged on the camera support, and the camera support is fixedly connected with the support base; the calibration device comprises a light source, a calibration board carrying platform, a motor Y shaft, a motor X shaft, a motor Y shaft, a calibration board carrying platform and a motor Y shaft, wherein the light source is arranged below the calibration board carrying platform.
A method for measuring the horizontal plane inclination angle of a camera comprises the following steps:
step 1, placing a calibration plate on a calibration plate carrying platform, setting a part of the calibration plate as a public area, controlling the movement of a working platform through a moving motor, enabling the calibration plate to enter the visual field of a camera, and shooting an image of the calibration plate, wherein the image comprises the public area of the calibration plate;
step 2, keeping the Y axis of the motor still, moving the working platform by a step length delta d along the X axis of the motor, and shooting an image of another calibration plate, wherein the image comprises a public area of the calibration plate; wherein, the images of the two calibration plates have the same public area;
step 3, processing the two shot calibration plate images, wherein the horizontal plane inclination angle of the camera is determined according to the image
Figure GDA0003633511960000021
And calculating, wherein Δ V is the difference of the V coordinate values of the common region points, and Δ U is the difference of the U coordinate values of the common region points.
Preferably, the pattern on the calibration plate is circular, the radius of the standard circle is r1, two positioning circles with the radius of r2 exist on the calibration plate, the positioning circles are located in different columns, the area between the columns where the positioning circles are located is a common area of the calibration plate, the point of the common area is the center of a circle in the common area, and r2 is larger than r1.
Preferably, in the step 1, the public area of the calibration plate is moved to the right side of the camera view field, and the shot calibration plate image is marked as a Left image; and (3) keeping the Y axis of the motor still in the step (2), moving the working platform by delta d step length along the negative direction of the X axis of the motor or leftwards, so that the public area of the calibration plate is moved to the left side of the visual field of the camera, and recording the shot image of the calibration plate as a Right image.
Further preferably, in the Left image, the center a of the common area is (U) l ，V l ) (ii) a When the working platform moves delta d step length along the negative direction of the X axis of the motor, the circle center A in the original public area is A' in the Right image (U) r ，V r ) And converting the circle center A' in the Right image into a Left image coordinate system, namely:
Figure GDA0003633511960000022
then it is determined that,
Figure GDA0003633511960000023
i.e. camera horizontal plane tilt angle
Figure GDA0003633511960000024
Further preferably, the method for calculating the horizontal tilt angle of the camera by processing the two captured images of the calibration board in step 3 is as follows:
3.1, calculating all circle parameters C (r, U and V) in the two images by using a circle Hough transformation method according to the Left image obtained in the step 1 and the Right image obtained in the step 2; wherein r represents the radius of the circle;
3.2, sorting the circle parameters in each image from small to large according to the radius, wherein the first two circles with the largest radius are positioning circles, dividing a common area according to the two positioning circles in each image, calculating the coordinates of the circle centers of all circles in the common area, and recording the circle center in the Left image as the circle center
Figure GDA0003633511960000025
The center of a circle in the Right image is
Figure GDA0003633511960000026
Wherein k is a subscript of the circle center, and N represents all circle centers in the public area;
step 3.3, calculating the horizontal plane inclination angle of the camera according to the corresponding coordinates of the circle center in the Left image and the Right image in the public area
Figure GDA0003633511960000031
Step 3.4 θ obtained according to step 3.3 k The average value of the camera horizontal tilt angle can be calculated as:
Figure GDA0003633511960000032
advantageous effects
The graphic manufacturing precision of the calibration plate can reach 1 micron, and the moving precision of the working platform can be less than 1 micron, so that the calibration precision is high; aiming at the calibration of different camera views and different motion ranges, the step size can be changed, and the method has strong adaptability; the invention is non-contact measurement, and has no damage to the calibration plate; the horizontal plane inclination angle obtained after calibration plays an important role in establishing an error model of a high-precision assembly system, positioning a large-size object, splicing images and transforming coordinates; the method has the advantages of simple and easy operation of the measuring process, simple and easy use of the design of the calibration plate, high precision of the measuring result and capability of meeting the requirement of calculating the planning coordinate of the image acquisition path.
Drawings
FIG. 1 is a schematic view of an apparatus for measuring a tilt angle of a camera in a horizontal plane according to the present invention;
FIG. 2 is a graphical illustration of a calibration plate according to the present invention;
FIG. 3 is a schematic diagram of calibration plate graphic image acquisition in accordance with the present invention;
FIG. 4 is a schematic diagram of calculating the horizontal tilt angle of the camera at the point of the common area according to the present invention.
Fig. 5 is a Left and Right image of the calibration board of the present embodiment.
In the figure, 1 work platform, 2 motor Y-axis, 3 calibration plate, 4 motor X-axis, 5 calibration plate carrier, 6 light source, 7 lens, 8 camera, 9 support base and 10 camera support.
Detailed Description
The invention is described in further detail below with reference to the accompanying drawings:
the invention provides a method for measuring the horizontal plane inclination angle of a camera, which controls a working platform to move through a motor, so that a calibration board graph enters the field of view of the camera, simultaneously adjusts the proper position of the calibration board graph in the field of view of the camera, shoots an image of the calibration board graph, then keeps a Y axis of the motor still, moves for a certain step length along an X axis of the motor, shoots an image of the other calibration board graph, and because the camera is installed on a horizontal plane, the coordinate values of common area points in the two calibration board graph images are different, and the two shot images are processed, so that the inclination angle of the camera on the horizontal plane can be obtained.
As shown in fig. 1, a device for measuring the horizontal inclination angle of a camera includes a working platform 1, a motor Y-axis 2, a calibration plate 3, a motor X-axis 4, a calibration plate carrier 5, a light source 6, a lens 7, a camera 8, a support base 9 and a camera support 10;
the light source 6 is mounted on a lens 7, the lens 7 is mounted on a camera 8, the camera 8 is fixedly mounted on a camera support 10, and the camera support 10 is fixedly connected with a support base 9; the calibration device is characterized in that a working platform 1 is arranged below the light source 6, a motor X shaft 4 is fixedly connected to the middle of the working platform 1, the motor X shaft 4 is in sliding connection with a motor Y shaft 2, and the calibration board carrying platform 5 is in sliding connection with the motor Y shaft 2.
Example (b):
a method for measuring the horizontal plane inclination angle of a camera comprises the following steps:
step 1, placing a calibration plate on a calibration plate carrying platform 5, setting a part of the calibration plate as a public area, controlling the movement of a working platform 1 through a moving motor, enabling the calibration plate to enter the visual field of a camera, and shooting an image of the calibration plate, wherein the image comprises the public area of the calibration plate;
step 2, keeping a Y axis 2 of the motor still, moving a working platform 1 by a step length delta d along an X axis 4 of the motor, and shooting an image of another calibration plate, wherein the image comprises a public area of the calibration plate; wherein, the images of the two calibration plates have the same public area;
step 3, processing the two shot calibration plate images, wherein the horizontal plane inclination angle of the camera is according to
Figure GDA0003633511960000041
And calculating, wherein Δ V is the difference of the V coordinate values of the common area points, and Δ U is the difference of the U coordinate values of the common area points.
As shown in fig. 2, the pattern on the calibration plate is circular, the calibration plate is made of glass material, the pattern on the calibration plate is made by electron beam lithography, the size deviation of the pattern is less than ± 0.5 micrometer, the thickness of the calibration plate is 1.6mm, the rectangular width and length of the calibration plate are 10mm × 20mm, the radius of the standard Circle is r1=0.5mm, the Circle distance is 1.5mm, two positioning circles with radius of r2=1mm are also present on the calibration plate, the common region where the positioning circles are located and the region between the positioning circles are 4x4, and the common region is used for calculating the inclination angle of the horizontal plane of the camera.
As shown in fig. 3, the Public area Public Region of the calibration board is first moved to the Right side of the camera view, the captured calibration board image is recorded as a Left image, and then the motor Y-axis 2 is kept still, and the calibration board image is moved by Δ d =5mm step length in the negative direction of the motor X-axis 4 or Left direction (equivalent to that the calibration board is still, and the camera is moved by Δ d =5mm step length in the positive direction of the X-axis or Right direction), so that the Public area Public Region of the calibration board is moved to the Left side of the camera view, and the captured calibration board image is recorded as a Right image.
In fig. 4, it is assumed that θ is counterclockwise (θ > 0), and in the Left image, the center a of the circle in the common area is (U) l ，V l ) (ii) a When the working platform 1 moves along the negative direction of the X axis 4 of the motor by delta d =5mm step length, the circle center A in the original common area is A' in the Right image (U) r ，V r ) And converting the circle center A' in the Right image into a Left image coordinate system, namely:
Figure GDA0003633511960000051
then it is determined that,
Figure GDA0003633511960000052
i.e. camera horizontal plane tilt angle
Figure GDA0003633511960000053
When theta is clockwise (theta)<0) In the Left image, the center of a circle A in the common area is (U) l ，V l ) (ii) a When the working platform 1 moves along the negative direction of the X axis 4 of the motor by delta d =5mm step length, the circle center A in the original common area is A' in the Right image (U) r ，V r ) And converting the circle center A' in the Right image into a Left image coordinate system, namely:
Figure GDA0003633511960000054
then it is determined that,
Figure GDA0003633511960000055
i.e. camera horizontal plane tilt angle
Figure GDA0003633511960000056
The method for calculating the horizontal tilt angle of the camera by processing the two captured calibration plate images in step 3 is as follows, as shown in fig. 5:
step 3.1, calculating all circle parameters C (r, U and V) in the two images by using a circle Hough transformation method according to the Left image obtained in the step 1 and the Right image obtained in the step 2; wherein r represents the radius of the circle;
step 3.2, setting the common area point in the Left image as
Figure GDA0003633511960000057
i and j respectively represent the row and the column of the common area, C is sorted from large to small according to the radius, and the first two elements are taken as the circle parameters of the two positioning circles
Figure GDA0003633511960000058
And
Figure GDA0003633511960000059
if it is not
Figure GDA00036335119600000510
Then
Figure GDA00036335119600000511
If not, then the mobile terminal can be switched to the normal mode,
Figure GDA00036335119600000512
m and n are subscripts of the two positioning circles respectively;
step 3.3, remove C l L in (1) 21 And L 34 If, if
Figure GDA00036335119600000513
(T p A coordinate difference threshold for a circle parameter), then if
Figure GDA00036335119600000514
Then L is obtained 11 If, if
Figure GDA00036335119600000515
Then L is obtained 31 And L 41 Likewise, L can be obtained 14 、 L 24 And L 44 ；
Step 3.4, remove C l L in (1) 11 、L 31 、L 41 、L 14 、L 24 And L 44 If it is determined that
Figure GDA00036335119600000516
Then if
Figure GDA00036335119600000517
Then according to
Figure GDA00036335119600000518
Can be given L 12 And L 13 Likewise, L can be obtained 22 、L 23 、L 32 、L 33 、L 42 And L 43 ；
Step 3.5, similarly, the common area point in the Right image corresponding to the Left image one by one can be obtained from steps 3.2, 3.3 and 3.4 as R ij (r rij ,U rij ,V rij )；
Step 3.6, which can be obtained from step 3.5, the horizontal plane inclination angle of the camera is:
Figure GDA0003633511960000061
Figure GDA0003633511960000062
the above description is only a preferred embodiment of the present invention, and is not intended to limit the present invention in any way, but any modifications or equivalent variations made according to the technical spirit of the present invention are within the scope of the present invention as claimed.

Claims (2)

1. The method for measuring the horizontal plane inclination angle of the camera is characterized in that the used device for measuring the horizontal plane inclination angle of the camera comprises a working platform, a motor Y axis, a calibration plate, a motor X axis, a calibration plate carrying platform, a light source, a lens, the camera, a support base and a camera support;
the camera comprises a light source, a camera support, a support base and a camera, wherein the light source is arranged on the camera, the camera is fixedly arranged on the camera support, and the camera support is fixedly connected with the support base; a working platform is arranged below the light source, a motor X shaft is fixedly connected in the middle of the working platform, the motor X shaft is in sliding connection with a motor Y shaft, and the calibration board carrying platform is in sliding connection with the motor Y shaft;
the measuring method comprises the following steps:
step 1, placing a calibration plate on a calibration plate carrying platform, setting a part of the calibration plate as a public area, controlling the movement of a working platform through a moving motor, enabling the calibration plate to enter the visual field of a camera, and shooting an image of the calibration plate, wherein the image comprises the public area of the calibration plate;
firstly, moving a public area of a calibration plate to the right side of the visual field of a camera, and recording a shot calibration plate image as a Left image;
step 2, keeping the Y axis of the motor still, moving the working platform along the X axis of the motor by a step length delta d, and shooting an image of another calibration plate, wherein the image comprises a public area of the calibration plate; wherein, the images of the two calibration plates have the same public area;
keeping the Y axis of the motor still, moving the working platform by delta d step length along the negative direction of the X axis of the motor or leftwards, moving the public area of the calibration plate to the left side of the visual field of the camera, and recording the shot image of the calibration plate as a Right image;
in the Left image, the center of a circle A in the common area is (U) l ，V l ) (ii) a When the working platform moves delta d step length along the negative direction of the X axis of the motor, the circle center A in the original public area is A' in the Right image (U) r ，V r ) And converting the circle center A' in the Right image into a Left image coordinate system, namely:
Figure FDA0003633511950000011
then the user can either, for example,
Figure FDA0003633511950000012
i.e. camera horizontal plane tilt angle
Figure FDA0003633511950000013
Step 3, processing the two shot calibration plate images, wherein the horizontal plane inclination angle of the camera is according to
Figure FDA0003633511950000014
Calculating, wherein Δ V is the difference of V coordinate values of the common region points, and Δ U is the difference of U coordinate values of the common region points;
the method comprises the following specific steps:
step 3.1, calculating all circle parameters C (r, U and V) in the two images by using a circle Hough transformation method according to the Left image obtained in the step 1 and the Right image obtained in the step 2; wherein r represents the radius of the circle;
3.2, sorting the circle parameters in each image from small to large according to the radius, wherein the first two circles with the largest radius are positioning circles, dividing a common area according to the two positioning circles in each image, calculating the coordinates of the circle centers of all circles in the common area, and recording the circle center in the Left image as the circle center
Figure FDA0003633511950000023
The center of a circle in the Right image is
Figure FDA0003633511950000024
Wherein k is a subscript of the circle center, and N represents all circle centers in the public area;
step 3.3, calculating the horizontal plane inclination angle of the camera according to the corresponding coordinates of the circle center in the Left image and the Right image in the public area
Figure FDA0003633511950000021
Step 3.4 θ obtained according to step 3.3 k The average value of the camera horizontal plane tilt angle can be calculated as:
Figure FDA0003633511950000022
2. the method of measuring the tilt angle of the horizontal plane of a camera according to claim 1, wherein: the pattern on the calibration plate is circular, the radius of the standard circle is r1, two positioning circles with the radius of r2 exist on the calibration plate, the positioning circles are in different rows, the area where the positioning circles are located and the area between the rows are the common area of the calibration plate, the point of the common area is the center of a circle in the common area, and r2 is larger than r1.
CN202011588254.1A 2020-12-29 2020-12-29 Method and device for measuring horizontal plane inclination angle of camera Active CN112697112B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011588254.1A CN112697112B (en) 2020-12-29 2020-12-29 Method and device for measuring horizontal plane inclination angle of camera

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011588254.1A CN112697112B (en) 2020-12-29 2020-12-29 Method and device for measuring horizontal plane inclination angle of camera

Publications (2)

Publication Number Publication Date
CN112697112A CN112697112A (en) 2021-04-23
CN112697112B true CN112697112B (en) 2022-10-14

Family

ID=75511408

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202011588254.1A Active CN112697112B (en) 2020-12-29 2020-12-29 Method and device for measuring horizontal plane inclination angle of camera

Country Status (1)

Country Link
CN (1) CN112697112B (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114935309A (en) * 2022-04-02 2022-08-23 杭州汇萃智能科技有限公司 Method, system and readable storage medium for correcting installation error in machine vision measurement
CN115930784B (en) * 2023-01-09 2023-08-25 广州市易鸿智能装备有限公司 Point inspection method of visual inspection system
CN116503493B (en) * 2023-06-27 2023-10-20 季华实验室 Multi-camera calibration method, high-precision equipment and computer readable storage medium
CN117268342B (en) * 2023-09-22 2024-03-29 合肥筑创空间建筑设计有限公司 Auxiliary mapping device for scene simulation distance

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN100384220C (en) * 2006-01-17 2008-04-23 东南大学 Video camera rating data collecting method and its rating plate
CN106341956B (en) * 2016-09-30 2019-04-30 哈尔滨工业大学 A kind of fixed camera bearing calibration
CN109300162B (en) * 2018-08-17 2021-08-03 浙江工业大学 Multi-line laser radar and camera combined calibration method based on refined radar scanning edge points
CN110148174A (en) * 2019-05-23 2019-08-20 北京阿丘机器人科技有限公司 Scaling board, scaling board recognition methods and device
CN111652942B (en) * 2020-05-29 2024-03-22 维沃移动通信有限公司 Calibration method of camera module, first electronic equipment and second electronic equipment

Also Published As

Publication number Publication date
CN112697112A (en) 2021-04-23

Similar Documents

Publication Publication Date Title
CN112697112B (en) Method and device for measuring horizontal plane inclination angle of camera
CN110497187B (en) Sun flower pattern assembly system based on visual guidance
CN111880164B (en) Laser radar calibration device and method
CN107942949B (en) A kind of lathe vision positioning method and system, lathe
CN112381827B (en) Rapid high-precision defect detection method based on visual image
CN105427288A (en) Calibration method and device of machine vision alignment system
US20220092330A1 (en) Image processing device, work robot, substrate inspection device, and specimen inspection device
Wang et al. Error analysis and improved calibration algorithm for LED chip localization system based on visual feedback
US10535157B2 (en) Positioning and measuring system based on image scale
CN112365502B (en) Calibration method based on visual image defect detection
CN114331924B (en) Large workpiece multi-camera vision measurement method
CN113496523A (en) System and method for three-dimensional calibration of visual system
CN115555648A (en) Numerical control cutting method, system, equipment and medium
CN113189568A (en) Laser radar calibration device and method
CN114708164B (en) Method for correcting image large and small head distortion caused by object inclination in machine vision measurement
CN109541626B (en) Target plane normal vector detection device and detection method
CN106735869B (en) The contactless localization method of laser vision for numerically controlled processing equipment
CN109472778B (en) Appearance detection method for towering structure based on unmanned aerial vehicle
CN205333535U (en) Metal strainometer defect automatic check out system
CN111983896B (en) High-precision alignment method for 3D exposure machine
CN110658215B (en) PCB automatic splicing detection method and device based on machine vision
CN112082511A (en) Large object rapid calibration and measurement method based on rotary table
CN109506626A (en) A kind of measurement method of camera tilt angles
CN113592962B (en) Batch silicon wafer identification recognition method based on machine vision
CN114535825B (en) Laser marking vehicle identification code system based on manipulator

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant