CN113538594A - Vehicle-mounted camera calibration method based on direction sensor - Google Patents
Vehicle-mounted camera calibration method based on direction sensor Download PDFInfo
- Publication number
- CN113538594A CN113538594A CN202110738903.XA CN202110738903A CN113538594A CN 113538594 A CN113538594 A CN 113538594A CN 202110738903 A CN202110738903 A CN 202110738903A CN 113538594 A CN113538594 A CN 113538594A
- Authority
- CN
- China
- Prior art keywords
- vehicle
- camera
- calibrated
- reference sample
- calibration
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Studio Devices (AREA)
Abstract
The invention discloses a vehicle-mounted camera calibration method based on a direction sensor, which comprises the following steps: s1, selecting a calibration site; s2, selecting a mark; s3, parking the reference sample car in the calibration site; s4, placing marks at the splicing lines of the cameras of the reference sample car; s5, acquiring the attitude data of the reference sample car, and further acquiring a reference value of the camera arrangement attitude angle of the reference sample car; collecting marked images shot by each camera; s6, parking the vehicle to be calibrated in the calibration site; s7, acquiring attitude data of the vehicle to be calibrated, and further acquiring an actual value of the arrangement attitude angle of the camera of the vehicle to be calibrated; s8, correcting the camera attitude angle of the vehicle to be calibrated according to the reference value and the actual value of the camera arrangement attitude angle; and S9, correcting the real-time image shot by the camera of the vehicle to be calibrated according to the mark image obtained by the reference sample vehicle. The vehicle-mounted camera calibration method realizes vehicle-mounted camera calibration under the conditions of lower site requirements and no additional sensor.
Description
Technical Field
The invention relates to the field of vehicle-mounted camera calibration, in particular to a vehicle-mounted camera calibration method based on a direction sensor.
Background
In a traditional vehicle-mounted camera calibration scheme, a series of ground marks need to be arranged on a flat ground, a vehicle is further parked at a specific position in a mark area, an image of a camera is collected and compared with a standard image built in the camera, and calibration parameters are calculated. The conventional scheme has the following disadvantages: 1. the ground condition of a calibration site, the coating precision of a ground mark and the maintenance requirements are high; 2. the parking position of the vehicle in the marking area is high, which brings high process cost especially for a production line running at high speed; 3. even on the premise of accurate calibration site and vehicle parking, the calibration effect still theoretically cannot reach mathematical accuracy due to slight change of the posture of each vehicle caused by accumulation of various mechanical parts of the whole vehicle.
Disclosure of Invention
The invention aims to provide a vehicle-mounted camera calibration method based on a direction sensor so as to realize vehicle-mounted camera calibration under the condition of weakening dependence of an external field.
In order to solve the technical problem, the invention provides a technical scheme that: a calibration method for a vehicle-mounted camera of a direction sensor comprises the following steps:
s1, selecting a calibration site, wherein the calibration site is a flat ground, and the front of the calibration site faces a specified direction;
s2, selecting a mark, wherein the mark is a pattern with certain characteristics and rules;
s3, driving the reference sample vehicle into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the reference sample vehicle, and parking;
s4, respectively placing marks at the splicing lines of the cameras of the reference sample car, and enabling the marks to be away from the car body by the distance of one standard unit;
s5, acquiring attitude data of the reference sample car by using the vehicle-mounted direction sensor of the reference sample car, obtaining a reference value of the camera arrangement attitude angle of the reference sample car by resolving, and recording the reference value; collecting a marked image shot by a camera;
s6, driving the vehicle to be calibrated into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the vehicle to be calibrated, and parking;
s7, acquiring attitude data of the vehicle to be calibrated respectively by using a vehicle-mounted direction sensor of the vehicle to be calibrated, and calculating to obtain an actual value of the arrangement attitude angle of the camera of the vehicle to be calibrated;
s8, correcting the attitude angle according to the reference value of the camera arrangement attitude angle and the actual value of the camera arrangement attitude angle;
and S9, according to the marked image shot by the reference sample vehicle camera, carrying out zoom correction and translation correction on the real-time image shot by the vehicle camera to be calibrated.
According to the scheme, the mark is an equilateral triangle with the side length being one standard unit.
According to the scheme, the standard unit is 1 m.
According to the above scheme, the zoom correction and the pan correction in S9 specifically include:
obtaining an image point displacement formula generated by the exterior orientation element according to a conformation equation:
wherein X and y are image plane coordinates, f is focal length, and Xs、Ys、ZsThe coordinate is a ground object coordinate, H is the height from the origin of the image plane coordinate to the ground, and psi, omega and kappa are respectively the rotation angles around the y axis, the x axis and the ground plane normal;
according to the scheme, the translation correction in the step S9 is specifically as follows: and performing translation correction on one camera of the vehicle to be calibrated, and then performing translation correction on the next camera by taking the image shot by the corrected camera as a reference so as to finish the correction of all the cameras.
An automobile comprising a processor, a memory and a computer program stored on said memory and executable on said processor, said computer program, when executed by said processor, implementing the steps of the orientation sensor based on in-vehicle camera calibration method as described above.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the orientation sensor based vehicle camera calibration method as described above.
The invention has the beneficial effects that: acquiring attitude data of a reference sample vehicle and a vehicle to be calibrated by using a vehicle-mounted direction sensor so as to acquire a reference value of a camera arrangement attitude angle of the reference sample vehicle and an actual value of the camera arrangement attitude angle of the vehicle to be calibrated, so as to correct the camera attitude angle of the vehicle to be calibrated, and performing zoom correction and translation correction on the vehicle to be calibrated according to acquired marked images of the reference sample vehicle and the vehicle to be calibrated; compared with the traditional vehicle-mounted camera calibration scheme, the method and the device have the advantages that the requirements on external fields, equipment and testing personnel are weakened, and the calibration cost of vehicle-mounted camera calibration is reduced.
Drawings
FIG. 1 is a diagram of the elements outside the constellation equation in accordance with one embodiment of the present invention;
FIG. 2 is a flowchart illustrating an attitude angle calibration process according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of zoom correction and pan correction according to an embodiment of the present invention;
FIG. 4 is a schematic view of a translational correction according to an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings of the embodiments of the present disclosure. It is to be understood that the described embodiments are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the disclosure without any inventive step, are within the scope of protection of the disclosure.
A calibration method for a vehicle-mounted camera based on a direction sensor comprises the following steps:
s1, selecting a calibration site, wherein the calibration site is a flat ground, and the front of the calibration site faces a specified direction;
s2, selecting a mark, wherein the mark is a pattern with certain characteristics and rules;
s3, driving the reference sample vehicle into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the reference sample vehicle, and parking;
s4, respectively placing marks at the splicing lines of the cameras of the reference sample car, and enabling the marks to be away from the car body by the distance of one standard unit;
s5, acquiring attitude data of the reference sample car by using the vehicle-mounted direction sensor of the reference sample car, obtaining a reference value of the camera arrangement attitude angle of the reference sample car by resolving, and recording the reference value; collecting a marked image shot by a camera;
s6, driving the vehicle to be calibrated into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the vehicle to be calibrated, and parking;
s7, acquiring attitude data of the vehicle to be calibrated respectively by using a vehicle-mounted direction sensor of the vehicle to be calibrated, and calculating to obtain an actual value of the arrangement attitude angle of the camera of the vehicle to be calibrated;
s8, correcting the attitude angle according to the reference value of the camera arrangement attitude angle and the actual value of the camera arrangement attitude angle;
s9, according to the marked image shot by the reference sample vehicle camera, carrying out zoom correction and translation correction on the real-time image shot by the vehicle camera to be calibrated;
wherein the attitude data includes the three-dimensional rotation and true azimuth acquired by the accelerometer and magnetometer onboard the vehicle.
Further, the marks are equilateral triangles with a side length of one standard unit.
Further, the standard unit is 1 m.
Further, the zoom correction and the pan correction in S9 are specifically:
obtaining an image point displacement formula generated by the exterior orientation element according to a conformation equation:
wherein X and y are image plane coordinates, f is camera focal length, and Xs、Ys、ZsThe coordinate is a ground object coordinate, H is the height from the origin of the image plane coordinate to the ground, and psi, omega and kappa are respectively the rotation angles around the y axis, the x axis and the ground plane normal;
when only attitude angle correction is performed, the influence of translation is ignored, Xs、Ys、ZsTaking 0 out of the variation amount of (1);
when only the translational correction is performed, the influence of the attitude angle is ignored, and the variation amounts of ψ, ω, κ take 0.
Further, the translation correction in S9 is specifically: and performing translation correction on one camera of the vehicle to be calibrated, and then performing translation correction on the next camera by taking the image shot by the corrected camera as a reference so as to finish the correction of all the cameras.
An automobile comprising a processor, a memory and a computer program stored on said memory and executable on said processor, said computer program, when executed by said processor, implementing the steps of the orientation sensor based on in-vehicle camera calibration method as described above.
A computer-readable storage medium, having stored thereon a computer program which, when being executed by a processor, carries out the steps of the orientation sensor based vehicle camera calibration method as described above.
For ease of understanding, the scheme in this example is further explained:
1) and a direction sensor.
The direction Sensor (Orientation Sensor) is a necessary component of modern vehicles, particularly automobiles equipped with a vehicle body stabilizing system or an intelligent driving system, and can be conveniently reused in the invention; or directly integrated inside the camera to be calibrated. And solving the included angles (psi, omega and kappa) of the object in the three-dimensional space in 3 axial directions by detecting data, wherein the unit is degree. From the physical aspect, the sub-sensors comprise: accelerometer (Accelerometer), magnetometer (geographical Field Sensor).
An Accelerometer (Accelerometer) is used to detect movement in the axial direction. Acceleration can be measured by measuring the axial force according to the law of inertia. Accelerometers typically comprise 3 axes, with the output being axial acceleration (including the effects of gravitational acceleration) in m/s2。
Magnetometers (geographic Field sensors) are used to detect ambient magnetic fields, often in practice directly as a form of electronic compass. From the geomagnetic field data, the angle in the horizontal direction with respect to the ground can be measured, and the output is the true azimuth angle (i.e., the angle in the clockwise direction with respect to north and south), in degrees.
2) The calibration site is only needed to be arranged on a flat ground and simply marked.
Because the field of vision is assisted by the vehicle-mounted camera, the field of vision is specific to the ground object or character, and the image calibration of the field of vision is based on the flat ground.
To facilitate reading the k angle of the exterior orientation element directly from the magnetometer, the front of the field should be oriented to true north. Meanwhile, the magnetometer outputs the compass to the screen and serves as a center line to guide parking.
The marks should be provided with patterns with obvious characteristics and rules, and considering that the size order of the vehicle body is meter-level, and the visual field range after the vehicle-mounted panoramic system is spliced is generally 2 m-3 m, so that an equilateral triangle with the side length of 1m is adopted in the embodiment (actually, the shapes of the marks are clear, and no precise requirements are required on the outlines of the marks, so that the difficulty of site construction is reduced). For a panoramic system with front/rear/left/right camera splicing, 1 panoramic camera needs to be placed on the ground 1m away from a vehicle body in four directions respectively, namely on splicing lines in the four directions respectively.
3) And calibrating by reversely solving the deviation of the elements of the exterior orientation of the camera.
The outer orientation element is used to characterize the spatial position of the camera with respect to the external reference frame, i.e. at the object side (corresponding to the inner orientation element, at the image side), and contains 6 parameters (x, y, z, ψ, ω, κ), which respectively represent 3 translation amounts and 3 rotation amounts along 3 orthogonal axes in three-dimensional space.
Firstly, directly measuring 3 rotation amounts through a direction sensor, comparing the rotation amounts with a default value to calculate a deviation value and correcting the deviation value, wherein the obtained image only has zooming and translation deformation;
then, a zooming deviation value is calculated and corrected by collecting the triangular mark in the actual environment and comparing the triangular mark with a default value, and the obtained image only has translational deformation;
then, the forward-looking camera is usually arranged in the middle and fixed on a front bumper, and is slightly influenced by translation actually; the rear-view camera is fixed on the back door moving part and is always offset to one side, the actual translational influence is large, and the picture quality of two sides is different, so the front-view camera is selected as an initial value, the translational deviation value is calculated and corrected by sequentially referring to the homonymous triangular marks in the previous visual angle according to the sequence of front, right, left and back (assuming that the rear-view camera is arranged to be offset to the left), and finally the calibration is finished.
The above description is only an embodiment of the present invention, and not intended to limit the scope of the present invention, and all modifications of equivalent structures and equivalent processes, which are made by using the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (7)
1. A calibration method for a vehicle-mounted camera based on a direction sensor is characterized by comprising the following steps: the method comprises the following steps:
s1, selecting a calibration site, wherein the calibration site is a flat ground, and the front of the calibration site faces a specified direction;
s2, selecting a mark, wherein the mark is a pattern with certain characteristics and rules;
s3, driving the reference sample vehicle into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the reference sample vehicle, and parking;
s4, respectively placing marks at the splicing lines of the cameras of the reference sample car, and enabling the marks to be away from the car body by the distance of one standard unit;
s5, acquiring attitude data of the reference sample car by using the vehicle-mounted direction sensor of the reference sample car, obtaining a reference value of the camera arrangement attitude angle of the reference sample car by resolving, and recording the reference value; collecting a marked image shot by a camera;
s6, driving the vehicle to be calibrated into a calibration site from the specified direction according to the vehicle-mounted direction sensor carried by the vehicle to be calibrated, and parking;
s7, acquiring attitude data of the vehicle to be calibrated respectively by using a vehicle-mounted direction sensor of the vehicle to be calibrated, and calculating to obtain an actual value of the arrangement attitude angle of the camera of the vehicle to be calibrated;
s8, correcting the attitude angle according to the reference value of the camera arrangement attitude angle and the actual value of the camera arrangement attitude angle;
and S9, according to the marked image shot by the reference sample vehicle camera, carrying out zoom correction and translation correction on the real-time image shot by the vehicle camera to be calibrated.
2. The orientation sensor-based vehicle camera calibration method according to claim 1, wherein: the marks are equilateral triangles with a side length of one standard unit.
3. The orientation sensor-based on-vehicle camera calibration method according to claim 1 or 2, wherein: one standard unit is 1 m.
4. The orientation sensor-based vehicle camera calibration method according to claim 1, wherein: the scaling correction and the translation correction of S9 specifically include:
obtaining an image point displacement formula generated by the exterior orientation element according to a conformation equation:
wherein X and y are image plane coordinates, f is focal length, and Xs、Ys、ZsThe coordinate is a ground object coordinate, H is the height from the origin of the image plane coordinate to the ground, and psi, omega and kappa are respectively the rotation angles around the y axis, the x axis and the ground plane normal.
5. The orientation sensor-based vehicle camera calibration method according to claim 1, wherein: s9 the translation correction specifically includes: and performing translation correction on one camera of the vehicle to be calibrated, and then performing translation correction on the next camera by taking the image shot by the corrected camera as a reference so as to finish the correction of all the cameras.
6. An automobile, characterized in that: comprising a processor, a memory and a computer program stored on said memory and executable on said processor, said computer program realizing the steps of the orientation sensor based on vehicle camera calibration method according to any of the claims 1 to 5 when executed by said processor.
7. A computer-readable storage medium characterized by: the computer readable storage medium has stored thereon a computer program which, when being executed by a processor, carries out the steps of the method for orientation sensor based on a vehicle camera calibration according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110738903.XA CN113538594B (en) | 2021-06-30 | 2021-06-30 | Vehicle-mounted camera calibration method based on direction sensor |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110738903.XA CN113538594B (en) | 2021-06-30 | 2021-06-30 | Vehicle-mounted camera calibration method based on direction sensor |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113538594A true CN113538594A (en) | 2021-10-22 |
| CN113538594B CN113538594B (en) | 2022-04-22 |
Family
ID=78126334
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110738903.XA Active CN113538594B (en) | 2021-06-30 | 2021-06-30 | Vehicle-mounted camera calibration method based on direction sensor |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113538594B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101750619A (en) * | 2010-01-18 | 2010-06-23 | 武汉大学 | Method for directly positioning ground target by self-checking POS |
| CN101763640A (en) * | 2009-12-31 | 2010-06-30 | 无锡易斯科电子技术有限公司 | Online calibration processing method for vehicle-mounted multi-view camera viewing system |
| US20110115922A1 (en) * | 2009-11-17 | 2011-05-19 | Fujitsu Limited | Calibration apparatus and calibration method |
| CN102982526A (en) * | 2011-06-01 | 2013-03-20 | 哈曼贝克自动系统股份有限公司 | Method of calibrating a vehicle vision system and vehicle vision system |
| CN103871071A (en) * | 2014-04-08 | 2014-06-18 | 北京经纬恒润科技有限公司 | Method for camera external reference calibration for panoramic parking system |
| CN107146255A (en) * | 2017-04-05 | 2017-09-08 | 纵目科技（上海）股份有限公司 | Panoramic picture error calibration method and device |
| CN107481292A (en) * | 2017-09-05 | 2017-12-15 | 百度在线网络技术（北京）有限公司 | The attitude error method of estimation and device of vehicle-mounted camera |
| CN112653889A (en) * | 2020-12-23 | 2021-04-13 | 北汽福田汽车股份有限公司 | Camera self-calibration method and device, camera equipment and vehicle |
| CN112702532A (en) * | 2020-12-29 | 2021-04-23 | 佛山科学技术学院 | Control method and device for autonomous image acquisition of unmanned vehicle |
| CN113002531A (en) * | 2021-02-08 | 2021-06-22 | 深圳市海沃德科技有限公司 | Vehicle panoramic looking-around method, system, equipment and computer readable storage medium |
-
2021
- 2021-06-30 CN CN202110738903.XA patent/CN113538594B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110115922A1 (en) * | 2009-11-17 | 2011-05-19 | Fujitsu Limited | Calibration apparatus and calibration method |
| CN101763640A (en) * | 2009-12-31 | 2010-06-30 | 无锡易斯科电子技术有限公司 | Online calibration processing method for vehicle-mounted multi-view camera viewing system |
| CN101750619A (en) * | 2010-01-18 | 2010-06-23 | 武汉大学 | Method for directly positioning ground target by self-checking POS |
| CN102982526A (en) * | 2011-06-01 | 2013-03-20 | 哈曼贝克自动系统股份有限公司 | Method of calibrating a vehicle vision system and vehicle vision system |
| CN103871071A (en) * | 2014-04-08 | 2014-06-18 | 北京经纬恒润科技有限公司 | Method for camera external reference calibration for panoramic parking system |
| CN107146255A (en) * | 2017-04-05 | 2017-09-08 | 纵目科技（上海）股份有限公司 | Panoramic picture error calibration method and device |
| CN107481292A (en) * | 2017-09-05 | 2017-12-15 | 百度在线网络技术（北京）有限公司 | The attitude error method of estimation and device of vehicle-mounted camera |
| CN112653889A (en) * | 2020-12-23 | 2021-04-13 | 北汽福田汽车股份有限公司 | Camera self-calibration method and device, camera equipment and vehicle |
| CN112702532A (en) * | 2020-12-29 | 2021-04-23 | 佛山科学技术学院 | Control method and device for autonomous image acquisition of unmanned vehicle |
| CN113002531A (en) * | 2021-02-08 | 2021-06-22 | 深圳市海沃德科技有限公司 | Vehicle panoramic looking-around method, system, equipment and computer readable storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113538594B (en) | 2022-04-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109767475B (en) | External parameter calibration method and system for sensor | |
| CN110501024B (en) | Measurement error compensation method for vehicle-mounted INS/laser radar integrated navigation system | |
| CN101216937B (en) | Parameter calibration method for moving containers on ports | |
| US8310684B2 (en) | System and method for localizing a carrier, estimating a posture of the carrier and establishing a map | |
| DE102017207696B4 (en) | Calibration device and calibration method | |
| CN110411457B (en) | Positioning method, system, terminal and storage medium based on stroke perception and vision fusion | |
| CN105594203B (en) | The method of adjustment of vehicle-surroundings image display device and camera | |
| EP2990765A1 (en) | Image processing device, image processing method, and program therefor | |
| DE10229336A1 (en) | Method and device for calibrating image sensor systems | |
| JP2008509619A (en) | Method for automatically calibrating stereo vision equipment | |
| CN104641394A (en) | Image processing device and storage medium | |
| CN109345591B (en) | Vehicle posture detection method and device | |
| CN109118547A (en) | Multi-cam combined calibrating system and method | |
| CN112489136B (en) | Calibration method, position determination device, electronic equipment and storage medium | |
| CN110458885B (en) | Positioning system and mobile terminal based on stroke perception and vision fusion | |
| WO2016129704A1 (en) | Onboard camera calibration apparatus for identifying mounting orientation | |
| CN106709222B (en) | IMU drift compensation method based on monocular vision | |
| CN111243029A (en) | Calibration method and device of vision sensor | |
| CN110956665A (en) | Vehicle turning track bidirectional calculation method, system and device | |
| CN115641385A (en) | Vehicle-mounted panoramic camera calibration method, device, equipment and medium | |
| CN112819711A (en) | Monocular vision-based vehicle reverse positioning method utilizing road lane line | |
| CN113538594B (en) | Vehicle-mounted camera calibration method based on direction sensor | |
| CN110426011A (en) | Vehicular turn angle measuring system and method | |
| CN111563936A (en) | Camera external parameter automatic calibration method and automobile data recorder | |
| CN108961337B (en) | Vehicle-mounted camera course angle calibration method and device, electronic equipment and vehicle |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |