CN114675657B - Infrared camera fuzzy control algorithm based homing charging method - Google Patents
Infrared camera fuzzy control algorithm based homing charging method Download PDFInfo
- Publication number
- CN114675657B CN114675657B CN202210574597.5A CN202210574597A CN114675657B CN 114675657 B CN114675657 B CN 114675657B CN 202210574597 A CN202210574597 A CN 202210574597A CN 114675657 B CN114675657 B CN 114675657B
- Authority
- CN
- China
- Prior art keywords
- robot
- charging
- charging seat
- pose
- pose information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 17
- 238000011478 gradient descent method Methods 0.000 claims abstract description 4
- 238000005457 optimization Methods 0.000 claims abstract description 4
- 238000001514 detection method Methods 0.000 claims description 7
- 238000011176 pooling Methods 0.000 claims description 3
- 238000004364 calculation method Methods 0.000 description 4
- 238000005516 engineering process Methods 0.000 description 4
- 238000003032 molecular docking Methods 0.000 description 4
- 210000001503 joint Anatomy 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000004069 differentiation Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0242—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using non-visible light signals, e.g. IR or UV signals
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01M—PROCESSES OR MEANS, e.g. BATTERIES, FOR THE DIRECT CONVERSION OF CHEMICAL ENERGY INTO ELECTRICAL ENERGY
- H01M10/00—Secondary cells; Manufacture thereof
- H01M10/42—Methods or arrangements for servicing or maintenance of secondary cells or secondary half-cells
- H01M10/44—Methods for charging or discharging
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E60/00—Enabling technologies; Technologies with a potential or indirect contribution to GHG emissions mitigation
- Y02E60/10—Energy storage using batteries
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Remote Sensing (AREA)
- Electromagnetism (AREA)
- Automation & Control Theory (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Aviation & Aerospace Engineering (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Chemical & Material Sciences (AREA)
- Multimedia (AREA)
- Chemical Kinetics & Catalysis (AREA)
- Manufacturing & Machinery (AREA)
- General Chemical & Material Sciences (AREA)
- Electrochemistry (AREA)
- Control Of Position, Course, Altitude, Or Attitude Of Moving Bodies (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a nest returning charging method based on an infrared camera fuzzy control algorithm, which comprises the following steps: step 1: the robot receives a nest returning charging instruction and positions an aruco code on a charging seat; step 2: acquiring the current pose information of the robot through the artco code on the charging seat
，
，
(ii) a And 3, step 3: the loss function for obtaining the final pose information of the robot through the pose information definition of the robot is as follows:
(ii) a And 4, step 4: and (3) adjusting the acquired pose information of the robot by using a gradient descent method, wherein an optimization formula of the pose information is as follows:
(ii) a And 5: repeating the step 3 and the step 4 until the minimum value of the loss function is less than or equal to a preset value, and stopping pose correction; namely:
Description
Technical Field
The invention belongs to the technical field of robots, and particularly relates to a nest returning charging method based on an infrared camera fuzzy control algorithm.
Background
With the wide application of artificial intelligence in the field of robots, the application scenes and the work tasks of the robots are more and more complex, and the autonomous operation time of the robots is also gradually and correspondingly increased. However, the onboard storage battery adopted by the robot cannot meet the working requirement of the robot, and increasing the capacity of the storage battery brings unnecessary burden to the robot. Therefore, the autonomous charging docking technology of the mobile robot becomes an important link of the continuous operation of the mobile robot. Because the problem of robot navigation accuracy, the robot is difficult to charge to the position of filling electric pile place through static map navigation, and most independent charging technology lets the robot navigation to the region of charging earlier at present, then adopts modes such as infrared signal, laser range finder or vision to guide the accurate butt joint of robot to fill electric pile. The method specifically comprises the following modes: 1. the infrared correlation technology is applied to an autonomous charging algorithm of an indoor robot, an autonomous charging task of the robot is completed by tracking infrared rays, and the pure infrared signal transmission distance is small and the transmission direction is single, so that the pure infrared guide butt joint charging efficiency is low and the range is small; 2. the SICK laser range finder is used for guiding the robot to automatically charge and butt, but the precision of the laser range finder is reduced along with the increase of the measuring range, so that the robot cannot be guided in a large range; 3. an automatic charging and docking mode based on infrared and ultrasonic waves is adopted, and an automatic charging and docking system is designed by utilizing infrared regression reflection and ultrasonic distance detection functions, so that the design scheme is simplified, and the guide range is expanded; 4. aiming at the problems of poor site adaptability, small guide range and the like of the robot autonomous charging technology, an autonomous charging mode based on a visual target is designed, the site deployment mode is effectively improved, the guide range and the success rate of autonomous charging are improved, but the phenomenon of target loss still exists when the robot moves at an excessively high speed, and the charging efficiency is low.
The Quick Response (QR) code of the mark charging pile is identified through a camera installed on the robot, the relative position of the robot and the QR code is quickly acquired, and the robot directly moves to the charging pile through the position information. Meanwhile, in order to eliminate the influence on positioning information caused by the fact that the robot loses a QR code in the moving process and cannot acquire a complete graph when the QR code is close to the robot, the relative angle between the QR code and the positioning information is determined by utilizing an infrared receiver when the QR code cannot be used for positioning, and the autonomous charging docking task of the robot is completed.
However, the existing nesting charging method still has the following problems: 1. under the condition of no illumination, the nest returning charging device cannot realize automatic nest returning charging at night; 2. the technical means is complex, and a QR code of a camera, an infrared transmitter and an infrared receiver are required to jointly form a path for calculating and planning robot returning and charging; and 3, adjusting the pose of the robot by using a PID (proportion integration differentiation) algorithm, wherein the calculated amount is large, and the accumulated error is relatively large.
In view of the above, the present invention is particularly proposed.
Disclosure of Invention
In order to solve the problems in the prior art, the invention provides a homing charging method based on an infrared camera fuzzy control algorithm.
In order to achieve the purpose, the invention provides a method for homing and charging based on an infrared camera fuzzy control algorithm, which comprises the following steps:
step 1: the robot receives a nest returning charging instruction and positions an aruco code on a charging seat;
and 2, step: acquiring pose information x, z and theta of the robot through the aruco code on the charging seat;
and step 3: and obtaining the final pose information of the robot through the pose information of the robot, wherein the loss function of the final pose information of the robot is as follows:
wherein m represents the times of pose correction of the robot; y is i Representing an actual target value of the ith correction pose information, and f (x, z, theta) representing a prediction function;
and 4, step 4: adjusting the obtained pose information of the robot by a gradient descent method, wherein an optimization formula of the pose information is as follows:
wherein x represents the offset, z represents the vertical distance between the charging seat and the robot, theta represents the relative angle between the charging seat and the robot, a represents the step length,
represents the derivative of the loss function L (ω);
and 5: repeating the step 3 and the step 4 until the minimum value of the loss function is less than or equal to a preset value, and stopping pose correction; namely:
(x，z，θ)＝argminL(x，z，θ)
wherein argmin represents a variable value at which the target function L (x, z, θ) is minimized;
and 6: and after the homing is successful, sending a command of ending the homing to end the pose adjustment of the robot.
Preferably, the step 1 specifically includes the following steps:
s1: the robot rotates the base, and an infrared camera is used for collecting image information;
s2: judging whether the image information has a charging seat, and returning to the step S1 when the image information has no charging seat; when the cradle is present, proceed to S3;
s3: whether the image information has the aruco code or not is judged, when the image information does not have the aruco code, the robot walks for a circle around the charging seat, and the number of the walking circles and the angle of each circle of the walking are used for shooting the charging seat to meet the formula:
wherein: l represents the number of turns of the robot walking around the charging seat; the first circle of the image is shot at every pi/2, the second circle of the image is shot at every pi/4, and the rest can be done in the same way until an image code appears in the image; when the aroco code exists, entering a step 2;
preferably, the formula of the prediction function in step 3 is: f (x, z, theta) ═ theta k 0 +xk 1 +zk 2 ；
Preferably, the step 1 adopts a target detection algorithm to locate the position of the charging seat so as to acquire the information of the aruco code on the charging seat;
preferably, the target detection algorithm adopts a Faster-R-CNN algorithm, and specifically comprises the following steps:
(1) extracting feature maps for the full graph by using the shared convolution layer;
(2) sending the obtained feature maps into an RPN (resilient packet network), generating a frame to be detected at the position of the appointed RoI by the RPN, and correcting the bounding frame of the RoI for the first time;
(3) the RoI Pooling Layer selects the characteristic corresponding to each RoI on the feature map according to the output of the RPN, and sets the dimension as a fixed value;
(4) the box is classified using the fully connected layer and a second correction of the target bounding box is made.
The invention provides a nest-returning charging method based on an infrared camera fuzzy control algorithm, which has the following beneficial effects:
the invention effectively solves the technical problems that the robot can not normally work in a dark environment, a plurality of matched devices are needed by a robot homing charging path, the calculation is complex, the parameters are difficult to obtain, the efficiency is low and the accuracy is low in the prior art. Meanwhile, a Fuzzy Control Algorithm (FCA) is adopted, the calculation complexity is low, the optimal point can be reached quickly, and the quick and accurate nest returning charging is further realized.
Detailed Description
The present invention will be further described with reference to specific examples to assist understanding of the invention.
The invention provides a nest-returning charging method based on an infrared camera fuzzy control algorithm, which comprises the following steps:
step 1: the robot receives a nest returning charging instruction and positions an aruco code on a charging seat;
step 2: acquiring pose information x, z and theta of the robot through the aruco code on the charging seat;
and step 3: the loss function for acquiring the final pose information of the robot through the pose information of the robot is as follows:
wherein m represents the times of pose correction of the robot; y is i The actual target value of the ith correction pose information is shown, f (x, z, theta) is a prediction function, and the formula of the prediction function is as follows: f (x, z, theta) ═ theta k 0 +xk 1 +zk 2 Wherein k is a variable, and x, z and theta are parameters;
and 4, step 4: and adjusting the obtained pose information of the robot by a gradient descent method, wherein an optimization formula of the pose information is as follows:
wherein x represents the offset, z represents the vertical distance between the charging seat and the robot, theta represents the relative angle between the charging seat and the robot, alpha represents the step length,
represents the derivative of the loss function L (ξ), where ω is the generic term for the variables in the function;
and 5: repeating the step 3 and the step 4 until the minimum value of the loss function is less than or equal to a preset value, and stopping pose correction; namely:
(x，z，θ)＝argminL(x，z，θ)
wherein argmin represents a variable value at which the target function L (x, z, θ) is minimized;
step 6: and after the homing is successful, sending a command of ending the homing to end the pose adjustment of the robot.
Wherein, the step 1 specifically comprises the following steps:
s1: the robot rotates the base, and an infrared camera is used for collecting image information;
s2: judging whether a charging seat exists in the image information or not, returning to S1 when the charging seat does not exist, and if the charging seat enters S1 for three times continuously, determining that the charging seat is not in a charging area, and exiting the nest-returning charging program; when the cradle is present, proceed to S3;
s3: whether the image information has the arbuo code or not is judged, when the image information does not have the arbuo code, the robot uses the charging seat as a circle center, uses the distance from the robot to the charging seat as a radius to surround the charging seat for one circle of walking, and the walking circles and the walking angle of each circle are used for shooting the charging seat to meet the formula:
wherein: l represents the number of turns of the robot walking around the charging seat; namely, the first circle of the charging seat is shot at every pi/2, the second circle of the charging seat is shot at every pi/4, and so on until the image has an aruco code, thenEntering the step 2; and when the walking angle is less than pi/16, if the image does not have the arico code, exiting the nest returning and charging program.
Before the robot returns the nest through the fuzzy control algorithm, the robot needs to be positioned at the position of a charging seat through a target detection algorithm so as to acquire the information of the aruco code on the charging seat, and the detection of the charging seat is performed by applying the Faster-R-CNN algorithm because the characteristics of the charging seat are obvious relative to the characteristics of the surrounding environment, and the method specifically comprises the following steps:
(1) extracting feature maps for the full graph by using the shared convolution layer;
(2) sending the obtained feature maps into an RPN (resilient packet network), generating a frame to be detected at the position of the appointed RoI by the RPN, and correcting the bounding frame of the RoI for the first time;
(3) the RoI Pooling Layer selects the characteristic corresponding to each RoI on the feature map according to the output of the RPN, and sets the dimension as a fixed value;
(4) the box is classified using the full connected layer and a second correction of the target bounding box is made.
Note: because the color information that the charging seat appears in the camera is different daytime and evening, so in order to improve the recognition efficiency, two models for detecting the charging seat in daytime and evening are trained respectively and are used for detecting the charging seat under different scenes respectively.
The invention effectively solves the technical problems that the robot can not normally work in dark environment, the matching devices required by the robot homing charging path are more, the calculation is complex, the parameters are difficult to obtain, the efficiency is low and the accuracy is low in the prior art. Meanwhile, a Fuzzy Control Algorithm (FCA) is adopted, the calculation complexity is low, the optimal point can be reached quickly, and the quick and accurate nest returning charging is further realized.
The inventive concept is explained in detail herein using specific examples, which are given only to aid in understanding the core concepts of the invention. It should be understood that any obvious modifications, equivalents and other improvements made by those skilled in the art without departing from the spirit of the present invention are included in the scope of the present invention.
Claims (4)
1. A nest returning charging method based on an infrared camera fuzzy control algorithm is characterized by comprising the following steps:
step 1: the robot receives a nest returning charging instruction and positions an aruco code on a charging seat, and the method specifically comprises the following steps:
s1: the robot rotates the base, and an infrared camera is used for collecting image information;
s2: judging whether the image information has a charging seat, and returning to the step S1 when the image information has no charging seat; when the cradle is present, proceed to S3;
s3: whether the image information has the aruco code or not is judged, when the image information does not have the aruco code, the robot walks for a circle around the charging seat, and the number of the walking circles and the angle of each circle of the walking are used for shooting the charging seat to meet the formula:
wherein: l represents the number of turns of the robot walking around the charging seat; i.e. each first turn is taken pi- 2 A shooting charging seat, and every second turn of the charging seat moves pi- 4 Shooting a charging seat, and repeating the steps until an aruco code appears in an image; when the aroco code exists, entering a step 2;
step 2: acquiring pose information x, z and theta of the robot through the aruco code on the charging seat;
and step 3: and defining and acquiring a loss function of the final pose information of the robot through the pose information of the robot as follows:
wherein m represents the times of pose correction of the robot; y is i Representing an actual target value of the ith correction pose information, and f (x, z, theta) representing a prediction function;
and 4, step 4: and (3) adjusting the acquired pose information of the robot by using a gradient descent method, wherein an optimization formula of the pose information is as follows:
wherein x represents the offset, z represents the vertical distance between the charging seat and the robot, theta represents the relative angle between the charging seat and the robot, alpha represents the step length,
represents the derivative of the loss function L (ω);
and 5: repeating the step 3 and the step 4 until the minimum value of the loss function is less than or equal to a preset value, and stopping pose correction; namely:
(x，z，θ)＝argminL(x，z，θ)
wherein argmin represents a variable value at which the target function L (x, z, θ) is minimized;
step 6: and after the homing is successful, sending a command of ending the homing to end the pose adjustment of the robot.
2. The method for infrared camera fuzzy control algorithm based homing charging according to claim 1, wherein the formula of the prediction function in the step 3 is: f (x, z, theta) ═ theta k 0 +xk 1 +zk 2 Wherein k is variable, and x, z and theta are parameters.
3. The infrared camera fuzzy control algorithm-based nest-returning charging method as claimed in claim 1, wherein the step 1 adopts a target detection algorithm to locate to the position of the charging stand to acquire the information of the arico code on the charging stand.
4. The method for fuzzy control of the homing charging algorithm based on the infrared camera as claimed in claim 1, wherein the target detection algorithm adopts a fast-R-CNN algorithm, comprising the following steps:
(1) extracting feature maps for the full graph by using the shared convolution layer;
(2) sending the obtained feature maps into an RPN (resilient packet network), generating a frame to be detected at the position of the appointed RoI by the RPN, and correcting the bounding frame of the RoI for the first time;
(3) the RoI Pooling Layer selects the characteristic corresponding to each RoI on the feature map according to the output of the RPN, and sets the dimension as a fixed value;
(4) the box is classified using the full connected layer and a second correction of the target bounding box is made.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210574597.5A CN114675657B (en) | 2022-05-25 | 2022-05-25 | Infrared camera fuzzy control algorithm based homing charging method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210574597.5A CN114675657B (en) | 2022-05-25 | 2022-05-25 | Infrared camera fuzzy control algorithm based homing charging method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114675657A CN114675657A (en) | 2022-06-28 |
| CN114675657B true CN114675657B (en) | 2022-09-23 |
Family
ID=82079576
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210574597.5A Active CN114675657B (en) | 2022-05-25 | 2022-05-25 | Infrared camera fuzzy control algorithm based homing charging method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114675657B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021184218A1 (en) * | 2020-03-17 | 2021-09-23 | 华为技术有限公司 | Relative pose calibration method and related apparatus |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106826821A (en) * | 2017-01-16 | 2017-06-13 | 深圳前海勇艺达机器人有限公司 | The method and system that robot auto-returned based on image vision guiding charges |
| CN110020633B (en) * | 2019-04-12 | 2022-11-04 | 腾讯科技（深圳）有限公司 | Training method of posture recognition model, image recognition method and device |
| CN110116407B (en) * | 2019-04-26 | 2021-03-30 | 哈尔滨工业大学（深圳） | Flexible robot position and posture measuring method and device |
| CN111625005A (en) * | 2020-06-10 | 2020-09-04 | 浙江欣奕华智能科技有限公司 | Robot charging method, robot charging control device and storage medium |
| CN112183133B (en) * | 2020-08-28 | 2022-05-31 | 同济大学 | Aruco code guidance-based mobile robot autonomous charging method |
-
2022
- 2022-05-25 CN CN202210574597.5A patent/CN114675657B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021184218A1 (en) * | 2020-03-17 | 2021-09-23 | 华为技术有限公司 | Relative pose calibration method and related apparatus |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114675657A (en) | 2022-06-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111958591B (en) | Autonomous inspection method and system for semantic intelligent substation inspection robot | |
| CN108983807B (en) | Unmanned aerial vehicle fixed-point landing method and system | |
| CN104049634B (en) | Intelligent body fuzzy dynamic obstacle avoidance method based on Camshift algorithm | |
| US20190243376A1 (en) | Actively Complementing Exposure Settings for Autonomous Navigation | |
| CN109901590B (en) | Recharging control method of desktop robot | |
| US20070150097A1 (en) | Localization system and method of mobile robot based on camera and landmarks | |
| KR100854653B1 (en) | Localization system of the mobile robot using the charging station | |
| CN105823478A (en) | Autonomous obstacle avoidance navigation information sharing and using method | |
| CN110026979B (en) | Mobile robot and return charging method | |
| CN110254258B (en) | Unmanned aerial vehicle wireless charging system and method | |
| CN108535279A (en) | A kind of detection method detecting robot based on sewage pipeline | |
| US20210089772A1 (en) | Method of using a heterogeneous position information acquisition mechanism in an operating space and robot and cloud server implementing the same | |
| CN108015764A (en) | A kind of zero priori target acquistion method of space based on the fusion of multi-source visual information | |
| CN112033391B (en) | Robot repositioning method and device based on charging pile | |
| JP2020126647A (en) | Method for correcting wrong arrangement of cameras by selectively using information generated by itself and information generated by other objects and device for using the same | |
| CN111199677A (en) | Automatic work map establishing method and device for outdoor area, storage medium and working equipment | |
| CN114675657B (en) | Infrared camera fuzzy control algorithm based homing charging method | |
| CN112373486A (en) | Unmanned driving method and system for small robot | |
| CN107305385B (en) | Butt joint method of automatic walking equipment and automatic walking equipment | |
| CN111931832B (en) | Optimal data acquisition method and system for substation inspection equipment | |
| CN112650304B (en) | Unmanned aerial vehicle autonomous landing system and method and unmanned aerial vehicle | |
| Armingol et al. | Landmark perception planning for mobile robot localization | |
| CN113498667A (en) | Intelligent mowing robot based on panoramic machine vision | |
| CN116202489A (en) | Method and system for co-locating power transmission line inspection machine and pole tower and storage medium | |
| CN111860461B (en) | Autonomous zooming method for built-in optical sensor of photoelectric pod |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |