CN112904879B - Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks - Google Patents
Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks Download PDFInfo
- Publication number
- CN112904879B CN112904879B CN202110062036.2A CN202110062036A CN112904879B CN 112904879 B CN112904879 B CN 112904879B CN 202110062036 A CN202110062036 A CN 202110062036A CN 112904879 B CN112904879 B CN 112904879B
- Authority
- CN
- China
- Prior art keywords
- landmark
- landing
- quadrotor
- color block
- color
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 239000003086 colorant Substances 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 230000008859 change Effects 0.000 claims description 5
- 238000004904 shortening Methods 0.000 claims description 3
- 230000000007 visual effect Effects 0.000 claims description 3
- 238000010586 diagram Methods 0.000 description 3
- WHXSMMKQMYFTQS-UHFFFAOYSA-N Lithium Chemical compound [Li] WHXSMMKQMYFTQS-UHFFFAOYSA-N 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000011835 investigation Methods 0.000 description 1
- 229910052744 lithium Inorganic materials 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000002195 synergetic effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/08—Control of attitude, i.e. control of roll, pitch, or yaw
- G05D1/0808—Control of attitude, i.e. control of roll, pitch, or yaw specially adapted for aircraft
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/10—Simultaneous control of position or course in three dimensions
- G05D1/101—Simultaneous control of position or course in three dimensions specially adapted for aircraft
Abstract
The invention discloses an autonomous tracking landing system and method of a quadrotor based on color block landmark recognition, comprising a central main control unit, a positioning recognition module and a following landing module which are arranged on a quadrotor body, and color block landmarks which are arranged on the ground, wherein the positioning recognition module is used for obtaining the central target value of the quadrotor, and an output signal of the positioning recognition module is sent to the central main control unit; the central main control unit is used for reading the sensing data information, ensuring the attitude control and operation of the quadrotor, and sending the output signals to the following landing module; the following landing module adopts a PID algorithm following positioning recognition system to realize the adjustment of the machine head direction when the tracking precision requirement is met. Compared with the prior art, the invention not only can solve the problems, but also can provide the direction standard for the head of the quadrotor, so that the quadrotor can accurately land according to the required direction.
Description
Technical Field
The invention belongs to the field of navigation and positioning of four-rotor aircraft, and particularly relates to an autonomous tracking landing system and method of a four-rotor aircraft.
Background
As quad-rotor aircraft evolve in military and civilian use, more and more quad-rotor aircraft are being used for cargo transportation, high altitude investigation, ground mapping, and space-ground collaboration tasks, among others. To achieve these functions, quad-rotor aircraft often require identification, tracking, landing, or even synergistic cooperation of specific targets. Therefore, an efficient landmark recognition method and a stable landing tracking method are important. Because the accuracy provided by conventional GPS positioning during four-rotor aircraft tracking and landing is not satisfactory, the combination of four-rotor aircraft with vision sensors is of great value both military and commercial importance.
Today, many quadrotors detect with fixed focus cameras, and during landing, their field of view can be pulled into the ground gradually, for detecting the color lump, can lead to the color lump to be full of the camera field of view, for detecting the two-dimensional code, can make the field of view unable to fully contain the two-dimensional code to lead to the target to lose.
The above-mentioned drawbacks of the prior art are also the technical problems to be solved in the present invention.
Disclosure of Invention
The invention aims to provide a four-rotor aircraft autonomous tracking landing system and method based on color block landmarks, and designs a square color block landmark containing four different colors so as to obtain a central target value of the four-rotor aircraft, thereby solving the following landing problem of the four-rotor aircraft.
The invention relates to a four-rotor aircraft autonomous tracking landing system based on color block landmark recognition, which comprises a central main control unit, a positioning recognition module and a following landing module which are arranged on a four-rotor aircraft body, and color block landmarks arranged on the ground, wherein: the positioning identification module is used for obtaining a central target value of the quadrotor, and an output signal of the positioning identification module is sent to the central main control unit; the central main control unit is used for reading the sensing data information, ensuring the attitude control and operation of the quadrotor, and sending the output signals to the following landing module;
the following landing module adopts a PID algorithm to follow the positioning recognition system to obtain the central target value of the color block landmark, and is used for realizing the adjustment of the machine head direction when the tracking precision requirement is met, and then automatically landing.
The invention discloses a four-rotor aircraft autonomous tracking landing method based on color block landmark recognition, which specifically comprises the following steps:
firstly, placing a color block landmark on a target place or a moving platform with a flat surface, and starting a quadrotor to reach a proper height, wherein the color block landmark is required to be ensured to be in the visual field range of a positioning and identifying system in the process;
step 2, detecting and identifying the color block, adopting PID algorithm to follow the center target point of the color block landmark obtained by the positioning and identifying system,
the PID controller output u (t) is calculated as follows:
wherein e (t) represents the difference between the target value and the current value, k p Represents proportional gain, k i Represents the integral gain, k d Representing differential gain;
step 3, when the number of the detected effective color blocks is more than or equal to 2, the color block landmarks are successfully identified, and the detected color block coordinates are averaged to be used as target points to follow; along with the shortening of the distance between the four-rotor aircraft and the color block landmark, the positioning recognition system completely recognizes the four color blocks with different colors, and further obtains the central target point;
when the quadrotor flies to the position right above the color block landmark and tracking meets the precision, the quadrotor is rotated according to the requirement
The yaw angle of the aircraft is adjusted in the aircraft nose direction, and then the aircraft automatically drops, so that the aircraft drops in the forward direction; in the landing process, a uniform-speed landing method is adopted, so that overlarge ground impact caused by too fast landing of the quadrotor aircraft is avoided.
Compared with the prior art, the invention not only can solve the problems, but also can provide the direction standard for the head of the quadrotor, so that the quadrotor can accurately land according to the required direction.
Drawings
FIG. 1 is a block landmark-based architecture diagram of an autonomous tracking landing system for a quad-rotor aircraft in accordance with the present invention;
FIG. 2 is a diagram of an example of a color patch landmark according to the present invention;
FIG. 3 is an overall flowchart of a four-rotor aircraft autonomous tracking landing method based on color block landmarks of the present invention;
fig. 4 is an illustration of a four-rotor aircraft following a patch landmark that is prone to target loss problems.
Detailed Description
The technical scheme of the invention is described in detail below with reference to the accompanying drawings and examples.
As shown in figure 1, the invention discloses a four-rotor aircraft autonomous tracking landing system architecture diagram based on color block landmarks. The system comprises a four-rotor aircraft body, a central main control unit, color block landmarks, a positioning identification module and a following landing module. The system comprises a central main control unit, a positioning identification module and a landing following module, wherein the central main control unit, the positioning identification module and the landing following module are all positioned on a four-rotor aircraft body, and color block landmarks are arranged on the ground.
Specifically, the four-rotor aircraft body comprises a rack with a wheel base not lower than 360mm, a brushless motor with a propeller, an electronic speed regulator, a lithium battery power system, a laser height ranging module and a landing gear. The specific modules of each section are not limited in their type of use, and most open source quadrotors include these sections.
Specifically, the central main control unit takes STM32 as a processor chip and comprises a peripheral working circuit and an attitude sensor module, and the functions of the central main control unit comprise reading data information such as a sensor and the like, ensuring attitude control and related operations.
Specifically, a color patch landmark is a square area that includes four small square areas of different colors, while requiring that the detection background of the color patch landmark not be identical to the four colors. As shown in fig. 2, an example plot of color patches designed in accordance with the present invention is shown. Taking a white background as an example, the upper left side of the example graph of color block landmarks is a red color block landmark, the upper right side is a yellow color block landmark, the lower left side is a blue color block landmark, and the lower right side is a green color block landmark. And obtaining a central target point of the whole color block landmark by judging the central coordinate of each color block landmark. Because the color block landmark has the specific azimuth of four colors, the four-rotor aircraft can judge the deflection angle of the four-rotor aircraft and the color block landmark according to the specific azimuth, and further provide a direction reference for the head of the four-rotor aircraft, thereby meeting the control requirement of the four-rotor aircraft on the yaw angle during landing. The above problems can be well solved by taking the central target point of fig. 2 as a reference, and the color patches of four colors can be always in the field of view of the camera in the dropping process. Note that the coordinate axes and the central target point in fig. 2 are only for clearly describing the working principle of the color block landmark, and should not appear in the actual landmark. Since the actual size of the color patch landmarks is highly correlated with a quad-rotor aircraft, some of the dimensional references are specifically shown in table 1.
TABLE 1
| Color block landmark size | Fly height |
| 200mm*200mm | 30cm-100cm |
| 500mm*500mm | 100cm-200cm |
| 1000mm*1000mm | 200cm-500cm |
Specifically, the positioning and identifying module is an OpenMV camera module installed at the right center of the bottom of the quadrotor, the OpenMV camera module is connected with the central main control unit through a serial port, and the central target value of the quadrotor is obtained by identifying the central coordinate position of four color block landmarks in an image. The OpenMV camera can quickly and accurately identify the required color block and obtain its center coordinates, so that the center target point described in fig. 2 can be conveniently calculated. To reduce its conservation, it is initially thought that only two different colored color patches are successfully identified.
Specifically, the following landing module is a controller in the central main control unit, and has the functions of adopting a PID algorithm to follow the central target value of the color block landmark obtained by the positioning and identifying system, firstly adjusting the machine head direction when the tracking precision requirement is met, and then automatically landing, wherein the PID algorithm is as follows
Wherein e (t) represents the difference between the target value and the current value, k p Represents the proportional gain, plays the roles of accelerating the response speed of the system and improving the adjustment precision of the system, and k i Represents integral gain, plays roles of eliminating residual error and adjusting steady state time, k d The differential gain is expressed, and the dynamic performance of the system is improved, the error trend is predicted and the system is corrected in advance. These parameters need to be adjusted according to the actual situation. Particularly, when the following control is performed, e (t) is the deviation between the central target point calculated by the positioning recognition system and the position of the quadrotor; when the automatic landing is performed, e (t) is the difference between the desired landing speed and the landing speed. Unlike conventional methods of altitude control, the four-rotor aircraft maintains a constant landing speed while landing, avoiding excessive ground impact caused by the excessive speed of landing of the four-rotor aircraft.
Fig. 3 is a flowchart showing an overall method for autonomous tracking landing of a quad-rotor aircraft based on landmark recognition according to the present invention. The method specifically comprises the following steps:
firstly, placing a color block landmark on a target place or a moving platform with a flat surface, and starting a quadrotor to reach a proper height, wherein the color block landmark is required to be ensured to be in the visual field range of a positioning and identifying system in the process;
step 2, detecting and identifying the color block, adopting PID algorithm to follow the center target point of the color block landmark obtained by the positioning and identifying system,
the PID controller output u (t) is calculated as follows:
wherein e (t) represents the difference between the target value and the current value, k p 、k i And k d Representing control parameters which need to be adjusted according to the actual situation. Particularly, when the following control is performed, e (t) is the deviation between the central target point calculated by the positioning recognition system and the position of the quadrotor, and when the automatic landing is performed, e (t) is the difference between the expected landing speed and the landing speed;
step 3, when the number of the detected effective color blocks is more than or equal to 2, the color block landmarks are successfully identified, and the detected color block coordinates are averaged to be used as target points to follow; along with the shortening of the distance between the four-rotor aircraft and the color block landmark, the positioning recognition system completely recognizes the four color blocks with different colors, and further obtains the central target point;
when the quadrotor flies to the position right above the color block landmark and tracking meets the precision, the yaw angle of the quadrotor is rotated according to the requirement, the direction of the machine head is adjusted first, and then automatic landing is carried out, so that the directional landing is realized; in the landing process, a uniform-speed landing method is adopted, so that overlarge ground impact caused by too fast landing of the quadrotor aircraft is avoided.
When the four-rotor aircraft follows the color block landmark, the problem that the color block landmark is lost in the field of view of the positioning and identifying system caused by the easy angular change is solved. As shown in fig. 4, an illustration of a four-rotor aircraft following a color patch landmark is prone to target loss problems. It can be seen that the color lump landmarks can be successfully identified when the quad-rotor aircraft is in a horizontal state. After the center target point is calculated, the quadrotor needs to make a change in its own attitude to track the color block landmark, so that the problem of field of view loss occurs. For this problem, the following landing system is solved by using a gesture decoupling method, and the following steps are included:
delay judgment, namely determining a lost target after losing a color block landmark for one second;
posture compensation, namely when target loss caused by angle change of the quadrotor is detected, compensating angle multiplication coefficient of e (t);
to avoid abrupt changes, e (t) slowly drops to zero in one second when confirming the missing target.
The invention is not a matter of the known technology.
The foregoing detailed description is provided to illustrate the technical spirit and features of the present invention and is not intended to limit the scope of the present invention for those skilled in the art to understand the present invention and implement it accordingly. All equivalent changes or modifications made in accordance with the spirit of the present invention should be construed to be included in the scope of the present invention.
Claims (2)
1. The utility model provides a four rotor craft independently trail landing method based on colour lump landmark discernment, four rotor craft independently trail landing system realization based on colour lump landmark discernment, the system is including setting up central master control unit, location recognition module and the landing module that follows on four rotor craft organism to and set up colour lump landmark on subaerial, wherein: the positioning identification module is used for obtaining a central target value of the quadrotor, and an output signal of the positioning identification module is sent to the central main control unit; the central main control unit is used for reading the sensing data information, ensuring the attitude control and operation of the quadrotor, and sending the output signals to the following landing module; the following landing module adopts a PID algorithm to follow a positioning recognition system to obtain a central target value of a color block landmark, and is used for realizing the adjustment of the machine head direction when the tracking precision requirement is met and then automatically landing, and is characterized in that the method specifically comprises the following steps:
firstly, placing a color block landmark on a target place or a moving platform with a flat surface, and starting a quadrotor to reach a proper height, wherein the color block landmark is required to be ensured to be in the visual field range of a positioning and identifying system in the process;
step 2, detecting and identifying the color block, adopting PID algorithm to follow the center target point of the color block landmark obtained by the positioning and identifying system,
the PID controller output u (t) is calculated as follows:
wherein e (t) represents the difference between the target value and the current value, k p Represents proportional gain, k i Represents the integral gain, k d Representing differential gain;
step 3, when the number of the detected effective color blocks is more than or equal to 2, the color block landmarks are successfully identified, and the detected color block coordinates are averaged to be used as target points to follow; along with the shortening of the distance between the four-rotor aircraft and the color block landmark, the positioning recognition system completely recognizes the four color blocks with different colors, and further obtains the central target point;
when the quadrotor flies to the position right above the color block landmark and tracking meets the precision, the yaw angle of the quadrotor is rotated according to the requirement, the direction of the machine head is adjusted first, and then automatic landing is carried out, so that the directional landing is realized; in the landing process, a uniform-speed landing method is adopted, so that overlarge ground impact caused by too fast landing of the quadrotor aircraft is avoided.
2. The method for autonomous tracking landing of a quad-rotor vehicle based on color-patch landmark recognition according to claim 1, wherein the method further comprises a gesture decoupling procedure in the case that the color-patch landmark is lost from the field of view of the positioning recognition system due to an angular change occurring when the quad-rotor vehicle follows the color-patch landmark; the specific process is as follows:
performing delay judgment, namely considering a lost target after losing a color-removing block landmark for one second;
performing attitude compensation, namely performing angle multiplication coefficient compensation on e (t) when target loss caused by angle change of the quadrotor aircraft is detected;
to avoid abrupt changes, e (t) slowly drops to zero in one second when confirming the missing target.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062036.2A CN112904879B (en) | 2021-01-18 | 2021-01-18 | Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110062036.2A CN112904879B (en) | 2021-01-18 | 2021-01-18 | Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112904879A CN112904879A (en) | 2021-06-04 |
| CN112904879B true CN112904879B (en) | 2024-04-12 |
Family
ID=76114633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110062036.2A Active CN112904879B (en) | 2021-01-18 | 2021-01-18 | Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112904879B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103809598A (en) * | 2014-03-12 | 2014-05-21 | 北京航空航天大学 | Rotor unmanned aircraft independent take-off and landing system based on three-layer triangle multi-color landing ground |
| CN104166854A (en) * | 2014-08-03 | 2014-11-26 | 浙江大学 | Vision grading landmark locating and identifying method for autonomous landing of small unmanned aerial vehicle |
| WO2015142166A1 (en) * | 2014-03-20 | 2015-09-24 | Lely Patent N.V. | Method and system for navigating an agricultural vehicle on a land area |
| CN105278541A (en) * | 2015-09-02 | 2016-01-27 | 蔡兆旸 | Aircraft auxiliary landing control method and system |
| CN107063261A (en) * | 2017-03-29 | 2017-08-18 | 东北大学 | The multicharacteristic information terrestrial reference detection method precisely landed for unmanned plane |
| CN107463181A (en) * | 2017-08-30 | 2017-12-12 | 南京邮电大学 | A kind of quadrotor self-adoptive trace system based on AprilTag |
| CN108710383A (en) * | 2018-05-25 | 2018-10-26 | 哈尔滨工业大学 | A kind of quadrotor Autonomous landing control method planned based on destination with tracking |
| CN110109469A (en) * | 2019-03-19 | 2019-08-09 | 南京理工大学泰州科技学院 | It is a kind of with color, identification, positioning, following function quadrotor drone control system |
| CN110989687A (en) * | 2019-11-08 | 2020-04-10 | 上海交通大学 | Unmanned aerial vehicle landing method based on nested square visual information |
| CN111307157A (en) * | 2020-03-12 | 2020-06-19 | 上海交通大学 | Navigation information resolving method for unmanned aerial vehicle autonomous landing based on vision |
-
2021
- 2021-01-18 CN CN202110062036.2A patent/CN112904879B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103809598A (en) * | 2014-03-12 | 2014-05-21 | 北京航空航天大学 | Rotor unmanned aircraft independent take-off and landing system based on three-layer triangle multi-color landing ground |
| WO2015142166A1 (en) * | 2014-03-20 | 2015-09-24 | Lely Patent N.V. | Method and system for navigating an agricultural vehicle on a land area |
| CN104166854A (en) * | 2014-08-03 | 2014-11-26 | 浙江大学 | Vision grading landmark locating and identifying method for autonomous landing of small unmanned aerial vehicle |
| CN105278541A (en) * | 2015-09-02 | 2016-01-27 | 蔡兆旸 | Aircraft auxiliary landing control method and system |
| CN107063261A (en) * | 2017-03-29 | 2017-08-18 | 东北大学 | The multicharacteristic information terrestrial reference detection method precisely landed for unmanned plane |
| CN107463181A (en) * | 2017-08-30 | 2017-12-12 | 南京邮电大学 | A kind of quadrotor self-adoptive trace system based on AprilTag |
| CN108710383A (en) * | 2018-05-25 | 2018-10-26 | 哈尔滨工业大学 | A kind of quadrotor Autonomous landing control method planned based on destination with tracking |
| CN110109469A (en) * | 2019-03-19 | 2019-08-09 | 南京理工大学泰州科技学院 | It is a kind of with color, identification, positioning, following function quadrotor drone control system |
| CN110989687A (en) * | 2019-11-08 | 2020-04-10 | 上海交通大学 | Unmanned aerial vehicle landing method based on nested square visual information |
| CN111307157A (en) * | 2020-03-12 | 2020-06-19 | 上海交通大学 | Navigation information resolving method for unmanned aerial vehicle autonomous landing based on vision |
Non-Patent Citations (2)
| Title |
|---|
| 基于扩张状态观测器的飞行器姿态控制研究;檀姗姗;杨洪玖;路继勇;于洋;;燕山大学学报(第06期);55-61 * |
| 基于视觉引导的无人机自主降落研究;徐滔宏;王志胜;鹿山山;;机械与电子(第08期);46-48，53 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112904879A (en) | 2021-06-04 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109911231B (en) | Unmanned aerial vehicle autonomous carrier landing method and system based on GPS and image recognition hybrid navigation | |
| CN109270953B (en) | Multi-rotor unmanned aerial vehicle autonomous landing method based on concentric circle visual identification | |
| Wenzel et al. | Automatic take off, tracking and landing of a miniature UAV on a moving carrier vehicle | |
| CN105867397B (en) | A kind of unmanned plane exact position landing method based on image procossing and fuzzy control | |
| CN106527491B (en) | A kind of fixed-wing unmanned aerial vehicle control system and horizontal crabbing method for controlling trajectory | |
| US10514710B2 (en) | Unmanned aerial vehicle alignment system | |
| US8666571B2 (en) | Flight control system for flying object | |
| CN111324126B (en) | Vision unmanned ship | |
| CN111596687A (en) | Landing guide device and method for mobile platform of vertical take-off and landing unmanned aerial vehicle | |
| CN109992006A (en) | A kind of accurate recovery method and system of power patrol unmanned machine | |
| CN105652879B (en) | A kind of no aileron unmanned plane autonomous flight control method | |
| CN103226356A (en) | Image-processing-based unmanned plane accurate position landing method | |
| CN105335733A (en) | Autonomous landing visual positioning method and system for unmanned aerial vehicle | |
| Chen et al. | System integration of a vision-guided UAV for autonomous landing on moving platform | |
| US20220017235A1 (en) | Autonomous landing systems and methods for vertical landing aircraft | |
| CN106500699B (en) | A kind of position and orientation estimation method suitable for Autonomous landing in unmanned plane room | |
| CN112161567A (en) | Positioning method and system for fully mechanized coal mining face | |
| CN108362296A (en) | The end visual guidance method of AUV and connection station underwater mating | |
| CN110850889B (en) | Unmanned aerial vehicle autonomous inspection system based on RTK navigation | |
| CN108196538B (en) | Three-dimensional point cloud model-based field agricultural robot autonomous navigation system and method | |
| CN115568332A (en) | Automatic following transportation platform for field environment and control method thereof | |
| CN205574253U (en) | Unmanned aerial vehicle fuselage structure and recess assistance -localization real -time platform | |
| CN112904879B (en) | Autonomous tracking landing system and method for four-rotor aircraft based on color block landmarks | |
| CN112859923B (en) | Unmanned aerial vehicle vision formation flight control system | |
| CN109101041A (en) | The dynamic following and dynamic of a kind of unmanned aerial vehicle make a return voyage method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |