CN114879673A - Visual navigation device based on TinyML technology - Google Patents
Visual navigation device based on TinyML technology Download PDFInfo
- Publication number
- CN114879673A CN114879673A CN202210506580.6A CN202210506580A CN114879673A CN 114879673 A CN114879673 A CN 114879673A CN 202210506580 A CN202210506580 A CN 202210506580A CN 114879673 A CN114879673 A CN 114879673A
- Authority
- CN
- China
- Prior art keywords
- tinyml
- visual navigation
- navigation device
- device based
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000000007 visual effect Effects 0.000 title claims abstract description 39
- 238000005516 engineering process Methods 0.000 title claims abstract description 16
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000004891 communication Methods 0.000 claims description 13
- 230000003993 interaction Effects 0.000 claims description 5
- 101100498818 Arabidopsis thaliana DDR4 gene Proteins 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- LHMQDVIHBXWNII-UHFFFAOYSA-N 3-amino-4-methoxy-n-phenylbenzamide Chemical compound C1=C(N)C(OC)=CC=C1C(=O)NC1=CC=CC=C1 LHMQDVIHBXWNII-UHFFFAOYSA-N 0.000 claims description 3
- 238000013480 data collection Methods 0.000 claims description 3
- 238000003708 edge detection Methods 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 3
- 230000009467 reduction Effects 0.000 claims description 3
- 238000010801 machine learning Methods 0.000 abstract description 7
- 238000000034 method Methods 0.000 abstract description 5
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007123 defense Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/30—Noise filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/94—Hardware or software architectures specially adapted for image or video understanding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Electromagnetism (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Artificial Intelligence (AREA)
- Aviation & Aerospace Engineering (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Instructional Devices (AREA)
- Navigation (AREA)
Abstract
A visual navigation device based on a TinyML technology is a visual navigation device and a method based on micro machine learning (TinyML), an SOC (field programmable gate array + ARM architecture) processor is adopted on hardware, certain advantages are achieved when huge image data in a visual system are processed, the TinyML is deployed to an ARM embedded application platform, and secondary processing is performed on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.
Description
Technical Field
The invention relates to the field of computer vision, in particular to a visual navigation device based on a TinyML technology.
Background
The development of computer hardware promotes the continuous upgrade of computer image processing capability and computer related technologies, the existing vision sensor has the advantages of wide signal detection range, complete target information, quick acquisition of environmental information and the like, and meanwhile, the vision system has autonomy and passivity, can normally work in places where radio signals cannot reach and has a very good navigation effect. The visual navigation system has wide application in the fields of mobile communication networks, power systems, rail transit, financial systems, smart cities, aerospace, national defense systems and the like. Meanwhile, with the development of machine learning, TinyML becomes an emerging development subject, and can realize a machine learning algorithm with low resource consumption and low power consumption on a microcontroller with limited resources.
In conventional visual navigation generic devices, high performance based computer frameworks are mostly employed. Therefore, the visual navigation system is huge in size and high in power consumption, and meanwhile, a CPU processor is adopted in the traditional visual navigation system, and the processing speed of the CPU is obviously not capable of following the current use in the face of binocular visual images with very large data volume.
Disclosure of Invention
In order to overcome the defects of the technology, the invention provides the visual navigation device based on the TinyML technology, which has the advantages of small error, strong anti-interference performance and strong real-time performance.
The technical scheme adopted by the invention for overcoming the technical problems is as follows:
a visual navigation device based on TinyML technology comprises a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises:
the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data;
the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time;
the visual navigation unit comprises:
the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit;
the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information;
and the memory unit is used for caching the data processed by the SOC.
Preferably, the SOC includes an FPGA chip and an ARM chip.
Preferably, the communication module is a WIFI wireless communication module.
Furthermore, the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm.
Further, the ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user.
Preferably, the memory cell is a DDR4 memory cell.
Further, a display is included for displaying a map of the unknown environment and location information of the user.
The invention has the beneficial effects that: a visual navigation device and method based on micro machine learning (TinyML) adopts an SOC (field programmable gate array + ARM architecture) processor on hardware, has certain advantages when processing huge image data in a visual system, deploys the TinyML to an ARM embedded application platform, and carries out secondary processing on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.
Drawings
FIG. 1 is a block diagram of the present invention;
FIG. 2 is a flow chart of information processing according to the present invention.
Detailed Description
The invention will be further explained with reference to fig. 1 and 2.
A visual navigation device based on TinyML technology comprises a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises: the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data; the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time; the visual navigation unit comprises: the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit; the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information; and the memory unit is used for caching the data processed by the SOC.
A visual navigation device and method based on micro machine learning (TinyML) adopts an SOC (field programmable gate array + ARM architecture) processor on hardware, has certain advantages when processing huge image data in a visual system, deploys the TinyML to an ARM embedded application platform, and carries out secondary processing on the data processed by the FPGA for the first time on the ARM platform. The system can perform navigation work such as real-time visual image acquisition, image processing, subsequent programming and path planning on an unknown environment. The device can work in any environment, and has the advantages of small error, strong anti-interference performance and strong real-time performance.
An IMU is typically composed of three parts, a gyroscope, an accelerometer, and a magnetometer. The gyroscope is internally provided with a gyroscope, and the axis of the gyroscope is always parallel to the initial direction due to the gyroscopic effect, so that the actual direction can be calculated through the deviation from the initial direction. The accelerometer is used for detecting the magnitude and direction of the acceleration received by the unmanned aerial vehicle, and the magnetometer is used for testing the magnetic field intensity and the direction. The earth's magnetic field is a weak magnetic field having a strength of about 0.5 to 0.6 gauss, which gradually increases from the equator toward the poles. The IMU is mainly used to measure the yaw direction and rotational angular velocity of the user, convert these information into digital signals, and transmit the signals to the communication module.
Preferably, the SOC is composed of an FPGA chip and an ARM chip. The communication module is a WIFI wireless communication module. The memory unit is a DDR4 memory unit, and the DDR4 memory unit has high-speed data read-write capability and can meet the requirements of the design.
Furthermore, the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm. The ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user. The invention adopts an SOC (FPGA + ARM architecture) processor to run a TinyML software stack. In the face of very much image data, the invention adopts the method of 'area conversion speed' in FPGA to process, namely, a large number of logic gate arrays in FPGA are used for processing the bottom layer (combined image filtering) and the middle layer (image enhancement, image segmentation and image identification) of the visual image. The logic gate arrays in the FPGA all run in parallel, so the processing speed of the basic image information is far better than that of a CPU. In the invention, the realization of human-computer interaction and various machine learning algorithms is required, so the functions of map construction, path planning, human-computer interaction and the like are realized by adopting the ARM. Traditionally, machine learning models are always deployed in resource-rich environments, where data is sent from a local device to a cloud for processing. There are currently concerns about privacy, latency, storage, and energy efficiency for this approach. Since TinyML models can be embedded into microcontrollers, they are not resource intensive. This approach is the most efficient and cost effective way to inject artificial intelligence into the visual navigation device.
Further, a display is included for displaying a map of the unknown environment and location information of the user.
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. The visual navigation device based on the TinyML technology is characterized by comprising a data acquisition unit and a visual navigation unit, wherein the data acquisition unit comprises:
the binocular camera is arranged on two sides of the intelligent terminal and used for acquiring real-time image data;
the IMU is arranged in the intelligent terminal and is used for measuring the turning angle and the course indication of a user in real time;
the visual navigation unit comprises:
the communication module is used for carrying out information interaction on information acquired by the binocular camera and the IMU and the visual navigation unit;
the SOC is used for processing information acquired by the binocular camera and the IMU into an environment map and position information;
and the memory unit is used for caching the data processed by the SOC.
2. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the SOC is composed of an FPGA chip and an ARM chip.
3. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the communication module is a WIFI wireless communication module.
4. A visual navigation device based on TinyML technology according to claim 2, characterized in that: the FPGA chip receives signals collected by the data collection unit from the communication module, the signals are subjected to data caching and clock domain crossing signal synchronization through a one-step clock dual-port FIFO, and the processed data are subjected to image noise reduction and filtering enhancement and then are identified by using an edge detection image identification algorithm.
5. A visual navigation device based on TinyML technology as claimed in claim 4, wherein: the ARM chip receives the image identified by the FPGA chip through a high-speed AMBA bus protocol, and carries out TinyML identification and mapping algorithm on the image to obtain a map of an unknown environment and position information of a user.
6. A visual navigation device based on TinyML technology according to claim 1, characterized in that: the memory cell is a DDR4 memory cell.
7. A visual navigation device based on TinyML technology as claimed in claim 5, wherein: a display is also included for displaying a map of the unknown environment and location information of the user.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210506580.6A CN114879673A (en) | 2022-05-11 | 2022-05-11 | Visual navigation device based on TinyML technology |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210506580.6A CN114879673A (en) | 2022-05-11 | 2022-05-11 | Visual navigation device based on TinyML technology |
Publications (1)
Publication Number | Publication Date |
---|---|
CN114879673A true CN114879673A (en) | 2022-08-09 |
Family
ID=82676291
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210506580.6A Pending CN114879673A (en) | 2022-05-11 | 2022-05-11 | Visual navigation device based on TinyML technology |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114879673A (en) |
Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102411535A (en) * | 2011-08-02 | 2012-04-11 | 上海交通大学 | Navigating-SoC (System On Chip) simulating, verifying and debugging platform |
CN103116175A (en) * | 2013-01-18 | 2013-05-22 | 东南大学 | Embedded type navigation information processor based on DSP (digital signal processor) and FPGA (field programmable gata array) |
CN103914071A (en) * | 2014-04-02 | 2014-07-09 | 中国农业大学 | Visual navigation path recognition system of grain combine harvester |
CN108594851A (en) * | 2015-10-22 | 2018-09-28 | 飞智控(天津)科技有限公司 | A kind of autonomous obstacle detection system of unmanned plane based on binocular vision, method and unmanned plane |
CN108733064A (en) * | 2017-04-18 | 2018-11-02 | 中交遥感载荷(北京)科技有限公司 | A kind of the vision positioning obstacle avoidance system and its method of unmanned plane |
CN110196053A (en) * | 2019-06-13 | 2019-09-03 | 内蒙古大学 | A kind of real-time field robot vision navigation method and system based on FPGA |
US20200125961A1 (en) * | 2018-10-19 | 2020-04-23 | Oracle International Corporation | Mini-machine learning |
CN111343379A (en) * | 2020-02-25 | 2020-06-26 | 华南理工大学 | FPGA-based high-speed data acquisition device and method |
CN111524177A (en) * | 2020-04-16 | 2020-08-11 | 华中科技大学 | Micro-miniature high-speed binocular stereoscopic vision system of robot |
CN212012915U (en) * | 2020-02-25 | 2020-11-24 | 华南理工大学 | Binocular image and IMU data high-speed acquisition device |
US20210133607A1 (en) * | 2019-10-31 | 2021-05-06 | Shoreline Iot, Inc. | Systems and methods for self-learning artificial intelligence of things (aiot) devices and services |
CN113391695A (en) * | 2021-06-11 | 2021-09-14 | 山东浪潮科学研究院有限公司 | Low-power-consumption face recognition method based on TinyML |
CN113407583A (en) * | 2021-06-24 | 2021-09-17 | 山东浪潮科学研究院有限公司 | Method for emotion analysis and food recommendation based on AIot and TinyML technology |
US20210319894A1 (en) * | 2020-04-08 | 2021-10-14 | CareBand Inc. | Wearable electronic device and system using low-power cellular telecommunication protocols |
-
2022
- 2022-05-11 CN CN202210506580.6A patent/CN114879673A/en active Pending
Patent Citations (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN102411535A (en) * | 2011-08-02 | 2012-04-11 | 上海交通大学 | Navigating-SoC (System On Chip) simulating, verifying and debugging platform |
CN103116175A (en) * | 2013-01-18 | 2013-05-22 | 东南大学 | Embedded type navigation information processor based on DSP (digital signal processor) and FPGA (field programmable gata array) |
CN103914071A (en) * | 2014-04-02 | 2014-07-09 | 中国农业大学 | Visual navigation path recognition system of grain combine harvester |
CN108594851A (en) * | 2015-10-22 | 2018-09-28 | 飞智控(天津)科技有限公司 | A kind of autonomous obstacle detection system of unmanned plane based on binocular vision, method and unmanned plane |
CN108733064A (en) * | 2017-04-18 | 2018-11-02 | 中交遥感载荷(北京)科技有限公司 | A kind of the vision positioning obstacle avoidance system and its method of unmanned plane |
US20200125961A1 (en) * | 2018-10-19 | 2020-04-23 | Oracle International Corporation | Mini-machine learning |
CN110196053A (en) * | 2019-06-13 | 2019-09-03 | 内蒙古大学 | A kind of real-time field robot vision navigation method and system based on FPGA |
US20210133607A1 (en) * | 2019-10-31 | 2021-05-06 | Shoreline Iot, Inc. | Systems and methods for self-learning artificial intelligence of things (aiot) devices and services |
CN111343379A (en) * | 2020-02-25 | 2020-06-26 | 华南理工大学 | FPGA-based high-speed data acquisition device and method |
CN212012915U (en) * | 2020-02-25 | 2020-11-24 | 华南理工大学 | Binocular image and IMU data high-speed acquisition device |
US20210319894A1 (en) * | 2020-04-08 | 2021-10-14 | CareBand Inc. | Wearable electronic device and system using low-power cellular telecommunication protocols |
CN111524177A (en) * | 2020-04-16 | 2020-08-11 | 华中科技大学 | Micro-miniature high-speed binocular stereoscopic vision system of robot |
CN113391695A (en) * | 2021-06-11 | 2021-09-14 | 山东浪潮科学研究院有限公司 | Low-power-consumption face recognition method based on TinyML |
CN113407583A (en) * | 2021-06-24 | 2021-09-17 | 山东浪潮科学研究院有限公司 | Method for emotion analysis and food recommendation based on AIot and TinyML technology |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN109435955B (en) | Performance evaluation method, device and equipment for automatic driving system and storage medium | |
CN109931945B (en) | AR navigation method, device, equipment and storage medium | |
WO2018217663A1 (en) | Image capture direction recognition method and server, surveillance method and system and image capture device | |
US20230394756A1 (en) | Three-dimensional reconstruction method, three-dimensional reconstruction apparatus and storage medium | |
CN110276972A (en) | A kind of object cognitive method and system based on car networking | |
CN111125283B (en) | Electronic map construction method and device, computer equipment and storage medium | |
CN112967392A (en) | Large-scale park mapping and positioning method based on multi-sensor contact | |
CN113807470B (en) | Vehicle driving state determination method and related device | |
CN111694012A (en) | Three-dimensional terrain online generation method and system based on airborne laser radar | |
EP4206731A1 (en) | Target tracking method and device | |
CN103033836A (en) | Vehicle navigation pointing device and method | |
Wang et al. | Remote sensing monitoring method based on BDS-based maritime joint positioning model | |
CN115585818A (en) | Map construction method and device, electronic equipment and storage medium | |
CN111121755B (en) | Multi-sensor fusion positioning method, device, equipment and storage medium | |
CN110515110A (en) | Method, apparatus, equipment and the computer readable storage medium of data assessment | |
CN116539053A (en) | Real-time global point cloud map construction method, system, equipment and storage medium | |
Cunha et al. | Hardware-accelerated data decoding and reconstruction for automotive LiDAR sensors | |
CN111652276B (en) | All-weather portable multifunctional bionic positioning and attitude-determining viewing system and method | |
CN114879673A (en) | Visual navigation device based on TinyML technology | |
CN211878189U (en) | Ground-air cooperative vision navigation device based on improved map optimization SLAM | |
CN103335645B (en) | The pinpoint image acquiring method in space towards landscape planning | |
CN113715817B (en) | Vehicle control method, vehicle control device, computer equipment and storage medium | |
CN214492786U (en) | Vehicle-mounted automatic driving hardware circuit and automatic driving automobile | |
CN106443798A (en) | Road underground data acquisition and fusion method and system | |
Chen et al. | Fully Onboard Single Pedestrian Tracking On Nano-UAV Platform |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |