CN110084167B - Scale data image processing method and device - Google Patents
Scale data image processing method and device Download PDFInfo
- Publication number
- CN110084167B CN110084167B CN201910321371.2A CN201910321371A CN110084167B CN 110084167 B CN110084167 B CN 110084167B CN 201910321371 A CN201910321371 A CN 201910321371A CN 110084167 B CN110084167 B CN 110084167B
- Authority
- CN
- China
- Prior art keywords
- white line
- angle
- determining
- current frame
- roll angle
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V2201/00—Indexing scheme relating to image or video recognition or understanding
- G06V2201/02—Recognising information on displays, dials, clocks
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a scale data image processing method and device. The method comprises the following steps: acquiring a current frame of a received video; determining the position and the inclination angle of a white line on the current frame; determining a pitch angle according to the distance from the white line to a preset aircraft symbol center pixel point; and determining a roll angle according to the inclination angle. The invention can directly extract the display values of the current key parameters from the display picture and send the display results to the application program responsible for monitoring so as to meet the identification requirements of different scale images and ensure the data integrity of the parameters in the image generation process.
Description
Technical Field
The embodiment of the invention relates to the technical field of aviation onboard and hardware graphic processing design, in particular to a scale data image processing method and device.
Background
Along with the fact that ultra-complex commercial goods shelf electronic components such as a CPU (central Processing Unit) and an image Processing Unit (GPU) are widely applied to an aviation cabin display system, the integration level of functions of a display Unit is higher, and the challenges are brought to the safety and the reliability of products while the performance and the economy are brought. Thus, airworthiness regulations put high demands on the development process of cockpit display units:
on one hand, the data integrity of the key flight parameters in the display unit is required to reach a level less than 1.0E-7/flight hour, and on the other hand, the development guarantee grade of each component module in the whole display unit is required to reach B grade or above.
Through the analysis of the current display system architecture, various input data can be sent to a display screen for display only by the data processing of a CPU and the image generation of a GPU in a display unit and then by the image superposition of a Field-Programmable Gate Array (FPGA), and according to the experience analysis generally accepted in the industry, the data processing channel can only reach 1.0E-6/flight hour in the integrity. Meanwhile, due to the lack of effective historical service data of the CPU and the GPU, abnormal states caused by the failure of the CPU and the GPU and errors generated in the design process cannot be completely described, so that the reliability and the safety of the abnormal states cannot be accurately evaluated, and finally the airworthiness reliability of the generated data graph cannot be effectively guaranteed.
Disclosure of Invention
The invention provides a scale data image processing method and device, which can directly extract the display values of the current key parameters from a display picture and send the display results to an application program responsible for monitoring so as to meet the identification requirements of different scale images and ensure the data integrity of the parameters in the image generation process.
In a first aspect, an embodiment of the present invention provides a scale data image processing method, including:
acquiring a current frame of a received video;
determining the position and the inclination angle of a white line on the current frame;
determining a pitch angle according to the distance from the white line to a preset aircraft symbol center pixel point;
and determining a roll angle according to the inclination angle.
Optionally, the determining the position and the inclination angle of the white line on the current frame includes:
respectively identifying a day, a ground and a white line from the current frame;
obtaining a contour of the white line from the identified day, ground and white line;
taking the position of the outline of the white line as the position of the white line;
determining a slope angle of the outline of the white line as a slope angle of the white line.
Optionally, the obtaining the outline of the white line from the identified day, ground and white line includes:
intercepting an image including the day, the ground and a white line from the current frame;
detecting the edge of the image to obtain a contour binary image;
carrying out Hough transformation on the coordinates of each pixel point in the contour binary image to obtain Hough coordinates of the corresponding pixel point;
determining the number of pixel points corresponding to the same Hough coordinate;
and taking the outline formed by the pixels with the maximum number as the outline of the white line.
Optionally, the determining the pitch angle according to the distance between the white line and the preset pixel point of the aircraft symbol center includes:
determining the vertical distance from the white line to a preset aircraft symbol center pixel point;
and taking the angle corresponding to the vertical distance as the pitch angle according to the preset corresponding relation between the distance and the angle.
Optionally, the determining the roll angle according to the inclination angle includes:
determining the roll angle according to a first formula, wherein the first formula is as follows: θ' =180 ° - (θ +90 °);
wherein θ is the skew angle and θ' is the roll angle.
Optionally, the method further includes:
receiving the collected actual pitch angle and actual roll angle;
comparing the actual pitch angle with the pitch angle, and comparing the actual roll angle with the roll angle;
the comparison result is sent back to the processor.
In a second aspect, there is provided a scale class data image processing apparatus comprising:
the acquisition module is used for acquiring the current frame of the received video;
the first determining module is used for determining the position and the inclination angle of a white line on the current frame;
the second determining module is used for determining a pitch angle according to the distance from the white line to a preset aircraft symbol center pixel point;
and the third determining module is used for determining a roll angle according to the inclination angle.
Optionally, the first determining module includes:
the identification unit is used for respectively identifying a day, a ground and a white line from the current frame;
an acquisition unit configured to acquire a contour of a white line from the identified sky, ground, and white line;
the processing unit is used for taking the position of the outline of the white line as the position of the white line; determining a slope angle of the outline of the white line as a slope angle of the white line.
In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, which computer program, when executed by a processor, implements the scale-like data image processing method according to any one of the first aspect.
The embodiment of the invention skillfully calculates the pitch angle and the roll angle in the current display picture by utilizing the FPGA image processing function and the parallel operation characteristic, combining the display characteristic of the scale type flight parameters of the cockpit display system and adopting a linear detection method taking the sky line as the reference. The display image of each frame can be covered, the digital and graphic processing process of the flight parameters is ensured to be in a monitoring state, the completeness of the scale type flight parameter display is improved, and the airworthiness evidence-obtaining confidence coefficient of the application of ultra-complex commercial shelf electronic components such as a CPU (central processing unit), a GPU (graphic processing unit) and the like in the field of aviation is improved, so that the safety of the airplane is improved.
Drawings
Fig. 1 is a schematic diagram of an overall architecture design according to an embodiment of the present invention.
Fig. 2 is a schematic diagram of a scale-type flight parameter display feature according to an embodiment of the present invention.
Fig. 3 is a flowchart of a scale data image processing method according to an embodiment of the present invention.
Fig. 4 is a flowchart of hough transform calculation according to an embodiment of the present invention.
Fig. 5 is a diagram of a data comparison monitoring architecture according to an embodiment of the present invention.
Fig. 6 shows the simulation result of the algorithm according to the embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the invention and are not limiting of the invention. It should be further noted that, for the convenience of description, only some of the structures related to the present invention are shown in the drawings, not all of the structures.
In order to satisfy the airworthiness confidence of the display data, the whole data path from data processing to image generation must be covered through monitoring so as to prove the correctness of the data content in the data path. Therefore, it is necessary to design and monitor a corresponding identification method for key parameters in different display forms, so that the generation process of the key parameter graph can be more intuitively and conveniently ensured to meet the airworthiness requirement.
FIG. 1 depicts the overall design architecture of an embodiment of the present invention, including the interaction of the cockpit display unit internal data processing path with the present monitor. Firstly, input data enters a cockpit display system from an airplane network through an I/O bus interface 11, then enters an I/O data processing unit 12, the interior of the I/O data processing unit contains an I/O data processing application program and a monitoring processing application program, wherein the data processing application program analyzes the input data and then carries out display application processing, the process comprises application software such as PFD (field programmable data processing), EICAS (enhanced interactive application systems) and the like, the application software processes the input data and sends the input data to a GPU (graphics processing unit) for graphics rendering and symbol generation, and as display windows from different applications may be contained in a display picture, video data generated by each application software through the GPU need to be overlaid with video images to form a frame of display picture. After the video images are superimposed, the monitoring functional unit 16 of the embodiment of the present invention receives the superimposed images, and also receives the original data for generating the images sent by the monitoring processing application program with serial number 12, and returns the flight data monitoring result to the monitoring processing application program with serial number 12, thereby finally forming a data closed loop. Therefore, the closed loop formed by the monitoring function of the embodiment of the invention can cover three functional units with serial numbers of 13, 14 and 15, which are the main components of the graphic display function of the display unit.
FIG. 2 illustrates display characteristics of a scale-like flight parameter for which embodiments of the present invention are directed. Wherein 21 is a pitch angle scale bar, the middle position is a 0-degree line, more than 0-degree line represents a positive number, and the lower part represents a negative number; 23 is an airplane symbol mark, the airplane symbol mark is fixedly positioned in the center of a picture in the display process, and 22 is an airplane symbol central point; 24 is a roll angle scale bar, the roll angle scale bar is displayed as an arc scale bar, the middle is 60 degrees respectively at the left and right of 0 degree, the roll angle arc scale bar takes the central point of the 22 airplane symbol as the center of a circle, the roll angle arc scale bar is fixed in the display process, and the roll angle indicator rolls left and right on the scale bar along with the airplane posture; 25 is an invalid display mark of the pitch angle and roll angle data of the airplane; and 26 is a sky line in PFD software, wherein the sky line is overlapped with a pitch angle 0-degree line and rotates along with the rolling of the airplane.
From the above display characteristics, we can find that the pitch and roll display is closely related to the sky and ground. The rotation angle of the natural ground wire depends on the current roll angle of the airplane, a fixed relation exists between the angle of the natural ground wire and the roll angle, and in addition, the numerical values of the pitch angle scales are linearly and uniformly distributed, namely, the pitch angle numerical value indicated in the current display picture can be calculated according to the distance from the center point of the airplane symbol to the natural ground wire (namely, the pitch angle 0-degree line). Therefore, the method is based on the space-ground line, and realizes the identification and monitoring algorithm of the scale key flight parameters.
Fig. 3 depicts the flow of pitch and roll angle identification functions in an embodiment of the present invention. The specific process is as follows:
step one, a video input 31 unit enters an image recognition module, and day, ground and white line recognition and range interception are carried out on the video in a 32 unit.
According to the difference of RGB values, the ground, the day and the white line in the image are distinguished. For example, if the RGB values of the pixel point satisfy x "9B" < = R < = x "A5" and x "5F" < = G < = x "69" and x "24" <= B < = x "2E", respectively, then ground is identified; identifying as sky if the RGB values of the pixel point respectively satisfy x "5e" < = R < = x "68" and x "99" < = G < = x "a3" and x "f5" < = B; if the RGB values of the pixel point satisfy R > = x "f0" and G > = x "f0" and B > = x "f0", respectively, it is identified as a white line. Then, in order to reduce the amount of computation and complexity of the later period, the PFD image is cut out from the complete image. Images with line range 25.
And step two, performing edge detection on the input image, obtaining a binary image after the edge detection, wherein the value of an edge point pixel is 1, and the value of a non-edge point pixel is 0, and storing the progress of the edge image. The binary image is a contour binary image of a scene in the image.
Reading pixel points of the binary image from the memory, calculating coordinate positions of the pixel points, performing Hough transform on edge pixel points, calculating rho values 36, and simultaneously performing statistical accumulation on parameter space to realize the parallelism of the Hough transform on time.
The most critical operation in Hough transform is to calculate rho value according to the coordinate position of the pixel point. The calculation related to the trigonometric function in the formula (1) is realized by adopting a lookup table, the numbers in the lookup table are respectively the result of multiplying a sine function and a cosine function value of 0-89 degrees by 1024 and then rounding, the step length delta theta of the angle is =1 degree, and thus only unsigned multiplication and addition operation are required to be carried out in the FPGA.
And calculating the coordinates of the pixel points in real time under the condition of continuous input, and performing Hough transformation when the selected pixel points are edge points. When theta is more than or equal to 0 and less than 90 degrees, the calculation is carried out by adopting the formula (1). When theta is less than or equal to 90 DEG and less than 180 DEG, alpha =180 DEG-theta and is calculated by the formula (2).
ρ＝xcosθ+ysinθ, 0≤θ<90° (1)
ρ＝ycosα-xsinα, 0≤α<90° (2)
Step four, specific parameters 40 of the sky line and the earth line can be obtained after the rho value is completed, and then specific numerical values of the pitch angle 41 and the roll angle 42 are respectively calculated. And the pitch angle can be calculated by carrying out geometric conversion according to the distance from the pixel point at the center of the aircraft symbol to the straight line. The roll angle can be calculated by adding 90 degrees to the calculated angle of the antenna and the ground wire, and the roll angle is calculated by a formula (3) in consideration of the range of the roll angle from +60 degrees to-60 degrees.
θ'＝180°-(θ+90°) (3)
Fig. 4 illustrates a specific flow of the calculation of the ρ value in the embodiment of the present invention. Mainly comprises a lookup table, a multiplier and an adder. With a pipeline computing structure, only one clock cycle is needed to complete rho value computation of one angle in Hough transform. Firstly, performing coordinate calculation 52 on a stored binary image 51 to obtain coordinates (x, y), then performing multiplier operation (53 and 54) on the coordinates and a sine function 57 and a cosine function 58 respectively, then performing accumulation statistics 55 on rho and theta in a Hough parameter space, and finally storing 56 calculation data.
Fig. 5 depicts the data comparison functional architecture in this patent. The identified pitch angle 61 and roll angle 62 are sent to a monitoring and comparing module 66 to wait for processing, meanwhile, the CPU monitoring and processing application 63 sends the original data of the pitch angle and roll angle to the FPGA, after the SPI data is analyzed, the original data are also sent to the monitoring and comparing module 66 to be compared with the identified pitch angle 61 and roll angle 62 respectively, and finally, the comparison result is packaged by the SPI data and sent to the CPU monitoring and processing application. And at this point, the integrity of the scale key parameters (pitch angle and roll angle) of the display system is monitored.
FIG. 6 depicts the results of an algorithm simulation test of an embodiment of the present invention. 71 is the indication state of the pitch angle and the roll angle in the display screen, and 72 is the recognition result of the algorithm of the embodiment of the invention, wherein the detection value of the roll angle is C0EE61B8, the result of conversion into decimal according to the formula is-7.45 degrees, which is similar to the display result of visual observation, the detection value of the pitch angle is 41316768, the result of conversion into decimal according to the formula is 11.09 degrees, which is similar to the display result of visual observation. The simulation result proves the correctness and the availability of the display system scale key parameter integrity monitoring algorithm provided by the embodiment of the invention.
It is to be noted that the foregoing description is only exemplary of the invention and that the principles of the technology may be employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. Therefore, although the present invention has been described in some detail by the above embodiments, the invention is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the invention, and the scope of the invention is determined by the scope of the appended claims.
Claims (3)
1. A scale data image processing method is characterized by comprising the following steps:
acquiring a current frame of a received video;
determining the position and the inclination angle of a white line on the current frame;
determining a pitch angle according to the distance from the white line to a preset aircraft symbol center pixel point;
determining a roll angle according to the inclination angle;
wherein the determining the position and the inclination angle of the white line on the current frame comprises:
respectively identifying a day, a ground and a white line from the current frame;
obtaining the outline of the white line from the identified day, ground and white lines;
taking the position of the outline of the white line as the position of the white line;
determining a bevel angle of the outline of the white line as a bevel angle of the white line;
wherein the obtaining the outline of the white line from the identified day, ground and white line comprises:
intercepting an image including the day, the ground and a white line from the current frame;
detecting the image at the edge to obtain a contour binary image; carrying out Hough transformation on the coordinates of each pixel point in the contour binary image to obtain Hough coordinates of the corresponding pixel point;
determining the number of pixel points corresponding to the same Hough coordinate; taking the outline formed by the pixels with the maximum number as the outline of the white line;
wherein, according to the distance between the white line and the preset aircraft symbol center pixel point, determining the pitch angle comprises:
determining the vertical distance from the white line to a preset aircraft symbol center pixel point;
according to a preset corresponding relation between the distance and the angle, taking the angle corresponding to the vertical distance as the pitch angle;
wherein, according to the bevel angle, confirm the roll angle, include:
determining the roll angle according to a first formula, wherein the first formula is as follows:
θ''＝180°-(θ+90°)；
wherein the θ is the skew angle, the θ ″ is the roll angle;
wherein the method further comprises:
receiving the collected actual pitch angle and actual roll angle;
comparing the actual pitch angle with the pitch angle, and comparing the actual roll angle with the roll angle;
the comparison result is sent back to the processor.
2. A scale-like data image processing apparatus, characterized by comprising:
the acquisition module is used for acquiring the current frame of the received video;
the first determining module is used for determining the position and the inclination angle of a white line on the current frame;
the second determining module is used for determining a pitch angle according to the distance from the white line to a preset aircraft symbol center pixel point;
the third determining module is used for determining a roll angle according to the inclination angle;
wherein the first determining module comprises:
the identification unit is used for respectively identifying a day, a ground and a white line from the current frame;
an acquisition unit configured to acquire a contour of a white line from the identified day, ground, and white line;
the processing unit is used for taking the position of the outline of the white line as the position of the white line and determining the inclination angle of the outline of the white line as the inclination angle of the white line;
wherein the obtaining unit is specifically configured to:
intercepting an image including the day, the ground and a white line from the current frame;
detecting the edge of the image to obtain a contour binary image; carrying out Hough transformation on the coordinates of each pixel point in the contour binary image to obtain Hough coordinates of the corresponding pixel point;
determining the number of pixel points corresponding to the same Hough coordinate; taking the outline formed by the pixels with the maximum number as the outline of the white line;
the second determining module is specifically configured to:
determining the vertical distance from the white line to a preset aircraft symbol center pixel point;
according to a preset corresponding relation between the distance and the angle, taking the angle corresponding to the vertical distance as the pitch angle;
wherein the third determining module is specifically configured to:
determining the roll angle according to a first formula, wherein the first formula is as follows:
θ''＝180°-(θ+90°)；
wherein θ is the skew angle and θ ″ is the roll angle;
wherein the apparatus further comprises:
the receiving module is used for receiving the collected actual pitch angle and the actual roll angle;
the comparison module is used for comparing the actual pitch angle with the pitch angle and comparing the actual roll angle with the roll angle;
and the sending module is used for sending the comparison result back to the processor.
3. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method according to claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910321371.2A CN110084167B (en) | 2019-04-19 | 2019-04-19 | Scale data image processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910321371.2A CN110084167B (en) | 2019-04-19 | 2019-04-19 | Scale data image processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110084167A CN110084167A (en) | 2019-08-02 |
| CN110084167B true CN110084167B (en) | 2022-11-04 |
Family
ID=67415964
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910321371.2A Active CN110084167B (en) | 2019-04-19 | 2019-04-19 | Scale data image processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110084167B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111506074B (en) * | 2020-05-08 | 2022-08-26 | 佳木斯大学 | Machine control method of crop tedding dust collection device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008011135A (en) * | 2006-06-29 | 2008-01-17 | Fuji Xerox Co Ltd | Image processing device and image processing program |
| CN108052942A (en) * | 2017-12-28 | 2018-05-18 | 南京理工大学 | A kind of visual pattern recognition methods of aircraft flight attitude |
-
2019
- 2019-04-19 CN CN201910321371.2A patent/CN110084167B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008011135A (en) * | 2006-06-29 | 2008-01-17 | Fuji Xerox Co Ltd | Image processing device and image processing program |
| CN108052942A (en) * | 2017-12-28 | 2018-05-18 | 南京理工大学 | A kind of visual pattern recognition methods of aircraft flight attitude |
Non-Patent Citations (2)
| Title |
|---|
| 一种足球机器人场地白线的提取方法;孙虹等;《科学技术与工程》;20100108(第01期);全文 * |
| 民航航班跟踪视频关键帧提取方法研究;宁煜西等;《空军预警学院学报》;20180615(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110084167A (en) | 2019-08-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3621034A1 (en) | Method and apparatus for calibrating relative parameters of collector, and storage medium | |
| US20180268601A1 (en) | Three-dimensional pose estimation of symmetrical objects | |
| EP3346445B1 (en) | Methods and devices for extracting an object from a video sequence | |
| US20140355820A1 (en) | Estimating a pose of a camera for volume estimation | |
| CN111627015B (en) | Small sample defect identification method, device, equipment and storage medium | |
| US11348275B2 (en) | Methods and apparatuses for determining bounding box of target object, media, and devices | |
| CN110222641B (en) | Method and apparatus for recognizing image | |
| EP3633606A1 (en) | Information processing device, information processing method, and program | |
| US20190318535A1 (en) | Display data processing method and apparatus | |
| CN110084167B (en) | Scale data image processing method and device | |
| EP4068210A1 (en) | System and method for automated estimation of 3d orientation of a physical asset | |
| CN113077548A (en) | Collision detection method, device, equipment and storage medium for object | |
| CN113743385A (en) | Unmanned ship water surface target detection method and device and unmanned ship | |
| US20150269454A1 (en) | Extraction device, method, and computer program product | |
| US5586204A (en) | Integrity checking procedure for high throughput data transformations | |
| US20120106868A1 (en) | Apparatus and method for image correction | |
| CN107992607B (en) | Typhoon forecast picture drawing method and device, computer equipment and storage medium | |
| CN111489433A (en) | Vehicle damage positioning method and device, electronic equipment and readable storage medium | |
| CN112651931B (en) | Building deformation monitoring method and device and computer equipment | |
| CN110636373B (en) | Image processing method and device and electronic equipment | |
| US20180330514A1 (en) | Selective 3d registration | |
| CN114495105A (en) | Image tilt correction method, device, storage medium and computer equipment | |
| CN110084164B (en) | Digital data image processing method and device | |
| US20240153274A1 (en) | Artificial intelligence enabled distance event detection using image analysis | |
| CN116385952B (en) | Distribution network line small target defect detection method, device, equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |