CN105450950A - Method for removing jitter from aerial video of unmanned aerial vehicle - Google Patents
Method for removing jitter from aerial video of unmanned aerial vehicle Download PDFInfo
- Publication number
- CN105450950A CN105450950A CN201510888639.2A CN201510888639A CN105450950A CN 105450950 A CN105450950 A CN 105450950A CN 201510888639 A CN201510888639 A CN 201510888639A CN 105450950 A CN105450950 A CN 105450950A
- Authority
- CN
- China
- Prior art keywords
- angle
- image
- straight line
- road
- aerial
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 28
- 238000012545 processing Methods 0.000 claims abstract description 7
- 238000007781 pre-processing Methods 0.000 claims abstract description 5
- 238000012937 correction Methods 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 claims description 5
- 238000003708 edge detection Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 3
- 238000001514 detection method Methods 0.000 description 5
- 206010044565 Tremor Diseases 0.000 description 4
- 238000012544 monitoring process Methods 0.000 description 4
- 230000006641 stabilisation Effects 0.000 description 3
- 238000011105 stabilization Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 238000013507 mapping Methods 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000011426 transformation method Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
- H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
The invention discloses a method for removing jitter from an aerial video of an unmanned aerial vehicle. The method comprises the following steps: a step 1: preprocessing a road aerial image; a step 2: extracting a straight line based on the aerial image; a step 3: calculating a straight line direction histogram; and a step 4: correcting and removing jitter from the aerial image. According to the method disclosed by the invention, by means of the property that most straight lines extracted by detecting the road aerial image of the unmanned aerial vehicle are parallel to the road direction, a relative angle histogram of the straight lines is established, the angle corresponding to the maximum peak point of the histogram is detected to obtain the road direction in the aerial image, the image is rotated according to the angle to adjust the road to the horizontal direction, in order to remove jitter from the aerial image. The method disclosed by the invention is based on the image processing technology and is used for intelligently sensing the road direction and rotating the aerial image to remove the jitter from the road aerial image of the unmanned aerial vehicle.
Description
Technical Field
The invention belongs to the technical field of image processing, relates to an unmanned aerial vehicle aerial video jitter removal method, and particularly relates to a straight line extraction method based on image edge detection and Hough transformation, a road direction detection method based on a straight line angle histogram and a correction method based on a road direction rotation image, so as to achieve the aim of image jitter removal.
Background
The unmanned aerial vehicle has the advantages of strong maneuverability, wide visual field, no limitation of a flight route by the terrain and the like, and is widely applied to the fields of surveying and mapping, aerial photography, traffic monitoring and the like. The application of the two-axis and three-axis stable holder eliminates the problem of image jitter of aerial images caused by attitude adjustment of an unmanned aerial vehicle, external conditions (gusts) and the like, but in many application scenes, such as unmanned aerial vehicle highway traffic monitoring, the problem of road jitter in the aerial images is caused because the unmanned aerial vehicle does not have the capability of intelligently identifying roads and the included angle between the flight advancing direction of the unmanned aerial vehicle and the road direction is changed all the time, and no matter the jitter is the traffic situation of ground personnel through image monitoring, or the problem of great obstacle is brought to the extraction of traffic parameters based on an image processing method. Therefore, the intelligent method for removing the tremble of the road aerial image is particularly important.
At present, the image shake removing method of the aerial image mainly comprises two modes of mechanical shake removing and registration shake removing. Hardware removes trembling, is to install image acquisition equipment on the mechanical cloud platform that has from increasing steady function, what this kind of mode was eliminated is the image shake that aerial image caused because unmanned aerial vehicle attitude adjustment, external condition (gust) etc.. The image registration method is characterized in that the motion of an image background is obtained by tracking feature points in an image, so that the shaking is eliminated by an affine transformation method, and a basic registration frame needs to be selected in advance for the shaking elimination. In actual unmanned aerial vehicle traffic monitoring application, a user is interested in a road area, and a stable road image needs to be acquired.
Disclosure of Invention
The invention provides an image de-jittering method based on a road direction histogram, which aims at solving the problem that the existing image de-jittering method cannot intelligently eliminate road jitter in unmanned aerial vehicle road aerial images. The invention relates to a road aerial image de-jittering method based on a road direction histogram. The image de-jittering method can intelligently sense the road direction, thereby realizing the image de-jittering which cannot be realized by the existing method.
The invention provides a universally applicable road aerial image shaking removing method based on a brand-new research approach point, and the method is realized by the following steps:
step 1: preprocessing of road aerial images
The unmanned aerial vehicle aerial video is unframed, a single-frame RGB color road aerial image is obtained, and the RGB color image is converted into a gray-scale image;
step 2: extracting straight lines based on aerial images
Processing the gray-scale image in the previous step by adopting a Canny edge detection operator to obtain an edge contour image, detecting the edge contour image based on Hough transform, and obtaining a straight line;
and step 3: calculating a straight-line direction histogram
The angles of the straight lines detected in the calculation step are calculated, and then a straight line relative angle histogram is calculated. And extracting the angle corresponding to the maximum peak point in the histogram, wherein the angle is the direction of the road in the frame of aerial image.
And 4, step 4: correction debouncing of aerial images
The road aerial image is rotated clockwise by the angle obtained in the previous step, so that the road with different orientation in the aerial image can be uniformly adjusted to be in the horizontal direction, and the road aerial image is debounced.
The invention has the advantages that:
(1) according to the method, a relative angle histogram of straight lines is established by utilizing the characteristic that most straight lines extracted by unmanned aerial vehicle aerial image detection are parallel to the direction of a road, the direction of the road in the aerial image can be obtained by detecting the angle corresponding to the maximum peak point of the histogram, and the road is adjusted to be in the horizontal direction according to the angle rotation image, so that the aerial image is subjected to debouncing;
(2) the unmanned aerial vehicle road aerial image stabilization method is based on an image processing technology, and the unmanned aerial vehicle road aerial image stabilization is realized by intelligently sensing the road direction and rotating the aerial image;
(3) the method can be suitable for removing the tremble of the aerial images in various road scenes, has good robustness and high operation speed, does not need external data support (such as GIS map data), and has multipoint innovation.
Drawings
FIG. 1 is a grayscale image of an aerial image;
FIG. 2 is a diagram of edge contour detection based on a Canny edge detector;
FIG. 3 is a graph of a line of sight based on Hough transform detection;
FIG. 4 is a diagram of image coordinate system and straight line angle direction definition;
FIG. 5 is a linear direction relative histogram;
FIG. 6 is a horizontal correction map based on road direction;
FIG. 7 is a flow chart of the method of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
The invention provides an aerial image de-jittering method based on a road direction histogram, which comprises the steps of firstly preprocessing an extracted road aerial image and converting the preprocessed road aerial image into a gray-scale image; then detecting a gray level image based on a Canny edge detection operator to obtain an edge contour image, and then carrying out Hough transform on the edge contour image to detect a straight line; calculating the angle of the detected straight line, calculating a relative angle histogram of the direction of the straight line, and extracting the angle of the straight line corresponding to the maximum peak value of the relative angle histogram, namely obtaining the direction of the road; and then, performing instantaneous needle rotation on the original aerial image based on the detected road direction, and rotating the road into the horizontal direction to realize the stabilization of the aerial image. The process of the method for removing the jitter of the aerial image based on the road direction histogram is shown in fig. 7, and the specific processing steps are as follows:
step 1: preprocessing of road aerial images
The road aerial photography video is deframed to obtain an RGB color image, and the RGB color image is converted into a gray scale image as shown in figure 1.
Step 2: extracting straight lines based on aerial images
After the gray image is obtained, the gray image is processed by a Canny edge detection operator to obtain a binary edge contour map, as shown in fig. 2, the contour map of fig. 2 is subjected to hough transform, and a line map is detected and obtained, as shown in fig. 3.
And step 3: calculating a straight-line direction histogram
Calculating the angle of each line in FIG. 1, wherein the angle of each line is defined as shown in FIG. 4, where O (0,0) is the origin of the pixel coordinates of the image, O (0,0) is the starting point, the column coordinate axis of the image is to the right, and the image is to the bottomThe row coordinate axis of the image. For any straight line i in FIG. 3, the two endpoints are P1 and P2, respectively, where the pixel coordinates of P1 and P2 are (c)i_1,ri_1) And (c)i_2,ri_2)，Is the included angle between the straight line i and the horizontal direction,indicating the angle rotated by rotating a horizontal straight line counterclockwise to be parallel to the straight line i, the angle of any straight line iThe calculation method of (2) is shown in the following formula (1):
wherein,is an integer, is calculated by rounding, and
based on the calculated straight line angle, a relative angle histogram is calculated, which is detailed as follows:
(1) identifying the number n of lines detected in fig. 3;
(2) set 180 cell intervals theta1～θ180Wherein: the interval is as follows: theta1＝[0°,1°),θ2＝[1°,2°),…,θi＝[(i-1)°,i°),…,θ180＝[179°,180°)；
(3) For n straight lines detected in FIG. 3 (n is confirmed in step (1)), the angles of the straight lines are counted, and if the angle is θiM straight lines of (a) are included, the angle theta is considerediThe number of occurrences is m, here by h (θ)i) Representing the angle theta of a straight lineiThe number of occurrences, i.e., the frequency.
(4) The frequency h (theta) of the occurrence of the linear angle counted in the step (3)i) Performing normalization processing to calculate each linear angle thetaiRelative frequency of occurrence H (θ)i) The calculation method is H (theta)i)＝h(θi) And/n. Frequency h (theta)i) The purpose of normalization is to simplify the computation, reducing the magnitude.
(5) A histogram of relative line angles is plotted, as shown in fig. 5, with the horizontal axis of the histogram representing the direction angle of the line in the range of [0 °,180 °), and the vertical axis representing the relative frequency of occurrence of the line angle in the range of [0,1 °]. In the histogram shown in FIG. 5, each angle θiThe height value of the corresponding straight line is the relative frequency value H (theta) calculated in the step (4)i) The relative frequency value corresponding to the straight line with the highest value in fig. 5 is referred to as the maximum histogram peak.
(6) Maximum peak H (θ) in fig. 5k) By the formula H (theta)k)＝Max{H(θ1),H(θ2),…,H(θi),…,H(θ180) Is calculated, wherein thetakThe angle of the straight line corresponding to the maximum histogram peak.
Angle thetakI.e. the direction of the road in the frame image. The detection principle of the road direction is as follows: in the aerial image, the contour lines of the road and the surrounding structures are parallel to the road direction, so that the angle with the highest relative frequency of the appearance of the straight line angle in fig. 3 is the road direction, and the principle is applicable to general straight lines.
And 4, step 4: correction debouncing of aerial images
Rotating the road aerial image clockwise by thetakIn other words, the road in the image can be corrected to the horizontal direction, and the corrected image is as shown in fig. 6.
By repeating the steps 1 to 4, the purpose of removing the trembles of the aerial video can be achieved based on the road direction correction.
Claims (2)
1. An unmanned aerial vehicle aerial video jitter removal method comprises the following steps:
step 1: preprocessing of road aerial images
The unmanned aerial vehicle aerial video is unframed, a single-frame RGB color road aerial image is obtained, and the RGB color image is converted into a gray-scale image;
step 2: extracting straight lines based on aerial images
Acquiring a gray-scale image edge contour map by adopting a Canny edge detection operator, detecting the edge contour map based on Hough transform, and acquiring a straight line;
and step 3: calculating a straight-line direction histogram
Calculating the angle of the straight line, then calculating the histogram of the relative angle of the straight line, and extracting the angle theta corresponding to the maximum peak point in the histogramkThe angle is the direction of the road in the frame of aerial image;
and 4, step 4: correction debouncing of aerial images
Rotating the road aerial image clockwise by an angle thetakThe method and the device can be used for adjusting the road in the aerial image towards different directions to be in the horizontal direction, so that the road aerial image is subjected to shake removal.
2. The method for de-jittering an aerial video of an unmanned aerial vehicle according to claim 1, wherein the step 3 specifically comprises:
let the upper left corner of the edge contour map be the origin O (0,0) of the pixel coordinates of the image, take O (0,0) as the starting point, the right side as the column coordinate axis of the image, and the down side as the row coordinate axis of the image, and let an arbitrary straight line i, whose two endpoints are P1 and P2, respectively, where the pixel coordinates of P1 and P2 are (c) respectivelyi_1,ri_1) And (c)i_2,ri_2)，Is the included angle between the straight line i and the horizontal direction,indicating the angle rotated by rotating a horizontal straight line counterclockwise to be parallel to the straight line i, the angle of any straight line iThe calculation method of (2) is shown in the following formula (1):
wherein,is an integer, is rounded off when calculated, and
based on the calculated straight line angle, calculating a relative angle histogram, specifically:
(1) setting the number of the straight lines obtained in the step 2 as n;
(2) set 180 cell intervals theta1～θ180Wherein: the interval is as follows: theta1＝[0°,1°),θ2＝[1°,2°),...,θi＝[(i-1)°,i°),...,θ180＝[179°,180°)；
(3) For n straight lines, counting the angle of the straight line, and if the angle is thetaiM straight lines of (a) are included, the angle theta is considerediThe number of occurrences is m times, and h (theta) is usedi) Representing the angle theta of a straight lineiThe number of occurrences, i.e., the frequency;
(4) the frequency h (theta) of the linear angle obtained in the step (3)i) Performing normalization processing to calculate each linear angle thetaiRelative frequency of occurrence H (θ)i) The calculation method is H (theta)i)＝h(θi)/n；
(5) Drawing a relative straight line angle histogram, wherein the horizontal axis of the histogram represents a straight line direction angle and has a definition domain of [0 degrees and 180 degrees ], and the vertical axis represents the relative frequency of the occurrence of the straight line angle and has a value domain of [0,1 ]; the relative frequency value corresponding to the straight line with the maximum height value in the relative angle histogram is the peak value of the histogram;
(6) maximum peak H (θ) of histogramk) By the formula H (theta)k)＝Max{H(θ1),H(θ2),...,H(θi),...,H(θ180) Is calculated, wherein thetakThe angle of the straight line corresponding to the peak value of the histogram;
angle thetakI.e. the direction of the road in the frame image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510888639.2A CN105450950B (en) | 2015-12-07 | 2015-12-07 | Unmanned plane video jitter removing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510888639.2A CN105450950B (en) | 2015-12-07 | 2015-12-07 | Unmanned plane video jitter removing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105450950A true CN105450950A (en) | 2016-03-30 |
| CN105450950B CN105450950B (en) | 2018-07-27 |
Family
ID=55560680
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510888639.2A Expired - Fee Related CN105450950B (en) | 2015-12-07 | 2015-12-07 | Unmanned plane video jitter removing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105450950B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107705301A (en) * | 2017-09-29 | 2018-02-16 | 南京中设航空科技发展有限公司 | A kind of highway graticule damage testing method based on unmanned plane highway map picture |
| CN110320934A (en) * | 2018-03-27 | 2019-10-11 | 杭州海康机器人技术有限公司 | Cloud platform control method, device, holder and mobile device |
| CN111309048A (en) * | 2020-02-28 | 2020-06-19 | 重庆邮电大学 | Method for detecting autonomous flight along road by combining multi-rotor unmanned aerial vehicle with road |
| CN113838313A (en) * | 2021-11-29 | 2021-12-24 | 中国民用航空总局第二研究所 | Obstacle identification method for course beacon channel clearance jitter |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6819779B1 (en) * | 2000-11-22 | 2004-11-16 | Cognex Corporation | Lane detection system and apparatus |
| CN101608924A (en) * | 2009-05-20 | 2009-12-23 | 电子科技大学 | A kind of method for detecting lane lines based on gray scale estimation and cascade Hough transform |
| CN101639893A (en) * | 2008-07-31 | 2010-02-03 | 东软集团股份有限公司 | Method and device for identifying road direction |
| CN101714252A (en) * | 2009-11-26 | 2010-05-26 | 上海电机学院 | Method for extracting road in SAR image |
| CN103500322A (en) * | 2013-09-10 | 2014-01-08 | 北京航空航天大学 | Automatic lane line identification method based on low-altitude aerial images |
| CN103810462A (en) * | 2012-11-14 | 2014-05-21 | 中国科学院沈阳自动化研究所 | High voltage transmission line detection method based on linear targets |
| CN103971081A (en) * | 2013-01-25 | 2014-08-06 | 株式会社理光 | Multi-lane detection method and system |
-
2015
- 2015-12-07 CN CN201510888639.2A patent/CN105450950B/en not_active Expired - Fee Related
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6819779B1 (en) * | 2000-11-22 | 2004-11-16 | Cognex Corporation | Lane detection system and apparatus |
| CN101639893A (en) * | 2008-07-31 | 2010-02-03 | 东软集团股份有限公司 | Method and device for identifying road direction |
| CN101608924A (en) * | 2009-05-20 | 2009-12-23 | 电子科技大学 | A kind of method for detecting lane lines based on gray scale estimation and cascade Hough transform |
| CN101714252A (en) * | 2009-11-26 | 2010-05-26 | 上海电机学院 | Method for extracting road in SAR image |
| CN103810462A (en) * | 2012-11-14 | 2014-05-21 | 中国科学院沈阳自动化研究所 | High voltage transmission line detection method based on linear targets |
| CN103971081A (en) * | 2013-01-25 | 2014-08-06 | 株式会社理光 | Multi-lane detection method and system |
| CN103500322A (en) * | 2013-09-10 | 2014-01-08 | 北京航空航天大学 | Automatic lane line identification method based on low-altitude aerial images |
Non-Patent Citations (2)
| Title |
|---|
| 唐亮、谢维信、黄建军、黄敬雄: "一种新的道路描述子：对称边缘方向直方图", 《电子学报》 * |
| 张道兵、张慧、张正、刘波、王宏琦: "一种稳健的道路主方向提取算法", 《光子学报》 * |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107705301A (en) * | 2017-09-29 | 2018-02-16 | 南京中设航空科技发展有限公司 | A kind of highway graticule damage testing method based on unmanned plane highway map picture |
| CN107705301B (en) * | 2017-09-29 | 2021-04-13 | 南京中设航空科技发展有限公司 | Highway marking damage detection method based on unmanned aerial vehicle aerial highway image |
| CN110320934A (en) * | 2018-03-27 | 2019-10-11 | 杭州海康机器人技术有限公司 | Cloud platform control method, device, holder and mobile device |
| CN110320934B (en) * | 2018-03-27 | 2022-11-22 | 杭州海康机器人技术有限公司 | Cloud deck control method and device, cloud deck and mobile device |
| CN111309048A (en) * | 2020-02-28 | 2020-06-19 | 重庆邮电大学 | Method for detecting autonomous flight along road by combining multi-rotor unmanned aerial vehicle with road |
| CN111309048B (en) * | 2020-02-28 | 2023-05-26 | 重庆邮电大学 | Method for detecting autonomous flight along road by combining multi-rotor unmanned aerial vehicle with road |
| CN113838313A (en) * | 2021-11-29 | 2021-12-24 | 中国民用航空总局第二研究所 | Obstacle identification method for course beacon channel clearance jitter |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105450950B (en) | 2018-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109785291B (en) | Lane line self-adaptive detection method | |
| CN105488501B (en) | The method of license plate sloped correction based on rotation projection | |
| WO2018177026A1 (en) | Device and method for determining road edge | |
| CN105450950B (en) | Unmanned plane video jitter removing method | |
| Noda et al. | Vehicle ego-localization by matching in-vehicle camera images to an aerial image | |
| US11205284B2 (en) | Vehicle-mounted camera pose estimation method, apparatus, and system, and electronic device | |
| CN103714538B (en) | Road edge detection method and device and vehicle | |
| CN105205785B (en) | A kind of orientable oversize vehicle operation management system and its operation method | |
| CN106845552B (en) | The matched low dynamic carrier speed calculation method of fusion light stream and SIFT feature under the uneven environment of light distribution | |
| US11430199B2 (en) | Feature recognition assisted super-resolution method | |
| CN109493385A (en) | Autonomic positioning method in a kind of mobile robot room of combination scene point line feature | |
| Zhang et al. | Robust inverse perspective mapping based on vanishing point | |
| CN114419165B (en) | Camera external parameter correction method, camera external parameter correction device, electronic equipment and storage medium | |
| CN112014856B (en) | Road edge extraction method and device suitable for cross road section | |
| CN113988112B (en) | Method, device and equipment for detecting lane line and storage medium | |
| CN111368797A (en) | Target real-time ranging method based on road end monocular camera | |
| US11417080B2 (en) | Object detection apparatus, object detection method, and computer-readable recording medium | |
| CN105606123A (en) | Method for automatic correction of digital ground elevation model for low-altitude aerial photogrammetry | |
| CN110718068A (en) | Road monitoring camera installation angle estimation method | |
| CN103514587A (en) | Ship-based image-stabilizing method based on sea-sky boundary detecting | |
| CN111754550B (en) | Method and device for detecting dynamic obstacle in movement state of agricultural machine | |
| CN108615242B (en) | High-speed guardrail tracking method | |
| Cho et al. | Improved lane detection system using Hough transform with super-resolution reconstruction algorithm and multi-ROI | |
| WO2019007508A1 (en) | Advanced driver assistance system and method | |
| CN112115778A (en) | Intelligent identification method for lane line under loop simulation condition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180727 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |