CN115218801A - Machine vision-based wind driven generator clearance distance measuring method and device - Google Patents
Machine vision-based wind driven generator clearance distance measuring method and device Download PDFInfo
- Publication number
- CN115218801A CN115218801A CN202210830550.0A CN202210830550A CN115218801A CN 115218801 A CN115218801 A CN 115218801A CN 202210830550 A CN202210830550 A CN 202210830550A CN 115218801 A CN115218801 A CN 115218801A
- Authority
- CN
- China
- Prior art keywords
- picture
- calibration
- clearance
- video stream
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000004364 calculation method Methods 0.000 claims abstract description 23
- 238000005259 measurement Methods 0.000 claims description 27
- 238000004422 calculation algorithm Methods 0.000 claims description 13
- 238000013144 data compression Methods 0.000 claims description 10
- 238000000691 measurement method Methods 0.000 claims description 6
- 238000000605 extraction Methods 0.000 claims description 3
- 238000010408 sweeping Methods 0.000 abstract description 7
- 238000009434 installation Methods 0.000 description 11
- 238000007906 compression Methods 0.000 description 6
- 230000006835 compression Effects 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000012544 monitoring process Methods 0.000 description 4
- 229910052799 carbon Inorganic materials 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 230000001502 supplementing effect Effects 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 235000017166 Bambusa arundinacea Nutrition 0.000 description 1
- 235000017491 Bambusa tulda Nutrition 0.000 description 1
- 241001330002 Bambuseae Species 0.000 description 1
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 235000015334 Phyllostachys viridis Nutrition 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 239000011425 bamboo Substances 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011217 control strategy Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000407 epitaxy Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000009347 mechanical transmission Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/14—Measuring arrangements characterised by the use of optical techniques for measuring distance or clearance between spaced objects or spaced apertures
-
- F—MECHANICAL ENGINEERING; LIGHTING; HEATING; WEAPONS; BLASTING
- F03—MACHINES OR ENGINES FOR LIQUIDS; WIND, SPRING, OR WEIGHT MOTORS; PRODUCING MECHANICAL POWER OR A REACTIVE PROPULSIVE THRUST, NOT OTHERWISE PROVIDED FOR
- F03D—WIND MOTORS
- F03D17/00—Monitoring or testing of wind motors, e.g. diagnostics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/10—Image enhancement or restoration using non-spatial domain filtering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02E—REDUCTION OF GREENHOUSE GAS [GHG] EMISSIONS, RELATED TO ENERGY GENERATION, TRANSMISSION OR DISTRIBUTION
- Y02E10/00—Energy generation through renewable energy sources
- Y02E10/70—Wind energy
- Y02E10/72—Wind turbines with rotation axis in wind direction
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Chemical & Material Sciences (AREA)
- Sustainable Energy (AREA)
- Combustion & Propulsion (AREA)
- Mechanical Engineering (AREA)
- General Engineering & Computer Science (AREA)
- Sustainable Development (AREA)
- Life Sciences & Earth Sciences (AREA)
- Quality & Reliability (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
The invention discloses a method and a device for measuring the clearance distance of a wind driven generator based on machine vision, wherein the method comprises the following measuring steps: analyzing a video stream shot by a camera, and extracting a plurality of pictures in the video stream for calibration; and extracting video stream information in real time, tracking the track of the blade in the video stream, and calculating the clearance distance of the blade tip. The method can monitor the clearance of the wind driven generator in real time, is convenient for adjusting the propeller angle of the fan in time, prevents the accident of the blade sweeping tower, and ensures the safe and stable operation of the wind driven generator set. The method adopts the video shot by the camera in real time, firstly calculates the pixel clearance of the blade in the picture, and then calculates the actual clearance, and has accurate calculation and low cost. According to the real-time change of the clearance, the measures of reducing the clearance such as pitch variation and yaw are adopted, the number of times of protection shutdown is reduced, the availability of the fan is improved, and the loss of the generated energy of the wind driven generator is avoided.
Description
Technical Field
The invention relates to a method and a device for measuring the clearance distance of a wind driven generator based on machine vision, and belongs to the technical field of distance measurement.
Background
At present, a green, environment-friendly and low-carbon life style is advocated, the carbon emission reduction pace is accelerated, and the global competitiveness of industry and economy is favorably improved. The large-scale fan is the core path for reducing cost in the future, and offshore wind power has a good development market.
With the development of the technology, the power of the wind generating set is larger and larger, and in order to capture more wind power, the blades are longer and longer, and the blades are softer and softer. The blade clearance refers to the distance from the blade tip to the tower barrel when the blade sweeps across the tower barrel when the hub of the wind driven generator rotates.
In order to avoid the occurrence of the blade sweeping, CN201910148507 and CN201910398383 respectively provide clearance monitoring methods, the two methods both limit the safe clearance of the fan through a laser range finder, the two methods are only safety triggering protection methods, when the clearance of the blade is reduced, the safety clearance calibrated by the laser range finder is triggered, corresponding shutdown protection is performed, and then the machine is restarted. The two methods lack real-time monitoring of the clearance distance of the blade, so that effective fan control cannot be carried out, the loss of the generated energy of the wind driven generator can be caused, and meanwhile, only one point can be monitored by adopting a laser range finder, so that the monitoring range is small, and the protection of the wind driven generator is poor.
The patent application with application number CN202110217287 uses a camera and laser detector combination and detects the clearance of the blade by calibrating the reflective strips on the tower. The camera is used as clearance measuring equipment, and the laser detector and the reflecting strips on the tower barrel are used as installation and calibration equipment. And calibrating by repeating measurement for multiple times, and performing offline data analysis. And solving the safety interval of the clearance of the blades, and setting a multi-gear protection threshold value for protection. The method is also only a safety triggering protection device, the clearance distance of the blades is not monitored in real time, and the loss of the generated energy of the wind driven generator is also caused.
The patent application No. CN202110217287 is used to detect the blade running track and clearance by installing multiple radars around the tower. Because the radar is installed at the tower section of thick bamboo outer wall, in order to can satisfy 360 not measuring blade positions and clearance in four corners, the mutually supporting of many radars detects, and is numerous, and the cost is higher. And the installation position of radar is usually more than 20m above ground, and installation in earlier stage, middle and later maintenance are all difficult, and the transmission of signal is also unstable.
The patent application with application number CN202110217287 detects the deformation angle of the blade in real time through a mechanical clearance detection device, and fits the blade attitude through a geometric algorithm to obtain simulated clearance value information. The detection method adopts a contact type measuring device, and a responsive mechanical transmission device is required to be arranged in 3 blades of the wind driven generator to detect the deformation quantity of the blades. The equipment structure adopted by the method is complex, the installation is difficult due to the large quantity of equipment required to be installed, meanwhile, the measurement method does not directly measure the clearance distance, and the accuracy of the control information obtained by fitting simulation needs to be improved.
In summary, the existing clearance monitoring method has the defects of difficult installation, high cost, incapability of accurately calculating the clearance and the like.
Disclosure of Invention
The invention provides a machine vision-based wind driven generator clearance distance measuring method, which aims to monitor the clearance distance of a wind driven generator in real time, prevent tower sweeping accidents and guarantee the safe and stable operation of a wind turbine generator.
The invention further provides a device for measuring the clearance of the wind driven generator based on machine vision.
The invention is realized by the following technical scheme:
a wind driven generator clearance distance measuring method based on machine vision adopts a camera to shoot the distance between a blade tip and a tower drum in real time, adopts a control unit to store shot video stream and calculate the blade tip clearance distance, and comprises the following measuring steps:
s1, analyzing a video stream shot by a camera, extracting a plurality of pictures in the video stream for calibration, and the method comprises the following steps:
s11, collecting a video stream shot by a camera, and extracting a plurality of pictures;
s12, identifying the leaves in the extracted picture by adopting a dynamic tracking algorithm, and integrating the leaves into a picture as a calibration picture;
s13, identifying the tower footing extension in the calibration picture, and obtaining the pixel coordinate of the center point of the tower footing as a first calibration point; identifying a horizontal projection point of the blade tip on the surface of the tower drum at the lowest point in the calibration picture, and taking the pixel coordinate of the point as a second calibration point; connecting all the blade tips in the calibration picture to form a blade tip track straight line, and taking a straight line which passes through the second calibration point and is parallel to the blade tip track straight line as a calibration line L1; acquiring an included angle B between a straight line of a blade tip track and the lower edge of the calibration picture; identifying the number of pixel points occupied by the tower footing diameter in the calibration picture, calculating the ratio of the actual diameter of the tower footing to the number of the pixel points occupied, and defining the ratio as a proportionality coefficient A1; calculating a proportionality coefficient A2 of the actual distance of the height of the second calibration point and the number of the pixel points according to the height H1 between the camera and the ground, the height H2 of the second calibration point and the ground and the proportionality coefficient A1;
s14, storing the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B as calibration information;
s2, extracting video stream information in real time, identifying blades in the video stream, and calculating the clearance distance of the blade tip, comprising the following steps:
s21, extracting the video stream and storing the video stream as a picture packet with a fixed period, identifying the leaves in the picture by adopting a dynamic tracking algorithm, and integrating the leaves into a picture as a measurement and calculation picture;
s22, calling the stored information of the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B;
s23, rotating the measured picture by an angle B by taking the first calibration point or the second calibration point as a circle center, and carrying out rotation coordinate change;
s24, performing fast Fourier transform on the measurement and calculation picture line by line after the rotation coordinate changes, calculating the amplitude of each line of pixels under different frequencies, and finding out the frequency corresponding to the maximum amplitude; and then finding out a pixel row where the amplitude change inflection point under the frequency is located, wherein the pixel distance between the pixels of the row and the calibration line L1 is the tip pixel clearance D, and calculating the actual tip clearance D according to a proportional coefficient A2.
Further, in s12, not less than 3 pictures are extracted, and the pictures are pictures of adjacent frames or similar frame moments, and each extracted picture includes a blade and a tower of the wind turbine generator.
Further, the pictures extracted at the close frame time meet the condition that the extraction time interval corresponding to the pictures does not exceed 1 second.
Further, s21 also includes judging and extracting wind speed information in the video stream, and when the wind speed is greater than 2 times of the starting wind speed of the fan, adopting a dynamic tracking algorithm to identify the blades in the picture; otherwise, the video stream is re-extracted.
Further, before the rotation coordinate change of the measurement and calculation picture, the data compression is carried out on the synthesized measurement and calculation picture, and N columns of pixels are expanded in the left and right directions by taking the first fixed landmark point as the center to be cut; and taking the M-th row of pixels right above the second calibration point as the upper edge of the measured picture, and expanding J rows of pixels downwards from the upper edge as the lower edge to cut.
Furthermore, the N columns of pixels are 30-40% of the total number of pixels in the length direction of the synthesized measured picture; the M rows of pixels are 5-10% of the total number of pixels in the width direction of the calculated picture; the J-line pixels take 60-80% of the total number of pixels in the width direction of the measured picture.
Further, data compression is performed on the measurement and calculation picture rotated by the angle B in s23, and clipping is performed by taking the calibration line L1 as the upper edge and the K lines of pixels expanded downward as the lower edge.
Further, the K rows of pixels are 50% -70% of the total number of pixels in the width direction of the calculated picture.
Further, in s24, the inflection point of the amplitude variation is set to be 5% to 15% of the maximum amplitude among the amplitudes of the pixels in each row.
A wind driven generator clearance distance measuring device based on machine vision comprises a camera and a control unit; the camera is installed at the position of the wind driven generator cabin, and vertically shoots the distance between the blade tip and the tower barrel downwards in real time; the control unit is used for storing shot video streams and calculating the blade tip clearance distance; the control unit comprises at least the following modules,
the calibration module is used for analyzing the video stream shot by the camera and extracting a plurality of pictures in the video stream for calibration;
and the clearance distance measuring module is used for extracting the video stream information, tracking the track of the blade in the video stream and calculating the clearance distance of the blade tip.
Compared with the prior art, the invention has the following beneficial effects:
1. the invention provides a machine vision-based wind driven generator clearance measuring method, which can monitor the clearance of a wind driven generator in real time, is convenient for timely adjusting the propeller angle of a fan, prevents the occurrence of a blade tower sweeping accident and ensures the safe and stable operation of a wind turbine generator.
2. In the method, firstly, a calibration point, a calibration line, a proportionality coefficient and a deviation angle of the camera installation are determined; and tracking the track of the blade in the video stream according to the calibration information, and calculating the clearance distance of the blade tip in real time.
3. The invention adopts the video shot by the camera in real time, firstly calculates the pixel clearance of the blade in the picture, and then calculates the actual clearance, and has accurate calculation and low cost. According to the real-time change of the clearance, the measures of reducing the clearance such as pitch variation and yaw are adopted, the number of times of protection shutdown is reduced, the availability of the fan is improved, and the loss of the generated energy of the wind driven generator is avoided.
4. When the method is used for identifying the blades in the video stream, the blade track picture is analyzed, the blade running frequency is extracted as a main characteristic value, so that the frequency difference between the blade running frequency and the picture background is distinguished, and the interference of background noise (ground, a tower drum, vegetation and the like) on the blade identification is eliminated. And scanning the picture line by adopting a fast Fourier analysis algorithm to obtain the frequency characteristics of the blade track. And finding out the edge of the moving track of the blade according to the amplitude change to obtain the clearance distance of the blade. The method is simple and reliable, and has low operation requirement on the computing unit.
Drawings
FIG. 1 is a side view of a measurement system;
FIG. 2 is a bottom view of the measurement system;
FIG. 3 is a flow chart of calibrating pictures in a video stream;
FIG. 4 is a flow chart of clearance calculation;
FIG. 5 (a) is a schematic diagram of a tower footing center point and a blade tip trajectory in a calibration picture;
FIG. 5 (b) is a schematic diagram of the deviation angle of the webcam installation in the calibration picture;
FIG. 6 (a) is a diagram illustrating pictures in a picture packet;
FIG. 6 (b) is a schematic view of a measurement and calculation picture;
FIG. 6 (c) is a schematic view of a measurement and calculation picture after the rotation coordinate is changed;
FIG. 6 (d) is a schematic diagram showing the amplitudes of the pixels in each row at different frequencies;
FIG. 6 (e) is a schematic diagram showing the amplitude of each row of pixels at the frequency corresponding to the maximum amplitude;
FIG. 7 is a graph of measured clearance distance versus wind speed;
in the figure, 1-web camera; 2-an industrial personal computer; 3-a wind driven generator main controller; 4-a first punctuation; 5-tower base epitaxy; d-tip pixel clearance; b, an included angle between the straight line of the blade tip track and the lower edge of the calibration picture is included; l1-a calibration line; l2-tip trajectory straight line.
Detailed Description
The first embodiment is as follows:
in the method for measuring the clearance distance of the wind driven generator based on machine vision, as shown in fig. 1 and fig. 2, the adopted equipment in the method comprises a network camera 1 and an Industrial Personal Computer (IPC); the network camera is installed at the bottom of the outer side of the cabin and is installed along the central axis of the cabin, and vertically and downwards shoots the distance between the blade tip and the tower barrel in real time; the industrial personal computer 2 is respectively connected with the network camera 1 and the wind driven generator main controller 3 and is used for storing shot video streams and calculating the blade tip clearance distance based on the stored video streams; in order to enable the network camera to shoot clear video stream under the condition of dark light, the light supplementing lamps are further arranged around the network camera, and the light irradiation direction of the light supplementing lamps is parallel to the direction of the visual angle of the network camera; the measuring method comprises the following steps:
s1, analyzing a video stream shot by a network camera, extracting a plurality of pictures in the video stream for calibration, as shown in fig. 3, comprising the following steps:
s11, collecting a video stream shot by a network camera, and extracting a plurality of pictures;
s12, identifying the leaves in the extracted picture by adopting a dynamic tracking algorithm, and integrating the leaves into a picture as a calibration picture; the number of the pictures extracted in the step is not less than 3, the pictures belong to adjacent frames or pictures extracted at close frame time, and the extracted pictures comprise the blades and the tower barrel of the wind driven generator. The pictures extracted at the close frame time meet the condition that the extraction time interval corresponding to the pictures does not exceed 1 second. Wherein, for a video stream with a refresh rate of 10FPS, pictures of adjacent frames are adopted; for a video stream with a refresh rate of 30FPS, pictures of similar frames are adopted;
and s13, identifying the tower footing extension in the calibration picture, and obtaining the pixel coordinate of the center point of the tower footing as a first calibration point, wherein 5 in the graph (a) in fig. 5 represents the tower footing extension, and 4 represents the first calibration point.
Identifying a horizontal projection point of the blade tip on the surface of the tower drum at the lowest point in the calibration picture, and taking the pixel coordinate of the horizontal projection point as a second calibration point;
connecting the blade tips in the calibration picture to form a blade tip track straight line, taking a straight line which passes through the second calibration point and is parallel to the blade tip track straight line as a calibration line L1, wherein L1 in the figure 5 (a) represents the calibration line, and L2 represents the blade tip track straight line;
acquiring an included angle B, namely a deviation angle of the installation of the network camera, wherein the included angle B in the graph (B) in FIG. 5 represents an included angle between a blade tip track straight line and the lower edge of the calibration picture, and d represents a blade tip pixel clearance distance;
identifying the number of pixel points occupied by the tower footing diameter in the calibration picture, calculating the ratio of the actual diameter of the tower footing to the number of pixel points occupied, and defining the ratio as a proportional coefficient A1 of the actual distance of the height of the tower footing to the number of pixel points, wherein the unit is as follows: meters per pixel;
calculating a proportionality coefficient A2 between the actual distance of the height of the second calibration point and the number of the pixel points according to the vertical height H1 between the network camera and the ground, the vertical height H2 between the second calibration point and the ground and the proportionality coefficient A1, wherein the unit is as follows: the calculation formula of the scale factor A2 is as follows
s14, storing the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B in an industrial personal computer as calibration information;
s2, extracting video stream information in real time, identifying blades in the video stream, and calculating a clearance distance of a blade tip, as shown in fig. 4, including the following steps:
s21, the industrial personal computer is respectively communicated with the network camera and the wind driven generator main controller, the industrial personal computer receives the video stream transmitted by the network camera and the wind speed information transmitted by the wind driven generator main controller in real time, extracts the video stream information in real time and judges the wind speed corresponding to the extracted video stream, when the wind speed is more than 2 times of the wind speed of the fan, the received video stream is stored in the industrial personal computer in a picture packet mode with a fixed period, a dynamic tracking algorithm is adopted to identify the blades of each picture in the picture packet, and the picture is combined into a picture to be used as a measured and calculated picture; otherwise, the video stream is re-extracted; the extracted video stream of this step has a plurality of leaves in the picture packet that satisfy the storage.
s22, calling the stored information of the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B;
s23, rotating the image to measure and calculate the coordinate change, rotating the included angle B by taking the first calibration point or the second calibration point as a circle center, and removing the deviation angle of the installation of the network camera; and the rotating direction of the included angle B is equivalent to the rotating direction of the tip trajectory in the step s13, which is the direction of the linear coincidence with the lower edge of the calibration picture.
s24, performing fast Fourier transform on each row of pixels in the measurement and calculation picture after the rotation coordinate changes line by line, calculating to obtain the amplitude of each row of pixels under different frequencies, finding out the frequency corresponding to the maximum amplitude, then finding out the pixel row where the amplitude change inflection point under the frequency is located, wherein the pixel distance between the row of pixels and the calibration line L1 is the tip pixel clearance D, and then calculating the actual clearance D of the wind driven generator according to the proportional coefficient A2; d = D × A2; the inflection point of the variation of the amplitude is set to be 5% -15% of the maximum amplitude among the amplitudes of the pixels of each row.
s25, transmitting the actual clearance distance D to the wind turbine master controller, and returning to s21.
Example two:
the optional step in this example is to perform data compression on the synthesized reckoning picture before performing the rotation coordinate change on the reckoning picture, i.e. after s 22. During compression, the left and right directions take the first fixed mark point as the center, and N columns of pixels are respectively expanded in the left and right directions for clipping; and in the up-down direction, the M-th line of pixels right above the second calibration point is used as the edge of the measured picture, and J lines of pixels are expanded downwards from the upper edge to be used as the lower edge for clipping. Wherein, the N rows of pixels take 30 to 40 percent of the total number of pixels in the length direction of the measured picture; the M rows of pixels are 5-10% of the total number of pixels in the width direction of the measured picture; the total number of pixels in the J lines in the width direction of the image is measured and calculated to be 60-80%. The data compression can improve the calculation speed and reduce the calculation amount and the storage space of the industrial personal computer.
Example three:
in this example, an optional step is to perform data compression on the measurement and calculation picture rotated by the angle B in s23, and perform cropping by taking the calibration line L1 as the upper edge and expanding K rows of pixels downward as the lower edge. And K, taking 50-70% of the total number of pixels in the width direction of the measured picture. The data compression can improve the calculation speed and reduce the calculation amount and the storage space of the industrial personal computer.
Example four:
the embodiment provides a machine vision-based wind driven generator clearance distance measuring device which comprises a camera and a control unit, wherein the camera is installed at the position of a cabin of a wind driven generator, and vertically shoots the distance between a blade tip and a tower in real time. The control unit is used for storing the shot video stream and calculating the blade tip clearance. The control unit is embedded with at least the following software modules,
the calibration module is used for analyzing the video stream shot by the camera and extracting a plurality of pictures in the video stream for calibration; calibration may be performed off-line. The calibration module is used for determining information of a first calibration point, a second calibration point, a calibration line L1, a proportionality coefficient A2 and an included angle B in the method.
And the clearance distance measuring module is used for extracting the video stream information, tracking the track of the blade in the video stream and calculating the clearance distance of the blade tip. The module is used for completing the processes of identifying the picture by adopting a dynamic tracking algorithm, measuring and calculating the coordinate rotation change of the picture, performing fast Fourier transform and determining the blade tip pixel clearance D and the actual clearance D in the method. Compression of the reckoning pictures may also be included.
The first application embodiment:
in the embodiment, the clearance distance of a certain wind driven generator is measured, the diameter of a tower footing of a fan is 7.4 meters, and the vertical height H1 between a network camera and the ground is 76 meters; the vertical height H2 of the blade tip between the horizontal projection point of the surface of the tower barrel at the lowest point and the tower base is 21 meters. The process for measuring the clearance distance of the wind driven generator is shown in fig. 3 and 4, and the specific steps are as follows:
s1, analyzing a video stream shot by a network camera, and calibrating 3 pictures in the video stream, as shown in fig. 3, the method specifically comprises the following steps:
s11, collecting a video stream shot by a network camera, and extracting 3 pictures; the refresh rate of the video stream taken in this example is 30FPS, and the extracted 3 pictures belong to close frame pictures.
s12, identifying the leaves in the extracted picture by adopting a dynamic tracking algorithm, and synthesizing a picture as a calibration picture;
s13, identifying the tower footing extension in the calibration picture to obtain a tower footing center point pixel coordinate as a first calibration point;
identifying a horizontal projection point of the blade tip, which is positioned at the lowest point on the surface of the tower drum, in the calibration picture, and taking the pixel coordinate of the horizontal projection point as a second calibration point;
connecting all the blade tips in the calibration picture to form a blade tip track straight line, and taking a straight line which passes through the second calibration point and is parallel to the blade tip track straight line as a calibration line L1;
acquiring an included angle B between a blade tip track straight line and the lower edge of a calibration picture, namely a deviation angle of installation of the network camera; angle B =7.6 ° in this example.
Identifying that the number of pixel points occupied by the diameter of the tower footing in the calibration picture is 55, calculating the ratio of the actual diameter of the tower footing to the number of the pixel points occupied, and taking the ratio as the proportional coefficient A1=0.1346 m/pixel of the actual distance of the height of the tower footing and the number of the pixel points;
identifying a horizontal projection point on the surface of the tower drum when the blade tip is positioned at the lowest point in the calibration picture, and taking the pixel coordinate of the point as a second calibration point; calculating a proportionality coefficient A2=0.0974 m/pixel between the actual distance of the height of the second calibration point and the number of the pixel points according to the vertical height H1 between the network camera and the ground, the vertical height H2 between the second calibration point and the ground and the proportionality coefficient A1;
s14, storing the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B as calibration information;
s2, extracting video stream information in real time, identifying blades in the video stream, and calculating a clearance distance of a blade tip, as shown in fig. 4, including the following steps:
and s21, the industrial personal computer is respectively communicated with the network camera and the wind driven generator main controller, the industrial personal computer receives the video stream transmitted by the network camera and the corresponding wind speed information transmitted by the wind driven generator main controller in real time, extracts the video stream information in real time and judges the wind speed corresponding to the extracted video stream, when the wind speed is more than 6m/s, the video stream is stored in the industrial personal computer in a picture packet mode with a fixed period of 1s, and as shown in fig. 6 (a), picture pixels in the stored picture packet are 1024 x 576. Identifying the leaves of each picture in the picture packet by adopting a dynamic tracking algorithm, and synthesizing a picture as a measuring and calculating picture; when the wind speed is less than or equal to 6m/s, re-extracting the video stream;
s22, calling the stored information of the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B;
and s23, performing data compression on the synthesized measuring and calculating picture. During compression, taking a first fixed mark point in the measurement picture as a center, and clipping 307 rows of pixels expanded in the left direction and the right direction; and then, taking the 58 th line of pixels right above the second calibration point as the upper edge of the measurement picture, expanding the 370 th line of pixels downwards, and cutting the measurement picture, as shown in fig. 6 (b), so as to obtain 614 × 370 pixels of the measurement picture after the first compression.
Rotating coordinate changes are carried out on the measured and calculated picture, the measured and calculated picture rotates anticlockwise by 7.6 degrees by taking the first fixed standard point as a circle center, and a deviation angle of installation of the network camera is removed;
and then, carrying out data compression on the measurement and calculation picture after the rotation coordinate changes: taking the calibration line L1 as the upper edge and the 205 rows of pixels expanded downward as the lower edge, clipping the measurement picture after the rotation coordinate change, as shown in fig. 6 (c), to obtain 614 × 205 pixels of the measurement picture after the second compression.
s24, performing fast fourier transform on each line of pixels of the calculated picture after the second compression row by row, and calculating to obtain the amplitude of each line of pixels at different frequencies, as shown in fig. 6 (d), the higher the brightness, the larger the amplitude, and finding the frequency corresponding to the maximum amplitude, where the frequency corresponding to the maximum amplitude in fig. 6 (d) is about 17HZ. As shown in fig. 6 (e), a pixel row where the amplitude change inflection point under the frequency is located is found, the pixel row corresponds to a blade tip track connection line in the measurement and calculation picture, a pixel distance between the pixel row and the stored calibration line L1 is a blade tip pixel clearance D, and then the actual clearance D of the wind driven generator is calculated according to a proportionality coefficient A2; the inflection point of the amplitude variation is 5% -10% of the maximum amplitude in the amplitudes of the pixels in each row, in this embodiment, the maximum amplitude is 9.2, and the inflection point of the amplitude variation is 1.
s25, transmitting the actual clearance distance D to the wind driven generator main controller, and returning to s21; the wind turbine clearance is measured again.
The wind driven generator main controller can draw a relation graph of clearance and wind speed according to data received in real time. Fig. 7 is a plot of clearance versus wind speed taken over 80 seconds. The clearance at each moment can be obtained from fig. 7, the clearance varies with the wind speed, and the higher the wind speed, the shorter the clearance, and the more likely the "tower-sweeping" accident occurs. Technicians can set means for adjusting the clearance distance such as variable pitch yawing according to real-time change of the clearance distance, so that the occurrence of tower sweeping accidents is avoided, the availability of the wind driven generator is improved, and the loss of the generated energy of the wind driven generator is avoided.
Application example two:
in the embodiment, a clearance test is carried out on a certain wind power plant in Henan on a certain wind power plant. The field adopts a 2.2MW machine type, the diameter of an impeller is 110 meters, and the height of a tower barrel is 90 meters. Different from a northern plain type wind field, the local area of the wind field belongs to a hilly wind field, the terrain is complex, the ground wind turbulence is large, the wind direction changes rapidly, and stronger negative shear is accompanied. The load stress of the local fan unit is increased, and the situation that the blades are occasionally swept by the tower is caused. Install the testing arrangement of this application additional and measure, can calculate the blade clearance that goes out the fan in real time and provide fan main control unit, fan main control unit carries out corresponding change paddle and falling load control strategy according to the clearance value, can effectually increase blade clearance, avoids sweeping the tower condition and takes place.
By adopting the clearance distance measuring device and the clearance distance measuring method, the actually controlled clearance distance is increased by about 2 meters, and compared with the original condition that the clearance distance is not measured in real time, the safety of the running of the fan is improved.
Claims (10)
1. A method for measuring the clearance distance of a wind driven generator based on machine vision is characterized in that: the method adopts a camera to shoot the distance between the blade tip and the tower drum in real time, adopts a control unit to store shot video stream and calculates the clearance distance of the blade tip, and comprises the following measurement steps:
s1, analyzing a video stream shot by a camera, extracting a plurality of pictures in the video stream for calibration, and the method comprises the following steps:
s11, collecting a video stream shot by a camera, and extracting a plurality of pictures;
s12, identifying the leaves in the extracted picture by adopting a dynamic tracking algorithm, and integrating the leaves into a picture as a calibration picture;
s13, identifying the tower footing extension in the calibration picture, and obtaining the pixel coordinate of the center point of the tower footing as a first calibration point; identifying a horizontal projection point of the blade tip, which is positioned at the lowest point on the surface of the tower drum, in the calibration picture, and taking the pixel coordinate of the point as a second calibration point; connecting the blade tips in the calibration picture to form a blade tip track straight line, and taking a straight line which passes through the second calibration point and is parallel to the blade tip track straight line as a calibration line L1; acquiring an included angle B between a straight line of a blade tip track and the lower edge of the calibration picture; identifying the number of pixel points occupied by the tower footing diameter in the calibration picture, calculating the ratio of the actual diameter of the tower footing to the number of the pixel points occupied, and defining the ratio as a proportional coefficient A1; calculating a proportionality coefficient A2 of the actual distance of the height of the second calibration point and the number of the pixel points according to the height H1 between the camera and the ground, the height H2 of the second calibration point and the ground and the proportionality coefficient A1;
s14, storing the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B as calibration information;
s2, extracting video stream information in real time, identifying blades in the video stream, and calculating the clearance distance of the blade tip, wherein the method comprises the following steps:
s21, extracting video stream and storing the video stream as a picture packet with a fixed period, identifying the leaves in the picture by adopting a dynamic tracking algorithm, and integrating the leaves into a picture as a measurement picture;
s22, calling the stored information of the first calibration point, the second calibration point, the calibration line L1, the proportionality coefficient A2 and the included angle B;
s23, rotating the measured picture by an angle B by taking the first calibration point or the second calibration point as a circle center, and carrying out rotation coordinate change;
s24, performing fast Fourier transform on the measurement and calculation picture line by line after the rotation coordinate changes, calculating the amplitude of each line of pixels under different frequencies, and finding out the frequency corresponding to the maximum amplitude; and then finding out a pixel row where the amplitude change inflection point under the frequency is located, wherein the pixel distance between the pixels of the row and the calibration line L1 is the tip pixel clearance D, and calculating the actual tip clearance D according to a proportional coefficient A2.
2. The machine-vision-based wind turbine clearance measurement method of claim 1, wherein: in s12, not less than 3 extracted pictures are taken, and the extracted pictures are pictures of adjacent frames or similar frames, and each extracted picture comprises a blade and a tower of the wind driven generator.
3. The machine-vision-based wind turbine clearance measurement method of claim 2, wherein: the pictures extracted at the similar frame time meet the condition that the extraction time interval corresponding to the pictures does not exceed 1 second.
4. The machine-vision-based wind turbine clearance measuring method of claim 1, wherein: s21, judging and extracting wind speed information in the video stream, and identifying blades in the picture by adopting a dynamic tracking algorithm when the wind speed is more than 2 times of the starting wind speed of the fan; otherwise, the video stream is re-extracted.
5. The machine-vision-based wind turbine clearance measurement method of claim 1, wherein: before the rotation coordinate change of the measurement and calculation picture is carried out, data compression is carried out on the synthesized measurement and calculation picture, and N columns of pixels are expanded in the left direction and the right direction respectively by taking a first fixed coordinate point as a center for cutting; and taking the M-th line of pixels right above the second calibration point as the upper edge of the measured picture, and expanding the J-line of pixels downwards from the upper edge as the lower edge to cut.
6. The machine-vision-based wind turbine clearance measuring method of claim 5, wherein: the N rows of pixels are combined to measure 30% -40% of the total number of pixels in the length direction of the picture; the M rows of pixels are 5-10% of the total number of pixels in the width direction of the calculated picture; and the J-line pixels are used for measuring and calculating 60-80% of the total number of pixels in the width direction of the picture.
7. The machine-vision-based wind turbine clearance measurement method of claim 5, wherein:
and (5) performing data compression on the measurement and calculation picture rotated by the angle B in the s23, taking the calibration line L1 as an upper edge, and expanding K rows of pixels downwards as a lower edge, and performing clipping.
8. The machine-vision-based wind turbine clearance measuring method of claim 7, wherein: and the number of the K rows of pixels is 50-70% of the total number of pixels in the width direction of the measured and calculated picture.
9. The machine-vision-based wind turbine clearance measurement method of claim 7, wherein: in s24, the inflection point of the amplitude variation is set to be 5% to 15% of the maximum amplitude among the amplitudes of the pixels in each row.
10. The utility model provides a aerogenerator headroom distance measuring device based on machine vision which characterized in that: comprises a camera and a control unit; the camera is installed at the position of the wind driven generator cabin and vertically shoots the distance between the blade tip and the tower barrel downwards in real time; the control unit is used for storing shot video streams and calculating the blade tip clearance distance; the control unit comprises at least the following modules,
the calibration module is used for analyzing the video stream shot by the camera and extracting a plurality of pictures in the video stream for calibration;
and the clearance distance measuring module is used for extracting the video stream information, tracking the track of the blade in the video stream and calculating the clearance distance of the blade tip.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210830550.0A CN115218801B (en) | 2022-07-15 | 2022-07-15 | Wind driven generator clearance distance measuring method and device based on machine vision |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210830550.0A CN115218801B (en) | 2022-07-15 | 2022-07-15 | Wind driven generator clearance distance measuring method and device based on machine vision |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115218801A true CN115218801A (en) | 2022-10-21 |
| CN115218801B CN115218801B (en) | 2023-06-02 |
Family
ID=83611945
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210830550.0A Active CN115218801B (en) | 2022-07-15 | 2022-07-15 | Wind driven generator clearance distance measuring method and device based on machine vision |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115218801B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116008970A (en) * | 2023-03-27 | 2023-04-25 | 南京牧镭激光科技股份有限公司 | Method for verifying radar null value inversion accuracy based on video image |
| CN116027314A (en) * | 2023-02-21 | 2023-04-28 | 湖南联智监测科技有限公司 | Fan blade clearance distance monitoring method based on radar data |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012017989A (en) * | 2010-07-06 | 2012-01-26 | East Japan Railway Co | Movable platform fence measuring device and measuring method |
| CN103514308A (en) * | 2012-06-20 | 2014-01-15 | 华锐风电科技(集团)股份有限公司 | Method and device for designing wind driven generator blades |
| CN107448360A (en) * | 2016-06-01 | 2017-12-08 | 森维安有限公司 | Adapter apparatus and facility for horizontal pre-assembled wind turbine rotor |
| CN108687005A (en) * | 2018-04-03 | 2018-10-23 | 无锡太湖学院 | A kind of cleaning apparatus for self of reflector |
| CN109812390A (en) * | 2019-02-28 | 2019-05-28 | 明阳智慧能源集团股份公司 | A kind of blade headroom monitoring method of wind power generating set |
| CN110939549A (en) * | 2019-11-18 | 2020-03-31 | 陈伟春 | Blade clearance monitoring system and blade clearance monitoring method |
| CN111336073A (en) * | 2020-03-04 | 2020-06-26 | 南京航空航天大学 | Wind driven generator tower clearance visual monitoring device and method |
| CN111911364A (en) * | 2020-09-11 | 2020-11-10 | 上海电气风电集团股份有限公司 | Blade tip tower barrel clearance monitoring method |
| CN112267980A (en) * | 2020-10-26 | 2021-01-26 | 无锡风电设计研究院有限公司 | Blade clearance monitoring system and method of wind generating set |
| CN113309674A (en) * | 2021-03-31 | 2021-08-27 | 新疆金风科技股份有限公司 | Method and device for determining clearance distance of wind generating set |
| CN113390436A (en) * | 2020-03-13 | 2021-09-14 | 北京金风科创风电设备有限公司 | Verification system, method and medium for video ranging device of wind generating set |
| CN113757051A (en) * | 2021-09-26 | 2021-12-07 | 新疆金风科技股份有限公司 | Wind generating set tower clearance monitoring method, device and system |
| EP3979201A1 (en) * | 2020-10-02 | 2022-04-06 | Baker Hughes Oilfield Operations LLC | Automated turbine blade to shroud gap measurement |
-
2022
- 2022-07-15 CN CN202210830550.0A patent/CN115218801B/en active Active
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012017989A (en) * | 2010-07-06 | 2012-01-26 | East Japan Railway Co | Movable platform fence measuring device and measuring method |
| CN103514308A (en) * | 2012-06-20 | 2014-01-15 | 华锐风电科技(集团)股份有限公司 | Method and device for designing wind driven generator blades |
| CN107448360A (en) * | 2016-06-01 | 2017-12-08 | 森维安有限公司 | Adapter apparatus and facility for horizontal pre-assembled wind turbine rotor |
| CN108687005A (en) * | 2018-04-03 | 2018-10-23 | 无锡太湖学院 | A kind of cleaning apparatus for self of reflector |
| CN109812390A (en) * | 2019-02-28 | 2019-05-28 | 明阳智慧能源集团股份公司 | A kind of blade headroom monitoring method of wind power generating set |
| CN110939549A (en) * | 2019-11-18 | 2020-03-31 | 陈伟春 | Blade clearance monitoring system and blade clearance monitoring method |
| CN111336073A (en) * | 2020-03-04 | 2020-06-26 | 南京航空航天大学 | Wind driven generator tower clearance visual monitoring device and method |
| CN113390436A (en) * | 2020-03-13 | 2021-09-14 | 北京金风科创风电设备有限公司 | Verification system, method and medium for video ranging device of wind generating set |
| CN111911364A (en) * | 2020-09-11 | 2020-11-10 | 上海电气风电集团股份有限公司 | Blade tip tower barrel clearance monitoring method |
| EP3979201A1 (en) * | 2020-10-02 | 2022-04-06 | Baker Hughes Oilfield Operations LLC | Automated turbine blade to shroud gap measurement |
| CN112267980A (en) * | 2020-10-26 | 2021-01-26 | 无锡风电设计研究院有限公司 | Blade clearance monitoring system and method of wind generating set |
| CN113309674A (en) * | 2021-03-31 | 2021-08-27 | 新疆金风科技股份有限公司 | Method and device for determining clearance distance of wind generating set |
| CN113757051A (en) * | 2021-09-26 | 2021-12-07 | 新疆金风科技股份有限公司 | Wind generating set tower clearance monitoring method, device and system |
Non-Patent Citations (2)
| Title |
|---|
| 张乐: "铁路机车设备检修管理信息化系统的研究", 无线互联科技, no. 6, pages 79 - 82 * |
| 王文亮: "风电机组叶尖净空分析与控制", 水力发电, vol. 48, no. 4, pages 94 - 98 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116027314A (en) * | 2023-02-21 | 2023-04-28 | 湖南联智监测科技有限公司 | Fan blade clearance distance monitoring method based on radar data |
| CN116008970A (en) * | 2023-03-27 | 2023-04-25 | 南京牧镭激光科技股份有限公司 | Method for verifying radar null value inversion accuracy based on video image |
| CN116008970B (en) * | 2023-03-27 | 2024-01-05 | 南京牧镭激光科技股份有限公司 | Method for verifying radar null value inversion accuracy based on video image |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115218801B (en) | 2023-06-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112267980B (en) | Blade clearance monitoring system and method of wind generating set | |
| US20230213021A1 (en) | Method of condition monitoring one or more wind turbines and parts thereof and performing instant alarm when needed | |
| CN115218801B (en) | Wind driven generator clearance distance measuring method and device based on machine vision | |
| CN111911364B (en) | Blade tip tower barrel clearance monitoring method | |
| EP2369176A1 (en) | Wind turbine and method for measuring the pitch angle of a wind turbine rotor blade | |
| CN113759960A (en) | Unmanned aerial vehicle-based fan blade and tower barrel inspection identification system and method | |
| US11680556B2 (en) | Methods and systems of advanced yaw control of a wind turbine | |
| US6993965B2 (en) | Horizontal axis wind turbine and method for measuring upflow angle | |
| CN114623049A (en) | Wind turbine generator tower clearance monitoring method and computer program product | |
| Guo et al. | Detecting and positioning of wind turbine blade tips for uav-based automatic inspection | |
| CN113339205A (en) | Method and system for monitoring running track of blade of wind generating set | |
| CN116484652B (en) | Wake flow interference detection method in wind power plant based on blade root load | |
| US20200309093A1 (en) | Method of determining a power curve of a wind turbine | |
| CN114398842A (en) | Method for evaluating generated energy of operating wind power plant | |
| CN114439703A (en) | Wind turbine generator blade imbalance fault monitoring method based on vehicle-mounted device | |
| CN109917422B (en) | Wind power plant wind resource condition prediction method and system | |
| CN113565697A (en) | Impeller pneumatic unbalance optimization system and method based on laser and video measurement | |
| EP4009272A1 (en) | Unmanned airborne visual diagnosis of an operating wind turbine generator | |
| CN117830294B (en) | Method for acquiring images and detecting defects of offshore wind power pile blades | |
| CN116008970B (en) | Method for verifying radar null value inversion accuracy based on video image | |
| CN116308180B (en) | Wind power structure health monitoring system and method based on unmanned aerial vehicle and machine vision | |
| US20210254603A1 (en) | Method for ascertaining a wind direction at a wind power installation, system for ascertaining a wind direction, and a wind power installation | |
| Xiao et al. | Test research on accuracy of laser clearance and video clearance | |
| CN113417812A (en) | Wind turbine generator system cabin displacement monitoring system and monitoring method | |
| CN117780573A (en) | Multi-data-fusion real-time monitoring method for impeller unbalance of wind turbine generator |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |