CN117911363A - Rotary machine vibration displacement detection method based on vision - Google Patents
Rotary machine vibration displacement detection method based on vision Download PDFInfo
- Publication number
- CN117911363A CN117911363A CN202410068456.5A CN202410068456A CN117911363A CN 117911363 A CN117911363 A CN 117911363A CN 202410068456 A CN202410068456 A CN 202410068456A CN 117911363 A CN117911363 A CN 117911363A
- Authority
- CN
- China
- Prior art keywords
- vibration displacement
- input
- input area
- pixel points
- rotary machine
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000006073 displacement reaction Methods 0.000 title claims abstract description 55
- 238000001514 detection method Methods 0.000 title claims abstract description 11
- 238000000034 method Methods 0.000 claims abstract description 20
- 238000012545 processing Methods 0.000 claims abstract description 7
- 238000012544 monitoring process Methods 0.000 claims abstract description 5
- 238000003745 diagnosis Methods 0.000 claims abstract description 3
- 238000001914 filtration Methods 0.000 claims abstract description 3
- 238000010586 diagram Methods 0.000 claims description 8
- 238000009499 grossing Methods 0.000 claims description 3
- 238000005259 measurement Methods 0.000 abstract description 20
- 238000000691 measurement method Methods 0.000 abstract description 4
- 230000000007 visual effect Effects 0.000 abstract description 2
- 230000001186 cumulative effect Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000001228 spectrum Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 239000000523 sample Substances 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000282461 Canis lupus Species 0.000 description 1
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 1
- 239000010931 gold Substances 0.000 description 1
- 229910052737 gold Inorganic materials 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01M—TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
- G01M13/00—Testing of machine parts
- G01M13/02—Gearings; Transmission mechanisms
- G01M13/028—Acoustic or vibration analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V10/443—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components by matching or filtering
- G06V10/449—Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Computational Linguistics (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Biophysics (AREA)
- Biodiversity & Conservation Biology (AREA)
- Acoustics & Sound (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Measurement Of Mechanical Vibrations Or Ultrasonic Waves (AREA)
Abstract
The invention discloses a visual-based rotary mechanical vibration displacement detection method, which comprises the steps of carrying out gray processing on a mechanical vibration video acquired by a high-speed industrial camera, extracting data of a t frame and a first frame in a video frame as an input part, identifying an edge area of each input part by using a sobel operator, taking the identified area as an input area, carrying out spatial filtering on the input area, extracting vibration displacement of pixel points in the input area, calculating an average value to obtain the vibration displacement of each input part, carrying out smooth processing on the vibration displacement of three continuous input parts, generating a time domain graph and a spectrogram, and monitoring the running state of the rotary machine according to the time domain graph and the spectrogram to realize real-time diagnosis of the rotary machine; the method of the invention overcomes some of the drawbacks of conventional measurement methods, such as mass loading; under the condition of excluding the measurement by using the traditional sensor, the vibration state of the rotary machine can be accurately detected, and the calculation efficiency is high.
Description
Technical Field
The invention relates to a vision-based rotary mechanical vibration displacement detection method, and belongs to the technical field of mechanical equipment state monitoring and image processing.
Background
In the operation of rotary machines, vibrations not only affect the working conditions of the equipment, but also are directly related to the service life thereof, so accurate measurement and monitoring of vibrations is a key to ensuring stable operation of the mechanical system. Vibration measurement plays a key role in the field of rotary machinery, and provides necessary information for improving the operating efficiency and safety of equipment by identifying system parameters, monitoring the operating condition of the equipment and diagnosing potential faults.
In the field of vibration measurement, there are two measuring methods, contact and non-contact. Touch measurements use physically attached sensors, such as accelerometers and displacement sensors, which are arranged on the measurement object and connected to the upper system. At present, some non-contact measurement modes such as a laser vibration meter and an eddy current sensor can well solve the problems existing in contact measurement. Although these non-contact measurement methods avoid the mass loading effect, different non-contact methods have respective problems, such as that the methods can only measure the corresponding signal of a single point, time and effort are wasted and the structure is limited when all wires and meters are processed. In contrast, another non-contact measurement method, a digital camera focuses visible light on a photosensitive element through a lens, records the intensity and color of the light, digitizes the light into an image, and provides excellent flexibility and high efficiency in vibration measurement, thereby providing a convenient and accurate scheme for measurement.
At present, the traditional optical flow method based on intensity is sensitive to noise and interference, while the digital image correlation method provides a smooth pattern shape with high precision, but has strict requirements on calculation and image quality, and can not extract small sub-pixel movements. The optical flow method based on the phase is an emerging vision measurement method, and can realize the extraction of sub-pixel small movements. However, these methods still face challenges in terms of time complexity, noise sensitivity, and image quality. Therefore, in order to more effectively apply the visual method to detect the vibration displacement of the rotating machinery, the problems need to be overcome, and the adaptability, the automation degree and the precision of the measuring system are improved.
Disclosure of Invention
Aiming at the problems, the invention provides a vision-based rotary mechanical vibration displacement detection method, which introduces a non-contact measurement mode for rotary mechanical vibration measurement, optimizes time calculation cost and measurement precision, and provides a reliable solution for rotary mechanical vibration displacement detection.
The method for detecting the vibration displacement of the rotary machine based on vision comprises the following steps:
1. shooting the running rotary machine by using an industrial high-speed camera, and obtaining clear video images at a frame rate of 1000-6000 fps;
2. Carrying out gray scale processing on video frames in the video in the step (1), extracting data of a t frame I (x, y, t) and a first frame I (x, y, 1) in the video frames as an input part, and obtaining n input parts;
3. Identifying the edge area of each input part by using a sobel operator, taking the identified area as an input area, performing spatial filtering on the input area, extracting vibration displacement delta (x, y, t) of pixel points in the input area, and calculating the average value of the vibration displacement of the pixel points in the input area to obtain the vibration displacement of each input part; smoothing the vibration displacement of the three continuous input parts by smoothdata functions to obtain n vibration displacement data of n input parts;
the step of extracting the vibration displacement of the pixel points in the input area is as follows:
(1) Setting a center frequency coefficient set W and a Gaussian kernel function coefficient set S of the orthogonal Gabor filter;
(2) Respectively calculating sine and cosine parts F Cx(s)、FCy(s)、FSx(s) and F Sy(s) of the filter in the horizontal direction and the vertical direction, and respectively carrying out convolution calculation on the input area in the horizontal direction and the vertical direction to obtain a sine filter result A S (x, y, t) and a cosine filter result A C (x, y, t);
(3) Phase compensation is carried out on A S (x, y, t) and A C (x, y, t) through arc tangent operation, phase phi (x, y, t) is obtained, and phase difference delta phi (x, y, t) =phi (x, y, t) -phi (x, y, 1) of the pixel points is calculated;
(4) Calculating pixel point speeds V (x, y, t) through pixel point phase differences, and multiplying the pixel point speeds V (x, y, t) by a frame rate to obtain vibration displacement delta (x, y, t) of a plurality of pixel points in an input area;
Calculating the average value of the vibration displacement of the pixel points in the input area, namely firstly calculating the average value and standard deviation of the vibration displacement of all the pixel points in the input area, then eliminating the vibration displacement data of the pixel points which are not in the range of the average value and the standard deviation, and then calculating the average value of the vibration displacement of the residual pixel points;
4. Based on the n vibration displacement data obtained in the step 3, a time domain diagram and a spectrogram are generated, and the running state of the rotary machine is monitored according to the time domain diagram and the spectrogram, so that the real-time diagnosis of the rotary machine is realized.
The invention has the advantages and technical effects that:
the vision-based rotary mechanical vibration displacement detection method introduces a non-contact measurement mode for rotary mechanical vibration measurement, and effectively solves the problems of mass loading effect, single-point measurement limitation and complicated instrument installation in the contact method. Meanwhile, the invention obviously improves the time calculation efficiency and the measurement precision by selecting the detection area and processing the data, can accurately track the vibration displacement of the rotary machine, and provides an advanced and reliable solution for the vibration displacement detection of the rotary machine.
Drawings
FIG. 1 is a high-speed rotor vibration test bed, wherein a is a rotor, b is a high-speed industrial camera, c is an adjustable light source, d is a high-speed rotor vibration platform, e is an eddy current sensor, and f is a rotation speed regulator;
FIG. 2 is a time domain waveform diagram and a spectrogram of a vibration signal acquired by an eddy current sensor;
FIG. 3 is a gray scale image of a first frame of an image of a rotor acquired by a high speed industrial camera;
FIG. 4 is an edge recognition result of the Sobel operator on the first frame gray scale map;
FIG. 5 is a schematic diagram of the edge area results of rotor image recognition;
FIG. 6 is a time domain waveform and spectrum diagram of vibration displacement extracted by the method of the present invention;
FIG. 7 is a graph showing the comparison of the time domain waveform and the frequency spectrum of vibration signals collected by the method and the eddy current sensor.
Detailed Description
The present invention will be described in further detail by way of examples, but the scope of the present invention is not limited to the above description;
Example 1: the experimental device shown in fig. 1 is adopted to verify the reliability of the method for detecting the vibration of the rotary machine, a rotor a of a high-speed rotor vibration platform is used as an object, a high-speed industrial camera b (Qianjin wolf 5F 01) is used as a video image acquisition device, an eddy current sensor e is used as a comparison acquisition device, and vibration video and voltage displacement signals are synchronously acquired on a high-speed rotor vibration test bed d (Nanjing Dongda Z-03); in order to generate slight vibration of the rotor during rotation, a fastening screw is installed at the left side of the rotor to cause mass misalignment; the contrast ratio of an image screen can be effectively improved by properly adjusting the brightness of a light source c (gold cup EF-200 LED), and an image sequence acquired by a high-speed industrial camera b can be directly stored in built-in equipment at a set frame rate; before vibration displacement data acquisition, the vertical distance between the lens of the high-speed industrial camera b and the rotor is set to 1500mm, the probe direction of the eddy current sensor coincides with the horizontal center line of the rotor, and the distance between the probe direction and the horizontal center line of the rotor is 2.5mm. Setting the resolution of the image acquired by the high-speed industrial camera b to 512×512, the frame rate to 2000fps, and setting the signal sampling rate of the eddy current sensor to 2048Hz; at the rotating speed of 2475r/min, 180000 data points are acquired by using an eddy current sensor, 6144 continuous frames are acquired by using a high-speed industrial camera, and the time domain waveform and spectrogram of the data acquired by the eddy current sensor are shown in FIG. 2;
the method for extracting the vibration displacement of the rotor comprises the following steps:
(1) Gray processing is carried out on the 6144 video frames, data of a t frame I (x, y, t) and a first frame I (x, y, 1) in each video frame are extracted to be used as an input part, and 6144 input parts are obtained;
(2) Identifying an edge region (fig. 4) of the first frame (fig. 3) with a sobel operator, and taking the identified region as an input region (fig. 5);
In Matlab, the input region is spatially filtered by an orthogonal Gabor filter, and the set of orthogonal Gabor filter center frequency coefficients W and the set of gaussian kernel coefficients S are set as follows:
The gaussian function is expressed as:
Wherein sigma is the standard deviation of the filter, and s is the Gaussian kernel function coefficient;
The sine part (F Cx(s)、FCy (s)) and the cosine part (F Sx(s)、FSy (s)) of the filter in the horizontal and vertical directions are calculated, respectively:
FCx(s)=G(s)cos(2πω1s)
FCy(s)=G(s)cos(2πω2s)
FSx(s)=G(s)sin(2πω1s)
FSy(s)=G(s)sin(2πω2s)
wherein ω 1 and ω 2 are the horizontal direction and vertical direction coefficients of the filter frequency, respectively;
Next, convolution calculations are performed on the input regions in the horizontal and vertical directions, respectively, resulting in a sine filter result a S (x, y, t), a cosine filter result a C (x, y, t):
AS(x,y,t)=(FCy*Tsx)+(FSy*Tcx)
AC(x,y,t)=(FCy*Tcx)-(FSy*Tsx)
Where, represents the convolution operation, T sx and T sx are the lateral and longitudinal gradients of the image.
Phase compensation is carried out on A S (x, y, t) and A C (x, y, t) through arctangent operation, so as to obtain phase phi (x, y, t):
calculating the phase difference of the pixel points: Δφ (x, y, t) =φ (x, y, t) - φ (x, y, 1)
Calculating pixel speed from the phase difference:
Where st denotes a length of the time series, S xy (x, y, t) denotes a sum of products of the phase information Φ (x, y, t) and the time series index st, S xy (x, y) denotes a cumulative sum of pixel-point phase information at the position (x, y), S x (x, y) denotes a cumulative sum of phase information at the position (x, y) in a horizontal direction of the pixel point, S y (x, y) denotes a cumulative sum of phase information at the position (x, y) in a vertical direction of the pixel point, W i is a frequency component of the filter, and N denotes the number of filters;
The pixel velocity V (x, y, t) is multiplied by the frame rate to obtain the vibration displacement delta (x, y, t) =v (x, y, t) ·Δt of the pixel;
where Δt is the time between two frames being input, i.e., the frame rate.
(3) Firstly calculating the average value and standard deviation of vibration displacement of all pixel points in an input area, then eliminating the vibration displacement data of the pixel points which are not in the range of the average value and the standard deviation, and then calculating the average value of vibration displacement of the rest pixel points to obtain the average value of vibration displacement of the pixel points in the input area, and taking the average value as the vibration displacement of the input part; smoothing the vibration displacement of each three continuous input parts in Matlab by using smoothdata functions to obtain 6144 vibration displacement data of 6144 input parts;
(4) Normalizing 6144 vibration displacement data, wherein the normalized data time domain waveform and spectrogram are shown in figure 6;
Comparing the time domain diagram of the data in fig. 6 with the signal acquired by the eddy current sensor with a spectrogram, wherein the comparison result is shown in fig. 7; since the vibration of the object shot by the high-speed industrial camera is the projection of the vibration of the three-dimensional space to the two-dimensional space, the time domain waveform of the final result is not exactly the same as the time domain waveform of the actual rotor vibration, and the extraction result of the method is approximately the same as the measurement result of the sensor on the time domain waveform, and the extracted vibration frequencies are 41.2598Hz and 41.25Hz respectively on the frequency spectrum, and the error is less than 0.01Hz, so that the method can accurately capture the frequency characteristics of the vibration. It is noted that the present process includes the related calculation of 6144 frame images, and the total running time of Matlab R2022a is 506s, which has higher running efficiency. The result verifies the reliability and accuracy of the method provided by the invention in tracking the vibration of the rotary machine, and provides a new non-contact measurement mode for the vibration measurement of the rotary machine.
Claims (3)
1. The method for detecting the vibration displacement of the rotary machine based on vision is characterized by comprising the following steps:
(1) Shooting the running rotary machine by using a high-speed industrial camera, and obtaining clear video images at a frame rate of 1000-6000 fps;
(2) Carrying out gray scale processing on video frames in the video in the step (1), extracting data of a t frame I (x, y, t) and a first frame I (x, y, 1) in the video frames as an input part, and obtaining n input parts;
(3) Identifying the edge area of each input part by using a sobel operator, taking the identified area as an input area, performing spatial filtering on the input area, extracting vibration displacement delta (x, y, t) of pixel points in the input area, and calculating the average value of the vibration displacement of the pixel points in the input area to obtain the vibration displacement of each input part;
Smoothing the vibration displacement of the three continuous input parts by smoothdata functions to obtain n vibration displacement data of n input parts;
(4) Based on the n vibration displacement data obtained in the step (3), generating a time domain diagram and a spectrogram, and monitoring the running state of the rotary machine according to the time domain diagram and the spectrogram to realize real-time diagnosis of the rotary machine.
2. The vision-based rotary mechanical vibration displacement detection method according to claim 1, wherein the step of extracting the vibration displacement of the pixel point in the input area in step (3) is as follows:
(1) Setting a center frequency coefficient set W and a Gaussian kernel function coefficient set S of the orthogonal Gabor filter;
(2) Respectively calculating sine and cosine parts F Cx(s)、FCy(s)、FSx(s) and F Sy(s) of the filter in the horizontal direction and the vertical direction, and respectively carrying out convolution calculation on the input area in the horizontal direction and the vertical direction to obtain a sine filter result A S (x, y, t) and a cosine filter result A C (x, y, t);
(3) Phase compensation is carried out on A S (x, y, t) and A C (x, y, t) through arc tangent operation, phase phi (x, y, t) is obtained, and phase difference delta phi (x, y, t) =phi (x, y, t) -phi (x, y, 1) of the pixel points is calculated;
(4) And calculating a pixel speed V (x, y, t) through the pixel phase difference, and multiplying the pixel speed V (x, y, t) by the frame rate to obtain vibration displacement delta (x, y, t) of a plurality of pixel points in the input area.
3. The vision-based rotary mechanical vibration displacement detection method according to claim 1, characterized in that: and (3) calculating the average value of the vibration displacement of the pixel points in the input area, namely calculating the average value and standard deviation of the vibration displacement of all the pixel points in the input area, removing the vibration displacement data of the pixel points which are not in the average value and standard range, and calculating the average value of the vibration displacement of the rest pixel points.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410068456.5A CN117911363A (en) | 2024-01-17 | 2024-01-17 | Rotary machine vibration displacement detection method based on vision |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202410068456.5A CN117911363A (en) | 2024-01-17 | 2024-01-17 | Rotary machine vibration displacement detection method based on vision |
Publications (1)
Publication Number | Publication Date |
---|---|
CN117911363A true CN117911363A (en) | 2024-04-19 |
Family
ID=90683397
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202410068456.5A Pending CN117911363A (en) | 2024-01-17 | 2024-01-17 | Rotary machine vibration displacement detection method based on vision |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117911363A (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118209312A (en) * | 2024-05-21 | 2024-06-18 | 远届测控设备(青岛)有限公司 | Shaft dynamic compensation measuring device based on visual perception |
-
2024
- 2024-01-17 CN CN202410068456.5A patent/CN117911363A/en active Pending
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN118209312A (en) * | 2024-05-21 | 2024-06-18 | 远届测控设备(青岛)有限公司 | Shaft dynamic compensation measuring device based on visual perception |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN117911363A (en) | Rotary machine vibration displacement detection method based on vision | |
CN107271025B (en) | Device and method for synchronously measuring three-dimensional vibration of rotating shaft | |
Zhong et al. | Measurement of instantaneous rotational speed using double-sine-varying-density fringe pattern | |
US8900143B2 (en) | Intraocular pressure detecting device and detecting method thereof | |
KR20070012503A (en) | Small displacement measuring method and instrument | |
CN110866892B (en) | Offline vibration measurement analysis method and system based on vision | |
CN110084127A (en) | A kind of magnetic suspension rotor vibration measurement method of view-based access control model | |
Wang et al. | Rotational speed measurement through image similarity evaluation and spectral analysis | |
JP2001324305A (en) | Image correspondent position detector and range finder equipped with the same | |
CN109945955B (en) | Water meter visual online detection method and system | |
CN112683382B (en) | Structure three-dimensional vibration measurement system and method based on monocular vision | |
Yang et al. | Monocular vision-based calibration method for determining frequency characteristics of low-frequency vibration sensors | |
JP5444879B2 (en) | Chassis dynamometer | |
Jin et al. | An experimental investigation of spindle rotary error on high-speed machining center | |
CN110118582B (en) | Fault diagnosis method and system for rotary mechanical equipment | |
Yang et al. | Monocular vision-based Earth’s graviation method used for low-frequency vibration calibration | |
CN111912361A (en) | Surface measuring instrument | |
JP2004325155A (en) | Method, apparatus, and program for inspecting vehicle instrument | |
Deng et al. | Synchronous monitoring of axial vibration and rotation speed of rotating cylinder by linear array scanning | |
CN110440898A (en) | A kind of mechanical vibration measurement method of rotation | |
CN113465549A (en) | Mark point-free bearing retainer contact angle measuring system and method based on vision | |
WO2013091766A1 (en) | A method of assessing the technical condition of a rotating shaft | |
CN113758695B (en) | System and method for realizing rotation axis order analysis by adopting visual composite position-sensing stripes | |
Zhang et al. | Vibration displacement measurement method based on vision Gaussian fitting and edge optimisation for rotating shafts | |
He et al. | Automatic calibration system for digital-display vibrometers based on machine vision |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination |