CN110619337A - Method for extracting and identifying sub-pixel edge of pantograph slide plate - Google Patents
Method for extracting and identifying sub-pixel edge of pantograph slide plate Download PDFInfo
- Publication number
- CN110619337A CN110619337A CN201910831149.7A CN201910831149A CN110619337A CN 110619337 A CN110619337 A CN 110619337A CN 201910831149 A CN201910831149 A CN 201910831149A CN 110619337 A CN110619337 A CN 110619337A
- Authority
- CN
- China
- Prior art keywords
- line
- edge
- lower edge
- auxiliary
- point
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 35
- 238000000605 extraction Methods 0.000 claims abstract description 30
- 230000000694 effects Effects 0.000 claims abstract description 8
- 238000012216 screening Methods 0.000 claims abstract description 3
- 229910052799 carbon Inorganic materials 0.000 claims description 5
- 238000005299 abrasion Methods 0.000 abstract description 8
- 238000005259 measurement Methods 0.000 abstract description 4
- 238000007689 inspection Methods 0.000 abstract 1
- 238000001514 detection method Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 4
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000010191 image analysis Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 241001270131 Agaricus moelleri Species 0.000 description 1
- 241001391944 Commicarpus scandens Species 0.000 description 1
- 241000733322 Platea Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- XAGFODPZIPBFFR-UHFFFAOYSA-N aluminium Chemical compound [Al] XAGFODPZIPBFFR-UHFFFAOYSA-N 0.000 description 1
- 229910052782 aluminium Inorganic materials 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000012790 confirmation Methods 0.000 description 1
- 230000004392 development of vision Effects 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 239000012212 insulator Substances 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000008439 repair process Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60L—PROPULSION OF ELECTRICALLY-PROPELLED VEHICLES; SUPPLYING ELECTRIC POWER FOR AUXILIARY EQUIPMENT OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRODYNAMIC BRAKE SYSTEMS FOR VEHICLES IN GENERAL; MAGNETIC SUSPENSION OR LEVITATION FOR VEHICLES; MONITORING OPERATING VARIABLES OF ELECTRICALLY-PROPELLED VEHICLES; ELECTRIC SAFETY DEVICES FOR ELECTRICALLY-PROPELLED VEHICLES
- B60L5/00—Current collectors for power supply lines of electrically-propelled vehicles
- B60L5/18—Current collectors for power supply lines of electrically-propelled vehicles using bow-type collectors in contact with trolley wire
- B60L5/20—Details of contact bow
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/44—Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Abstract
The invention discloses a method for extracting and identifying a sub-pixel edge of a pantograph slide plate, which comprises the steps of triggering a photoelectric sensor to enable a fixed camera to work when a train reaches a fixed position, determining an interested area of the slide plate in a pantograph image shot by the camera, adding an auxiliary line in the area, accurately positioning an image edge point by using a sub-pixel edge extraction algorithm based on partial area effect, screening and updating points on a target edge line according to constraint relations between inner edge lines of the slide plate and between the inner edge line and an outer edge line, and further determining the target edge line. The method eliminates the interference of external edge lines, is more suitable for more complicated edge extraction conditions in practical conditions, increases the inspection constraint on edge points, further improves the accuracy of edge extraction and identification, obtains the sub-pixel characteristic information of the upper edge line and the lower edge line of the pantograph by the method, and has important significance for the online measurement of the abrasion of the pantograph slide plate.
Description
Technical Field
The invention belongs to the technical field of pantograph fault detection, and particularly relates to a method for extracting and identifying sub-pixel edges of a pantograph slide plate.
Background
The pantograph slide plate is important electrical equipment for an electric train to contact with a contact net and obtain current from the contact net. As shown in fig. 1, which is a schematic structural view of a pantograph assembly contacting with a contact line, a pantograph slide plate and the contact line have a certain contact pressure when contacting, and a main force point between the pantograph slide plate and the contact line is a lower surface of the contact line and an upper surface of the pantograph slide plate. Therefore, sliding wear and electrical wear inevitably occur on the upper surface of the carbon-based sliding plate. In order to avoid or reduce the occurrence of pantograph-catenary accidents, railway workers need to perform regular maintenance, repair, replacement and other work on the wear condition of the pantograph, particularly the sliding plate, so as to ensure that the pantograph-catenary can maintain a good contact state and a good current collection condition when a train runs. Therefore, the method for accurately extracting the upper edge and the lower edge of the pantograph slide plate is researched, and the method has important significance for improving the overhauling efficiency of the pantograph slide plate of the train.
With the maturity of image processing technology and the rapid development of vision measurement technology, pantograph slide plate abrasion detection based on machine vision becomes an effective method. As shown in fig. 2, because the contact abrasion and the electrical abrasion exist between the carbon strip 101 of the slide plate and the contact line 105, the accurate extraction and identification of the upper and lower edges of the slide plate in a complex environment are completed through image processing and analysis, which is a key link for realizing the wear measurement of the pantograph slide plate based on the vision technology.
However, in actual image processing, information on the edge of an image near the pantograph pan is complicated, and a large number of disturbance factors are involved. The pantograph bow comprises various background information such as a pantograph supporting frame, a sliding plate bracket, a contact net, a catenary and the like near the pantograph bow, projection of a contact line on the side surface of the sliding plate, false edges generated by the unevenness of the side surface of the sliding plate, strong light reflection phenomena of a background plate and the side surface of the sliding plate caused by high-power strong light irradiation, and even complex conditions such as aluminum edge wrapping of the side surfaces of some sliding plates. As shown in fig. 3, which is an image of a pantograph in actual operation captured by an on-site camera, the complex edge environment around the skateboard causes certain interference to the accurate extraction and identification of the target edge, and in this complex situation, it is difficult to accurately identify the upper and lower edges of the skateboard.
In the prior art, the edge extraction of the pantograph slide plate usually only considers the extraction of the upper edge and the lower edge, but neglects the interference of other edges on the slide plate on the accurate extraction of the upper edge and the lower edge; the edge is also extracted accurately by using sub-pixels, but the extraction precision is influenced by the lack of verification of edge points after extraction.
Disclosure of Invention
The purpose of the invention is as follows: aiming at the problems in the prior art, the invention provides a method for extracting and identifying the sub-pixel edge of the pantograph pan, which can be used for extracting the upper edge and the lower edge of the pan and simultaneously considering the influence of other edges; the method for extracting the edge points of the sliding plate target is adopted to accurately extract the edge points of the sliding plate target, the sub-pixel characteristics of the target edge points are obtained, meanwhile, the extracted complex edge points are subjected to geometric constraint analysis, the target edge of the sliding plate is accurately identified, the precision of the edge detection of the sliding plate is further improved, and an important foundation is laid for three-dimensional measurement of the abrasion of the sliding plate.
The technical scheme is as follows: a method for extracting and identifying sub-pixel edges of a pantograph pan is characterized by comprising the following steps: comprises the following steps of (a) carrying out,
step A, starting a trigger camera to work when a train passes through the position of a trigger sensor, shooting an image of a pantograph at a fixed position, and determining an interested area of a sliding plate in the image;
b, adding a group of single-pixel vertical auxiliary lines in the region of interest of the step A, solving the target edge lines of the auxiliary lines and the skateboard image, the feature information of the interference edge lines and the intersection point set information intersected with the vertical auxiliary lines by adopting a sub-pixel edge extraction method based on partial area effect, and establishing an identification model of the target edge of the skateboard;
and step C, according to the constraint relationship between the inner edge lines and the outer edge lines of the skateboard image, screening out intersection points which meet the constraint condition in the intersection point set in the step A, updating the positions of the intersection points which do not meet the constraint condition on the auxiliary line where the intersection points are located, and determining the upper edge reference line and the lower edge reference line of the skateboard image together by the intersection points which meet the constraint condition and the updated intersection points, namely the upper edge line and the lower edge line of the skateboard which are required to be taken.
Further, the step B includes:
b1, accurately positioning the real edge line of the skateboard in the region of interest by adopting a sub-pixel edge extraction algorithm based on partial area effect, wherein the real edge line comprises a target edge line and an interference edge line;
step B2, adding a group of single-pixel vertical auxiliary lines in the region of interest, and obtaining the set of all the points of intersection of the sub-pixel edge lines and the auxiliary lines obtained in the step B1 through the equation of the auxiliary linesWherein i represents the ith auxiliary line in the vertical direction, j represents the jth intersection point generated by the intersection of the ith auxiliary line and the real edge line detected by the image, N is the number of the auxiliary lines in the region of interest, NiThe number of intersection points of the i-th auxiliary line and the real edge line of the image is determined;
and step B3, compared with the real extracted sub-pixel edge image, the bottom interference edge of the slide plate interested area and the short edge of the side face of the slide plate are omitted, and the identification model of the target edge of the slide plate is established.
Further, the step C includes:
step C1, determining a reference line of the lower edge of the sliding plate according to the constraint relation that the lower edge line of the sliding plate is approximately parallel to the bottom surface projection edge line and has a certain thickness;
and C2, determining a reference line of the upper edge of the sliding plate according to the constraint relation that the upper edge and the lower edge of the sliding plate are approximately parallel and have a certain thickness and the reference line of the lower edge determined in the step C1.
Further, the step C1 includes:
step C11, using the target lower edge line of the slide plate as the lower edge reference line and the bottom surface projection edge line as the lower edge reference line, and searching a first pair of intersection points meeting the constraint condition from the first auxiliary line according to the constraint relation that the thickness between the target lower edge line of the slide plate and the bottom surface projection edge line, namely the lower edge reference line and the reference line, is constant and approximately parallelAndrespectively as reference lines of the lower edge of the sliding plateAnd a reference lineAt the i-th0A starting point on the bar auxiliary line;
step C12, searching the intersection point of the next auxiliary line and the lower edge reference line and the reference line according to the constraint relation in the step C11Andaccording to the constraint condition that the lower edge datum line is approximately parallel to the reference line, if the constraint condition is met, the lower edge datum line is approximately parallel to the reference lineRespectively the reference point and the lower edge of the sliding plate on the auxiliary line, if not, the step is transferred to the step C13, and the step C12 is repeated until the ith step0All the auxiliary lines after the bar are searched, and the step C14 is carried out;
step C13, utilizing the slope between the lower edge reference points on the first two auxiliary lines to perform straight line fitting at the lower edge reference point on the previous auxiliary line, wherein the intersection point of the fitted straight line and the current auxiliary line is the lower edge reference point on the auxiliary lineUtilizing the slope between the lower edge reference points on the first two auxiliary lines to perform straight line fitting at the lower edge reference point on the previous auxiliary line, wherein the intersection point of the fitted straight line and the current auxiliary line is the lower edge reference point on the auxiliary line
Step C14, lower edge reference point set on all auxiliary lines obtained from steps C11, C12 and C13And reference point setThe reference line of the lower edge of the sliding plate can be determinedAnd a reference line
Further, in the step C11, the constraint relationship of the thickness between the reference line of the lower edge of the sliding plate and the reference line is:
wherein ,denotes the ith0The intersection point of the strip auxiliary line and the reference line of the lower edge of the sliding plate,is the ith0Intersection point, n, of the strip auxiliary line and the slide plate lower edge reference lineyThe gradient direction component of the intersection point in the vertical direction is represented,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,indicating the initial number of pixels between the reference line and the lower edge reference line of the skateboard image obtained at different times, thick1 being the average value of the number of pixels between the reference line and the lower edge reference line of the skateboard, thick1 beingIs determined.
Further, the constraint condition that the reference line of the lower edge of the sliding plate is approximately parallel to the reference line in the step C12 is as follows:
wherein ,the edge slope between the reference line of the lower edge of the skateboard and the corresponding intersection point of the current auxiliary line and the previous auxiliary line,the slope of the edge between the reference line of the lower edge of the sliding plate and the corresponding intersection point of the current auxiliary line and the previous auxiliary line is theta, and theta is a threshold value of the difference value of the two slopes.
Further, the step C2 includes:
step C21, according to the thickness constraint condition between the upper edge and the lower edge of the two ends of the skateboard image, and according to the extraction result of the reference point of the lower edge of the skateboard in the step C1, in the ith step0Searching intersection points meeting constraint conditions on auxiliary lineAs a reference line of the upper edge of the skateboardAt the i-th0A starting point on the bar auxiliary line;
step C22, searching the intersection point of the next auxiliary line and the slide plate upper edge reference line according to the constraint relation in the step C21Using the constraint condition of the characteristic that the difference of the pixel numbers between the upper edge datum line and the lower edge datum line of the sliding plate of the adjacent auxiliary lines is not large, if the constraint condition is met, thenIf the reference point is not the reference point of the upper edge of the slide plate on the auxiliary line, the step C23 is executed, and the step C22 is repeated until the reference point is reached to the ith0All the auxiliary lines after the bar are searched, and the step C24 is carried out;
step C23, according to the number d of pixels between the upper and lower edge reference lines of the slide plate on the previous auxiliary linei-1And the vertical coordinate of the intersection point of the current auxiliary line and the lower edge reference lineFitting the upper edge point reference point of the current auxiliary line wherein
Step C24, obtaining the product according to the steps C21, C22 and C23Set of upper edge reference points on all the auxiliary linesCan determine the reference line of the upper edge of the sliding plate
Further, the thickness constraint condition between the upper and lower edges of the two ends of the sliding plate in the step C21 is as follows:
wherein ,d0The number of pixels between the upper and lower edges at the two ends of the slide,is the ith0The number of intersections of the strip auxiliary lines with all the edges of the image,is the ith to be determined0The reference point of the upper edge of the slide plate on the strip auxiliary line,is the ith0The lower edge reference point of the slide plate on the strip auxiliary line.
Further, the constraint condition that the number of pixels between the upper edge reference line and the lower edge reference line of the slider in the step C22 is as follows:
wherein ,diThe number of pixels between the upper edge reference line and the lower edge reference line of the slide plate on the current auxiliary line, di-1The number of pixels between the upper edge reference line and the lower edge reference line of the slide plate on the previous auxiliary line, and sigma is the number of pixels surrounded on the current auxiliary line and the number of pixels surrounded on the previous auxiliary lineIs constrained.
Further, in the step C, in a shadow area of the intersection part of the sliding plate and the contact network, fitting the upper edge point of the shadow left part by using a RANSAC method by using a point on the determined upper edge line of the sliding plate on the left side of the contact line projection area, and calculating a difference value of the reference point of the shadow area of the sliding plate, thereby realizing the positioning of the upper edge point of the sliding plate in the contact network projection area.
Has the advantages that:
(1) compared with the prior art, the method has the advantages that when the upper edge and the lower edge of the pantograph slide plate are extracted, the interference and the influence of other edge lines are considered, and the method is more suitable for extracting and identifying the more complicated edge of the pantograph slide plate in actual conditions;
(2) after the sub-pixel feature extraction is carried out on the edge points, the detection link is added, the non-conforming edge points are eliminated according to the geometric constraint relation of the upper edge and the lower edge of the pantograph slide plate, and the detection precision is further improved;
(3) the method and the device can obtain the sub-pixel characteristic information of the upper edge line and the lower edge line of the slide plate while extracting the edges, and provide an important basis for measuring the abrasion of the pantograph slide plate.
Description of the drawings:
fig. 1 is a schematic structural view of a pantograph assembly in contact with a catenary;
FIG. 2 is a schematic structural view of a pantograph pan;
FIG. 3 is an image of a pantograph in actual operation taken by a live camera;
FIG. 4 is a flow chart of a method of the present invention;
FIG. 5 is a schematic diagram of the theory of sub-pixel extraction of image edges;
FIG. 6 is a model of the identification of the target edge of the skateboard;
FIG. 7 is a schematic diagram of the extraction of the reference line of the lower edge of the slide plate and the starting point of the reference line;
FIG. 8 is a schematic diagram of the recognition result of the reference line and the lower edge reference line of the slide plate;
FIG. 9 is a diagram illustrating the detection result of the reference line of the upper edge of the slide plate;
wherein, 1 is a pantograph slide plate, 2 is a pantograph head current connection assembly, 3 is an upper frame, 4 is a pull rod, 5 is a balance rod, 6 is a lower arm rod, 7 is a system damper, 8 is a bottom frame, 9 is an air bag assembly, 10 is an insulator assembly, 11 is a contact line, 12 is a dropper and 13 is a carrier cable;
101 is a carbon strip of a pantograph slide plate, 102 is a bonding layer, 103 is a bracket assembly, 104 is an air channel pipe joint, and 105 is a contact line;
31 is the upper edge of the sliding plate, 32 is the lower edge of the sliding plate, 33 is the edge of the contact line, 34 is the edge of the side surface of the sliding plate, 35 is the projected edge of the bottom surface of the sliding plate, and 36 is the interference edge of the background plate;
61 is the intersection point of the edge line of the sliding plate and the auxiliary line, 62 is the edge of the contact line, 63 is the upper edge of the sliding plate, 64 is the lower edge of the sliding plate, and 65 is the projection edge of the bottom surface of the sliding plate;
71 is the starting point of the reference line of the lower edge of the skateboard, and 72 is the starting point of the reference line of the lower edge of the skateboard;
identified slider lower edge reference line 81, identified slider lower edge reference line 82, and identified slider upper edge reference line 83.
The specific implementation mode is as follows:
the present invention will be further described with reference to the accompanying drawings.
The invention discloses a method for extracting and identifying sub-pixel edges of a pantograph pan, which comprises the following steps of:
step A, starting a trigger camera to work when a train passes through the position of a trigger sensor, shooting an image of a pantograph at a fixed position, and determining an interested area of a sliding plate in the image;
step B, adding a group of single-pixel vertical auxiliary lines in the region of interest of the step A, solving the target edge line of the auxiliary line and the skateboard image, the feature information of the interference edge line and the intersection point set information intersected with the vertical auxiliary lines by adopting a sub-pixel edge extraction method based on partial area effect, and establishing an identification model of the target edge of the skateboard, wherein the identification model specifically comprises the following steps:
b1, accurately positioning the real edge line of the skateboard in the region of interest by adopting a sub-pixel edge extraction algorithm based on partial area effect, wherein the real edge line comprises a target edge line, an interference edge line and characteristic information of each edge;
image Ii，jGenerally, the gray value of each pixel is related to the gray value of the pixels in the neighborhood of the pixel by a certain amount. As shown in fig. 5(a), assuming that there is a straight line edge passing through at the pixel (i, j), the gray value of the pixel can be defined as:
wherein A and B are gray values at two sides of the edge, SA and SBThe area occupied by the parts with the gray values A and B on two sides in the pixel respectively, h represents the length and width value of each pixel, and the above parameters also satisfy h2＝SA+SB。
Without loss of generality, by a quadratic curve y ═ cx2The + bx + a approximation is used as an image edge model, as shown in fig. 5 (b). The curve is arranged through a pixel (i, j), an image is divided into an upper gray part and a lower gray part, the gray value of the pixel above the curve is marked as B, the gray value of the pixel below the curve is marked as A, and the accurate extraction process of the image edge is analyzed through the parameter solution of the quadratic curve model.
Taking pixel (i, j) as the center, selecting 5 × 3 frame, SL、SM and SRIs the sum of the grey values of the individual pixels of all the rows in the first, second and third columns of the rectangular frame. In a rectangular frame, the area of the pixel under the edge line of each column can be determined by the following equation
For convenience of calculation, let h equal to 1, and the coefficients of the edge quadratic curve model can be obtained from the above relation
The gray values a and B can be determined as follows:
image is vertically distant from pixel Ii，jThe distance of the center, and also the sub-pixel position of the edge in the vertical direction of the pixel (i, j) is
lo＝(0，a) (5)
Normalized normal vector of edge point at (i, j) is
The curvature of the point is calculated by the formula
By using the above algorithm for solving the sub-pixel edge based on partial area effect, the sub-pixel position of the image edge, the normal vector and curvature of the edge point, the image gray values at two sides of the edge point and other features can be accurately solved, and the method has important significance for accurately positioning the upper edge and the lower edge of the subsequent slide plate.
Step B2, adding a group of single-pixel vertical auxiliary lines in the region of interest, and obtaining a set of all points of intersection of the jth edge line and the ith auxiliary line in the sub-pixel edge lines obtained in the step B1 through an equation of the auxiliary lines
And establishing a rectangular coordinate system by taking the upper left corner of the pantograph image as an origin, wherein the added auxiliary line equation can be defined as follows:
wherein ,x0Is the column coordinate of the leftmost column of pixels in the original image, Δ d is the distance between two adjacent auxiliary lines, r1Is the line coordinate, r, of the 1 st line pixel of the interested area in the original whole image2I represents the ith auxiliary line, n is the number of the auxiliary lines in the interested region, and assuming that the length of the pixel in the horizontal direction of the interested region is L, the pixels coexist in the regionAuxiliary lines, symbols, or the like, in the vertical direction of the stripTo round-down operations.
In the image analysis process, the auxiliary lines are added only for conveniently solving the intersection point set of each auxiliary line and the real edge line solved in the image.
On the left side of the bow net contact point, the upper edge and the lower edge of the auxiliary line and the contact line and the interference edge line of the side face of the sliding plate are intersected, the intersection point set of the auxiliary line and the image real edge line in each vertical direction is recorded and recorded asWherein i represents the ith auxiliary line in the vertical direction, j represents the jth intersection point generated by the intersection of the ith auxiliary line and the real edge line detected by the image, N is the number of the auxiliary lines in the region of interest, NiThe number of the intersection points, x, of the i-th auxiliary line and the image real edge linei＝x0+ i · Δ d is the pixel column coordinate of the i-th auxiliary line in the image horizontal direction,and the pixel row coordinate of the corresponding jth intersection point on the ith auxiliary line in the vertical direction of the image is defined. Point setIs NiX 2 intersection coordinate matrix, intersection ordinate in each columnAre combined in a gradually increasing manner from top to bottom in the order of intersecting with the true edge line of the image.
The image edge extraction method introduced in the above section can accurately calculate the sub-pixel coordinates of the image edge points, and then easily obtain the set of all the points where the calculated sub-pixel edges in the image of the region of interest intersect with the ith auxiliary line through the equation of the auxiliary line.
Step B3, according to the imaging characteristics of the pantograph and the extraction condition of the sub-pixel edge of the area of interest of the skateboard, taking close-range image analysis of the skateboard as an example, the identification model of the target edge of the skateboard as shown in fig. 6 is established, and compared with the actually extracted sub-pixel edge image, the bottom interference edge of the area of interest of the skateboard and the short edge of the side surface of the skateboard are omitted, and actually, experiments can prove that the edges have little influence on the correct identification of the target edge of the skateboard.
This scheme is fit for adding of various types of pantograph slide upper and lower edge line and surveys: aiming at the same type of pantograph, because the camera images the pantograph head at a fixed position, the edge positions of the pantograph passing at different times are relatively fixed; aiming at different types of pantographs, the target edge of the pantograph head of the pantograph can be accurately positioned by adjusting the designed parameters in the method.
And C, solving an intersection point set of each auxiliary line and the edge extracted in the image horizontal direction in the step B, and accurately identifying the upper edge and the lower edge of the skateboard according to the geometric constraint of the skateboard edge by using the discrete intersection point information.
According to the constraint relation between the inner edge lines and the outer edge lines of the skateboard image, the intersection points meeting the constraint condition in the intersection point set of the step A are screened out, the intersection point positions which do not meet the constraint condition are updated on the auxiliary line where the intersection points are located, the intersection points meeting the constraint condition and the updated intersection points jointly determine the upper edge reference line and the lower edge reference line of the skateboard image, namely the upper edge line and the lower edge line of the skateboard required to be taken, and the method comprises the following steps:
step C1, determining the reference line of the lower edge of the sliding plate according to the constraint relation that the lower edge line of the sliding plate is approximately parallel to the bottom surface projection edge line and has a certain thickness, and specifically comprises the following steps:
step C11, because the pantograph is running, the bottom of the sliding plate is not worn, the target lower edge line of the sliding plate is used as the lower edge reference line, the bottom surface projection edge line is used as the lower edge reference line, and according to the constraint relation that the thickness between the target lower edge line of the sliding plate and the bottom surface projection edge line, namely the lower edge reference line and the reference line is constant and approximately parallel, the first pair of intersection points meeting the constraint condition is searched from the first auxiliary lineAndrespectively as reference lines of the lower edge of the sliding plateAnd a reference lineAt the i-th0Starting point on the bar auxiliary line. As shown in fig. 7, where 71 is the starting point of the slider lower edge reference line and 72 is the starting point of the slider lower edge reference line.
The certain constraint relation of thickness between the lower edge datum line of the sliding plate and the reference line is as follows:
wherein ,denotes the ith0The intersection point of the strip auxiliary line and the reference line of the lower edge of the sliding plate,is the ith0Intersection point, n, of the strip auxiliary line and the slide plate lower edge reference lineyThe gradient direction component of the intersection point in the vertical direction is represented,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,the number of pixels between the reference line and the lower edge reference line of the skateboard image obtained at different times is shown, thick1 is the average value of the number of pixels between the reference line and the lower edge reference line of the skateboard, and thick1 isThe allowable error of (2) is generally 5 to 10 pixels.
In the step, according to the actual situation, an interested area of the sliding plate is selected, and the interference of other edges except the sliding plate is eliminated as much as possible; when edge interference is still encountered, the interference edge can be basically eliminated by further judging the gradient direction of the vertical direction of the edge.
Step C12, sliding plate lower edge reference lineBecause the light is supplemented to the side of the sliding plate by a high-power auxiliary light source, clear imaging can be realized. The gray values of the two sides of the lower edge of the sliding plate are greatly influenced by the change of external light, and according to the method, the extracted lower edge of the sliding plate is broken sometimes, even the edge is lost, and certain difficulty is brought to the confirmation of the corresponding intersection point. Considering the reference edge of the lower edge of the slideThe reference edge is an ideal reference edge, the edge is not easy to break or lose, and the reference edge is easy to extract. Thus, the reference edge of the lower edge of the slide is usedReference edge to lower edge of slide plateTo realize the lower edge of the sliding plateThe rapid detection of (2).
Searching the intersection point of the next auxiliary line and the lower edge reference line and the reference line according to the constraint relation in the step C11Andbecause the complex edge condition of the side surface of the sliding plate has the interference intersection point, according to the constraint condition that the lower edge datum line is approximately parallel to the reference line, if the constraint condition is met, the reference line is parallel to the reference lineRespectively the reference point and the lower edge of the sliding plate on the auxiliary line, if not, the step is transferred to the step C13, and the step C12 is repeated until the ith step0All the auxiliary lines after the bar are searched, and the step C14 is carried out;
the constraint condition that the reference line of the lower edge of the sliding plate is approximately parallel to the reference line is as follows:
wherein ,the edge slope between the reference line of the lower edge of the skateboard and the corresponding intersection point of the current auxiliary line and the previous auxiliary line,the slope is the edge slope between the lower edge reference line of the sliding plate and the intersection point corresponding to the current auxiliary line and the previous auxiliary line, theta is the threshold value of the difference value of the two slopes, and the value range is 0-0.03.
Step C13, because of the lower edge reference line of the slide plateAnd lower edge reference lineApproximately parallel, the slope between the lower edge reference points on the first two auxiliary lines is utilized to perform straight line fitting at the lower edge reference point on the previous auxiliary line, and the intersection point of the fitted straight line and the current auxiliary line is the lower edge reference point on the auxiliary lineUtilizing the slope between the lower edge reference points on the first two auxiliary lines to perform straight line fitting at the lower edge reference point on the previous auxiliary line, wherein the intersection point of the fitted straight line and the current auxiliary line is the lower edge reference point on the auxiliary line
Step C14, lower edge reference point set on all auxiliary lines obtained from steps C11, C12 and C13And reference point setThe reference line of the lower edge of the sliding plate can be determinedAnd a reference lineAs shown in fig. 8, where 81 is the identified slider lower edge reference line and 82 is the identified slider lower edge reference line.
Step C2, sliding plate upper edge lupThe extraction environment is complex, on one hand, due to the abrasion condition between the sliding plate and the contact line, on the other hand, due to the action of external light, the gray scales on two sides of the upper edge are not obviously changed sometimes, and the edge of the extracted sliding plate is broken. Also, the contact line and the contact portion of the slider are exposed to the light of the supplementary light source, and the contact line and the like cause the contact portion to generate the interference edge condition on the side surface of the slider, all of which cause the identification of the upper edge of the slider to be more complicated than the identification of the lower edge.
Determining a reference line of the upper edge of the slide plate according to the constraint relationship that the upper edge and the lower edge of the slide plate are approximately parallel and have a certain thickness and the reference line of the lower edge determined in the step C1, including:
step C21, combining with the actual situation, because the slide plate abrasion at the two ends of the pantograph is small, according to the constraint condition that the number of pixels between the upper edge and the lower edge of the slide plate image at the two ends is basically kept unchanged, and according to the extraction result of the reference point of the lower edge of the slide plate in the step C1, in the ith step0Searching intersection points meeting constraint conditions on auxiliary lineAs a reference line of the upper edge of the skateboardAt the i-th0A starting point on the bar auxiliary line;
the constraint condition that the number of pixels between the lower edges of the upper edges at the two ends of the sliding plate is basically kept unchanged is as follows:
wherein ,d0The number of pixels between the upper and lower edges at the two ends of the slide,is the ith0The number of intersections of the strip auxiliary lines with all the edges of the image,is the ith to be determined0The reference point of the upper edge of the slide plate on the strip auxiliary line,is the ith0The lower edge reference point of the slide plate on the strip auxiliary line.
Step C22, searching the intersection point of the next auxiliary line and the slide plate upper edge reference line according to the constraint relation in the step C21Using the constraint condition of the characteristic that the difference of the pixel numbers between the upper edge datum line and the lower edge datum line of the sliding plate of the adjacent auxiliary lines is not large, if the constraint condition is met, thenIf the reference point is not the reference point of the upper edge of the slide plate on the auxiliary line, the step C23 is executed, and the step C22 is repeated until the reference point is reached to the ith0All the auxiliary lines after the bar are searched, and the step C24 is carried out;
the certain constraint condition of the pixel number between the upper edge datum line and the lower edge datum line of the sliding plate is as follows:
wherein ,diThe number of pixels between the upper edge reference line and the lower edge reference line of the slide plate on the current auxiliary line, di-1The number of pixels between the upper edge reference line and the lower edge reference line of the slide plate on the previous auxiliary line is sigma of the number of pixels on the current auxiliary lineConstrained by the difference between the number of pixels on the previous auxiliary line and the value range of sigma is [0, 7 ]]。
Step C23, according to the number d of pixels between the upper and lower edge reference lines of the slide plate on the previous auxiliary linei-1And the vertical coordinate of the intersection point of the current auxiliary line and the lower edge reference lineFitting the upper edge point reference point of the current auxiliary line wherein
Step C24, upper edge reference point set on all auxiliary lines obtained from steps C21, C22 and C23Can determine the reference line of the upper edge of the sliding plateAs shown in fig. 9, 81 is a detected slide lower edge reference line, 82 is a recognized slide lower edge reference line, and 83 is a recognized slide upper edge reference line.
It should be noted that, due to the shadow caused by the irradiation of the contact line by the auxiliary light source and the interference of the irregular edge of the side surface of the slider, the upper edge point of the contact portion of the slider with the contact line is determined, and if only the constraint of the thickness of the slider between the front and rear adjacent auxiliary lines is adopted, the upper edge of the slider is easily positioned in the shadow area of the slider. In order to accurately detect the accurate and consistent edge of the sliding plate as far as possible, in a shadow area of the intersection part of the sliding plate and a contact net, the determined upper edge point of the sliding plate on the left side of a contact line projection area is utilized, the RANSAC method is used for fitting the upper edge point of the shadow left side part, and then the edge of the contact line is combined to calculate two intersection points of the upper edge and the lower edge of the contact line and the upper edge of the sliding plate. The projected area of the contact line is the area between the two intersection points. And then, a difference value is calculated for the reference point of the shadow area of the sliding plate by utilizing the linear equation of the upper edge of the sliding plate, so that the upper edge of the sliding plate in the projection area of the contact net is positioned.
The input to the RANSAC algorithm is a set of observations, a parameterized model that can be interpreted or adapted to the observations, some trusted parameters. RANSAC achieves this goal by iteratively selecting a set of random subsets in the observed data.
The above description is only of the preferred embodiments of the present invention, and it should be noted that: it will be apparent to those skilled in the art that various modifications and adaptations can be made without departing from the principles of the invention and these are intended to be within the scope of the invention.
Claims (10)
1. A method for extracting and identifying sub-pixel edges of a pantograph pan is characterized by comprising the following steps:
step A, starting a trigger camera to work when a train passes through the position of a trigger sensor, shooting an image of a pantograph at a fixed position, and determining an interested area of a sliding plate in the image;
b, adding a group of single-pixel vertical auxiliary lines in the region of interest of the step A, solving the target edge lines of the auxiliary lines and the skateboard image, the feature information of the interference edge lines and the intersection point set information intersected with the vertical auxiliary lines by adopting a sub-pixel edge extraction method based on partial area effect, and establishing an identification model of the target edge of the skateboard;
and step C, according to the constraint relationship between the inner edge lines and the outer edge lines of the skateboard image, screening out intersection points which meet the constraint condition in the intersection point set in the step A, updating the positions of the intersection points which do not meet the constraint condition on the auxiliary line where the intersection points are located, and determining the upper edge reference line and the lower edge reference line of the skateboard image together by the intersection points which meet the constraint condition and the updated intersection points, namely the upper edge line and the lower edge line of the skateboard which are required to be taken.
2. The pantograph pan sub-pixel edge extraction and identification method according to claim 1, wherein the step B comprises:
b1, accurately positioning the real edge line of the skateboard in the region of interest by adopting a sub-pixel edge extraction algorithm based on partial area effect, wherein the real edge line comprises a target edge line and an interference edge line;
step B2, adding a group of single-pixel vertical auxiliary lines in the region of interest, and obtaining the set of all the points of intersection of the sub-pixel edge lines and the auxiliary lines obtained in the step B1 through the equation of the auxiliary linesWherein i represents the ith auxiliary line in the vertical direction, j represents the jth intersection point generated by the intersection of the ith auxiliary line and the real edge line detected by the image, N is the number of the auxiliary lines in the region of interest, NiThe number of intersection points of the i-th auxiliary line and the real edge line of the image is determined;
and step B3, compared with the real extracted sub-pixel edge image, the bottom interference edge of the slide plate interested area and the short edge of the side face of the slide plate are omitted, and the identification model of the target edge of the slide plate is established.
3. The pantograph pan sub-pixel edge extraction and identification method according to claim 1, wherein the step C comprises:
step C1, determining a reference line of the lower edge of the sliding plate according to the constraint relation that the lower edge line of the sliding plate is approximately parallel to the bottom surface projection edge line and has a certain thickness;
and C2, determining a reference line of the upper edge of the sliding plate according to the constraint relation that the upper edge and the lower edge of the sliding plate are approximately parallel and have a certain thickness and the reference line of the lower edge determined in the step C1.
4. The pantograph pan sub-pixel edge extraction and identification method according to claim 3, wherein the step C1 comprises:
step C11, using the target lower edge line of the skateboard as the lower edge reference lineThe bottom surface projection edge line is a lower edge reference line, and a first pair of intersection points meeting constraint conditions are searched from a first auxiliary line according to the constraint relation that the thickness between the target lower edge line of the sliding plate and the bottom surface projection edge line, namely the lower edge reference line, and the reference line is constant and approximately parallelAndrespectively as reference lines of the lower edge of the sliding plateAnd a reference lineAt the i-th0A starting point on the bar auxiliary line;
step C12, searching the intersection point of the next auxiliary line and the lower edge reference line and the reference line according to the constraint relation in the step C11Andaccording to the constraint condition that the lower edge datum line is approximately parallel to the reference line, if the constraint condition is met, the lower edge datum line is approximately parallel to the reference lineRespectively the reference point and the lower edge of the sliding plate on the auxiliary line, if not, the step is transferred to the step C13, and the step C12 is repeated until the ith step0All the auxiliary lines after the bar are searched, and the step C14 is carried out;
step C13, utilizing the slope between the lower edge reference points on the first two auxiliary lines to perform straight line fitting at the lower edge reference point on the previous auxiliary line, wherein the intersection point of the fitted straight line and the current auxiliary line is the auxiliary lineLower edge datum pointUtilizing the slope between the lower edge reference points on the first two auxiliary lines to perform straight line fitting at the lower edge reference point on the previous auxiliary line, wherein the intersection point of the fitted straight line and the current auxiliary line is the lower edge reference point on the auxiliary line
Step C14, lower edge reference point set on all auxiliary lines obtained from steps C11, C12 and C13And reference point setThe reference line of the lower edge of the sliding plate can be determinedAnd a reference line
5. The method for extracting and identifying the sub-pixel edge of the pantograph pan according to claim 4, wherein the constraint relation of the thickness between the reference line of the lower edge of the pan and the reference line in the step C11 is:
wherein ,denotes the ith0The intersection point of the strip auxiliary line and the reference line of the lower edge of the sliding plate,is the ith0Intersection point, n, of the strip auxiliary line and the slide plate lower edge reference lineyThe gradient direction component of the intersection point in the vertical direction is represented,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,is the pixel row coordinate of the starting point of the lower edge reference line in the vertical direction,indicating the initial number of pixels between the reference line and the lower edge reference line of the skateboard image obtained at different times, thick1 being the average value of the number of pixels between the reference line and the lower edge reference line of the skateboard, thick1 beingIs determined.
6. The method for extracting and identifying the sub-pixel edge of the pantograph pan according to claim 4, wherein the constraint condition that the reference line of the lower edge of the pantograph pan is approximately parallel to the reference line in the step C12 is as follows:
wherein ,the edge slope between the reference line of the lower edge of the skateboard and the corresponding intersection point of the current auxiliary line and the previous auxiliary line,the slope of the edge between the reference line of the lower edge of the sliding plate and the corresponding intersection point of the current auxiliary line and the previous auxiliary line is theta, and theta is a threshold value of the difference value of the two slopes.
7. The pantograph pan sub-pixel edge extraction and identification method according to claim 3, wherein the step C2 comprises:
step C21, according to the thickness constraint condition between the upper edge and the lower edge of the two ends of the skateboard image, and according to the extraction result of the reference point of the lower edge of the skateboard in the step C1, in the ith step0Searching intersection points meeting constraint conditions on auxiliary lineAs a reference line of the upper edge of the skateboardAt the i-th0A starting point on the bar auxiliary line;
step C22, searching the intersection point of the next auxiliary line and the slide plate upper edge reference line according to the constraint relation in the step C21Using the constraint condition of the characteristic that the difference of the pixel numbers between the upper edge datum line and the lower edge datum line of the sliding plate of the adjacent auxiliary lines is not large, if the constraint condition is met, thenIf the reference point is not the reference point of the upper edge of the slide plate on the auxiliary line, the step C23 is executed, and the step C22 is repeated until the reference point is reached to the ith0All the auxiliary lines after the bar are searched, and the step C24 is carried out;
step C23, according to the number d of pixels between the upper and lower edge reference lines of the slide plate on the previous auxiliary linei-1And the vertical coordinate of the intersection point of the current auxiliary line and the lower edge reference lineFitting the upper edge point reference point of the current auxiliary line wherein
Step C24, upper edge reference point set on all auxiliary lines obtained from steps C21, C22 and C23Can determine the reference line of the upper edge of the sliding plate
8. The method for extracting and identifying the sub-pixel edge of the pantograph pan according to claim 7, wherein the thickness constraint between the upper and lower edge lines at the two ends of the pan in the step C21 is as follows:
wherein ,d0The number of pixels between the upper and lower edges at the two ends of the slide,is the ith0The number of intersections of the strip auxiliary lines with all the edges of the image,is the ith to be determined0The reference point of the upper edge of the slide plate on the strip auxiliary line,is the ith0Auxiliary barThe lower edge reference point of the sliding plate on the assistant line.
9. The method for extracting and identifying the subpixel edge of a pantograph pan according to claim 7, wherein the constraint condition that the number of pixels between the upper edge reference line and the lower edge reference line of the pan in the step C22 is as follows:
wherein ,diThe number of pixels between the upper edge reference line and the lower edge reference line of the slide plate on the current auxiliary line, di-1And sigma is the difference constraint of the number of pixels surrounded on the current auxiliary line and the number of pixels surrounded on the previous auxiliary line.
10. The method for extracting and identifying the subpixel edge of the pantograph pan according to claim 1, wherein in the step C, in a shadow area of an intersection part of the pan and the catenary, a RANSAC method is used for fitting an upper edge point of the shadow left part by using a determined point on an upper edge line of the pan on the left side of a contact line projection area, and a difference value is calculated for a reference point of the shadow area of the pan, so that the positioning of the upper edge point of the pan on the contact line projection area is realized.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910831149.7A CN110619337B (en) | 2019-09-06 | 2019-09-06 | Sub-pixel edge extraction and identification method of pantograph slide plate |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910831149.7A CN110619337B (en) | 2019-09-06 | 2019-09-06 | Sub-pixel edge extraction and identification method of pantograph slide plate |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110619337A true CN110619337A (en) | 2019-12-27 |
| CN110619337B CN110619337B (en) | 2023-04-28 |
Family
ID=68922200
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910831149.7A Active CN110619337B (en) | 2019-09-06 | 2019-09-06 | Sub-pixel edge extraction and identification method of pantograph slide plate |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110619337B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111238393A (en) * | 2020-01-20 | 2020-06-05 | 成都铁安科技有限责任公司 | Pantograph carbon slide plate detecting system and its control method |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001134771A (en) * | 1999-11-04 | 2001-05-18 | Honda Motor Co Ltd | Object recognizing device |
| CN108801134A (en) * | 2018-05-07 | 2018-11-13 | 南京理工大学 | A kind of abrasion of pantograph pan monitoring device and method based on sub-pixel edge detection |
-
2019
- 2019-09-06 CN CN201910831149.7A patent/CN110619337B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001134771A (en) * | 1999-11-04 | 2001-05-18 | Honda Motor Co Ltd | Object recognizing device |
| CN108801134A (en) * | 2018-05-07 | 2018-11-13 | 南京理工大学 | A kind of abrasion of pantograph pan monitoring device and method based on sub-pixel edge detection |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111238393A (en) * | 2020-01-20 | 2020-06-05 | 成都铁安科技有限责任公司 | Pantograph carbon slide plate detecting system and its control method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110619337B (en) | 2023-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102298778B (en) | Estimation system, estimation method, and estimation program for estimating object state | |
| CN105674880B (en) | Contact net geometric parameter measurement method and system based on binocular principle | |
| CN109584238B (en) | Stereoscopic vision-based bow net running state online detection system and method | |
| CN107703513B (en) | Non-contact net relative position detection method based on image processing | |
| CN109635737B (en) | Auxiliary vehicle navigation positioning method based on road marking line visual identification | |
| CN106996748A (en) | A kind of wheel footpath measuring method based on binocular vision | |
| CN109523583B (en) | Infrared and visible light image registration method for power equipment based on feedback mechanism | |
| CN109300125B (en) | Method for extracting pantograph slide plate image of urban rail train in complex environment | |
| CN109740609A (en) | A kind of gauge detection method and device | |
| CN115482195B (en) | Train part deformation detection method based on three-dimensional point cloud | |
| CN112950532B (en) | Train pantograph state detection method | |
| CN109580137A (en) | A kind of bridge structure displacement influence line measurement method based on computer vision technique | |
| CN112261390B (en) | Vehicle-mounted camera equipment and image optimization device and method thereof | |
| CN112785654A (en) | Calibration method and device for track geometry detection system | |
| CN108765393B (en) | Method for detecting vibration behavior of high-speed railway contact net | |
| Lu et al. | Automatic wear measurement of pantograph slider based on multiview analysis | |
| CN111414931A (en) | Multi-branch multi-scale small target detection method based on image depth | |
| CN113223050A (en) | Robot motion track real-time acquisition method based on Aruco code | |
| CN111563896A (en) | Image processing method for catenary anomaly detection | |
| CN115931874A (en) | Carrying type magnetic suspension intelligent dynamic inspection system and gap width detection method | |
| CN110619337A (en) | Method for extracting and identifying sub-pixel edge of pantograph slide plate | |
| CN113269729B (en) | Assembly body multi-view detection method and system based on depth image contrast | |
| CN117474873B (en) | Surface treatment system before brazing of high-chromium wear-resistant castings | |
| CN101685000B (en) | Computer system and method for image boundary scan | |
| CN110717910B (en) | CT image target detection method based on convolutional neural network and CT scanner |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |