CN114119555B - Large-caliber element edge detection method based on object distance focusing method - Google Patents
Large-caliber element edge detection method based on object distance focusing method Download PDFInfo
- Publication number
- CN114119555B CN114119555B CN202111428157.0A CN202111428157A CN114119555B CN 114119555 B CN114119555 B CN 114119555B CN 202111428157 A CN202111428157 A CN 202111428157A CN 114119555 B CN114119555 B CN 114119555B
- Authority
- CN
- China
- Prior art keywords
- edge
- image
- camera
- value
- focusing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000003708 edge detection Methods 0.000 title claims abstract description 14
- 230000008859 change Effects 0.000 claims abstract description 51
- 238000012545 processing Methods 0.000 claims abstract description 20
- 230000000007 visual effect Effects 0.000 claims abstract description 11
- 230000008569 process Effects 0.000 claims description 12
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 description 19
- 230000003287 optical effect Effects 0.000 description 7
- 230000008439 repair process Effects 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000009434 installation Methods 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 230000008901 benefit Effects 0.000 description 2
- FFBHFFJDDLITSX-UHFFFAOYSA-N benzyl N-[2-hydroxy-4-(3-oxomorpholin-4-yl)phenyl]carbamate Chemical compound OC1=C(NC(=O)OCC2=CC=CC=C2)C=CC(=C1)N1CCOCC1=O FFBHFFJDDLITSX-UHFFFAOYSA-N 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- VYPSYNLAJGMNEJ-UHFFFAOYSA-N Silicium dioxide Chemical compound O=[Si]=O VYPSYNLAJGMNEJ-UHFFFAOYSA-N 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003090 exacerbative effect Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 239000005350 fused silica glass Substances 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 238000011900 installation process Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000000691 measurement method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/13—Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10056—Microscopic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Landscapes
- Engineering & Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
Abstract
A large-caliber element edge detection method based on an object distance focusing method relates to the technical field of engineering optics and is used for solving the problem that a globally clear focusing position cannot be obtained before an image is acquired in the prior art. The technical key points of the invention include: respectively moving a plurality of edges of the element into the visual field range of the camera, changing the object distance, and acquiring a plurality of images corresponding to each edge under different focal planes; automatically and clearly focusing each edge according to variance change curves of a plurality of images corresponding to each edge; after focusing is completed, acquiring a plurality of images containing each edge, and processing the plurality of images so as to acquire the positions of the plurality of edges; the edge automatic focusing strategy is designed to automatically focus according to the variance change curve of the image, so that the acquired edge image is clearer, and the coordinate position of the element edge can be acquired more accurately. The method is easy to realize automation and can be used for edge detection of large-caliber elements.
Description
Technical Field
The invention relates to the technical field of engineering optics, in particular to a large-caliber element edge detection method based on an object distance focusing method.
Background
The construction of high power solid state laser devices requires a large number of optical elements that are susceptible to damage to the surface in a strong laser environment, which can weaken the material properties and further exacerbate the damage process. Studies have shown that if damage is not repaired or inhibited in time, the damage size of the front and rear surfaces of the component will increase linearly and exponentially, respectively, under laser irradiation. In addition to further exacerbating the damage to the element itself, the damage also degrades the quality of the beam passing through the element, affecting the focal spot quality, modulating the light field to form a strong region, causing damage to downstream elements, which is a process of a vicious circle. In order to improve the service life of the element and reduce the maintenance cost of the device, a CO 2 laser repair method is often adopted in engineering to repair the damage on the surface of the fused quartz element, and the method heals the damaged cracks through the thermal effect of laser so as to improve the damage threshold of the element.
The laser repair of the large-caliber optical element is carried out on a detection repair machine tool, and the element is required to be put down and mounted on the machine tool during repair. Because the clamping accuracy is limited, the surface position of the element after installation is uncertain, and the pose of the element needs to be redetermined. Calculating the geometric center coordinates of the component by acquiring the edge positions is an important element in the automated determination of the component surface positions. To automatically obtain the exact edge position, a clear image of the element edge needs to be acquired and the edge is extracted by image processing. In order to improve the detection accuracy of the edge position, a microscopic camera is used for image acquisition. Because of the limited precision of assembly adjustment during installation, the focusing positions of the elements are inconsistent every time, and automatic focusing is needed.
The autofocus method is active and passive. The active focusing realizes automatic focusing by measuring the distance between the lens and the object, and when the measured distance exceeds the depth of field range, the movement mechanism is controlled to adjust the focusing position so as to obtain a clear image. The distance measurement accuracy of the method is limited, the depth of field of a microscopic imaging system is small, and the distance measurement method is difficult to meet the use requirement of microscopic detection. Therefore, in engineering, a passive method is generally adopted to perform automatic focusing on a microscopic image, and the method obtains the position with the highest definition as a focusing position to realize automatic focusing by performing definition evaluation on images with different focusing positions. However, the depth of field of the microscopic camera is far smaller than the depth of the element chamfer in the depth of field direction, and the element edge image can only be clearly imaged in the depth of field range near the focusing plane, so that the microscopic images of different focusing positions can only be locally clear, and the focusing position with clear edge can not be obtained through global definition evaluation.
Disclosure of Invention
In view of the above problems, the invention provides a large-caliber element edge detection method based on an object distance focusing method, which is used for solving the problem that a globally clear focusing position cannot be obtained before an image is acquired in the prior art.
A large-caliber element edge detection method based on an object distance focusing method comprises the following steps:
step one, respectively moving a plurality of edges of an element into a camera visual field range, changing object distances, and acquiring a plurality of images corresponding to each edge under different focal planes; automatically and clearly focusing each edge according to variance change curves of a plurality of images corresponding to each edge;
and step two, after focusing is completed, collecting a plurality of images containing each edge, and processing the images to obtain the positions of the edges.
Further, the process of acquiring a plurality of images corresponding to each edge in different focal planes in the first step includes: the two search steps are set so that the camera moves close to the edge of the element along the positive direction or the negative direction of the Z axis of the coordinate system of the machine tool according to the search steps, and a plurality of images under different focal planes are acquired.
Further, the cross section of the element is a regular rectangle, and the plurality of edges of the element comprise a left edge, a right edge, an upper edge and a lower edge.
Further, the process of automatically and clearly focusing each edge according to the variance change curves of the plurality of images corresponding to each edge in the first step includes:
Dividing each acquired image into a plurality of sub-region images, wherein the sub-region images have consistent focusing states;
step two, calculating the gray variance value of each sub-area image;
Drawing and obtaining a horizontal direction variance change curve corresponding to each image according to the gray variance value of each sub-area image;
and step four, performing automatic focusing according to an edge automatic focusing strategy according to the variance change curve in the horizontal direction.
Further, in step two, the gray variance value of each sub-area image x is calculated according to the following formula:
Wherein, I (I, j) represents the gray value of the pixel point (I, j); μ represents the average gray value of the sub-region image.
Further, the specific steps of the step one four include:
Step four, firstly, a camera is moved to be close to the edge of the element along the positive direction or the negative direction of the Z axis of a machine tool coordinate system according to a searching step distance s 1, a horizontal direction variance change curve is obtained by collecting images according to the steps one to three, for the image collected under one focal plane, the corresponding horizontal direction variance change curve comprises a fluctuation starting point O and one or more peaks, the gray level variance value corresponding to the camera when focusing on the element chamfering area is set as a peak value I, and the gray level variance value corresponding to the camera when focusing on the boundary position of the element chamfering area and the element surface area is set as a peak value II:
a1 When the value of the peak value II is larger than a preset threshold value t 1, judging whether the difference between the position of the peak value I and the position of the fluctuation starting point O in the variance change curve is smaller than a preset threshold value t 2 or not;
a11 If smaller, calculate the slope between the fluctuation starting point O and the peak value i: if the slope is greater than the preset threshold t 3, the slope is saved and is recorded as k OI; if the slope is smaller than a preset threshold t 3, enabling the camera to move close to the edge of the element along the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and recalculating the slope between the fluctuation starting point O and the peak value I until the slope is larger than the preset threshold t 3; executing the first step and the second step;
a12 If not, enabling the camera to move close to the edge of the element along the positive direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and judging whether the difference between the position of the peak value I in the variance change curve and the position of the fluctuation starting point O is smaller than a preset threshold t 2 or not again, if so, executing a 11); if not, repeatedly executing a 12);
a2 When the value of the peak value II is not greater than the preset threshold value t 1, comparing the value of the current peak value II with the value of the peak value II obtained by collecting images after the camera is moved last time: if the value of the current peak value II is greater than the value of the peak value II obtained by acquiring an image after the camera is moved last time, the camera is moved close to the edge of the element along the last moving direction according to the searching step s 1; otherwise, moving the element to be close to the edge of the element along the direction opposite to the previous moving direction; and acquiring images, and repeatedly executing a 1) according to the steps one to three to obtain a horizontal direction variance change curve;
Step one, four, then enabling the camera to move close to the edge of the element along the positive direction or the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 2, acquiring images, and obtaining a horizontal direction variance change curve according to the steps one to three: calculating the slope between the fluctuation starting point O and the peak value I, and judging whether the current slope is larger than the slope k OI stored in a 11): if the gradient is larger than the preset gradient, the camera is moved to be close to the edge of the element along the positive direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the gradient k OI is not increased any more; if not, the camera is moved to be close to the edge of the element along the negative direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the slope k OI is not increased any more;
auto-focusing is accomplished according to a step one four to step one four edge auto-focusing strategy, wherein the search step s 1 is greater than the search step s 2.
Further, in the fourth step, the calculation formula of the slope between the fluctuation starting point O and the peak value i is:
In the formula, (n 0,T0)、(n1,T1) represents the coordinates of the fluctuation starting point O and the peak value I in the horizontal direction variance change curve, respectively.
Further, in the second step, a plurality of images including each edge are acquired, and the specific steps of processing the plurality of images include: for each image, firstly, convolving the image with a Sobel operator to obtain a gradient image in the horizontal direction of the image; then presetting a first fixed threshold value, and carrying out binarization processing on the gradient image to obtain a binarized image; then presetting a second fixed threshold value, counting the number of pixels with the pixel value of 255 in each column in the binarized image, and determining the column as a pixel column with the edge when the number of pixels exceeds the preset second fixed threshold value, namely determining an edge line; then, the distance between the edge line and the center line of the image is calculated, and the element edge position is obtained according to the distance calculation.
Further, the plurality of edge positions acquired in the second step include:
The left edge midpoint is at the machine coordinate system at X' L:
X'L＝XL+kpixelΔXL
Wherein X L is the X-axis coordinate when the midpoint of the left edge calibrated in advance moves to the center of the field of view of the camera; k pixel is the actual size represented by a single pixel in the calibrated image; Δx L is the pixel distance between the midpoint of the left edge and the image centerline;
The right edge midpoint is at the machine coordinate system at the X-axis coordinate X' R:
X'R＝XR+kpixelΔXR
wherein X R is the X-axis coordinate when the midpoint of the right edge calibrated in advance moves to the center of the field of view of the camera; Δx R is the pixel distance between the midpoint of the right edge and the image centerline;
The Y-axis coordinate Y' T of the midpoint of the upper edge in the machine coordinate system is:
Y′T＝YT+kpixelΔYT
Wherein Y T is a Y-axis coordinate when a midpoint of the pre-calibrated upper edge moves to the center of the field of view of the camera; ΔY T is the pixel distance between the midpoint of the upper edge and the image centerline;
the lower edge midpoint is at machine coordinate system Y' D:
Y′D＝YD+kpixelΔYD
Wherein Y D is a Y-axis coordinate when a midpoint of a pre-calibrated lower edge moves to the center of a camera visual field; ΔY D is the pixel distance between the midpoint of the lower edge and the image centerline.
The beneficial technical effects of the invention are as follows:
the invention adopts an object distance focusing method based on an image variance change curve to realize the automatic focusing of the element edge and obtain a clear image of the edge; the accurate position of the element edge is obtained by processing the edge clear image; the method realizes high-precision detection of the edges of the large-caliber elements, is easy to realize automation, and can be used for the automatic determination process of the pose of the elements.
Drawings
The invention may be better understood by reference to the following description taken in conjunction with the accompanying drawings, which are included to provide a further illustration of the preferred embodiments of the invention and to explain the principles and advantages of the invention, together with the detailed description below.
FIG. 1 is a schematic diagram of a large-caliber element edge detection device according to an embodiment of the invention;
FIG. 2 is a schematic diagram of edge image sub-region division in an embodiment of the present invention;
FIG. 3 is a graph of variance variation of images at different focal plane positions in an embodiment of the invention;
FIG. 4 is a flow chart of edge autofocus in an embodiment of the invention;
FIG. 5 is a schematic diagram of the result of auto-focusing the edges of a component in an embodiment of the invention; wherein, figure (a) is the left edge; fig. (b) is right edge; figure (c) is an upper edge; fig. (d) is a lower edge;
FIG. 6 is a schematic diagram of the result of detecting the edge of a component according to an embodiment of the present invention; wherein, figure (a) is an image acquired after edge focusing is clear; fig. (b) is a gradient binary image after image processing; fig. (c) is an edge detection result diagram.
Detailed Description
In order that those skilled in the art will better understand the present invention, exemplary embodiments or examples of the present invention will be described below with reference to the accompanying drawings. It is apparent that the described embodiments or examples are only implementations or examples of a part of the invention, not all. All other embodiments or examples, which may be made by one of ordinary skill in the art without undue burden, are intended to be within the scope of the present invention based on the embodiments or examples herein.
The object distance focusing method based on the image gray variance change curve provided by the invention realizes automatic focusing of the element edge position, and the edge contour position is obtained by processing the edge clear image, so that the high-precision automatic detection of the element edge position is realized.
The embodiment of the invention provides a large-caliber element edge detection method based on an object distance focusing method, which comprises the following steps: step one, respectively moving a plurality of edges of an element into a camera visual field range, changing object distances, and acquiring a plurality of images corresponding to each edge under different focal planes; automatically and clearly focusing each edge according to variance change curves of a plurality of images corresponding to each edge; and step two, after focusing is completed, collecting a plurality of images containing each edge, and processing the images to obtain the positions of the edges.
In this embodiment, optionally, the acquiring the plurality of images corresponding to each edge in the different focal planes in the step one includes: the two search steps are set so that the camera moves close to the edge of the element along the positive direction or the negative direction of the Z axis of the coordinate system of the machine tool according to the search steps, and a plurality of images under different focal planes are acquired.
In this embodiment, the cross section of the element is optionally regular rectangular, and the plurality of edges of the element include a left edge, a right edge, an upper edge, and a lower edge.
In this embodiment, optionally, the automatically and clearly focusing each edge according to the variance curves of the multiple images corresponding to each edge in the first step includes:
Dividing each acquired image into a plurality of sub-region images, wherein the sub-region images have consistent focusing states;
step two, calculating the gray variance value of each sub-area image;
Drawing and obtaining a horizontal direction variance change curve corresponding to each image according to the gray variance value of each sub-area image;
and step four, performing automatic focusing according to an edge automatic focusing strategy according to the variance change curve in the horizontal direction.
In this embodiment, optionally, in step two, the gray variance value of each sub-area image x is calculated according to the following formula:
Wherein, I (I, j) represents the gray value of the pixel point (I, j); μ represents the average gray value of the sub-region image.
In this embodiment, optionally, the specific steps of the step one four include:
Step four, firstly, a camera is moved to be close to the edge of the element along the positive direction or the negative direction of the Z axis of a machine tool coordinate system according to a searching step distance s 1, a horizontal direction variance change curve is obtained by collecting images according to the steps one to three, for the image collected under one focal plane, the corresponding horizontal direction variance change curve comprises a fluctuation starting point O and one or more peaks, the gray level variance value corresponding to the camera when focusing on the element chamfering area is set as a peak value I, and the gray level variance value corresponding to the camera when focusing on the boundary position of the element chamfering area and the element surface area is set as a peak value II:
a1 When the value of the peak value II is larger than a preset threshold value t 1, judging whether the difference between the position of the peak value I and the position of the fluctuation starting point O in the variance change curve is smaller than a preset threshold value t 2 or not;
a11 If smaller, calculate the slope between the fluctuation starting point O and the peak value i: if the slope is greater than the preset threshold t 3, the slope is saved and is recorded as k OI; if the slope is smaller than a preset threshold t 3, enabling the camera to move close to the edge of the element along the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and recalculating the slope between the fluctuation starting point O and the peak value I until the slope is larger than the preset threshold t 3; executing the first step and the second step;
a12 If not, enabling the camera to move close to the edge of the element along the positive direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and judging whether the difference between the position of the peak value I in the variance change curve and the position of the fluctuation starting point O is smaller than a preset threshold t 2 or not again, if so, executing a 11); if not, repeatedly executing a 12);
a2 When the value of the peak value II is not greater than the preset threshold value t 1, comparing the value of the current peak value II with the value of the peak value II obtained by collecting images after the camera is moved last time: if the value of the current peak value II is greater than the value of the peak value II obtained by acquiring an image after the camera is moved last time, the camera is moved close to the edge of the element along the last moving direction according to the searching step s 1; otherwise, moving the element to be close to the edge of the element along the direction opposite to the previous moving direction; and acquiring images, and repeatedly executing a 1) according to the steps one to three to obtain a horizontal direction variance change curve;
Step one, four, then enabling the camera to move close to the edge of the element along the positive direction or the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 2, acquiring images, and obtaining a horizontal direction variance change curve according to the steps one to three: calculating the slope between the fluctuation starting point O and the peak value I, and judging whether the current slope is larger than the slope k OI stored in a 11): if the gradient is larger than the preset gradient, the camera is moved to be close to the edge of the element along the positive direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the gradient k OI is not increased any more; if not, the camera is moved to be close to the edge of the element along the negative direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the slope k OI is not increased any more;
auto-focusing is accomplished according to a step one four to step one four edge auto-focusing strategy, wherein the search step s 1 is greater than the search step s 2.
In this embodiment, optionally, in the fourth step, the calculation formula of the slope between the starting point O and the peak value i is:
In the formula, (n 0,T0)、(n1,T1) represents the coordinates of the fluctuation starting point O and the peak value I in the horizontal direction variance change curve, respectively.
In this embodiment, optionally, in the second step, a plurality of images including each edge are acquired, and the specific step of processing the plurality of images includes: for each image, firstly, convolving the image with a Sobel operator to obtain a gradient image in the horizontal direction of the image; then presetting a first fixed threshold value, and carrying out binarization processing on the gradient image to obtain a binarized image; then presetting a second fixed threshold value, counting the number of pixels with the pixel value of 255 in each column in the binarized image, and determining the column as a pixel column with the edge when the number of pixels exceeds the preset second fixed threshold value, namely determining an edge line; then, the distance between the edge line and the center line of the image is calculated, and the element edge position is obtained according to the distance calculation.
In this embodiment, optionally, the plurality of edge positions acquired in the second step include:
the left edge midpoint is at the machine coordinate system at X' L:
X′L＝XL+kpixelΔXL
Wherein X L is the X-axis coordinate when the midpoint of the left edge calibrated in advance moves to the center of the field of view of the camera; k pixel is the actual size represented by a single pixel in the calibrated image; Δx L is the pixel distance between the midpoint of the left edge and the image centerline;
The right edge midpoint is at the machine coordinate system at the X-axis coordinate X' R:
X'R＝XR+kpixelΔXR
wherein X R is the X-axis coordinate when the midpoint of the right edge calibrated in advance moves to the center of the field of view of the camera; Δx R is the pixel distance between the midpoint of the right edge and the image centerline;
The Y-axis coordinate Y' T of the midpoint of the upper edge in the machine coordinate system is:
Y′T＝YT+kpixelΔYT
Wherein Y T is a Y-axis coordinate when a midpoint of the pre-calibrated upper edge moves to the center of the field of view of the camera; ΔY T is the pixel distance between the midpoint of the upper edge and the image centerline;
the lower edge midpoint is at machine coordinate system Y' D:
Y′D＝YD+kpixelΔYD
Wherein Y D is a Y-axis coordinate when a midpoint of a pre-calibrated lower edge moves to the center of a camera visual field; ΔY D is the pixel distance between the midpoint of the lower edge and the image centerline.
In another embodiment of the present invention, as shown in fig. 1, a method for detecting an edge of a large-caliber element is provided, where the detecting device is composed of a motion platform and a microscopic detecting system. The motion platform comprises X, Y, Z motion axes, and the motion directions of the motion axes are respectively consistent with the directions of X, Y, Z coordinate axes of a machine tool coordinate system; the motion platform can carry an optical large-caliber element to realize the movement along the X, Y axis direction, and carry a microscopic detection system to realize the movement along the Z axis direction. The microscopic detection system consists of an area array CCD camera, a variable-focus microscope lens, a coaxial light source and an annular light source; the resolution of the area array CCD camera is 2456 multiplied by 2056, the detection range is 1.5mm multiplied by 1.3mm, and the detection precision is 0.63 mu m/pixel. During detection, firstly, four edges of the element are moved to a microscopic detection visual field range one by one according to coordinates calibrated in advance, automatic focusing is performed by controlling a microscopic detection system to move along a Z axis to change object distance, and finally, acquired edge images are processed to obtain edge positions. The method comprises the following specific steps:
Step1, initializing a repair platform to finish the installation of an optical element;
according to the embodiment of the invention, the positioning precision of the motion platform is +/-10 mu m, and comprises X, Y, Z motion axes, the platform can realize the two-dimensional high-precision movement of the optical element by controlling the X, Y axis, and the object distance adjustment of the microscopic detection system is realized by controlling the Z axis.
Step 2, controlling a motion platform to move along X, Y, Z axes according to pre-calibrated coordinates, moving the edge of the optical element to a microscopic visual field range, and automatically focusing a plurality of edges of the element according to a variance change curve of an image; wherein the plurality of edges includes a left edge, a right edge, an upper edge, and a lower edge;
According to the embodiment of the invention, the microscopic camera is not focused at the edge position due to deflection of the element in the installation process, and the microscopic camera is required to be automatically focused for improving the edge detection precision. The detection range of a camera of the microscopic detection system is far larger than the element installation error, so that the edge to be detected can be ensured to enter a microscopic field through the position coordinates calibrated in advance. The illumination mode adopted by the microscopic detection system is superposition illumination of the coaxial light source and the annular light source, the coaxial light source can improve the surface brightness of the element, so that the element can be conveniently distinguished from the background, and the annular light source can reflect the fine characteristics of the chamfer area, so that the camera can conveniently focus on the edge.
The adopted method is an object distance focusing method based on an image gray level variance change curve. The gray variance curves of the horizontal or vertical direction of the image have different characteristics at different focus positions. With these features, auto-focusing of the edge is achieved by adjusting the focal plane by controlling the stage Z-axis motion to change the object distance. Taking the left edge as an example, the image including the left edge acquired by the microscope camera is divided into 245 subregions as shown in fig. 2, the subregions are 10 pixels×2056 pixels in size, the subregions are far smaller than the depth of field of the microscope camera, and the images in the regions can be considered to have consistent focusing states. The image gray scale variance value of each sub-region is calculated by the following equation (1), and a variance change curve in the horizontal direction of the image is drawn.
Wherein x=1, 2, …,245 represents a subregion; i (I, j) represents the gray value of the pixel point (I, j); μ represents the average of gray values of the small rectangular area (sub-area).
Fig. 3 is a graph showing variance variation of images collected by cameras in different focal planes in a horizontal direction, wherein red lines in the graph represent positions of the focal planes, horizontal coordinates represent positions of the images in the horizontal direction, and vertical coordinates represent gray variances. The difference between the gray values of the images of the chamfer and the element surface is larger, and a larger peak value (peak value II) appears on the gray value variance curve of the boundary position of the two areas, and the value can be used for judging whether the focal plane enters the vicinity of the microscopic image area. As shown in fig. 3 (a) and (g), when the focal plane is far from the edge, the variance is small except for the peak ii; as shown in (a) to (b) and (g) to (f) of fig. 3, when the focal plane is gradually approaching the edge, the peak value ii gradually increases, and the variation of the chamfer area variance gradually becomes remarkable; as can be seen from comparing (f) to (c) in fig. 3, as the focal plane gradually approaches the best focus position along the Z-axis, the position of the peak i gradually approaches the element edge; as can be seen from comparing fig. 3 (b) to (c), as the focal plane gradually approaches the best focus position along the negative Z-axis direction, the peak value i in the curve gradually increases, and the peak value reaches the maximum when the focal plane reaches the best focus position.
Based on the above characteristics, the present invention designs an edge autofocus strategy as shown in fig. 4, wherein (n 0,T0)、(n1,T1)、(n2,T2) represents the coordinates of the fluctuation starting point O, the curve peak value i, and the curve peak value ii of the variance change curve in the variance curve, respectively; t 1、t2、t3 is a preset threshold, t 1 is related to the peak value II when the microscope camera focal plane enters the camera field of view, which is set to bring the focus position close to the edge, providing enough information for subsequent focusing; t 2 relates to the chamfer area width, t 3 relates to the slope value peak k OI when the camera is focused at the edge position, both values being for bringing the focal plane further towards the edge; the preset threshold values are set according to experience values of multiple experiments; s 1、s2 is the searching pace moving along the Z axis, s 1 takes a larger pace to increase the searching speed to prevent the local extreme point from being trapped, and s 2 takes a smaller pace to increase the searching precision. The strategy is divided into ①～④ parts:
① The focal plane is moved to the vicinity of the edge, so that the gray variance change curve can provide enough information for automatic focusing, and the specific steps are as follows:
A) The camera collects the edge image and calculates an image gray variance change curve, a value T 2-now of a peak value II is obtained according to the curve, if the value is larger than a threshold value T 1, the process ① is completed, otherwise, the step B is entered;
B) If T 2-now>T2-pre, let s=s 1, otherwise let s= -s 1. And controlling the Z-axis movement step length s to enable T 2-pre＝T2-now, collecting an edge image by the camera, calculating an image gray variance change curve, and obtaining a value T 2-now of a peak value II according to the curve. If T 2-now>t1, then process ① is completed, otherwise, repeat step B);
② When the focal plane gradually approaches the edge along the positive direction of the Z axis, the focal plane can be further moved to the vicinity of the edge by judging the position of the peak I. The method comprises the following specific steps: the camera collects the edge image and calculates the gray variance change curve of the image, and the position n 1 of the peak value I is obtained according to the curve. If n 1≤no+t2 then the process ② is completed, otherwise the control platform moves s 1 in the forward direction until n 1≤no+t2 is satisfied.
③ When the focal plane gradually approaches the edge along the negative Z-axis direction, the focal plane can be further moved to the vicinity of the edge by determining the slope between the point O and the peak I. The method comprises the following specific steps: the camera collects edge images and calculates an image gray variance change curve, and the slope between the point O and the peak value I is obtained according to the curve:
if k OI≥t3 then the process ③ is completed, otherwise the control platform moves s 1 in the negative Z-axis direction until k OI≥t3 is satisfied.
④ The best focus position is found by determining the slope change between point O and peak i, and the camera can be considered to be focused on the edge of the optical element when the slope between point O and peak i reaches a maximum. The method comprises the following specific steps:
A) The slope between point O and peak I at the end of step ③ is noted as k OI-pre and step B) is entered;
B) And controlling the Z-axis movement step length s 2, acquiring an edge image by the obtained camera, calculating an image gray variance change curve, and obtaining a slope k OI-now between the point O and the peak value I according to the curve. Let s=s 2 if k OI-now>kOI-pre, otherwise let s= -s 2, let k OI-pre＝kOI-now and go to step C);
C) And controlling the Z axis to move in a step length s, controlling the camera to acquire an edge image, calculating an image gray variance change curve, and obtaining a slope k OI-now between the point O and the peak value I according to the curve. If T 2-now>t1 or k OI-now≤kOI-pre, then auto-focusing is completed, otherwise let k OI-pre＝kOI-now and repeat step C).
And 3, after focusing is finished, controlling a camera to acquire microscopic images with clear edges, and processing the images to acquire edge positions.
According to the embodiment of the invention, the position of the edge in the image is obtained by adopting an edge extraction algorithm based on image gradients, the method utilizes gradients to describe gray level mutation of the edge position of the image, the contour of the edge can be obtained by carrying out binarization processing on an edge gradient map, and the machine tool coordinates of the edge can be calculated according to the contour position, so that the edge detection of the large-caliber element is completed.
After obtaining a sharp image of the edge, the distance of the edge from the microscopic center needs to be calculated. Still taking the left edge as an example, since the edge is an approximately vertical straight line in the image, there is a sudden change in gray value in the direction perpendicular to the edge, the left edge is detected by using the Sobel operator in the horizontal direction as shown in the following formula:
Gx＝Sobelx*I (3)
In the formula, sobel x represents a horizontal Sobel operator; i represents an original image; g x represents a gradient image in the horizontal direction, which is obtained by planar convolution of Sobel x with the original image I.
In order to facilitate subsequent processing, the gradient image G x is subjected to binarization processing of a fixed threshold T shown in formula (5) to obtain an image I B, and at the same time, morphological open operation is performed on the image I B to eliminate noise generated by gray value fluctuation, so as to obtain a final binarization result diagram.
Counting the number of pixels with the pixel value of 255 in each column of the binarized image from left to right, and recognizing the column as the pixel column with the edge when the number of pixels exceeds a set threshold for the first time, wherein the threshold is half of the number of pixels in the column in actual use, namely 1028. The distance Δx L of the edge from the center of the microscopic field is calculated and the element left edge position X' L is obtained using equation (6).
X'L＝XL+kpixelΔXL (6)
Wherein X L is a machine tool coordinate of a pre-calibrated left edge moving to the center of a visual field, and the value can be calibrated by manually moving the left edge to the center of the visual field of a microscopic camera and reading a grating ruler on a machine tool; k pixel is the actual size represented by a single pixel in the calibrated image, and the coefficient can be obtained by calibrating a standard scale plate.
By adopting the process, the position coordinates of the upper edge, the lower edge, the left edge and the right edge of the element can be obtained, so that the edge detection of the large-caliber element is completed.
Another embodiment of the present invention provides an example analysis of a method for detecting edges of a large-caliber element based on an object distance focusing method, where the method is used to detect a batch of elements, and the caliber of the element is 430mm×430mm. The automatic detection of the element edge is realized by using autonomously developed 'large-caliber element surface defect automatic detection and repair control software', and the specific process is as follows:
(1) And carrying out zero return and error compensation on the motion platform and moving the motion platform to the installation station to complete the installation of the element.
(2) The four edges of the element are sequentially moved into microscopic view and the edges are auto-focused. Fig. 5 is an image of the element before and after focusing of four edges stored during an automatic edge finding process, with images of the area near the edges taken for comparison. As can be seen from the figure, the object distance focusing method based on the image variance change curve can achieve the acquisition of the clear-edge image.
(3) And processing the acquired sharp-edge image. Fig. 6 (a) is a clear image of the left edge acquired by the camera, the horizontal gradient map thereof is calculated, and the horizontal gradient map is binarized, and the processing result is shown in fig. 6 (b). Counting the pixel number of which the pixel value is 255 in each column, wherein the column which reaches the threshold value first is regarded as the column where the edge is located, and the red line in fig. 6 (b) and (c) is the detected edge line. The edge lines of other edges are obtained by adopting the same method, and the distance from the center of the microscopic vision is calculated, so that the result is that:
ΔXL,ΔXR,ΔYT,ΔYD＝0.38mm,0.36mm,0.27mm,0.27mm
(4) The accurate position of the element edge can be obtained by acquiring the machine tool coordinates when the camera collects the edge and superposing the deviation value on the corresponding machine tool coordinates, and the accurate position is as follows:
X′L,X′R,Y′T,Y′D＝-55.595mm,373.474mm,215.426mm,-213.623mm
The invention realizes high-precision detection of the edge of the large-caliber element through the process, and provides accurate position reference for the determination of the pose of the subsequent element.
While the invention has been described with respect to a limited number of embodiments, those skilled in the art, having benefit of the above description, will appreciate that other embodiments are contemplated within the scope of the invention as described herein. The disclosure of the present invention is intended to be illustrative, but not limiting, of the scope of the invention, which is defined by the appended claims.
Claims (8)
1. The large-caliber element edge detection method based on the object distance focusing method is characterized by comprising the following steps of:
Step one, respectively moving a plurality of edges of an element into a camera visual field range, changing object distances, and acquiring a plurality of images corresponding to each edge under different focal planes; automatically and clearly focusing each edge according to variance change curves of a plurality of images corresponding to each edge, wherein the automatic and clear focusing comprises the following steps:
Dividing each acquired image into a plurality of sub-region images, wherein the sub-region images have consistent focusing states;
step two, calculating the gray variance value of each sub-area image;
Drawing and obtaining a horizontal direction variance change curve corresponding to each image according to the gray variance value of each sub-area image;
Step four, automatic focusing is carried out according to an edge automatic focusing strategy according to a horizontal direction variance change curve;
and step two, after focusing is completed, collecting a plurality of images containing each edge, and processing the images to obtain the positions of the edges.
2. The method for detecting edges of a large-caliber element based on an object distance focusing method according to claim 1, wherein the process of acquiring a plurality of images corresponding to each edge in different focal planes in the first step comprises the following steps: the two search steps are set so that the camera moves close to the edge of the element along the positive direction or the negative direction of the Z axis of the coordinate system of the machine tool according to the search steps, and a plurality of images under different focal planes are acquired.
3. The method for detecting edges of a large-caliber element based on an object distance focusing method according to claim 2, wherein the cross section of the element is a regular rectangle, and the edges of the element comprise a left edge, a right edge, an upper edge and a lower edge.
4. The method for detecting edges of large-caliber elements based on an object distance focusing method according to claim 1, wherein in the step two, gray variance values of each sub-area image x are calculated according to the following formula:
Wherein, I (I, j) represents the gray value of the pixel point (I, j); μ represents the average gray value of the sub-region image.
5. The method for detecting edges of a large-caliber element based on an object distance focusing method as claimed in claim 4, wherein the specific steps of the step one four include:
Step four, firstly, a camera is moved to be close to the edge of the element along the positive direction or the negative direction of the Z axis of a machine tool coordinate system according to a searching step distance s 1, a horizontal direction variance change curve is obtained by collecting images according to the steps one to three, for the image collected under one focal plane, the corresponding horizontal direction variance change curve comprises a fluctuation starting point O and one or more peaks, the gray level variance value corresponding to the camera when focusing on the element chamfering area is set as a peak value I, and the gray level variance value corresponding to the camera when focusing on the boundary position of the element chamfering area and the element surface area is set as a peak value II:
a1 When the value of the peak value II is larger than a preset threshold value t 1, judging whether the difference between the position of the peak value I and the position of the fluctuation starting point O in the variance change curve is smaller than a preset threshold value t 2 or not;
a11 If smaller, calculate the slope between the fluctuation starting point O and the peak value i: if the slope is greater than the preset threshold t 3, the slope is saved and is recorded as k OI; if the slope is smaller than a preset threshold t 3, enabling the camera to move close to the edge of the element along the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and recalculating the slope between the fluctuation starting point O and the peak value I until the slope is larger than the preset threshold t 3; executing the first step and the second step;
a12 If not, enabling the camera to move close to the edge of the element along the positive direction of the Z axis of the machine tool coordinate system according to the searching step s 1, collecting images, and judging whether the difference between the position of the peak value I in the variance change curve and the position of the fluctuation starting point O is smaller than a preset threshold t 2 or not again, if so, executing a 11); if not, repeatedly executing a 12);
a2 When the value of the peak value II is not greater than the preset threshold value t 1, comparing the value of the current peak value II with the value of the peak value II obtained by collecting images after the camera is moved last time: if the value of the current peak value II is larger than the value of the peak value II obtained by acquiring an image after the camera is moved last time, the camera is moved to be close to the edge of the element along the last moving direction according to the searching step s 1, otherwise, the camera is moved to be close to the edge of the element along the direction opposite to the last moving direction; and acquiring images, and repeatedly executing a 1) according to the steps one to three to obtain a horizontal direction variance change curve;
Step one, four, then enabling the camera to move close to the edge of the element along the positive direction or the negative direction of the Z axis of the machine tool coordinate system according to the searching step s 2, acquiring images, and obtaining a horizontal direction variance change curve according to the steps one to three: calculating the slope between the fluctuation starting point O and the peak value I, and judging whether the current slope is larger than the slope k OI stored in a 11): if the gradient is larger than the preset gradient, the camera is moved to be close to the edge of the element along the positive direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the gradient k OI is not increased any more; if not, the camera is moved to be close to the edge of the element along the negative direction of the Z axis of the coordinate system of the machine tool according to the searching step s 2 until the slope k OI is not increased any more;
auto-focusing is accomplished according to a step one four to step one four edge auto-focusing strategy, wherein the search step s 1 is greater than the search step s 2.
6. The method for detecting edges of large caliber element based on object distance focusing method according to claim 5, wherein in step four, a calculation formula of a slope between the fluctuation starting point O and the peak value i is:
In the formula, (n 0,T0)、(n1,T1) represents the coordinates of the fluctuation starting point O and the peak value I in the horizontal direction variance change curve, respectively.
7. The method for detecting edges of a large-caliber element based on an object distance focusing method according to claim 6, wherein the step two is to collect a plurality of images including each edge, and the specific step of processing the plurality of images comprises: for each image, firstly, convolving the image with a Sobel operator to obtain a gradient image in the horizontal direction of the image; then presetting a first fixed threshold value, and carrying out binarization processing on the gradient image to obtain a binarized image; then presetting a second fixed threshold value, counting the number of pixels with the pixel value of 255 in each column in the binarized image, and determining the column as a pixel column with the edge when the number of pixels exceeds the preset second fixed threshold value, namely determining an edge line; then, the distance between the edge line and the center line of the image is calculated, and the element edge position is obtained according to the distance calculation.
8. The method for detecting edges of a large-caliber element based on an object distance focusing method according to claim 7, wherein the plurality of edge positions obtained in the step two comprise:
The left edge midpoint is at the machine coordinate system at X' L:
X'L＝XL+kpixelΔXL
Wherein X L is the X-axis coordinate when the midpoint of the left edge calibrated in advance moves to the center of the field of view of the camera; k pixel is the actual size represented by a single pixel in the calibrated image; Δx L is the pixel distance between the midpoint of the left edge and the image centerline;
The right edge midpoint is at the machine coordinate system at the X-axis coordinate X' R:
X'R＝XR+kpixelΔXR
wherein X R is the X-axis coordinate when the midpoint of the right edge calibrated in advance moves to the center of the field of view of the camera; Δx R is the pixel distance between the midpoint of the right edge and the image centerline;
The Y-axis coordinate Y' T of the midpoint of the upper edge in the machine coordinate system is:
Y′T＝YT+kpixelΔYT
Wherein Y T is a Y-axis coordinate when a midpoint of the pre-calibrated upper edge moves to the center of the field of view of the camera; ΔY T is the pixel distance between the midpoint of the upper edge and the image centerline;
the lower edge midpoint is at machine coordinate system Y' D:
Y′D＝YD+kpixelΔYD
Wherein Y D is a Y-axis coordinate when a midpoint of a pre-calibrated lower edge moves to the center of a camera visual field; ΔY D is the pixel distance between the midpoint of the lower edge and the image centerline.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111428157.0A CN114119555B (en) | 2021-11-29 | 2021-11-29 | Large-caliber element edge detection method based on object distance focusing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111428157.0A CN114119555B (en) | 2021-11-29 | 2021-11-29 | Large-caliber element edge detection method based on object distance focusing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114119555A CN114119555A (en) | 2022-03-01 |
| CN114119555B true CN114119555B (en) | 2024-05-17 |
Family
ID=80370699
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111428157.0A Active CN114119555B (en) | 2021-11-29 | 2021-11-29 | Large-caliber element edge detection method based on object distance focusing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114119555B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107179645A (en) * | 2017-06-30 | 2017-09-19 | 天津市亚安科技有限公司 | A kind of auto focusing method for camera |
| WO2018059158A1 (en) * | 2016-09-29 | 2018-04-05 | 华为技术有限公司 | Auto-focusing method and apparatus |
| CN111505793A (en) * | 2020-04-15 | 2020-08-07 | 于兴虎 | Microscopic operation automatic focusing method and system based on variance and global search strategy |
| CN111770271A (en) * | 2020-06-28 | 2020-10-13 | 青岛海洋地质研究所 | Automatic focusing method based on image processing |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8194995B2 (en) * | 2008-09-30 | 2012-06-05 | Sony Corporation | Fast camera auto-focus |
| US8027582B2 (en) * | 2009-12-21 | 2011-09-27 | Sony Corporation | Autofocus with confidence measure |
-
2021
- 2021-11-29 CN CN202111428157.0A patent/CN114119555B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2018059158A1 (en) * | 2016-09-29 | 2018-04-05 | 华为技术有限公司 | Auto-focusing method and apparatus |
| CN107179645A (en) * | 2017-06-30 | 2017-09-19 | 天津市亚安科技有限公司 | A kind of auto focusing method for camera |
| CN111505793A (en) * | 2020-04-15 | 2020-08-07 | 于兴虎 | Microscopic operation automatic focusing method and system based on variance and global search strategy |
| CN111770271A (en) * | 2020-06-28 | 2020-10-13 | 青岛海洋地质研究所 | Automatic focusing method based on image processing |
Non-Patent Citations (1)
| Title |
|---|
| 大功率激光驱动器镜组性能影响因素分析;柏占伟;谢志江;袁晓东;;沈阳工业大学学报;20130506(第03期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114119555A (en) | 2022-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107356608B (en) | Rapid dark field detection method for surface microdefects of large-caliber fused quartz optical element | |
| CN110006905B (en) | Large-caliber ultra-clean smooth surface defect detection device combined with linear area array camera | |
| CN110411346B (en) | Method for quickly positioning surface micro-defects of aspheric fused quartz element | |
| EP3387370B1 (en) | Focusing system for a telecentric optical measuring machine | |
| JPH0736613B2 (en) | Focus adjustment method for imaging device | |
| US7120286B2 (en) | Method and apparatus for three dimensional edge tracing with Z height adjustment | |
| CN114113114B (en) | Automatic process method for detecting and repairing micro defects on surface of large-caliber element | |
| CN111239164B (en) | Defect detection device and method thereof | |
| CN113467067B (en) | Automatic focusing method and device of microscopic imaging system based on multi-image area relation | |
| JP2000009991A (en) | Device and method of automatic focusing | |
| CN113538583A (en) | Method for accurately positioning position of workpiece on machine tool and vision system | |
| CN114113116B (en) | Accurate detection process method for surface microdefect of large-caliber element | |
| CN209992407U (en) | Large-caliber ultra-clean smooth surface defect detection device combined with linear array camera | |
| CN110653016B (en) | Pipetting system and calibration method thereof | |
| CN114111578B (en) | Automatic pose determining method for large-caliber element | |
| CN114119555B (en) | Large-caliber element edge detection method based on object distance focusing method | |
| CN115278072B (en) | Automatic focusing method and system for Micro LED detection | |
| CN114113115B (en) | High-precision automatic positioning method for micro defects on surface of large-caliber element | |
| TW201520669A (en) | Bevel-axial auto-focus microscopic system and method thereof | |
| CN115103124B (en) | Active alignment method for camera module | |
| CN114113112B (en) | Surface micro defect positioning and identifying method based on three-light-source microscopic system | |
| JP3180091B2 (en) | Non-contact dimension measurement method by laser autofocus | |
| JP2008058133A (en) | Measuring device for curvature radius of long tool edge and its method | |
| US9658444B2 (en) | Autofocus system and autofocus method for focusing on a surface | |
| CN114001676A (en) | Optical axis automatic alignment method for detecting optical element by knife edge instrument |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |