CN117953032A - Defect size calibration method for correcting large-breadth material - Google Patents
Defect size calibration method for correcting large-breadth material Download PDFInfo
- Publication number
- CN117953032A CN117953032A CN202311701938.1A CN202311701938A CN117953032A CN 117953032 A CN117953032 A CN 117953032A CN 202311701938 A CN202311701938 A CN 202311701938A CN 117953032 A CN117953032 A CN 117953032A
- Authority
- CN
- China
- Prior art keywords
- image
- defect
- size
- flaw
- segmented
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000007547 defect Effects 0.000 title claims abstract description 263
- 239000000463 material Substances 0.000 title claims abstract description 133
- 238000000034 method Methods 0.000 title claims abstract description 34
- 238000012545 processing Methods 0.000 claims abstract description 75
- 238000007906 compression Methods 0.000 claims abstract description 36
- 230000006835 compression Effects 0.000 claims abstract description 35
- 230000011218 segmentation Effects 0.000 claims abstract description 28
- 238000005520 cutting process Methods 0.000 claims abstract description 18
- 230000002950 deficient Effects 0.000 claims description 43
- 238000012937 correction Methods 0.000 claims description 30
- 238000000605 extraction Methods 0.000 claims description 21
- 238000001514 detection method Methods 0.000 claims description 17
- 238000003709 image segmentation Methods 0.000 claims description 10
- 238000012544 monitoring process Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 4
- 230000000007 visual effect Effects 0.000 abstract description 9
- 230000009286 beneficial effect Effects 0.000 description 9
- 238000004519 manufacturing process Methods 0.000 description 7
- 238000010586 diagram Methods 0.000 description 6
- 238000004364 calculation method Methods 0.000 description 5
- 239000002994 raw material Substances 0.000 description 4
- 238000009434 installation Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 239000000109 continuous material Substances 0.000 description 1
- 230000001186 cumulative effect Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/136—Segmentation; Edge detection involving thresholding
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Geometry (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
- Image Processing (AREA)
Abstract
The invention provides a flaw defect size calibration method for correcting a large-width material. The method relates to the technical field of defect size calibration, and comprises the following steps: compressing an image of a large-breadth material to obtain a first image; determining a flaw defect area, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation to obtain a second image; determining the resolution of each segmented field of view image of the camera; acquiring the position of a second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating to obtain the defect size of the second image in each sectional view image, and carrying out merging processing to obtain the accurate defect size of the large-breadth material; the method meets the actual operation environment, ensures that the flaw and defect size calibration is not influenced by image compression errors, and corrects the flaw and defect size calibration errors caused by different visual field resolutions of cameras.
Description
Technical Field
The invention relates to the technical field of defect size calibration, in particular to a defect size calibration method for correcting a large-breadth material.
Background
In the process of producing and processing large-breadth materials, an industrial machine vision detection system is often adopted to detect flaw and defect sizes of the materials, the operation speed of a production and processing machine is high, and a linear array camera is often adopted by camera equipment for image data acquisition, so that compression reduction treatment is carried out on detected images of the materials, and errors of dimensional deviation often occur in the process; in addition, the absolute level of camera installation is difficult to achieve, so that the resolution ratio of the detected images from left to right is inconsistent, and further errors occur in flaw and defect size calibration, and quality judgment of materials is affected.
Therefore, the invention provides a flaw size calibration method for correcting a large-width material.
Disclosure of Invention
The invention provides a flaw size calibration method for correcting a large-width material, which is used for obtaining a first image by compressing an image of the large-width material; determining a flaw defect area, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation to obtain a second image; determining the resolution of each segmented field of view image of the camera; acquiring the position of a second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating to obtain the defect size of the second image in each sectional view image, and carrying out merging processing to obtain the accurate defect size of the large-breadth material; the method meets the actual operation environment, ensures that the flaw and defect size calibration is not influenced by image compression errors, and corrects the flaw and defect size calibration errors caused by different visual field resolutions of cameras.
The invention provides a flaw and defect size calibration method for correcting a large-width material, which comprises the following steps:
step 1: monitoring and acquiring an image of a large-breadth material, and performing compression treatment to acquire a first image of the large-breadth material;
Step 2: determining a flaw defect area in the first image, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation on the defect image to obtain a second image of the large-breadth material;
step 3: obtaining a field image of a camera, carrying out segmentation processing according to a preset image segmentation calibration size, and determining the resolution of each segmented field image;
Step 4: and acquiring the position of the second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating the defect size of the second image in each sectional view image according to the resolution ratio, and carrying out merging processing to acquire the accurate defect size of the large-breadth material.
Preferably, obtaining a first image of a wide-width material includes:
Monitoring and scanning the large-breadth material according to the pre-deployment camera to obtain an initial image of the large-breadth material;
Acquiring an application scene of the large-breadth material, and determining the image compression quality of an initial image of the large-breadth material according to a scene running speed-image compression quality mapping table;
and carrying out compression processing on the initial image, and when the initial image after the compression processing is monitored to meet the image compression quality, judging that the compression processing of the initial image is finished, and marking the initial image after the compression processing as a first image.
Preferably, determining a defective area of the flaw in the first image, and performing expansion cutting to obtain a defective image, including:
determining a flaw defect area of the first image and performing primary pixel extraction operation to obtain primary pixel number of the primary pixel extraction operation, and judging that the primary pixel extraction operation is correct when the primary pixel number is monitored to be consistent with the pixel number of the first image;
otherwise, carrying out primary pixel extraction operation on the flaw defect area of the first image again;
Performing outward expansion operation on the flaw defect area according to the preset flaw defect outward expansion size to obtain an outward expansion flaw defect area;
and cutting the outward-expanding flaw defect area to obtain a flaw image.
Preferably, gray statistics and threshold segmentation are performed on the defect image, and a second image of the large-breadth material is obtained, including:
performing secondary pixel extraction on the defect image, and judging that the defect image is qualified when the number of the secondary extraction pixels is monitored to be consistent with the pixel quality of the first image;
Otherwise, re-acquiring the defect image;
Converting a qualified defect image into a gray image, and initializing variances, mean values and weights of foreground pixels and background pixels in the gray image;
According to a preset defect threshold, threshold division is carried out on defective pixels and background pixels, pixels which are larger than or equal to the preset defect threshold in a gray scale range related to the gray scale image are judged to be the background pixels, and pixels which are smaller than the preset defect threshold in the gray scale range are judged to be the defective pixels;
Respectively obtaining the pixel quantity of the defect pixel and the background pixel, and updating the variance, the mean value and the weight;
determining an optimal defect threshold of the gray level image according to the updated variance, the updated average value and the updated weight;
and according to the optimal defect threshold, performing binarization processing on the gray level image, and performing threshold segmentation on the defect pixels and the background pixels to obtain a second image of the large-breadth material.
Preferably, determining the resolution of each segmented view image comprises:
obtaining a field of view image of a camera, and processing the field of view image into a plurality of independent vertical segments according to the preset image segment calibration size;
the pixel number of each segmented image is obtained respectively, and the resolution of each segmented image is determined according to the preset image segmented calibration size, wherein the resolution of each segmented image=the preset image segmented calibration size/the pixel number of each segmented image.
Preferably, obtaining accurate flaw defect sizes for large-width materials includes:
Acquiring the position of a second image in the view image, and determining a calibration plate for performing horizontal correction processing on the second image in the view image according to the pre-deployment position of the camera, the initial image size, the second image size and independent horizontal segmentation processing on the view image of the camera when the second image and the view image are monitored to be not in absolute level;
keeping the pre-deployment position of the camera unchanged, respectively placing the calibration plates at different positions in the view field image, and shooting by the camera;
Acquiring calibration images containing complete calibration plates, performing image processing, extracting calibration points of each calibration image, determining internal and external parameters of a camera after all the calibration points of each calibration image are detected to be extracted correctly, and performing distortion correction processing on a second image;
Randomly selecting a segmented view image from the view image, acquiring the left distances from each left pixel of the second image after distortion correction to the segmented view image, and acquiring the upper distances from each upper pixel of the second image after distortion correction to the segmented view image;
performing horizontal correction processing on the second image according to all left side distances and all upper side distances;
when the left side distances are monitored to be consistent and the upper side distances are monitored to be consistent, judging that the second image horizontal correction processing is completed;
determining the number of defective pixels of the second image in each segmented view image according to the horizontal correction processing position of the second image in the view image;
determining the defect width and defect height of the second image in each segmented field image according to the resolution of each segmented field image;
the longitudinal resolution of the preset view images is consistent, and the defect size of the second image in each segmented view image is calculated;
and accumulating and combining the defect sizes of the segments to obtain the accurate defect size of the large-breadth material.
Preferably, the step of calculating and combining the defect sizes of the second image in each segmented view image includes:
calculating and combining the defect sizes of the second image in each sectional view image to obtain the accurate defect sizes of the large-breadth materials;
; wherein f is the accurate flaw defect size of the large-breadth material; ry is the longitudinal resolution of the field of view image of the camera; px (i 1) is the number of defective width pixels of the i1 st segment segmented view image related to the second image; py (i 1) is the defect-height pixel of the i1 st segment segmented view image related to the second image; l is the preset image segmentation calibration size; p (i 1) is the number of pixels of the i1 st segment segmented view image related to the second image.
Preferably, the method further comprises:
Obtaining the historical detection length and the actual length of the flaw defects of the large-width material, and calculating to obtain the flaw defect size detection deviation of the large-width material;
Calculating to obtain the flaw defect size detection deviation of the corrected large-width material according to the accurate flaw defect size and the actual length of the large-width material;
And when the detected flaw and defect size detection deviation of each corrected large-width material is lower than that of the large-width material, judging that the flaw and defect size calibration correction of the large-width material is successful.
Compared with the prior art, the application has the following beneficial effects:
Compressing an image of a large-breadth material to obtain a first image; determining a flaw defect area, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation to obtain a second image; determining the resolution of each segmented field of view image of the camera; acquiring the position of a second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating to obtain the defect size of the second image in each sectional view image, and carrying out merging processing to obtain the accurate defect size of the large-breadth material; the method meets the actual operation environment, ensures that the flaw and defect size calibration is not influenced by image compression errors, and corrects the flaw and defect size calibration errors caused by different visual field resolutions of cameras.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention may be realized and attained by the structure particularly pointed out in the written description and drawings.
The technical scheme of the invention is further described in detail through the drawings and the embodiments.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate the invention and together with the embodiments of the invention, serve to explain the invention. In the drawings:
FIG. 1 is a flow chart of a method for calibrating a defect size of a large-width material according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of determining a defective area of a first image according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of performing out-expansion on a defective area according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a defect image obtained by cutting a defect area according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of performing gray statistics and threshold segmentation on a defect image to obtain a second image according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of a camera view image and a measured target area of a wide-width material according to an embodiment of the present invention;
FIG. 7 is a schematic diagram of obtaining a defect size of a second image in each segmented view image according to the resolution calculation and performing merging processing according to an embodiment of the present invention;
FIG. 8 is an analysis chart of the expansion operation of the defective area in the embodiment of the present invention.
Detailed Description
The preferred embodiments of the present invention will be described below with reference to the accompanying drawings, it being understood that the preferred embodiments described herein are for illustration and explanation of the present invention only, and are not intended to limit the present invention.
The embodiment of the invention provides a flaw and defect size calibration method for correcting a large-width material, which is shown in fig. 1 and comprises the following steps:
step 1: monitoring and acquiring an image of a large-breadth material, and performing compression treatment to acquire a first image of the large-breadth material;
Step 2: determining a flaw defect area in the first image, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation on the defect image to obtain a second image of the large-breadth material;
step 3: obtaining a field image of a camera, carrying out segmentation processing according to a preset image segmentation calibration size, and determining the resolution of each segmented field image;
Step 4: and acquiring the position of the second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating the defect size of the second image in each sectional view image according to the resolution ratio, and carrying out merging processing to acquire the accurate defect size of the large-breadth material.
In this embodiment, the large-width material is a large width of a material used for production processing, and the large width of the raw material is subjected to flaw and defect size calibration to determine the quality grade of the raw material.
In this embodiment, the first image of the wide-width material refers to an image obtained by compressing the monitoring image of the wide-width material, and is used to adapt to the running speed of the processing of the raw material.
In this embodiment, the defective area in the first image refers to an area containing a defective defect in the first image, and the defective area is not unique and is dynamically adjusted according to the size of the defective defect.
In this embodiment, the outward expanding and cutting refers to amplifying and reducing the defective area, so as to facilitate the subsequent defect calibration operation.
In this embodiment, the defect image is an image obtained by performing a flaring cutting process on a defective region.
In this embodiment, the gray level statistics refers to statistics according to gray values of pixels in the defect image, so as to determine the number of pixels with different gray levels in the defect image.
In this embodiment, threshold segmentation refers to segmentation processing of the defect image according to the gray level statistics result, and segmentation of the defect and the background is performed, so as to facilitate subsequent defect size calibration.
In this embodiment, the second image of the large-width material is an image obtained by performing gray-scale statistics and threshold segmentation processing on the defective image.
In this embodiment, the field of view image of the camera refers to a monitorable image of a pre-deployed camera for monitoring large-format materials.
In this embodiment, the resolution of each segmented field image refers to the ratio of the preset image segmentation calibration size of each segmented image obtained by segmenting the field image of the camera according to the preset image segmentation calibration size to the number of corresponding pixels, wherein the preset image segmentation calibration size is usually 10mm.
In this embodiment, the precise defect size of the large-width material refers to the cumulative processing of the defect sizes of each segment, and the precise defect size obtained is the correction processing of the existing defect size.
The beneficial effects of the technical scheme are as follows: obtaining a first image by compressing an image of a wide-width material; determining a flaw defect area, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation to obtain a second image; determining the resolution of each segmented field of view image of the camera; acquiring the position of a second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating to obtain the defect size of the second image in each sectional view image, and carrying out merging processing to obtain the accurate defect size of the large-breadth material; the method meets the actual operation environment, ensures that the flaw and defect size calibration is not influenced by image compression errors, and corrects the flaw and defect size calibration errors caused by different visual field resolutions of cameras.
The embodiment of the invention provides a method for calibrating the size of a flaw defect of a large-width material, which comprises the following steps of:
Monitoring and scanning the large-breadth material according to the pre-deployment camera to obtain an initial image of the large-breadth material;
Acquiring an application scene of the large-breadth material, and determining the image compression quality of an initial image of the large-breadth material according to a scene running speed-image compression quality mapping table;
and carrying out compression processing on the initial image, and when the initial image after the compression processing is monitored to meet the image compression quality, judging that the compression processing of the initial image is finished, and marking the initial image after the compression processing as a first image.
In this embodiment, according to the characteristics of raw material production and processing, a line camera, that is, a camera using a line image sensor, is often used as a pre-deployment camera, so as to detect continuous materials, and is suitable for measurement and other occasions.
In this embodiment, the initial image of the wide-width material refers to a monitoring image of the wide-width material by the pre-deployment camera, and includes all parameter data of the wide-width material without processing.
In this embodiment, the application scenario of the large-breadth material refers to a production speed of the large-breadth material, and the higher the image compression quality is, the faster the subsequent image processing speed is, and the application scenario of the large-breadth material can be satisfied, for example, the scenario running speed 3 (1-10), and the corresponding image compression quality is determined to be 2 (1-5) according to the scenario running speed-image compression quality mapping table, under the image compression quality, the image compression processing satisfies both the current scenario running speed and the rapid compression processing under the condition that the image quality is unchanged.
In this embodiment, the initial image is compressed, and the compression process needs to ensure that the image quality is unchanged.
The beneficial effects of the technical scheme are as follows: the linear array camera is adopted, so that the characteristic requirements of material production and processing are met, the production cost is reduced, and the deployment efficiency is improved; the image is compressed, so that the defect size calibration can be conveniently carried out later, the processing efficiency is improved, and the high-speed production requirement of materials is met.
The embodiment of the invention provides a calibration method for correcting the defect size of a large-breadth material, which is used for determining a defect area of a defect in a first image and performing outward expansion cutting to obtain a defect image, and comprises the following steps:
determining a flaw defect area of the first image and performing primary pixel extraction operation to obtain primary pixel number of the primary pixel extraction operation, and judging that the primary pixel extraction operation is correct when the primary pixel number is monitored to be consistent with the pixel number of the first image;
otherwise, carrying out primary pixel extraction operation on the flaw defect area of the first image again;
Performing outward expansion operation on the flaw defect area according to the preset flaw defect outward expansion size to obtain an outward expansion flaw defect area;
and cutting the outward-expanding flaw defect area to obtain a flaw image.
In this embodiment, as shown in fig. 2, a defective area in the first image is determined, for example, an area encircled by a line segment in fig. 2 is the defective area in the first image.
In this embodiment, when it is detected that the number of primary pixels coincides with the number of pixels of the first image, the image quality of the defective area of the first image coincides with the image quality of the first image.
In this embodiment, as shown in fig. 3, the expansion operation is performed on the defective area, for example, the inner area of the line segment in fig. 3 includes all the defective areas, and the inner area of the line segment is the image obtained by the expansion operation on the defective areas.
In this embodiment, the preset defect expansion size is, for example, 20 pixels, that is, the defect area is subjected to an outward expansion operation of 20 pixels, so as to ensure that all the defect areas are cut when the subsequent cutting process is performed, that is, the subsequent defect image includes all the defect areas, for example, as shown in fig. 4, the defect areas are cut, and a cut image is obtained.
In this embodiment, the performing the expanding operation on the defective area specifically includes:
Extracting region boundary point coordinates of a flaw defect region, and obtaining a fluent line and a non-fluent line;
if the number of the non-smooth lines is 0, performing the outward expansion operation on the flaw defect area according to the preset flaw defect outward expansion size;
If the number of the non-fluent lines is not 0, extracting the outermost expansion value and the innermost value of each non-fluent line, and simultaneously, connecting the non-fluent lines in a straight line based on the starting point and the ending point of the corresponding non-fluent line to obtain a first number based on points in the straight line and a second number based on points outside the straight line;
According to the outermost expansion value, the innermost value, the first number and the second number, an outer expansion coefficient is given to the corresponding non-fluent line;
; wherein ZY is the outermost spread value; ZN is the innermost measured value; n01 is a first number; n02 is a second number; /(I) Fluency at the j01 point; /(I)Fluency at the j02 th point; y01 represents a fluency threshold;
determining the final expansion size according to all expansion coefficients and combining the fluency threshold value of the preset flaw defect expansion size corresponding to the fluency line;
; wherein W0 represents the preset flaw defect expansion size; w1 represents the final flare size; /(I) The number of the non-fluent lines is represented; n04 represents the number of smooth lines; a fluency coefficient Y01 obtained by mapping the fluency threshold Y01 is represented; /(I) The expansion coefficient of the j3 th non-fluent line; /(I)Represents N03/>And N03/>Is a variance of (2);
and performing expansion operation on the flaw defect area according to the final expansion size.
As shown in fig. 8, u01 is a non-fluent line, u02 is a straight line, u03 is a first point, u04 is a second point, the fluent line is a line formed by points without protrusions or depressions, the non-fluent line is a line formed by points with protrusions or depressions, u05 is regarded as an outermost point, the value is a corresponding outermost expansion value, u06 is regarded as an innermost point, the value is a corresponding innermost value, the value is mainly related to fluency of the line caused by the corresponding point, and the larger the relation is, and the larger the corresponding value is.
In the embodiment, the effective expansion size is determined by effectively analyzing the smooth boundary and the non-smooth boundary of the flaw defect area, so that the reasonable expansion of the area is realized, and the integrity of the defect image obtained later and the reliability of the subsequent analysis are ensured.
The beneficial effects of the technical scheme are as follows: carrying out primary pixel extraction operation on the flaw defect area of the first image, ensuring that the number of the extracted pixels is consistent with that of the first image, and avoiding flaw defect size calibration errors caused by pixel extraction processing; and the flaw defect area is subjected to outward expansion cutting treatment, so that the flaw defect size calibration can be conveniently carried out subsequently.
The embodiment of the invention provides a flaw and defect size calibration method for correcting a large-breadth material, which is used for carrying out gray level statistics and threshold segmentation on a defect image to obtain a second image of the large-breadth material, and comprises the following steps:
performing secondary pixel extraction on the defect image, and judging that the defect image is qualified when the number of the secondary extraction pixels is monitored to be consistent with the pixel quality of the first image;
Otherwise, re-acquiring the defect image;
Converting a qualified defect image into a gray image, and initializing variances, mean values and weights of foreground pixels and background pixels in the gray image;
According to a preset defect threshold, threshold division is carried out on defective pixels and background pixels, pixels which are larger than or equal to the preset defect threshold in a gray scale range related to the gray scale image are judged to be the background pixels, and pixels which are smaller than the preset defect threshold in the gray scale range are judged to be the defective pixels;
Respectively obtaining the pixel quantity of the defect pixel and the background pixel, and updating the variance, the mean value and the weight;
determining an optimal defect threshold of the gray level image according to the updated variance, the updated average value and the updated weight;
and according to the optimal defect threshold, performing binarization processing on the gray level image, and performing threshold segmentation on the defect pixels and the background pixels to obtain a second image of the large-breadth material.
In this embodiment, the secondary pixel fetch operation coincides with the primary pixel fetch operation.
In this embodiment, the gradation image refers to gradation processing of a qualified defective image.
In this embodiment, the variance, the mean and the weight of the foreground pixel and the background pixel in the grayscale image are initialized, for example, the variance, the mean and the weight of the foreground pixel and the background pixel are respectively initialized to 0.
In this embodiment, the grayscale image is divided into two pixel categories: foreground pixels and background pixels.
In this embodiment, the defective pixel and the background pixel are thresholded according to a preset defect threshold, for example, the preset defect threshold is a gray value 100 (the gray range 0-255, 0 represents black and 255 represents white), and the pixels with the gray value greater than or equal to the preset defect threshold are determined as the background pixels, for example, the pixel a1 gray value 200, the pixel a2 gray value 50 and the pixel a3 gray value 125, the pixel a1 and the pixel a3 are determined as the background pixels, and the pixel a2 is determined as the defective pixel.
In this embodiment, the preset defect threshold is not unique.
In this embodiment, an optimal defect threshold of the gray image is determined according to the updated variance, average value and weight, for example, after pixel classification is performed according to all preset defect thresholds, a corresponding preset variance with the largest pixel variance is used as the optimal defect threshold, where pixel variance=before the background weight is Jing Quan (background average value-before Jing Junzhi)/(2).
In this embodiment, binarizing the gray-scale image means image processing of the gray-scale image according to an optimal defect threshold, for example, as shown in fig. 5, threshold segmentation is performed on the image, the image inside the line segment is used as a foreground, and the image outside the line segment is used as a background.
The beneficial effects of the technical scheme are as follows: the defective image is subjected to secondary pixel extraction, so that the image quality is ensured; determining an optimal defect threshold of the gray level image, and facilitating the subsequent threshold segmentation processing of the defect pixels and the background pixels; and the gray level image is subjected to binarization processing and threshold segmentation, so that a second image of a large-breadth material is obtained, image data is simplified, and defective pixels are accurately identified.
The embodiment of the invention provides a calibration method for correcting flaw and defect sizes of a large-breadth material, which comprises the following steps of:
obtaining a field of view image of a camera, and processing the field of view image into a plurality of independent vertical segments according to the preset image segment calibration size;
the pixel number of each segmented image is obtained respectively, and the resolution of each segmented image is determined according to the preset image segmented calibration size, wherein the resolution of each segmented image=the preset image segmented calibration size/the pixel number of each segmented image.
In this embodiment, as shown in fig. 6, the image segmentation calibration size is preset, and is determined according to a film with 10mm between black and white, namely, segmentation is performed every 10mm, and resolution calculation is performed every 10 mm.
In this embodiment, the large-breadth material is generally 1500 mm-180 mm, so that the resolution of the 150-180 segment segmented field-of-view image is calculated for subsequent flaw size calibration.
In this embodiment, since a large-width material has a defective region, the number of pixels of each segment visual field image is not uniform.
The beneficial effects of the technical scheme are as follows: the visual field image is processed into a plurality of independent vertical segments according to the preset image segment calibration size, the resolution of each segment visual field image is determined, and the defect size calibration error problem caused by inconsistent pixel resolution in the camera visual field image due to difficulty in horizontal installation of the camera is solved.
The embodiment of the invention provides a method for calibrating the size of a flaw defect of a large-width material, which is used for obtaining the accurate flaw defect size of the large-width material and comprises the following steps:
Acquiring the position of a second image in the view image, and determining a calibration plate for performing horizontal correction processing on the second image in the view image according to the pre-deployment position of the camera, the initial image size, the second image size and independent horizontal segmentation processing on the view image of the camera when the second image and the view image are monitored to be not in absolute level;
keeping the pre-deployment position of the camera unchanged, respectively placing the calibration plates at different positions in the view field image, and shooting by the camera;
Acquiring calibration images containing complete calibration plates, performing image processing, extracting calibration points of each calibration image, determining internal and external parameters of a camera after all the calibration points of each calibration image are detected to be extracted correctly, and performing distortion correction processing on a second image;
Randomly selecting a segmented view image from the view image, acquiring the left distances from each left pixel of the second image after distortion correction to the segmented view image, and acquiring the upper distances from each upper pixel of the second image after distortion correction to the segmented view image;
performing horizontal correction processing on the second image according to all left side distances and all upper side distances;
when the left side distances are monitored to be consistent and the upper side distances are monitored to be consistent, judging that the second image horizontal correction processing is completed;
determining the number of defective pixels of the second image in each segmented view image according to the horizontal correction processing position of the second image in the view image;
determining the defect width and defect height of the second image in each segmented field image according to the resolution of each segmented field image;
the longitudinal resolution of the preset view images is consistent, and the defect size of the second image in each segmented view image is calculated;
and accumulating and combining the defect sizes of the segments to obtain the accurate defect size of the large-breadth material.
In this embodiment, determining that the second image is subjected to the horizontal correction processing in the view image means that the second image and the view image are subjected to the horizontal processing, so that a resolution error caused by that the view image and the second image are not relatively horizontal in the following process is avoided.
In this embodiment, the calibration board for performing the horizontal correction processing on the second image in the view image refers to a tool for calibrating and calibrating the camera, for example, calibration boards with black and white squares of 10mm x 10mm.
In this embodiment, the size of the calibration plate and the relative camera position are known, and are used to calculate the internal parameters and the external parameters of the camera, and establish an accurate mapping relationship between the camera and the second image.
In this embodiment, the internal parameters of the camera are used to describe the conversion relationship between the positions of the points of the wide material and the pixel points of the corresponding image positions, and the internal parameters of the camera are not changed due to the placement position of the camera, such as focal length, principal point and distortion parameters; the external parameters of the camera refer to representing the coordinate position, such as position and attitude, of the wide-width material by using a camera coordinate system.
In this embodiment, the second image is subjected to horizontal correction processing, so that each defective area is conveniently divided into a plurality of segments, the resolution of each segment and the number of corresponding defective pixels are calculated to obtain the defective size of each segment, and then the defective sizes of each segment are accumulated and combined to obtain the accurate defective size of the large-width material.
In this embodiment, the defect width and the defect height of the second image in each segment view image are determined according to the defect pixels, for example, the number of transverse pixels of the view segment a1 is 10, the segment length is 10mm, the maximum number of transverse defect pixels of the second image in the view segment a1 is 7, the maximum number of longitudinal defect pixels is 8, and then the defect width of the second image in the view segment a1 is 7mm and the defect height is 8mm.
In this embodiment, the defect sizes of the segments are accumulated and combined to obtain the accurate defect size of the large-width material, for example, as shown in fig. 7, if the defect area is in the 3 rd, 4 th and 5 th segment view images, the length of the defect area is the sum of the defect widths in the corresponding segment view images.
In this embodiment, as shown in fig. 7, the defect heights in the sectional view images are identical, and if they are not identical, the largest defect height in the sectional view image is selected as the defect height of the defect area for subsequent calculation.
The beneficial effects of the technical scheme are as follows: the second image is subjected to distortion correction and horizontal correction, so that the defect size in each field of view section can be accurately determined; the defect sizes of the segments are accumulated and combined to obtain the accurate defect sizes of the large-breadth materials, the problem that the visual field resolution is inconsistent due to the movement of the large-breadth materials, and further the defect size calibration error is caused is solved, and the defect size calibration is effectively corrected.
The embodiment of the invention provides a flaw size calibration method for correcting a large-breadth material, which is used for calculating the flaw size of a second image in each sectional view image and carrying out merging processing, and comprises the following steps:
calculating and combining the defect sizes of the second image in each sectional view image to obtain the accurate defect sizes of the large-breadth materials;
; wherein f is the accurate flaw defect size of the large-breadth material; ry is the longitudinal resolution of the field of view image of the camera; px (i 1) is the number of defective width pixels of the i1 st segment segmented view image related to the second image; py (i 1) is the defect-height pixel of the i1 st segment segmented view image related to the second image; l is the preset image segmentation calibration size; p (i 1) is the number of pixels of the i1 st segment segmented view image related to the second image.
In this embodiment, the precise flaw size of the wide material should be consistent over each time period.
In this embodiment, the flaw defect size in each segmented field image is calculated from the resolution of each segmented field image.
In this embodiment, the longitudinal resolution of each segmented field of view image is preset to be uniform for simplifying the calculation without affecting the calculation accuracy.
The beneficial effects of the technical scheme are as follows: the defect sizes of the second images in the sectional view images are calculated and combined to obtain the accurate defect sizes of the large-breadth materials, so that the accurate measurement of the defect sizes is realized, the calibration errors caused by inconsistent resolutions in the camera view images are corrected, and the accuracy requirements for camera installation are further reduced.
The embodiment of the invention provides a flaw and defect size calibration method for correcting a large-width material, which further comprises the following steps:
Obtaining the historical detection length and the actual length of the flaw defects of the large-width material, and calculating to obtain the flaw defect size detection deviation of the large-width material;
Calculating to obtain the flaw defect size detection deviation of the corrected large-width material according to the accurate flaw defect size and the actual length of the large-width material;
And when the detected flaw and defect size detection deviation of each corrected large-width material is lower than that of the large-width material, judging that the flaw and defect size calibration correction of the large-width material is successful.
In this embodiment, the deviation of the detected size of the defect of the large-width material is calculated, for example, the history of the detected size of the defect of the large-width material is 0.489127, and the actual length is 0.3768, and the deviation of the detected size of the defect is 0.298.
In this embodiment, the corrected defect size detection deviation of the large-width material is calculated, for example, the accurate defect size is 0.4329635 and the actual length is 0.3768, and then the corrected defect size detection deviation is 0.056.
The beneficial effects of the technical scheme are as follows: and calculating to obtain the flaw and defect size detection deviation of the large-width material and the flaw and defect size detection deviation of the corrected large-width material, and intuitively displaying the improvement of the precision of flaw and defect size calibration, so as to be convenient for correcting the flaw and defect size calibration.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present invention without departing from the spirit or scope of the invention. Thus, it is intended that the present invention also include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.
Claims (8)
1. A method for calibrating flaw size of a large-width material, comprising:
step 1: monitoring and acquiring an image of a large-breadth material, and performing compression treatment to acquire a first image of the large-breadth material;
Step 2: determining a flaw defect area in the first image, performing outward expansion cutting to obtain a defect image, and performing gray level statistics and threshold segmentation on the defect image to obtain a second image of the large-breadth material;
step 3: obtaining a field image of a camera, carrying out segmentation processing according to a preset image segmentation calibration size, and determining the resolution of each segmented field image;
Step 4: and acquiring the position of the second image in the view image, determining the defect width and the defect pixel number of the second image in each sectional view image, calculating the defect size of the second image in each sectional view image according to the resolution ratio, and carrying out merging processing to acquire the accurate defect size of the large-breadth material.
2. The method for calibrating a flaw size of a wide material according to claim 1, wherein obtaining a first image of the wide material comprises:
Monitoring and scanning the large-breadth material according to the pre-deployment camera to obtain an initial image of the large-breadth material;
Acquiring an application scene of the large-breadth material, and determining the image compression quality of an initial image of the large-breadth material according to a scene running speed-image compression quality mapping table;
and carrying out compression processing on the initial image, and when the initial image after the compression processing is monitored to meet the image compression quality, judging that the compression processing of the initial image is finished, and marking the initial image after the compression processing as a first image.
3. The method for calibrating a defect size of a large-width material according to claim 1, wherein determining a defective area of the defect in the first image and performing a flaring cutting to obtain a defect image comprises:
determining a flaw defect area of the first image and performing primary pixel extraction operation to obtain primary pixel number of the primary pixel extraction operation, and judging that the primary pixel extraction operation is correct when the primary pixel number is monitored to be consistent with the pixel number of the first image;
otherwise, carrying out primary pixel extraction operation on the flaw defect area of the first image again;
Performing outward expansion operation on the flaw defect area according to the preset flaw defect outward expansion size to obtain an outward expansion flaw defect area;
and cutting the outward-expanding flaw defect area to obtain a flaw image.
4. The method for calibrating a defect size of a large-width material according to claim 1, wherein the step of performing gray statistics and threshold segmentation on the defect image to obtain a second image of the large-width material comprises the steps of:
performing secondary pixel extraction on the defect image, and judging that the defect image is qualified when the number of the secondary extraction pixels is monitored to be consistent with the pixel quality of the first image;
Otherwise, re-acquiring the defect image;
Converting a qualified defect image into a gray image, and initializing variances, mean values and weights of foreground pixels and background pixels in the gray image;
According to a preset defect threshold, threshold division is carried out on defective pixels and background pixels, pixels which are larger than or equal to the preset defect threshold in a gray scale range related to the gray scale image are judged to be the background pixels, and pixels which are smaller than the preset defect threshold in the gray scale range are judged to be the defective pixels;
Respectively obtaining the pixel quantity of the defect pixel and the background pixel, and updating the variance, the mean value and the weight;
determining an optimal defect threshold of the gray level image according to the updated variance, the updated average value and the updated weight;
and according to the optimal defect threshold, performing binarization processing on the gray level image, and performing threshold segmentation on the defect pixels and the background pixels to obtain a second image of the large-breadth material.
5. A method for calibrating a flaw size for correcting large-width materials according to claim 1, wherein determining the resolution of each segmented field-of-view image comprises:
obtaining a field of view image of a camera, and processing the field of view image into a plurality of independent vertical segments according to the preset image segment calibration size;
the pixel number of each segmented image is obtained respectively, and the resolution of each segmented image is determined according to the preset image segmented calibration size, wherein the resolution of each segmented image=the preset image segmented calibration size/the pixel number of each segmented image.
6. The method for calibrating a defect size of a wide-width material according to claim 1, wherein obtaining an accurate defect size of the wide-width material comprises:
Acquiring the position of a second image in the view image, and determining a calibration plate for performing horizontal correction processing on the second image in the view image according to the pre-deployment position of the camera, the initial image size, the second image size and independent horizontal segmentation processing on the view image of the camera when the second image and the view image are monitored to be not in absolute level;
keeping the pre-deployment position of the camera unchanged, respectively placing the calibration plates at different positions in the view field image, and shooting by the camera;
Acquiring calibration images containing complete calibration plates, performing image processing, extracting calibration points of each calibration image, determining internal and external parameters of a camera after all the calibration points of each calibration image are detected to be extracted correctly, and performing distortion correction processing on a second image;
Randomly selecting a segmented view image from the view image, acquiring the left distances from each left pixel of the second image after distortion correction to the segmented view image, and acquiring the upper distances from each upper pixel of the second image after distortion correction to the segmented view image;
performing horizontal correction processing on the second image according to all left side distances and all upper side distances;
when the left side distances are monitored to be consistent and the upper side distances are monitored to be consistent, judging that the second image horizontal correction processing is completed;
determining the number of defective pixels of the second image in each segmented view image according to the horizontal correction processing position of the second image in the view image;
determining the defect width and defect height of the second image in each segmented field image according to the resolution of each segmented field image;
the longitudinal resolution of the preset view images is consistent, and the defect size of the second image in each segmented view image is calculated;
and accumulating and combining the defect sizes of the segments to obtain the accurate defect size of the large-breadth material.
7. The method for calibrating a defect size of a wide-width material according to claim 6, wherein the step of calculating the defect size of the second image in each of the segmented view images and performing the merging process comprises:
calculating and combining the defect sizes of the second image in each sectional view image to obtain the accurate defect sizes of the large-breadth materials;
; wherein f is the accurate flaw defect size of the large-breadth material; ry is the longitudinal resolution of the field of view image of the camera; px (i 1) is the number of defective width pixels of the i1 st segment segmented view image related to the second image; py (i 1) is the defect-height pixel of the i1 st segment segmented view image related to the second image; l is the preset image segmentation calibration size; p (i 1) is the number of pixels of the i1 st segment segmented view image related to the second image.
8. A method for calibrating a flaw size of a large-width material according to claim 1, further comprising:
Obtaining the historical detection length and the actual length of the flaw defects of the large-width material, and calculating to obtain the flaw defect size detection deviation of the large-width material;
Calculating to obtain the flaw defect size detection deviation of the corrected large-width material according to the accurate flaw defect size and the actual length of the large-width material;
And when the detected flaw and defect size detection deviation of each corrected large-width material is lower than that of the large-width material, judging that the flaw and defect size calibration correction of the large-width material is successful.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311701938.1A CN117953032B (en) | 2023-12-12 | 2023-12-12 | Defect size calibration method for correcting large-breadth material |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202311701938.1A CN117953032B (en) | 2023-12-12 | 2023-12-12 | Defect size calibration method for correcting large-breadth material |
Publications (2)
Publication Number | Publication Date |
---|---|
CN117953032A true CN117953032A (en) | 2024-04-30 |
CN117953032B CN117953032B (en) | 2024-10-18 |
Family
ID=90804669
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202311701938.1A Active CN117953032B (en) | 2023-12-12 | 2023-12-12 | Defect size calibration method for correcting large-breadth material |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN117953032B (en) |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0742431A1 (en) * | 1995-05-10 | 1996-11-13 | Mahlo GmbH & Co. KG | Method and apparatus for detecting flaws in moving fabrics or the like |
CN109141232A (en) * | 2018-08-07 | 2019-01-04 | 常州好迪机械有限公司 | A kind of circle plate casting online test method based on machine vision |
CN113610774A (en) * | 2021-07-16 | 2021-11-05 | 广州大学 | Glass scratch defect detection method, system, device and storage medium |
CN115330770A (en) * | 2022-10-12 | 2022-11-11 | 南通宝江家用纺织品有限公司 | Cloth area type defect identification method |
US20230005130A1 (en) * | 2020-07-16 | 2023-01-05 | Boe Technology Group Co., Ltd. | Method and device for detecting display panel defect |
CN115797872A (en) * | 2023-01-31 | 2023-03-14 | 捷易(天津)包装制品有限公司 | Machine vision-based packaging defect identification method, system, equipment and medium |
CN116485752A (en) * | 2023-04-23 | 2023-07-25 | 信利光电股份有限公司 | Display screen AOI detection method, system, electronic device and storage medium for removing edge misjudgment |
CN116559183A (en) * | 2023-07-11 | 2023-08-08 | 钛玛科(北京)工业科技有限公司 | Method and system for improving defect judging efficiency |
CN116990311A (en) * | 2023-08-03 | 2023-11-03 | 杭州海康机器人股份有限公司 | Defect detection system, method and device |
-
2023
- 2023-12-12 CN CN202311701938.1A patent/CN117953032B/en active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0742431A1 (en) * | 1995-05-10 | 1996-11-13 | Mahlo GmbH & Co. KG | Method and apparatus for detecting flaws in moving fabrics or the like |
CN109141232A (en) * | 2018-08-07 | 2019-01-04 | 常州好迪机械有限公司 | A kind of circle plate casting online test method based on machine vision |
US20230005130A1 (en) * | 2020-07-16 | 2023-01-05 | Boe Technology Group Co., Ltd. | Method and device for detecting display panel defect |
CN113610774A (en) * | 2021-07-16 | 2021-11-05 | 广州大学 | Glass scratch defect detection method, system, device and storage medium |
CN115330770A (en) * | 2022-10-12 | 2022-11-11 | 南通宝江家用纺织品有限公司 | Cloth area type defect identification method |
CN115797872A (en) * | 2023-01-31 | 2023-03-14 | 捷易(天津)包装制品有限公司 | Machine vision-based packaging defect identification method, system, equipment and medium |
CN116485752A (en) * | 2023-04-23 | 2023-07-25 | 信利光电股份有限公司 | Display screen AOI detection method, system, electronic device and storage medium for removing edge misjudgment |
CN116559183A (en) * | 2023-07-11 | 2023-08-08 | 钛玛科(北京)工业科技有限公司 | Method and system for improving defect judging efficiency |
CN116990311A (en) * | 2023-08-03 | 2023-11-03 | 杭州海康机器人股份有限公司 | Defect detection system, method and device |
Non-Patent Citations (3)
Title |
---|
ZIHAO LIU ET.AL.: ""Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds"", 《2019 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR)》, 9 January 2020 (2020-01-09) * |
郝丽;杨旭;王峥;黄亚平;宋元涛;: "复杂背景下基于多阈值的光滑工件表面瑕疵检测", 计算机应用与软件, no. 09, 10 September 2020 (2020-09-10), pages 189 - 193 * |
陈红星: ""透明玻璃平板表面微划痕在线检测方法研究"", 《中国优秀硕士学位论文全文数据库 信息科技辑》, vol. 2021, no. 06, 15 June 2021 (2021-06-15), pages 136 - 36 * |
Also Published As
Publication number | Publication date |
---|---|
CN117953032B (en) | 2024-10-18 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200364849A1 (en) | Method and device for automatically drawing structural cracks and precisely measuring widths thereof | |
CN116559183B (en) | Method and system for improving defect judging efficiency | |
CN112651968B (en) | Wood board deformation and pit detection method based on depth information | |
CN103051872B (en) | Based on the belt deflection detection method of Edge extraction | |
CN107490582B (en) | Assembly line workpiece detection system | |
CN107798293A (en) | A kind of crack on road detection means | |
CN118475952A (en) | Method and device for detecting stability of vision system | |
JP2012002531A (en) | Crack detection method | |
CN113077437B (en) | Workpiece quality detection method and system | |
CN111539927B (en) | Detection method of automobile plastic assembly fastening buckle missing detection device | |
CN111798423B (en) | Concrete crack picture splicing and detecting method | |
CN113487563B (en) | EL image-based self-adaptive detection method for hidden cracks of photovoltaic module | |
JP2013238449A (en) | Crack detection method | |
CN117893457B (en) | PCB intelligent detection method, device and computer equipment | |
CN109767426B (en) | Shield tunnel water leakage detection method based on image feature recognition | |
CN117058411B (en) | Method, device, medium and equipment for identifying edge appearance flaws of battery | |
CN114199127A (en) | Automobile part size detection system and method based on machine vision | |
CN117953032B (en) | Defect size calibration method for correcting large-breadth material | |
CN116026861A (en) | Glass bottle detection method and system | |
CN115761468A (en) | Water level detection system and method based on image segmentation and target detection technology | |
CN114708262A (en) | Visual detection method for pin of connector | |
CN110458840B (en) | Method, system and terminal equipment for reducing panel defect over-detection rate | |
CN116503408B (en) | Scanning technology-based steel structure surface defect detection method | |
CN117593302B (en) | Defective part tracing method and system | |
CN113870237B (en) | Composite material image shadow detection method based on horizontal diffusion |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant |