CN117392079A - Appearance defect detection method and device, visual detection system and electronic equipment - Google Patents
Appearance defect detection method and device, visual detection system and electronic equipment Download PDFInfo
- Publication number
- CN117392079A CN117392079A CN202311316780.6A CN202311316780A CN117392079A CN 117392079 A CN117392079 A CN 117392079A CN 202311316780 A CN202311316780 A CN 202311316780A CN 117392079 A CN117392079 A CN 117392079A
- Authority
- CN
- China
- Prior art keywords
- defect detection
- light field
- appearance defect
- image
- sample
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000007547 defect Effects 0.000 title claims abstract description 271
- 238000001514 detection method Methods 0.000 title claims abstract description 197
- 230000000007 visual effect Effects 0.000 title abstract description 6
- 230000004927 fusion Effects 0.000 claims abstract description 53
- 238000000034 method Methods 0.000 claims abstract description 33
- 238000012549 training Methods 0.000 claims abstract description 32
- 238000012545 processing Methods 0.000 claims description 25
- 238000012795 verification Methods 0.000 claims description 24
- 238000013135 deep learning Methods 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 13
- 238000004590 computer program Methods 0.000 claims description 12
- 230000011218 segmentation Effects 0.000 claims description 11
- 241001292396 Cirrhitidae Species 0.000 claims description 8
- 238000000605 extraction Methods 0.000 claims description 5
- 238000011179 visual inspection Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 3
- 230000007797 corrosion Effects 0.000 description 14
- 238000005260 corrosion Methods 0.000 description 14
- 208000025962 Crush injury Diseases 0.000 description 12
- 230000008569 process Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 11
- 238000003384 imaging method Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 6
- 230000000694 effects Effects 0.000 description 6
- 241001085205 Prenanthella exigua Species 0.000 description 4
- 238000004422 calculation algorithm Methods 0.000 description 3
- 230000003287 optical effect Effects 0.000 description 3
- 230000001360 synchronised effect Effects 0.000 description 3
- 230000003190 augmentative effect Effects 0.000 description 2
- 238000012512 characterization method Methods 0.000 description 2
- 238000007499 fusion processing Methods 0.000 description 2
- 238000005286 illumination Methods 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 230000003796 beauty Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 238000013145 classification model Methods 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000877 morphologic effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000007500 overflow downdraw method Methods 0.000 description 1
- 230000003647 oxidation Effects 0.000 description 1
- 238000007254 oxidation reaction Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/80—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level
- G06V10/806—Fusion, i.e. combining data from various sources at the sensor level, preprocessing level, feature extraction level or classification level of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10052—Images from lightfield camera
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Quality & Reliability (AREA)
- Investigating Materials By The Use Of Optical Means Adapted For Particular Applications (AREA)
Abstract
The application discloses an appearance defect detection method, an appearance defect detection device, a visual detection system and electronic equipment, and belongs to the field of defect detection. The method comprises the following steps: acquiring a plurality of first light field images, wherein the light fields corresponding to the plurality of first light field images are different; channel fusion is carried out on a plurality of first light field images to obtain a target image, wherein the target image comprises light field image characteristics of multiple channels; inputting the target image into an appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model; the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images. According to the appearance defect detection method, through channel superposition and fusion of single-channel images in a plurality of light fields, target images containing characteristics of defects in different light fields are obtained, and the target images are detected, so that high-precision defect detection is realized, and meanwhile, the detection efficiency is improved.
Description
Technical Field
The application belongs to the field of defect detection, and particularly relates to an appearance defect detection method, an appearance defect detection device, a visual detection system and electronic equipment.
Background
Along with the improvement of the living standard of people and the continuous development and optimization of the electronic product industry, the requirements of consumers on beauty are pursued extremely, the appearance quality becomes an important factor for consumers to select electronic products, and the requirements of manufacturers on the defect detection capability of the product surfaces are promoted to be higher and higher, and even the limit of human eyes is reached.
The defect form of the product appearance shows the characteristics of diversity, multi-dimension and multi-angle. The morphological diversity refers to diversity of the center, edge, dead angle and the like of the defect, diversity of the shapes of points, lines, planes and the like, and diversity of colors of gray, white, black, blue, green and the like; multi-dimensionality refers to the need to determine defect specifications from multiple measurement dimensions such as length, width, gray scale, color, depth, etc.; the multi-angle performance means that the appearance defect cannot be observed at a certain angle, the multi-angle performance needs to be selected by manpower to comprehensively observe the appearance defect, and the multi-angle performance needs to be simulated by human eyes for the defect detection.
The existing appearance defect detection technology cannot meet the requirements of complexity and accuracy of defect form detection, and is poor in detection precision and low in detection efficiency.
Disclosure of Invention
The present application aims to solve at least one of the technical problems existing in the prior art. Therefore, the application provides an appearance defect detection method, an appearance defect detection device, a visual detection system and electronic equipment, and the detection efficiency is improved while the defects are detected with high precision.
In a first aspect, the present application provides an appearance defect detection method, including:
acquiring a plurality of first light field images, wherein the first light field images comprise objects to be detected, and light fields corresponding to the plurality of first light field images are different;
channel fusion is carried out on the plurality of first light field images to obtain a target image, wherein the target image comprises light field image characteristics of multiple channels;
inputting the target image into an appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model;
the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
According to the appearance defect detection method, through channel superposition fusion of the single-channel images in the multiple light fields, the target image which can contain the characteristics of defects in different light fields is obtained, so that different defect types are easy to distinguish in the target image, the target image is detected by using the appearance defect detection model, a more comprehensive defect detection result can be obtained, and the detection efficiency is improved while synchronous high-precision detection of the images in different light fields is realized.
According to one embodiment of the application, the target image is obtained by channel fusion based on at least three first light field images, and the light fields corresponding to the at least three first light field images comprise a coaxial light field, a left light field and a right light field.
According to one embodiment of the application, after the inputting of the target image into the appearance defect detection model, the appearance defect detection model performs the following steps on the target image:
performing pixel segmentation and feature extraction on the target image to obtain pixel features of each pixel point in the target image;
and classifying each pixel characteristic, generating the defect detection result and outputting the defect detection result.
According to an embodiment of the application, before the inputting the target image into the appearance defect detection model, the method further comprises:
acquiring a plurality of sample images containing sample objects;
determining a defect detection label of each sample image according to the defect position information, the defect type and the defect area of each sample image;
combining each sample image and the defect detection label corresponding to each sample image to obtain a sample corresponding to each sample image;
Constructing a sample image set based on the plurality of samples;
constructing a training set and a verification set based on the sample image set;
inputting training samples in the training set into the appearance defect detection model to train the appearance defect detection model;
inputting the verification sample in the verification set to the trained appearance defect detection model, and obtaining a defect estimation result corresponding to the verification sample, which is output by the appearance defect detection model;
calculating a loss value according to a defect estimation result of the verification sample and a defect detection label based on a preset loss function;
and under the condition that the loss value is smaller than a preset threshold value or the training frequency of the appearance defect detection model reaches the preset frequency, the training of the appearance defect detection model is completed.
According to an embodiment of the present application, the performing channel fusion on the plurality of first light field images to obtain a target image includes:
and carrying out channel fusion on the plurality of first light field images through a Halcon machine vision image processing system to obtain the target image.
According to one embodiment of the application, the appearance defect detection model is trained based on a structural framework of a deep learning segmentation network.
According to an embodiment of the present application, the performing channel fusion on the plurality of first light field images to obtain a target image further includes:
performing size normalization on the plurality of first light field images;
and merging the normalized first light field images according to a target fusion sequence to obtain the target image.
In a second aspect, the present application provides an appearance defect detection apparatus, the apparatus comprising:
the acquisition module is used for acquiring a plurality of first light field images, wherein the first light field images comprise objects to be detected, and light fields corresponding to the plurality of first light field images are different;
the first processing module is used for carrying out channel fusion on the plurality of first light field images to obtain a target image, wherein the target image comprises light field image characteristics of multiple channels;
the second processing module is used for inputting the target image into an appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model;
the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
In a third aspect, the present application provides a visual inspection system comprising:
the image acquisition device is used for acquiring a plurality of first light field images of the object to be detected;
and the controller is electrically connected with the image acquisition device and is used for executing the appearance defect detection method of the first aspect.
In a fourth aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor implements the method for detecting an appearance defect according to the first aspect when executing the computer program.
In a fifth aspect, the present application provides a non-transitory computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the appearance defect detection method as described in the first aspect above.
In a sixth aspect, the present application provides a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being configured to execute a program or instructions to implement the appearance defect detection method according to the first aspect.
In a seventh aspect, the present application provides a computer program product comprising a computer program which, when executed by a processor, implements the appearance defect detection method as described in the first aspect above.
Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, wherein:
FIG. 1 is a schematic diagram of a scratch on the surface of an object in the open field provided by the related art;
FIG. 2 is a schematic illustration of a scratch on the surface of an object in a dark field provided by the related art;
FIG. 3 is a schematic flow chart of an appearance defect detection method according to an embodiment of the present disclosure;
FIG. 4 is a schematic diagram of an object surface crush injury under a coaxial light field provided in an embodiment of the present application;
FIG. 5 is a schematic diagram of an object surface crush injury in a left light field provided in an embodiment of the present application;
FIG. 6 is a schematic diagram of an object surface crush injury in a right light field provided in an embodiment of the present application;
FIG. 7 is a schematic illustration of corrosion points on the surface of an object in a coaxial light field provided in an embodiment of the present application;
FIG. 8 is a schematic illustration of corrosion points on the surface of an object in a left light field provided by an embodiment of the present application;
FIG. 9 is a schematic illustration of corrosion points on the surface of an object in the right light field provided by an embodiment of the present application;
FIG. 10 is a schematic diagram of object surface crush injury in a multi-channel image provided in an embodiment of the present application;
FIG. 11 is a schematic illustration of corrosion points on the surface of an object in a multi-channel image provided in an embodiment of the present application;
FIG. 12 is a schematic diagram of a deep learning result of an appearance defect detection model provided in an embodiment of the present application on a crush injury in a multi-channel image;
FIG. 13 is a second schematic diagram of the result of deep learning of a crush injury in a multi-channel image by using an appearance defect detection model according to an embodiment of the present disclosure;
FIG. 14 is a schematic diagram of a deep learning result of an appearance defect detection model provided in an embodiment of the present application on corrosion points in a multi-channel image;
FIG. 15 is a second schematic view of the results of deep learning of corrosion points in a multi-channel image by using the appearance defect detection model according to the embodiment of the present application;
FIG. 16 is a second flow chart of an appearance defect detection method according to the embodiment of the present disclosure;
FIG. 17 is a schematic diagram of an appearance defect detecting device according to an embodiment of the present disclosure;
FIG. 18 is a schematic diagram of a visual inspection system provided in an embodiment of the present application;
Fig. 19 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Technical solutions in the embodiments of the present application will be clearly described below with reference to the drawings in the embodiments of the present application, and it is apparent that the described embodiments are some embodiments of the present application, but not all embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments in the present application are within the scope of the protection of the present application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged, as appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the objects identified by "first," "second," etc. are generally of a type and not limited to the number of objects, e.g., the first object may be one or more. Furthermore, in the description and claims, "and/or" means at least one of the connected objects, and the character "/", generally means that the associated object is an "or" relationship.
In the related art, due to complexity and accuracy requirements of appearance defects, the defects cannot be comprehensively and accurately imaged and represented under a light field formed by a single light source and a single angle, and most of the defects are imaged and represented by using multiple light fields formed by multiple light sources and multiple angles.
At present, imaging of multiple light field cameras aims at independent photographing and imaging of defects in different light fields, independent detection is carried out, and the lack of association of imaging characterization forms among images of each light field possibly causes that certain defect types cannot be accurately detected.
For example, for the scratch appearance defect of the object surface, as shown in fig. 1, in the image shot in bright field, the scratch is a black long line with burr in light background, as shown in fig. 2, in the image shot in dark field, the scratch is a white long line with burr in black background, and the features of different light field images lack connection.
Aiming at the problems existing in the defect detection, the embodiment of the application provides a brand-new appearance defect detection method based on multiple light fields, and the detection logic of the method is not limited to application scenes such as appearance defects, size defects and the like.
The method for detecting the appearance defects, the device for detecting the appearance defects, the visual detection system, the electronic device and the readable storage medium provided by the embodiment of the application are described in detail below by means of specific embodiments and application scenes thereof with reference to the accompanying drawings.
The appearance defect detection method can be applied to the terminal, and can be specifically executed by hardware or software in the terminal.
The terminal includes, but is not limited to, a portable communication device such as a mobile phone or tablet having a touch sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the terminal may not be a portable communication device, but rather a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).
In the following various embodiments, a terminal including a display and a touch sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and joystick.
The implementation main body of the appearance defect detection method provided in the embodiment of the present application may be an electronic device or a functional module or a functional entity capable of implementing the appearance defect detection method in the electronic device, where the electronic device in the embodiment of the present application includes, but is not limited to, a mobile phone, a tablet computer, a camera, a wearable device, and the like, and the appearance defect detection method provided in the embodiment of the present application is described below by taking the electronic device as an implementation main body as an example.
As shown in fig. 3, the appearance defect detection method includes: step 310, step 320 and step 330.
Step 310, a plurality of first light field images are acquired, wherein the first light field images comprise objects to be detected, and light fields corresponding to the plurality of first light field images are different.
The object to be detected can be an electronic product such as a mobile phone, a tablet personal computer and the like, or an object such as a household appliance which needs appearance detection.
It should be noted that the light sources with different illumination intensities and different illumination angles may form different light fields.
Since the imaging characterization morphology of defects in different light fields is inconsistent, for example, crush is gray-black in imaging on the on-axis light field as shown in fig. 4, bright white in imaging on the left light field as shown in fig. 5, and bright white in imaging on the right light field as shown in fig. 6.
As another example, corrosion sites are all characterized as gray-black in the on-axis light field as shown in fig. 7, bright-white in the left light field as shown in fig. 8, and bright-white in the right light field as shown in fig. 9, and corrosion sites and pressure flaws cannot be distinguished individually from each other from the light fields.
Thus, defect detection under a single light field can distinguish a portion of defects, but is quite difficult to distinguish the types of defects.
The first light field images can be obtained by using image acquisition equipment such as an RGB camera, a 3D camera or an infrared camera to independently shoot and image an object to be detected in different light fields, and each first light field image corresponds to a light field at the image acquisition moment.
In actual execution, under different light fields, the image acquisition equipment performs independent shooting and imaging on an object to be detected at the same shooting position and angle, so that first light field images corresponding to each light field can be obtained, and each first light field image is used as a single-channel image.
Step 320, performing channel fusion on the plurality of first light field images to obtain a target image, where the target image includes light field image features of multiple channels.
And channel fusion is carried out on the first light field image of each single channel, a plurality of one-dimensional images can be overlapped into a multi-dimensional target image, and the target image comprises light field image characteristics of the first light field image of each channel.
The light field image features can represent the imaging of defects in the first light field image of each channel, and conventional image processing algorithms such as pyramid fusion or generalized intensity-Hue-Saturation (GIHS) fusion algorithm can be used for channel fusion of the first light field image under the coaxial light field, the left light field and the right light field to generate a multi-channel target image.
For example, a general full-color image has one-dimensional components of three channels R (red), G (green) and B (blue), and has three single-channel images R, G and B, by fusing the three single-channel images in a channel manner, the dimension of the image is increased to obtain a three-channel three-dimensional full-color image, and based on the same principle as the channel superposition of the full-color image, in the present application, channel superposition fusion is performed on a first light field image of multiple single-channel channels in one dimension to obtain a multi-dimensional target image of multiple channels as shown in fig. 10 and 11, where the light texture region in the box of fig. 10 is the region where the crush defect is located; the light textured portion of the box of fig. 11 is the area where the etch point defect is located.
Wherein the brightness characteristics of the gray portion may characterize different defect types.
According to the embodiment of the application, the images of different light fields can be synchronously detected by combining the images of the multiple light fields into the multi-channel image for the first time.
Step 330, inputting the target image into the appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model; the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
In actual execution, the obtained target image is input into an appearance defect detection model, the appearance defect detection model detects defects of an object to be detected in the target image, and under the condition that defects exist on the surface of the object to be detected, the appearance defect detection model positions and judges the types of the defects, so that the physical indexes such as the defect type, the defect position, the defect area and the like of each defect on the surface of the object to be detected can be obtained, and the physical indexes of the defects are output as defect detection results.
For example, the defect type may include crush and etch points, etc. that are common to the appearance of the product, the defect area may be represented by how many pixels in the target image, and the defect location may be the coordinates of the defect in the target image.
Among them, defect types include, but are not limited to, defects common in industrial products such as crush injury, corrosion point and oxidation, and detection of crush injury and corrosion point is taken as an example in the subsequent examples of the present application, and are not considered as limiting the protection scope of the present application.
According to the appearance defect detection method provided by the embodiment of the application, the single-channel images in the multiple light fields are subjected to channel superposition fusion to obtain the target image which can contain the characteristics of the defects in different light fields, so that different defect types are easy to distinguish in the target image, the target image is detected by using the appearance defect detection model, a more comprehensive defect detection result can be obtained, and the detection efficiency is improved while the synchronous high-precision detection of the images in different light fields is realized.
In some embodiments, channel fusion is performed on the plurality of first light field images to obtain a target image, and the method further includes:
performing size normalization on the plurality of first light field images;
and merging the normalized multiple first light field images according to the target fusion sequence to obtain a target image.
The target fusion sequence characterizes the sequence of each channel of the first light field image in the image fusion process.
In the actual execution process, before channel combination, each first light field image needs to be adjusted to the same size, and then the first light field images with the adjusted sizes are fused according to the target fusion sequence, so that the target image is obtained.
In this embodiment, the size of each first light field image is normalized, so as to ensure that the first light field image proceeds smoothly in the channel fusion process.
In some embodiments, the target image is obtained by channel fusion based on at least three first light field images, and the light fields corresponding to the at least three first light field images include a coaxial light field, a left light field and a right light field.
It should be noted that, the coaxial light field is mainly used for detecting planar objects with very high reflection degree, such as a mobile phone shell of a mirror surface, the left light field and the right light field can carry out complementary detection on defect types which cannot be distinguished under the coaxial light field, the purpose of distinguishing is achieved, the embodiment can cope with defect detection scenes of most defect types such as crush injury, corrosion points and the like, and has the characteristics of high precision and good detection effect.
For example, the same principle as the channel superposition of the full-color image is that the first light field image corresponding to the coaxial light field is used for replacing the single-channel image of the R channel, the first light field image corresponding to the left light field is used for replacing the single-channel image of the G channel, the first light field image corresponding to the right light field is used for replacing the single-channel image of the B channel, and the first light field images corresponding to the three light fields are subjected to channel fusion, so that the target images of the three channels can be obtained. The number of channels of the plurality of first light field images includes, but is not limited to, 3 channels, 5 channels, 6 channels, etc., and correspondingly, the target image may be a 3 channel image, a 5 channel image, or a 6 channel image.
In addition, under the condition that the target image is a color image, the crush injury in the target image is a blue-green mixed defect form, the corrosion points in the target image are gray-green uniform defect forms, and the representation of the defects is up-scaled on the multi-channel image, so that the defect forms of the crush injury and the corrosion points are obviously distinguished.
In the embodiment, channel fusion is performed on the first light field images of the single channels of the coaxial light field, the left light field and the right light field to obtain the multi-channel target image after dimension rising, so that the defect morphology is more obviously distinguished.
In some embodiments, channel fusion is performed on a plurality of first light field images to obtain a target image, including:
and carrying out channel fusion on the plurality of first light field images by using a Halcon machine vision image processing system to obtain a target image.
The Halcon machine vision image processing system comprises a plurality of operators with image processing functions, such as channel_to_image operators, and can fuse a plurality of single-channel images into a multi-channel image.
In actual execution, inputting a plurality of first light field images into a Halcon machine vision image processing system, and calling a read_image operator in the Halcon machine vision image processing system to read the plurality of first light field images; and then, an operator with a channel fusion function such as a channels_to_image operator is called, channel fusion is carried out on a plurality of first light field images, and a target image is generated and output.
In the embodiment, the Halcon machine vision image processing system is utilized to perform channel fusion on a plurality of first light field images, so that a large number of first light field images can be processed simultaneously while the image processing precision is ensured, further the defect detection efficiency is improved, and the channel fusion is efficient, accurate and reliable.
In some embodiments, after inputting the target image into the appearance defect detection model, the appearance defect detection model performs the following steps on the target image:
performing pixel segmentation and feature extraction on the target image to obtain pixel features of each pixel point in the target image;
classifying the characteristics of each pixel, generating a defect detection result and outputting the defect detection result.
The defect detection result may include physical indexes such as a defect position, a defect type, a defect area, and the like of each defect on the appearance of the object to be detected, and the defect area may be determined by an area composed of the number of continuous pixels.
In the actual execution process, loading an appearance defect detection model, inputting the target image after channel fusion into the appearance defect detection model, dividing and classifying the target image at the pixel level by the appearance defect detection model, and then forming the classified pixels to generate a final defect detection result.
In some embodiments, the appearance defect detection model is trained based on a structural framework of a deep learning segmentation network.
The deep learning segmentation network includes, but is not limited to, a classification network, a semantic segmentation network, an object detection network, and an instance segmentation network.
In addition, the appearance defect detection module is not limited to the segmentation model in practical application, and may be a classification model, object detection, or the like.
For example, the appearance defect detection model may include a backbone network, a classifier, and an output layer connected in sequence; the backbone network may be a CSP-DarkNet35 model network, which is a deep learning network with image segmentation and feature extraction functions.
In practical implementation, the backbone network is used for performing pixel segmentation and feature extraction on the target image, and generating pixel features corresponding to each pixel in the target image and position information of each pixel feature.
The classifier is used for classifying the characteristics of each pixel and generating a defect type corresponding to each pixel in the target image.
The output layer is used for classifying the defect type of each pixel in the target image according to the position information and the defect type of each pixel in the target image, and combining adjacent pixels of the same defect type to generate a defect detection result and outputting, wherein the gray scale of the gray scale represents different defect types, the defect type of the light texture area in the boxes of fig. 12 and 13 is a crush injury, and the defect type of the light texture area in the boxes of fig. 14 and 15 is a corrosion point.
In the defect detection industry, aiming at deep learning, even a multi-channel image is used, only a color three-channel image formed by a single light field of a color camera is formed, and the image is not separated from a nest of the single light field image, so that the application scene and effect of the deep learning are greatly limited. In the embodiment, after the single-channel images under the multiple light fields are subjected to channel fusion, a target image is obtained, and the target image is subjected to defect detection by using a deep learning network, namely an appearance defect detection model, so that the types of defects can be effectively distinguished, and the defect detection result is more accurate.
In some embodiments, prior to inputting the target image into the appearance defect detection model, the method further comprises:
acquiring a plurality of sample images containing sample objects;
determining a defect detection label of each sample image according to the defect position information, the defect type and the defect area of each sample image;
combining each sample image and the defect detection label corresponding to each sample image to obtain a sample corresponding to each sample image;
constructing a sample image set based on the plurality of samples;
constructing a training set and a verification set based on the sample image set;
Inputting training samples in the training set into the appearance defect detection model to train the appearance defect detection model;
inputting the verification sample in the verification set into the trained appearance defect detection model, and obtaining a defect estimation result corresponding to the verification sample, which is output by the appearance defect detection model;
calculating a loss value according to a defect estimation result of the verification sample and the defect detection label based on a preset loss function;
and under the condition that the loss value is smaller than a preset threshold value or the training times of the appearance defect detection model reach the preset times, the appearance defect detection model is trained.
In actual execution, when the object to be detected is a mobile phone, taking a plurality of mobile phones with appearance specific defects as sample objects, and collecting sample light field images under different light fields for each sample image.
And carrying out channel fusion on a plurality of sample light field images in different light fields corresponding to each sample object by adopting the same channel fusion method as that of the first light field image, so as to obtain the sample image corresponding to each sample object. Training the deep learning network by the fused multi-channel sample images.
The defect position information, the defect type and the defect area of the sample image can be used as appearance defect labels of the sample image; and combining each sample image and the appearance defect label corresponding to each sample image to obtain a sample corresponding to each sample object as one sample, so as to construct a sample image set, wherein the sample image set comprises a plurality of samples.
The samples in the sample image set can be divided into a training set and a verification set, the proportion of the number of the samples in the training set and the verification set can be flexibly adjusted, the samples in the training set are training samples, and the samples in the verification set are verification samples.
And sequentially inputting training samples in the training set into the appearance defect detection model to train the appearance defect detection model.
And sequentially inputting the verification samples in the verification set into the trained appearance defect detection model to obtain a defect estimation result corresponding to the verification samples, which is output by the appearance defect detection model.
On the basis, a preset loss function is utilized to calculate a loss value according to the defect estimation result corresponding to the verification sample in the verification set and the appearance defect label. The preset loss function may be a root mean square error (Root Mean Square Error, RMSE) function, and the corresponding loss value is RMSE, and the preset loss function may be set according to actual requirements, which is not limited herein specifically.
After the loss value is obtained through calculation, the training process of the round is finished, model parameters in the appearance defect detection model are updated, and then the next round of training is carried out. In the training process, if the loss value obtained by calculating a certain trained model is smaller than a preset threshold value, the appearance defect detection model is trained.
Or, when the training times of the appearance defect detection model reach the preset times, selecting a trained model with the minimum loss value. The preset number of iterations may be a preset number of iterations.
In this embodiment, the appearance defect detection model is constructed and trained, so that a basis is provided for accurate detection of appearance defects.
A specific embodiment is described below.
As shown in fig. 16, first, under different light fields, an object to be measured is photographed and imaged separately at the same photographing angle, so that a first light field image corresponding to each light field can be obtained, and each first light field image is used as a single channel image.
And then, carrying out channel fusion on the first light field image of each single channel, so that a plurality of one-dimensional images can be overlapped into a multi-dimensional target image, wherein the target image contains the light field image characteristics of the first light field image of each channel.
At present, logic for detecting multi-channel images formed by combining multiple light field images is not used at home and abroad, and belongs to the first example of industry.
And inputting the target image into a deep learning network, wherein the deep learning network is an appearance defect detection model. The appearance defect detection model detects defects of the object to be detected in the target image, and positions and types of the defects are judged by the appearance defect detection model under the condition that the surface of the object to be detected has the defects, so that the physical indexes such as the defect type, the defect position, the defect area and the like of each defect on the surface of the object to be detected can be obtained.
And finally, the deep learning network outputs the physical index of each defect on the surface of the object to be detected as a defect detection result.
In addition, the application scenario of the appearance defect detection method provided in the embodiment of the present application includes, but is not limited to, appearance detection, size detection, traditional image processing, deep learning detection, and the like, and may include all cases where image processing is used.
According to the embodiment of the application, a traditional image processing algorithm is combined with deep learning, a multi-channel fusion image appearance defect detection method based on a multi-light field is designed for the first time, the appearance detection method is truly expanded from one dimension to a multi-dimension level, and the deep learning is developed for the second time, so that the processing of the image can include 3 channels, 5 channels, 6 channels and the like.
According to the appearance defect detection method provided by the embodiment of the application, the execution main body can be an appearance defect detection device. In the embodiment of the present application, an appearance defect detection device is described by taking an example in which an appearance defect detection device performs an appearance defect detection method.
The embodiment of the application also provides an appearance defect detection device.
As shown in fig. 17, the appearance defect detecting device includes: an acquisition module 1710, a first processing module 1720, and a second processing module 1730.
An obtaining module 1710, configured to obtain a plurality of first light field images, where the first light field images include an object to be detected, and light fields corresponding to the plurality of first light field images are different;
a first processing module 1720, configured to perform channel fusion on the plurality of first light field images to obtain a target image, where the target image includes light field image features of multiple channels;
the second processing module 1730 is configured to input the target image to the appearance defect detection model, and obtain a defect detection result of the object to be detected output by the appearance defect detection model; the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
According to the appearance defect detection device provided by the embodiment of the application, through carrying out channel superposition fusion on the single-channel images in the plurality of light fields, the target image which can contain the characteristics of defects in different light fields is obtained, so that different defect types are easy to distinguish in the target image, the target image is detected by utilizing the appearance defect detection model, a more comprehensive defect detection result can be obtained, and the detection efficiency is improved while synchronous high-precision detection on the images in different light fields is realized.
In some embodiments, a first processing module 1720 to size normalize the plurality of first light field images;
and merging the normalized multiple first light field images according to the target fusion sequence to obtain a target image.
In some embodiments, the first processing module 1720 is configured to perform channel fusion on the plurality of first light field images by using the Halcon machine vision image processing system to obtain the target image.
The appearance defect detection device in the embodiment of the application may be an electronic device, or may be a component in the electronic device, for example, an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices than a terminal. By way of example, the electronic device may be a mobile phone, tablet computer, notebook computer, palm computer, vehicle-mounted electronic device, mobile internet appliance (Mobile Internet Device, MID), augmented reality (augmented reality, AR)/Virtual Reality (VR) device, robot, wearable device, ultra-mobile personal computer, UMPC, netbook or personal digital assistant (personal digital assistant, PDA), etc., but may also be a server, network attached storage (Network Attached Storage, NAS), personal computer (personal computer, PC), television (TV), teller machine or self-service machine, etc., and the embodiments of the present application are not limited in particular.
The appearance defect detection device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which is not specifically limited in the embodiments of the present application.
The appearance defect detection device provided in the embodiment of the present application can implement each process implemented in the embodiment of the appearance defect detection method in fig. 1 to 16, and in order to avoid repetition, a detailed description is omitted here.
In some embodiments, as shown in fig. 18, embodiments of the present application further provide a visual inspection system, including:
the image acquisition device 1810, the image acquisition device 1810 is used for acquiring a plurality of first light field images of the object to be detected;
the controller 1820 is electrically connected to the image capturing device 1810, where the controller 1820 is configured to implement the processes of the above-mentioned embodiment of the appearance defect detection method, and achieve the same technical effects, so that repetition is avoided and no further description is given here.
In some embodiments, as shown in fig. 19, the embodiment of the present application further provides an electronic device 1900, including a processor 1901, a memory 1902, and a computer program stored in the memory 1902 and capable of running on the processor 1901, where the program, when executed by the processor 1901, implements the respective processes of the embodiment of the appearance defect detection method, and can achieve the same technical effects, and for avoiding repetition, will not be repeated herein.
The electronic device in the embodiment of the application includes the mobile electronic device and the non-mobile electronic device.
The embodiment of the present application further provides a non-transitory computer readable storage medium, on which a computer program is stored, where the computer program when executed by a processor implements each process of the embodiment of the appearance defect detection method, and the same technical effects can be achieved, so that repetition is avoided, and no further description is given here.
The processor is a processor in the electronic device in the above embodiment. Readable storage media include computer readable storage media such as computer readable memory ROM, random access memory RAM, magnetic or optical disks, and the like.
The embodiment of the application also provides a computer program product, which comprises a computer program, and the computer program realizes the appearance defect detection method when being executed by a processor.
The processor is a processor in the electronic device in the above embodiment. Readable storage media include computer readable storage media such as computer readable memory ROM, random access memory RAM, magnetic or optical disks, and the like.
The embodiment of the application further provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled with the processor, the processor is used for running a program or instructions, the processes of the embodiment of the appearance defect detection method can be realized, the same technical effect can be achieved, and in order to avoid repetition, the description is omitted here.
It should be understood that the chips referred to in the embodiments of the present application may also be referred to as system-on-chip chips, chip systems, or system-on-chip chips, etc.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solutions of the present application may be embodied essentially or in a part contributing to the prior art in the form of a computer software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to perform the methods of the embodiments of the present application.
The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.
In the description of the present specification, reference to the terms "one embodiment," "some embodiments," "illustrative embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiments or examples. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.
While embodiments of the present application have been shown and described, it will be understood by those of ordinary skill in the art that: many changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the application, the scope of which is defined by the claims and their equivalents.
Claims (10)
1. An appearance defect detection method, comprising:
acquiring a plurality of first light field images, wherein the first light field images comprise objects to be detected, and light fields corresponding to the plurality of first light field images are different;
Channel fusion is carried out on the plurality of first light field images to obtain a target image, wherein the target image comprises light field image characteristics of multiple channels;
inputting the target image into an appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model;
the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
2. The method for detecting an appearance defect according to claim 1, wherein the target image is obtained by channel fusion based on at least three first light field images, and the light fields corresponding to the at least three first light field images include a coaxial light field, a left light field and a right light field.
3. The method according to claim 1, wherein after the inputting of the target image into an appearance defect detection model, the appearance defect detection model performs the following steps on the target image:
performing pixel segmentation and feature extraction on the target image to obtain pixel features of each pixel point in the target image;
And classifying each pixel characteristic, generating the defect detection result and outputting the defect detection result.
4. The method of claim 1, wherein prior to said inputting the target image into an appearance defect detection model, the method further comprises:
acquiring a plurality of sample images containing sample objects;
determining a defect detection label of each sample image according to the defect position information, the defect type and the defect area of each sample image;
combining each sample image and the defect detection label corresponding to each sample image to obtain a sample corresponding to each sample image;
constructing a sample image set based on the plurality of samples;
constructing a training set and a verification set based on the sample image set;
inputting training samples in the training set into the appearance defect detection model to train the appearance defect detection model;
inputting the verification sample in the verification set to the trained appearance defect detection model, and obtaining a defect estimation result corresponding to the verification sample, which is output by the appearance defect detection model;
calculating a loss value according to a defect estimation result of the verification sample and a defect detection label based on a preset loss function;
And under the condition that the loss value is smaller than a preset threshold value or the training frequency of the appearance defect detection model reaches the preset frequency, the training of the appearance defect detection model is completed.
5. The method of claim 1, wherein the performing channel fusion on the plurality of first light field images to obtain a target image includes:
and carrying out channel fusion on the plurality of first light field images through a Halcon machine vision image processing system to obtain the target image.
6. The method according to claim 1, wherein the appearance defect detection model is trained based on a structural framework of a deep learning segmentation network.
7. The method for detecting an appearance defect according to any one of claims 1 to 6, wherein the performing channel fusion on the plurality of first light field images to obtain a target image further comprises:
performing size normalization on the plurality of first light field images;
and merging the normalized first light field images according to a target fusion sequence to obtain the target image.
8. An appearance defect detecting device, comprising:
The acquisition module is used for acquiring a plurality of first light field images, wherein the first light field images comprise objects to be detected, and light fields corresponding to the plurality of first light field images are different;
the first processing module is used for carrying out channel fusion on the plurality of first light field images to obtain a target image, wherein the target image comprises light field image characteristics of multiple channels;
the second processing module is used for inputting the target image into an appearance defect detection model to obtain a defect detection result of the object to be detected, which is output by the appearance defect detection model;
the appearance defect detection model is obtained based on training of a sample image set, and sample images of the sample image set are obtained after channel fusion of a plurality of sample light field images.
9. A visual inspection system, comprising:
the image acquisition device is used for acquiring a plurality of first light field images of the object to be detected;
a controller electrically connected to the image pickup device, the controller being configured to perform the appearance defect detection method of any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the appearance defect detection method of any of claims 1-7 when the program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311316780.6A CN117392079A (en) | 2023-10-10 | 2023-10-10 | Appearance defect detection method and device, visual detection system and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311316780.6A CN117392079A (en) | 2023-10-10 | 2023-10-10 | Appearance defect detection method and device, visual detection system and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117392079A true CN117392079A (en) | 2024-01-12 |
Family
ID=89464126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311316780.6A Pending CN117392079A (en) | 2023-10-10 | 2023-10-10 | Appearance defect detection method and device, visual detection system and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117392079A (en) |
-
2023
- 2023-10-10 CN CN202311316780.6A patent/CN117392079A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111667520B (en) | Registration method and device for infrared image and visible light image and readable storage medium | |
| CN110544258B (en) | Image segmentation method and device, electronic equipment and storage medium | |
| JP2011214903A (en) | Appearance inspection apparatus, and apparatus, method and program for generating appearance inspection discriminator | |
| CN108416902A (en) | Real-time object identification method based on difference identification and device | |
| CN112052186A (en) | Target detection method, device, equipment and storage medium | |
| Tan et al. | Separation of highlight reflections on textured surfaces | |
| EP3561776A1 (en) | Method and apparatus for processing a 3d scene | |
| CN111325107A (en) | Detection model training method and device, electronic equipment and readable storage medium | |
| CN116071315A (en) | Product visual defect detection method and system based on machine vision | |
| CN111259710A (en) | Parking space structure detection model training method adopting parking space frame lines and end points | |
| CN116152166A (en) | Defect detection method and related device based on feature correlation | |
| CN113947613B (en) | Target area detection method, device, equipment and storage medium | |
| Żak et al. | Local image features matching for real-time seabed tracking applications | |
| CN111435429A (en) | Gesture recognition method and system based on binocular stereo data dynamic cognition | |
| CN113281780A (en) | Method and device for labeling image data and electronic equipment | |
| Chen et al. | Image segmentation in thermal images | |
| CN109064444B (en) | Track slab disease detection method based on significance analysis | |
| CN106402717B (en) | A kind of AR control method for playing back and intelligent desk lamp | |
| CN110310341A (en) | The generation method of default parameters, device, equipment and storage medium in color algorithm | |
| CN117392079A (en) | Appearance defect detection method and device, visual detection system and electronic equipment | |
| KR102015620B1 (en) | System and Method for detecting Metallic Particles | |
| WO2015136716A1 (en) | Image processing device, image sensor, and image processing method | |
| CN110910379A (en) | Incomplete detection method and device | |
| CN110728222A (en) | Pose estimation method for target object in mechanical arm grabbing system | |
| CN111862106A (en) | Image processing method based on light field semantics, computer device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |