CN114240939B - Method, system, equipment and medium for detecting appearance defects of mainboard components - Google Patents
Method, system, equipment and medium for detecting appearance defects of mainboard components Download PDFInfo
- Publication number
- CN114240939B CN114240939B CN202210171790.4A CN202210171790A CN114240939B CN 114240939 B CN114240939 B CN 114240939B CN 202210171790 A CN202210171790 A CN 202210171790A CN 114240939 B CN114240939 B CN 114240939B
- Authority
- CN
- China
- Prior art keywords
- image
- detected
- main board
- mainboard
- pixels
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007547 defect Effects 0.000 title claims abstract description 72
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000001514 detection method Methods 0.000 claims abstract description 50
- 238000012549 training Methods 0.000 claims abstract description 26
- 238000007781 pre-processing Methods 0.000 claims abstract description 20
- 238000010586 diagram Methods 0.000 claims description 37
- 238000013461 design Methods 0.000 claims description 25
- 230000004044 response Effects 0.000 claims description 14
- 230000002159 abnormal effect Effects 0.000 claims description 11
- 238000004422 calculation algorithm Methods 0.000 claims description 10
- 238000001914 filtration Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 5
- 230000002950 deficient Effects 0.000 claims description 5
- 238000004040 coloring Methods 0.000 claims description 3
- 238000000354 decomposition reaction Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 abstract description 7
- 238000013136 deep learning model Methods 0.000 abstract description 6
- 238000004519 manufacturing process Methods 0.000 description 19
- 239000003086 colorant Substances 0.000 description 13
- 238000013135 deep learning Methods 0.000 description 7
- 230000008859 change Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 5
- 238000012545 processing Methods 0.000 description 5
- 238000013528 artificial neural network Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 239000000463 material Substances 0.000 description 3
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000007689 inspection Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000003990 capacitor Substances 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000014509 gene expression Effects 0.000 description 1
- 230000002427 irreversible effect Effects 0.000 description 1
- 238000002372 labelling Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 229910000679 solder Inorganic materials 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
- G06T7/001—Industrial image inspection using an image reference approach
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30141—Printed circuit board [PCB]
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- General Health & Medical Sciences (AREA)
- Computational Linguistics (AREA)
- Biophysics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Biology (AREA)
- Quality & Reliability (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a method, a system, equipment and a medium for detecting appearance defects of main board components, wherein the method comprises the following steps: acquiring a high-definition image of a mainboard, preprocessing the high-definition image to obtain an image to be detected, comparing the image to be detected of the mainboard through a standard template image corresponding to the mainboard, and storing the image to be detected of the mainboard and a comparison result thereof; responding to the stored images to be detected of the main board and the comparison results thereof to be accumulated to a preset number, and training an image recognition model through the stored images to be detected of the main board and the comparison results thereof; and carrying out defect detection on the acquired image to be detected of the mainboard through the trained image recognition model. By the method, the system, the equipment and the medium for detecting the appearance defects of the mainboard components, the problem that deep learning model reasoning cannot be used due to the fact that defect samples are difficult to collect in the mainboard appearance defect detection process can be effectively solved.
Description
Technical Field
The invention belongs to the field of computers, and particularly relates to a method, a system, equipment and a medium for detecting appearance defects of a mainboard component.
Background
With the development of Artificial Intelligence (AI) technology in recent years, especially in the field of deep learning, more and more new technologies are applied to production and life of people. Currently, AI has achieved technology ground in a plurality of fields such as finance, medical treatment, industrial manufacturing and security, and application scenarios are increasingly abundant, which leads to a profound revolution in various industries. The future development of AI is the combination of technology and industry, realizes the AI technology to enable various industries, solves pain points, creates value, reduces cost and improves efficiency.
The current major outbreaks of artificial intelligence are caused by deep learning. Deep learning is to use a deep neural network to automatically learn the characteristics of an object, and then the deep neural network has the capability of identifying the object. Image recognition has been increasingly applied as an important direction of deep learning applications such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks). The method is more mature in the fields of intelligent driving, face recognition, medical image recognition, industrial quality inspection and the like.
In the actual factory production, quality detection plays an important role as the guarantee of product quality.
At present, the surface defect detection of most of main board components is basically carried out by naked eyes of workers or by means of naked eyes of a magnifier, the condition of missed detection and error detection is not only low in efficiency but also easy to occur, and meanwhile, the workers use eyes for a long time, fatigue damage is easy to cause, and the damage is irreversible.
Based on the reasons, a plurality of quality inspectors must be equipped on each production line to detect and rotate regularly, so that the quality of the product is ensured, and the damage to eyes is reduced, thereby not only reducing the production efficiency, but also needing to invest a large amount of manpower and material resources, lowering the product profit and influencing the product competitiveness. Meanwhile, due to the fact that eyes are used excessively, the loss rate of quality inspection workers is high, and a factory is in a state of always calling for workers.
Aiming at the problems, the invention provides a method for combining image registration, template matching and deep learning, which is a method for detecting defects by registration and template matching in the early period, automatically labeling the defects in the problems, training a model when the number of defect samples reaches a certain number, and detecting by using the deep learning model after training. The method overcomes the defects that defect samples need to be collected from the beginning to perform model training when deep learning is directly used for defect detection in the prior art. In normal production products, the number of defect products is relatively very small, all kinds of defect samples need to be collected, the method is very difficult to achieve in reality, and a plurality of model methods using deep learning are not practical in actual production.
Therefore, a detection method which is more convenient for detection is needed.
Disclosure of Invention
In order to solve the above problems, the present invention provides a method for detecting an appearance defect of a motherboard device, including:
acquiring a high-definition image of a mainboard, preprocessing the high-definition image to obtain an image to be detected, comparing the image to be detected of the mainboard through a standard template image corresponding to the mainboard, and storing the image to be detected of the mainboard and a comparison result thereof;
responding to the stored images to be detected of the mainboard and the comparison results thereof accumulated to a preset number, and training an image recognition model through the stored images to be detected of the mainboard and the comparison results thereof; and
and carrying out defect detection on the acquired image to be detected of the mainboard through the trained image recognition model.
In some embodiments of the present invention, obtaining a high-definition image of a motherboard and preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard motherboard image corresponding to the motherboard includes:
carrying out bit-wise subtraction on the image to be detected of the main board and the pixel corresponding to the standard template image corresponding to the main board to obtain a corresponding difference value, and judging whether the difference value is greater than a first threshold value or not;
and in response to the difference value being larger than a first threshold value, removing the isolated pixels of which the difference value is larger than the first threshold value through a median filtering algorithm, and automatically marking the area of which the difference value is larger than the first threshold value as a defect area.
In some embodiments of the present invention, obtaining a high-definition image of a motherboard and preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard motherboard image corresponding to the motherboard further includes:
generating a corresponding abstract picture by the image to be detected of the main board and the standard template image corresponding to the main board according to the difference of pixel values between adjacent images;
carrying out bit-wise subtraction on the generated abstract picture of the image to be detected of the main board and the generated standard template image corresponding to the main board to obtain a difference value of corresponding pixel bits;
removing the pixels with the difference value not being the first preset value through a median filtering algorithm, and judging whether the area of a region formed by the pixels with the difference value not being the first preset value is larger than a second preset value or not;
automatically marking the region as a defective region in response to the area of the region being greater than a second predetermined value.
In some embodiments of the present invention, generating the corresponding abstract picture according to the difference between the pixel values of the adjacent images of the main board and the standard template image corresponding to the main board includes:
acquiring RGB channel values of corresponding pixels, and calculating Euclidean distances between the RGB channel values of the pixels and the RGB channel values of the adjacent pixels according to the RGB channel values of the pixels;
judging Euclidean distances of RGB channel values of the pixels and the pixels between adjacent pixels, dividing the pixels with the same Euclidean distances or the difference smaller than a second threshold value into the same area, and uniformly coloring the color of the area according to the color of the most pixels in the area.
In some embodiments of the present invention, obtaining a high-definition image of a motherboard and preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard motherboard image corresponding to the motherboard further includes:
generating a standard vector diagram of the main board according to the standard template image corresponding to the main board and the design drawing of the main board;
generating a vector diagram of the main board from the image to be detected of the main board according to the design drawing of the main board;
scaling the vector diagram and the standard vector diagram of the main board to a preset size, carrying out bitwise subtraction on the vector diagram of the main board scaled to the preset size and the standard vector diagram of the main board, and simultaneously judging whether the bitwise subtraction value of each bitwise pixel is a first preset value or not;
and in response to the bit-wise subtraction value of the pixel being a first preset value, marking the component area to which the pixel belongs as a defect area according to the design drawing of the mainboard.
In some embodiments of the present invention, training an image recognition model according to the saved image to be detected of the motherboard and the comparison result thereof includes:
decomposing the stored image to be detected of the main board into a plurality of local images to be detected, and associating the comparison result of the images to be detected with the corresponding local images to be detected;
and respectively and independently training corresponding image recognition models according to the local images to be detected and the corresponding comparison results thereof.
In some embodiments of the present invention, the performing defect detection on the to-be-detected image of the acquired motherboard by using the trained image recognition model includes:
preprocessing the high-definition image of the mainboard according to the acquired high-definition image of the mainboard in the preprocessing mode of the image to be detected of the trained image recognition model to obtain the image to be detected;
decomposing the image to be detected into a plurality of local images to be detected according to the decomposition mode and respectively inputting the local images to be detected into the image recognition model for detection;
and marking the defect detection result of the main board as a defect in response to the fact that the detection result of any one of the image recognition models is abnormal.
In another aspect of the present invention, a system for detecting an appearance defect of a motherboard device is further provided, including:
the image registration detection module is configured to acquire a high-definition image of a main board and preprocess the high-definition image to obtain an image to be detected, compare the image to be detected of the main board with a standard template image corresponding to the main board, and store the image to be detected of the main board and a comparison result thereof;
the model training module is configured for responding to the stored images to be detected of the mainboard and the comparison results thereof, accumulating the images to be detected of the mainboard and the comparison results thereof to a preset number, and training an image recognition model through the stored images to be detected of the mainboard and the comparison results thereof; and
and the model detection module is configured and used for carrying out defect detection on the acquired to-be-detected image of the mainboard through the trained image recognition model.
Yet another aspect of the present invention is a computer apparatus comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of the above embodiments.
Yet another aspect of the present invention further provides a computer-readable storage medium, which stores a computer program, and the computer program realizes the steps of the method of any one of the above embodiments when executed by a processor.
According to the method, the system, the equipment and the medium for detecting the appearance defects of the mainboard components, the problem that deep learning model reasoning cannot be used due to the fact that defect samples are difficult to collect in the mainboard appearance defect detection process can be effectively solved, and due to the fact that the sizes of the mainboards are different and the types of the mainboard components are different in factory production line production, training of models is completed by collecting defect sample pictures of the components, and then defect detection is not practical. The method comprises the steps of firstly using a template for comparison to collect defect samples, collecting defect sample pictures while detecting defects, and then carrying out model training along with the increase of the sample pictures. The detection of any mainboard product can be used immediately under the condition of no defect sample, and the model training can be carried out along with the increase of the using time, and the using process can be simplified by introducing the model at the later stage.
The method and the device effectively solve the problem that the appearance defect detection of the factory mainboard is difficult to use deep learning model reasoning, accelerate the detection speed, save manpower and material resources and improve the production efficiency.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the drawings without creative efforts.
Fig. 1 is a flowchart of a method for detecting an appearance defect of a motherboard component according to an embodiment of the present invention;
fig. 2 is a schematic structural diagram of a system for detecting an appearance defect of a motherboard component according to an embodiment of the present invention;
FIG. 3 is a schematic structural diagram of a computer device according to an embodiment of the present invention;
fig. 4 is a schematic structural diagram of a computer-readable storage medium according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the following embodiments of the present invention are described in further detail with reference to the accompanying drawings.
It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are used for distinguishing two entities with the same name but different names or different parameters, and it should be noted that "first" and "second" are merely for convenience of description and should not be construed as limitations of the embodiments of the present invention, and they are not described in any more detail in the following embodiments.
As shown in fig. 1, the present invention provides a method for detecting an appearance defect of a motherboard device, including:
s1, acquiring a high-definition image of a mainboard, preprocessing the high-definition image to obtain an image to be detected, comparing the image to be detected of the mainboard through a standard template image corresponding to the mainboard, and storing the image to be detected of the mainboard and a comparison result thereof;
s2, responding to the stored images to be detected of the main board and the comparison results thereof, accumulating to a preset number, and training an image recognition model through the stored images to be detected of the main board and the comparison results thereof; and
and S3, carrying out defect detection on the acquired to-be-detected image of the mainboard through the trained image recognition model.
In the embodiment of the present invention, when step S1 of the present invention is implemented, a high-definition camera is used to photograph a newly produced main board on a product line to form a high-definition image of the main board, and a series of processing manners such as rotational translation and the like are performed on the high-definition image of the main board to correct the main board in the high-definition image, and pixels such as an environment unrelated to the main board in the high-definition image are removed to generate an image to be detected of the main board.
And further detecting the difference, namely the defect, between the main board and the standard template in a mode of registering, matching and comparing the image to be detected of the main board with the standard template image corresponding to the main board, automatically marking the difference area as a defect area, and simultaneously saving the marked image to be detected of the main board and the detection result (if the marked area exists, the detection result is the defect main board, otherwise, the detection result is the normal main board) as a training sample.
In step S2, when the number of the defective motherboards and the normal motherboards automatically labeled in the manner of step S1 reaches a certain number, for example, 2000, or more, an artificial intelligent image recognition model is trained by the above-mentioned sample automatically labeled with the images to be detected of the plurality of motherboards.
In step S3, the main board on the subsequent production line is rapidly detected through the image recognition model trained in step S2. When the mainboard on the production line is detected by using the image recognition model, the picture of the mainboard shot by the industrial camera still needs to be preprocessed to form an image to be detected.
In some embodiments of the present invention, obtaining a high-definition image of a motherboard and preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard motherboard image corresponding to the motherboard includes:
carrying out bit-wise subtraction on the image to be detected of the main board and the pixel corresponding to the standard template image corresponding to the main board to obtain a corresponding difference value, and judging whether the difference value is greater than a first threshold value or not;
and in response to the difference value being larger than a first threshold value, removing the isolated pixels of which the difference value is larger than the first threshold value through a median filtering algorithm, and automatically marking the area of which the difference value is larger than the first threshold value as a defect area.
In this embodiment, when the high-definition image of the acquired motherboard (the motherboard manufactured on the production line) is registered and compared with the image of the standard template of the motherboard of this type, the difference between the two images is calculated by subtracting the two images in a pixel-by-pixel manner. Specifically, three channel pixel values of R, G and B of the two frames of images are subtracted, the obtained maximum difference value is compared with a set threshold value, a pixel with large change is reserved, the reserved isolated pixel is removed through median filtering, and an area with obvious color change and large change range is used as a change area between the two frames of images, namely an abnormal area. For example, if the values of the RGB three-channel pixels are subtracted and then are all greater than 5, the value of the low pixel point is considered to be changed greatly, and if the values of the RGB three-channel pixels of the plurality of pixel points in the pixel range are subtracted and then are all greater than 5, and the area formed by the plurality of pixel points is converted according to the size of the motherboard and then is greater than the area of the minimum component on the motherboard, the area is considered to belong to an abnormal area.
And further automatically marking the area with obvious color change and large change range by adopting a general marking tool, and marking the main board as a defect sample.
It should be noted that, in the above embodiment, the image boundary of the acquired main board needs to be adjusted, and the size of the image to be detected, which is to be compared with the standard template, of the main board and the pixel content of the main board in the image are similar to the standard template. However, since the captured picture is taken by an industrial camera, the focusing and lighting conditions of the camera affect the content of the main board in the imaged picture. For example, in some cases, distances of the same device may be different in two captured main board images, where the difference does not refer to a real distance, but the number of rows or columns of pixels between two devices in two frame images is different, so that when comparing the to-be-detected image of the main board with the standard template, differences generated by bit subtraction due to differences of several rows or several different pixel differences need to be processed. To solve this problem, in some embodiments of the present invention, the difference is formed by ignoring, i.e. if the difference forms a difference of 1 row of pixels or 1 column of pixels, i.e. it cannot form a range of the area of the minimum device, the pixels of the difference can be completely ignored (there may be 1 row of pixels or two rows of pixels, i.e. according to the shape and size of the corresponding device), i.e. the difference resulting from the bitwise subtraction is not treated as a defect, but directly ignored.
In some embodiments of the present invention, obtaining a high definition image of a motherboard and preprocessing the high definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard template image corresponding to the motherboard further includes:
generating a corresponding abstract picture by the image to be detected of the main board and the standard template image corresponding to the main board according to the difference of pixel values between adjacent images;
carrying out bit-by-bit subtraction on the generated abstract picture of the image to be detected of the main board and the generated standard template image corresponding to the main board to obtain a difference value of corresponding pixel bits;
removing the pixels with the difference value not being the first preset value through a median filtering algorithm, and judging whether the area of a region formed by the pixels with the difference value not being the first preset value is larger than a second preset value or not;
automatically marking the region as a defective region in response to the area of the region being greater than a second predetermined value.
In this embodiment, when the acquired high-definition image of the motherboard (the motherboard manufactured on the production line) is registered and compared with the image of the standard template of the motherboard of this type, the high-definition image of the motherboard and the standard image of the motherboard are made into abstract pieces for comparison. The method includes the steps that a mainboard image originally representing a real object is rendered into an abstract picture through the difference of pixels of adjacent high-definition images of the collected mainboard, for example, a chip on the mainboard is usually packaged into black, relevant information about the model of the chip with lighter color is engraved on the outer surface of a packaging layer of the chip through laser, when the image of the mainboard is abstracted in the mode, the color of an area where the chip is located is coated into the color of the chip, namely, the most colors in the appearance colors of different devices on the mainboard are used as the abstract colors of the devices according to the appearance colors of the different devices, for example, the chips are processed, if the chips are black, the black is used as the abstract colors of the area where the chip is located on the mainboard, therefore, the area can hardly be recognized after abstraction without referring to a design drawing, the abstract colors of the area are abstracted according to the most pixels of the area, the information of the corresponding devices can not be recognized in the abstracted image of the mainboard to be detected, the different devices represent different color areas, and the image to be detected at the moment is completely abstracted by a plurality of color spots and does not have the information of the real components on the mainboard.
Similarly, the image of the standard template used for comparing with the image to be detected of the main board is also subjected to the abstraction processing, so as to form an abstracted picture of the standard template with different color areas.
Further, subtracting the abstract picture of the image to be detected of the main board from the abstract picture of the standard template of the main board according to the position to obtain a corresponding difference value, judging whether the difference value is 0, if the difference value is 0, indicating that the pixel in the area belongs to a normal pixel point, and if not, considering that the pixel point is an abnormal pixel point. And meanwhile, removing the pixel points with the difference value not being 0 through a median filtering algorithm, and judging the area formed by the abnormal pixel points. The area where the abnormal pixel points are located is larger than the area of the mainboard minimum component and the area of the abstracted mainboard after the conversion of the image to be detected, and then the area is considered to be abnormal.
In some embodiments of the present invention, when the above bit-wise subtraction is performed, since the camera may have a difference (a template that is earlier) from the color of the standard template after abstracting the high-definition image of the captured main board due to the light environment or due to the influence of the service life of the camera, the bit-wise subtraction is not 0, and therefore, pixels within a certain range of pixel difference values may be considered as normal, that is, considered as 0.
In addition, in some cases, the abstract picture image of the latest normal image to be detected can be dynamically replaced as the abstract picture image of the standard template according to a certain time.
It should be noted that, in the abstracted main board, the image to be detected may also have a condition of a misalignment of 1 pixel or several pixels when two frames of pictures are compared in the above embodiment, and the corresponding scheme in the above embodiment may still be adopted to be ignored. And the situation that the difference value of smaller ranges or isolated pixels is different due to bit-wise subtraction brought by other characters or patterns on the component can be effectively avoided by adopting an abstract mode.
In some embodiments of the present invention, generating the corresponding abstract picture according to the difference between the pixel values of the adjacent images of the main board and the standard template image corresponding to the main board includes:
acquiring RGB channel values of corresponding pixels, and calculating Euclidean distances of the RGB channel values of the pixels and the adjacent pixels according to the RGB channel values of the pixels;
judging Euclidean distances of RGB channel values of the pixels and the pixels between adjacent pixels, dividing the pixels with the same Euclidean distances or the difference smaller than a second threshold value into the same area, and uniformly coloring the color of the area according to the color of the most pixels in the area.
In this embodiment, when abstracting the image to be detected of the motherboard and the standard template image corresponding to the motherboard, what is needed is to determine the range of the abstracted colors or what colors are divided into one region, that is, the region corresponding to the component. For example, when an image regarded as green by naked eyes is represented in the form of an RGB channel, if a color variation range is ignored, combinations of green can be more than ten thousands, that is, (0, 255, 0) in RGB represents green, (150, 222, 159) also represents green (light green), and especially when colors carried by a certain component on a mainboard are gradually varied, color values actually shot by a camera at the same place are different. Therefore, the difference of colors cannot be accurately determined by the single contrast of the values on the RGB channels. Therefore, the embodiment of the invention adopts a mode of calculating the Euclidean distance, namely, the numerical values on the three channels are uniformly calculated.
Specifically, the euclidean distance between a certain pixel and 8 surrounding neighboring pixels is calculated for each pixel, for example, if the RGB values of the certain pixel and the surrounding pixels are the same (up, down, left, right, and obliquely up, down (including 4 pixels in total), the euclidean distance between the certain pixel and the surrounding pixels is 0, and the calculation method can be regarded as:
x1, y1, z1 respectively represent the values of the three RGB channels of the first pixel (current pixel), x2, y2, z2 represent the values of the three RGB channels of the second pixel (the other 8 pixels adjacent to the current pixel), and d represents the euclidean distance between two pixels on the three RGB color channels. The euclidean distance is calculated by comparing the value of any one pixel with the values of the other 8 pixels adjacent thereto, and the pixels having the euclidean distance smaller than 20 are divided into the same region.
Further, after dividing a specific pixel into corresponding regions, the RGB pixel values of the pixels of the region are counted, the square root is calculated for the sum of the pixel values of the same RBG or the pixels of three channels of a single pixel, and the similar pixels are taken as the pixels of the same color. And counting the number of pixel points of each color classification, taking the color with the largest number of pixel points as the main color of the region, and unifying the colors of other pixels in the region into the main color so as to finish abstract color representation of the region.
In some embodiments of the present invention, obtaining a high-definition image of a motherboard and preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the motherboard with a standard motherboard image corresponding to the motherboard further includes:
generating a standard vector diagram of the main board according to the standard template image corresponding to the main board and the design drawing of the main board;
generating a vector diagram of the main board from the image to be detected of the main board according to the design drawing of the main board;
scaling the vector diagram of the main board and the standard vector diagram to a preset size, carrying out bitwise subtraction on the vector diagram of the main board scaled to the preset size and the standard vector diagram of the main board, and simultaneously judging whether the bitwise subtraction value of each bitwise pixel is a first preset value or not;
and in response to the bit-wise subtraction value of the pixel being a first preset value, marking the component area to which the pixel belongs as a defect area according to the design drawing of the mainboard.
In this embodiment, in order to solve the influence of the bit-wise subtraction on the calculation result caused by the displacement of a plurality of pixels of the main board position in the main board high-definition image due to the imaging of the camera, a method for converting the image to be detected of the main board into a vector diagram is provided. The method comprises the steps of determining the distance between corresponding components according to the size of a design drawing on the basis of the design imaging resolution and the design drawing, estimating the position of the corresponding component on an image to be detected of a mainboard, which is converted into the resolution of the image to be detected of the mainboard, on the basis of the resolution of the image to be detected of the mainboard, which is shot by a camera, and then abstracting the color of the corresponding area on the image to be detected according to the processing mode of the color on the mainboard in the embodiment (for example, the color of the area is classified according to RGB channel values of adjacent pixels, and the color with the largest number of pixel points is selected as the abstract color of the area).
Likewise, the standard template image for the motherboard is still processed and converted into a corresponding vector image in the same manner as described above.
Furthermore, after the image to be detected of the main board and the standard template image are both converted into vector diagrams, the image to be detected and the standard template image can be converted into images with corresponding resolution ratios according to the computing power, and then bitwise subtraction is carried out. Due to the characteristics of the vector image, even if the resolution ratio of the vector image is reduced, the distribution of components of the mainboard can still be represented, the distribution of the components of the mainboard cannot be influenced no matter how many times the design drawing of the circuit board is amplified in 3D software, and operations such as bitwise subtraction of the image to be detected and the like consume extremely high computing resources, so that the efficiency of registration and comparison can be effectively improved through the method.
In some embodiments of the present invention, when generating the vector diagram for the image to be detected of the motherboard, the motherboard design file may be exported into a design diagram with the same size as the image to be detected of the motherboard, and all colors on the design diagram are removed, and under the condition of the same size, the corresponding component unit on the design diagram is colored according to the colors on the image to be detected of the motherboard, and the vector diagram for the image to be detected of the motherboard is generated by using the colored design diagram as the vector diagram for the image to be detected of the motherboard. And compared with the vector diagram of the standard template after being reduced to a proper resolution.
It should be noted that, if the design file of the motherboard is obtained and the corresponding device in the design file can be inserted into the real appearance picture of the device, the vector diagram of the corresponding standard template can be directly derived through the design file, and a manner of manually selecting a qualified motherboard as the standard template of the motherboard to regenerate the vector diagram of the corresponding standard template is not needed at the beginning, thereby completely realizing automation.
In some embodiments of the present invention, training an image recognition model according to the saved image to be detected of the motherboard and the comparison result thereof includes:
decomposing the stored image to be detected of the main board into a plurality of local images to be detected, and associating the comparison result of the images to be detected with the corresponding local images to be detected;
and respectively and independently training corresponding image recognition models according to the local images to be detected and the corresponding comparison results.
In some embodiments of the present invention, the performing defect detection on the acquired to-be-detected image of the motherboard by using the trained image recognition model includes:
preprocessing the high-definition image of the mainboard into an image to be detected according to the acquired high-definition image of the mainboard in a preprocessing mode of the image to be detected for training the image recognition model;
decomposing the image to be detected into a plurality of local images to be detected according to the decomposition mode and respectively inputting the images to be detected into the plurality of image recognition models for detection;
and marking the defect detection result of the main board as a defect in response to the fact that the detection result of any one of the image recognition models is abnormal.
In this embodiment, it should be noted that resolution of the output image of the industrial camera adopted by the present invention is different according to different sizes of the main boards, in the actual production, the circuit board on the production line may be the main board of the server, and the main board of the server is different in size due to different versions, therefore, in order to clearly present the appearance layout of some large main boards, a higher resolution must be used, and therefore, in order to ensure image quality, it is preferable to process images with a resolution specification of 5000 × 5000 pixels on the main board of the server. However, no matter what kind of image recognition model is, the calculation of enough large pixels cannot be supported for image recognition, generally, most common image recognition engines support data with a size of 640 × 640 convolution, and for recognition of a server motherboard, such a small resolution ratio can hardly recognize corresponding small components, such as some smaller patch capacitors or some other smaller components such as some solder joints, which are already eliminated during data preprocessing of the image recognition algorithm model or when the image recognition algorithm model passes through the first convolution layer.
Therefore, in order to improve the accuracy of the image recognition model, the image to be detected is split into a plurality of small images to respectively and correspondingly train a plurality of models, for example, for the image to be detected with 5000 × 5000 pixels, the image to be detected is split into 49 blocks with 7 × 7, 49 image recognition models are respectively trained, and then the image is recognized through 49 models.
And when the image recognition model is used for detection, when a plurality of recognition models exist, the detection result of the main board is abnormal as long as 1 detection result is abnormal.
In addition, it should be noted that in some small-sized motherboards or circuit boards, for example, the resolution of the images to be detected can be completely reduced, and the identification can be quickly completed through an image identification model.
In addition, when the image recognition model is trained, a plurality of image recognition models may be trained after splitting the to-be-detected image of the motherboard, or the to-be-detected image of the motherboard may be trained after being subjected to abstraction processing and the like according to the above embodiment, and when the corresponding image recognition model is used for recognition, the to-be-detected image of the motherboard to be detected is subjected to the same abstraction processing and the like in the above manner, and then is recognized by the image recognition model.
As shown in fig. 2, another aspect of the present invention further provides a system for detecting an appearance defect of a motherboard component, including:
the image registration detection module 1 is configured to acquire a high-definition image of a main board and preprocess the high-definition image to obtain an image to be detected, compare the image to be detected of the main board with a standard template image corresponding to the main board, and store the image to be detected of the main board and a comparison result thereof;
the model training module 2 is configured to respond to the stored images to be detected of the main board and the comparison results thereof, accumulate the images to be detected of the main board and the comparison results thereof to a preset number, and train an image recognition model through the stored images to be detected of the main board and the comparison results thereof; and
and the model detection module 3 is configured to perform defect detection on the acquired to-be-detected image of the main board through the trained image recognition model.
As shown in fig. 3, another aspect of the present invention also provides a computer device, including:
at least one processor 21; and
a memory 22, said memory 22 storing computer instructions 23 executable on said processor 21, said instructions 23 when executed by said processor 21 implementing the steps of the method of any of the above embodiments.
As shown in fig. 4, a further aspect of the present invention also provides a computer-readable storage medium 401, where the computer-readable storage medium 401 stores a computer program 402, and the computer program 402 implements the steps of the method according to any one of the above embodiments when being executed by a processor.
According to the method, the system, the equipment and the medium for detecting the appearance defects of the mainboard components, the problem that deep learning model reasoning cannot be used due to the fact that defect samples are difficult to collect in the mainboard appearance defect detection process can be effectively solved, and due to the fact that the sizes of the mainboards are different and the types of the mainboard components are different in factory production line production, training of models is completed by collecting defect sample pictures of the components, and then defect detection is not practical. The method comprises the steps of firstly using a template for comparison to collect defect samples, collecting defect sample pictures while detecting defects, and then carrying out model training along with the increase of the sample pictures. The detection of any mainboard product can be used immediately under the condition of no defect sample, and the model training can be carried out along with the increase of the using time, and the using process can be simplified by introducing the model at the later stage.
The method and the device effectively solve the problem that the appearance defect detection of the factory mainboard is difficult to use deep learning model reasoning, accelerate the detection speed, save manpower and material resources and improve the production efficiency.
In addition, as can be seen from the exemplary embodiment of the present invention, the present invention can have a completely automatic implementation capability, and in some cases, only at the beginning of the design of the motherboard, the real object image of the corresponding component is added to the design file and exported to the corresponding picture, so that a completely automatic detection process can be implemented without human intervention.
The foregoing is an exemplary embodiment of the present disclosure, but it should be noted that various changes and modifications could be made herein without departing from the scope of the present disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the disclosed embodiments described herein need not be performed in any particular order. Furthermore, although elements of the embodiments of the invention may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.
It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly supports the exception. It should also be understood that "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
The numbers of the embodiments disclosed in the embodiments of the present invention are merely for description, and do not represent the merits of the embodiments.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the storage medium may be a read-only memory, a magnetic disk or an optical disk.
Those of ordinary skill in the art will understand that: the discussion of any embodiment above is meant to be exemplary only, and is not intended to intimate that the scope of the disclosure, including the claims, of embodiments of the invention is limited to these examples; within the idea of an embodiment of the invention, also technical features in the above embodiment or in different embodiments may be combined and there are many other variations of the different aspects of an embodiment of the invention as described above, which are not provided in detail for the sake of brevity. Therefore, any omissions, modifications, substitutions, improvements, and the like that may be made without departing from the spirit and principles of the embodiments of the present invention are intended to be included within the scope of the embodiments of the present invention.
Claims (8)
1. A method for detecting appearance defects of a mainboard component is characterized by comprising the following steps:
acquiring a high-definition image of a mainboard, preprocessing the high-definition image to obtain an image to be detected, comparing the image to be detected of the mainboard through a standard template image corresponding to the mainboard, and storing the image to be detected of the mainboard and a comparison result thereof;
responding to the stored images to be detected of the main board and comparison results thereof are accumulated to a preset number, and training an image recognition model through the stored images to be detected of the main board and comparison results thereof; and
carrying out defect detection on the acquired image to be detected of the mainboard through the trained image recognition model;
the method comprises the following steps of obtaining a high-definition image of a main board, preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the main board through a standard template image corresponding to the main board, wherein the steps of:
generating a corresponding abstract picture by the image to be detected of the main board and the standard template image corresponding to the main board according to the difference of pixel values between adjacent images;
carrying out bit-wise subtraction on the generated abstract picture of the image to be detected of the main board and the generated standard template image corresponding to the main board to obtain a difference value of corresponding pixel bits;
removing the pixels with the difference value not being the first preset value through a median filtering algorithm, and judging whether the area of a region formed by the pixels with the difference value not being the first preset value is larger than a second preset value or not;
automatically marking the region as a defective region in response to the area of the region being greater than a second predetermined value; or
Generating a standard vector diagram of the main board according to the standard template image corresponding to the main board and the design drawing of the main board;
scaling the vector diagram and the standard vector diagram of the main board to a preset size, carrying out bitwise subtraction on the vector diagram of the main board scaled to the preset size and the standard vector diagram of the main board, and simultaneously judging whether the bitwise subtraction value of each bitwise pixel is a first preset value or not;
and in response to the bit-wise subtraction value of the pixel being a first preset value, marking the component area to which the pixel belongs as a defect area according to the design drawing of the mainboard.
2. The method according to claim 1, wherein the obtaining of the high-definition image of the main board and the preprocessing of the high-definition image are performed to obtain an image to be detected, and the comparing of the image to be detected of the main board with the standard template image corresponding to the main board comprises:
carrying out bit-wise subtraction on the image to be detected of the main board and the pixel corresponding to the standard template image corresponding to the main board to obtain a corresponding difference value, and judging whether the difference value is greater than a first threshold value or not;
and in response to the difference value being larger than a first threshold value, removing the isolated pixels with the difference value being larger than the first threshold value through a median filtering algorithm, and automatically marking the area with the difference value being larger than the first threshold value as a defect area.
3. The method according to claim 1, wherein the generating of the corresponding abstracted picture by the image to be detected of the main board and the standard template image corresponding to the main board according to the difference of the pixel values between the adjacent main boards comprises:
acquiring RGB channel values of corresponding pixels, and calculating Euclidean distances between the RGB channel values of the pixels and the RGB channel values of the adjacent pixels according to the RGB channel values of the pixels;
judging Euclidean distances of RGB channel values of the pixels and the pixels between adjacent pixels, dividing the pixels with the same Euclidean distances or the difference smaller than a second threshold value into the same area, and uniformly coloring the color of the area according to the color of the most pixels in the area.
4. The method according to claim 1, wherein the training of the image recognition model through the saved images to be detected of the main board and the comparison result thereof comprises:
decomposing the stored image to be detected of the main board into a plurality of local images to be detected, and associating the comparison result of the images to be detected with the corresponding local images to be detected;
and respectively and independently training corresponding image recognition models according to the local image to be detected and the corresponding comparison result.
5. The method according to claim 4, wherein the defect detection of the acquired image to be detected of the main board through the trained image recognition model comprises:
preprocessing the high-definition image of the mainboard according to the preprocessing mode of the image to be detected of the trained image recognition model of the acquired high-definition image of the mainboard to obtain the image to be detected;
decomposing the image to be detected into a plurality of local images to be detected according to the decomposition mode and respectively inputting the images to be detected into the image recognition model for detection;
and in response to the fact that the detection result of any one image recognition model is abnormal, marking the defect detection result of the mainboard as a defect.
6. The utility model provides a mainboard components and parts appearance defect detecting system which characterized in that includes:
the image registration detection module is configured to acquire a high-definition image of a main board and preprocess the high-definition image to obtain an image to be detected, compare the image to be detected of the main board with a standard template image corresponding to the main board, and store the image to be detected of the main board and a comparison result thereof;
the model training module is configured for responding to the stored images to be detected of the mainboard and the comparison results thereof, accumulating the images to be detected of the mainboard and the comparison results thereof to a preset number, and training an image recognition model through the stored images to be detected of the mainboard and the comparison results thereof; and
the model detection module is configured to perform defect detection on the acquired to-be-detected image of the mainboard through the trained image recognition model;
the method comprises the following steps of obtaining a high-definition image of a main board, preprocessing the high-definition image to obtain an image to be detected, and comparing the image to be detected of the main board through a standard template image corresponding to the main board, wherein the steps of:
generating a corresponding abstract picture by the image to be detected of the main board and the standard template image corresponding to the main board according to the difference of pixel values between adjacent images;
carrying out bit-wise subtraction on the generated abstract picture of the image to be detected of the main board and the generated standard template image corresponding to the main board to obtain a difference value of corresponding pixel bits;
removing the pixels of which the difference values are not the first preset value through a median filtering algorithm, and judging whether the area of a region formed by the pixels of which the difference values are not the first preset value is larger than a second preset value or not;
automatically marking the region as a defective region in response to the area of the region being greater than a second predetermined value; or
Generating a standard vector diagram of the main board according to the standard template image corresponding to the main board and the design drawing of the main board;
scaling the vector diagram and the standard vector diagram of the main board to a preset size, carrying out bitwise subtraction on the vector diagram of the main board scaled to the preset size and the standard vector diagram of the main board, and simultaneously judging whether the bitwise subtraction value of each bitwise pixel is a first preset value or not;
and in response to the bit-wise subtraction value of the pixel being a first preset value, marking the component area to which the pixel belongs as a defect area according to a design drawing of the mainboard.
7. A computer device, comprising:
at least one processor; and
a memory storing computer instructions executable on the processor, the instructions when executed by the processor implementing the steps of the method of any one of claims 1 to 5.
8. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210171790.4A CN114240939B (en) | 2022-02-24 | 2022-02-24 | Method, system, equipment and medium for detecting appearance defects of mainboard components |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN202210171790.4A CN114240939B (en) | 2022-02-24 | 2022-02-24 | Method, system, equipment and medium for detecting appearance defects of mainboard components |
Publications (2)
Publication Number | Publication Date |
---|---|
CN114240939A CN114240939A (en) | 2022-03-25 |
CN114240939B true CN114240939B (en) | 2023-04-07 |
Family
ID=80748036
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN202210171790.4A Active CN114240939B (en) | 2022-02-24 | 2022-02-24 | Method, system, equipment and medium for detecting appearance defects of mainboard components |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN114240939B (en) |
Families Citing this family (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN114782437B (en) * | 2022-06-20 | 2022-09-02 | 西南石油大学 | Computer mainboard quality detection method and system based on artificial intelligence |
CN115880288B (en) * | 2023-02-21 | 2023-11-14 | 深圳市兆兴博拓科技股份有限公司 | Detection method, system and computer equipment for electronic element welding |
CN116075148B (en) * | 2023-03-14 | 2023-06-20 | 四川易景智能终端有限公司 | PCBA board production line intelligent supervision system based on artificial intelligence |
CN115965646B (en) * | 2023-03-16 | 2023-07-04 | 深圳思谋信息科技有限公司 | Region dividing method, device, computer equipment and computer readable storage medium |
CN117129480B (en) * | 2023-10-25 | 2024-02-13 | 深圳市吉方工控有限公司 | Intelligent detection method and device for computer main board components based on machine vision |
CN117274258A (en) * | 2023-11-21 | 2023-12-22 | 深圳市研盛芯控电子技术有限公司 | Method, system, equipment and storage medium for detecting defects of main board image |
CN117974662A (en) * | 2024-03-29 | 2024-05-03 | 东莞市昌盛电子制品有限公司 | Chip detection method, electronic equipment and storage medium |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140126839A1 (en) * | 2012-11-08 | 2014-05-08 | Sharp Laboratories Of America, Inc. | Defect detection using joint alignment and defect extraction |
CN105160654A (en) * | 2015-07-09 | 2015-12-16 | 浙江工商大学 | Towel label defect detecting method based on feature point extraction |
CN112070747A (en) * | 2020-09-09 | 2020-12-11 | 深兰人工智能芯片研究院(江苏)有限公司 | LED lamp bead defect detection method and device |
-
2022
- 2022-02-24 CN CN202210171790.4A patent/CN114240939B/en active Active
Also Published As
Publication number | Publication date |
---|---|
CN114240939A (en) | 2022-03-25 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN114240939B (en) | Method, system, equipment and medium for detecting appearance defects of mainboard components | |
CN111179251B (en) | Defect detection system and method based on twin neural network and by utilizing template comparison | |
CN111612763B (en) | Mobile phone screen defect detection method, device and system, computer equipment and medium | |
CN110060237B (en) | Fault detection method, device, equipment and system | |
CN108562589A (en) | A method of magnetic circuit material surface defect is detected | |
CN109767445B (en) | High-precision PCB defect intelligent detection method | |
WO2022236876A1 (en) | Cellophane defect recognition method, system and apparatus, and storage medium | |
CN111815564B (en) | Method and device for detecting silk ingots and silk ingot sorting system | |
CN111738994B (en) | Lightweight PCB defect detection method | |
CN111445459A (en) | Image defect detection method and system based on depth twin network | |
CN110533654A (en) | The method for detecting abnormality and device of components | |
CN113989794B (en) | License plate detection and recognition method | |
CN115661161B (en) | Defect detection method, device, storage medium, apparatus and program product for parts | |
CN113780484B (en) | Industrial product defect detection method and device | |
CN115578585A (en) | Industrial image anomaly detection method, system, computer device and storage medium | |
CN111882547A (en) | PCB missing part detection method based on neural network | |
CN114187247A (en) | Ampoule bottle printing character defect detection method based on image registration | |
CN117078677A (en) | Defect detection method and system for starting sheet | |
CN105354833A (en) | Shadow detection method and apparatus | |
CN112861861B (en) | Method and device for recognizing nixie tube text and electronic equipment | |
CN111935480B (en) | Detection method for image acquisition device and related device | |
CN115410184A (en) | Target detection license plate recognition method based on deep neural network | |
CN114708247A (en) | Cigarette case packaging defect identification method and device based on deep learning | |
CN115330688A (en) | Image anomaly detection method considering tag uncertainty | |
CN113592789A (en) | Dim light image identification method, device, equipment and storage medium |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
GR01 | Patent grant | ||
GR01 | Patent grant |