TWI796681B - A method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning - Google Patents

Abstract

The present invention proposes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, including: training each type of defect in the PCB defect image library, dividing the PCB image to be tested into multiple image blocks; using PCB defect features to search for defects Identify the defects in the PCB picture to be tested and mark the suspected defective blocks; use the PCB defect category feature to classify the defects, judge the defect category, judge whether the suspected defects that are not in the category are false defects or defects of a new category, and add the new category of defects to Defect image library; use PCB defect level features to judge suspected defects. If it is a false defect, delete the corresponding defect and analyze the cause. If it is a true defect, mark it in the defect record. The method detects PCB defects with high speed and high precision.

Description

A method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning

The present invention relates to the technical field of defect detection, in particular to a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning.

Two-dimensional (2D) printed circuit board (PCB) is the basis for the production of various microcircuit boards and motherboards, and its correctness is the guarantee for the correctness of other subsequent processes. Due to the continuous development of modern technology and fine craftsmanship, the production of PCB is becoming more and more complex and precise. Traditional visual inspection is no longer suitable for complex PCB inspection.

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

For this reason, the object of the present invention is to propose a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, which has the advantages of fast speed, high precision, strong generalization ability, and clear structure.

In order to achieve the above purpose, the embodiment of the present invention proposes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, including:

Step S1, establishing a two-dimensional PCB defect block image library by collecting and marking PCB defect block images, the two-dimensional PCB defect block image library includes a two-dimensional PCB defect data pair image library and a two-dimensional PCB defect single data image library;

Step S2, using a convolutional neural network (CNN) to train each type of PCB defect in the two-dimensional PCB defect block image library to extract PCB defect category features;

Step S3, using a convolutional neural network to train the level of each type of PCB defect in the two-dimensional PCB defect block image library, and extracting features of the level of PCB defect;

Step S4, extracting a two-dimensional PCB image through the PCB production line, the two-dimensional PCB image includes a two-dimensional standard PCB image and a two-dimensional PCB image to be tested;

Step S5, performing denoising processing on the two-dimensional PCB image by filtering the denoising block;

Step S6, registering the two-dimensional standard PCB image with the two-dimensional standard PCB image through the image registration block;

Step S7, compare and analyze the gray value of the two-dimensional standard PCB image and the two-dimensional PCB image to be tested through the image gray value comparison block to obtain a two-dimensional PCB residual image;

Step S8: Segment and locate the two-dimensional standard PCB picture, the two-dimensional PCB-to-be-tested picture and the two-dimensional PCB residual picture through image segmentation and positioning blocks to obtain multiple picture blocks and corresponding coordinate;

Step S9, filter the image block of the two-dimensional PCB residual image through the residual filter block, and eliminate the comparison error caused by registration and other factors;

Step S10, defect searching and matching blocks, using the PCB defect features trained by the convolutional neural network, searching for defect blocks in multiple image blocks, and performing defect block matching;

Step S11, the CNN defect identification block identifies the defects in the two-dimensional PCB image to be tested through a convolutional neural network;

Step S12, the PCB defect classification block, classifies the defects marked in the two-dimensional PCB image to be tested through the PCB defect category features trained by the convolutional neural network, and judges the defects that are not in the category to determine whether it is false The defect is a defect of a new category, and the defect of the new category is added to the defect block picture library;

Step S13, marking each type of defect in the two-dimensional PCB-to-be-test image as a suspected defect through the PCB defect suspected block;

Step S14, the PCB defect verification block, judges the authenticity of the suspected defect block through the PCB defect level features and classification information trained by the convolutional neural network. If the defect is a false defect, delete the corresponding defect, and analyze the cause of the false defect. The reason, if the defect is a true defect, is marked in the defect record.

The real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning in the embodiment of the present invention uses PCB images, standard PCB images and convolutional neural networks to detect defects on PCBs, and uses PCB detection tools to detect defects in the assembly process Finding and eliminating errors early in the process can avoid sending defective PCB boards to subsequent assembly stages, while reducing repair costs and avoiding scrapping non-repairable boards. The use of convolutional neural networks has the advantages of fast speed, high precision, strong generalization ability, and clear structure.

In addition, the real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in one embodiment of the present invention, the two-dimensional PCB defect block image library is divided into two categories, one is a data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block images, and the other is A data image library for only a single 2D PCB defect block.

Further, in an embodiment of the present invention, the step S1 also includes:

Build a two-dimensional PCB defect block picture library. Two types of two-dimensional PCB defect block image libraries contain various labeled, classified and graded PCB defect block images of PCBs, and the PCB defect block images include multi-scale images.

Further, in an embodiment of the present invention, the steps S2 and S3 also include:

For each type of PCB defect in the two-dimensional PCB defect block image library, train with a convolutional neural network to extract PCB defect category features;

For the level of each type of PCB defect in the two-dimensional PCB defect block image library, a convolutional neural network is used to train and extract PCB defect level features.

Further, in one embodiment of the present invention, in step S4, the two-dimensional PCB pictures extracted by the PCB picture extraction block include two types: one is to extract two-dimensional standard PCB pictures and two-dimensional PCB pictures to be tested; the other is To extract only two-dimensional PCB pictures to be tested.

Further, in one embodiment of the present invention, the step S4 also includes:

In comparing the detection method of the standard PCB and the hybrid detection method of the PCB, extract the two-dimensional picture of the PCB to be tested and the corresponding two-dimensional standard PCB picture;

In the non-standard board comparison PCB detection method, only two-dimensional pictures of the PCB to be tested are extracted.

Further, in one embodiment of the present invention, in the step S5, the filtering and denoising block performs denoising processing on the two-dimensional PCB picture, including:

Using an algorithm to denoise the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested, in the comparison of the standard PCB detection method and the PCB hybrid detection method, the two-dimensional standard PCB picture and the two-dimensional standard PCB picture are denoised. Dimensional denoising of the PCB image to be tested, removing the noise in the image, and correcting the influence of the color of the bottom plate on the image, so that the color of the bottom plate of the standard PCB image, the PCB image to be tested, and the pictures in the defect library are consistent;

In the non-standard board comparison PCB inspection method, only the two-dimensional image of the PCB to be tested is denoised, the noise in the image is removed, and the influence of the color of the bottom plate on the image is corrected, so that the bottom plate of the picture of the PCB to be tested and the picture in the defect library The color is consistent.

Further, in one embodiment of the present invention, registration, gray value comparison, and image segmentation are performed on the two-dimensional standard PCB picture and the two-dimensional test PCB picture, further comprising:

In comparing the detection method of the standard PCB and the mixed detection method of the PCB, the steps S6, S7 and S8 include:

In comparing the detection method of the standard PCB and the mixed detection method of the PCB, the two-dimensional standard PCB image and the two-dimensional PCB image to be tested are registered, and the two-dimensional standard PCB image is compared according to the denoising processed two-dimensional standard PCB image. The two-dimensional standard PCB picture to be tested is corrected, and the two-dimensional standard PCB picture is compared with the gray value of the two-dimensional PCB picture to be tested, and the picture residual is calculated to obtain a two-dimensional PCB residual picture; the two-dimensional standard PCB picture is , dividing the two-dimensional PCB image to be tested and the two-dimensional PCB residual image to obtain a plurality of image blocks and corresponding coordinates;

Registering the two-dimensional standard PCB picture, detecting whether there is rotation, deformation, light unevenness and light reflection in the two-dimensional standard PCB picture, and if there is, using an algorithm to correct it;

The two-dimensional PCB image to be tested is compared with the processed two-dimensional standard PCB image to determine whether the position, light and color are consistent, and if not, the algorithm is used to correct the two-dimensional PCB image to be tested.

Further, in one embodiment of the present invention, in the non-comparison reference detection method, step S8 further includes: segmenting the two-dimensional PCB-to-be-test image to obtain a plurality of image blocks and corresponding coordinates.

Further, in one embodiment of the present invention, in the step S9, filter the image block of the two-dimensional PCB residual image, eliminate the comparison error caused by registration and other factors, and delete the two-dimensional PCB There are no residual picture blocks in the residual picture;

The step S10 and step S12 further include:

In the non-standard board comparison PCB detection method, the two-dimensional PCB defect single data picture library is used;

In the non-standard board comparison PCB detection method and the PCB hybrid detection method, for each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, use convolutional neural network training , extracting PCB defect features and PCB defect category features;

Step S10 uses the PCB defect features trained by the convolutional neural network, and the step S8 performs PCB defect search and discrimination in multiple image blocks, and uses a single two-dimensional PCB defect image library convolutional neural network to train the category features of PCB defects, Step S12 classifies the defects marked in the two-dimensional PCB image to be tested, judges the defects that are not in the category, judges whether it is a false defect or a defect of a new category, and adds the defects of the new category to the defect photo gallery.

Further, in one embodiment of the present invention, in comparing the standard PCB detection method and the PCB hybrid detection method, each of the two-dimensional PCB defect image library and the two-dimensional PCB defect image library Types of PCB defects, trained with convolutional neural networks, to extract PCB defect features and PCB defect category features. Step S11 and step S12 also include:

Using the two-dimensional PCB defect data pair picture library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, step S11 uses the PCB defect feature pairs trained by the convolutional neural network, and step S9 performs defect identification on the residual images , using the two-dimensional PCB defect picture library convolutional neural network to train the category feature pairs of PCB defects, step S12 classifies the defects marked in the two-dimensional PCB picture to be tested, and judges the defects that are not in the category, and judges whether Whether it is a false defect or a defect of a new category, the defect of the new category is added to the defect image library.

Further, in one embodiment of the present invention, in the PCB hybrid detection method, after step S12, it also includes:

Using the PCB defect features of the two-dimensional PCB defect image library trained by the convolutional neural network to search and match the marked suspected defect blocks, delete false defects therein, and improve recognition accuracy;

After step S13, the suspected defect is searched for a match in the defect picture through the PCB defect feature trained by the convolutional neural network.

Further, in one embodiment of the present invention, for PCB defect verification, the PCB defect level feature information trained by the convolutional neural network in step S3 is used to judge the authenticity of the PCB defect, and if the defect is a false defect, delete the defect. False defects, and analyze the cause of false defects, if the defect is a true defect, it will be marked in the defect record.

Furthermore, in an embodiment of the present invention, in the process of searching and judging defects, the defects are eliminated successively through multiple identification iterations.

Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.

Embodiments of the present invention are described in detail below, examples of which are shown in the drawings, wherein the same or similar reference numerals designate the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the figures are exemplary and are intended to explain the present invention and should not be construed as limiting the present invention.

The following describes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to an embodiment of the present invention with reference to the drawings.

With the development of technology, the production of PCB is becoming more and more complex and precise. The existing defect detection methods are not efficient and have low precision. The embodiment of the present invention proposes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, which specifically includes three embodiments of defect detection methods for PCB defect detection. Among them, one is a CNN detection method that relies on standard PCB board comparison (called method 1), the second is a CNN detection method without standard board comparison (called method 2), and the third is the first two Method Hybrid CNN's PCB detection method (referred to as method three).

As shown in Figure 1, the real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning in the embodiment of the present invention includes:

Step S1, the PCB defect block image library, establishes a two-dimensional PCB defect block (patch) image library, which respectively has a two-dimensional PCB defect data pair image library and a two-dimensional PCB defect single data image library;

Step S2, CNN defect feature training block, for each type of PCB defect in the two-dimensional PCB defect block image library, use convolutional neural network (CNN) training to extract PCB defect category features;

Step S3, CNN defect level feature training block, for each type of PCB defect level in the two-dimensional PCB defect block image library, use convolutional neural network training to extract PCB defect level features;

Step S4, the PCB image extraction block extracts two-dimensional PCB images, which respectively include a two-dimensional standard PCB image and a two-dimensional PCB image to be tested;

Step S5, filter the denoising block, and perform denoising processing on the two-dimensional PCB image;

Step S6, the image registration block, registering the two-dimensional standard PCB image and the two-dimensional PCB image to be tested, so that the two-dimensional standard PCB image and the two-dimensional PCB image to be tested are aligned;

Step S7, image gray value comparison block, compare and analyze the gray value of the two-dimensional standard PCB image and the two-dimensional PCB image to be tested to obtain the two-dimensional PCB residual image;

Step S8, image segmentation and positioning block, segmenting and positioning the two-dimensional standard PCB picture, the two-dimensional PCB to be tested picture and the two-dimensional PCB residual picture, and obtaining the blocks (patches) and corresponding coordinates of multiple pictures;

Step S9, the residual filter block is used to patch filter the two-dimensional PCB residual image to eliminate comparison errors caused by registration and other factors;

Step S10, defect searching and matching blocks, using the PCB defect features trained by the convolutional neural network, searching for defect blocks in multiple image blocks, and performing defect block matching;

Step S11, the CNN defect identification block identifies defects in the PCB image to be tested through a convolutional neural network;

Step S12, the PCB defect classification block, classifies the defects marked in the PCB picture to be tested through the PCB defect category features trained by the convolutional neural network, and judges the defects that are not in the category to determine whether it is a false defect or a new category Defects of new categories are added to the defect image library;

Step S13, PCB defect suspected block, marking each type of defect in the two-dimensional PCB image to be tested as a suspected defect;

Step S14, the PCB defect verification block, judges the authenticity of the suspected defect block through the PCB defect level features and classification information trained by the convolutional neural network. If the defect is a false defect, delete the corresponding defect, and analyze the cause of the false defect. The reason, if the defect is a true defect, is marked in the defect record.

In the embodiment of the present invention, the two-dimensional PCB defect block image library is divided into two categories, one is a data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block images, and the other is only a single two-dimensional PCB defect Block data image library.

In an embodiment of the present invention, on the basis of the above embodiments, step S1 further includes:

Two types of two-dimensional PCB defect block image libraries contain a variety of PCB defect block images that have been marked, classified and graded, and the PCB defect block images include multi-scale images.

In an embodiment of the present invention, step S2 and step S3 also include:

For each type of PCB defect in the two-dimensional PCB defect block image library, use convolutional neural network (CNN) training to extract PCB defect category features;

For the level of each type of PCB defect in the two-dimensional PCB defect block image library, use convolutional neural network training to extract PCB defect level features.

In an embodiment of the present invention, it is characterized in that step S4 includes extracting a two-dimensional PCB picture. There are two situations, extracting 2D standard PCB pictures and 2D PCB pictures to be tested, and extracting only PCB pictures to be tested.

Step S4 also includes:

For detection methods 1 and 3 that require comparison technology, it is necessary to extract two-dimensional images of the PCB to be tested and the corresponding standard PCB images. For method 2 of unsupervised learning detection (that is, non-comparison reference learning detection method), only two-dimensional PCB images to be tested need to be extracted.

In the embodiment of the present invention, step S5 denoises the two-dimensional PCB picture by filtering the denoising block.

Use the algorithm to denoise the standard PCB picture and the PCB picture to be tested. Method 1 and method 3 need to denoise the two-dimensional standard PCB picture and the PCB picture to be tested, remove the noise in the picture, and correct the influence of the color of the bottom plate on the picture, so that the standard PCB picture, the PCB picture to be tested, and the pictures in the defect library The color of the bottom plate is consistent; the second method only needs to denoise the picture of the PCB to be tested, remove the noise in the picture, and correct the influence of the color of the bottom plate on the picture, so that the color of the bottom plate of the picture of the PCB to be tested and the picture in the defect library is consistent.

Registration, gray value comparison, and image segmentation are performed on the standard PCB image and the PCB image to be tested.

Step S6, step S7 and step S8 are used for method one and method three. Register the standard PCB picture and the PCB picture to be tested, and correct the PCB picture to be tested according to the processed standard PCB picture. Compare the gray value of the standard PCB picture and the PCB picture to be tested, and calculate the picture residual to obtain the residual picture; then divide the standard PCB picture, the PCB picture to be tested and the residual picture to obtain multiple picture blocks and coordinates.

Register the standard PCB picture, detect whether there is rotation, deformation, uneven light and light reflection in the standard PCB picture, and if there is, use the algorithm to correct it.

Compare the picture of the PCB to be tested with the processed standard PCB picture to judge whether the position, light and color are consistent. If not, the picture of the PCB to be tested is corrected by the algorithm.

The image segmentation and positioning step S8 is used in the second method, and the PCB image to be tested is segmented to obtain multiple image blocks and coordinates.

In an embodiment of the present invention, step S9 filters the patch of the two-dimensional PCB residual image, eliminates comparison errors due to registration and other factors, and deletes picture blocks without residual in the residual image;

For each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, use convolutional neural network training to extract PCB defect features and PCB defect category features. Step S10 and step S12 also include:

The second method uses only a single two-dimensional PCB defect block data image library. Step S10 uses the PCB defect features trained by CNN, and step S8 performs PCB defect search and discrimination. Then use a single two-dimensional PCB defect picture library CNN to train the category features of PCB defects, and step S12 classifies the defects marked in the PCB picture to be tested, and judges the defects that are not in the category to determine whether it is a false defect or a new type of defect. , adding defects of a new category to the defect image library.

In the embodiment of the present invention, it is characterized in that, for each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, use convolutional neural network training to extract PCB defect features, PCB defect category features. Step S11 and step S12 also include:

Method 1 and Method 3 use a data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block images, step S11 uses the PCB defect features trained by CNN, and performs defect identification on the residual image in step S9. Then use the two-dimensional PCB defect picture library CNN to train the category characteristics of PCB defects, classify the defects marked in the PCB picture to be tested in step S12, and judge the defects that are not in the category, and judge whether it is a false defect or a new type of defect , adding defects of a new category to the defect image library.

After step S12, method three also includes:

Use the PCB defect features of the two-dimensional PCB defect image library trained by the convolutional neural network to search and match the marked suspected defect blocks, delete the false defects, and improve the recognition accuracy.

After step S13, the suspected defect is searched for a match in the defect picture through the PCB defect feature trained by the convolutional neural network.

In an embodiment of the present invention, for PCB defect verification, the PCB defect level feature information trained by the convolutional neural network in step S3 is used to judge the authenticity of the PCB defect, and if the defect is a false defect, then delete the false defect, and Analyze the cause of the false defect, if the defect is a true defect, mark it in the defect record.

In the embodiment of the present invention, it is characterized in that, in the process of searching and judging defects, the defects are eliminated successively through multiple identification iterations.

To sum up, the embodiment of the present invention proposes three defect detection methods, and the three detection methods are described in detail below through diagrams.

method one:

Detection based on deep learning, as shown in Figure 2, includes the following steps:

1-1) Step S1 of establishing a two-dimensional PCB defect picture library. The defect library contains a variety of labeled, classified and graded defect images for PCBs. The defect picture may be a multi-scale picture. The defect picture is a picture data pair composed of a standard picture (positive example) and a defect picture (negative example).

1-2) Step S2 of CNN defect feature training is connected with step S1. For each type of PCB defect data pair, use CNN training to extract PCB defect features and defect category features.

1-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defect data pair, use CNN training to extract PCB defect level features.

1-4) The step S4 of reading the standard PCB image reads the standard PCB image and the PCB image to be tested respectively.

1-5) Step S5 of filtering and denoising is connected with step S4, respectively filtering and denoising the standard PCB image and the PCB to be tested. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect recognition. Various algorithms and techniques are required to filter the noise.

1-6) Step S6 of image registration is connected with step S5, respectively registering images of the standard PCB and the PCB to be tested.

1-6-1) Check whether the standard PCB picture is correct. Whether there are changes such as rotation and deformation. If there is a change, it needs to be corrected; check whether the light of the standard PCB picture is abnormal, whether there is uneven light, reflection, etc. If there is an abnormal light, algorithm correction is required.

1-6-2) In addition to connecting with the step S5 of filtering and denoising, the picture of the PCB to be tested is also connected with the registered standard PCB picture. Check whether the picture of the PCB to be tested is correct and consistent with the position of the standard PCB, and check whether the position, light, and color of the standard picture are consistent. If inconsistent, use the algorithm to correct.

1-7) The step S7 of comparing the gray value of the image is connected with the step S6 of the image registration, comparing the gray value of the picture of the PCB to be tested with the picture of the standard PCB, and calculating the residual error of the picture.

1-8) The step S8 of image segmentation is connected with step S7, and the standard image, the image to be tested, and the residual image of the PCB are appropriately and identically segmented into small patches (patch), which are prepared for defect block positioning and marking.

1-9) The step S9 of residual filtering is connected with the step S8 of image segmentation to filter the comparison residual caused by registration error and interference in the residual. Delete the patch without residual in the residual image, and the patch with residual is to be detected and analyzed.

1-10) Step S11 of PCB defect discrimination is connected with step S9 of residual filtering and step S2 of PCB defect feature training. Using the PCB defect features trained by the CNN in step S2, identify the residual patch. Identify which patches are defects of the PCB to be tested, and mark suspected defective patches. This step can be identified multiple times, and the suspected defect patch can be gradually eliminated.

1-11) Step S12 of PCB defect classification is connected with step S11 of defect discrimination and step S2 of defect training. Using the PCB defect category feature trained by the CNN in step S2, classify the defect patch marked in step S11 of defect discrimination, and determine which defect the marked defect belongs to.

1-12) PCB defect verification S14 is connected with step S3 of defect classification S12 and defect level feature. Using the PCB defect level feature trained by CNN in step S3, check whether the suspected PCB defect obtained in step S12 is a real defect, delete false defects, and improve detection accuracy. If it is a true defect, flag it. At the same time, those defects that are not in the category are screened, and when the detected true defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspections.

Method Two:

Compared with the first detection method, the second defect detection method is a non-reference learning detection, and does not need to be referenced and compared with a standard PCB. As shown in Figure 3, the specific steps are:

2-1) Step S1 of establishing a PCB defect picture library. The defect library contains various marked, classified and graded defect pictures of PCBs. The defect picture may be a multi-scale picture. This defect library is different from the image library in the first detection method, it is not an image data pair, but a single defect image.

2-2) CNN defect feature training step S2 is connected with step S1. For each type of PCB defect, use CNN training to extract PCB defect features and defect category features.

2-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defects, use CNN training to extract PCB defect level features.

2-4) Step S4 of reading the standard PCB image, reading the image of the PCB to be tested.

2-5) The step S5 of filtering and denoising is connected with step S4, and the PCB to be tested is filtered and denoised. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect recognition. Various algorithms and techniques are required to filter the noise.

2-6) The step S8 of image segmentation is connected with the step S5 of filtering and denoising, and the image of the PCB to be tested is segmented into small patches, which are prepared for defect block positioning and marking.

2-7) Step S10 of defect matching search is connected with step S8 and step S2 of defect feature training. Using the PCB defect features trained by the CNN in step S2, search for a defect patch. Identify which patches are defects of the PCB to be tested, and mark suspected defective patches. This step can be identified multiple times, and the suspected defect patch can be gradually eliminated.

2-8) Step S12 of PCB defect classification is connected with step S10 of defect matching search and step S2 of defect training. Use the PCB defect category feature trained by CNN in step S2 to classify the defect patch marked in defect search step S10, determine which defect the marked defect belongs to, and screen out those defects that are not in the category to determine whether it is a false defect It is still a new type of defect; if it is a new type of defect, expand the defect into the defect library.

2-9) Step S14 of PCB defect verification is connected with step S12 of defect classification and step S3 of defect level characterization. Using the PCB defect level feature trained by CNN in step S3, check whether the suspected PCB defect obtained in step S12 is a real defect, delete false defects, and improve detection accuracy. If it is a true defect, flag it. When the detected true defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspections.

Method three:

The third detection method is a combination of the two detection methods described above, called the hybrid detection method, which can more accurately identify defects in the PCB, as shown in Figure 4, the specific steps are:

3-1) Step S1 of establishing a two-dimensional PCB defect picture library. The defect library contains a variety of labeled, classified and graded defect images for PCBs. The defect picture may be a multi-scale picture.

The PCB defect library is divided into two types of defect image libraries. One type is a picture data pair composed of standard pictures (positive examples) and defective pictures (negative examples) for defect pictures, and the second type is just a picture library composed of defect pictures.

3-2) Step S2 of CNN defect feature training is connected with step S1. For each type of PCB defect data pair in the first type of defect library, use CNN training to extract PCB defect features and defect category features. For each type of PCB defect in the second type of defect library, use CNN training to extract PCB defect features and defect category features.

3-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defect data pairs in the first type of defect library, use CNN training to extract PCB defect level features. For each level of PCB defects in the second type of defect library, use CNN training to extract PCB defect level features.

3-4) The step S4 of reading the standard PCB image reads the standard PCB image and the PCB image to be tested respectively.

3-5) Step S5 of filtering and denoising is connected with step S4, respectively filtering and denoising the standard PCB image and the PCB to be tested. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect recognition. Various algorithms and techniques are required to filter the noise.

3-6) Step S6 of image registration is connected with step S5, respectively registering images of the standard PCB and the PCB to be tested.

3-6-1) Check whether the standard PCB picture is correct. Whether there are changes such as rotation and deformation. If there is a change, it needs to be corrected; check whether the light of the standard PCB picture is abnormal, whether there is uneven light, reflection, etc. If there is an abnormal light, algorithm correction is required.

3-6-2) In addition to being connected with the filtering and denoising step S5, the PCB image to be tested is also connected with the registered standard PCB image. Check whether the picture of the PCB to be tested is correct and consistent with the position of the standard PCB, and check whether the position, light, and color of the standard picture are consistent. If inconsistent, use the algorithm to correct.

3-7) The step S7 of comparing the gray value of the image is connected with the step S6 of the image registration, comparing the gray value of the picture of the PCB to be tested with the standard PCB picture, and calculating the residual error of the picture.

3-8) The step S8 of image segmentation is connected with the step S7, and the standard image, the image to be tested, and the residual image of the PCB are appropriately and identically segmented into small patches (patch), which are prepared for defect block positioning and marking.

3-9) The step S9 of residual filtering is connected with the step S8 of image segmentation to filter the comparison residual caused by registration error and interference in the residual. Delete the patch without residual in the residual image, and the patch with residual is to be detected and analyzed.

3-10) Step S11 of PCB defect discrimination is connected with step S9 of residual filtering and step S2 of PCB defect feature training. Using the PCB defect features trained by the CNN in step S2, identify the residual patch. Identify which patches are defects of the PCB to be tested, and mark suspected defective patches. This step can be identified multiple times, and the suspected defect patch can be gradually eliminated.

3-11) Step S12 of PCB defect classification is connected with step S11 of defect discrimination and step S2 of defect training. Using the PCB defect category feature trained by CNN in step S2, classify the defect patch marked in step S11 of defect discrimination.

3-12) Step S13 of suspected defect is connected with step S12, marking the suspected defect of the PCB to be tested, and judging which defect the marked suspected defect belongs to.

3-13) Step S10 of defect search and matching is connected with step S13 of PCB suspected defect and step S2 of CNN defect feature training. For the step S13 of the suspected PCB defect, use the PCB defect features and category features trained by the second type of defect library CNN in step S2 to search and match. Filter false defects from suspected defects.

3-14) Step S14 of PCB defect verification is connected with step S10 of defect search and matching and step S3 of defect level feature training. The defects screened in step S10 are further filtered. Using the PCB defect level feature trained by the second type of defect library CNN in step S3, check whether the suspected PCB defect obtained in step S10 is a real defect, delete false defects, and improve detection accuracy. If it is a true defect, flag it. At the same time, those defects that are not in the category are screened, and when the detected true defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspections.

According to the real-time automatic detection method of two-dimensional PCB appearance defects based on deep learning proposed by the embodiment of the present invention, PCB images, standard PCB images and convolutional neural networks are used to detect defects on PCBs, and tools for detecting defects through PCBs are available in Finding and eliminating errors early in the assembly process can avoid sending bad boards to subsequent assembly stages, while reducing repair costs and avoiding scrapping unrepairable boards. The use of convolutional neural networks has the advantages of fast speed, high precision, strong generalization ability, and clear structure.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

In the description of this specification, descriptions referring to the terms "one embodiment", "some embodiments", "example", "specific examples", or "some examples" mean that specific features described in connection with the embodiment or example , structure, material or characteristic is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the described specific features, structures, materials or characteristics may be combined in any suitable manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine different embodiments or examples and features of different embodiments or examples described in this specification without conflicting with each other.

Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention, those skilled in the art can make the above-mentioned The embodiments are subject to changes, modifications, substitutions and variations.

S1~S14: Steps

The above and/or additional aspects and advantages of the present invention will become apparent and easy to understand from the following description of the embodiments in conjunction with the accompanying drawings, wherein:

Fig. 1 is a flowchart of a method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to an embodiment of the present invention.

Fig. 2 is a flowchart of a method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to a specific embodiment of the present invention.

Fig. 3 is a block diagram of a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to another specific embodiment of the present invention.

Fig. 4 is a flowchart of a method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to another specific embodiment of the present invention.

S1~S14: Steps

Claims (10)

A method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning, said method comprising the following steps: Step S1, establishing a two-dimensional PCB defect block image library through a PCB defect block image library, said two-dimensional PCB defect block image library It includes a two-dimensional PCB defect data pair image library and a two-dimensional PCB defect single data image library; wherein, the two-dimensional PCB defect block image library is divided into two categories, one is a two-dimensional PCB defect block and a corresponding defect-free standard PCB block image The formed data pair picture library, another kind is only the data picture library of single two-dimensional PCB defect block; Said step S1 also includes: setting up said two-dimensional PCB defect block picture library, two types of described two-dimensional PCB defect block pictures The library contains a variety of PCB defect block pictures that have been marked and classified and graded, and the PCB defect block pictures include multi-scale pictures; Step S2, through CNN defect feature training blocks, for the two-dimensional PCB defect block picture library. Each type of PCB defect is trained by using a convolutional neural network to extract PCB defect category features; step S3, through the CNN defect level feature training block, for each type of PCB defect in the two-dimensional PCB defect block picture library Level, using the convolutional neural network for training, extracting PCB defect level features; step S4, extracting two-dimensional PCB images through the PCB image extraction block, the two-dimensional PCB images include two-dimensional standard PCB images and two-dimensional to-be-tested PCB image; step S5, denoising the two-dimensional PCB image through the filter denoising block; step S6, through the image registration block, the two-dimensional PCB image to be tested and the two-dimensional standard PCB Image registration; step S7, by image gray value comparison block, the two-dimensional standard PCB image and the described The gray value of the two-dimensional PCB image to be tested is compared and analyzed to obtain a two-dimensional PCB residual image; step S8, through image segmentation and positioning blocks, the two-dimensional standard PCB image, the two-dimensional PCB image to be tested and The two-dimensional PCB residual picture is segmented and positioned to obtain picture blocks and corresponding coordinates of a plurality of pictures; Step S9, through the residual filtering block, the picture block of the two-dimensional PCB residual picture is filtered to eliminate the The comparison error produced by accuracy and other factors; step S10, defect search and match block through the PCB defect features trained by the convolutional neural network, search for defect blocks in multiple picture blocks, and perform defect block matching; step S11, The CNN defect identification block identifies defects in the two-dimensional PCB image to be tested through the convolutional neural network; step S12, the PCB defect classification block uses the PCB defect category features trained by the convolutional neural network to Classify the defects marked in the two-dimensional PCB picture to be tested, judge the defects that are not in the category, judge whether it is a false defect or a defect of a new category, and supplement the defects of the new category to the two-dimensional PCB defect block picture library; step S13, through the PCB defect suspected block, mark each type of defect in the two-dimensional PCB image to be tested as a suspected defect; and step S14, the PCB defect verification block is trained through the convolutional neural network PCB defect level features and classification information to judge the authenticity of the suspected defective block. If the defect is a false defect, delete the corresponding defect and analyze the cause of the false defect. If the defect is a true defect, mark it in the defect record. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 1, wherein, in the step S4, the two-dimensional PCB pictures extracted by the PCB picture extraction block include two types: One is to extract the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested; the other is to only extract the two-dimensional PCB picture to be tested; the step S4 also includes: In the detection method of comparing the standard PCB and the PCB mixed detection method, extract the two-dimensional PCB picture to be tested and the corresponding two-dimensional standard PCB picture; and in the non-standard board comparison PCB detection method, only extract the two A picture of the PCB to be tested. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 2, wherein, in the step S5, the denoising process is performed on the two-dimensional PCB image by filtering the denoising block, including: Using an algorithm to denoise the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested, in the comparison standard PCB detection method and PCB hybrid detection method, the two-dimensional standard PCB picture and the two-dimensional standard PCB picture and the tested PCB picture are denoised. Denoise the two-dimensional PCB image to be tested, remove the noise in the image, and correct the influence of the color of the bottom plate on the image, so that the color of the bottom plate of the two-dimensional standard PCB image, the two-dimensional PCB image to be tested, and the pictures in the defect library are consistent ; In the non-standard board comparison PCB detection method, only denoise the two-dimensional PCB image to be tested, remove the noise in the image, and correct the impact of the color of the bottom plate on the image, so that the two-dimensional PCB image to be tested , The color of the bottom plate of the picture in the defect library is the same. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 3, wherein registration and gray value comparison are performed on the two-dimensional standard PCB image and the two-dimensional PCB image to be tested , image segmentation, the method further includes the following steps: in the detection method of the comparison standard PCB and the mixed detection method of PCB, the step S6, the step S7 and the step S8 include: in the In comparing the detection method of the standard PCB and the mixed detection method of the PCB, the two-dimensional standard PCB picture is registered with the two-dimensional PCB picture to be tested, and the two-dimensional standard PCB picture after the denoising process is used for the registration of the two-dimensional standard PCB picture. The two-dimensional PCB image to be tested is corrected, and the two-dimensional standard PCB image and the two-dimensional PCB image to be tested are compared to the gray value, and the image residual is calculated to obtain the two-dimensional PCB residual image; the two-dimensional standard PCB picture, the two-dimensional PCB to be tested picture and the two-dimensional The PCB residual image is segmented to obtain multiple image blocks and corresponding coordinates; the two-dimensional standard PCB image is registered to detect whether there is rotation, deformation, uneven light and light reflection in the two-dimensional standard PCB image, if If it exists, then use the algorithm to correct it; compare the two-dimensional standard PCB picture to be tested with the processed two-dimensional standard PCB picture, and judge whether the position, light and color are consistent. Measure the PCB picture for correction. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 3, wherein, in the non-comparison reference detection method, the step S8 further includes: the two-dimensional to-be-tested PCB picture Segmentation is performed to obtain multiple picture blocks and corresponding coordinates. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim item 5, wherein, in the step S9, the image blocks of the two-dimensional PCB residual image are filtered to eliminate comparison errors and delete There is no residual image block in the two-dimensional PCB residual image; the step S10 and the step S12 further include: in the non-standard board comparison PCB detection method, using the two-dimensional PCB defect single data image library; in the non-standard board comparison PCB detection method and the PCB hybrid detection method, for each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, use the convolution Neural network training, extracting PCB defect features and PCB defect category features; the step S10 uses the PCB defect features trained by the convolutional neural network to perform PCB defect search and discrimination in multiple image blocks, using a single two-dimensional PCB Defect picture library convolutional neural network trains the category characteristics of PCB defects, and the step S12 classifies the defects marked in the two-dimensional PCB image to be tested, judges the defects not in the category, and judges whether it is a false defect or a defect. Defects of a new category are added to the defect image library. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 5 or 6, wherein, in the detection method of comparing the standard PCB and the mixed detection method of PCB, the two-dimensional PCB is respectively detected Each kind of PCB defect in the defect picture library and the two-dimensional PCB defect picture library is trained with the convolutional neural network to extract PCB defect features and PCB defect category features; said step S11 and said step S12 also Including: using a two-dimensional PCB defect data pair picture library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, the step S11 uses the PCB defect features trained by the convolutional neural network, and the steps S9 performs defect recognition on the residual image; adopts the two-dimensional PCB defect image library convolutional neural network to train the category characteristics of PCB defects, and classifies the defects marked in the two-dimensional PCB image to be tested in the step S12, and Defects that are not in the category are judged, whether it is a false defect or a defect of a new category is judged, and the defect of the new category is added to the defect image library. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim item 7, wherein, in the PCB hybrid detection method, after the step S12, it also includes: using the convolutional neural network trained The PCB defect features of the two-dimensional PCB defect image library search and match the marked suspected defect blocks, delete the false defects in it, and improve the recognition accuracy; and after the step S13, the PCB trained by the convolutional neural network Defect features, searching for matches in the defect pictures for suspected defects. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 1 or 8, wherein, for PCB defect verification, the PCB defect level features trained by the convolutional neural network described in step S3 message to judge the authenticity of the PCB defect, if the defect is a false defect, delete all Describe the false defect, and analyze the cause of the false defect, if the defect is a true defect, it will be marked in the defect record. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 1 or 8, wherein, in the process of searching and judging defects, the defects are eliminated successively through multiple identification iterations.

Images