TW202144766A - Two-dimensional PCB appearance defect real-time automatic detection technology based on deep learning - Google Patents

Abstract

The invention provides a two-dimensional PCB appearance defect real-time automatic detection technology based on deep learning. The PCB defect detection method comprises three PCB defect detection modes of CNN detection depending on standard PCB comparison, CNN detection (unsupervised learning detection) without standard PCB comparison and hybrid CNN detection, and comprises the following steps: training each type of defect in a PCB defect picture library, and segmenting a PCB picture to be detected into a plurality of picture blocks; utilizing the PCB defect characteristics to search defect blocks to identify defects in the PCB picture to be detected and marking suspected defect blocks; carrying out defect classification by utilizing PCB defect category characteristics, judging defect categories, judging whether suspected defects which are not in the categories are false defects or new categories of defects, and supplementing the new categories of defects to a defect picture library; and judging suspected defects by utilizing PCB defect level characteristics, if the suspected defects are false defects, deleting the corresponding defects and analyzing reasons, and if the suspected defects are true defects, marking the suspected defects in defect records. The method is high in PCB defect detection speed and high in precision.

Description

Real-time automatic detection method of two-dimensional PCB appearance defects based on deep learning

The invention relates to the technical field of defect detection, in particular to a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning.

Two-dimensional (2D) printed circuit board (PCB) is the basis for the production of various microcircuit boards and motherboards, and its correctness is the guarantee for the correctness of other subsequent processes. Due to the continuous development of modern technology and fine craftsmanship, the production of PCBs is becoming more and more complex and sophisticated. Traditional appearance inspection is not suitable for complex PCB inspection.

The present invention aims to solve one of the technical problems in the related art at least to a certain extent.

Therefore, the purpose of the present invention is to propose a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, which has the advantages of high speed, high precision, strong generalization ability, and clear structure.

In order to achieve the above purpose, an embodiment of the present invention proposes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, including:

In step S1, a two-dimensional PCB defect block picture library is established by collecting and marking PCB defect block pictures, and the two-dimensional PCB defect block picture library includes a two-dimensional PCB defect data pair picture library and a two-dimensional PCB defect sheet data picture library;

Step S2, utilizes the convolutional neural network (Convolutional neural network; CNN) to carry out training to each type of PCB defect in the described two-dimensional PCB defect block picture library, and extracts the PCB defect category feature;

Step S3, using the convolutional neural network to train the level of each type of PCB defect in the two-dimensional PCB defect block picture library, and extracting the level of PCB defect;

Step S4, extracting a two-dimensional PCB picture through the PCB production line, where the two-dimensional PCB picture includes a two-dimensional standard PCB picture and a two-dimensional PCB to be tested picture;

Step S5, performing denoising processing on the two-dimensional PCB picture by filtering the denoising block;

Step S6, through the image registration block, the two-dimensional PCB picture to be tested and the two-dimensional standard PCB picture are registered;

Step S7, through the image gray value comparison block, compare and analyze the gray value of the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested to obtain a two-dimensional PCB residual picture;

Step S8, through the image segmentation and positioning block, the two-dimensional standard PCB picture, the two-dimensional PCB to be tested picture and the two-dimensional PCB residual picture are divided and positioned to obtain the picture blocks of multiple pictures and corresponding coordinate;

Step S9, filter the picture block of the two-dimensional PCB residual picture through the residual filter block to eliminate the comparison error caused by registration and other factors;

Step S10, the defect search matching block, through the PCB defect feature trained by the convolutional neural network, search for the defect block in the plurality of picture blocks, and perform the defect block matching;

Step S11, the CNN defect identification block identifies defects in the two-dimensional PCB image to be tested through a convolutional neural network;

Step S12, the PCB defect classification block, classifies the defects marked in the two-dimensional PCB image to be tested by using the PCB defect category features trained by the convolutional neural network, and judges the defects that are not in the category to determine whether they are false or not. If the defect is still a defect of a new category, add the defect of the new category to the defect block picture library;

Step S13, through the PCB defect suspected block, mark each type of defect in the two-dimensional PCB image to be tested as a suspected defect;

Step S14, the PCB defect verification block, through the PCB defect level features and classification information trained by the convolutional neural network, to determine the authenticity of the suspected defect block, if the defect is a false defect, delete the corresponding defect, and analyze the false defect. The reason, if the defect is a true defect, is marked in the defect record.

The deep learning-based real-time automatic detection method for two-dimensional PCB appearance defects according to the embodiment of the present invention utilizes PCB images, standard PCB images and convolutional neural networks to perform defect detection on PCBs. Finding and eliminating errors early in the process can avoid sending defective PCB boards to subsequent assembly stages, while reducing repair costs will avoid scrapping unrepairable boards. The use of convolutional neural networks has the advantages of high speed, high accuracy, strong generalization ability, and clear structure.

In addition, the real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to the above-mentioned embodiments of the present invention may also have the following additional technical features:

Further, in an embodiment of the present invention, the two-dimensional PCB defect block picture library is divided into two categories, one is a data pair picture library composed of two-dimensional PCB defect blocks and corresponding defect-free standard PCB block pictures, and the other is a data pair picture library. A data picture library for only a single 2D PCB defect block.

Further, in an embodiment of the present invention, the step S1 further includes:

Build a two-dimensional PCB defect block picture library. The two-dimensional two-dimensional PCB defect block picture library contains a variety of labeled and classified and graded PCB defect block pictures of the PCB, the PCB defect block pictures including multi-scale pictures.

Further, in an embodiment of the present invention, the step S2 and the step S3 further include:

For each type of PCB defect in the two-dimensional PCB defect block image library, a convolutional neural network is used for training, and the characteristics of the PCB defect category are extracted;

For the level of each type of PCB defect in the two-dimensional PCB defect block image library, a convolutional neural network is used for training to extract the PCB defect level feature.

Further, in an embodiment of the present invention, in step S4, the two-dimensional PCB pictures extracted by the PCB picture extraction block include two types: one is to extract two-dimensional standard PCB pictures and two-dimensional PCB pictures to be tested; In order to extract only the two-dimensional PCB image to be tested.

Further, in an embodiment of the present invention, the step S4 further includes:

In comparing the standard PCB detection method and the PCB hybrid detection method, extract the two-dimensional PCB image to be tested and the corresponding two-dimensional standard PCB image;

In the non-standard board comparison PCB detection method, only the two-dimensional PCB picture to be tested is extracted.

Further, in an embodiment of the present invention, in the step S5, the filtering and denoising block performs denoising processing on the two-dimensional PCB picture, including:

The two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested are denoised by using an algorithm, and in the comparison of the standard PCB detection method and the PCB hybrid detection method, the two-dimensional standard PCB picture and the two-dimensional PCB picture are denoised. De-noise the PCB image to be tested, remove the noise in the image, and correct the influence of the color of the backplane on the image, so that the color of the backplane in the standard PCB image, the PCB image to be tested, and the image in the defect library is the same;

In the non-standard board comparison PCB detection method, only the two-dimensional PCB image to be tested is denoised, the noise in the image is removed, and the influence of the color of the bottom plate on the image is corrected, so that the image of the PCB to be tested and the bottom plate of the image in the defect library are Consistent color.

Further, in an embodiment of the present invention, performing registration, gray value comparison, and image segmentation on the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested, further comprising:

In the detection method for comparing the standard PCB and the PCB hybrid detection method, the step S6, the step S7 and the step S8 include:

In the detection method for comparing the standard PCB and the PCB hybrid detection method, the two-dimensional standard PCB picture and the two-dimensional PCB to be tested picture are registered, and the two-dimensional standard PCB picture after denoising is processed. Correcting the PCB picture to be measured, comparing the gray value of the two-dimensional standard PCB picture and the two-dimensional PCB picture to be measured, and calculating the residuals of the pictures to obtain a two-dimensional PCB residual picture; comparing the two-dimensional standard PCB picture , the two-dimensional PCB picture to be tested and the two-dimensional PCB residual picture are divided to obtain a plurality of picture blocks and corresponding coordinates;

The two-dimensional standard PCB picture is registered to detect whether the two-dimensional standard PCB picture has rotation, deformation, uneven light and light reflection, and if there is, use an algorithm to correct it;

The two-dimensional PCB image to be tested is compared with the processed two-dimensional standard PCB image to determine whether the position, light and color are consistent. If they are inconsistent, the two-dimensional PCB image to be tested is corrected by an algorithm.

Further, in an embodiment of the present invention, in the non-comparison reference detection method, step S8 further includes: segmenting the two-dimensional PCB picture to be tested to obtain a plurality of picture blocks and corresponding coordinates.

Further, in an embodiment of the present invention, in the step S9, the picture block of the two-dimensional PCB residual picture is filtered, the comparison error caused by registration and other factors is eliminated, and the two-dimensional PCB is deleted. The image block without residual in the residual image;

The steps S10 and S12 further include:

In the detection method for comparing PCBs without standard boards, the two-dimensional PCB defect single data picture library is used;

In the non-standard board comparison PCB detection method and the PCB hybrid detection method, the two-dimensional PCB defect image library and each type of PCB defect in the two-dimensional PCB defect image library are trained with a convolutional neural network , extract PCB defect features and PCB defect category features;

Step S10 uses the PCB defect feature trained by the convolutional neural network, and the step S8 performs PCB defect search and discrimination in a plurality of picture blocks, and uses a single two-dimensional PCB defect picture library convolutional neural network to train the category features of PCB defects, Step S12 classifies the defects marked in the two-dimensional PCB image to be tested, judges the defects that are not in the category, determines whether it is a false defect or a defect of a new category, and supplements the defect of the new category to the defect Photo gallery.

Further, in an embodiment of the present invention, in the detection method for comparing the standard PCB and the PCB hybrid detection method, each of the two-dimensional PCB defect picture library and the two-dimensional PCB defect picture library is respectively analyzed. Type PCB defects, using convolutional neural network training to extract PCB defect features and PCB defect category features. Step S11 and step S12 also include:

A two-dimensional PCB defect data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures is used. Step S11 uses the PCB defect feature pairs trained by the convolutional neural network, and step S9 performs defect identification on the residual image. , using the two-dimensional PCB defect image library convolutional neural network to train the category feature pairs of PCB defects, step S12 to classify the defects marked in the two-dimensional PCB image to be tested, to judge the defects that are not in the category, and to determine whether Whether a false defect or a new category of defects is added, the new category of defects is added to the defect picture library.

Further, in an embodiment of the present invention, in the PCB hybrid detection method, after step S12, the method further includes:

Use the PCB defect feature of the two-dimensional PCB defect image library trained by the convolutional neural network to search and match the marked suspected defect blocks, delete the false defects therein, and improve the recognition accuracy;

After step S13 , searching and matching the defect pictures for the suspected defects through the PCB defect features trained by the convolutional neural network.

Further, in one embodiment of the present invention, for PCB defect verification, the PCB defect level feature information trained by the convolutional neural network in step S3 is used to judge the authenticity of the PCB defect, and if the defect is a false defect, delete the defect. False defects, and analyze the cause of false defects, if the defect is a true defect, it will be marked in the defect record.

Further, in an embodiment of the present invention, in the process of searching and judging defects, the defects are eliminated one by one through multiple identification iterations.

Additional aspects and advantages of the present invention will be set forth, in part, from the following description, and in part will be apparent from the following description, or may be learned by practice of the invention.

The following describes in detail the embodiments of the present invention, examples of which are illustrated in the drawings, wherein the same or similar reference numerals refer to the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary, and are intended to explain the present invention and should not be construed as limiting the present invention.

The following describes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to an embodiment of the present invention with reference to the drawings.

With the development of the process, the fabrication of PCBs is becoming more and more complex and precise, and the existing defect detection methods are not efficient and have low precision. The embodiment of the present invention proposes a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning, which specifically includes three embodiments of defect detection methods for PCB defect detection. Among them, one is a CNN detection method that relies on standard PCB board comparison (called method 1), the second is a CNN detection method without standard board comparison (called method 2), and the third is the first two. Method Hybrid CNN's PCB detection method (called method three).

As shown in Figure 1, the deep learning-based real-time automatic detection method for two-dimensional PCB appearance defects according to an embodiment of the present invention includes:

Step S1, a picture library of PCB defect blocks is established, a two-dimensional PCB defect block (patch) picture library is established, and there are respectively a two-dimensional PCB defect data pair picture library and a two-dimensional PCB defect sheet data picture library;

Step S2, CNN defect feature training block, for each type of PCB defect in the two-dimensional PCB defect block image library, use convolutional neural network (CNN) training to extract PCB defect category features;

Step S3, the CNN defect level feature training block, for each type of PCB defect level in the two-dimensional PCB defect block image library, is trained with a convolutional neural network to extract the PCB defect level feature;

Step S4, the PCB image extraction block, extracts a two-dimensional PCB image, which includes a two-dimensional standard PCB image and a two-dimensional PCB image to be tested;

Step S5, filter the denoising block, and perform denoising processing on the two-dimensional PCB image;

Step S6, the image registration block registers the two-dimensional standard PCB image and the two-dimensional PCB image to be tested, so that the two-dimensional standard PCB image and the two-dimensional PCB image to be tested are aligned;

Step S7, the image gray value comparison block, compares and analyzes the gray value of the two-dimensional standard PCB image and the two-dimensional PCB image to be tested, and obtains a two-dimensional PCB residual image;

Step S8, image segmentation and positioning block, the two-dimensional standard PCB picture, the two-dimensional PCB picture to be tested and the two-dimensional PCB residual picture are divided and positioned to obtain patches of multiple pictures and corresponding coordinates;

Step S9, the residual filter block, filters the patch of the two-dimensional PCB residual image, and eliminates the comparison error due to registration and other factors;

Step S10, the defect search matching block, through the PCB defect feature trained by the convolutional neural network, search for the defect block in the plurality of picture blocks, and perform the defect block matching;

Step S11, the CNN defect identification block identifies defects in the PCB image to be tested through a convolutional neural network;

Step S12, the PCB defect classification block, classifies the defects marked in the PCB image to be tested through the PCB defect category features trained by the convolutional neural network, and judges the defects that are not in the category to determine whether it is a false defect or a new category. defects, and add new categories of defects to the defect picture library;

Step S13, the PCB defect suspected block, marks each type of defect in the two-dimensional PCB image to be tested as a suspected defect;

Step S14, the PCB defect verification block, through the PCB defect level features and classification information trained by the convolutional neural network, to determine the authenticity of the suspected defect block, if the defect is a false defect, delete the corresponding defect, and analyze the false defect. The reason, if the defect is a true defect, is marked in the defect record.

In the embodiment of the present invention, the two-dimensional PCB defect block picture library is divided into two categories, one is a data pair picture library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, and the other type has only a single two-dimensional PCB defect Block data picture library.

In the embodiment of the present invention, on the basis of the above-mentioned embodiment, step S1 further includes:

Two types of two-dimensional PCB defect block image libraries contain a variety of marked, classified and graded PCB defect block images of PCBs, and PCB defect block images include multi-scale images.

In the embodiment of the present invention, step S2 and step S3 further include:

For each type of PCB defect in the two-dimensional PCB defect block image library, use convolutional neural network (CNN) training to extract PCB defect category features;

For the level of each type of PCB defect in the two-dimensional PCB defect block image library, the convolutional neural network is used to train and extract the PCB defect level features.

In the embodiment of the present invention, it is characterized in that step S4 includes extracting a two-dimensional PCB picture. There are two cases, extracting two-dimensional standard PCB pictures and two-dimensional PCB pictures to be tested, and extracting only the PCB pictures to be tested.

Step S4 also includes:

For detection method 1 and method 3 that require comparison technology, it is necessary to extract the two-dimensional PCB image to be tested and the corresponding standard PCB image. For the second method of unsupervised learning detection (that is, the non-comparison reference learning detection method), it is only necessary to extract the two-dimensional PCB image to be tested.

In the embodiment of the present invention, the filtering and denoising block in step S5 performs denoising processing on the two-dimensional PCB picture.

The algorithm is used to denoise the standard PCB image and the PCB image to be tested. Method 1 and method 3 need to denoise the two-dimensional standard PCB picture and the PCB picture to be tested, remove the noise in the picture, and correct the influence of the color of the bottom plate on the picture, so that the standard PCB picture, the PCB picture to be tested, and the pictures in the defect library are The color of the bottom plate is the same; the second method only needs to denoise the PCB image to be tested, remove the noise in the image, and correct the impact of the color of the bottom plate on the image, so that the color of the PCB to be tested and the image in the defect library are the same.

Registration, gray value comparison, and image segmentation are performed on the standard PCB image and the PCB image to be tested.

For step S6, step S7 and step S8, method 1 and method 3 are used. The standard PCB picture and the PCB picture to be tested are registered, and the picture of the PCB to be tested is corrected according to the processed standard PCB picture. Compare the gray value of the standard PCB image and the PCB image to be tested, and calculate the residual image to obtain the residual image; then divide the standard PCB image, the PCB image to be tested, and the residual image to obtain multiple image blocks and coordinates.

The standard PCB picture is registered to detect whether the standard PCB picture has rotation, deformation, uneven light and light reflection. If there is, use the algorithm to correct it.

Compare the PCB image to be tested with the processed standard PCB image to determine whether the position, light and color are consistent. If they are inconsistent, correct the PCB image to be tested through an algorithm.

The image segmentation and positioning step S8 is used in the second method, and the PCB image to be tested is segmented to obtain a plurality of image blocks and coordinates.

In the embodiment of the present invention, step S9 filters the patch of the two-dimensional PCB residual picture, eliminates the comparison error caused by registration and other factors, and deletes the picture block without residual in the residual picture;

For each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, use convolutional neural network training to extract PCB defect features and PCB defect category features. Step S10 and step S12 also include:

Method 2 uses only a single two-dimensional PCB defect block data image library. Step S10 uses the PCB defect features trained by CNN, and step S8 performs PCB defect search and discrimination. Then, a single two-dimensional PCB defect image library CNN is used to train the category features of PCB defects. Step S12 is to classify the defects marked in the PCB image to be tested, and judge the defects that are not in the category to determine whether it is a false defect or a new category of defects. , add the new category of defects to the defect image library.

In the embodiment of the present invention, it is characterized in that, for each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library, convolutional neural network training is used to extract PCB defect features and PCB defects. Category features. Step S11 and step S12 also include:

Method 1 and method 3 use a data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, step S11 uses the PCB defect features trained by CNN, and performs defect identification on the residual image in step S9. Then use the two-dimensional PCB defect image library CNN to train the category features of PCB defects, classify the defects marked in the PCB image to be tested in step S12, judge the defects that are not in the category, and judge whether it is a false defect or a new category of defects. , add the new category of defects to the defect image library.

After step S12, the third method further includes:

Using the PCB defect features of the two-dimensional PCB defect image library trained by the convolutional neural network to search and match the marked suspected defect blocks, delete the false defects and improve the recognition accuracy.

After step S13 , a defect image is searched and matched for the suspected defect through the PCB defect feature trained by the convolutional neural network.

In the embodiment of the present invention, for PCB defect verification, the PCB defect level feature information trained by the convolutional neural network in step S3 is used to judge the authenticity of the PCB defect, if the defect is a false defect, the false defect is deleted, and Analyze the cause of the false defect, and mark it in the defect record if the defect is a true defect.

In the embodiment of the present invention, it is characterized in that, in the process of searching and judging defects, the defects are eliminated one by one through multiple identification iterations.

To sum up, the embodiment of the present invention proposes three defect detection methods, and the three detection methods are described in detail below with reference to the drawings.

method one:

Detection based on deep learning, as shown in Figure 2, includes the following steps:

1-1) Step S1 of establishing a two-dimensional PCB defect image library. The defect library contains various marked, classified and graded defect pictures of PCBs. Defect pictures can be multi-scale pictures. The defective picture is a picture data pair composed of a standard picture (positive example) and a defective picture (negative example).

1-2) Step S2 of CNN defect feature training is connected with step S1. For each type of PCB defect data pair, use CNN training to extract PCB defect features and defect category features.

1-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defect data pair, use CNN training to extract PCB defect level features.

1-4) Step S4 of reading the standard PCB picture, respectively reading the standard PCB image and the PCB image to be tested.

1-5) The step S5 of filtering and denoising is connected with step S4, and the standard PCB image and the PCB to be tested are filtered and denoised respectively. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect identification. Various algorithms and techniques are required to filter the noise.

1-6) Step S6 of image registration is connected with step S5, and the images of the standard PCB and the PCB to be tested are registered respectively.

1-6-1) Check whether the standard PCB picture is correct. Whether there are changes such as rotation, deformation, etc. If there is a change, it needs to be corrected; check whether the light of the standard PCB picture is abnormal, whether there is uneven light, reflection, etc. If there is an abnormal light, algorithm correction is required.

1-6-2) In addition to the connection with the step S5 of filtering and denoising, the PCB image to be tested should also be connected with the registered standard PCB image. Check whether the PCB picture to be tested is correct and consistent with the standard PCB position, and check whether the position, light and color of the standard picture are consistent. If not, use algorithm to correct.

1-7) The step S7 of comparing the gray value of the image is connected with the step S6 of image registration, and the gray value of the image is compared between the PCB image to be tested and the standard PCB image, and the residual of the image is calculated.

1-8) The step S8 of image segmentation is connected with step S7, and the standard image, the image to be tested, and the residual image of the PCB are appropriately and identically segmented, and divided into small patches (patches) to prepare for the location and marking of defective blocks.

1-9) The step S9 of residual filtering is connected with the step S8 of image segmentation, and the comparison residual caused by the registration error and interference in the residual is filtered. Delete the patch without residual in the residual image, and the patch with residual to be detected and analyzed.

1-10) Step S11 of PCB defect identification is connected with step S9 of residual filtering and step S2 of PCB defect feature training. Use the PCB defect features trained by the CNN in step S2 to identify residual patches. Identify which patches are defects of the PCB under test, and mark the suspected defect patches. In this step, multiple identification iterations can be performed to gradually eliminate suspected defect patches.

1-11) Step S12 of PCB defect classification is connected to step S11 of defect identification and step S2 of defect training. Using the PCB defect category features trained by the CNN in step S2, classify the defect patches marked in step S11 of defect discrimination, and determine which defect the marked defect belongs to.

1-12) PCB defect verification S14 is connected with step S3 of defect classification S12 and defect level characterization. Using the PCB defect level features trained by the CNN in step S3, check whether the suspected PCB defects obtained in step S12 are true defects, delete false defects, and improve detection accuracy. If it is a true defect, mark it. At the same time, the defects that are not in the category are screened. When the detected real defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspection.

Method Two:

Compared with the first detection method, the second defect detection method is non-reference learning detection and does not need to be referenced and compared with the standard PCB. As shown in Figure 3, the specific steps are:

2-1) Step S1 of establishing a PCB defect picture library. The defect library contains a variety of marked, classified and graded defect pictures of the PCB. Defect pictures can be multi-scale pictures. This defect library is different from the picture library in the first detection method, it is not a picture data pair, but a single defect picture.

2-2) CNN defect feature training step S2 is connected with step S1. For each type of PCB defect, use CNN training to extract PCB defect features and defect category features.

2-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defect, use CNN training to extract PCB defect level features.

2-4) Step S4 of reading the standard PCB picture, reading the PCB image to be tested.

2-5) The step S5 of filtering and denoising is connected with step S4, and the PCB to be tested is filtered and denoised. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect identification. Various algorithms and techniques are required to filter the noise.

2-6) The step S8 of image segmentation is connected with the step S5 of filtering and denoising, and the picture to be tested of the PCB is segmented and divided into patches, which are prepared for the location and marking of defective blocks.

2-7) Step S10 of defect matching search is connected with step S8 and step S2 of defect feature training. Use the PCB defect features trained by the CNN in step S2 to search for defect patches. Identify which patches are defects of the PCB under test, and mark the suspected defect patches. In this step, multiple identification iterations can be performed to gradually eliminate suspected defect patches.

2-8) Step S12 of PCB defect classification is connected with step S10 of defect matching search and step S2 of defect training. Using the PCB defect category features trained by the CNN in step S2, classify the defect patches marked in the defect search step S10, determine which defect the marked defect belongs to, and at the same time screen those defects that are not in the category to determine whether they are false defects. Or a new type of defect; if it is a new type of defect, expand the defect into the defect library.

2-9) The step S14 of PCB defect verification is connected with the step S12 of defect classification and the step S3 of defect level characterization. Using the PCB defect level feature trained by the CNN in step S3, check whether the suspected PCB defect obtained in step S12 is a real defect, delete the false defect, and improve the detection accuracy. If it is a true defect, mark it. When the detected true defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspection.

Method three:

The third detection method is a combination of the two detection methods described above, called a hybrid detection method, which can more accurately identify defects in the PCB, as shown in Figure 4. The specific steps are:

3-1) Step S1 of establishing a two-dimensional PCB defect image library. The defect library contains various marked, classified and graded defect pictures of PCBs. Defect pictures can be multi-scale pictures.

The PCB defect library is divided into two types of defect image libraries. One is a picture data pair composed of standard pictures (positive examples) and defective pictures (negative examples), and the second category is only a picture library composed of defective pictures.

3-2) Step S2 of CNN defect feature training is connected with step S1. For each type of PCB defect data pair in the first type of defect library, use CNN to train to extract PCB defect features and defect category features. For each type of PCB defect in the second type of defect library, use CNN to train to extract PCB defect features and defect category features.

3-3) Step S3 of CNN defect level training is connected with step S1. For each level of PCB defect data pair in the first type of defect library, use CNN training to extract PCB defect level features. For each level of PCB defect in the second type of defect library, use CNN training to extract PCB defect level features.

3-4) Step S4 of reading the standard PCB picture, respectively reading the standard PCB image and the PCB image to be tested.

3-5) The step S5 of filtering and denoising is connected with step S4, and the standard PCB image and the PCB to be tested are filtered and denoised respectively. There are inevitably many kinds of noise in the picture, as well as the color of the bottom plate, which has a great impact on defect identification. Various algorithms and techniques are required to filter the noise.

3-6) Step S6 of image registration is connected with step S5, and the images of the standard PCB and the PCB to be tested are registered respectively.

3-6-1) Check whether the standard PCB picture is correct. Whether there are changes such as rotation, deformation, etc. If there is a change, it needs to be corrected; check whether the light of the standard PCB picture is abnormal, whether there is uneven light, reflection, etc. If there is an abnormal light, algorithm correction is required.

3-6-2) In addition to the connection with the filtering and denoising step S5, the PCB image to be tested should also be connected with the registered standard PCB image. Check whether the PCB picture to be tested is correct and consistent with the standard PCB position, and check whether the position, light and color of the standard picture are consistent. If not, use algorithm to correct.

3-7) The step S7 of comparing the gray value of the image is connected with the step S6 of image registration, and the gray value of the image is compared between the PCB picture to be tested and the standard PCB picture, and the residual of the picture is calculated.

3-8) The step S8 of image segmentation is connected with step S7, and the standard image, the image to be tested, and the residual image of the PCB are appropriately and identically segmented, and divided into small patches (patches) to prepare for the location and marking of defective blocks.

3-9) The step S9 of residual filtering is connected with the step S8 of image segmentation, and the comparison residual caused by the registration error and interference in the residual is filtered. Delete the patch without residual in the residual image, and the patch with residual to be detected and analyzed.

3-10) Step S11 of PCB defect identification is connected with step S9 of residual filtering and step S2 of PCB defect feature training. Use the PCB defect features trained by the CNN in step S2 to identify residual patches. Identify which patches are defects of the PCB under test, and mark the suspected defect patches. In this step, multiple identification iterations can be performed to gradually eliminate suspected defect patches.

3-11) Step S12 of PCB defect classification is connected with step S11 of defect identification and step S2 of defect training. The defect patch marked in step S11 of defect discrimination is classified by using the PCB defect category feature trained by CNN in step S2.

3-12) The step S13 of the suspected defect is connected with the step S12 to mark the suspected defect of the PCB to be tested, and determine which kind of defect the marked suspected defect belongs to.

3-13) The step S10 of defect search and matching is connected with the step S13 of the suspected PCB defect and the step S2 of the CNN defect feature training. For step S13 of suspected PCB defects, use the PCB defect features and category features trained by the second-type defect library CNN in step S2 to search for matching. Filter false defects from suspected defects.

3-14) Step S14 of PCB defect verification is connected with step S10 of defect search and matching and step S3 of defect level feature training. The defects screened in step S10 are further filtered. Using the PCB defect level features trained by the second type of defect library CNN in step S3, check whether the suspected PCB defect obtained in step S10 is a real defect, delete false defects, and improve detection accuracy. If it is a true defect, mark it. At the same time, the defects that are not in the category are screened. When the detected real defect is a new type of defect, the new defect is marked and added to the defect library. If it is a false defect, analyze the cause of the false defect and provide a reference for subsequent inspection.

According to the deep learning-based real-time automatic detection method for two-dimensional PCB appearance defects proposed in the embodiment of the present invention, PCB images, standard PCB images and convolutional neural networks are used to detect defects on PCBs. Finding and eliminating errors early in the assembly process can avoid sending bad boards to subsequent assembly stages, while reducing repair costs will avoid scrapping unrepairable boards. The use of convolutional neural networks has the advantages of high speed, high accuracy, strong generalization ability, and clear structure.

In addition, the terms "first" and "second" are only used for descriptive purposes, and should not be construed as indicating or implying relative importance or implying the number of indicated technical features. Thus, a feature delimited with "first", "second" may expressly or implicitly include at least one of that feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise expressly and specifically defined.

In the description of this specification, description with reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples", etc., mean specific features described in connection with the embodiment or example , structure, material or feature is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, those skilled in the art may combine and combine the different embodiments or examples described in this specification, as well as the features of the different embodiments or examples, without conflicting each other.

Although the embodiments of the present invention have been shown and described above, it should be understood that the above-mentioned embodiments are exemplary and should not be construed as limiting the present invention. Embodiments are subject to variations, modifications, substitutions and variations.

S1~S14: Steps

The above and/or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of embodiments taken in conjunction with the accompanying drawings, wherein:

FIG. 1 is a flowchart of a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to an embodiment of the present invention.

FIG. 2 is a flowchart of a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to a specific embodiment of the present invention.

FIG. 3 is a flowchart of a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to another specific embodiment of the present invention.

FIG. 4 is a flow chart of a real-time automatic detection method for two-dimensional PCB appearance defects based on deep learning according to yet another specific embodiment of the present invention.

S1~S14: Steps

Claims (10)

A method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning, the method comprises the following steps: Step S1, establishing a two-dimensional PCB defect block image library through a PCB defect block image library, where the two-dimensional PCB defect block image library includes a two-dimensional PCB defect data pair image library and a two-dimensional PCB defect sheet data image library; Among them, the two-dimensional PCB defect block picture library is divided into two categories, one is a data pair picture library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, and the other is only a single two-dimensional PCB defect block data. image library; The step S1 also includes: establishing the two-dimensional PCB defect block picture library, the two types of the two-dimensional PCB defect block picture library include a variety of marked, classified and graded PCB defect block pictures of the PCB, and the PCB defect block pictures include multi-scale pictures; Step S2, through the CNN defect feature training block, for each type of PCB defect in the two-dimensional PCB defect block image library, use a convolutional neural network for training to extract PCB defect category features; Step S3, through the CNN defect level feature training block, the level of each type of PCB defect in the two-dimensional PCB defect block picture library is trained by using the convolutional neural network, and the PCB defect level feature is extracted; Step S4, extracting a two-dimensional PCB picture through a PCB picture extraction block, where the two-dimensional PCB picture includes a two-dimensional standard PCB picture and a two-dimensional PCB picture to be tested; Step S5, performing denoising processing on the two-dimensional PCB picture by filtering the denoising block; Step S6, through the image registration block, the two-dimensional PCB picture to be tested and the two-dimensional standard PCB picture are registered; Step S7, through the image gray value comparison block, compare and analyze the gray value of the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested to obtain a two-dimensional PCB residual picture; Step S8, through the image segmentation and positioning block, the two-dimensional standard PCB picture, the two-dimensional PCB to be tested picture and the two-dimensional PCB residual picture are divided and positioned to obtain the picture blocks of multiple pictures and corresponding coordinate; Step S9, filter the picture block of the two-dimensional PCB residual picture through the residual filter block to eliminate the comparison error caused by registration and other factors; Step S10, the defect search matching block searches for defect blocks in a plurality of picture blocks through the PCB defect feature trained by the convolutional neural network, and performs defect block matching; Step S11, the CNN defect identification block identifies the defects in the two-dimensional PCB picture to be tested through the convolutional neural network; Step S12, the PCB defect classification block classifies the defects marked in the two-dimensional PCB image to be tested by using the PCB defect category features trained by the convolutional neural network, and judges the defects that are not in the category to determine whether they are If the false defect is still a defect of a new category, the defect of the new category is supplemented to the two-dimensional PCB defect block picture library; Step S13, through the PCB defect suspected block, mark each type of defect in the two-dimensional PCB image to be tested as a suspected defect; and Step S14, the PCB defect verification block judges the authenticity of the suspected defect block through the PCB defect level features and classification information trained by the convolutional neural network, if the defect is a false defect, delete the corresponding defect, and analyze and generate a false defect If the defect is a true defect, it will be marked in the defect record. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 1, wherein, in the step S4, the two-dimensional PCB pictures extracted by the PCB picture extraction block include two types: One is to extract the two-dimensional standard PCB picture and the two-dimensional to-be-tested PCB picture; the other is to extract only the two-dimensional to-be-tested PCB picture; The step S4 also includes: In the detection method of comparing the standard PCB and the PCB hybrid detection method, extract the two-dimensional PCB image to be tested and the corresponding two-dimensional standard PCB image; and In the non-standard board comparison PCB detection method, only the two-dimensional PCB picture to be tested is extracted. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 2, wherein, in the step S5, denoising is performed on the two-dimensional PCB picture by filtering and denoising blocks, including: The two-dimensional standard PCB picture and the two-dimensional PCB to be tested picture are denoised by using an algorithm. In the detection method for comparing the standard PCB and the PCB hybrid detection method, the two-dimensional standard PCB picture and the all The two-dimensional PCB image to be tested is denoised, the noise in the image is removed, and the influence of the color of the bottom plate on the image is corrected, so that the color of the bottom plate of the two-dimensional standard PCB image, the two-dimensional PCB image to be tested, and the image in the defect library are consistent ; In the non-standard board comparison PCB detection method, only the two-dimensional PCB image to be tested is denoised, the noise in the image is removed, and the influence of the color of the bottom plate on the image is corrected, so that the two-dimensional PCB image to be tested, The color of the bottom plate of the pictures in the defect library is the same. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 3, wherein the two-dimensional standard PCB picture and the two-dimensional PCB picture to be tested are registered and compared with gray values , image segmentation, the method further comprises the following steps: In the detection method for comparing standard PCBs and the PCB hybrid detection method, the step S6, the step S7 and the step S8 include: In the detection method for comparing standard PCBs and the PCB hybrid detection method, the two-dimensional standard PCB picture and the two-dimensional PCB to be tested picture are registered, and the two-dimensional standard PCB picture after denoising is processed. Correcting the two-dimensional PCB picture to be measured, comparing the gray value of the two-dimensional standard PCB picture and the two-dimensional PCB to be measured picture, and calculating the residuals of the pictures to obtain a two-dimensional PCB residual picture; The two-dimensional standard PCB picture, the two-dimensional PCB to be tested picture and the two-dimensional PCB residual picture are divided to obtain a plurality of picture blocks and corresponding coordinates; The two-dimensional standard PCB picture is registered to detect whether the two-dimensional standard PCB picture has rotation, deformation, uneven light and light reflection, and if there is, use an algorithm to correct it; The two-dimensional PCB image to be tested is compared with the processed two-dimensional standard PCB image to determine whether the position, light and color are consistent. If they are inconsistent, the two-dimensional PCB image to be tested is corrected by an algorithm. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 3, wherein, in the non-comparison reference detection method, the step S8 further comprises: analyzing the two-dimensional PCB picture to be tested Perform segmentation to obtain multiple image blocks and corresponding coordinates. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 5, wherein, in the step S9, the picture blocks of the two-dimensional PCB residual pictures are filtered, the comparison errors are eliminated, and the deletion There is no residual picture block in the two-dimensional PCB residual picture; The step S10 and the step S12 further include: In the detection method for comparing PCBs without standard boards, the two-dimensional PCB defect single data picture library is used; In the non-standard board comparison PCB detection method and the PCB hybrid detection method, the convolutional neural network is used for each type of PCB defect in the two-dimensional PCB defect image library and the two-dimensional PCB defect image library. Road training, extracting PCB defect features and PCB defect category features; The step S10 uses the PCB defect features trained by the convolutional neural network to search and discriminate the PCB defects in a plurality of picture blocks, and use a single two-dimensional PCB defect picture library convolutional neural network to train the category features of the PCB defects, The step S12 classifies the defects marked in the two-dimensional PCB image to be tested, judges the defects that are not in the category, determines whether it is a false defect or a defect of a new category, and supplements the defect of the new category to the Defect Gallery. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 5 or 6, wherein, In the detection method for comparing standard PCBs and the PCB hybrid detection method, the convolutional neural Network training to extract PCB defect features and PCB defect category features; The step S11 and the step S12 further include: Using a two-dimensional PCB defect data pair image library composed of two-dimensional PCB defect blocks and corresponding non-defective standard PCB block pictures, the step S11 uses the PCB defect features trained by the convolutional neural network, and the step S9 pairs The residual image is used for defect identification; the two-dimensional PCB defect image library convolutional neural network is used to train the category features of PCB defects, and the step S12 is to classify the defects marked in the two-dimensional PCB image to be tested, and the defects that are not in the category are classified. The defect in the defect is judged to determine whether it is a false defect or a defect of a new type, and the defect of the new type is added to the defect picture library. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claim 7, wherein, In the PCB hybrid detection method, after the step S12, the method further includes: Use the PCB defect feature of the two-dimensional PCB defect image library trained by the convolutional neural network to search and match the marked suspected defect blocks, delete the false defects therein, and improve the recognition accuracy; and After the step S13, search and match the suspected defect in the defect picture by using the PCB defect feature trained by the convolutional neural network. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning as described in claim 1 or 8, wherein, For PCB defect verification, judge the authenticity of the PCB defect through the PCB defect level feature information trained by the convolutional neural network in the step S3, if the defect is a false defect, delete the false defect, and analyze and generate The cause of the false defect, if the defect is a true defect, is marked in the defect record. The method for real-time automatic detection of two-dimensional PCB appearance defects based on deep learning according to claims 1 to 9, wherein, in the process of searching and judging defects, the defects are successively eliminated through multiple identification iterations.

Images