TWI749714B - Method for defect detection, method for defect classification and system thereof - Google Patents

Abstract

A method for defect detection, a method for defect classification and a system are provided. The system obtains images of a plurality of non-defective units by one or more cameras. A computation device performs a machine-learning method upon the images so as to learn the image features of the non-defective unit. A non-defective unit model used to describe the image features of the non-defective unit is established. The non-defective unit model is applied to images of a device under test for retrieving defect data thereof. The defect data is referred to for determining whether or not the device under test is a defective unit. If the defect data is confirmed as defects, a deep-learning method is performed on the defects. The defects can be classified.

Description

Defect detection method, defect classification method and system

The manual discloses a flaw detection method, especially a method and system for flaw detection and flaw classification based on image features learned from good products.

In the industrial manufacturing industry, in addition to judging defects by human hands or eyes, with the development of computers and learning algorithms, the technology of learning defects in images is developed, and software methods can be used to perform defect judgments more efficiently. . However, judging from the current technology, using image recognition technology to detect product defects requires the collection of a large number of images, as well as the inability to collect complete images of defects, resulting in poor machine learning effects.

In addition, in machine learning technology, deep learning algorithms can be used to learn from a large number of sample images to obtain a defect judgment model. However, to obtain an effective defect judgment model, it is necessary to collect various types of defect images, such as In the learning process, you must mark each defect location, and then perform software learning. The operation is complicated and time-consuming. It still requires a lot of learning and time costs. Furthermore, the use of deep learning algorithms also has the problem of high hardware costs. If the learning results Not as expected, but need to learn again.

The specification discloses a defect detection method, a defect classification method and a system, which are different from the general method of learning defect image characteristics for defect products to determine the defect. According to an embodiment, the proposed defect detection system includes one or more image captures. The device is used to obtain images of multiple good products or an object under test. It is equipped with a computing device, including a processing circuit and an interface circuit. It can receive one or more image capture devices through the interface circuit to capture multiple good products or objects under test. The output image is processed by the processing circuit to perform a defect detection method for the one or more images.

In the defect detection method, a machine learning method is performed on the multiple images of the multiple good products obtained to learn the image characteristics of the multiple good products, and a good product model to describe the characteristics of the good product images is established, so this good product can be applied The model is applied to the image of the object to be tested to obtain the defect data in the image of the object to be tested, and then according to the detection parameters, it is judged whether the object to be tested is a defective product by comparing the defect data in the image of the object to be tested.

Further, the defect data of the object to be measured may include one or any combination of the position, area, color, brightness, and shape of one or more defects according to the properties of the object to be measured.

In this way, the various defect data of the object under test can be compared with the threshold set for each defect data in the detection parameters to determine whether the object under test is a defective product. The type of defect to be judged includes at least dirt, omission, damage, and color change. Change by area.

In one embodiment, after the system obtains the good product image, it can be cut into multiple block images, and then machine learning method is used to learn the good product image characteristics in each block image to form a good product model describing the good product image characteristics and become normal for each block. The distribution standard means that the normal distribution standard of each block includes the intensity, area and color distribution of each pixel.

According to the defect classification method embodiment proposed in the disclosure book, multiple images of multiple good products are also obtained first, and the image features of the multiple good products are learned by the machine learning method to establish a good product model to describe the image characteristics of the good products, and then shoot Multiple DUTs to obtain multiple images of multiple DUTs. The good product model can be used to obtain multiple images with defect data from these DUTs. According to the pre-defined defect categories, a deep learning method can be implemented. Defects in the image can be classified to generate defect categories, and create each classified defect item.

Furthermore, in the step of shooting images of the object to be measured, multiple images of the object to be measured can be continuously shot, and the position of the object to be measured can be dynamically located, and the same detection area in the multiple images can be accurately positioned to mark many of them. Information on the location and characteristics of each defect. In addition, when positioning the detection block, you can also select a fixed or arbitrary shape to select the detection range.

Further, after obtaining one or more defect locations, the image can be cropped to obtain multiple defect location blocks, so that when a deep learning method is subsequently performed, the efficiency of deep learning can be greatly improved.

After that, one or more images of another object to be tested are obtained, and after the defect data of the object to be tested is obtained using the good product model, the defects of the object to be tested can be classified according to the defect category.

In order to further understand the features and technical content of the present invention, please refer to the following detailed description and drawings about the present invention. However, the provided drawings are only for reference and description, and are not used to limit the present invention.

The following are specific specific examples to illustrate the implementation of the present invention. Those skilled in the art can understand the advantages and effects of the present invention from the content disclosed in this specification. The present invention can be implemented or applied through other different specific embodiments, and various details in this specification can also be based on different viewpoints and applications, and various modifications and changes can be made without departing from the concept of the present invention. In addition, the drawings of the present invention are merely schematic illustrations, and are not drawn according to actual size, and are stated in advance. The following embodiments will further describe the related technical content of the present invention in detail, but the disclosed content is not intended to limit the protection scope of the present invention.

It should be understood that although terms such as “first”, “second”, and “third” may be used herein to describe various elements or signals, these elements or signals should not be limited by these terms. These terms are mainly used to distinguish one element from another, or one signal from another signal. In addition, the term "or" used in this document may include any one or a combination of more of the associated listed items depending on the actual situation.

The manual discloses a defect detection method, defect classification method and system, which is different from the general method of establishing a defect detection model by learning the image characteristics of defective products. The defect detection method proposed in the disclosure book is based on non-defective unit images. Basic, learn the image characteristics of good products, as the basis for defect inspection.

Compared with the need to obtain defective products with various defect patterns, it is easier to obtain good samples. According to the technical concept of the defect detection process, the first stage of quality inspection requires sampling of multiple good products, but the collection of good samples is relatively easy. After that, the machine-learning method is used to learn the image features of the good product, the purpose is to find the correlation between the image features that define the good product and the various features, which is relatively more powerful than the hardware computing power required to learn the defect image characteristics Low, and better efficiency. The flawed image is screened out by machine learning, and then the accuracy can be judged manually or by machine. After parameter adjustment and multiple trainings, the resulting model can be refined. After that, according to the pre-defined defect categories, proceed to the second stage to establish the defect classification using the deep-learning method, which is used to point out the categories of various defects. One of the advantages is that after the suspicious flaws are screened out in the first stage, the range of flawed images to be classified can be greatly reduced, so that the deep learning classification can be calculated more quickly.

For an embodiment of the system for realizing the defect detection method, refer to the schematic diagram of the system shown in FIG. 1 for taking a sample with a camera.

The main components of the defect detection system include one or more image capture devices implemented by various forms of optical equipment to achieve the goal of automatic optical inspection. As shown in the figure, there are cameras 111, 112, 113 for obtaining good products or objects to be tested. The image of the sample 12 is taken under a light source 14, as shown in the schematic diagram. When the system wants to build a good product model, multiple good products are replaced as the sample 12, and the camera 111, 112, 113 shoots the sample 12 from different angles to obtain the good product. image. The defect detection system is provided with a computing device 10, the main components of which include a processing circuit 101, a storage device 103, and an interface circuit 105. Through the interface circuit 105, one or more cameras 111, 112, 113 can be used to photograph the good product or the object under test. For one or more images, the processing circuit 101 executes a defect detection method for the obtained one or more images. The storage device 103 is used to store image data and data obtained through analysis. According to an embodiment, the defect detection system may be provided with a control circuit 107, which is a driving circuit used by the computing device 10 to control the operation of the cameras 111, 112, 113, so that the user can operate the operation and shooting of the camera through the control circuit 107 .

According to the embodiment of the flaw detection system described above, the processing circuit 101 in the computing device 10 executes the flaw detection method and the flaw classification method, especially the machine learning method and the deep learning method applied therein. Common machine learning methods such as (but not limited to) linear regression, logistic regression, support vector machine, classification and regression tree, etc.

The above-mentioned machine learning method and other related learning algorithms can automatically analyze the image characteristics of specific targets (such as good products and defective products) in the image from a large amount of collected image data, including learning unknown features, and the goal is to be able to obtain various data The distribution (mainly for good products), and the relationship between the data. However, in order to ensure that machine learning can obtain a model that meets the requirements, it is necessary to provide a large number of samples (good and defective) image data in the process of learning training data. Take the method disclosed in the disclosure book as an example, first obtain the good product on the production line. The image data is used as the training data of the learning algorithm.

According to the embodiment, the machine learning method is used to learn the features in the image of a specific product on the production line, where the good product is mainly obtained from the production line, and the image characteristics of the good product are learned from the image of the good product, such as area characteristics and color characteristics. Features, brightness features, shape features, etc., build a good product model from these good product image characteristics, which in turn is used to detect defective products that do not match these characteristics. For example, the types of defects that can be detected by the good product model are oxidation, dirt, scratches, poor printing, and breakage.

FIG. 2 shows a flowchart of an embodiment of a model for learning image characteristics of a good product in a defect detection method.

First, obtain multiple images of multiple good products on the applied production line (step S201), and then learn the image features of the good products by machine learning (step S203). In the process of learning image features, according to an embodiment, the Perform image analysis on grayscale or color images, and use machine learning to obtain the location, area, color, brightness, and shape of the recognizable image of a specific product (good product). Obtain the good product rules in the image pixels, and establish a good product model (step S205). For example, the good product model describes the image characteristics of the good product, which can be represented by feature vectors, image attribute tags, and pixel values.

In order to be able to detect defects based on the good product model, the detection parameters are determined, including various thresholds for judging defective products (step S207). According to one of the embodiments, when performing defect detection, the defect detection parameters can be determined based on one or the other according to the properties of the object to be tested (for example, the object to be tested is an electronic product, a circuit board, a printed product, a plastic product, a metal product, etc.) One or any combination of the position, area, color, brightness, and shape of multiple flaws. In this way, when detecting flaws, you can compare the flaw data in one or more images of the object to be tested based on these detection parameters. Whether the object to be tested is a defective product.

According to the good product model established according to the process described in FIG. 2, FIG. 3 then shows a flow chart of an embodiment of the defect detection method.

Use the camera provided by the system to shoot an object under test (step S301), obtain one or more images of the object under test (step S303), and then apply the good product model to obtain the non-good product model in the image of the test object according to the properties of the test object Defect data in one or more images of the good image characteristics described in (step S305), obtain the position of the suspected defect and related defect data, such as one or any combination of area, color, brightness, and shape (step S307) , Then, according to the attributes of the object to be tested, the detection parameters, namely the image comparison detection parameters that are judged as suspected flaws, can be obtained according to the attributes of the test object (step S309), which will be based on the judgment conditions (strength, tolerance, and Need to adjust) to filter out the part that determines the defect. When screening, the defect data (one or any combination of area, color, brightness and shape) of the object to be tested can be compared with the setting for each defect data in the detection parameters Threshold to determine whether the object to be tested is a defective product (step S311).

It is mentioned here that the above method uses good product images as the training model material, and there is no need to judge and mark defects in the process, so a lot of computing time can be saved, and the establishment of a good product model can determine the defects on the object under test. , And then apply the defect judgment condition.

In the method of judging whether there is a defect, the image can also be divided into mxn blocks, and each block can obtain the image characteristics of the good product according to the learning result, such as the intensity, area and color of each pixel in each block (image Therefore, the good product model describing the characteristics of the good product image can be the normal distribution standard of each block. For related embodiments, please refer to the flowchart of the embodiment of establishing the defect judgment standard in the defect detection method shown in FIG. 4.

First obtain one or more good images (step S401), and cut each image into multiple block images (step S403), and then refer to the description of the above embodiment to learn the good images in each block image by machine learning Features (step S405), so that the normal distribution standard of each block can be established (step S407). In other words, the good product model established by the machine learning method can be represented by the normal distribution standard of each block, and the normal distribution standard of each block includes the distribution of the intensity, area and color of each pixel. Therefore, the distribution standard (good product model) can be used to obtain the defect data in one or more images of the object to be tested, and the detection parameters set according to the attributes of the object to be tested can be used to compare the images of each block in the image of the object to be tested. Based on the element image value, it can be judged whether each area has a defect, and then as a whole, it can be judged whether the object to be tested is a defective product.

According to one embodiment, after learning the image characteristics of good products, the system can generate multiple (for example, 3) screening rate values with different intensities, so that the user can use the system's suggested parameter values in the user interface provided by the system to pass The photos show different intensities of screening effects, and the appropriate values can be selected intuitively.

For the distribution of the image features of each block in the above embodiment, refer to the schematic diagram of the normal distribution standard of a block in the defect detection method shown in FIG. 5.

The figure shows the image feature distribution of a certain block in the image of the good product or the object to be tested. The horizontal axis is the intensity value of the pixel, and the vertical axis is a certain detection parameter, such as the frequency value of the good product parameter. The displayed curve can be various The intensity, area, and color distribution of the pixels. There are two vertical dotted lines. Threshold 1 501 and Threshold 2 502 are used to determine whether they belong to the normal range. Threshold 1 501 and Threshold 2 502 are used to screen defect images. Actual demand adjustment.

For good products, the normal distribution standard of the entire image can be obtained through the machine learning method, as shown in Figure 6. Then, the subsequent comparison of the test object also cuts the image into multiple blocks, and judges whether it is a suspected defect based on the distribution of each block, as a preliminary screening method, thus effectively reducing the calculation requirements for defect judgment.

Among them, the good product model established by the normal distribution standard of each image block of the good product can perform defect detection, as shown in FIG. 7 as a flowchart of another embodiment of the defect detection method.

First, take the image of the object to be measured (step S701), and then divide the image of the object to be measured into multiple block images (step S703), and obtain the image characteristics of each block, such as the intensity, the pattern area and the color feature distribution, etc. (Step S705), the image feature distribution of each block can also be obtained. At this time, after comparing the normal distribution standard of each block (step S707), in accordance with the threshold range described in the above embodiment, it can be judged whether each block is a defect. And then judge whether the whole meets the condition of the defective product (step S709).

It is worth mentioning that the defect detection method proposed in the manual can continuously capture multiple images of the object under test, and the system can dynamically locate the position of the object under test, and the image processing program for multiple block images can be high-speed Different images (image files) can be changed, and the same inspection area can be accurately positioned, and multiple defect locations and characteristic information can be marked at one time. According to the embodiment described in FIG. 4, multiple flaws can be defined after learning the characteristics of a good product image through a machine learning method. In this way, with only one inspection, multiple flaws caused by different causes in the image of the object to be tested can be marked.

Furthermore, because the types of objects to be tested are changeable, when locating the detection area, the detection range can be selected in any shape, and the non-detection area can be set independently, and the fixed or arbitrary shape can be selected through a single frame. Frame selection of any shape can increase the user's convenience in operating the software. In addition, by setting the function of not detecting the area, it can speed up the personnel to judge the scope of the defect and focus only on the inspection area.

For example, the image characteristics of each area (according to the attributes and requirements of the object to be measured) include the intensity change (brightness), area change, and color change calculated from the image characteristics of each area of the object to be measured. The comparison is performed by the system according to the requirements. The established conditions for judging defects, including the scope of defects (such as threshold one 501 and threshold two 502), if one of the blocks is judged to meet the defect conditions, but it may still not be considered as a defect, the further detection parameters can be set to meet the defect The number or location of the condition of the block, so all or part of the block still needs to be judged whether the object under test is a defective product.

The flaw detection methods described in the above embodiments are the first stage in the operation of the entire system. When the first stage uses machine learning to establish a good product model to screen out the suspicious flaws of the object to be tested, compare the pre-defined flaw categories and return In the second stage, the deep learning method can be used to establish the defect classification to point out the categories of various defects, and the results of the first stage can greatly reduce the range of the defect images to be classified, so that the deep learning classification can be calculated more quickly.

It is worth mentioning that after the defect classification is obtained through deep learning, it can be adjusted again to use the deep learning method to improve the judgment of the defect classification through multiple training.

Fig. 8 shows a flowchart of an embodiment of a defect classification method. The system can firstly specify defect items according to requirements, including previously collected defect patterns, as a basis for defect classification. In the process, the image of the object to be tested is first obtained (step S801). Preferably, multiple images of the object to be tested are taken to obtain multiple images of the plurality of objects to be tested, and then the good product model is applied (step S803). The good product model selected by object attributes obtains multiple images with defect data from multiple objects to be tested. Among them, the position and form of flaws, such as area, color, brightness, and shape, can be obtained for the entire image of the object under test, or the normal distribution of color or brightness can be compared after being cut into multiple image blocks. Standard, the defective block can be obtained (step S805).

It is mentioned here that in the defect classification method, before the classification is performed, after obtaining one or more defect positions, a plurality of defect position blocks can be obtained by a cropped image, so that the method is When a deep learning method is subsequently performed, the efficiency of deep learning can be greatly improved.

Next, learn various defect features by deep learning method to classify defects. Methods such as (but not limited to) logistic regression model and k-fold cross validation are used, and input Through learning, the defect image can obtain a function describing various defect categories, and establish each classified defect item, including the establishment of a defect detection model (step S807). After that, according to this flaw detection model, flaw detection and classification can be performed on the products on the production line (step S809), in which the image of the object to be tested is also input, in addition to judging whether there is a flaw, it can also be classified for the flaw to facilitate subsequent follow-up Correct the system parameters.

Figures 9A to 9E show schematic diagrams of several examples of defect classification, but the defect classification is not limited to this example. The example shown in the figure is a letter on a printed matter as an example. After shooting the printed matter to obtain an image, the defective part is judged by the good product model, and then the defect detection model is used for classification. For example, as shown in Figure 9A, it is shown that there is a dirty defect; Figure 9B has a defect caused by poor printing; Figure 9C is damaged due to mechanical scratches during the printing process; Figure 9D shows that because The problem of color change caused by poor printing pigments or missing in the process; and the area change caused by printing problems as shown in Figure 9E.

To sum up, in terms of existing technology, for example, in manufacturing, most product inspections still rely on manual inspection. Machine inspection technology is mainly assisted by manual labor. Therefore, it is labor intensive, inefficient, and difficult to perform more precise. The detection is even prone to errors. Therefore, according to the defect detection and classification method described in the above embodiment, the proposed defect detection system uses the good product image to perform the machine learning method, learns the image feature of the good product, and establishes it to describe the good product image feature The next application only needs to set the flaw strength and flaw size tolerance, and then adjust the inspection rigor according to actual needs. According to the set inspection parameters, it can detect the defects of the item. It can also use deep learning to classify the defects and strengthen the detection accuracy. In this way, in response to the current testing needs that occur on the production line, a powerful but simple testing capability can be achieved, and the product yield rate can be greatly improved.

The content disclosed above is only a preferred and feasible embodiment of the present invention, and does not limit the scope of the patent application of the present invention. Therefore, all equivalent technical changes made using the description and schematic content of the present invention are included in the application of the present invention. Within the scope of the patent.

10: Computing device

101: processing circuit

103: storage device

105: Interface circuit

107: Control circuit

111, 112, 113: camera

12: sample

14: light source

501: Threshold One

502: Threshold Two

Steps S201～S207: Steps to build a good product model

Steps S301～S311: Defect detection step

Steps S401～S407: Steps to establish the normal distribution standard of blocks

Steps S701～S709: Defect detection step

Steps S801~S809: Defect classification step

Figure 1 shows a schematic diagram of a system that uses a camera to photograph samples;

FIG. 2 shows a flowchart of an embodiment of learning image features of good products in a defect detection method to establish a model;

Figure 3 shows a flow chart of an embodiment of the defect detection method;

Figure 4 shows a flowchart of an embodiment of establishing a defect judgment standard in a defect detection method;

Figure 5 shows a schematic diagram of the normal distribution standard of a certain block in the defect detection method;

Figure 6 shows a schematic diagram of the normal distribution standard of multiple blocks in each image in the flaw detection method;

FIG. 7 shows a flowchart of the second embodiment of the defect detection method;

FIG. 8 shows a flowchart of an embodiment of a defect classification method; and

Figures 9A to 9E show schematic diagrams of several types of defects.

S301: Shoot the object under test

S303: Obtain the image of the object under test

S305: Apply good product model

S307: Get defect location

S309: Compare detection parameters

S311: Determine whether it is a defective product

Claims (14)

A defect detection method includes: obtaining multiple images of multiple good products; cutting each image of each good product into multiple regional images, and performing a machine learning method on each image of each good product to learn the good images in each regional image Feature to obtain the image features of the multiple good products, and establish a good product model to describe the image characteristics of the good product, and the good product model is represented by the normal distribution standard of each block; shoot a test object to obtain the test object One or more images of; using the good product model to obtain the one or Defect data in multiple images; and compare each defect data of the object under test with the threshold set for each defect data in the detection parameters to determine whether the object under test is a defective product, which is set according to each defect data The defect types judged by the threshold include at least dirt, omission, damage, color change and area change. The defect detection method according to claim 1, wherein after the one or more defect positions are obtained, a plurality of defect position blocks are obtained by cropping the image, and when a deep learning method is subsequently performed, it can greatly Improve the efficiency of deep learning. The defect detection method according to claim 1, wherein the deep learning method is executed according to the predetermined defect category according to the threshold set by each defect data to classify the defects. The flaw detection method according to claim 1, wherein in the step of shooting the image of the object to be measured, after obtaining multiple images of the object to be measured by continuous shooting, the position of the object to be measured is dynamically located, and the position of the object to be measured is accurately located The same detection block in multiple images is used to mark the location and characteristic information of multiple defects. The defect detection method according to claim 4, wherein, when locating the detection block, You can choose a fixed or arbitrary shape frame to select the detection range. The flaw detection method according to claim 1, wherein the normal distribution standard of each block includes the characteristic distribution of the intensity, area, and color of each pixel. A defect classification method includes: obtaining multiple images of multiple good products; cutting each image of each good product into multiple regional images, and performing a machine learning method on each image of each good product to learn the good images in each regional image Feature to obtain the image features of the multiple good products, and establish a good product model to describe the image characteristics of the good product, and the good product model is represented by the normal distribution standard of each block; shoot multiple objects to be tested to obtain the multiple Multiple images of the object under test; apply the good product model to obtain multiple images with defect data from the multiple objects to be tested; perform a deep learning on the defect images in the multiple images according to the predefined defect category The method is to classify defects, generate the defect categories in the multiple images, and create each classified defect item; obtain one or more images of another object to be tested; and apply the good product model according to one or more of the object to be tested One or any combination of the position, area, color, brightness, and shape of multiple defects, obtain the defect data in the one or more images of the object to be tested, and determine the defect based on the threshold set by each defect data The type includes at least dirt, omission, damage, color change and area change, and classify the defect of the object to be tested based on this. The defect classification method according to claim 7, wherein, in the step of shooting the image of the object to be measured, after obtaining multiple images of the object to be measured by continuous shooting, the position of the object to be measured is dynamically located, and the position of the object to be measured is accurately located The same detection block in multiple images is used to mark the location and characteristic information of multiple defects. The defect classification method according to claim 8, wherein, when locating the detection block, You can choose a fixed or arbitrary shape frame to select the detection range. A defect detection system includes: one or more image capturing devices for obtaining images of a plurality of good products or an object to be tested; a computing device including a processing circuit and an interface circuit through which the one is received One or more images obtained by or a plurality of image capturing devices photographing the plurality of good products or the object to be tested, and executing a defect detection method for the one or more images through the processing circuit, including: obtaining a plurality of good products Images; each image of each good product is cut into multiple regional images, and a machine learning method is performed on each image of each good product to learn the image characteristics of the good product in each regional image to obtain the image characteristics of the multiple good products , Establish a good product model to describe the image characteristics of the good product, and the good product model is represented by the normal distribution standard of each block; shoot a test object to obtain one or more images of the test object; apply the good product model according to One or any combination of the position, area, color, brightness, and shape of the one or more defects of the object to be tested is used to obtain the defect data in the one or more images of the object to be tested; Each defect data of the object is compared with the threshold set for each defect data in the detection parameters to determine whether the object to be tested is a defective product. Among them, the type of defect judged according to the threshold set by each defect data includes at least dirt, omission, Damage and color change vary by area. The defect detection system according to claim 10, wherein after the one or more defect positions are obtained, a plurality of defect position blocks are obtained by cropping the image, and when a deep learning method is subsequently performed, it can greatly Improve the efficiency of deep learning. The defect detection system according to claim 11, in which the deep learning method is executed according to the threshold set by each defect data and according to the pre-defined defect category, To classify blemishes. According to the flaw detection system of claim 10, in the implemented flaw detection method, after continuously shooting and obtaining multiple images of the object to be measured, the position of the object to be measured is dynamically located, and the location of the multiple images is accurately located The same inspection block is used to mark the location and characteristic information of multiple flaws. The defect detection system according to any one of claims 10 to 13, wherein the normal distribution standard of the block includes the characteristic distribution of the intensity, area, and color of each pixel.

Images