TWI722861B - Classification method and a classification system - Google Patents

Abstract

The invention discloses a classification method and a classification system. The classification system can use the classification method to classify to-be tested object into the good product area or the bad product area. The classification method comprise an automatic optical inspection device classification step. In the automatic optical inspection device classification step, the automatic optical inspection device will capture the image of to-be tested object to generate a to-be tested image, and the automatic optical inspection device will analyze the to-be tested image to classify it as good category or one of N preset defect categories. If the to-be tested image is classified into one of the N preset defect categories, a machine learning classification step is performed to use the one of the N machine learning models corresponding to the preset defect category to analyze to-be tested image to determine whether a defect corresponding to the preset defect category appears in the to-be tested image so as to determine whether the to-be tested object is a good or bad product.

Description

Classification method and classification system

The invention relates to a classification method and a classification system, in particular to a classification method and a classification system for classifying an object to be tested by using an automatic optical detection device and a machine learning model.

Automated Optical Inspection (AOI) has been widely used in various inspection occasions. However, in actual applications of automatic optical inspection devices, misjudgments often occur, which causes troubles for related industries.

The invention discloses a classification method classification system, which is mainly used to improve the problem that the automatic optical detection device (AOI) is prone to misjudgment when various types of test objects are detected by the automatic optical detection device (AOI).

One of the embodiments of the present invention discloses a classification method, which is suitable for a classification system. The classification system includes an automatic optical inspection device, a processing device and a classification device. The classification system can execute the classification method to classify a test object into A good product area or a defective product area, the classification method includes: an automatic optical inspection device classification step and a machine learning classification step. The classification step of the automatic optical inspection device includes: an image capturing step and an image classification step. Image capture step: use an automatic optical inspection device to capture an image of the object to be tested to generate an image to be tested; image classification step: use an automatic optical inspection device or processing device to analyze the image to be tested to determine whether the image to be tested belongs to A good product category and which of the N preset defect categories; where N is a positive integer greater than 1; if it is determined that the image to be tested belongs to the good product category, the classification device is used to move the object to be tested to the good product area; if judged If the image to be tested belongs to one of the N preset defect categories, a machine learning classification step is performed; the processing device stores N machine learning models corresponding to the N preset defect categories, and the N machine learning models are used separately A training set that is not exactly the same for model training, and each training image file includes multiple training images and at least one good product image file corresponding to one of the N preset defect categories; machine learning classification The steps include: a loading step: load the image to be tested into one of the corresponding machine learning models according to the classification result of the image classification step; a judging step: use the machine learning model loaded with the image to be tested to determine The image to be tested is a good product or a defective product; if the machine learning model loaded with the image to be tested is used to determine that the image to be tested is a good product, the classification device is used to transfer the object to be tested to the good product area; if used The machine learning model loaded with the image to be tested determines that the image to be tested is a defective product, and the classification device is used to transfer the object to be tested to the defective product area.

One of the embodiments of the present invention discloses a classification system, which includes: an automatic optical inspection device, a processing device, and a classification device. The classification system can execute a classification method to classify a test object into a good product area or a non-defective area. In the good product area, the classification method includes: an automatic optical inspection device classification step and a machine learning classification step. The automatic optical inspection device classification step includes: an image capture step: use the automatic optical inspection device to capture an image of the object to be tested to generate an image to be tested; an image classification step: use the automatic optical inspection device or processing device to analyze The image to be tested is used to determine which of a good product category and N preset defect categories the image to be tested belongs to; where N is a positive integer greater than 1; if it is determined that the image to be tested belongs to a good product category, the classification device will be used to The test object is transferred to the good product area; if it is determined that the image to be tested belongs to one of the N preset defect categories, a machine learning classification step is performed; the processing device stores N machine learnings corresponding to the N preset defect categories Model, N machine learning models use a different training set (training set) for model training, and each training image file contains multiple training image files corresponding to one of the N preset defect categories And at least one good product image file; the machine learning classification step includes: a loading step: according to the classification result of the image classification step, the image to be tested is loaded into one of the corresponding machine learning models; a judging step: using loading There is a machine learning model for the image to be tested to determine whether the image to be tested is a good product or a defective product; if the machine learning model loaded with the image to be tested is used to determine that the image to be tested is a good product, the classification device is used to The object to be tested is transferred to the good product area; if the machine learning model loaded with the image to be tested is used to determine that the image to be tested is defective, the classification device is used to transfer the object to be tested to the defective product area.

In summary, the classification method and classification system of the present invention are designed through automatic optical inspection device classification steps and machine learning classification steps, and make the images to be tested that are judged to belong to different preset defect categories, using corresponding machine learning models Designs such as reconfirmation can greatly improve the problem that the automatic optical inspection device is prone to misjudgment when the conventional automatic optical inspection device is used to classify the object to be tested.

In order to further understand the features and technical content of the present invention, please refer to the following detailed descriptions and drawings about the present invention, but these descriptions and drawings are only used to illustrate the present invention, and do not make any claims about the protection scope of the present invention. limit.

In the following description, if it is pointed out, please refer to the specific drawing or as shown in the specific drawing, it is only used to emphasize that in the subsequent description, most of the related content appears in the specific drawing. However, it is not limited that only the specific drawings can be referred to in this subsequent description.

Please refer to FIG. 1, which shows a block diagram of the classification system of the present invention. The classification system 100 of the present invention includes: an automatic optical inspection device 1, a processing device 2, a storage device 3 and a classification device 4. The Automated Optical Inspection (AOI) device includes an image capture device 11, and the image capture device 11 is used to capture an image of an object to be tested to generate an image to be tested 111. The processing device 2 is electrically connected to the automatic optical inspection device 1, and the processing device 2 can receive the image to be tested 111 transmitted by the automatic optical inspection device 1.

The processing device 2 may be, for example, various computers, servers, and other equipment, and the processing device 2 may, for example, include a central processing unit (CPU) or a graphics processing unit. The storage device 3 can be various types of memory, hard disk, etc., and the storage device 3 can be integrated with the processing device 2 into an independent device (such as server equipment, industrial computer equipment, etc.), which is not limited here. The sorting device 4 is electrically connected to the processing device 2, and the sorting device 4 can transfer the test object passing through the automatic optical inspection device 1 to a good product area or a defective product area according to the instructions transmitted by the processing device 2. In practical applications, the specific hardware components included in the classification device 4 may vary according to the type, appearance, size, etc. of the object to be measured, and it is not limited herein.

The classification system 100 of the present invention is used to implement a classification method of the present invention to classify a test object into the good product area or the defective product area. The classification method of the present invention includes: an automatic optical inspection device classification step S1 and a machine learning classification step S2. The classification system 100 sequentially executes the automatic optical inspection device classification step S1 and the machine learning classification step S2 to determine The test object is classified into the good product area or the bad product area.

Please refer to FIG. 1 and FIG. 2 together. FIG. 2 is a schematic flowchart of the automatic optical inspection device classification step of the classification method of the present invention. The automatic optical inspection device classification step S1 includes:

An image capturing step S11: using the automatic optical inspection device 1 to capture an image of the object to be tested to generate an image to be tested 111;

An image classification step S12: use the automatic optical inspection device 1 or the processing device 2 to analyze the image to be tested 111 to determine which of the good product category and the N preset defect categories the tested image 111 belongs to; where N is A positive integer greater than 1;

If it is determined that the image to be tested 111 belongs to the good product category, the classification device 4 is used to transfer the object to be tested to the good product area; if it is determined that the image to be tested 111 belongs to one of the N preset defect categories, the machine learning classification is performed Step S2. In other words, the automatic optical inspection device 1 and the processing device 2 can analyze the image to be tested to determine which of the N+1 categories (1 good product category + N preset defect categories) the test image belongs to.

Please refer to FIG. 3, which shows a schematic diagram of a practical example of the classification method of the present invention. It is assumed that the object to be tested is a contact lens, and the contact lens (the object to be tested) is prone to 3 types of defects during the production process, which are respectively For long scratches, holes, and irregular debris, the relevant personnel can define the three types of defects as a first preset type, a second preset type, and a third preset. Set the defect category.

In the above-mentioned image classification step S12, the automatic optical inspection device 1 (or the processing device 2) will determine whether there are defects belonging to the first preset defect category, defects belonging to the second preset defect category, or The defects belonging to the third preset defect category, that is, the automatic optical inspection device 1 (or the processing device 2) will use the image 111 to determine whether the object to be tested has long scratches, holes, or not. One of the defects of regular debris.

If the automatic optical inspection device 1 (or the processing device 2) determines that there are no defects belonging to the first preset defect category, defects belonging to the second preset defect category, or defects belonging to the third preset defect category in the image to be tested 111 Defects (that is, when the automatic optical inspection device 1 judges that the object to be tested does not have long strip scratches, holes, or irregularities based on the image 111 to be tested), the automatic optical inspection device 1 (or the processing device 2) ) It will be determined that the image to be tested 111 belongs to the good product category, and the automatic optical inspection device 1 (or the processing device 2) will accordingly control the classification device 4 to transfer the tested object to the good product area.

In contrast, if the automatic optical inspection device 1 (or the processing device 2) determines that a defect belonging to the first preset defect category, a defect belonging to the second preset defect category, or a third preset defect appears in the image to be tested 111 Type of defects (ie, the automatic optical inspection device 1 judges that the object to be tested has long scratches, holes, or irregular debris through the image 111 to be tested), the automatic optical inspection device 1 will determine the object to be tested The image belongs to the first preset flaw category, the second preset flaw category, or the third preset flaw category, and the post-processing device 2 will execute the machine learning classification step S2.

In practical applications, the processing device 2 may store N preset grayscale rules 21 corresponding to N preset defect categories. In the image classification step S12, the automatic optical inspection device 1 will first calculate a grayscale value of all pixels in at least one area of the image to be tested 111, and then, the automatic optical inspection device will determine all pixels in at least one area Whether the gray scale value of the pixel meets the N preset gray scale rule 22. If the automatic optical inspection device 1 determines that the grayscale values of all pixels in at least one area meet one of the preset grayscale rules 22, it is determined that the image to be tested 111 belongs to the preset defect category corresponding to the preset grayscale rule 22. Among them, depending on the actual object to be measured, the automatic optical detection device 1 may calculate the grayscale values of all pixels in the image to be measured, or only calculate the grayscale values of all pixels in a part of the area of the image to be measured.

Continuing the above example, assuming that the test object is a contact lens, the classification system 100 is to detect the central area A of the contact lens (that is, the area where the contact lens is actually used to correct the user’s vision) to determine whether the contact lens is in the central area Whether there are long strips of scratches, holes, and irregular debris in the scope of A; Of course, the types of defects are not limited to these three types, which can be changed according to requirements, for example, the defects can also be It includes straight line flaws, solid circle flaws, hollow circle flaws, etc. Assuming that the central area A of the contact lens does not have any of the above-mentioned defects, the average grayscale value of all pixels in the central area A of the image 111 to be tested is approximately 200, and the central area A of the contact lens has a striped scratch When a defect occurs, the grayscale value of all pixels corresponding to the defect is approximately between 120 and 150. When a hole appears in the central area A of the contact lens, the grayscale value of all pixels corresponding to the defect is approximately 70 ~80, and when irregular debris appears in the central area A of the contact lens, the grayscale values of all pixels corresponding to the defect are approximately between 20-50. The processing device 2 may have three preset grayscale rules, which are: Rule 1: At least 50-100 pixels in the central area A have grayscale values between 120-150; Rule 2: The part in the central area A The grayscale values of the pixels are between 70 and 80; Rule 3: The grayscale values of some pixels in the central area A are between 20 and 50. It should be noted that the above rules 1, rules 2, and 3 are designed by relevant personnel based on the types of defects that they actually want to classify. That is to say, in practical applications, the number of rules and the specific content of the rules can be based on the types of defects to be classified. Designed for the difference of the measured object, the type of defect, the appearance, etc.

In summary, when the classification system 100 executes the image classification step S12, the automatic optical inspection device 1 will calculate the grayscale value of all pixels in the central area A of the image to be tested 111, and determine the grayscale value of each pixel Whether it is between 120-150, 70-80, or 20-50, if the grayscale value of more than 50 pixels in the central area is between 120-150, the automatic optical inspection device 1 will determine that the image to be tested 111 belongs to the first A preset defect category; if the grayscale values of some pixels in the central area are between 70 and 80, the automatic optical inspection device 1 will determine that the image to be tested 111 belongs to the second preset defect category; if the central area is partially If the grayscale value of the pixel is between 20-50, the automatic optical inspection device 1 will determine that the image to be tested 111 belongs to the third preset defect category.

In different embodiments, the image to be measured may be an image processed by binarization, that is, after the image capture device 11 of the automatic optical inspection device 1 captures the image of the object to be measured, Binarize the image to generate the image to be tested. In the image to be tested, the flawed part will appear as white (grayscale value 255), and the rest will appear as black (grayscale value 0 ), or, the blemishes are black and the rest are white. As shown in Figures 4 and 5, they are respectively shown as schematic diagrams of two different images to be tested. In practical applications, the relevant personnel can make the automatic optical inspection device 1 classify the two images to be tested shown in FIGS. 4 and 5 into two different preset defects according to two different preset grayscale rules 21 Category. Specifically, one of the preset gray scale rules 21 of the automatic optical inspection device 1 may be: judging whether the length of the pattern formed by all pixels with a gray scale value of 255 in the image to be measured is greater than 100 pixels, and the length of the pattern is The aspect ratio is 100:1; another preset grayscale rule 21 can be: determine whether the area of the pattern formed by all pixels with a grayscale value of 255 in the image to be tested is greater than 500 pixels, and the aspect ratio of the pattern It is 1:1.

In addition, it is worth mentioning that after the image picker 11 of the automatic optical inspection device 1 captures the color image of the object to be tested, it can first separate the color image into three images according to the three primary colors (RGB), and only retain the red, Green and blue images, then the three images are binarized, and finally the preset grayscale value rules are used to determine whether there are patterns that meet the preset grayscale rules in the three images. If one of them If the image has a pattern that meets the preset gray-scale value rule, it is determined that the image belongs to the preset defect category corresponding to the preset gray-scale value rule.

In different embodiments, after the image picker 11 of the automatic optical inspection device 1 picks up the color image of the object to be tested, it may first take the three values of R, G, and B in the color image as the average value, the highest value, or After the lowest value, the color image is converted into a black-and-white image, and then binarization is performed to generate the image to be tested. Finally, the automatic optical inspection device 1 determines whether there is any image in the image to be tested. The pattern conforming to the preset grayscale value rule is used to determine which preset defect category or good product category the image to be tested belongs to.

Please refer to FIG. 1 and FIG. 6 together. FIG. 6 is a schematic flowchart of the machine learning classification step of the classification method of the present invention. In practical applications, the processing device 2 stores N preset grayscale rules 21 and N machine learning models 22 corresponding to the N preset defect categories, and the storage device 3 stores N training set ) 31, each training atlas 31 includes at least one good product image file 311 and a plurality of defective product training image files 312 corresponding to one of the N preset defect categories. The N machine learning models 22 correspond to the use of N training atlas 31 for model training, that is, the N machine learning models 22 respectively use a different training atlas 31 for model training. Wherein, in actual implementation, each training atlas 31 may include the same good product image files 311, but it is not limited to this.

After the classification system 100 has performed the automatic optical inspection device classification step S1, and the automatic optical inspection device 1 determines that the current image to be tested belongs to one of the N preset defect categories, the classification system 100 will execute the machine Learning classification step S2, the machine learning classification step S2 includes:

A loading step S21: load the image to be tested 111 into one of the corresponding machine learning models 22 according to the classification result of the image classification step S1;

A judging step S22: using the machine learning model 22 loaded with the image to be tested 111 to determine whether the image to be tested 111 is a good product or a defective product;

If the machine learning model 22 loaded with the image 111 to be tested is used to determine that the image 111 to be tested is a good product under review, the classification device 4 is used to transfer the object to be tested to the good product area; if a machine with the image 111 to be tested is used The learning model 22 determines that the image to be tested 111 is a defective product, and the classification device 4 is used to transfer the object to be tested to the defective product area. It is particularly noted that in a specific application, the classification system 100 may have N defective product areas, the N defective product areas corresponding to N preset defect categories, and when the machine learning model 22 determines that the image to be tested is defective At this time, the processing device 2 may control the sorting device 4 to transfer the object to be tested to the corresponding defective product area.

Please refer to FIG. 1, FIG. 7 and FIG. 8. FIG. 7 is a block diagram of one embodiment of the storage device of the classification system of the present invention, and FIG. 8 is an explanatory diagram of an actual example of the classification method of the present invention . Following the above example, in practical applications, the storage device 3 may store three machine learning models and three training atlases. The three machine learning models are respectively defined as a first machine learning model 22A, a second machine learning model 22B, and a third machine learning model 22C. The three training atlas 31 are respectively defined as a first training atlas 31A, a second training atlas 31B, and a third training atlas 31C. For example, the first training atlas 31A may store three defective training image files 312A1, 312A2, 312A3, and one of the defective training image files 312A1 has a strip-shaped scratch D11 at the bottom left of the central area A. And the length of the strip-shaped scratch D11 extends from upper left to lower right; one of the defective training image files 312A2 has a strip-shaped scratch D12 at the upper right of the central area A, and a strip-shaped scratch D12 The length direction of is extending from upper left to lower right; one of the defective product training files 312A3 has a long scratch D13 in the center of the central area A, and the length of the long scratch D13 is from upper right to Extends to the bottom left.

For example, the second training atlas 31B may store three defective training image files 312B1, 312B2, 312B3, and one of the defective training image files 312B1 has an irregular hole D21 in the lower left of the central area A; An irregular hole D22 appears in the upper right of the central area A of one of the defective product training files 312B2; an irregular hole D23 appears in the center of the central area A of one of the defective product training files 312B3; In addition, the irregular holes D21, D22, and D23 appearing in the three defective training image files 312B1, 312B2, and 312B3 have different shapes and sizes.

For example, the third training atlas 31C may store three defective training image files 312C1, 312C2, 312C3, and one of the defective training image files 312C1 has an irregular debris D31 at the lower left of the central area A; An irregular debris D32 appeared at the upper right of the central area A of one of the defective training image files 312C2; an irregular debris D33 appeared at the center of the central area A of one of the defective training image files 312C3; Moreover, the appearance and size of the irregular debris D31, D32, and D33 appearing in the three defective product training image files 312C1, 312C2, and 312C3 are not exactly the same.

Assuming that after the automatic optical inspection device 1 performs image capture of the object to be tested, the generated image to be tested 111 is as shown in FIG. 8, then in the above-mentioned automatic optical inspection device classification step S1, the to-be-tested image 111 will be determined to belong to The aforementioned first preset defect category, and in the aforementioned machine learning classification step S2, the processing device 2 will input the image to be tested 111 into the first machine learning model 22A, and then the processing device 2 will be able to use the first The machine learning model 22A determines that the image to be tested 111 has a defect of the first defect category, and the processing device 2 can determine that the image to be tested 111 is a defective product, and finally, the processing device 2 can control the classification The device 4 transfers the object to be tested to the defective product area.

As described above, since each defective product training image file 312 included in the first training image file set 31A only has defects of the first defect category, and each defective product training image collection 31 in the first training image file set 31A has only defects of the first defect category. It does not have defects of other types of defects. Therefore, when the image to be tested 111 that has never appeared in the first training image file set 31A as shown in FIG. 5 is input into the first machine learning model 22A, the first The machine learning model 22A can accurately determine that the image to be tested 111 in FIG. 5 has a defect belonging to the first defect category.

As explained above, on the contrary, assuming that the training atlas of a single machine learning model contains multiple types of images with different defect types, then the single machine learning model inputs an image that never appears in the training atlas At this time, the machine learning model will be prone to misjudge the type of the defect of the image file. In addition, in actual experiments, the applicant found that a single machine learning model is only trained for a single defect type. The training time required for the machine learning model to reach a predetermined judgment accuracy rate will be significantly lower than that of a single machine. The learning module trains multiple defect types at the same time, and the machine learning model judges the training time required for the various defect types to reach the same judgment accuracy rate.

In summary, the classification method and classification system of the present invention first use the automatic optical detection device 1 to capture images of the object to be tested to generate the image to be tested, and use the automatic optical detection device 1 to analyze the image to be tested to determine the object to be tested. Measure whether there is any defect in the N preset defect categories in the test image. If one of the N preset defect categories appears, then a machine learning model that only trains on specific types of defects is used. The image to be tested is judged again to confirm whether there are specific types of defects in the image to be tested. The classification method and classification system of the present invention can not only be accurate through the cooperation of the automatic optical detection device and the machine learning model. In addition to judging whether the object under test has defects, it can also accurately determine whether the object under test has defects of a preset type.

It should be noted that for manufacturers, it is a routine process to detect defects in their products, and it is more important for manufacturers to determine what defects are present in the object to be tested, so the manufacturer It is possible to adjust the production process according to the types of defects. Therefore, the classification method and classification system of the present invention are designed through automatic optical detection devices and N machine learning models, etc., not only can accurately determine whether the object to be tested has In addition to flaws, it is also possible to accurately determine which type of flaw belongs to.

Please refer to FIG. 1 and FIG. 9 together, which show a schematic flowchart of another embodiment of the machine learning classification step of the inventive classification method. The biggest difference between this embodiment and the previous embodiments is that in the determining step S22, the machine learning model 22 is used to determine whether the image to be tested is a good product, a defective product, or an unknown product.

If the machine learning model 22 determines that the image to be tested 111 is an unknown product, the classification device 4 is used to transfer the object to be tested to an unknown product area, and the processing device 2 will send out a prompt message 23 to remind the reviewers to The test objects in the unknown area are manually inspected. In practical applications, the prompt information 22 may be, for example, a control signal, and the related light emitting device and sound device of the classification method will emit a specific light beam and sound after receiving the control signal (prompt information 22); of course, the prompt information 22 can also be displayed on a display device.

As shown in FIG. 1, the classification system 100 may further include an input device 5 arranged adjacent to the unknown area, and the input device 5 is used to provide user operations to correspondingly generate a determination result information 51. If the test object set in the unknown area is manually inspected, and the input device 5 is operated to generate the determination result information 51, and the processing device 2 determines that the test object belongs to the N preset defect categories according to the determination result information 51 For one type, the processing device 2 stores the image 111 to be tested in the training atlas 31 of the machine learning model 22 corresponding to the corresponding preset defect category, and the processing device 2 will control the classification device 4 according to the determination result information. Move the object to be tested to the "good product area" or "defective product area". In practical applications, the classification system 100 can also be provided with a display device around the unknown area, the display device can display the image to be tested, and the user can view the corresponding object to be tested set in the unknown area through the display device Based on the image to be tested, determine which type of defect the image to be tested belongs to. Then, the user can generate the determination result information by operating the input device 5 (such as various touch screens, input buttons, etc.) 51.

Please refer to FIG. 10, which shows a schematic diagram of an actual example of the classification method of the present invention, which inherits the foregoing example of the classification method of the present invention applied to the classification of contact lenses. The defect is a "hole", but the automatic optical inspection device 1 erroneously determines that the image to be tested 111 has a "scratch", and the image to be tested 111 is correspondingly input to the first machine learning model 22A. At this time, the first machine learning model 22A A machine learning model 22A will determine the image to be tested 111 as "unknown". In specific implementation, each machine learning model can be a Sigmoid function using logistic regression to calculate the probability that the image to be tested is a good product and the probability that the image to be tested is a defective product. Assuming that the machine learning model calculates the probability of being tested The probability that the image is a good product and the probability that the image to be tested is a defective product does not exceed a predetermined threshold (for example, no more than 0.8), and the machine learning model will determine the image to be tested as an "unknown product."

As mentioned above, by adding the "unknown" judging mechanism to the machine learning model, the user will be able to understand more clearly what kind of misjudgment may occur when the automatic optical inspection device 1 faces the image to be tested. In this way, relevant personnel can correspondingly modify the relevant rules of the automatic optical inspection device 1 to determine the image to be tested.

Please refer to FIG. 11 and FIG. 12 together. FIG. 11 and FIG. 12 respectively show the image to be tested through the binarization process. As shown in Figure 11, in practical applications, if a discontinuous pattern similar to a linear defect appears in the image to be tested after the binarization process, it will be difficult for the automatic optical inspection device 1 to correctly determine the What kind of preset defect category does the flaw appearing in the image to be tested belong to? At this time, if the automatic optical inspection device 1 classifies the image to be tested in FIG. 11 into the preset flaw category of linear flaws, the image to be tested passes the phase After the judgment of the corresponding machine learning module, it can be correctly judged whether the image to be tested belongs to the preset defect category of linear defect. In addition, if the classification method in the determination step S22 is to use the machine learning model 22 to determine whether the image to be tested is a good product, a defective product, or an unknown product, the automatic optical inspection device 1 may also be a discontinuous similar The image to be tested with a pattern of linear flaws is classified as an unknown product.

As shown in FIG. 12, in practical applications, if the image to be tested has gray stripes (noise) in addition to linear flaws, it will be difficult for the automatic optical inspection device 1 to correctly determine the image to be tested. Which type of preset defect category does the defect appearing in in the optical system belong to? At this time, the automatic optical inspection device 1 either classifies the image to be tested as the preset defect category of linear defects, or classifies the image to be tested as The category of unknown products and the defects contained in the image to be tested can all be correctly judged in the subsequent process.

As mentioned above, the classification method and classification system of the present invention, even if discontinuous flaw images or interference patterns appear in the image to be tested, the subsequent machine learning model or the image to be tested can still be judged as unknown. And other steps, and correctly classify the object to be tested.

In the above-mentioned embodiments, the classification method of the present invention may further include a training set expansion step: using the processing device 2 to judge the image 111 to be tested as good or defective by the machine learning model 22, according to the image classification step The classification result is stored in the training atlas 31 corresponding to one of the corresponding machine learning models 22. Then, the processing device 2 can make each machine learning model use the expanded training atlas 31 to perform model training again when the image files included in the training atlas 31 increase to a predetermined number. In this way, each machine learning can be improved. The judgment accuracy of the model.

In a different embodiment, in the training set expansion step, the processing device 2 may also be the image to be tested 111 that is judged by the machine learning model 22 to be a good product or a defective product, and according to the classification result of the image classification step, Randomly stored in the training atlas 31 corresponding to one of the corresponding machine learning models 22 or in a validation dataset (Validation Dataset) corresponding to one of the corresponding machine learning models 22, and the validation atlas is stored in a centralized manner There are multiple images of good products and defective products that have been confirmed (for example, have been labeled). More specifically, during the model training process of each machine learning model, the corresponding training atlas is used for model training, and the corresponding verification atlas is used to verify the training results. By "randomly" storing the images to be tested that are judged as good or defective by the machine learning model into the corresponding training atlas or the design in the verification atlas, it will be possible to avoid adding too many waits to the training atlas. After the machine learning model is trained again, the problem of over-learning (Overfitting) occurs.

The N machine learning models in the above-mentioned classification method and classification system of the present invention can be trained by using a Convolutional Neural Network (CNN) model with N training atlases and N verification atlases. However, each machine learning model 21 is not limited to being generated using a convolutional neural network model. In actual implementation, the processing device 2 or the storage device 3 may be pre-stored with three convolutional neural network models of different depths. When performing model training on the N machine learning models 21, the processing device 2 may be based on The training atlas corresponding to the N preset defect categories to determine the difficulty of the defects in the N training images (for example, using the standard deviation of the defect area, the standard deviation of the gray scale value of the defect, the aspect ratio of the defect, etc.), and then The processing device 2 selects one of three convolutional neural network models of different depths as a basic model for generating N machine learning models 21 according to the judgment result.

If the processing device 2 selects one of the convolutional neural network models with relatively low depth to cooperate with N training atlases and N verification atlases to generate N machine learning models 21, and the accuracy of the predetermined number of machine learning models is exceeded (Accuracy), Precision (Precision), or Recall (Recall) does not reach the predetermined standard, the processing device 2 can switch to a relatively deep convolutional neural network model, and then re-coordinate with N training atlases and N verification atlases to regenerate new N machine learning models. In practical applications, the processing device 2 may repeatedly change the convolutional neural network model until N machine learning models 21 meeting the predetermined accuracy are generated.

In addition, the above-mentioned convolutional neural network models of different depths mean that the three convolutional neural network models have different numbers of convolution layers (Convolution Layer), pooling layer (Pooling Layer), and fully connected layer (Fully connection Layer), for example, a low-depth convolutional neural network model can be based on having a first convolutional layer, a first pooling layer, a second convolutional layer, a second pooling layer, and a fully connected layer. The deeper convolutional neural network model can be based on the first convolutional layer, the first pooling layer, the second convolutional layer, the second pooling layer, the third convolutional layer, the third pooling layer, and the first full The connection layer and the second fully connected layer.

It is worth mentioning that in different embodiments, between the image classification step S12 and the loading step S21, an image cutting step may also be included, and the image cutting step includes: using automatic optical inspection The device 1 or the processing device 2 divides the image to be tested to cut out a defect image from the image to be tested, and the image to be tested loaded in the loading step S21 is Describe the flawed image. For example, suppose that the automatic optical inspection device 1 performs image capture on the object to be tested and the image to be tested is 1000 pixels*1000 pixels, and then, if the automatic optical inspection device 1 determines that the image to be tested has a length of 100 pixels and If the defect corresponds to a defect with a 100:1 aspect ratio of the pattern, the automatic optical inspection device 1 cuts the image to be tested into a defect image of 200 pixels*200 pixels, and the defect image contains 100 pixels in length and The defect corresponds to the defect with the aspect ratio of the pattern of 100:1. Then, in the loading step S21, the defect image of 200 pixels*200 pixels is loaded into the corresponding machine learning model. As mentioned above, through the design of the image cutting step, the speed of image recognition by the machine learning model can be accelerated.

In addition, in an embodiment where the classification method includes the image cutting step and the training set expansion step, the image stored in the training atlas 31 corresponding to one of the machine learning models 22 is the defect image.

In summary, the classification method and classification system of the present invention are designed through automatic optical inspection device classification steps and machine learning classification steps, and make the images to be tested that are judged to belong to different preset defect categories, using corresponding machine learning models Designs such as reconfirmation can greatly improve the problem that the automatic optical inspection device is prone to misjudgment when the conventional automatic optical inspection device is used to classify the object to be tested.

The above descriptions are only the preferred and feasible embodiments of the present invention, which do not limit the scope of the present invention. Therefore, all equivalent technical changes made by using the description and drawings of the present invention are included in the protection scope of the present invention. .

100: Classification system 1: Automatic optical inspection device 11: Image grabber 111: Image to be tested 2: Processing device 21: Preset grayscale rules 22, 22A, 22B, 22C: machine learning model 23: Information 3: storage device 31: Training Atlas 311: Good product image file 312: Bad product training image file 4: Sorting device 5: Input device 51: Judgment result information

Figure 1 is a block diagram of the classification system of the present invention.

2 is a schematic flow chart of the classification steps of the automatic optical inspection device of the classification method of the present invention.

Fig. 3 is an explanatory diagram of an actual example of the classification method of the present invention.

FIG. 4 is a schematic diagram of an actual example of the image to be measured after the binarization process.

FIG. 5 is a schematic diagram of one embodiment of the image to be measured through binarization processing.

FIG. 6 is a schematic flowchart of the machine learning classification step of the classification method of the present invention.

FIG. 7 is a block diagram of one embodiment of the storage device of the classification system of the present invention.

Fig. 8 is an explanatory diagram of an actual example of the classification method of the present invention.

FIG. 9 is a schematic flowchart of another embodiment of the machine learning classification step of the inventive classification method.

Fig. 10 is an explanatory diagram of an actual example of the classification method of the present invention.

FIG. 11 is a schematic diagram of one embodiment of the image to be measured through binarization processing.

FIG. 12 is a schematic diagram of one embodiment of the image to be measured.

100: Classification system

1: Automatic optical inspection device

11: Image grabber

111: Image to be tested

2: Processing device

21: Preset grayscale rules

22: Machine learning model

23: Information

3: storage device

31: Training Atlas

311: Good product image file

312: Bad product training image file

4: Sorting device

5: Input device

51: Judgment result information

Claims (10)

A classification method, which is suitable for a classification system, the classification system includes an automatic optical detection device, a processing device and a classification device, the classification system can execute the classification method to classify a test object into a good product Area or a defective product area, the classification method includes: an automatic optical inspection device classification step, which includes: an image capturing step: using the automatic optical inspection device to perform an image capture of the object under test, To generate an image to be tested; an image classification step: use the automatic optical inspection device or the processing device to analyze the image to be tested to determine that the image to be tested belongs to a good product category and N preset defect categories Which type of; where N is a positive integer greater than 1; if it is determined that the image to be tested belongs to the good product category, the classification device is used to transfer the object to be tested to the good product area; If the image to be tested belongs to one of the N preset defect categories, a machine learning classification step is performed; the processing device stores N machine learning models corresponding to the N preset defect categories, and N Each of the machine learning models uses a training set (training set) that is not exactly the same for model training, and each of the training images includes a plurality of types corresponding to one of the N preset defect categories. Training image files and at least one good image file; the machine learning classification step includes: a loading step: according to the classification result of the image classification step, the image to be tested is loaded to one of the corresponding machines In the learning model; A determination step: using the machine learning model loaded with the image to be tested to determine whether the image to be tested is a good product or a defective product; if the machine loaded with the image to be tested is used If the learning model determines that the image to be tested is the good product for review, the classification device is used to transfer the object to be tested to the good product area; if the machine learning loaded with the image to be tested is used The model determines that the image to be tested is the defective product, and then uses the classification device to transfer the object to be tested to the defective product area. The classification method according to claim 1, wherein the classification system further includes a storage device storing a plurality of the training atlases, and the classification method further includes a training set expansion step: The machine learning model determines that the image to be tested is the good product or the defective product. According to the classification result of the image classification step, it is stored in the corresponding one of the machine learning models. Described in the training atlas. The classification method according to claim 1, wherein the classification system further includes a storage device storing a plurality of the training atlases, and the classification method further includes a training set expansion step: According to the classification result of the image classification step, the machine learning model determines that the image to be tested is the defective product, and randomly stores the corresponding training atlas corresponding to one of the machine learning models Or a verification atlas corresponding to one of the corresponding machine learning models; each of the verification atlases includes a plurality of verification images used by each of the machine learning models for model verification. The classification method according to claim 1, wherein the classification system can use the classification method to classify the test object into the good product area, the defective product area or an unknown product area, and the classification system Includes an input device, the The input device is arranged adjacent to the unknown product area; in the determining step, the machine learning model is used to determine that the image to be tested is the good product, the defective product, or an unknown product; The machine learning model determines that the image to be tested is the unknown product, and then uses the classification device to transfer the object to be tested to the unknown product area, and sends out a prompt message to remind reviewers to The object to be tested in the unknown area is manually inspected; if the object to be tested placed in the unknown area is manually inspected and the input device is operated to generate a determination result information, the processing The device determines that the object to be tested belongs to one of the N preset defect categories according to the determination result information, and the processing device stores the image to be tested in the corresponding preset defect category The training atlas of the machine learning model of, and the processing device will control the classification device to transfer the object to be tested to the defective product area according to the determination result information. The classification method according to claim 1, wherein the processing device stores N preset grayscale rules corresponding to the N preset defect categories; in the image classification step, the automatic optical detection The device will first calculate a grayscale value of all pixels in at least one area in the image to be measured, and then, the automatic optical detection device will determine whether the grayscale value of all pixels in at least one area is Meets N preset gray-scale rules; if the automatic optical detection device determines that the gray-scale values of all pixels in at least one of the regions meet one of the preset gray-scale rules, then determine the to-be-tested The image belongs to the preset defect category corresponding to the preset grayscale rule. A classification system, comprising: an automatic optical inspection device, a processing device, and a classification device. The classification system can execute the classification method described in claim 1. The classification system according to claim 6, wherein the classification system further includes a storage device storing a plurality of the training atlases, and the classification method further includes a training set expansion step: The machine learning model judges the image to be tested as the good product or the defective product, and according to the classification result of the image classification step, it is stored in the training corresponding to one of the machine learning models. Atlas. The classification system according to claim 6, wherein the classification system further includes a storage device storing a plurality of the training atlases, and the classification method further includes a training set expansion step: According to the classification result of the image classification step, the machine learning model determines that the image to be tested is the defective product, and randomly stores the corresponding training atlas corresponding to one of the machine learning models Or a verification atlas corresponding to one of the corresponding machine learning models; each of the verification atlases includes a plurality of verification images used by each of the machine learning models for model verification. The classification system according to claim 6, wherein the classification system can use the classification method to classify the test object into the good product area, the defective product area or an unknown product area, and the classification system An input device is included, and the input device is arranged adjacent to the unknown product area; in the determining step, the machine learning model is used to determine that the image to be tested is the good product, the defective product, or An unknown product; if the machine learning model determines that the image to be tested is the unknown product, the classification device is used to transfer the object to be tested to the unknown product area, and a prompt message is issued to Remind the judges to perform manual inspection of the test object set in the unknown area; if the test object set in the unknown area is manually inspected and the input device is operated to produce a judgment Result information, and the processing device determines that the object under test belongs to the N number of predictions based on the determination result information. Set one of the defect categories, the processing device stores the image to be tested in the training atlas of the machine learning model corresponding to the preset defect category, and the processing device will According to the determination result information, the classification device is controlled to transfer the object to be tested to the defective product area. The classification system according to claim 6, wherein the processing device stores N preset grayscale rules corresponding to the N preset defect categories; in the image classification step, the automatic optical detection The device will first calculate a grayscale value of all pixels in at least one area in the image to be measured, and then, the automatic optical detection device will determine whether the grayscale value of all pixels in at least one area is Meets N preset gray-scale rules; if the automatic optical detection device determines that the gray-scale values of all pixels in at least one of the regions meet one of the preset gray-scale rules, then determine the to-be-tested The image belongs to the preset defect category corresponding to the preset grayscale rule.

Images