TW202117664A - Optical inspection secondary image classification method which can effectively improve the accuracy of image recognition and classification - Google Patents

Abstract

An optical inspection secondary image classification method of the present invention is used in the image classification determination of good products or defective products provided by a processing line equipped with an automatic optical inspection system, mainly, after an image classification model is generated and verified, the predicted and classified results of input images are recorded. then the number of images that cannot be accurately predicted and classified are increased, labeled, and then re-entered into a neural network architecture of the image classification model for a reinforced training and then another image classification model is generated, through at least one reinforced training, the prediction and classification accuracy of the image classification model is improved, the recognition accuracy and efficiency of a detection device are improved by relatively more active and reliable means, and the cost of secondary screening is therefore reduced.

Description

Optical inspection secondary image classification method

The present invention is related to the deep learning image classification technology, and particularly refers to an optical detection secondary image classification method that is particularly suitable for being applied to the product after the automatic optical detection is passed, and then the product is judged as a good product or a defective product.

Automatic Optical Inspection (AOI) is a non-contact method that uses machine vision technology to capture images for analysis, and then determine whether the product (finished or semi-finished) is defective, and can be deployed in the middle of the node of the automated processing line Testing, without affecting production capacity, is a testing method widely used in the industry, and it is a necessary investment with a high proportion in the manufacturing process of the circuit board and display panel industry.

Taking the most widely used circuit board processing production line as an example, the AOI inspection system process mainly uses optical instruments to scan the circuit board to be inspected to obtain images, and then the system compares the qualified parameters in the database and passes the computer image The processing technology detects whether there are defects on the circuit board.

Usually the circuit board processing production line has extremely high yield requirements, so the AOI parameters are set very strictly, and the optical principle is easy to make the AOI sensitive to light and shadow interference. Therefore, as long as there is some slight external light and shadow influence, the equipment is It will be automatically judged as defective, which causes AOI inspection to often face the phenomenon of over-screening (misjudgment of good products as defective).

It is known that image data can be processed through artificial intelligence and deep learning neural networks, and an algorithm that uses rules to predict unknown data can identify unknown defects and assist the subsequent optimization of AOI detection, thereby improving the accuracy of detection equipment Rate, reduce the cost of manual second screening.

The reason why the image classification model based on the neural network architecture can fully and correctly learn the key to distinguishing images, in addition to the need to provide a large amount of labeled (Label) image data in the training process of deep learning (Training) In addition, improved neural network architecture may be a feasible approach.

However, the parameter options that can be adjusted in the neural network architecture are very cumbersome, and it is not easy to systematically find the best permutation and combination in the cumbersome parameter options; therefore, how to improve the image classification model under the condition of a limited number of samples The accuracy of the discrimination has long been a problem that the industry and academia desperately want to solve.

In view of this, the present invention is to provide a secondary image classification method for optical detection that can effectively improve the accuracy of image classification for automatic optical detection, as its main purpose.

The optical inspection secondary image classification method of the present invention is used in the image classification judgment of a good product or defective product provided by a processing production line equipped with an automatic optical inspection system; the optical inspection secondary image classification method , It includes the following steps: (a) Provide an image feature automatic identification system step, an image feature automatic identification system integrates a storage unit, a processing unit and an image capture unit; (b) Create a training data set Step, at least a training data set is established in the storage unit of the image feature automatic identification system, the training data set includes at least a plurality of good images labeled as label 0, a plurality of overscreened images labeled as label 0, A plurality of defective product images marked as label 1; (c) a step of establishing a verification data set, a verification data set is created in the storage unit of the image feature automatic identification system, and the verification data set includes at least a plurality of good product images, Multiple defective images; (d) a step of establishing a neural network-like architecture, in the storage unit of the image feature automatic identification system, creating an operation executed by the processing unit for performing good or defective images on the input image A type of neural network architecture for product prediction classification; (e) Image classification model training step, through the image capture unit in the training data set, the plural number is labeled as the good product image with label 0, and the plural number is labeled as the excess of label 0 Screened images and multiple defective images marked with label 1 are input to this type of neural network architecture one by one, and after comparing the prediction and classification output results of this type of neural network architecture, adjust the features and weights of this type of neural network architecture. And save the adjustment to reduce the error successfully, and then produce an image classification model that is better at predicting and classifying the input image with good or defective products; (f) the image classification model verification step, each time the image classification is generated When modeling, through the image capture unit, the multiple good images and multiple defective images in the verification data set are input into the image classification model generated this time one by one, and the image classification model generated this time compares the input Perform predictive classification for each good image and each defective image to evaluate the accuracy of the prediction and classification of the image classification model generated this time; (g) Statistical classification prediction error distribution and enhanced training steps, each time an image is generated Classification model, and when the verification of the image classification model generated this time is completed, record the input predicted classification results of each of the good images and each of the defective images, and calculate the predicted classification error distribution, and make each actual Images of good quality images that cannot be accurately predicted and classified are marked as 0 after number amplification, and the images that are actually defective images that cannot be accurately predicted and classified are marked as 1 after number amplification. Through the image capture unit, the amplified images marked as 0 and the amplified images marked as 1 are input again into the neural network that the generated image classification model belongs to. Enhance the training of the framework to generate another image classification model; and (h) the step of confirming the model, after performing step (f) to step (g) at least once, confirm at least one of the image classification models Image classification model, namely The at least one confirmed image classification model can be used to classify images of good or defective products provided by the processing line.

According to the above technical features, in the optical detection secondary image classification method, in the confirming model step, after repeating the step f to the step g at least once, confirming a better accuracy among the image classification models Use the confirmed image classification model to classify the products provided by the processing line as good or defective images.

According to the above technical features, in the optical detection secondary image classification method, in the confirming model step, after repeating the step f to the step g at least once, it is confirmed that the plural accuracy is better among the image classification models And merge the confirmed image classification models, and use the merged image classification models to classify the products provided by the processing line as good or defective images.

According to the above technical features, the number of good product images in the training data set and the verification data set is determined by the automatic optical detection system as good products and the original images of good products that are manually confirmed to be good are amplified by amplifying the original images of the training data set; the training data set The number of over-screened images is achieved by amplifying multiple original images of good products that are judged as defective by the automatic optical inspection system but confirmed as good products by manual; the number of defective images in the training data set and the verification data set It is achieved by amplifying a plurality of original images of defective products that are identified as defective by the automatic optical inspection system and manually confirmed as defective.

According to the above technical features, each of the good product images in the verification data set is not exactly the same as each of the good product images in the training data set, and each of the defective product images in the verification data set is the same as the training data The images of the defective products in the set are not exactly the same.

According to the above technical features, the number of good product images in the training data set and the verification data set is determined by the automatic optical detection system as good products and the original images of good products that are manually confirmed to be good are amplified by amplifying the original images of the training data set; the training data set The number of over-screened images is achieved by amplifying multiple original images of good products that are judged as defective by the automatic optical inspection system but confirmed as good products by manual; the number of defective images in the training data set and the verification data set It is achieved by amplifying the original images of defective products identified as defective by the automatic optical inspection system and manually confirmed as defective; and, each of the good product images in the verification data set and the training data set The images of the good products are not exactly the same, and the images of the defective products in the verification data set are not exactly the same as the images of the defective products in the training data set.

According to the above-mentioned technical characteristics, the predicted output value of each of the neural network architectures of each of the image classification models for each input image is presented in the output type of 0 or 1.

According to the above technical characteristics, the predicted output value of each of the neural network architectures of each of the image classification models for each input image is presented in the output type with the probability of prediction being 0 and the probability of prediction being 1.

According to the above technical features, the optical detection secondary image classification method, in the step (d), is to create a storage unit in the image feature automatic identification system for performing good or defective images on the input image Convolutional neural network architecture (Convolutional neural network, CNN) for predictive classification.

The optical detection secondary image classification method disclosed in the present invention mainly records the input image prediction classification results after the image classification model is generated and verified, and the number of images that cannot be accurately predicted and classified is increased, After labeling, input the neural network architecture to which the image classification model belongs to strengthen the training, and then generate another image classification model. Through at least one strengthened training, the predictive classification accuracy of the image classification model is improved, which is especially suitable for application in products. After the automatic optical inspection, the product is judged as good or defective again, using relatively more active and reliable means to improve the accuracy and efficiency of the detection equipment, and reduce the cost of secondary screening.

The present invention mainly provides automatic optical inspection, an optical inspection secondary image classification method that can effectively improve the accuracy of image classification. The optical inspection secondary image classification method of the present invention is used in a processing with an automatic optical inspection system The image classification of the products provided by the production line is good or defective; as shown in Figures 1 to 3, the optical inspection secondary image classification method includes the following steps:

(a) Provide an image feature automatic identification system step. An image feature automatic identification system 10 integrates a storage unit 11, a processing unit 12, and an image capture unit 13; in implementation, the processing unit 12 is It may be a graphics processor (Graphics Processing Unit, GPU).

(b) The step of creating a training data set is to create at least a training data set 20 in the storage unit 11 of the image feature automatic identification system 10, and the training data set 20 includes at least a plurality of good images with a label of 0 and a plurality of The over-screened image marked as label 0, and the defective product image marked as label 1 in multiple numbers.

(c) The step of establishing a verification data set is to create a verification data set 30 in the storage unit 11 of the image feature automatic identification system 10, and the verification data set 30 includes at least a plurality of good product images and a plurality of defective product images.

(d) The step of establishing a neural network-like architecture, in the storage unit 11 of the image feature automatic identification system 10, establishes an operation executed by the processing unit 12 for predicting and classifying the input image with good or defective products A type of neural network architecture 40; in implementation, a convolutional neural network can be established in the storage unit 11 of the image feature automatic recognition system 10 for predicting and classifying the input image for good or defective products Road architecture (Convolutional neural network, CNN).

(e) Image classification model training step, through the image capturing unit 13 in the training data set 20 in the training data set 20 that the plural number is labeled as the good image of the label 0, the plural number is labeled as the over-screened image of the label 0, and the plural number is labeled as The defective product images of label 1 are input into this type of neural network architecture 40 one by one, and after comparing the prediction of the classification output result of this type of neural network architecture 40, the features and weights of this type of neural network architecture are adjusted, and the results of successfully reducing errors are stored The adjustment starts to produce an image classification model that is better at predicting and categorizing the input image as good or defective.

(f) The image classification model verification step. Each time an image classification model is generated, the multiple good product images and the multiple defective product images in the verification data set 30 are inputted one by one through the image capture unit 13 The generated image classification model is used to predict and classify each input good product image and each defective product image by the generated image classification model, so as to evaluate the prediction classification accuracy of the image classification model generated this time.

(g) Statistical classification prediction error distribution and enhanced training steps, each time an image classification model is generated, and when the image classification model generated this time is verified, each input good image and each defective product are recorded The predicted classification result of the image, and the statistical prediction of the classification error distribution, the number of each image that is actually a good image but cannot be accurately predicted and classified is increased and then marked as 0, and each image that is actually a defective image is marked as 0. After the number of images that cannot be accurately predicted and classified is amplified, the images are marked as 1, and the images that are amplified and marked as 0 are completed one by one through the image capturing unit 13 and the amplified and marked as The image of 1 is input again into the neural network architecture of the image classification model generated this time to strengthen the training, and then another image classification model is generated.

(h) The step of confirming the model, after performing steps (f) to (g) at least once, confirming at least one image classification model among the image classification models, and then using the at least one confirmed image Image classification model to classify images of good or defective products provided by the processing line

In the optical detection secondary image classification method of the present invention, in the confirming model step, after repeating the steps (f) to (g) at least once, an accuracy can be confirmed among the image classification models A better image classification model uses the confirmed image classification model to classify images of good or defective products provided by the processing line.

In the optical detection secondary image classification method of the present invention, in the confirming model step, after repeating the step (f) to the step (g) at least once, the plural accuracy of the image classification models can be confirmed A better image classification model, and combine the confirmed image classification models, and use the combined image classification models to classify the products provided by the processing line for good or defective images Discriminate.

Since the optical detection secondary image classification method of the present invention records the input image prediction classification results after the image classification model is generated and verified, the number of images that cannot be accurately predicted and classified is increased and labeled After that, input the neural network architecture to which the image classification model belongs again for enhanced training, and then generate another image classification model. Therefore, through at least one enhanced training, the predictive classification accuracy of the image classification model can be improved, which is especially suitable for application in After the product passes the automatic optical inspection, the product is again judged as good or defective, and the detection accuracy and efficiency of the inspection equipment are improved by relatively more active and reliable means, and the cost of secondary screening is reduced.

As shown in Figures 1 and 4, the optical detection secondary image classification method of the present invention reveals the number of good images in the training data set 20 and the verification data set 30 in various possible implementation modes. It can be achieved by amplifying the original images of good products that are judged as good by the automatic optical inspection system and manually confirmed as good; the number of over-screened images in the training data set 20 can be obtained by the automatic optical inspection system It is judged as a defective product but achieved by augmenting the original image of a good product that is manually confirmed as a good product; the number of defective product images in the training data set 20 and the verification data set 30 is determined by the automatic optical inspection system as a defective product and Achieved by augmenting the original image of the defective product that was manually confirmed as a defective product.

And, in the optical detection secondary image classification method of the present invention, each of the good image in the verification data set 30 and each of the good image in the training data set 20 in the various possible implementation modes disclosed above The images are not exactly the same, each of the defective product images in the verification data set 30 and each of the defective product images in the training data set 20 are not completely the same.

Of course, the optical detection secondary image classification method of the present invention, in the various possible implementation modes, and the number of good images in the training data set 20 and the verification data set 30 can be determined by the automatic The optical inspection system judges the product as good and is achieved by augmenting the original image of the good product that is manually confirmed as good; the number of over-screened images in the training data set 20 can be determined by the automatic optical inspection system as a defective product by multiple numbers, but through manual The original image of the good product confirmed as good is amplified; the number of defective product images in the training data set 20 and the verification data set 30 is determined by the automatic optical inspection system as defective and manually confirmed as defective The original image of the defective product is amplified; and, each of the good product images in the verification data set 30 is not exactly the same as each of the good product images in the training data set 20, and each of the good product images in the verification data set 30 It is preferable that the defective product image is not exactly the same as each of the defective product images in the training data set 20.

Specifically, in the optical detection secondary image classification method of the present invention, in various possible implementation modes, the prediction output of each type of neural network architecture 40 of each image classification model for each input image The value can be selected to be presented as an output type of 0 or 1. Of course, the predicted output value of each of the neural network architectures 40 of each image classification model for each input image can also be selected to be 0. The probability and the prediction of the probability of 1 are presented in the output form.

The above-mentioned embodiments are only to illustrate the technical ideas and features of the present invention, and their purpose is to enable those who are familiar with the art to understand the content of the present invention and implement them accordingly. When they cannot be used to limit the patent scope of the present invention, That is, all equal changes or modifications made in accordance with the spirit of the present invention should still be covered by the patent scope of the present invention.

100: Optical inspection secondary image classification method 10: Image feature automatic identification system 11: storage unit 12: Processing unit 13: Image capture unit 20: Training data set 30: Validation data set 40: Neural Network Architecture

Figure 1 is a block diagram of the basic structure of the image feature automatic recognition system in the present invention. Figure 2 is the basic flow chart of the optical detection secondary image classification method of the present invention. Figure 3 is a schematic diagram of a possible implementation of the image classification model training process in the present invention. Figure 4 is a flow chart of the original image confirmation for one possible implementation of the present invention.

100: Optical inspection secondary image classification method

Claims (9)

An optical inspection secondary image classification method, which is used in the image classification judgment of a good product or defective product provided by a processing production line equipped with an automatic optical inspection system; the optical inspection secondary image classification method is It includes the following steps: (a) Provide an image feature automatic identification system step. An image feature automatic identification system (10) integrates a storage unit (11), a processing unit (12), and an image capture unit (13); (b) The step of creating a training data set, at least creating a training data set (20) in the storage unit (11) of the image feature automatic recognition system (10), the training data set (20) at least including plural marks Is the good product image with label 0, the over-screening image with the plural label as label 0, and the defective product image with the plural label as label 1; (c) The step of establishing a verification data set is to create a verification data set (30) in the storage unit (11) of the image feature automatic identification system (10), and the verification data set (30) includes at least a plurality of good product images Image, plural defective images; (d) The step of establishing a neural network-like architecture, in the storage unit (11) of the image feature automatic recognition system (10), establishes an operation executed by the processing unit (12) for the input image A neural network architecture for predicting and categorizing good or defective products (40); (e) Image classification model training step, through the image capture unit (13) in the training data set (20), the multiple-labeled good image with label 0, and the multiple-labeled over-screened image with label 0 , The defect images marked as label 1 are input into the neural network architecture (40) one by one, and the neural network architecture (40) is adjusted after comparing the prediction and classification output results of the neural network architecture (40) The features and weights of the images, and the adjustments that successfully reduce the error are stored, and then an image classification model that is better at predicting and classifying the input images for good or defective products will be produced; (f) The image classification model verification step, each time an image classification model is generated, the multiple good product images and multiple defective product images in the verification data set (30) are passed through the image capture unit (13) Input the image classification model produced this time one by one, and the image classification model produced this time will predict and classify each input good image and each defective image, so as to evaluate the prediction of the image classification model produced this time Classification accuracy (g) Statistical classification prediction error distribution and enhanced training steps, each time an image classification model is generated, and when the image classification model generated this time is verified, each input good image and each defective product are recorded The predicted classification result of the image, and the statistical prediction of the classification error distribution, the number of each image that is actually a good image but cannot be accurately predicted and classified is increased and then marked as 0, and each image that is actually a defective image is marked as 0. After the number of images that cannot be accurately predicted and classified is amplified, the images are marked as 1, and the images that have been amplified and marked as 0 are amplified one by one through the image capturing unit (13) and the images that have been amplified and The image marked 1 is re-input to the neural network architecture (40) to which the image classification model generated this time belongs to for enhanced training, and then another image classification model is generated; and (h) The step of confirming the model, after performing steps (f) to (g) at least once, confirming at least one image classification model among the image classification models, and then using the at least one confirmed image The image classification model classifies the products provided by the processing line as good or defective images. The optical detection secondary image classification method according to claim 1, wherein, in the optical detection secondary image classification method, in the step (h), the step (f) to the step ( g) Afterwards, confirm an image classification model with better accuracy among the image classification models, and use the confirmed image classification model to perform good or defective images of the products provided by the processing line Classification discrimination. The optical detection secondary image classification method according to claim 1, wherein, in the optical detection secondary image classification method, in the step (h), the step (f) to the step ( g) After that, among the image classification models, the image classification model with better plural accuracy is confirmed, and the confirmed image classification models are merged, and the merged image classification models are used for the The products provided by the processing line are classified into good or defective products by image classification. The optical detection secondary image classification method according to any one of Claims 1 to 3, wherein the number of good quality images in the training data set (20) and the verification data set (30) is determined by the automatic The optical inspection system judges it to be good and is achieved through the amplification of the original image of the good product that is manually confirmed as good; the number of over-screened images in the training data set (20) is determined by the automatic optical inspection system to be defective but through Amplification of the original image of the good product that was manually confirmed as a good product is achieved; the number of defective product images in the training data set (20) and the verification data set (30) is determined by the automatic optical detection system as defective products by a plurality of The original image of the defective product confirmed as a defective product is amplified. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein each of the good image in the verification data set (30) and each of the training data set (20) The good product images are not completely the same, and each of the defective product images in the verification data set (30) is not completely the same as each of the defective product images in the training data set (20). The optical detection secondary image classification method according to any one of Claims 1 to 3, wherein the number of good quality images in the training data set (20) and the verification data set (30) is determined by the automatic The optical inspection system judges it to be good and is achieved through the amplification of the original image of the good product that is manually confirmed as good; the number of over-screened images in the training data set (20) is determined by the automatic optical inspection system to be defective but through Amplification of the original image of the good product that was manually confirmed as a good product is achieved; the number of defective product images in the training data set (20) and the verification data set (30) is determined by the automatic optical detection system as defective products by a plurality of The original image of the defective product confirmed as a defective product is amplified; and, each of the good product images in the verification data set (30) is not exactly the same as each of the good product images in the training data set (20), Each of the defective product images in the verification data set (30) is not exactly the same as each of the defective product images in the training data set (20). The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the predicted output value of each of the neural network architectures (40) of each of the image classification models for each input image , Is presented in the output form of 0 or 1. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the predicted output value of each of the neural network architectures (40) of each of the image classification models for each input image , Is presented in the output form of the probability that the prediction is 0 and the probability that the prediction is 1. The optical detection secondary image classification method according to any one of claims 1 to 3, wherein the optical detection secondary image classification method in the step (d) is based on the automatic identification of the image feature The storage unit (11) of the system (10) establishes a convolutional neural network (CNN) architecture for predicting and classifying the input image with good or defective products.

Images