JP7074460B2 - Image inspection equipment and methods - Google Patents

Description

The present invention relates to an image inspection apparatus and method, and more particularly to an inspection technique using an image to be inspected.

Conventionally, as an inspection method for finished products, semi-finished products, products and parts in the processing process, there is a method called image inspection using images. In this method, an image to be inspected is acquired by an image pickup device, and the presence or absence of defects and defects, the type, and the like are determined based on the acquired image. For example, Patent Document 1 discloses a technique for determining the quality of the appearance of an inspection target.

There are a wide variety of image inspection methods for inspecting the presence or absence of defects and defects in inspection targets such as products, but the general operation of a conventional image inspection device is as shown in the flowchart of FIG. .. First, the image inspection device acquires an image to be inspected (step S800). Next, the image inspection device performs preprocessing of the acquired image and processes the image into a state in which it is easy to inspect (step S801).

Subsequently, the image inspection apparatus calculates a feature amount that quantifies the feature of the image from the image after the preprocessing (step S802). Regarding the calculated feature amount of the image, there may be one case or a plurality of feature amounts depending on the specific method adopted in the image inspection apparatus. Then, the image inspection device determines whether the inspection target is a non-defective product or a defective product based on the value of the feature amount of the image (step S803).

There are various feature quantities that quantify the features of an image, but in conventional image inspection devices, it was common for the designer of the image inspection device to design and select based on his or her own knowledge. .. In addition, as for the method of determining whether the inspection target is a good product or a defective product, various methods such as a judgment method based on a threshold value determined by the designer, a statistical test method, and a machine learning method are used. Is used.

By the way, in recent years, a method has been proposed in which an image inspection is performed by learning the feature amount of an image to be inspected by deep learning using a convolutional neural network. In such a method, the inspection is performed using the neural network 900 having the configuration shown in FIG. In this method, a plurality of elements called convolution layers 901a, 901b, and 901c that perform operations centered on the convolution operation are connected in series, and a characteristic pattern is extracted from the input image to be inspected, and the characteristics of the image are obtained. Calculate the amount. Then, the portion called the fully connected layer 902 following the convolution layers 901a, 901b, and 901c determines the quality of the inspection target based on the calculated feature amount of the image.

For example, Patent Document 2 targets medical images, but classifies inspection targets using a neural network having a convolution layer and a fully connected layer for a two-dimensional image generated from a three-dimensional image to be inspected. The technology is disclosed.

The feature of the image inspection method based on such a neural network is that the feature amount of the image is automatically acquired by learning. As described above, in the technique using the conventional image inspection method as described in Patent Document 1, the feature amount of the image is generally designed by the designer based on his / her own knowledge. The feature amount greatly affects the performance of image inspection, but the appropriate feature amount differs depending on the inspection target. Therefore, it was considered difficult to design features unless they were experts in image processing and image inspection.

On the other hand, in the image inspection method based on the neural network as described in Patent Document 2, if a large number of images to be inspected for non-defective products and defective products are prepared, the images more suitable for inspection can be obtained by learning them. A feature amount can be obtained. Therefore, it was possible to save a lot of labor in the design of features. It is a well-known fact that features are automatically acquired by deep learning using a convolutional neural network, as disclosed in Non-Patent Document 1, for example.

In the method described in Patent Document 1, it is necessary to make a neural network learn the distinction between a non-defective product and a defective product before performing an image inspection so that the quality of the inspection target can be determined with a sufficiently high correct answer rate. .. This learning is called "supervised learning" and requires an image to be inspected in which good products and defective products are distinguished in advance. On the other hand, image learning by deep learning requires a large amount of images for learning. Further, it is desirable that the number of good images and defective images to be inspected used for learning is close to the same number. This means that not only many good images but also many defective images are needed.

However, it is often difficult to prepare a large number of defective images as learning images. At the production site, production equipment and processes are usually designed so as not to produce defective products as much as possible, and it is not uncommon for the defective product rate to be low. However, most of the inspection images collected from such sites are good images, and it is often difficult to successfully learn the neural network as shown in FIG.

The reason is that, in general, supervised learning of a neural network proceeds when the output of the neural network is different from that of the teacher, that is, when the output is incorrect. At the beginning, there are many erroneous judgments, but as a result, learning progresses, erroneous judgments gradually decrease, and at the end, good or bad can be judged correctly, which is the normal learning process.

However, if there are only good images or if there are very few defective images, the correct answer rate will be high even if the neural network always outputs good images. Therefore, it becomes difficult to learn the distinction between good and bad that you really want to learn.

For this reason, it may be difficult to apply a conventional image inspection device using deep learning by a convolutional neural network when there are no defective products or there are few defective products. And this means that when a new production facility or equipment is started up, the inspection cannot be performed until a considerable number of defective images are accumulated. Therefore, the image inspection device using the deep learning by the conventional convolutional neural network has an advantage that the feature amount of the image can be automatically learned, but the scenes in which it can be exhibited are limited.

Japanese Unexamined Patent Publication No. 2017-174039 Japanese Unexamined Patent Publication No. 2016-999707 Japanese Unexamined Patent Publication No. 2008-310700

Yutaka Matsuo, "Is Artificial Intelligence Beyond Humans? Beyond Deep Learning," Kakugawa EPUB Selection, published in March 2015 Alec Radford et al. : Unsupervised Representation Learning with Deep Convolutional Generic Advanced Network, ICLR 2016 (arXiv: 1511.06434) Diederik P.M. Kingma et al. : Auto-Encoding Variational Bayes (arXiv: 1312.6114) Just Tobias Springenberg et al. : Driving for Syntax: The All Convolutional Net (arXiv: 1412.6806) Liu, Fei Tony and Ting, Kai Ming and Zhou, Zhi-Hua: Isolation forest, Data Mining, 2008. ICMP'08. Eighth IEEE International Convention on, IEEE, pp. 413-422 (2008) Watanabe, Satosi and Pakvasa, Nikhil: Subspace method of pattern recognition, Proc. 1st. IJCPR, pp. 25-32 (1973) Breunig, Markus Mand Kriegel, Hans-Peter and Ng, Raymond T and Sander, Jorg: LOF: identityfying density-based local outliers, ACM sigmod. 29-2, ACM, pp. 93-104 (2000) Cortes, Corinna and Vapnik, Vladimir: Support-vector network, Machine learning, Vol. 20-3, Springer, pp. 273-297 (1995) Breiman, Leo: Random forests, Machine learning, Vol. 45-1, Springer, pp. 5-32 (2001) Viola, Paul and Joes, Michael: Rapid object detection using a boosted cachedade of simple features, Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on, Vol. 1, IEEE (2001)

In the image inspection device using deep learning by the conventional convolutional neural network, if there are no or few defective products to be inspected, it is possible to construct an image inspection device that can automatically learn the feature amount of the image to be inspected. It was difficult.

The present invention has been made to solve the above-mentioned problems, and an image capable of automatically learning the feature amount of an image to be inspected even when there are no or few defective products to be inspected. Provide an inspection device.

In order to solve the above-mentioned problems, the image inspection device according to the present invention is an image inspection device that inspects the quality of an inspection target by an image, and learns a neural network based on a learning image including the inspection target. The feature amount learning unit that constructs a trained neural network that outputs a feature amount that can restore the learning image, and the learning unit that is output by the trained neural network after the learning of the feature amount learning unit is completed. A discriminator learning unit that generates a discriminator for determining the quality of the inspection target based on the feature amount of the image by learning, and a determination image including the inspection target are input to the trained neural network, and the above is described. The feature amount calculation unit that outputs the feature amount of the determination image and the feature amount of the determination image output by the feature amount calculation unit are input to the classifier generated by the discriminator learning unit. It is characterized by including an identification unit for determining the quality of the inspection target.

Further, in the image inspection device according to the present invention, the feature amount learning unit has a first input image storage unit that stores the learning image as an input image and a feature amount of the input image using the input image as an input. The first convolutional neural network calculation unit that outputs A convolutional neural network calculation unit, an output image storage unit that stores the image output from the reverse convolutional neural network calculation unit as an output image, and an image difference calculation unit that calculates the difference between the input image and the output image. , The feature amount calculation setting update unit that updates the parameter values of the first convolutional neural network calculation unit and the reverse convolutional neural network calculation unit so that the difference becomes smaller than the calculated value, and the first A feature amount calculation setting storage unit for storing the values of the parameters of each of the convolutional neural network calculation unit and the reverse convolutional neural network calculation unit is provided, and the first convolutional neural network calculation unit and the reverse convolutional neural network calculation unit are provided. Each unit performs an operation using the value of the parameter stored in the feature amount calculation setting storage unit, and the feature amount calculation setting storage unit stores the updated value of the parameter, and the first unit. The convolutional neural network calculation unit of the above may output the feature amount of each of the learning images by using the updated value of the parameter.

Further, in the image inspection apparatus according to the present invention, the feature amount calculation unit is an updated input image storage unit that stores the determination image and an updated feature amount calculation setting storage unit. It may be provided with a second convolutional neural network calculation unit that inputs the determination image using the value of the parameter and outputs the feature amount of the determination image.

Further, in the image inspection device according to the present invention, the discriminator learning unit includes a first feature amount input unit for inputting the feature amount of the learning image output by the feature amount learning unit, and the learning image. The discriminant operation unsupervised learning unit is provided with an unsupervised learning unit that performs unsupervised learning and generates the discriminator by using the feature amount of the above as an input. You may learn by any of the following methods: Class support vector machine, subspace method, Local Outlier Factor, and statistical test.

Further, in the image inspection apparatus according to the present invention, the identification unit includes a second feature amount input unit for inputting the feature amount of the determination image output by the feature amount calculation unit, and the determination unit. The discriminator generated by the discriminating calculation unsupervised learning unit may be provided with a discriminating calculation unit that determines the quality of the inspection target by using the feature amount as an input.

Further, in the image inspection apparatus according to the present invention, the discriminator learning unit further includes a discriminator adjusting unit that adjusts the value of the adjustment parameter of the discriminating calculation unsupervised learning unit, and the discriminator adjusting unit is a defective product. The value of the adjustment parameter may be adjusted based on the feature amount calculated from the image to be inspected in which the image showing the above and the image showing the non-defective product are distinguished.

Further, in the image inspection apparatus according to the present invention, the discriminator learning unit includes a first feature amount input unit for inputting the feature amount of the learning image output by the feature amount learning unit, and the input. A pass / fail information input unit for inputting information indicating the quality of the learning image corresponding to the feature amount of the learning image, and information indicating the quality of the feature amount of the learning image and the corresponding learning image. As an input, the discriminant calculation supervised learning unit that generates the discriminator by supervised learning is provided, and the discriminant calculation supervised learning unit is a support vector machine, a random forest, or a boosting method. You may learn.

Further, in the image inspection apparatus according to the present invention, the identification unit includes a second feature amount input unit for inputting the feature amount of the determination image output by the feature amount calculation unit, and the determination unit. The discriminator generated by the discriminating calculation supervised learning unit may be provided with a discriminating calculation unit that determines the quality of the inspection target by using the feature amount as an input.

Further, the image inspection apparatus according to the present invention further includes an image acquisition unit having a first image acquisition unit for acquiring the learning image and a second image acquisition unit for acquiring the determination image. May be good.

Further, in the image inspection method according to the present invention, a feature amount learning step of learning a neural network based on a learning image including an inspection target and constructing a trained neural network that outputs the feature amount of the learning image. A discriminator learning step that generates a discriminator for determining the quality of the inspection target by learning based on the feature amount of the learning image output by the trained neural network, and a determination including the inspection target. The feature amount calculation step of inputting the image for use into the trained neural network and outputting the feature amount of the determination image, and the feature amount of the determination image output in the feature amount calculation step are subjected to the discriminator. It is characterized by comprising an identification step for determining the quality of the inspection target by inputting to the classifier generated in the learning step.

According to the present invention, learning of the folding layer for extracting the feature amount in the image to be inspected and learning of the discriminator for determining the quality of the inspection target are performed separately, so that there is no or few defective products to be inspected. Even so, the feature amount of the image to be inspected can be automatically learned.

FIG. 1 is a diagram illustrating the principle of the image inspection apparatus according to the present invention. FIG. 2 is a functional block diagram of the image inspection apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of a neural network structure used in the feature amount learning unit according to the first embodiment of the present invention. FIG. 4 is a diagram showing another example of the neural network structure used in the feature amount learning unit according to the first embodiment of the present invention. FIG. 5 is a functional block diagram of a feature amount learning unit and a feature amount calculation unit according to the first embodiment of the present invention. FIG. 6 is a functional block diagram of the discriminator learning unit and the discriminating unit according to the first embodiment of the present invention. FIG. 7 is a block diagram showing a hardware configuration of the image inspection device according to the first embodiment of the present invention. FIG. 8 is a flowchart of the feature amount learning process according to the first embodiment of the present invention. FIG. 9 is a flowchart of the discriminator-unsupervised learning process according to the first embodiment of the present invention. FIG. 10 is a flowchart of the feature amount calculation process according to the first embodiment of the present invention. FIG. 11 is a flowchart of the determination output process according to the first embodiment of the present invention. FIG. 12 is a functional block diagram of the image inspection apparatus according to the second embodiment of the present invention. FIG. 13 is a diagram illustrating adjustment of adjustment parameters by the classifier adjusting unit according to the second embodiment of the present invention. FIG. 14 is a flowchart of the parameter adjustment process according to the second embodiment of the present invention. FIG. 15 is a functional block diagram of the discriminator learning unit and the discriminating unit according to the third embodiment of the present invention. FIG. 16 is a flowchart of the discriminator supervised learning process according to the third embodiment of the present invention. FIG. 17 is a flowchart illustrating an outline of the operation of the conventional image inspection device. FIG. 18 is a diagram showing a deep learning model using a convolutional neural network used in a conventional image inspection device.

[Principle of invention]
FIG. 1 is a diagram illustrating the principle of the image inspection apparatus according to the present invention. In the image inspection apparatus according to the present invention, the convolutional layer for learning the feature amount of the image to be inspected and the discriminator for determining the quality of the inspection target are separately learned. In the learning of the convolution layer, unsupervised learning based on the self-encoder is performed. In the classifier, a learning method that can generate a classifier is used even when there are no defective products to be inspected or there are few defective products to be inspected.

As shown in FIG. 1, in an image inspection device according to a conventional technique, a neural network consisting of a combination of a convolution layer for extracting features of an image to be inspected and a fully connected layer for determining the quality of the inspection target is learned as a unit. Was there. Therefore, a certain number of defective images were required to learn the feature amount in the image to be inspected.

However, it can be said that it is not always necessary to learn the two parts of the convolution layer and the fully connected layer as one. First, in learning the convolution layer for extracting the feature amount of the image to be inspected, it is not always necessary to distinguish whether the image to be inspected is an image showing a good product or an image showing a defective product. In addition, it is possible to learn by separating the convolutional layer from the fully connected layer.

If only the convolutional layer can be learned, it is possible to replace the fully connected layer for determining the quality of the inspection target with a more suitable classifier and perform learning by the classifier alone. Further, in the learning of this classifier, a method that can be performed even when there is no defective image to be inspected or when there are very few defective products is used.

By using the above method, it is possible to realize an image inspection device that automatically learns the feature amount of the image to be inspected even when there are no defective products or the number of defective products is small. Hereinafter, each of the convolutional layer for extracting the feature amount of the image and the discriminator for determining the quality of the inspection target used in the image inspection apparatus according to the present invention will be described in more detail.

For the learning of the convolutional layer for extracting the features, the inspection target can be obtained by using a method such as a self-encoder (Auto Encoder), a variational self-encoder (Variational Auto Encoder), or a hostile generation network (Generative Adversarial Networks). Even if there is no or few defective images, the feature amount of the image can be learned by unsupervised learning. In particular, it is described in Non-Patent Documents 2 and 3 that the variable self-encoder and the hostile generation network have the ability to generate an image not included in the image used at the time of learning, and it is an unknown good product. It can be expected to have the effect of being applicable to images of defective products.

There are various methods for determining the quality of the inspection target in addition to the neural network consisting of fully connected layers used in the prior art. Among them, there is a method that can be used even in a situation where there is no defective product to be inspected or the number of defective products assumed in the present invention is small. Specifically, a statistical test using a mixed normal distribution model, an isolation forest, a Local Outlier Factor (hereinafter referred to as "LOF"), a support vector machine (hereinafter referred to as "SVM"), a random forest, and a booth. It is a technique such as ting.

Whereas a neural network consisting of fully connected layers needs to be learned using many images of both good and defective products, in the above method, only the good products to be inspected or the number of defective products is very small. Even in a small number of situations, it is possible to build a classifier by learning the features of the image. Therefore, the image inspection of the inspection target can be performed even in a situation where there is no or few defective products to be inspected as assumed in the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS. 2 to 16. The components that are common to each figure are designated by the same reference numerals.

[First Embodiment]
FIG. 2 is a functional block diagram of the image inspection device 1 according to the first embodiment of the present invention. In the first embodiment, the situation where there is no defective image to be inspected, or even if there is, the image used for learning is not distinguished into a good product and a defective product, that is, a situation where the product is not labeled is targeted. There is.

[Functional block of the entire image inspection device]
The image inspection device 1 includes an image pickup unit 2, an image acquisition unit 3, a learning unit 4, and a determination unit 5. The image inspection device 1 includes a learning unit 4 for learning the feature amount of an image of a product or the like to be inspected, and a determination unit 5 for determining the quality of the inspection target after learning.

The image pickup unit 2 captures an image including an inspection target. The image pickup unit 2 is used when the learning unit 4 performs learning, an image for learning including an inspection target (hereinafter, referred to as “learning image”), and when the determination unit 5 determines the quality of the inspection target. An image for determination (hereinafter referred to as "determination image") including the inspection target used for the above is taken. It is preferable that the image shooting conditions used for the learning unit 4 and the image shooting conditions used for the determination unit 5 match as much as possible. This is to prevent the correct inspection from being impossible due to the large difference in shooting conditions between the time of learning and the time of judgment.

The image acquisition unit 3 acquires an image taken by the image pickup unit 2 by electronic means such as network communication.

The learning unit 4 includes an image storage unit 41, a feature amount learning unit 42, and a discriminator learning unit 43. The learning unit 4 learns a large number of images of the product to be inspected taken by the imaging unit 2, and determines the calculation content so that the determination unit 5 can correctly determine the quality of the inspection target.
A plurality of learning images used by the learning unit 4 are stored in the image storage unit 41.

The feature amount learning unit 42 learns a neural network based on the image for learning, and constructs a trained neural network that outputs the feature amount of the image to be inspected. Further, the feature amount learning unit 42 inputs the learning image into the trained neural network and outputs the feature amount of the learning image.

The feature amount learning unit 42 acquires an operation for compressing the image to be inspected into a feature amount of less information by learning. The feature amount learning unit 42 performs unsupervised learning based on a self-encoder as learning of a neural network, and realizes the feature amount learning by the convolution layer described above. The details of the feature amount learning unit 42 will be described later.

The discriminator learning unit 43 generates a discriminator for determining the quality of the inspection target based on the feature amount of the learning image output by the feature amount learning unit 42 by learning. In the present embodiment, the discriminator learning unit 43 performs unsupervised learning to learn the discriminator. The discriminator generated by learning is used in the discriminating unit 52 of the determination unit 5 described later. The details of the classifier learning unit 43 and the classifier will be described later.

The determination unit 5 includes a feature amount calculation unit 51, an identification unit 52, and an output unit 53. The determination unit 5 determines the quality of the inspection target based on the determination image sent from the image acquisition unit 3.

The feature amount calculation unit 51 inputs the determination image into the trained neural network and outputs the feature amount of the determination image. The details of the feature amount calculation unit 51 will be described later.
The identification unit 52 inputs the feature amount of the determination image output by the feature amount calculation unit 51 into the discriminator generated by the discriminator learning unit 43, and determines the quality of the inspection target. The details of the identification unit 52 will be described later.

The output unit 53 displays the determination result by the identification unit 52 on a display screen or sends the determination result to the manufacturing apparatus of the product to be inspected, for example. The determination result output by the output unit 53 is used for sorting defective products and the like.

[Learning using a self-encoder by the feature learning unit]
The feature amount learning unit 42 included in the learning unit 4 aims to acquire a convolutional neural network operation such as compressing an image to be inspected into a feature amount of less information by learning. This feature amount is not limited to any information, and must be useful information for determining the quality of the inspection target in the image inspection.

In order to realize the feature amount learning unit 42 as described above, in this embodiment, a self-encoder having a convolutional neural network as shown in FIG. 3 is used. Then, the convolutional neural network constructed by the feature amount learning unit 42 is used by the feature amount calculation unit 51 to calculate the feature amount of the determination image. More specifically, the feature amount extractor shown in the portion surrounded by the broken line in FIG. 3 is constructed and used in the feature amount calculation unit 51.

The self-encoder has a structure that converts an input image into a latent variable having a smaller number of dimensions than the number of dimensions of the input image, and restores the input image to an image of the same size as the input image. In the self-encoder, when learning a neural network as shown in FIG. 3, the difference between the input image and the output image is made as small as possible. If the learning is successful and a self-encoder that outputs an image close to the original image can be generated, it means that the image can be represented by latent variables with a smaller number of dimensions than the original image.

Such latent variables can be a powerful feature in imaging tests. What is important for the present invention is that it is not necessary to distinguish whether the image to be inspected is a good image or a defective image when performing this learning. This is because the self-encoder learns so that the input image and the output image are as identical as possible, and does not learn whether the image is good or bad. .. This leads to the advantage of the present invention, that is, even if there are no defective products to be inspected or even if there are few defective products, the feature amount of the image to be inspected is automatically learned.

Also, as shown in FIG. 4, it is possible to add a mechanism of a hostile generation network to the self-encoder. The hostile generation network is a neural network that generates an image called a generator (hereinafter referred to as "Generator") and a discriminator (hereinafter referred to as "Discriminator") in which the image is a real image or a Generator. Has a neural network that determines whether the image is generated by.

In FIG. 4, a self-encoder having a convolutional neural network shown in FIG. 3 is used as a Generator. As shown in FIG. 4, the Discriminator learns to make as little mistake as possible whether it is a real image or not. On the other hand, the Generator learns that the Discriminator misidentifies the image as a real image. If learning progresses well by competing these two neural networks hostilely and the accuracy of the image generated by the Generator is improved, the part of the Generator that is the feature extractor will also be an excellent feature extractor. It is expected.

The variational self-encoder is also a technique having a similar mechanism and can be used in the same form. Neither the hostile generation network nor the variational self-encoder needs to distinguish between good and bad products in the data used for learning.

[Learning by the discriminator learning department and generation of the discriminator]
When there is no defective image to be inspected and only a sufficient number of good images can be secured, a discriminator can be generated by a method called unsupervised learning. For example, if the distribution of the feature amount in the good product image is estimated by the mixed normal distribution model and then the statistical test is used, it is possible to generate a discriminator in which the image deviating from the distribution is regarded as the defective product image.

Further, by using the subspace method, it is possible to learn the distribution of the feature amount in the image of a good product and generate a discriminator in which the product deviating from the distribution is regarded as a defective product. Further, there is also a means for generating a classifier by generating pseudo defective product data at a position sufficiently far from all non-defective product images, such as 1-class SVM (One-Class SVM).

On the other hand, even if a defective image to be inspected can be secured but a label indicating good or bad is not attached to the image for learning, there is a means for generating a classifier by unsupervised learning. For example, there is a classifier such as an isolation forest that classifies images for learning into a large number of groups and treats those belonging to a group composed of a small number of images as defective products. Further, there is a classifier such as a LOF that evaluates the distance between images for learning and regards an image having a large distance from other images for learning as a defective product.

[Functional block of feature amount learning unit and feature amount calculation unit]
FIG. 5 is a functional block diagram of the feature amount learning unit 42 and the feature amount calculation unit 51. As described above, the feature amount learning unit 42 constructs a self-encoder having a convolutional neural network that compresses the image to be inspected into the feature amount of less information by learning. The feature amount calculation unit 51 calculates the feature amount of the determination image by using the trained convolutional neural network constructed by the feature amount learning unit 42.

As shown in FIG. 5, the feature amount learning unit 42 includes an input image storage unit 421, a preprocessing unit 422, a convolutional neural network calculation unit (hereinafter referred to as “convolutional NN calculation unit”) 423, and a feature amount storage unit 424. Reverse convolutional neural network calculation unit (hereinafter referred to as "reverse convolutional NN calculation unit") 425, output image storage unit 426, feature amount calculation setting storage unit 427, image difference calculation unit 428, feature amount calculation setting update unit 429, and A feature amount output unit 430 is provided.

The feature amount calculation unit 51 includes an input image storage unit 511, a preprocessing unit 512, a convolutional NN calculation unit 513, a feature amount calculation setting storage unit 514, and a feature amount output unit 515.

First, each functional block included in the feature amount learning unit 42 will be described.
The input image storage unit 421 stores a large number of learning images required for learning executed by the feature amount learning unit 42 as input images.

The preprocessing unit 422 adjusts the learning image stored in the input image storage unit 421. Although the pre-processing unit 422 can be omitted, it is actually preferable to provide the pre-processing unit 422 to perform pre-processing of the image such as normalization of the brightness of the image. For the image preprocessing by the preprocessing unit 422, a generally known and known technique may be used.

The convolutional NN calculation unit 423 corresponds to the feature amount extractor shown in FIGS. 3 and 4. The convolutional NN calculation unit 423 takes a learning image (input image) as an input and outputs a feature amount of the learning image (input image). More specifically, the convolutional NN calculation unit 423 performs an operation centered on the convolutional operation on the input image using the value of the parameter stored in the feature amount calculation setting storage unit 427 described later, and inputs the input image. Outputs the feature amount of. This operation usually consists of multiple stages. When one step (one time) of calculation is performed, data that is smaller than the number of pixels of the input image or data is output.

The convolutional NN calculation unit 423 converts the original input image into a latent variable indicating the number of pixels smaller than the number of pixels by repeating such a convolutional calculation a plurality of times. This latent variable is obtained as a feature quantity that quantifies the features of the image to be inspected.

Further, the convolutional NN calculation unit 423 calculates the feature amount of all the learning images stored in the image storage unit 41 by using the value of the parameter updated by the feature amount calculation setting update unit 429 described later.

In the calculation and design of the convolutional neural network by the convolutional NN calculation unit 423, known techniques described in Patent Documents 2, 3 and the like may be used. In addition, a plurality of known software frameworks used for the calculation and design of convolutional neural networks are also known, and they may be used.

It is described in Non-Patent Document 4 that the performance of pooling (thinning out) used in a general convolutional neural network is improved by replacing it with a convolution with a stride. Therefore, it is preferable that the convolution NN calculation unit 423 also performs convolution with a stride. The feature amount of the input image calculated by the convolutional NN calculation unit 423 is temporarily stored in the feature amount storage unit 424 described later.

The feature amount storage unit 424 temporarily stores the feature amount of the input image output from the convolutional NN calculation unit 423. Further, the feature amount of the input image stored in the feature amount storage unit 424 is used by the deconvolutional NN calculation unit 425, which will be described later.

The deconvolutional NN calculation unit 425 receives the feature amount of the input image output by the convolutional NN calculation unit 423 as an input, and outputs an image having the same size as the input image. More specifically, the deconvolutional NN calculation unit 425 uses the values of the neural network parameters stored in the feature amount calculation setting storage unit 427 to perform the reverse operation of the convolutional neural network calculation by the convolutional NN calculation unit 423. conduct. The deconvolution NN calculation unit 425 performs the deconvolutional neural network calculation a plurality of times until the feature amount of the image becomes an image of the same size as the image stored in the input image storage unit 421.

As the deconvolutional neural network constituting the deconvolutional NN calculation unit 425, a network structure obtained by transposing the convolutional neural network constituting the convolutional NN calculation unit 423 may be used. Further, for the deconvolutional neural network calculation by the deconvolutional NN calculation unit 423, for example, a known calculation method described in Non-Patent Document 2 or the above-mentioned known software framework may be used. The calculation result of the last stage of the deconvolutional neural network by the deconvolution NN calculation unit 423 is stored in the output image storage unit 246 described later.

The output image storage unit 426 stores the calculation result output from the deconvolution NN calculation unit 425 as an output image.

The feature amount calculation setting storage unit 427 stores the parameter values of the convolutional neural network and the deconvolutional neural network used by the convolutional NN calculation unit 423 and the deconvolutional NN calculation unit 425. More specifically, the feature amount calculation setting storage unit 427 stores the initially set parameter values and the parameter values updated by the feature amount calculation setting update unit 429 described later.

The image difference calculation unit 428 calculates the difference between the input image (learning image) stored in the input image storage unit 421 and the output image stored in the output image storage unit 426 corresponding to the input image. do.

The feature amount calculation setting update unit 429 is a parameter of the convolutional neural network used by the convolutional NN calculation unit 423 and the reverse convolutional neural network used by the reverse convolutional NN calculation unit 425 stored in the feature amount calculation setting storage unit 427. Update the value. When a hostile generation network is used as the network, the parameter values of the neural network are updated based on the discrimination result of the Discriminator.

As a method for updating the parameter value in the neural network, a known method such as a reverse error propagation method (backpropagation) may be used. Further, the parameter of the neural network may be updated by the feature amount calculation setting update unit 429 every time one input image to be learned is input, or may be performed after a certain number of input images are accumulated. ..

The feature amount output unit 430 outputs the calculation result by the convolutional neural network calculation output from the convolutional NN calculation unit 423. More specifically, the feature amount output unit 430 outputs the feature amount corresponding to all the learning images calculated by the convolutional NN calculation unit 423 using the updated parameter values described later.

Next, each functional block included in the feature amount calculation unit 51 will be described. The feature amount calculation unit 51 calculates the feature amount of the determination image acquired by the image acquisition unit 3.

The input image storage unit 511 stores a determination image taken by the image pickup unit 2 and acquired by the image acquisition unit 3.

The pre-processing unit 512 has the same function as the pre-processing unit 422 of the feature amount learning unit 42, and performs pre-processing of the determination image stored in the input image storage unit 511 as necessary. The preprocessing unit 512 preprocesses the determination image and processes it so that the quality determination can be easily performed. Although the preprocessing unit 512 may be omitted as in the preprocessing unit 422 of the feature amount learning unit 42, the preprocessing unit 512 is provided to improve the determination accuracy in the feature amount calculation unit 51. May be possible.

The convolutional NN calculation unit 513 has the same configuration as the convolutional NN calculation unit 423 of the feature amount learning unit 42. Further, the convolutional NN calculation unit 513 outputs the feature amount of the determination image by inputting the determination image according to the updated value of the parameter of the convolutional neural network stored in the feature amount calculation setting storage unit 514. The convolutional NN calculation unit 513 performs the same calculation as the convolutional NN calculation unit 423 when the learning process of the feature amount learning unit 42 is completed.

The feature amount calculation setting storage unit 514 stores the values of the parameters of the convolutional neural network in the convolutional NN calculation unit 423 whose values have been updated by the feature amount learning unit 42. More specifically, the updated value of the parameter of the convolutional neural network is transferred from the feature amount calculation setting storage unit 427.

The feature amount output unit 515 outputs the feature amount of the determination image as a result of the convolutional neural network calculation by the convolutional NN calculation unit 513. The feature amount of the determination image output from the feature amount output unit 515 is input to the identification unit 52.

[Functional block of discriminator learning unit and discriminator unit]
FIG. 6 is a functional block diagram of the discriminator learning unit 43 and the discriminating unit 52. The discriminator learning unit 43 includes a feature amount input unit 431, a dimension reduction unit 432, and a discriminating calculation unsupervised learning unit 433. Further, the identification unit 52 includes a feature amount input unit 521, a dimension reduction unit 522, and an identification calculation unit 523. As described above, the discriminator generated by the discriminator learning unit 43 is used in the discriminator unit 52 to determine the quality of the inspection target.

First, each functional block included in the classifier learning unit 43 will be described.
The feature amount of the learning image output from the feature amount output unit 430 of the feature amount learning unit 42 is input to the feature amount input unit 431. More specifically, the feature amount input unit 431 collectively receives the feature amount calculated for the learning image stored in the image storage unit 41 after the learning process of the feature amount learning unit 42 is completed.

The dimension reduction unit 432 performs an operation to reduce the dimension of the feature amount of the image while keeping the information amount of the feature amount in the learning image input to the feature amount input unit 431 as small as possible. The dimension reduction unit 432 compresses the dimension of the feature amount of the image to a smaller dimension by using a known method such as principal component analysis.

The reasons for providing the dimension reduction unit 432 include the following reasons. Normally, there are a plurality of feature quantities of the learning image input to the feature quantity input unit 431 for each image. Therefore, when the number of features of the input image is large, the discriminator 52 may not be able to generate the classifier as it is, or the calculation amount required for the generation may be very large.

Therefore, by providing the dimension reduction unit 432, it is possible to avoid the so-called "curse of dimensionality", in which the calculation cost increases exponentially as the dimension of the mathematical space increases. If the dimension of the feature amount of the image output by the convolutional NN calculation unit 423 of the feature amount learning unit 42 is sufficiently low, the dimension reduction unit 432 may be omitted.

The discriminating calculation unsupervised learning unit 433 determines the quality of the inspection target by a known unsupervised learning method using the feature amount of the learning image input via the feature amount input unit 431 and the dimension reduction unit 432 as input. Generate a classifier. The discriminator generated by the discriminant unsupervised learning unit 433 is used in the discriminating calculation unit 523, which will be described later, included in the discriminating unit 52 of the determination unit 5.

The discriminant operation unsupervised learning unit 433 may use, for example, a method such as an isolation forest, a One-Class SVM, a subspace method, or a LOF as a known unsupervised learning method. As an example of the isolation forest, the method described in Non-Patent Document 5 may be used. As an example of the subspace method, the method described in Non-Patent Document 6 may be used. As an example of LOF, the method described in Non-Patent Document 7 may be used.

Next, each functional block included in the identification unit 52 will be described.
The feature amount input unit 521 has the same function and configuration as the feature amount input unit 431 of the discriminator learning unit 43. That is, the feature amount of the determination image output from the feature amount output unit 515 of the feature amount calculation unit 51 is input to the feature amount input unit 521.

The dimension reduction unit 522 has the same function and configuration as the dimension reduction unit 432 of the classifier learning unit 43. It is necessary to use completely the same calculation method for the dimension reduction unit 432 of the discriminator learning unit 43 and the dimension reduction unit 522 of the discriminator unit 52.

The identification calculation unit 523 uses the feature amount of the determination image input via the feature amount input unit 521 and the dimension reduction unit 522 as input, and the identification is generated by the identification calculation unsupervised learning unit 433 of the classifier learning unit 43. The quality of the inspection target is judged using a device. The determination result by the identification calculation unit 523 is output from the output unit 53.

[Hardware configuration of image inspection equipment]
FIG. 7 is a block diagram showing a hardware configuration of the image inspection device 1 according to the present embodiment. The image inspection device 100 includes a computer including a control unit 102, a communication control device 105, an image pickup device 106, a storage device 107, and a display device 108 connected via a bus 101, and a program for controlling these hardware resources. It can be realized.

The control unit 102 includes a CPU 103 and a main storage unit 104. The main storage unit 104 stores in advance a program for the CPU 103 to perform various controls and operations. The control unit 102 realizes the functions of the image inspection device 1 such as the learning unit 4 and the determination unit 5 shown in FIG.

The communication control device 105 is a control device for connecting a network between the image inspection device 100 and various external electronic devices.
The image pickup apparatus 106 can convert an optical signal into an image signal to generate a still image. More specifically, the image pickup device 106 has an image pickup device such as a CCD (charge-coupled device) image sensor or a CMOS image sensor, and forms an image of light incident from the image pickup region on the light receiving surface. , Convert to an electrical signal.

The storage device 107 includes a readable / writable storage medium and a drive device for reading / writing various information such as programs and data to the storage medium. A semiconductor memory such as a flash memory or a hard disk can be used as the storage medium in the storage device 107. The storage device 107 is an image storage unit 107a, an input image storage unit 107b, a feature amount calculation setting storage unit 107c, a feature amount storage unit 107d, an output image storage unit 107e, a program storage unit 107f, and other storage devices (not shown), for example. , A storage device for backing up programs, data, etc. stored in the storage device 107 can be provided.

The image storage unit 107a functions as the image storage unit 41 of the learning unit 4 and stores the learning image.
The input image storage unit 107b functions as an input image storage unit 421 of the feature amount learning unit 42 and an input image storage unit 511 of the feature amount calculation unit 51.

The feature amount calculation setting storage unit 107c functions as the feature amount calculation setting storage unit 427 of the feature amount learning unit 42, and stores the parameter values of the neural network.
The feature amount storage unit 107d functions as the feature amount storage unit 424 of the feature amount learning unit 42, and stores the feature amount of the learning image (input image) calculated by the feature amount learning unit 42.

The output image storage unit 107e functions as the output image storage unit 426 of the feature amount learning unit 42, and stores the output image of the calculation result of the deconvolution NN calculation unit 425.
The program storage unit 107f stores various programs for executing processing necessary for image inspection, such as learning processing by the learning unit 4 and determination processing by the determination unit 5 in the present embodiment.

The display device 108 functions as an output unit 53 of the image inspection device 1. The display device 108 is realized by a liquid crystal display or the like.

[Operation of image processing device]
Next, the operation of the image inspection device 1 according to the present embodiment will be described. In the following, the learning process by the learning unit 4 and the determination process by the determination unit 5 will be described separately. Further, the learning process will be described separately as a feature amount learning process by the feature amount learning unit 42 and a discriminator unsupervised learning process by the discriminator learning unit 43. The determination process will be described separately as a feature amount calculation process by the feature amount calculation unit 51 and a determination output process by the identification unit 52.

[Feature amount learning process]
First, the feature amount learning process by the feature amount learning unit 42 will be described with reference to the flowchart of FIG. First, the image pickup unit 2 captures a learning image including the inspection target (step S100). The image acquisition unit 3 acquires the captured image. The acquired learning image is stored in the image storage unit 41. The input image storage unit 421 reads the learning image acquired by the image acquisition unit 3 from the image storage unit 41 and stores it (step S101).

Next, the convolutional NN calculation unit 423 performs the convolutional neural network calculation a plurality of times using the learning image stored in the input image storage unit 421 as the input image, and outputs the feature amount of the input image (step S102). More specifically, the convolutional NN calculation unit 423 performs a calculation using the value of the parameter stored in the feature amount calculation setting storage unit 427. The input image (learning image) input to the convolutional NN calculation unit 423 may be preprocessed in advance by the preprocessing unit 422, such as normalizing the luminance.

The feature amount of the input image output from the convolutional NN calculation unit 423 is stored in the feature amount storage unit 424 (step S103). Next, the deconvolution NN calculation unit 425 executes the deconvolutional neural network calculation by using the feature amount of the input image stored in the feature amount storage unit 424 as an input (step S104).

More specifically, the deconvolutional NN calculation unit 425 performs a plurality of operations opposite to the convolutional neural network calculation by the convolutional NN calculation unit 423 using the value of the parameter stored in the feature amount calculation setting storage unit 427. ..

Further, the deconvolution NN calculation unit 425 performs the calculation until the calculated feature amount becomes an image of the same size as the input image stored in the input image storage unit 421. The image restored by the deconvolution NN calculation unit 425 is stored in the output image storage unit 426 as an output image.

Next, the image difference calculation unit 428 calculates the difference between the input image stored in the input image storage unit 421 and the output image stored in the output image storage unit 426 corresponding to the input image (step). S105).

Next, if the difference between the input image and the output image calculated by the image difference calculation unit 428 is not sufficiently small (step S106: NO), the feature amount calculation setting update unit 429 stores the feature amount calculation setting. The value of the parameter of the convolutional neural network stored in the part 427 is updated (step S107). The feature amount calculation setting update unit 429 may update the parameter values of the convolutional neural network by using a known method such as backpropagation.

The updated parameter values of the convolutional neural network are overwritten and stored in the feature amount calculation setting storage unit 427. After that, the convolutional NN calculation unit 423 performs the convolutional neural network calculation again using the updated parameter values (step S102). After that, the calculation result is stored as a feature amount of the input image (learning image) (step S103). Then, the deconvolution NN calculation unit 425 performs a deconvolution neural net calculation, restores the image, obtains an output image, and stores it in the output image storage unit 426 (step S104).

Next, the image difference calculation unit 428 calculates the difference between the input image and the output image again (step S105), and if the difference is sufficiently small (step S106: YES), the convolutional NN calculation unit 423 most recently. The value of the set parameter is transferred from the feature amount calculation setting storage unit 427 of the feature amount learning unit 42 to the feature amount calculation setting storage unit 514 of the feature amount calculation unit 51 (step S108).

The feature amount learning unit 42 repeatedly executes the processes from step S102 to step S107 several times. The convolutional NN calculation unit 423 when the difference between the input image and the output image is sufficiently small functions as a feature amount extractor (learned convolutional neural network) described with reference to FIGS. 3 and 4.

Further, in the present embodiment, the case where the feature amount learning unit 42 performs the above-mentioned processing from step S102 to step S107 every time one image is input has been described, but as described in [0078]. In addition, it is also possible to perform these processes every time a certain number of images are accumulated.

Next, the convolutional NN calculation unit 423 calculates the feature amount for all the learning images stored in the image storage unit 41 and outputs the feature amount to the feature amount output unit 430 (step S109).

As described above, the feature amount learning unit 42 learns the convolutional neural network using the learning image as an input image, updates the parameter values, and generates a feature amount extractor (learned convolutional neural network). do. Then, the feature amount learning unit 42 calculates the feature amount of the learning image by using the feature amount extractor generated by the learning.

[Unsupervised learning process]
Next, the unsupervised learning process of the classifier by the classifier learning unit 43 will be described with reference to the flowchart of FIG. First, the feature amount input unit 431 receives the feature amount of the learning image transmitted from the feature amount output unit 430 of the feature amount learning unit 42 (step S110).

Next, the dimension reduction unit 432 performs an operation to reduce the dimension while not reducing the amount of information of the feature amount of the received image (step S111). The dimension reduction unit 432 compresses the received image into features having a smaller dimension by using a known method such as principal component analysis.

Next, the discriminant calculation unsupervised learning unit 433 performs unsupervised learning (step S112) based on the feature amount of the learning image dimensionally compressed by the dimension reduction unit 432, and generates a discriminator (step S113). ). More specifically, the discriminant operation unsupervised learning unit 433 uses known unsupervised learning methods such as isolation forest, One-Class SVM, subspace method, and LOF to determine whether the inspection target is within the range of good products. Generate a classifier to determine if.

The discriminator generated by the learning by the discriminating calculation unsupervised learning unit 433 is used in the discriminating calculation unit 523 possessed by the discriminating unit 52 of the determination unit 5.
As described above, the discriminator learning unit 43 performs unsupervised learning based on the feature amount of the learning image to generate a discriminator.

[Feature amount calculation process]
Next, the feature amount calculation process by the feature amount calculation unit 51 possessed by the determination unit 5 will be described with reference to the flowchart of FIG. First, the determination image received from the image acquisition unit 3 is stored in the input image storage unit 511 (step S114).

Next, the convolutional NN calculation unit 513 reads out the values of the learned convolutional neural network parameters stored in the feature amount calculation setting storage unit 514 (step S115). After that, the convolutional NN calculation unit 513 uses the value of the read parameter and inputs the determination image stored in the input image storage unit 511 to perform the convolutional neural network calculation (step S116). The convolutional NN calculation unit 513 has the same configuration as the convolutional NN calculation unit 423 of the learning unit 4.

Next, the calculation result by the convolutional NN calculation unit 513 is output from the feature amount output unit 515 as the feature amount of the determination image (step S117). The feature amount of the determination image output from the feature amount output unit 515 is input to the identification unit 52.

As described above, the feature amount calculation unit 51 calculates the feature amount of the determination image using the values of the parameters of the trained convolutional neural network obtained by the feature amount learning process by the feature amount learning unit 42. ..

[Judgment output processing]
Next, the determination output process by the identification unit 52 and the output unit 53 of the determination unit 5 will be described with reference to the flowchart of FIG. First, the feature amount of the determination image output from the feature amount output unit 515 of the feature amount calculation unit 51 of the determination unit 5 is input to the feature amount input unit 521 of the identification unit 52 (step S118).

Next, the dimension reduction unit 522 performs dimensional compression of the input feature amount (step S119). After that, the identification calculation unit 523 is generated by a learning process by the identification calculation unsupervised learning unit 433 of the discriminator learning unit 43, using the feature amount in the dimension-compressed determination image output from the dimension reduction unit 522 as an input. Using the classifier, the quality of the inspection target is determined (step S120).

Then, the determination result by the identification calculation unit 523 is output from the output unit 53 (step S121). More specifically, the output unit 53 displays the determination result on a display screen such as a liquid crystal display, or transmits the determination result to an apparatus that manufactures the product to be inspected. In this way, the determination result regarding the quality of the inspection target is output from the output unit 53, which is used for sorting defective products and the like.

In this way, the discriminator 52 determines the quality of the inspection target by using the discriminator generated by the discriminator learning unit 43 in the learning process. Then, the output unit 53 outputs the determination result by the identification unit 52.

As described above, according to the first embodiment, the image inspection device 1 generates unsupervised learning of the convolution layer for extracting the feature amount of the image and a classifier for determining the quality of the inspection target. Do unsupervised learning separately. As a result, an image inspection device capable of automatically learning the feature amount of the image to be inspected even when there is no defective image to be inspected or even if the quality of the image to be inspected is not distinguished. can get.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the following description, the same reference numerals will be given to the same configurations as those of the first embodiment described above, and the description thereof will be omitted.

In the first embodiment, the discriminator learning unit 43 learns the discriminator by using a known unsupervised learning method such as isolation forest, One-Class SVM, subspace method, LOF, and the quality of the inspection target. The case of generating a classifier for determining the above is described. The second embodiment is common to the first embodiment in that a discriminator is generated by unsupervised learning.

However, in the second embodiment, the image inspection device 1A is different from the first embodiment in that the learning unit 4A further includes the classifier adjusting unit 44. Further, in the second embodiment, it is assumed that there are a very small amount of defective image images to be inspected for learning, and they are labeled so as to be distinguished from non-defective products.

[Identifier adjustment parameters]
The above-mentioned various classifiers described in the first embodiment usually have parameters (hereinafter, referred to as “adjustment parameters”) related to the determination of the quality of the inspection target. For example, in a method using a statistical test, the boundary between a non-defective product and a defective product changes when the significance level is changed. The classifier according to the method described in the first embodiment also has such an adjustment parameter. As shown by the arrow in FIG. 13, the boundary between the non-defective product and the defective product can be adjusted by adjusting the adjustment parameter of the discriminator. In FIG. 13, "white circles" indicate feature quantities of non-defective images to be inspected. Further, "X" indicates the feature amount of the defective product image to be inspected.

Adjustment of such adjustment parameters is important in unsupervised learning. If the boundary is set closer to the non-defective product side shown by the broken line on the right side of FIG. 13, the inspection target that was actually a defective product is mistakenly judged as a non-defective product and overlooked (hereinafter referred to as “missing”) is reduced. However, the number of cases where an inspection target that was actually a good product is excessively judged as a defective product (hereinafter referred to as "excessive judgment") increases. On the other hand, if the boundary is set closer to the defective product side shown by the broken line on the left side of FIG. 13, such an excessive determination is reduced, but the risk of missing a true defective product is increased.

It is important to adjust the adjustment parameters of the discriminator to the optimum values, but the adjustment can be difficult. However, when there are even a small number of images that are definitely known to be defective, the adjustment parameters can be easily adjusted as compared with the case where there are no defective images. For example, in the case of the distribution of the feature amount as shown in FIG. 13, if the boundary between the non-defective product and the defective product is set at the position indicated by the solid line in the center, the defective product is overlooked at least for the image data collected so far. It can be seen that the over-judgment can be suppressed to a small amount.

[Functional block of image inspection device]
Next, the functional block of the image inspection apparatus 1A according to the second embodiment will be described with reference to FIG. Hereinafter, the discriminator learning unit 43 and the discriminator adjusting unit 44, which are components different from the first embodiment, will be mainly described.

The discriminator learning unit 43 is provided with an adjustment parameter related to the accuracy of the quality determination in the discriminator generated by learning from the discriminator adjusting unit 44. The discriminator learning unit 43 generates a discriminator using the value of the adjustment parameter given by the discriminator adjusting unit 44.

The discriminator adjusting unit 44 adjusts the adjusting parameters of the discriminator to balance the risk of overlooking a defective product in the inspection target and the risk of overdetermining a non-defective product as a defective product. For example, the classifier adjusting unit 44 may adjust so that the ratio of over-determining a non-defective product as a defective product is equal to or less than a certain threshold value while avoiding overlooking defective products as much as possible. The value of the discriminator adjustment parameter adjusted by the discriminator adjusting unit 44 is input to the discriminator learning unit 43.

[Parameter adjustment processing]
Next, the parameter adjustment process for adjusting the adjustment parameter of the classifier will be described with reference to the flowchart of FIG. First, the discriminator adjusting unit 44 sets the discriminator adjusting parameter to a predetermined value (step S200). The discriminator adjusting unit 44 inputs the set adjustment parameter value to the discriminator learning unit 43. Then, the discriminant operation unsupervised learning unit 433 adjusts the discriminator with the value of the set adjustment parameter.

Next, the feature amount of the learning image calculated by the feature amount learning unit 42 (convolutional NN calculation unit 423) is input to the feature amount input unit 431 of the discriminator learning unit 43 (step S200). If necessary, the dimension reduction unit 432 performs dimension compression of the feature amount of the learning image.

Next, the unsupervised learning unit 433 of the discriminator learning unit 43 inputs the feature amount of the learning image into the discriminator whose adjustment parameters have been adjusted, and determines the quality of the inspection target (step S202). ). After that, the discriminant operation unsupervised learning unit 433 collates the judgment result of the quality of the inspection target with the distinction between the actual non-defective product and the defective product, and determines that the inspection target that was actually a defective product is a non-defective product (missed). The ratio between the presence or absence of the product and the case where the inspection target that was actually a non-defective product is determined to be a defective product (excessive determination) is calculated (step S203).

The discriminant operation unsupervised learning unit 433 is used when it is determined that there is no oversight (step S204: YES) and the excess determination ratio is equal to or less than the threshold value (step S205: YES) and is within the acceptance range. The value of the adjustment parameter is stored (step S206). After that, the identification unit 52 of the determination unit 5 uses the identification device set by the value of the adjustment parameter after adjustment.

On the other hand, if there is an oversight (step S204: NO) and the excess determination ratio is equal to or less than the threshold value (step S207: YES), the discriminator adjusting unit 44 is a non-defective product as described with reference to FIG. The value of the adjustment parameter is changed in the direction in which the ratio of determination is reduced (step S208). After that, the processes from step S201 to step S203 are executed again, and the adjustment parameter values are repeatedly changed until the excess determination ratio becomes equal to or less than the threshold value (step S205: YES) without overlooking (step S204: YES). ..

Further, when there is no oversight (step S204: YES) and the excess determination ratio is equal to or higher than the threshold value (step S205: NO), the discriminator adjusting unit 44 is a non-defective product as described with reference to FIG. The value of the adjustment parameter is changed in the direction in which the ratio of determination is increased (step S209). After that, the processes from step S201 to step S203 are executed again, and the adjustment parameter values are changed until the overlooked ratio is equal to or less than the threshold value (step S205: YES) without overlooking (step S204: YES). repeat.

Further, if there is an oversight (step S204: NO) and the excess determination ratio is equal to or higher than the threshold value (step S207: NO), the adjustment parameter can be overlooked no matter how it is adjusted. It means that it is difficult to obtain a target value such that the excess judgment ratio is equal to or less than the threshold value.

In this case, if another adjustment parameter exists in the discriminator, the parameter adjustment process may be performed again by using the other adjustment parameter. If the quality determination in the discriminator does not improve even if another adjustment parameter is adjusted by the discriminator adjusting unit 44, the feature quantity learning unit 42 may re-learn the feature quantity in the image to be inspected. You may consider it.

As described above, the discriminator obtained by adjusting the adjustment parameters by the discriminator adjusting unit 44 is used by the discriminating calculation unit 523 of the discriminating unit 52 possessed by the determination unit 5. Then, the identification calculation unit 523 determines the quality of the inspection target by using the discriminator adjusted with the value of the adjustment parameter.

As described above, according to the second embodiment, the discriminator adjusting unit 44 adjusts the adjustment parameters in the discriminator, and there is a risk that a defective product in the inspection target may be overlooked, and a non-defective product is excessively determined as a defective product. Balance with fear. Thereby, the adjustment of the image inspection device 1A can be performed more easily. In addition, the accuracy of the quality determination of the image inspection device 1A can be further improved.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the following description, the same components as those of the first and second embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

In the first and second embodiments, the case where unsupervised learning is used for learning the classifier has been described. On the other hand, in the third embodiment, the discriminator learning unit 43A has the discriminant operation supervised learning unit 435, and the discriminator is generated by supervised learning in the first and second embodiments. Is different. In the third embodiment, as in the second embodiment, there are a very small amount of defective images to be inspected for learning, and they are labeled so as to be distinguished from the non-defective images. Is assumed.

[Generation of classifier by supervised learning]
If a small number of defective images with labels indicating the quality of the image to be inspected and images of good products can be secured, a small amount of learning is performed by supervised learning such as SVM, random forest, and boosting. A robust classifier can be generated from the image sample for.

For example, the SVM uses only the minimum amount of data involved in discrimination and generates a discriminator that creates a margin around the discriminant boundary as much as possible. Therefore, stable identification results can be obtained even for unknown data.

Random forest is a classifier that combines a large number of child classifiers, but since a decision tree is used as these child classifiers, even a small number of samples can be correctly discriminated without dwarfing.

Boosting is also a classifier that combines a large number of child classifiers, but it also has the feature that each image for learning can be weighted, so by giving a large weight to a small amount of defective images, a small amount of defective products can be added. It can be expected that stable identification will be performed even for the image of.

[Functional block of discriminator learning unit and discriminator unit]
FIG. 15 is a functional block diagram of the discriminator learning unit 43A and the discriminating unit 52.
The classifier learning unit 43A includes a feature quantity input unit 431, a dimension reduction unit 432, a pass / fail information input unit 434, and a learning unit 435 with a discriminant calculation teacher.

The quality information input unit 434 inputs information indicating the quality of the image to be inspected corresponding to the feature amount of the learning image input to the feature amount input unit 431.

The learning unit 435 with a discrimination calculation supervised learning unit is known by inputting information on the feature amount of the learning image and whether the learning image based on the feature amount is an image showing a good product or an image showing a defective product. Generate a classifier by the supervised learning method of.

As known supervised learning methods used by the discriminating calculation supervised learning unit 435, for example, SVM described in Non-Patent Document 8, Random Forest described in Non-Patent Document 9, and Non-Patent Document 10 are described. There are boosting etc.

The discriminator generated by the discriminating calculation supervised learning unit 435 is used by the discriminating calculation unit 523 of the discriminating unit 52. Then, the identification calculation unit 523 uses the discriminator generated by supervised learning as an input for the feature amount of the determination image, and determines the quality of the inspection target.

[Learning process with discriminator supervised learning]
Next, the learning process with discriminator supervised learning will be described with reference to the flowchart of FIG. First, the feature amount input unit 431 of the discriminator learning unit 43A obtains the feature amount in the image to be inspected for learning obtained by the calculation result by the convolutional NN calculation unit 423 of the feature amount learning unit 42, and the feature amount output unit 430. Received from (step S300).

Next, the dimension reduction unit 432 performs dimensional compression of the feature amount of the learning image received by the feature amount input unit 431 (step S301). After that, the learning unit 435 with the identification calculation supervised learning unit receives the feature amount of the dimension-compressed learning image and the information indicating the quality of the corresponding learning image input via the quality information input unit 434 as input. Supervised learning is performed to learn the classifier (step S302).

The discriminant operation supervised learning unit 435 learns the discriminator by using a known supervised learning method such as SVM. Then, the learning unit 435 with a discriminating calculation supervised learning unit generates a discriminator for determining the quality of the inspection target (step S303).

The generated classifier is used in the discriminating calculation unit 523 of the discriminating unit 52 possessed by the determination unit 5. The discrimination calculation unit 523 uses the classifier generated by supervised learning to input the feature amount of the judgment image and determines the quality of the inspection target. Then, the output unit 53 displays the determination result on a display screen such as a liquid crystal display.

As described above, according to the third embodiment, since supervised learning is performed to generate a discriminator, even if there are only a small number of defective images to be inspected, the quality of the inspection target is determined. It is possible to learn and generate a classifier that can make a more accurate judgment.

Although the embodiments of the image inspection apparatus and the image inspection method of the present invention have been described above, the present invention is not limited to the described embodiments and is assumed by those skilled in the art within the scope of the invention described in the claims. It is possible to make various possible modifications.

For example, in the described embodiment, the case where the image pickup unit 2 and the image acquisition unit 3 are shared by the learning unit 4 and the determination unit 5 has been described, but the learning unit 4 and the determination unit 5 are independent of each other. It may have an image pickup unit 2 and an image acquisition unit 3.

Further, in the described embodiment, the case where the learning unit 4 and the determination unit 5 are mounted on the same computer has been described, but the learning unit 4 and the determination unit 5 are mounted on different computers. May be good.

1, 1A, 100 ... Image inspection device, 2 ... Imaging unit, 3 ... Image acquisition unit, 4 ... Learning unit, 5 ... Judgment unit, 41 ... Image storage unit, 42 ... Feature quantity learning unit, 43 ... Discriminator learning unit , 51 ... feature amount calculation unit, 52 ... identification unit, 53 ... output unit, 101 ... bus, 102 ... control unit, 103 ... CPU, 104 ... main storage unit, 105 ... communication control device, 106 ... image pickup device, 107 ... Storage device, 108 ... Display device.

Claims (7)

It is an image inspection device that inspects the quality of the inspection target by images.
A feature amount learning unit that performs unsupervised learning of a neural network based on a learning image including the inspection target and constructs a trained neural network that outputs a feature amount that can restore the learning image.
After the learning of the feature amount learning unit is completed, the discriminator learning unit that generates a discriminator for determining the quality of the inspection target based on the feature amount of the learning image output by the trained neural network by learning. When,
A feature amount calculation unit that inputs a judgment image including the inspection target into the trained neural network and outputs a feature amount of the judgment image, and a feature amount calculation unit.
It is provided with an identification unit for inputting the feature amount of the determination image output by the feature amount calculation unit into the classifier generated by the discriminator learning unit to determine the quality of the inspection target .
The classifier learning unit
An unsupervised learning unit is provided, which performs unsupervised learning by inputting the feature amount of the learning image and generates the discriminator.
The identification unit is
Using the feature amount of the determination image as an input and the discriminator generated by the unsupervised learning unit for discrimination calculation, the quality of the inspection target is determined.
An image inspection device characterized by that. In the image inspection apparatus according to claim 1,
The feature amount learning unit is
A first input image storage unit that stores the learning image as an input image,
A first convolutional neural network calculation unit that takes the input image as an input and outputs the feature amount of the input image.
A deconvolutional neural network calculation unit that outputs an image of the same size as the input image by using the feature amount of the input image output by the first convolutional neural network calculation unit as an input.
An output image storage unit that stores the image output from the deconvolutional neural network calculation unit as an output image, and an output image storage unit.
An image difference calculation unit that calculates the difference between the input image and the output image,
A feature amount calculation setting update unit that updates the parameter values of the first convolutional neural network calculation unit and the deconvolutional neural network calculation unit so that the difference becomes smaller than the calculated value.
A feature amount calculation setting storage unit that stores the values of the parameters of each of the first convolutional neural network calculation unit and the deconvolutional neural network calculation unit, and
Equipped with
The first convolutional neural network calculation unit and the deconvolutional neural network calculation unit perform calculations using the values of the parameters stored in the feature amount calculation setting storage unit, respectively.
The feature amount calculation setting storage unit stores the updated value of the parameter, and stores the updated value.
The first convolutional neural network calculation unit is an image inspection apparatus characterized in that it outputs the feature amount of each of the learning images by using the updated value of the parameter. In the image inspection apparatus according to claim 2,
The feature amount calculation unit is
A second input image storage unit that stores the determination image, and
A second convolutional neural network calculation unit that outputs the feature amount of the determination image by inputting the determination image using the updated value of the parameter stored in the feature amount calculation setting storage unit. An image inspection device characterized by being provided with. In the image inspection apparatus according to any one of claims 1 to 3, the image inspection apparatus
The discriminant operation unsupervised learning unit is characterized in that the discriminator is trained by any of the following methods: isolation forest, One-Class support vector machine, subspace method, Local Outlier Factor, and statistical test. Image inspection equipment. In the image inspection apparatus according to any one of claims 1 to 4 .
The discriminator learning unit further includes a discriminator adjusting unit that adjusts the value of the adjustment parameter of the discriminating operation unsupervised learning unit.
The discriminator adjusting unit is characterized in that the value of the adjustment parameter is adjusted based on the feature amount calculated from the image to be inspected in which the image showing a defective product and the image showing a non-defective product are distinguished. Image inspection equipment. In the image inspection apparatus according to any one of claims 1 to 5 .
Further, an image inspection apparatus including an image acquisition unit having a first image acquisition unit for acquiring the learning image and a second image acquisition unit for acquiring the determination image. A feature amount learning step of performing unsupervised learning of a neural network based on a learning image including an inspection target and constructing a trained neural network that outputs the feature amount of the learning image.
A discriminator learning step of generating a discriminator for determining the quality of the inspection target by learning based on the feature amount of the learning image output by the trained neural network.
A feature amount calculation step of inputting a judgment image including the inspection target into the trained neural network and outputting the feature amount of the judgment image.
It is provided with an identification step in which the feature amount of the determination image output in the feature amount calculation step is input to the classifier generated in the discriminator learning step to determine the quality of the inspection target .
The discriminator learning step
An unsupervised learning step is provided in which unsupervised learning is performed by using the feature amount of the learning image as an input to generate the discriminator.
The identification step is
Using the feature amount of the determination image as an input and the discriminator generated by the discriminant operation unsupervised learning step, the quality of the inspection target is determined.
An image inspection method characterized by that.

Images