JP2019087181A - Device and method for image inspection - Google Patents

Abstract

To provide an image inspection device that can automatically learn the feature amount of an image of an inspection target even if there are no defective items or less items in the inspection targets.SOLUTION: An image inspection device 1 includes: a feature amount learning unit 42 for learning a neural network based on a learning image including an inspection target and constructing the learned neural network, which outputs the feature amount of the learning image; a discriminator learning unit 43 for generating, by learning, a discriminator for determining whether the inspection target is defective, on the basis of the feature amount of the learning image output from the learned neural network; a feature amount calculation unit 51 for inputting a determination image including the inspection target into the learned neural network and outputting the feature amount of the determination image; and an identification unit 52 for determining whether the inspection target is defective by inputting the feature amount of the determination image output from the feature amount calculation unit 51 into the discriminator generated by the discriminator learning unit 43.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image inspection apparatus and method, and more particularly to an inspection technique using an image of an object to be inspected.

  2. Description of the Related Art Conventionally, as a method of inspecting a finished product, a semi-finished product, or a product or part in a processing process, there is one called image inspection using an image. In this method, an image to be inspected is acquired by an imaging device, and the presence or absence of a defect or a defect is determined based on the acquired image. For example, Patent Document 1 discloses a technique for determining the quality of the appearance of an inspection object.

  Although there are a wide variety of image inspection methods for inspecting the presence or absence of defects and defects in inspection objects such as products, the general operation in the conventional image inspection apparatus is generally the operation shown in the flowchart of FIG. . First, the image inspection apparatus acquires an image of an inspection object (step S800). Next, the image inspection apparatus preprocesses the acquired image, and processes the image into a state easy to inspect (step S801).

  Subsequently, the image inspection apparatus calculates a feature amount obtained by quantifying the feature of the image from the image after the pre-processing (step S802). The calculated feature amounts of the image may be one or more depending on the specific method adopted in the image inspection apparatus. Then, the image inspection apparatus determines whether the inspection target is a non-defective product or a non-defective product based on the value of the feature amount of the image (step S803).

  There are various feature quantities that quantify the image features, but in the conventional image inspection apparatus, it is common for the designer of the image inspection apparatus to design and select based on his / her knowledge . In addition, various methods, such as those based on the threshold determined by the designer, those based on statistical tests, those based on machine learning, etc., can be used to determine whether the inspection target is a good product or a defective product. Is used.

  In recent years, there has been proposed a method of performing image inspection by learning feature quantities of an image to be inspected by deep learning using a convolutional neural network. In such a method, the inspection is performed using a neural network 900 configured as shown in FIG. In this method, a plurality of elements called convolution layers 901a, 901b, and 901c, which perform calculations centering on a convolution operation, are connected in series, and characteristic patterns are extracted from the input inspection target image, and the image features Calculate the quantity. Then, based on the calculated feature amount of the image, a portion called a total joint layer 902 subsequent to the convolutional layers 901a, 901b, and 901c determines the quality of the inspection target.

  For example, Patent Document 2 is directed to a medical image, but classifies an examination object with respect to a two-dimensional image generated from a three-dimensional image to be examined using a neural network having a convolution layer and a total connection layer. It discloses technology.

  The feature of the image inspection method based on such a neural network is that the feature amount of the image is automatically acquired by learning. As described above, in the technique using the conventional image inspection method as described in Patent Document 1, the feature amount of the image is generally designed based on the knowledge of the designer. Although the feature amount greatly affects the performance of the image inspection, the appropriate feature amount differs depending on the inspection target. Therefore, it has been considered that the design of feature quantities is difficult unless it is an expert in image processing and image inspection.

  On the other hand, in the image inspection method based on a neural network as described in Patent Document 2, if a large number of images to be inspected of non-defective products and defective products are prepared, the images more suitable for inspection by learning them. Feature quantities are obtained. As a result, the design of feature quantities can be significantly reduced. It is a well-known fact, for example, as disclosed in Non-Patent Document 1, that feature values are automatically acquired by deep learning using convolutional neural networks.

  In the method described in Patent Document 1, it is necessary to make the neural network learn the distinction between good and bad products before image inspection, and to be able to determine whether the test object is good or bad with a sufficiently high correct answer rate. . This learning is called "supervised learning", and requires an image to be inspected in which good and bad products are distinguished in advance. On the other hand, learning of images by deep learning requires a large number of images for learning. Further, it is desirable that the numbers of non-defective images to be inspected and non-defective images used for learning be close to the same number. This means that not only many good images but also many bad images are needed.

  However, it is often difficult to prepare a large number of defective product images as learning images. In production sites, production facilities and processes are usually designed so as to minimize the occurrence of defective products, and it is not uncommon for the defective product rate to be low. However, images for inspection collected from such a site are mostly non-defective images, and in many cases, it is difficult to learn a neural network as shown in FIG.

  The reason is that, in general, supervised learning of a neural network proceeds in the case where the output of the neural network is different from the teacher, ie, in the case of an incorrect output. Although there are many false positives at the beginning, it is the normal learning process that learning advances and the false negatives decrease gradually and the final judgment becomes correct.

  However, when there are only non-defective images or when there are very few non-defective images, the correct answer rate is high even if the neural network always outputs non-defective products. Therefore, it becomes difficult to learn the distinction between good and bad that you really want to learn.

  For these reasons, there have been cases where it has been difficult to apply an image inspection apparatus using deep learning based on the conventional convolutional neural network when there are no defective products or few defective products. And, this means that when a new production facility or device is started up, inspection can not be performed until a considerable number of images of defective products are accumulated. Therefore, although the image inspection apparatus using deep learning based on the conventional convolutional neural network has an advantage that it can automatically learn the feature amount of the image, the scene where it can be exhibited is limited.

JP, 2017-174039, A JP, 2016-99707, A JP 2008-310700 A

Yu Matsuo "Artificial Intelligence Beyond Humans? Things Beyond Deep Learning", Kadokawa EPUB Selected Book, March 2015 Alec Radford et al. : Unsupervised Representation Learning with Deep Convolutional Generative Adversalial Networks, ICLR 2016 (arXiv: 1511.06434) Diederik P. Kingma et al. : Auto-Encoding Variational Bayes (arXiv: 1312.6114) Jost Tobias Springenberg et al. : Striving for Simplicity: The All Convolutional Net (arXiv: 1412.6806) Liu, Fei Tony and Ting, Kai Ming and Zhou, Zhi-Hua: Isolation forest, Data Mining, 2008. ICDM ‘08. Eighth IEEE International Conference on, IEEE, pp. 413-422 (2008) Watanabe, Satosi and Pakvasa, Nikhil: Subspace method of pattern recognition, Proc. 1st. IJCPR, pp. 25-32 (1973) Breunig, Markus M and Kriegel, Hans-Peter and Ng, Raymond T and Sander, Jorg: LOF: identifying density-based local outliers, ACM sigmod record, Vol. 29-2, ACM, pp. 93-104 (2000) Cortes, Corinna and Vapnik, Vladimir: Support-vector networks, Machine learning, Vol. 20-3, Springer, pp. 273-297 (1995) Breiman, Leo: Random forests, Machine learning, Vol. 45-1, Springer, pp. 5-32 (2001) Viola, Paul and Jones, Michael: Rapid object detection using a boosted cascade of simple features, Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on Vol. 1, IEEE (2001)

  In the conventional image inspection apparatus using deep learning by convolutional neural network, it is possible to construct an image inspection apparatus capable of automatically learning the feature quantity of the image to be inspected when there are no defective products to be inspected or when there are few defective products to be inspected. It was difficult.

  The present invention has been made to solve the above-described problem, and an image that can automatically learn feature quantities of an image to be inspected even if there are no or few defective products to be inspected. Provide an inspection device.

  In order to solve the problems described above, an image inspection apparatus according to the present invention is an image inspection apparatus which inspects the quality of an inspection object by an image, and learning of a neural network based on a learning image including the inspection object. And a feature amount learning unit for constructing a learned neural network that outputs a feature amount capable of restoring the learning image, and the learning output from the learned neural network after the learning of the feature amount learning unit is completed. A classifier learning unit that generates a classifier that determines whether the inspection object is good or bad based on the feature amount of the image by learning, a determination image including the inspection object is input to the learned neural network, and A feature amount calculation unit for outputting a feature amount of the determination image, and the feature amount of the determination image output from the feature amount calculation unit Enter the discriminator generated by section, characterized in that it comprises an identification unit for judging quality of said object.

  Further, in the image inspection apparatus according to the present invention, the feature amount learning unit is a first input image storage unit that stores the learning image as an input image, and the feature amount of the input image using the input image as an input. A first convolutional neural network computing unit for outputting the first and second feature quantities of the input image output from the first convolutional neural network computing unit as an input and outputting an image of the same size as the input image A convolutional neural network operation unit; an output image storage unit which stores the image output from the deconvolution neural network operation unit as an output image; and an image difference calculation unit which calculates a difference between the input image and the output image The first convolutional neural network operation unit such that the difference is smaller than a calculated value A feature amount calculation setting updating unit for updating the value of each parameter of the deconvoluted neural network operation unit, and a feature for storing the value of the parameter of each of the first convolutional neural network operation unit and the deconvoluted neural network operation unit An amount calculation setting storage unit, and the first convolutional neural network calculation unit and the deconvolution neural network calculation unit respectively calculate using the value of the parameter stored in the feature quantity calculation setting storage unit And the feature amount calculation setting storage unit stores the updated value of the parameter, and the first convolutional neural network operation unit uses the updated value of the parameter for all the learning The feature amount of each image may be output.

  Further, in the image inspection apparatus according to the present invention, the feature amount calculation unit is a second input image storage unit for storing the determination image, and the updated feature amount stored in the feature amount calculation setting storage unit. The image processing apparatus may further include a second convolutional neural network computing unit that receives the determination image as an input using the parameter value and outputs the feature amount of the determination image.

  In the image inspection apparatus according to the present invention, the classifier learning unit may be a first feature input unit for inputting the feature of the learning image output by the feature learning unit, and the learning image. And an unsupervised learning unit that performs unsupervised learning and generates the discriminator, the unsupervised learning unit performing the unsupervised learning by using the discriminator as an isolation forest, One- It may be learned by any of Class support vector machine, subspace method, Local Outlier Factor, and statistical test.

  Further, in the image inspection apparatus according to the present invention, the identification unit is a second feature amount input unit for inputting the feature amount of the determination image output by the feature amount calculation unit; The image processing apparatus may further include a discrimination operation unit that determines the quality of the inspection object using the classifier generated by the discrimination operation unsupervised learning unit with the feature amount as an input.

  In the image inspection apparatus according to the present invention, the classifier learning unit further includes a classifier adjustment unit for adjusting the value of the adjustment parameter of the classification arithmetic unsupervised learning unit, and the classifier adjustment unit is a defective product. The value of the adjustment parameter may be adjusted based on the feature value calculated from the image of the inspection target in which the image indicating the image and the image indicating the non-defective item are distinguished.

  In the image inspection apparatus according to the present invention, the classifier learning unit may be a first feature input unit for inputting the feature of the image for learning output from the feature learning unit; A quality information input unit for inputting information indicating the quality of the learning image corresponding to the feature amount of the learning image; information indicating the quality of the learning image corresponding to the feature amount of the learning image And a classification operation supervised learning unit that generates the classifier by supervised learning as an input, wherein the classification operation supervised learning unit uses one of a support vector machine, a random forest, and a boosting method. You may learn.

  Further, in the image inspection apparatus according to the present invention, the identification unit is a second feature amount input unit for inputting the feature amount of the determination image output by the feature amount calculation unit; The image processing apparatus may further include an identification operation unit that determines the quality of the inspection target using the identifier generated by the identification operation supervised learning unit using a feature amount as an input.

  The image inspection apparatus according to the present invention further includes an image acquisition unit having a first image acquisition unit for acquiring the learning image, and a second image acquisition unit for acquiring the determination image. It is also good.

  In the image inspection method according to the present invention, a feature amount learning step of learning a neural network based on a learning image including an inspection target and constructing a learned neural network that outputs the feature amount of the learning image. And a discriminator learning step of generating a discriminator for judging whether the inspection object is good or bad based on the feature amount of the image for learning output from the learned neural network, and determining the inspection object. For the feature amount calculating step of inputting the learning image to the learned neural network and outputting the feature amount of the determination image, and the feature amount of the determination image output in the feature amount calculating step, the discriminator Providing an identification step of determining whether the inspection object is good or bad by inputting to the identifier generated in the learning step; And butterflies.

  According to the present invention, the learning of the convolutional layer for extracting the feature amount in the image to be inspected and the learning of the discriminator to determine the quality of the inspection object are separately performed. Even in this case, the feature amount of the image to be inspected can be learned automatically.

FIG. 1 is a view for explaining the principle of the image inspection apparatus according to the present invention. FIG. 2 is a functional block diagram of the image inspection apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram showing an example of a neural network structure used in the feature amount learning unit according to the first embodiment of the present invention. FIG. 4 is a view showing another example of the neural network structure used in the feature amount learning unit according to the first embodiment of the present invention. FIG. 5 is a functional block diagram of a feature amount learning unit and a feature amount calculating unit according to the first embodiment of the present invention. FIG. 6 is a functional block diagram of a classifier learning unit and a discrimination unit according to the first embodiment of the present invention. FIG. 7 is a block diagram showing the hardware configuration of the image inspection apparatus according to the first embodiment of the present invention. FIG. 8 is a flowchart of feature amount learning processing according to the first embodiment of the present invention. FIG. 9 is a flowchart of classifier unsupervised learning processing according to the first embodiment of the present invention. FIG. 10 is a flowchart of feature quantity calculation processing according to the first embodiment of the present invention. FIG. 11 is a flowchart of determination output processing according to the first embodiment of the present invention. FIG. 12 is a functional block diagram of an image inspection apparatus according to a second embodiment of the present invention. FIG. 13 is a diagram for explaining adjustment of adjustment parameters by the classifier adjusting unit according to the second embodiment of the present invention. FIG. 14 is a flowchart of parameter adjustment processing according to the second embodiment of the present invention. FIG. 15 is a functional block diagram of a classifier learning unit and a classifier according to a third embodiment of the present invention. FIG. 16 is a flowchart of discriminator supervised learning processing according to the third embodiment of the present invention. FIG. 17 is a flow chart for explaining the outline of the operation of the conventional image inspection apparatus. FIG. 18 is a diagram showing a deep learning model by a convolutional neural network used in a conventional image inspection apparatus.

[Principle of the invention]
FIG. 1 is a view for explaining the principle of the image inspection apparatus according to the present invention. The image inspection apparatus according to the present invention separates and learns separately the convolutional layer for learning the feature amount of the image to be inspected and the classifier for judging the quality of the inspection object. In convolutional layer learning, unsupervised learning based on a self-coder is performed. In the discriminator, a learning method is used which can generate the discriminator even in a situation where there is no defective product to be inspected or a situation in which there are few defective products to be inspected.

  As shown in FIG. 1, in an image inspection apparatus according to the prior art, a neural network consisting of a combination of a convolutional layer for extracting feature quantities of an image to be inspected and all connected layers for determining the quality of the inspection object is integrally learned. It was Therefore, in order to learn the feature amount in the image to be inspected, a fixed number of images of defective products are required.

  However, it can be said that it is not always necessary to learn two parts of the convolutional layer and the total joint layer as one. First, in learning of the convolutional layer for extracting the feature amount of the image to be inspected, it is not always necessary to distinguish whether the image to be inspected is an image indicating a non-defective item or an image indicating a non-defective item. It is also possible to learn the convolutional layer separately from all the coupled layers.

  If only the convolutional layer can be learned, it is also possible to perform learning with the discriminator alone by replacing all the joint layers which determine the quality of the inspection object with a more suitable discriminator. In addition, for learning of the classifier, a method that can be performed even when there is no defective image to be inspected or even a very small number is used.

  By using the method as described above, it is possible to realize an image inspection apparatus that automatically learns the feature amount of the image to be inspected even when there is no defective product or when the number of defective products is small. Hereinafter, each of the convolution layer used to extract the feature amount of the image and the classifier that determines the quality of the inspection target used in the image inspection apparatus according to the present invention will be described in more detail.

  With regard to the learning of the convolutional layer for extracting the feature value, the inspection object can be checked by using a method such as an auto encoder, an variational auto encoder, or a generative adaptive networks. Even if there are no or few defective images of the above, it is possible to learn the feature amount of the image by unsupervised learning. In particular, it is described in Non-Patent Documents 2 and 3 etc. that the variational self-coder and the hostile generation network have the ability to generate an image not included in the image used at the time of learning. The effect of being able to adapt to the image of a defective product can be expected.

  There are various classifiers for judging the quality of the inspection object other than the neural network consisting of all connection layers used in the prior art. Among them, there is also a method which can be used even in a situation where there are no defective products to be inspected which are assumed in the present invention or the number of defective products is small. Specifically, statistical tests using mixed normal distribution models, isolation forests, local outlier factors (hereinafter referred to as “LOF”), support vector machines (hereinafter referred to as “SVM”), random forests, booths It is a method such as

  The neural network consisting of all connection layers needs to learn by using many images of both good and bad products, whereas with the above method, only the number of good images to be inspected or the number of bad images is extremely small. Even in a small number of situations, a classifier can be constructed by learning feature quantities of an image. Therefore, it is possible to perform the image inspection of the inspection target even in a situation where there are no or few defective products to be inspected as assumed in the present invention.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to FIGS. 2 to 16. The components common to the respective drawings are denoted by the same reference numerals.

First Embodiment
FIG. 2 is a functional block diagram of the image inspection apparatus 1 according to the first embodiment of the present invention. In the first embodiment, a situation in which there is no defective image to be inspected or an image used for learning even if there is no distinction between good and defective items, that is, a situation in which it is not labeled There is.

[Function block of the whole image inspection device]
The image inspection apparatus 1 includes an imaging unit 2, an image acquisition unit 3, a learning unit 4, and a determination unit 5. Image inspection apparatus 1 includes a learning unit 4 that learns feature amounts of an image of a product to be inspected and the like, and a determination unit 5 that determines the quality of the inspection object after learning, when the function is roughly divided.

  The imaging unit 2 captures an image including an inspection target. When the imaging unit 2 uses a learning image including an examination target (hereinafter referred to as a “learning image”) used when the learning unit 4 performs learning, and when the determination unit 5 determines the quality of the examination object An image for determination (hereinafter referred to as “image for determination”) including an inspection target used for It is preferable that the shooting conditions of the image used in the learning unit 4 and the shooting conditions of the image used in the determination unit 5 match as much as possible. This is to prevent that the inspection can not be performed correctly because the imaging conditions greatly differ between the time of learning and the time of determination.

  The image acquisition unit 3 acquires an image captured by the imaging unit 2 by electronic means such as network communication, for example.

The learning unit 4 includes an image storage unit 41, a feature amount learning unit 42, and a classifier learning unit 43. The learning unit 4 learns a large number of images of the product to be inspected which is photographed by the imaging unit 2 and determines the operation content so that the determination unit 5 can correctly determine the quality of the inspection object.
The image storage unit 41 stores a plurality of learning images used by the learning unit 4.

  The feature amount learning unit 42 learns a neural network based on the image for learning, and constructs a learned neural network that outputs the feature amount of the image to be inspected. Further, the feature amount learning unit 42 inputs the learning image into the learned neural network, and outputs the feature amount of the learning image.

  The feature amount learning unit 42 obtains, by learning, an operation that compresses an image to be inspected into a feature amount of less information. The feature amount learning unit 42 performs unsupervised learning based on a self-coder as learning of a neural network, and realizes the learning of the feature amount by the convolution layer described above. The details of the feature amount learning unit 42 will be described later.

  The classifier learning unit 43 generates, by learning, a classifier that determines the quality of the inspection target based on the feature amount of the learning image output from the feature amount learning unit 42. In the present embodiment, the classifier learning unit 43 performs unsupervised learning to learn a classifier. The classifier generated by learning is used by the classification unit 52 of the determination unit 5 described later. The details of the classifier learning unit 43 and the classifier will be described later.

  The determination unit 5 includes a feature amount calculation unit 51, an identification unit 52, and an output unit 53. The determination unit 5 determines the quality of the inspection object based on the determination image sent from the image acquisition unit 3.

The feature amount calculation unit 51 inputs the determination image to the learned neural network, and outputs the feature amount of the determination image. The details of the feature amount calculation unit 51 will be described later.
The identification unit 52 inputs the feature amount of the determination image output from the feature amount calculation unit 51 to the classifier generated by the classifier learning unit 43, and determines the quality of the inspection target. The details of the identification unit 52 will be described later.

  The output unit 53 displays the determination result by the identification unit 52 on a display screen, for example, or sends the result to a manufacturing apparatus of a product to be inspected. The determination result output by the output unit 53 is used for sorting out defective products.

[Learning using self-coder by feature amount learning unit]
The feature amount learning unit 42 included in the learning unit 4 aims to obtain, by learning, a convolutional neural network operation that compresses an image to be inspected into a feature amount of less information. This feature amount is not necessarily any information, and should be useful information when determining the quality of an inspection object in image inspection.

  In order to realize the feature quantity learning unit 42 as described above, in the present embodiment, a self-coder having a convolutional neural network as shown in FIG. 3 is used. The convolutional neural network constructed by the feature amount learning unit 42 is used by the feature amount calculation unit 51 to calculate the feature amounts of the determination image. More specifically, a feature quantity extractor shown in a portion surrounded by a broken line in FIG. 3 is constructed and used in the feature quantity calculation unit 51.

  The self-coder has a structure that converts an input image into a latent variable having a smaller number of dimensions compared to the dimensionality of the input image, and restores from that into an image of the same size as the input image. In the self-coder, when learning a neural network as shown in FIG. 3, the difference between the input image and the output image is made as small as possible. If learning is successful and a self-coder can be generated that produces an image close to the original image, it means that the image can be represented by latent variables with the number of dimensions smaller than the number of dimensions of the original image.

  Such latent variables can be powerful feature quantities in image inspection. And, what is important for the present invention is that it is not necessary to distinguish whether the image to be inspected is a non-defective image or a non-defective image when performing this learning. This is because the self-coder learns so that the input image and the output image are as equal as possible, and does not learn whether the image is good or not. . This leads to the automatic learning of the feature amount of the image to be inspected, even when there are no defective products to be inspected, which is an advantage of the present invention.

  Also, as shown in FIG. 4, it is possible to add a mechanism of a hostile generation network to the self-coder. The hostile generation network is a neural network that generates an image called a generator (hereinafter referred to as "Generator") and a classifier called a discriminator (hereinafter referred to as "Discriminator"). And a neural network that determines whether the image is a generated image.

  In FIG. 4, a self-coder having a convolutional neural network shown in FIG. 3 is used as a generator. As shown in FIG. 4, the Discriminator learns to make the determination as to whether it is a real image as incorrect as possible. On the other hand, the Generator learns that the Discriminator misjudges from the real image. The learning progresses well by making the two neural networks compete against each other in a hostile manner, and if the accuracy of the image generated by the Generator is improved, the part serving as the feature extractor in the Generator will also become an excellent feature extractor. It is expected.

  In addition, the variational self-coder is also a technology with a similar mechanism, and can be used in a similar manner. It should be noted that neither the hostile generation network nor the variational self-coder need to distinguish between good and bad in the data used for learning.

[Learning by classifier learning unit and generation of classifiers]
If there are no defective images to be inspected and only a sufficient number of non-defective images can be secured, a classifier can be generated by a method called unsupervised learning. For example, if a distribution of feature quantities in a non-defective image is estimated using a mixed normal distribution model and a statistical test is used, it is possible to generate a discriminator in which one out of the distribution is an image of a defective product.

  Also, by using the subspace method, it is possible to learn the distribution of feature quantities in a non-defective image, and to generate a discriminator in which one out of the distribution is regarded as a defective product. There is also a means for generating pseudo-defective defective data at a position sufficiently far from the images of all non-defective products such as one-class SVM (One-Class SVM) to generate a discriminator.

  On the other hand, there is a means for generating a discriminator by unsupervised learning even in the case where a defective product image to be inspected can be secured but a label indicating good or bad is not attached to a learning image. For example, as in an isolation forest, there is a classifier that classifies learning images into a large number of groups, and sets those belonging to a group consisting of a small number of images as defective products. In addition, there is a classifier that evaluates the distance between learning images as in the LOF, and makes a product having a large distance to another learning image a defective product.

[Function block of feature amount learning unit and feature amount calculating unit]
FIG. 5 is a functional block diagram of the feature amount learning unit 42 and the feature amount calculating unit 51. As described above, the feature quantity learning unit 42 constructs a self-coder having a convolutional neural network by learning so as to compress an image to be examined into feature quantities of less information. The feature quantity calculation unit 51 calculates the feature quantity of the determination image using the trained convolutional neural network constructed by the feature quantity learning unit 42.

  As shown in FIG. 5, the feature amount learning unit 42 includes an input image storage unit 421, a pre-processing unit 422, a convolutional neural network operation unit (hereinafter referred to as "convolution NN operation unit") 423, a feature amount storage unit 424, Deconvoluted neural network operation unit (hereinafter referred to as "deconvoluted NN operation unit") 425, output image storage unit 426, feature value calculation setting storage unit 427, image difference calculation unit 428, feature value calculation setting update unit 429, and A feature amount output unit 430 is provided.

  The feature amount calculation unit 51 includes an input image storage unit 511, a preprocessing unit 512, a convolutional NN operation unit 513, a feature amount calculation setting storage unit 514, and a feature amount output unit 515.

First, functional blocks included in the feature amount learning unit 42 will be described.
The input image storage unit 421 stores, as input images, a large number of learning images necessary for learning performed by the feature amount learning unit 42.

  The preprocessing unit 422 adjusts the learning image stored in the input image storage unit 421. Although the preprocessing unit 422 may be omitted, in practice, it is preferable to provide the preprocessing unit 422 and perform preprocessing of the image, such as normalization of the luminance of the image. In addition, about the pre-processing of the image by the pre-processing part 422, the well-known technique generally known may be used.

  The convolutional NN calculator 423 corresponds to the feature quantity extractor shown in FIGS. 3 and 4. The convolutional NN calculation unit 423 receives the learning image (input image) as an input, and outputs the feature amount of the learning image (input image). More specifically, the convolutional NN operation unit 423 performs an operation centered on the convolution operation on the input image using the values of the parameters stored in the feature value calculation setting storage unit 427 described later, and the input image Output the feature quantity of. This operation is usually composed of multiple stages. When one stage (one) of operation is performed, data smaller than the number of pixels of the input image or data is output.

  The convolutional NN operation unit 423 converts the original input image into a latent variable indicating the number of pixels smaller than the number of pixels by repeating such convolution operation a plurality of times. This latent variable is obtained as a feature that quantifies features of the image to be examined.

  Further, the convolutional NN calculating unit 423 calculates feature amounts of all the learning images stored in the image storage unit 41 using the values of the parameters updated by the feature amount calculation setting updating unit 429 described later.

  In the operation and design of the convolutional neural network by the convolutional NN operation unit 423, a known technique described in Patent Documents 2 and 3 may be used. Also, a plurality of software frameworks used for the operation and design of known convolutional neural networks are known, and these may be used.

  Non-Patent Document 4 describes that the pooling (thinning) used in a general convolutional neural network is improved in performance by replacing it with a strided convolution. Therefore, it is preferable that the stride convolution be performed also in the convolutional NN operation unit 423. The feature quantities of the input image calculated by the convolutional NN calculation unit 423 are temporarily stored in a feature quantity storage unit 424 described later.

  The feature amount storage unit 424 temporarily stores the feature amounts of the input image output from the convolutional NN operation unit 423. Also, the feature amount of the input image stored in the feature amount storage unit 424 is used by the deconvoluted NN operation unit 425 described later.

  The deconvoluted NN operation unit 425 receives the feature amount of the input image output from the convolutional NN operation unit 423 and outputs an image having the same size as the input image. More specifically, the deconvoluted NN operation unit 425 uses the values of the neural network parameters stored in the feature amount calculation setting storage unit 427 to perform the inverse operation of the convolutional neural network operation by the convolution NN operation unit 423. Do. The deconvoluted NN operation unit 425 performs the deconvolution neural network operation a plurality of times, and performs the operation until the feature amount of the image becomes an image of the same size as the image stored in the input image storage unit 421.

  The deconvoluted neural network constituting the deconvoluted NN operation unit 425 may use a network structure in which the convolutional neural network constituting the convolutional NN operation unit 423 is transposed. In addition, for the deconvoluted neural network operation by the deconvoluted NN operation unit 423, for example, a known operation method described in Non-Patent Document 2 or a known software framework described above may be used. The operation result of the last stage of the deconvoluted neural network by the deconvoluted NN operation unit 423 is stored in an output image storage unit 246 described later.

  The output image storage unit 426 stores the calculation result output from the deconvoluted NN calculation unit 425 as an output image.

  The feature value calculation setting storage unit 427 stores values of parameters of the convolutional neural network and the deconvoluted neural network used by the convolutional NN operation unit 423 and the deconvoluted NN operation unit 425. More specifically, the feature amount calculation setting storage unit 427 stores the value of the parameter that has been initialized and the value of the parameter updated by the feature amount calculation setting updating unit 429 described later.

  The image difference calculation unit 428 calculates the difference between the input image (the learning image) stored in the input image storage unit 421 and the output image stored in the output image storage unit 426 corresponding to the input image. Do.

  The feature quantity calculation setting update unit 429 is a parameter of the parameters of the convolutional neural network used by the convolutional NN operation unit 423 and the deconvoluted neural network used by the deconvoluted NN operation unit 425 stored in the feature amount calculation setting storage unit 427. Update the value When using a hostile generation network as the network, the values of the parameters of the neural network are updated based on the determination result of the Discriminator.

  As a method of updating parameter values in the neural network, a known method such as back propagation may be used. Also, updating of the parameters of the neural network by the feature value calculation setting updating unit 429 may be performed each time one input image to be learned is input, or may be performed after the number of sheets collected to some extent is accumulated. .

  The feature amount output unit 430 outputs the calculation result by the convolutional neural network calculation output from the convolutional NN calculation unit 423. More specifically, the feature quantity output unit 430 outputs feature quantities corresponding to all the learning images calculated by the convolutional NN calculation unit 423 using the values of the updated parameters described later.

  Next, each functional block included in the feature amount calculation unit 51 will be described. The feature amount calculation unit 51 calculates the feature amount of the determination image acquired by the image acquisition unit 3.

  The input image storage unit 511 stores the determination image captured by the imaging unit 2 and acquired by the image acquisition unit 3.

  The preprocessing unit 512 has the same function as the preprocessing unit 422 included in the feature amount learning unit 42, and performs preprocessing of the determination image stored in the input image storage unit 511 as necessary. The pre-processing unit 512 pre-processes the determination image and processes it so that the quality determination can be easily performed. Although the preprocessing unit 512 may be omitted as in the preprocessing unit 422 of the feature amount learning unit 42, provision of the preprocessing unit 512 improves the determination accuracy of the feature amount calculation unit 51. There are times when you can.

  The convolutional NN operation unit 513 has the same configuration as the convolutional NN operation unit 423 which the feature amount learning unit 42 has. Further, the convolutional NN operation unit 513 receives the determination image according to the updated value of the parameter of the convolutional neural network stored in the feature amount calculation setting storage unit 514, and outputs the feature amount of the determination image. The convolutional NN operation unit 513 performs the same operation as the convolutional NN operation unit 423 when the learning process of the feature amount learning unit 42 is completed.

  The feature amount calculation setting storage unit 514 stores values of parameters of the convolutional neural network in the convolutional NN operation unit 423 whose values have been updated by the feature amount learning unit 42. More specifically, the updated value of the parameter of the convolutional neural network is transferred from the feature value calculation setting storage unit 427.

  The feature amount output unit 515 outputs the feature amount of the determination image as a result of the convolutional neural network operation by the convolutional NN operation unit 513. The feature amount of the determination image output from the feature amount output unit 515 is input to the identification unit 52.

[Function block of classifier learning unit and classifier]
FIG. 6 is a functional block diagram of the classifier learning unit 43 and the discrimination unit 52. The classifier learning unit 43 includes a feature amount input unit 431, a dimension reduction unit 432, and a classification operation unsupervised learning unit 433. The identification unit 52 further includes a feature amount input unit 521, a dimension reduction unit 522, and an identification operation unit 523. As described above, the classifier generated by the classifier learning unit 43 is used by the classification unit 52 to determine whether the inspection target is good or bad.

First, each functional block included in the classifier learning unit 43 will be described.
The feature amount of the learning image output from the feature amount output unit 430 of the feature amount learning unit 42 is input to the feature amount input unit 431. More specifically, the feature amount input unit 431 collectively receives the feature amounts calculated for the learning image stored in the image storage unit 41 after the learning process of the feature amount learning unit 42 is completed.

  The dimension reducing unit 432 performs an operation to reduce the dimension of the feature amount of the image while preventing the information amount of the feature amount in the learning image input to the feature amount input unit 431 as much as possible. The dimension reduction unit 432 compresses the dimensions of the feature amount of the image into fewer dimensions using, for example, a known method such as principal component analysis.

  The reason for providing the dimension reduction unit 432 is as follows. Usually, there are a plurality of feature amounts of the learning image input to the feature amount input unit 431 for one image. Therefore, when the number of feature amounts of the input image is large, there are cases where the identification unit 52 can not generate the discriminator as it is, or the amount of calculation required for the generation may be extremely large.

  Therefore, by providing the dimension reduction unit 432, it is possible to avoid a so-called “dimension curse” problem in which the calculation cost increases exponentially with the increase of the dimensionality of the mathematical space. If the dimension of the feature amount of the image output from the convolutional NN calculating unit 423 of the feature amount learning unit 42 is sufficiently low, the dimension reducing unit 432 may be omitted.

  The discrimination operation unsupervised learning unit 433 receives the feature amounts of the learning image input through the feature amount input unit 431 and the dimension reducing unit 432 as input, and determines the quality of the inspection object by a known unsupervised learning method. Generate a classifier. The classification operation generated by the classification operation unsupervised learning unit 433 is used by a classification operation unit 523 which will be described later, which is included in the identification unit 52 of the determination unit 5.

  The discrimination calculation unsupervised learning unit 433 may use, for example, a method such as isolation forest, One-Class SVM, subspace method, LOF, or the like as a known unsupervised learning method. Note that as an example of the isolation forest, the method described in Non-Patent Document 5 may be used. As an example of the subspace method, the method described in Non-Patent Document 6 may be used. As an example of LOF, the method described in Non-Patent Document 7 may be used.

Next, each functional block of the identification unit 52 will be described.
The feature amount input unit 521 has the same function and configuration as the feature amount input unit 431 that the classifier learning unit 43 has. That is, the feature amount of the determination image output from the feature amount output unit 515 of the feature amount calculation unit 51 is input to the feature amount input unit 521.

  The dimension reduction unit 522 has the same function and configuration as the dimension reduction unit 432 that the classifier learning unit 43 has. Note that it is necessary to use completely the same calculation method in the dimension reduction unit 432 of the classifier learning unit 43 and the dimension reduction unit 522 of the identification unit 52.

  The discrimination operation unit 523 receives the feature amount of the determination image input through the feature amount input unit 521 and the dimension reduction unit 522 as an input, and generates the discrimination generated by the discrimination operation unsupervised learning unit 433 of the classifier learning unit 43. The quality of the inspection object is determined using the The determination result by the identification operation unit 523 is output from the output unit 53.

[Hardware configuration of the image inspection apparatus]
FIG. 7 is a block diagram showing the hardware configuration of the image inspection apparatus 1 according to the present embodiment. Image inspection apparatus 100 includes a computer including control unit 102, communication control device 105, imaging device 106, storage device 107, and display device 108 connected via bus 101, and a program for controlling these hardware resources. It can be realized.

  The control unit 102 includes a CPU 103 and a main storage unit 104. In the main storage unit 104, programs for the CPU 103 to perform various controls and calculations are stored in advance. The control unit 102 implements the functions of the image inspection apparatus 1 such as the learning unit 4 and the determination unit 5 shown in FIG.

The communication control device 105 is a control device for making a network connection between the image inspection apparatus 100 and various external electronic devices.
The imaging device 106 may convert the light signal to an image signal to generate a still image. More specifically, the imaging device 106 includes an imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS image sensor, and forms an image of light incident from an imaging area on a light receiving surface. , Convert to electrical signals.

  The storage device 107 includes a readable and writable storage medium, and a drive device for reading and writing various information such as programs and data from and to the storage medium. For the storage device 107, a semiconductor memory such as a flash memory or a hard disk can be used as a storage medium. The storage device 107 is an image storage unit 107a, an input image storage unit 107b, a feature amount calculation setting storage unit 107c, a feature amount storage unit 107d, an output image storage unit 107e, a program storage unit 107f, and other storage devices not shown. A storage device for backing up programs, data, and the like stored in the storage device 107 can be included.

The image storage unit 107 a functions as the image storage unit 41 included in the learning unit 4 and stores the learning image.
The input image storage unit 107 b functions as an input image storage unit 421 of the feature amount learning unit 42 and an input image storage unit 511 of the feature amount calculation unit 51.

The feature amount calculation setting storage unit 107c functions as the feature amount calculation setting storage unit 427 of the feature amount learning unit 42, and stores the value of the neural network parameter.
The feature amount storage unit 107d functions as the feature amount storage unit 424 of the feature amount learning unit 42, and stores the feature amounts of the learning image (input image) calculated by the feature amount learning unit 42.

The output image storage unit 107e functions as an output image storage unit 426 of the feature amount learning unit 42, and stores an output image of the calculation result of the deconvoluted NN calculation unit 425.
The program storage unit 107f stores various programs for executing processes necessary for image inspection, such as learning processing by the learning unit 4 in the present embodiment and determination processing by the determination unit 5.

  The display device 108 functions as the output unit 53 of the image inspection apparatus 1. The display device 108 is realized by a liquid crystal display or the like.

[Operation of image processing apparatus]
Next, the operation of the image inspection apparatus 1 according to the present embodiment will be described. In the following, the learning process by the learning unit 4 and the determination process by the determination unit 5 will be separately described. In the learning process, the feature quantity learning process by the feature quantity learning unit 42 and the classifier unsupervised learning process by the classifier learning unit 43 will be separately described. In the determination process, the feature quantity calculation process by the feature quantity calculation unit 51 and the determination output process by the identification unit 52 will be separately described.

[Feature amount learning process]
First, feature amount learning processing by the feature amount learning unit 42 will be described with reference to the flowchart in FIG. First, a learning image including an examination target is captured by the imaging unit 2 (step S100). The image acquisition unit 3 acquires a captured image. The acquired learning image is stored in the image storage unit 41. The input image storage unit 421 reads out the learning image acquired by the image acquisition unit 3 from the image storage unit 41 and stores it (step S101).

  Next, with the learning image stored in the input image storage unit 421 as an input image, the convolutional NN calculation unit 423 performs convolutional neural network calculation a plurality of times, and outputs the feature amount of the input image (step S102). More specifically, the convolutional NN operation unit 423 performs an operation using the values of the parameters stored in the feature value calculation setting storage unit 427. Note that the preprocessing unit 422 may perform preprocessing such as normalization of luminance in advance for an input image (learning image) input to the convolutional NN operation unit 423.

  The feature amount of the input image output from the convolutional NN operation unit 423 is stored in the feature amount storage unit 424 (step S103). Next, the deconvoluted NN operation unit 425 executes deconvolution neural network operation with the feature amount of the input image stored in the feature amount storage unit 424 as an input (step S104).

  More specifically, the deconvoluted NN operation unit 425 performs the inverse operation to the convolutional neural network operation by the convolutional NN operation unit 423 multiple times using the values of the parameters stored in the feature value calculation setting storage unit 427. .

  In addition, the deconvoluted NN operation unit 425 performs the operation until the calculated feature amount becomes an image of the same size as the input image stored in the input image storage unit 421. The image restored by the deconvoluted NN calculation unit 425 is stored in the output image storage unit 426 as an output image.

  Next, the image difference calculation unit 428 calculates the difference between the input image stored in the input image storage unit 421 and the output image stored in the output image storage unit 426 corresponding to the input image (step S105).

  Next, when the difference between the input image and the output image calculated by the image difference calculating unit 428 is not sufficiently small (step S106: NO), the feature amount calculation setting updating unit 429 stores the feature amount calculation setting storage. The values of the parameters of the convolutional neural network stored in the unit 427 are updated (step S107). The feature value calculation setting update unit 429 may update the values of the parameters of the convolutional neural network using, for example, a known method such as back propagation.

  The updated parameter values of the convolutional neural network are overwritten and stored in the feature value calculation setting storage unit 427. Thereafter, the convolutional NN operation unit 423 performs the convolutional neural network operation again using the updated parameter value (step S102). Thereafter, the calculation result is stored as the feature amount of the input image (learning image) (step S103). Then, the deconvoluted NN operation unit 425 performs deconvoluted neural network operation, restores the image to obtain an output image, and stores it in the output image storage unit 426 (step S104).

  Next, the image difference calculation unit 428 calculates again the difference between the input image and the output image (step S105), and if the difference is sufficiently small (step S106: YES), the convolution NN calculation unit 423 immediately The value of the set parameter is transferred from the feature amount calculation setting storage unit 427 of the feature amount learning unit 42 to the feature amount calculation setting storage unit 514 of the feature amount calculation unit 51 (step S108).

  The feature amount learning unit 42 repeatedly executes the processing from step S102 to step S107 several times. The convolutional NN operation unit 423 when the difference between the input image and the output image is sufficiently small functions as the feature quantity extractor (learned convolutional neural network) described in FIGS. 3 and 4.

  In the present embodiment, the feature amount learning unit 42 performs the process from step S102 to step S107 described above each time one image is input, but as described in [0078] It is also possible to perform these processes each time an image is accumulated to some extent.

  Next, the convolutional NN operation unit 423 calculates feature amounts for all the learning images stored in the image storage unit 41, and outputs the feature amounts to the feature amount output unit 430 (step S109).

  As described above, the feature quantity learning unit 42 learns the convolutional neural network using the learning image as the input image, updates the parameter values, and generates a feature quantity extractor (learned convolutional neural network) Do. Then, the feature quantity learning unit 42 calculates the feature quantity of the learning image using the feature quantity extractor generated by learning.

[Classifier unsupervised learning process]
Next, unsupervised learning processing of the classifier by the classifier learning unit 43 will be described with reference to the flowchart in FIG. First, the feature amount input unit 431 receives the feature amount of the learning image transmitted from the feature amount output unit 430 of the feature amount learning unit 42 (step S110).

  Next, the dimension reducing unit 432 performs an operation to reduce the dimension while preventing the information amount of the feature amount of the received image from being reduced (step S111). The dimension reduction unit 432 compresses the received image into feature quantities with fewer dimensions using, for example, a known method such as principal component analysis.

  Next, the discrimination operation unsupervised learning unit 433 performs unsupervised learning based on the feature amount of the learning image which is dimension-reduced by the dimension reducing unit 432 (step S112), and generates a classifier (step S113). ). More specifically, the discrimination operation unsupervised learning unit 433 uses, for example, a known unsupervised learning method such as isolation forest, One-Class SVM, subspace method, LOF, and the inspection object is within the non-defective range. Generate a classifier to determine the

The classifier generated by learning by the classification operation unsupervised learning unit 433 is used by the classification operation unit 523 of the discrimination unit 52 of the determination unit 5.
As described above, the classifier learning unit 43 performs unsupervised learning based on the feature amount of the learning image, and generates a classifier.

[Feature amount calculation processing]
Next, feature amount calculation processing by the feature amount calculation unit 51 included in the determination unit 5 will be described with reference to the flowchart in FIG. First, the determination image received from the image acquisition unit 3 is stored in the input image storage unit 511 (step S114).

  Next, the convolutional NN operation unit 513 reads the values of the parameters of the learned convolutional neural network stored in the feature amount calculation setting storage unit 514 (step S115). Thereafter, the convolutional NN operation unit 513 performs convolutional neural network operation using the read parameter value and the determination image stored in the input image storage unit 511 as an input (step S116). The convolutional NN operation unit 513 has the same configuration as the convolutional NN operation unit 423 of the learning unit 4.

  Next, the calculation result by the convolutional NN calculation unit 513 is output from the feature amount output unit 515 as the feature amount of the determination image (step S117). The feature amount of the determination image output from the feature amount output unit 515 is input to the identification unit 52.

  As described above, the feature quantity calculation unit 51 calculates the feature quantity of the determination image using the values of the parameters of the learned convolutional neural network obtained by the feature quantity learning processing by the feature quantity learning unit 42. .

Judgment output processing
Next, determination output processing by the identification unit 52 and the output unit 53 that the determination unit 5 has will be described with reference to the flowchart in FIG. First, the feature amount of the determination image output from the feature amount output unit 515 of the feature amount calculation unit 51 of the determination unit 5 is input to the feature amount input unit 521 of the identification unit 52 (step S118).

  Next, the dimension reduction unit 522 performs dimension compression of the input feature amount (step S119). After that, the discrimination operation unit 523 is generated by learning processing by the discrimination operation unsupervised learning unit 433 of the discriminator learning unit 43 with the feature amount in the dimension-compressed image for judgment output from the dimension reduction unit 522 as an input. It is determined whether the inspection object is good or bad using a classifier (step S120).

  Then, the determination result by the identification operation unit 523 is output from the output unit 53 (step S121). More specifically, the output unit 53 displays the determination result on a display screen such as a liquid crystal display, for example, or transmits the determination result to an apparatus that manufactures a product to be inspected. As described above, when the determination result on the quality of the inspection object is output from the output unit 53, it is used for sorting out defective products.

  As described above, the identification unit 52 determines the quality of the inspection target using the classifier generated by the classifier learning unit 43 in the learning process. Then, the output unit 53 outputs the determination result by the identification unit 52.

  As described above, according to the first embodiment, the image inspection apparatus 1 generates the unsupervised learning of the convolutional layer that extracts the feature amount of the image and the classifier that determines the quality of the inspection target. And unsupervised learning separately. Thus, the image inspection apparatus can automatically learn the feature amount of the image to be inspected even if there is no defective product image to be inspected or if the quality is not distinguished in the learning image. can get.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the following description, the same components as those in the first embodiment described above are denoted by the same reference numerals, and the description thereof is omitted.

  In the first embodiment, the classifier learning unit 43 learns the classifier using a known unsupervised learning method such as isolation forest, One-Class SVM, subspace method, LOF, etc. The case of generating a classifier that determines. The second embodiment is common to the first embodiment in that a classifier is generated by unsupervised learning.

  However, in the second embodiment, the image inspection apparatus 1A is different from the first embodiment in that the learning unit 4A further includes a classifier adjusting unit 44. Further, in the second embodiment, it is assumed that a very small amount of defective images to be inspected for learning exist and they are labeled as being distinguished from non-defective products.

Discriminator adjustment parameters
The above-described various classifiers described in the first embodiment usually have parameters (hereinafter referred to as “adjustment parameters”) related to the determination of the quality of the inspection object. For example, in the method using a statistical test, changing the significance level changes the boundary between good and bad products. The classifier according to the method described in the first embodiment also has such adjustment parameters. As shown by the arrows in FIG. 13, the boundary between the non-defective product and the non-defective product can be adjusted by adjusting the adjustment parameter of the identifier. In FIG. 13, “white circle” indicates the feature amount of the non-defective item image to be inspected. Also, “cross” indicates the feature amount of the defective image to be inspected.

  Such adjustment of adjustment parameters is important in unsupervised learning. If the boundary is set closer to the non-defective side as shown by the broken line on the right side of FIG. 13, the fact that an inspection object that is actually a non-defective product is misjudged as a non-defective item will be missed (hereinafter referred to as "missing"). However, there is an increase in the determination of an inspection object which is actually a non-defective item as an inferior product as excessive (hereinafter referred to as “excess determination”). On the other hand, if the boundary is set closer to the defective product side, which is shown by the broken line on the left side of FIG. 13, such an overdetermination is reduced, but the risk of missing a real defective product increases.

  Although it is important to adjust the adjustment parameter of the discriminator to an optimal value, the adjustment may be difficult. However, in the case where even a very small number of images surely known as defectives exist, adjustment of adjustment parameters becomes easier as compared with the case where there is no defective image at all. For example, in the case of the distribution state of the feature amount as shown in FIG. 13, if the boundary between the non-defective product and the non-defective product is set at the position indicated by the solid line in the middle, the defective product is missed for at least the image data collected so far It can be seen that the overdetermination can be suppressed to a small amount.

[Function block of image inspection device]
Next, functional blocks of the image inspection apparatus 1A according to the second embodiment will be described with reference to FIG. Hereinafter, the discriminator learning unit 43 and the discriminator adjustment unit 44 which are components different from those of the first embodiment will be mainly described.

  The classifier learning unit 43 is configured to receive from the classifier adjusting unit 44 adjustment parameters related to the accuracy of the quality determination in the classifier generated by performing learning. The classifier learning unit 43 generates a classifier using the value of the adjustment parameter supplied from the classifier adjustment unit 44.

  The discriminator adjustment unit 44 adjusts the adjustment parameter of the discriminator to achieve a balance between the risk of missing a defective product to be inspected and the possibility of excessively determining a non-defective product as a defective product. For example, the discriminator adjustment unit 44 may adjust so that the ratio of determining a non-defective product as an over-defective product becomes equal to or less than a certain threshold value while avoiding missing of defective products as much as possible. The value of the adjustment parameter of the classifier adjusted by the classifier adjustment unit 44 is input to the classifier learning unit 43.

[Parameter adjustment process]
Next, parameter adjustment processing for adjusting the adjustment parameter of the classifier will be described with reference to the flowchart of FIG. First, the discriminator adjustment unit 44 sets the adjustment parameter of the discriminator to a predetermined value (step S200). The classifier adjustment unit 44 inputs the set adjustment parameter value to the classifier learning unit 43. Then, the classification operation unsupervised learning unit 433 adjusts the classifier with the value of the set adjustment parameter.

  Next, the feature amount of the learning image calculated by the feature amount learning unit 42 (convoluted NN operation unit 423) is input to the feature amount input unit 431 of the classifier learning unit 43 (step S200). Note that, as necessary, the dimension reducing unit 432 performs dimension compression of the feature amount of the learning image.

  Next, the discrimination operation unsupervised learning unit 433 of the discriminator learning unit 43 inputs the feature amount of the learning image to the discriminator whose adjustment parameter has been adjusted, and determines the quality of the inspection target (step S202). ). After that, the discrimination operation unsupervised learning unit 433 collates the discrimination result between the good and bad of the inspection object with the distinction between the actual non-defective product and the defective product, and determines that the inspection object which is actually defective product is the non-defective product (missed) The ratio between the presence or absence of and the case where it is determined that the inspection object that is actually a non-defective product is a non-defective product (excess determination) is calculated (step S203).

  The discrimination calculation unsupervised learning unit 433 is used when there is no missing (step S204: YES) and when the excess determination ratio is equal to or less than the threshold (step S205: YES) and it is determined that it is within the acceptance range. The value of the adjustment parameter is stored (step S206). Thereafter, the identification unit 52 of the determination unit 5 uses the identifier set by the value of the adjustment parameter after adjustment.

  On the other hand, in the case where there is missed (step S204: NO) and the excess determination ratio is less than or equal to the threshold (step S207: YES), the discriminator adjustment unit 44 is good as described in FIG. The value of the adjustment parameter is changed in the direction in which the ratio determined to be smaller decreases (step S208). Thereafter, the processing from step S201 to step S203 is executed again, and the change of the adjustment parameter is repeated until there is no missing (step S204: YES) and the excess determination ratio becomes equal to or less than the threshold (step S205: YES). .

  In addition, when there is no miss (step S204: YES) and the excess determination ratio is equal to or more than the threshold (step S205: NO), the discriminator adjustment unit 44, as described in FIG. The value of the adjustment parameter is changed in the direction in which the proportion determined to increase is increased (step S209). After that, the processing from step S201 to step S203 is executed again, and there is no missing (step S204: YES), and the change of the adjustment parameter value is made until the ratio of the excess determination becomes less than the threshold (step S205: YES). repeat.

  Also, if there is a miss (step S204: NO) and the excess determination ratio is equal to or more than the threshold (step S207: NO), the miss is missed regardless of how the adjustment parameter is adjusted. It means that it is difficult to obtain a target value such that the excess judgment ratio is less than or equal to the threshold.

  In this case, if there is another adjustment parameter in the discriminator, the other adjustment parameter may be used to perform the parameter adjustment process again. In addition, even if another adjustment parameter is adjusted by the discriminator adjustment unit 44, if the quality determination by the discriminator does not improve, the feature amount learning unit 42 performs learning of the feature amount in the image to be inspected again. You may consider.

  As described above, the discriminator obtained by adjusting the adjustment parameter by the discriminator adjustment unit 44 is used by the discrimination calculation unit 523 of the discrimination unit 52 included in the determination unit 5. Then, the discrimination calculation unit 523 determines the quality of the inspection object using the classifier in which the value of the adjustment parameter is adjusted.

  As described above, according to the second embodiment, the discriminator adjustment unit 44 adjusts the adjustment parameter in the discriminator, and there is a risk that the defective product in the inspection object may be missed, and the non-defective product is determined to be defective. Balance with fear. Thereby, adjustment of image inspection device 1A can be performed more easily. In addition, it is possible to further improve the accuracy of the quality determination of the image inspection apparatus 1A.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the following description, the same components as those in the first and second embodiments described above are designated by the same reference numerals, and the description thereof will be omitted.

  In the first and second embodiments, the case where unsupervised learning is used for learning of the classifier has been described. On the other hand, in the third embodiment, the classifier learning unit 43A includes the classification operation supervised learning unit 435, and generates the classifier by supervised learning in the first and second embodiments. It is different from In the third embodiment, as in the second embodiment, a very small amount of defective images to be inspected for learning exist, and they are labeled and distinguished from non-defective images. Is assumed.

[Generation of classifiers by supervised learning]
If the images of a few defective products with a label indicating the quality of the image to be inspected and the images of non-defective products have been secured, a small amount of learning is possible through supervised learning such as SVM, random forest, or boosting. Robust classifiers can be generated from image samples for

  For example, the SVM uses only a minimal amount of data involved in the identification and produces a discriminator that preferably produces a margin around the identification boundary. For this reason, stable identification results can be obtained even for unknown data.

  A random forest is a classifier that combines a large number of child classifiers. However, since a decision tree is used as these child classifiers, a small number of samples can be correctly identified without reduction.

  Although boosting is also a classifier that combines a large number of child classifiers, it also has a feature that weights can be given to each image for learning, so by giving a large weight to images of small quantities of defective products, small quantities of defective products It can be expected to perform stable identification also for the image of.

[Function block of classifier learning unit and classifier]
FIG. 15 is a functional block diagram of the classifier learning unit 43A and the identification unit 52.
The classifier learning unit 43A includes a feature amount input unit 431, a dimension reduction unit 432, a quality information input unit 434, and a classification operation supervised learning unit 435.

  The quality information input unit 434 inputs information indicating the quality of the image to be inspected corresponding to the feature amount of the learning image input to the feature amount input unit 431.

  The identification operation supervised learning section 435 is known by using as input the feature amount of the learning image and the information indicating whether the learning image that is the source of the feature amount is an image indicating a non-defective item or an image indicating a non-defective item The classifier is generated by the supervised learning method of

  As a known supervised learning method used by the identification operation supervised learning unit 435, for example, SVM described in Non-Patent Document 8, Random Forest described in Non-Patent Document 9, Non-Patent Document 10 There are boosting and so on.

  The classifier generated by the classification operation supervised learning unit 435 is used by the classification operation unit 523 of the classification unit 52. Then, using the feature amount of the determination image as an input, the identification operation unit 523 determines the quality of the inspection object using the classifier generated by supervised learning.

[Classifier supervised learning process]
Next, the classifier supervised learning process will be described with reference to the flowchart of FIG. First, the feature quantity input unit 431 of the discriminator learning unit 43A performs the feature quantity output unit 430 on the feature quantity in the image of the inspection target for learning obtained by the calculation result by the convolutional NN calculation unit 423 of the feature quantity learning unit 42. From (step S300).

  Next, the dimension reducing unit 432 performs dimension compression of the feature amount of the learning image received by the feature amount input unit 431 (step S301). After that, the discrimination operation supervised learning unit 435 receives, as input, the feature amount of the learning image subjected to dimension compression and the information indicating the quality of the corresponding learning image input through the quality information input unit 434. The supervised learning is performed to learn the classifier (step S302).

  The classification operation supervised learning unit 435 learns the classifier using, for example, a known supervised learning method such as SVM. Then, the classification operation supervised learning unit 435 generates a classifier that determines the quality of the inspection target (step S303).

  The generated discriminator is used by the discrimination calculation unit 523 of the discrimination unit 52 of the determination unit 5. The discrimination operation unit 523 determines the quality of the inspection object using the feature amount of the determination image as an input, using the classifier generated by supervised learning. Then, the output unit 53 displays the determination result on a display screen such as a liquid crystal display.

  As described above, according to the third embodiment, since supervised learning is performed to generate a classifier, even if there are only a few defective images to be inspected, the quality of the inspection object can be determined It is possible to learn and generate classifiers that can be determined more correctly.

  Although the embodiments of the image inspection apparatus and the image inspection method of the present invention have been described above, the present invention is not limited to the described embodiments, and those skilled in the art can assume the scope of the invention described in the claims. It is possible to make various possible modifications.

  For example, in the embodiment described above, the case where the imaging unit 2 and the image acquisition unit 3 are shared by the learning unit 4 and the determination unit 5 has been described, but the learning unit 4 and the determination unit 5 are independent of each other The imaging unit 2 and the image acquisition unit 3 may be provided.

  In the embodiment described above, the learning unit 4 and the determination unit 5 are implemented on the same computer, but the learning unit 4 and the determination unit 5 are implemented on separate computers. It is also good.

  1, 1A, 100 ... image inspection device, 2 ... imaging unit, 3 ... image acquisition unit, 4 ... learning unit, 5 ... determination unit, 41 ... image storage unit, 42 ... feature amount learning unit, 43 ... classifier learning unit , 51: feature amount calculation unit, 52: identification unit, 53: output unit, 101: bus, 102: control unit, 103: CPU, 104: main storage unit, 105: communication control device, 106: imaging device, 107: 107 Storage device, 108 ... display device.

Claims (10)

An image inspection apparatus for inspecting the quality of an inspection object by an image,
A feature amount learning unit that constructs a learned neural network that performs learning of a neural network based on a learning image including the inspection target and outputs a feature that can restore the learning image;
A discriminator learning unit that generates a discriminator that determines the quality of the inspection object by learning based on the feature amount of the learning image output by the learned neural network after the learning of the feature amount learning unit is completed. When,
A feature amount calculation unit which inputs a determination image including the inspection target to the learned neural network and outputs a feature amount of the determination image;
An identification unit that inputs the feature amount of the determination image output from the feature amount calculation unit into the identifier generated by the identifier learning unit and determines whether the inspection object is good or bad Image inspection device characterized by In the image inspection apparatus according to claim 1,
The feature quantity learning unit
A first input image storage unit that stores the learning image as an input image;
A first convolutional neural network operation unit that receives the input image and outputs a feature of the input image;
A deconvoluted neural network operation unit that receives an input of the feature amount of the input image output by the first convolutional neural network operation unit, and outputs an image having the same size as the input image;
An output image storage unit which stores the image output from the deconvoluted neural network operation unit as an output image;
An image difference calculation unit that calculates a difference between the input image and the output image;
A feature amount calculation setting update unit that updates the values of the parameters of the first convolutional neural network computing unit and the deconvoluted neural network computing unit such that the difference is smaller than the calculated value;
A feature value calculation setting storage unit for storing values of the parameters of the first convolutional neural network operation unit and the deconvoluted neural network operation unit;
Equipped with
The first convolutional neural network operation unit and the deconvoluted neural network operation unit respectively perform operations using the values of the parameters stored in the feature value calculation setting storage unit,
The feature amount calculation setting storage unit stores the updated value of the parameter,
The image inspection apparatus, wherein the first convolutional neural network operation unit outputs the feature quantities of each of all the learning images using the updated values of the parameters. In the image inspection apparatus according to claim 2,
The feature quantity calculation unit
A second input image storage unit storing the determination image;
A second convolutional neural network computing unit that outputs the feature amount of the determination image using the updated determination parameter stored in the feature amount calculation setting storage unit as an input of the determination image And an image inspection apparatus characterized by comprising. The image inspection apparatus according to any one of claims 1 to 3.
The classifier learning unit
A first feature amount input unit that inputs the feature amount of the learning image output from the feature amount learning unit;
A discrimination operation unsupervised learning unit that performs unsupervised learning by using the feature amount of the learning image as input and generates the classifier;
Equipped with
The discrimination operation unsupervised learning unit learns the classifier by any one of an isolation forest, a One-Class support vector machine, a subspace method, a local outlier factor, and a statistical test. Image inspection device. In the image inspection apparatus according to claim 4,
The identification unit
A second feature amount input unit that inputs the feature amount of the determination image output by the feature amount calculation unit;
And a discrimination operation unit for judging whether the inspection object is good or bad using the discriminator generated by the discrimination operation unsupervised learning unit with the feature amount of the judgment image as an input. Image inspection device. In the image inspection apparatus according to claim 4 or 5,
The discriminator learning unit further includes a discriminator adjustment unit configured to adjust the value of the adjustment parameter of the discriminant calculation unsupervised learning unit,
The discriminator adjustment unit adjusts the value of the adjustment parameter based on a feature value calculated from the image of the inspection target in which an image indicating a defective product and an image indicating a non-defective product are distinguished. Image inspection device. The image inspection apparatus according to any one of claims 1 to 3.
The classifier learning unit
A first feature amount input unit that inputs the feature amount of the learning image output from the feature amount learning unit;
A quality information input unit that inputs information indicating the quality of the learning image corresponding to the feature amount of the input learning image;
A discrimination operation supervised learning unit that generates the discriminator by supervised learning with the feature amount of the learning image and the information indicating the quality of the corresponding learning image as an input;
Equipped with
The image processing apparatus characterized in that the identification arithmetic supervised learning unit learns by any one of a support vector machine, a random forest, and a boosting method. In the image inspection apparatus according to claim 7,
The identification unit
A second feature amount input unit that inputs the feature amount of the determination image output by the feature amount calculation unit;
And a discrimination operation unit for judging whether the inspection object is good or bad using the discriminator generated by the discrimination operation supervised learning unit with the feature amount of the discrimination image as an input. Image inspection device. The image inspection apparatus according to any one of claims 1 to 8.
An image inspection apparatus, further comprising: an image acquisition unit having a first image acquisition unit for acquiring the learning image; and a second image acquisition unit for acquiring the determination image. A feature amount learning step of constructing a learned neural network which performs learning of a neural network based on a learning image including an inspection object and outputs a feature amount of the learning image;
A discriminator learning step of generating a discriminator for judging whether the inspection object is good or bad based on the learning based on the feature amount of the image for learning output from the learned neural network;
A feature amount calculating step of inputting a determination image including the inspection target to the learned neural network and outputting a feature amount of the determination image;
An identification step of determining the quality of the inspection object by inputting the feature amount of the determination image output in the feature amount calculation step to the identifier generated in the identifier learning step. Image inspection method characterized by

Images