JP7404817B2 - Learning device, detection device, learning method, and learning program - Google Patents

Description

The present disclosure relates to a learning device, a detection device, a learning method, and a learning program.

2. Description of the Related Art In recent years, in production scenes such as production lines, technology has been developed to photograph the manufactured products using a photographing device and automatically inspect the quality of the products based on the obtained observation images. When there are multiple samples of both non-defective products and defective products, machine learning using images of non-defective products (hereinafter referred to as non-defective product images) and images of defective products (hereinafter referred to as defective product images) The quality of the product can be determined using the discriminator obtained by performing this process. However, it is usually difficult to collect more defective products than good products. For this reason, techniques have been developed for determining quality using a generative model obtained by machine learning using a plurality of images of non-defective products (Non-Patent Documents 1 to 3).

Bergmann, et al., “Improving Unsupervised Defect Segmentation by Applying Structural Similarity To Autoencoders”, arXiv:1807.02011v3, February 1, 2019. Schlegl, et al., “Unsupervised Anomaly Detection with Generative Adversarial Networks to Guide Marker Discovery”, arXiv:1703.05921v1, March 17, 2017. Kenta Toyota, Kazuhiro Hotta, "Automatic identification of defective locations using subspace method and robust statistics", SSII2016, IS3-22, June 10, 2016

The detection methods in Non-Patent Documents 1 to 3 use a generation model that converts a given image into a feature amount and generates an image that is a restored image of the given image from the feature amount obtained by the conversion. As this generation model, a machine learning model such as a neural network or an eigenvector derived by principal component analysis can be used. A generative model is trained by machine learning to generate a restored image that matches the given training image when it is given a training image. As a result of this training, the generative model produces reconstructed images that have high reproducibility for features that appear in the training images and less reproducibility for other features that are unlikely to appear in the training images (e.g., have no chance). Acquire the ability to generate.

Therefore, by training a generative model as described above using images of non-defective products as training images, the generative model will be able to: It is possible to obtain the ability to generate a restored image that reproduces the defect-free state of the product depicted in a given image (or with less reproducibility in terms of defects). As a result of acquiring this ability, when a defect is included in a product shown in an observed image, there will be a difference in the area where the defect exists between the restored image generated by the trained generative model and the observed image. arise. Therefore, the presence or absence of a defect can be detected based on the difference between the restored image generated by the trained generative model and the observed image.

The inventors of the present invention have discovered that such conventional detection methods have the following problems. In other words, if the number of training images used for machine learning of the generative model is small, the generative model will generate a restored image that reproduces the product in the given image (or has low reproducibility for defects). There is a possibility that the ability will be insufficient. If a generative model with insufficient ability is given an observed image of a product without defects, there may be areas in the restored image generated by the generative model where the state of the product shown in the observed image is not correctly reproduced. As a result, there are differences between the observed image and the restored image in areas where the state of the product is not correctly reproduced, and even though there is no defect in the product shown in the observed image, the product may be defective. If it exists, there is a possibility of false detection. Furthermore, even if an observation image showing a product with a defect is given, there will be a difference between the observed image and the restored image in areas other than the area where the defect is shown (the true defect location), resulting in the defect location being may be falsely detected.

On the other hand, if the number of training images used for machine learning of the generative model is excessive, the processing time for the machine learning and the machine resources (for example, CPU, memory) used for the machine learning will become enormous. Therefore, a problem arises in that the processing cost of machine learning increases. In addition, the ability of the generative model to reproduce the state of the product that appears in the given image becomes excessive.In other words, the reproducibility of defects (defective parts) in the restored image generated by the generative model becomes excessive. It gets expensive. For example, assume that a large number of training images containing random patterns are used for machine learning of a generative model. In this case, by learning how to reconstruct random patterns, the generative model acquires the ability to faithfully reproduce any given pattern. If the ability to reproduce the state of the product shown in a given image becomes excessive, if there is a defect in the product shown in the observed image, the restored image generated by the generative model will show that defect. The location is reproduced with high precision. As a result, there arises a problem in that there is no difference between the observed image and the restored image (that is, the defective locations cancel each other out), making it difficult to detect the defective locations.

Therefore, in order to improve the accuracy of detecting defective parts based on the difference between the observed image and the restored image, it is necessary to generate a restored image that reproduces the product shown in the given image (or with low reproducibility for defects). It is desirable to appropriately select images used for learning so that the generative model acquires the ability to However, in conventional methods, the images used for this learning are determined based on the results of manual trial and error. Therefore, there is a problem in that it is costly to select learning images used for machine learning of a generative model used for detecting defects. Note that this problem may occur not only when detecting defects in a product, but also in any situation where some feature is detected from an image of an object. For example, when using the above-mentioned generative model to detect whether or not a predetermined object appears in an observation image, manual trial and error cannot be used to generate A problem may arise in that selecting images to be used for machine learning of a model is costly.

The present disclosure has been made in view of these circumstances, and its purpose is to provide a technique for improving the performance of a generative model by appropriately selecting images used for learning.

According to an example of the present disclosure, a learning device includes a first acquisition unit, a learning unit, a second acquisition unit, a generation unit, an evaluation unit, an extraction unit, and an additional learning unit. The first acquisition unit acquires one or more first learning images showing predetermined features. The learning unit performs machine learning of the generative model using one or more first learning images. The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image in which the given image is restored from the feature amount. The learning unit trains the generative model by machine learning so that when each of the one or more first learning images is given, it generates a restored image that matches each of the one or more given first learning images. The second acquisition unit acquires a plurality of additional images showing predetermined characteristics. The generation unit generates each of the plurality of restored images by restoring each of the plurality of images by supplying each of the plurality of additional images to the generation model. The evaluation unit calculates an evaluation value of each additional image according to the difference between each additional image and each generated restored additional image, and determines the degree of restoration of each additional image in each restored additional image. Calculate the evaluation value for. The extraction unit extracts one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the calculated evaluation value. By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, the additional learning unit adapts to each of the one or more given second learning images. The generative model is further trained to generate a restored image.

First, the learning device performs machine learning of a generative model using one or more first learning images in which predetermined features are captured. The number of first learning images may be selected as appropriate, such as from one to several, to the extent that the processing cost of machine learning does not become enormous (that is, not excessive). Thereby, the generative model is trained to convert the first learning image into a feature amount and generate a restored image that matches the first learning image from the feature amount obtained by the conversion. In other words, when an input image containing a predetermined feature is given, regardless of whether or not other features other than the predetermined feature that are unlikely to appear in the training image or are not included in the training image appear in the input image, A trained learning model that has acquired the ability to generate a restored image with predetermined features from the input image is provisionally constructed.

Next, the learning device acquires a plurality of additional images each showing a predetermined feature. Each acquired additional image is a training image candidate used for further machine learning (additional learning) of the generative model. If the trained generative model has sufficient ability to reproduce images with predetermined features, then when an additional image is given, the trained generative model will adapt to the given additional image. It should generate a restored image. On the other hand, if the ability to reproduce images with predetermined features is insufficient, the trained generative model should generate restored images that are unsuitable (i.e., incongruent) with the given additional image. .

Therefore, the learning device generates each restored additional image by restoring each additional image by giving each acquired additional image to the generation model. Then, the learning device calculates, for each additional image, an evaluation value for the degree of restoration of each additional image in each restored additional image, according to the difference between each additional image and each restored additional image. .

The larger the difference between the additional image and the restored additional image, the lower the degree of restoration of the additional image in the restored additional image, that is, the machine learning of the generative model for such an additional image is insufficient. shows. Therefore, an additional image that has a large difference from the restored additional image, in other words, an additional image that has been restored to a low degree, is highly likely to contribute to appropriately improving the ability of the generative model to reproduce an image with predetermined characteristics.

On the other hand, the smaller the difference between the additional image and the restored additional image, the higher the degree of restoration of the additional image in the restored additional image, that is, the machine learning of the generative model for the additional restored image is sufficient. shows. Therefore, an additional image with a small difference from the restored additional image, in other words, a highly restored additional image is unlikely to contribute to an appropriate improvement in the ability of the generative model. In other words, such an additional image with a high degree of restoration has a low degree of contribution to improving the ability to restore an image showing predetermined characteristics. Therefore, if you try to improve the ability of the generative model by adding one or more second learning images included in the additional images with a high degree of restoration to the learning images used for machine learning, it is possible to improve the ability of the generative model. Excessive addition may lead to an enormous increase in the number of learning images used for machine learning, which may increase the processing cost of the machine learning. Furthermore, by improving the ability to restore patterns for parts other than predetermined features, there is a possibility that the generative model will acquire the ability to faithfully reproduce any pattern. In this case, the trained generative model makes it impossible to detect features other than the predetermined features.

Taking these points into consideration, the learning device extracts one or more additional images that are evaluated to have a low degree of restoration from the plurality of additional images based on the calculated evaluation value. A method for evaluating that the degree of restoration is low may be selected as appropriate depending on the embodiment. For example, the learning device may extract the additional image with the lowest evaluation. Further, for example, the learning device may extract a predetermined number of additional images in order from the one with the lowest evaluation. The number of additional images to be extracted may be determined as appropriate. Further, for example, the learning device according to the configuration may extract an additional image with a lower evaluation than a threshold value. The threshold value serving as the criterion for determination may be determined as appropriate. Then, by machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, the learning device The generative model is further trained to generate matching reconstructed images. A trained generative model is a reconstructed image that shows a predetermined feature for a given input image, regardless of whether the input image contains other features that are unlikely to appear or are not seen in the training image. Acquire the ability to generate. Therefore, the trained generative model can be used to detect the other features.

Therefore, according to the above disclosure, based on the results of the tentatively constructed trained generative model demonstrating the ability for each candidate, from the training image candidates used for machine learning, the appropriate ability is determined. One or more candidates that are highly likely to contribute to improvement can be automatically extracted. Therefore, images used for machine learning (learning images) can be appropriately selected. In addition, a trained generative model built using appropriately selected training images can accurately detect whether or not features other than the predetermined features appear in the input image (observed image). can do. In other words, the performance of the generative model can be improved.

Note that the generation model is constituted by a module that includes calculation parameters used in calculation processing to convert an input image into a feature amount and generate an image obtained by restoring the input image from the feature amount obtained by the conversion. The type of generation model is not particularly limited as long as it can convert images, and may be selected as appropriate depending on the embodiment. For example, the generative model may be constructed by a neural network. In this case, parameters regarding each neuron (node) constituting the neural network (for example, the weight of the connection between each neuron, the threshold value of each neuron, etc.) are examples of calculation parameters. Further, for example, the generative model may be configured by an eigenvector (orthonormal vector) matrix derived by principal component analysis. In this case, each component (term) of the eigenvector matrix is an example of a calculation parameter. Hereinafter, the "eigenvector matrix" will also be simply referred to as "eigenvector." Note that the "feature amount" may also be referred to as a "feature vector." The feature amount basically has lower dimensions than the input image. The data format of the feature amount and the information indicated by the feature amount are not particularly limited, and may be determined as appropriate depending on the embodiment.

The type of the predetermined feature is not particularly limited as long as it can be seen in an image, and may be selected as appropriate depending on the embodiment. The predetermined feature may be a feature that itself can be extracted, a background that is not extracted, or a combination thereof. Similarly, the types of other features do not need to be particularly limited as long as they can be seen in the image, and may be selected as appropriate depending on the embodiment. A predetermined feature may be capable of taking two states: a state in which it includes other features and a state in which it does not include other features. When a predetermined feature and other features have such a relationship, it is desirable that the probabilities of each state be biased. When the possible probabilities of each state are biased, it means that the possible probabilities of any state are higher than the possible probabilities of other states to the extent that it affects the reproducibility of the generative model. In addition, it is desirable that a feature not included in a state with a high probability of occurrence of a predetermined feature but included in a state with a low probability of occurrence is selected as the other feature. Furthermore, it is desirable that the predetermined features appearing in each learning image used for machine learning do not include the other features. As a result, by using a training image that shows a predetermined feature that does not include other features for machine learning of a generative model, the generative model can determine whether or not the predetermined feature that appears in the input image contains other features. Regardless, it is trained to generate a restored image that shows a predetermined feature that does not include the other features. Therefore, based on the difference between the restored image generated by the generative model and the input image, it is possible to detect whether a predetermined feature appearing in the input image includes another feature. As an example, the predetermined feature may be a product and the other feature may be a defect in the product. That is, each learning image may include a product that can take two states: a state containing a defect and a state containing no defect. The type of image does not need to be particularly limited, and may be selected as appropriate depending on the embodiment. The image may be, for example, a binary image such as a black and white image, an RGB image, a depth image, an infrared image, or the like.

When performing a visual inspection of a product, the predetermined feature may be, for example, a product transported on a production line of electronic equipment, electronic parts, automobile parts, medicine, food, and the like. The electronic component may be, for example, a board, a chip capacitor, a liquid crystal, a relay winding, or the like. Automotive parts may be, for example, connecting rods, shafts, engine blocks, power window switches, panels, etc. The drug may be, for example, a packaged tablet, an unpackaged tablet, or the like. A product may be a final product produced after the manufacturing process is completed, an intermediate product produced during the manufacturing process, or an initial product prepared before the manufacturing process. It's okay. Accordingly, other features than the predetermined features may be, for example, defects such as scratches, dirt, cracks, dents, burrs, color unevenness, and foreign matter contamination. In addition, the predetermined feature may be, for example, a movable object such as a person or a vehicle (eg, a car). Features other than the predetermined features may be, for example, the three-dimensional shape of the object, optical flow (direction of movement), etc. that can change due to the movement. Note that the "predetermined feature" may be replaced with "predetermined subject" or "background feature," and the "other feature" may be replaced with "predetermined feature" or "foreground feature." Further, the "learning device" may be read as a "training device" or a "model generating device."

In the above disclosure, calculating the evaluation value according to the difference means generating each of the plurality of difference images indicating the difference between each of the plurality of additional images and each of the plurality of restored and additional images, and calculating the evaluation value according to the difference. , the larger the pixel value of each pixel, the higher the evaluation value, and the contribution of pixels having a reference pixel value or more to the evaluation value is greater than the contribution of pixels less than the reference pixel value to the evaluation value. The evaluation value may be calculated as follows. The larger the difference between a pixel between each additional image and each restored additional image, the larger the pixel value of that pixel in each difference image. Therefore, according to the configuration, it is possible to appropriately calculate the evaluation value according to the degree of restoration of the additional image. Specifically, the larger the obtained evaluation value, the lower the degree of restoration of the corresponding additional image. Therefore, in the process of the extraction unit, an additional image that is determined to have a high evaluation value is extracted according to a predetermined criterion.

In the above disclosure, the learning device may further include a display control unit that causes a display device to display each of the extracted one or more additional images. For example, in addition to predetermined features, an image that includes other features that are not desired to be reproduced in the generative model may be acquired as an additional image. In the above evaluation, the degree of restoration of the additional image may be low even when other features appear in the additional image in addition to the predetermined feature. According to this configuration, the additional image automatically extracted by the learning device as a learning image used for machine learning (additional learning) is displayed on the display device. Based on this display, the operator can visually confirm whether the additional image includes other features that are not desired to be reproduced in the generated model.

In the above disclosure, the learning device includes a selection reception unit that receives, for each of the one or more additional images displayed on the display device, a selection as to whether or not to use each of the one or more additional images for machine learning. It may further include. Then, the additional learning unit may determine the additional images selected to be used for machine learning as one or more second learning images. According to this configuration, by reflecting the result of visual inspection by the operator, it is possible to exclude additional images that include other features as described above from the targets of machine learning of the generative model. As a result, it is possible to appropriately construct a trained generative model that has acquired the ability to generate a restored image for an input image that has low reproducibility for other features and high reproducibility for a predetermined feature. Therefore, according to the constructed trained generative model, it is possible to accurately detect whether features other than the predetermined features appear in the input image (observed image).

In the above disclosure, the learning device may further include a third acquisition unit that acquires a plurality of evaluation images showing a predetermined feature and features other than the predetermined feature. No other features are shown in the multiple additional images. The generation unit may further generate each of a plurality of restored evaluation images obtained by restoring each of the plurality of evaluation images by supplying each of the plurality of acquired evaluation images to a trained generation model. The evaluation unit may further determine whether or not other features are reflected in each additional image based on the degree of difference between each additional image and each generated restored additional image, and Based on the degree of difference between each evaluation image and each generated restored evaluation image, it may be determined whether other features are included in each evaluation image. The evaluation unit determines whether the number or percentage of additional images determined to include other features is equal to or higher than a first threshold, or the number or percentage of evaluation images determined to include no other features. An evaluation value may be calculated for one or more additional images that are determined to include other features when the second threshold value or more is equal to or higher than the second threshold. The extraction unit adds one or more images that are evaluated to have a low degree of restoration from among the one or more additional images that are determined to include other features based on the calculated evaluation value. You may also extract images for

In this configuration, other features are subject to detection processing using restored images generated by trained generative models, and are basically not included in the learning images used for training. It is a characteristic. The predetermined features shown in each additional image do not include other features. In the above visual inspection example, the predetermined feature is the product, and the other features are defects that may occur in the product. Therefore, in this configuration, the number or percentage of additional images that were determined to include other features was incorrectly determined to include other features even though the other features were not included. Corresponds to the number or proportion of images. In addition, the number or percentage of evaluation images that were determined to not contain other features is the number or percentage of images that were incorrectly determined to not contain other features even though they did contain other features. corresponds to

Therefore, the learning device uses the number or percentage of additional images that are determined to include other features and the number or percentage of evaluation images that are determined to not include other features. We evaluate the performance of detection processing using restored images generated by a generative model. Then, when the performance of the detection process is low, the learning device according to the configuration extracts one or more additional images that are evaluated to have a low degree of restoration from among the additional images that have been incorrectly determined. Then, machine learning (additional learning) of the generative model is performed using one or more second learning images included in the extracted one or more additional images. As a result, according to the configuration, the performance of the detection process using the generative model is evaluated, and by improving the ability of the generative model to restore an image containing a predetermined feature, it is possible to improve the performance of the detection process using the generative model. A suitable trained generative model can be constructed. Note that the first threshold value and the second threshold value may be the same value or may be different values. Further, the first threshold value and the second threshold value may be specified by an operator's input, or may be fixed values specified by a program.

In the above disclosure, the learning device is provided with a plurality of parameter candidates for determining whether or not other features appear in each additional image and whether or not other features appear in each evaluation image. It's okay to be rejected. The evaluation unit uses each of the plurality of parameter candidates to determine whether or not other features appear in each additional image, and to determine whether or not other features appear in each evaluation image. Based on the results of each determination, optimization of parameters for determining whether other features are included in the image is performed. For example, the evaluation unit selects one parameter candidate that has the smallest number of additional images that are determined to include other features and evaluation images that are determined to not include other features among the plurality of parameter candidates. Identify. The evaluation unit determines whether, in the results of each determination using the optimized parameters, the number or proportion of additional images determined to include other features is greater than or equal to a first threshold, or the other features are When the number or proportion of evaluation images that are determined not to be captured is equal to or higher than a second threshold, an evaluation value is calculated for one or more additional images that are determined to include other features. Good too.

When detecting whether or not other features appear in the input image based on the differences that occur between the input image and the restored image, what kind of difference occurs to determine whether the other features appear in the input image? Standards will be established to determine what is recognized as being accredited. This standard is the parameter for the detection (hereinafter also referred to as detection parameter). The optimal detection parameters for detecting other features may vary depending on the restoration ability of the generative model. If the values of the detection parameters are adjusted manually, human costs for adjusting the detection parameters will be incurred each time machine learning of the generative model is performed. Moreover, manual adjustment of the values of detection parameters can lead to errors due to human factors. In contrast, with this configuration, the detection parameters can be automatically optimized according to the restoration ability of the generative model, and the performance of the detection process using the generative model can be evaluated. Therefore, it is possible to construct a trained generative model that is more suitable for detection processing of other features without causing problems due to human factors.

In the above disclosure, the learning device further includes a third acquisition unit that acquires a plurality of first evaluation images and a plurality of second evaluation images. The plurality of first evaluation images include a predetermined feature and features other than the predetermined feature. The plurality of second evaluation images show predetermined features and do not show other features. The evaluation unit is given a plurality of parameter candidates for determining whether or not other features appear in the image. The evaluation unit uses each of the plurality of parameter candidates to determine whether or not other features are shown in each first evaluation image, and to determine whether or not other features are shown in each second evaluation image. Execute. The evaluation unit optimizes parameters for determining whether other features are included in the image based on the results of each determination. The selection accepting unit further accepts a selection as to whether or not other features are included in each of the one or more additional images displayed on the display device. The third acquisition unit acquires each additional image selected as showing other features as one of the plurality of first evaluation images, and acquiring each additional image selected as not showing other features. is acquired as one of the plurality of second evaluation images.

According to the above configuration, the operator can omit the effort of preparing the first evaluation image and the second evaluation image.

In the above disclosure, the second acquisition unit communicates with the detection device. The detection device acquires an observed image obtained by observing a predetermined feature, and inputs the observed image to a generation model to generate a restored observed image corresponding to the observed image. The detection device detects whether features other than the predetermined features are included in the observed image based on the difference between the generated restored observed image and the observed image. The second acquisition unit acquires the observed image acquired by the detection device as one of the plurality of additional images.

According to the above configuration, the observed image is acquired as an additional image. Therefore, the operator can omit the effort of preparing additional images.

In the above disclosure, the learning device further includes a storage processing unit that stores one or more additional images extracted by the extraction unit among the plurality of additional images in the storage unit.

Generally, the number of observation images acquired by a detection device becomes enormous in a short period of time. However, according to the above configuration, although the observed image is acquired as an additional image, only one or more additional images extracted based on the evaluation value are stored in the storage unit. Therefore, it is possible to avoid a situation where the free space in the storage unit becomes insufficient.

In the above disclosure, each of the first learning image and the additional image may be divided into a plurality of patch images. A generative model may be provided for each patch image. The learning unit performs machine learning for each patch image so that, when each of the one or more first learning images is given, it generates a restored image that matches each of the one or more given first learning images. You can also train the model. The generation unit may generate each restored additional image corresponding to each additional image for each patch image. The evaluation unit may calculate the evaluation value of each additional image for each patch image. The extraction unit may extract one or more additional images from the plurality of additional images based on the calculated evaluation value for each patch image. The additional learning unit generates a restored image that matches each of the given one or more second learning images by machine learning for each patch image, when each of the one or more second learning images is given. The generative model may be further trained.

When each training image is divided into multiple patch images and a generative model is prepared for each patch image, it is necessary to select the learning image to be used for machine learning of the generative model used to detect features other than the predetermined features. Costs will further increase. According to this configuration, it is possible to at least partially automate the selection of training images suitable for machine learning of the generative model, and therefore it is possible to significantly reduce the cost of selecting training images used for machine learning. . Further, according to the configuration, the ability of the generative model can be optimized for each patch image, so it is possible to improve the accuracy of detecting features other than the predetermined features.

In the above disclosure, the generative model may be configured by a neural network. According to this configuration, a trained generative model can be easily constructed. Alternatively, in the above disclosure, machine learning may include principal component analysis, and the generative model may be constructed from eigenvectors derived by principal component analysis.

In the above disclosure, the predetermined characteristic may be a product conveyed on a manufacturing line that is defect-free. According to this configuration, images used for machine learning of a generative model that can be used for product appearance inspection can be appropriately selected, and the performance of the generative model can be improved.

In the above disclosure, the learning unit constructs a provisional trained generative model by machine learning using the first learning image. However, the form of the learning device of the present invention does not have to be limited to such an example. In the learning device according to the above aspect, the process of the learning unit may be omitted. In this case, the learning device may acquire a provisional trained generative model constructed by another computer.

For example, according to an example of the present disclosure, a learning device includes a model acquisition section, a generation section, an evaluation section, an extraction section, and an additional learning section. The model acquisition unit acquires a generative model trained by machine learning using one or more first learning images showing predetermined features. The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image in which the given image is restored from the feature amount. Through machine learning, the generative model is trained to, given each of the one or more first training images, generate a restored image that matches each of the one or more given first training images. The image acquisition unit acquires a plurality of additional images showing predetermined characteristics. The generation unit generates each of the plurality of restored additional images by restoring each of the plurality of additional images by supplying each of the plurality of additional images to the generation model. The evaluation unit calculates an evaluation value of each additional image according to the difference between each additional image and each generated restored additional image, and determines the degree of restoration of each additional image in each restored additional image. Calculate the evaluation value for. The extraction unit extracts one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value. By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, the additional learning unit adapts to each of the one or more given second learning images. The generative model is further trained to generate a restored image.

The embodiments of the present disclosure do not need to be limited to learning devices. According to an example of the present disclosure, the detection device includes an acquisition unit that acquires an observed image obtained by observing a predetermined feature, and a generative model trained by the learning device according to any of the above forms. By inputting , the generator generates a restored observation image corresponding to the observed image, and other features other than the predetermined features are added to the observed image based on the difference between the generated restored observation image and the observed image. A detection unit that detects whether or not the image is captured. According to the configuration, it is possible to accurately detect whether features other than the predetermined features appear in the observed image.

As another aspect of each of the learning device and the detection device according to each of the above embodiments, one aspect of the present disclosure may be an information processing method that realizes each of the above configurations, a program, or the like. It may also be a computer-readable storage medium that stores such a program. Here, a computer-readable storage medium is a medium that stores information such as programs through electrical, magnetic, optical, mechanical, or chemical action. Further, a detection system according to one aspect of the present disclosure may be configured by a learning device and a detection device according to any of the above embodiments.

For example, a learning method according to one aspect of the present disclosure includes the steps of a computer acquiring one or more first learning images in which predetermined features are captured, and using the one or more first learning images to create a generative model. The generative model is configured to, when an image is given, convert the given image into a feature amount and generate an image that restores the given image from the feature amount. , training a generative model to generate restored images that match each of the given one or more first learning images when each of the one or more first learning images is given by machine learning; a step of acquiring a plurality of additional images showing predetermined characteristics, and a step of generating each of a plurality of restored additional images by restoring each of the plurality of additional images by giving each of the plurality of additional images to a generation model. and an evaluation value of each additional image according to the difference between each additional image and each generated restored additional image, and an evaluation of the degree of restoration of each additional image in each restored additional image. a step of calculating a value, a step of extracting one or more additional images that are evaluated as having a low degree of restoration based on the evaluation value from a plurality of additional images, and one or more images extracted by machine learning. When each of the one or more second learning images included in the above additional images is given, the generative model is further trained to generate a restored image that matches each of the given one or more second learning images. An information processing method that executes steps.

Further, for example, a learning program according to one aspect of the present disclosure includes the steps of acquiring, in a computer, one or more first learning images in which a predetermined feature is captured, and using the one or more first learning images, A step of implementing machine learning of a generative model, the generative model converts the given image into features when an image is given, and generates an image that restores the given image from the features. training the generative model to generate a restored image that matches each of the given one or more first learning images, using machine learning; and a step of obtaining a plurality of additional images showing predetermined characteristics, and providing each of the plurality of additional images to a generation model to generate each of a plurality of restored additional images by restoring each of the plurality of additional images. and the evaluation value of each additional image according to the difference between each additional image and each generated restored additional image, and the degree of restoration of each additional image in each restored additional image. a step of calculating an evaluation value for the image; a step of extracting one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value; When each of one or more second learning images included in one or more additional images is given, the generative model is further modified to generate a restored image that matches each of the given one or more second learning images. This is a program for executing training steps.

According to the present disclosure, the performance of a generative model can be improved by appropriately selecting images used for learning.

FIG. 3 is a diagram schematically illustrating an example of an application screen of the detection system according to the embodiment. FIG. 1 is a diagram schematically illustrating an example of the hardware configuration of a learning device 1 according to the present embodiment. FIG. 2 is a diagram schematically illustrating an example of the hardware configuration of a detection device 2 according to the present embodiment. FIG. 1 is a diagram schematically illustrating an example of the software configuration of the learning device 1 according to the present embodiment. FIG. 1 is a diagram schematically illustrating an example of the software configuration of the learning device 1 according to the present embodiment. FIG. 2 is a diagram schematically illustrating an example of the software configuration of the detection device 2 according to the present embodiment. It is a flowchart showing an example of a processing procedure of the learning device 1 according to the present embodiment. It is a flowchart showing an example of a processing procedure of the learning device 1 according to the present embodiment. 3 is a diagram schematically illustrating an example of a generative model 5 configured by a neural network. FIG. 3 is a diagram schematically illustrating an example of a process for detecting a defect L. FIG. It is a figure which shows an example of a hypothetical determination result of step S107 and S108. FIG. 3 is a diagram schematically illustrating an example of a processing process for calculating an evaluation value 125. FIG. 3 is a diagram schematically illustrating an example of a display screen according to the present embodiment. FIG. 3 is a diagram schematically illustrating an example of a display screen according to the present embodiment. It is a flowchart which shows an example of the processing procedure of the detection device 2 concerning this embodiment. The results of evaluating the performance of a trained generative model constructed in the process of repeating machine learning processing using this experimental example are shown. 12 is a diagram schematically illustrating an example of the software configuration of a learning device 1B according to modification 5. FIG. 12 is a diagram schematically illustrating an example of the software configuration of a learning device according to Modification 8. FIG. 12 is a flowchart illustrating an example of a procedure for extracting an additional image in a learning device according to Modification 9. FIG. 12 is a flowchart illustrating an example of a procedure for extracting an additional image in a learning device according to Modification 9. FIG. 12 is a flowchart illustrating an example of the procedure of additional learning processing in the learning device according to Modification 9. FIG. 12 is a flowchart illustrating an example of the procedure of additional learning processing in the learning device according to Modification 9. FIG. 5 is a diagram showing an example of a display screen displayed in step S502. FIG.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment (hereinafter also referred to as "this embodiment") according to one aspect of the present invention will be described below based on the drawings. However, this embodiment described below is merely an illustration of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the invention. That is, in implementing the present invention, specific configurations depending on the embodiments may be adopted as appropriate. Although the data that appears in this embodiment is explained using natural language, more specifically, it is specified using pseudo language, commands, parameters, machine language, etc. that can be recognized by a computer.

§1 Application Example First, an example of a situation where the present invention is applied will be described using FIG. 1. FIG. 1 is a diagram schematically illustrating an example of an application screen of a detection system according to an embodiment. In the example of FIG. 1, an image of product R is used as an example of a situation in which it is detected whether or not an image of a predetermined feature includes features other than the predetermined feature. Let's assume a situation where we want to detect whether something exists. Product R, which is a good product, is an example of the predetermined feature, and a defect is an example of another feature. However, the types of the predetermined feature and other features do not need to be limited to such examples, and may be appropriately selected depending on the embodiment. The present disclosure is applicable to any situation where some feature in an image is detected.

As shown in FIG. 1, a detection system 100 according to this embodiment includes a learning device 1 and a detection device 2. The learning device 1 is a computer configured to construct a learned generative model 5 by machine learning. The detection device 2 is a computer configured to detect features other than the predetermined features (in this embodiment, defects) using the trained generative model 5. In this embodiment, the learning device 1 and the detection device 2 are connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, and the like.

However, the method of exchanging data between the learning device 1 and the detection device 2 does not need to be limited to this example, and may be selected as appropriate depending on the embodiment. For example, data may be exchanged between the learning device 1 and the detection device 2 using a storage medium. Moreover, in this embodiment, the learning device 1 and the detection device 2 are mutually separate computers. However, the configuration of the detection system 100 does not need to be limited to such an example, and may be determined as appropriate depending on the embodiment. For example, the learning device 1 and the detection device 2 may be an integrated computer. Further, for example, at least one of the learning device 1 and the detection device 2 may be configured by a plurality of computers.

The learning device 1 according to the present embodiment acquires one or more first learning images 121 in which a good product R is captured as an example of a predetermined feature. The product R may be, for example, a product transported on a manufacturing line for electronic equipment, electronic parts, automobile parts, medicine, food, and the like. The electronic component may be, for example, a board, a chip capacitor, a liquid crystal, a relay winding, or the like. Automotive parts may be, for example, connecting rods, shafts, engine blocks, power window switches, panels, etc. The drug may be, for example, a packaged tablet, an unpackaged tablet, or the like. Product R may be a final product produced after the manufacturing process is completed, an intermediate product produced during the manufacturing process, or an initial product prepared before the manufacturing process. There may be.

The learning device 1 performs machine learning of the generative model 5 using the acquired one or more first learning images 121. The generative model 5 is configured to, when an input image is given, convert the given input image into a feature amount, and generate an image in which the given input image is restored from the feature amount obtained by the conversion. . Through machine learning, the learning device 1 uses a generative model 5 to generate a restored image that matches each of the given one or more first learning images 121 when each of the one or more first learning images 121 is given. train. As a result, the generative model 5 acquires the ability to restore the image of the product R, which is a good product, within the range that appears in the first learning image 121.

However, product R does not always appear in the same way in the image. For example, individual differences may occur in the product R. Furthermore, for example, photographing conditions such as the timing of photographing may change (for example, the position of the product R may shift or be tilted). Depending on these factors, the appearance of the product R in the obtained image may differ. At the stage where machine learning using the first learning image 121 is completed, the trained generative model 5 has the ability to restore the image of the good product R for a range other than the range that appears in the first learning image 121. It is unknown whether they have been acquired or not.

Therefore, the learning device 1 acquires a plurality of additional images 123 each showing the product R. The learning device 1 generates each of the plurality of restored additional images 124 by restoring each of the plurality of additional images 123 by giving each of the acquired plural additional images 123 to the trained generation model 5. Then, the learning device 1 calculates the evaluation value 125 of each additional image 123 according to the difference between each additional image 123 and each generated restored additional image 124, and calculates the evaluation value 125 for each additional restored image 124. An evaluation value 125 for the degree of restoration of each additional image 123 is calculated.

The larger the difference between the additional image 123 and the restored additional image 124, the lower the degree of restoration of the additional image 123 in the restored additional image 124. In other words, the state of the product R appearing in this additional image 123 is This indicates that the ability of the generative model 5 to reproduce is insufficient. Therefore, the additional image 123 that has a large difference from the restored additional image 124, in other words, the additional image 123 that has been restored to a low degree, may contribute to an appropriate improvement in the ability of the generative model 5 to reproduce the image of the product R. expensive.

On the contrary, the additional image 123 with a high degree of restoration is unlikely to contribute to appropriately improving the ability of the generative model 5. In addition, if you try to improve the ability of the generative model 5 by adding additional images 123 with a high degree of restoration to the learning images used for machine learning, the excessive addition of additional images 123 will cause problems in machine learning. This may lead to an enormous increase in the number of learning images used, increasing the processing cost of the machine learning. Furthermore, by improving the ability to restore patterns other than product R, there is a possibility that the generative model 5 acquires the ability to faithfully reproduce any pattern.

Taking these points into consideration, the learning device 1 extracts one or more additional images 126 that are evaluated to have a low degree of restoration from the plurality of additional images 123 based on the calculated evaluation value. The predetermined criteria for evaluating that the degree of restoration is low does not need to be particularly limited, and may be set as appropriate depending on the embodiment. For example, the learning device 1 may extract the additional image 123 with the lowest evaluation as the additional image 126. Further, for example, the learning device 1 may extract a predetermined number of additional images 123 as additional images 126 in order from the one with the lowest evaluation. The number of additional images 126 to be extracted may be determined as appropriate based on operator input, setting values, and the like. Further, for example, the learning device 1 may extract the additional image 123 with a lower evaluation than the threshold value as the additional image 126. The threshold value that serves as the criterion for determination may be determined as appropriate based on operator input, set values, and the like. Then, the learning device 1 selects one or more images included in the one or more extracted additional images 126 as the second learning image 126A. The learning device 1 may determine all of the one or more additional images 126 as the second learning images 126A. Alternatively, the learning device 1 may determine a part of the one or more additional images 126 as the second learning image 126A. Through machine learning, the learning device 1 uses a generative model 5 to generate restored images that match each of the given one or more second learning images 126A, when each of the one or more second learning images 126A is given. further training.

The detection device 2 according to this embodiment acquires an observation image 221 obtained by observing the product R. In this embodiment, the detection device 2 is connected to a camera CA arranged to observe the product R. The detection device 2 acquires an observation image 221 from the camera CA. Note that the type of each image (121, 123, 221) does not need to be particularly limited, and may be appropriately selected depending on the embodiment. Each image (121, 123, 221) may be, for example, a binary image such as a black and white image, an RGB image, a depth image, an infrared image, or the like. Accordingly, the camera CA may be, for example, a general digital camera, a depth camera, an infrared camera, etc.

Next, the detection device 2 generates a restored observed image 223 corresponding to the observed image 221 by inputting the observed image 221 to the generative model 5 trained by the learning device 1. Then, based on the difference between the generated restored observation image 223 and observation image 221, the detection device 2 determines whether or not a defect is shown in the observation image 221 as an example of a feature other than the predetermined feature. Then, it is detected whether or not product R includes a defect. The defects may be, for example, scratches, dirt, cracks, dents, burrs, color unevenness, foreign matter contamination, and the like. Thereby, the detection device 2 uses the learned generative model 5 to perform an external appearance inspection of the product R (that is, determines whether the product R is good or bad).

As described above, in this embodiment, the learned generative model 5 is provisionally constructed by machine learning using the first learning image 121. In the series of processes of the learning device 1, based on the results of the provisionally constructed learned generation model 5 demonstrating the restoration ability for each additional image 123, from a plurality of additional images 123, One or more additional images 126 that are likely to contribute to appropriate improvement of the restoration ability can be automatically extracted. Therefore, images used for machine learning (learning images) can be appropriately selected. In addition, according to the trained generative model 5 constructed using appropriately selected learning images, it is possible to determine whether or not a defect is captured in the observed image 221, that is, whether or not there is a defect in the product R. can be detected with high accuracy.

§2 Specific example [Hardware configuration]
<Learning device>
Next, an example of the hardware configuration of the learning device 1 according to this embodiment will be described using FIG. 2. FIG. 2 is a diagram schematically illustrating an example of the hardware configuration of the learning device 1 according to the present embodiment.

As shown in FIG. 2, the learning device 1 according to the present embodiment is a computer to which a control section 11, a storage section 12, a communication interface 13, an input device 14, a display device 15, and a drive 16 are electrically connected. . In addition, in FIG. 2, the communication interface is described as "communication I/F."

The control unit 11 includes a CPU (Central Processing Unit), which is a hardware processor, a RAM (Random Access Memory), a ROM (Read Only Memory), etc., and is configured to execute information processing based on programs and various data. Ru. The storage unit 12 is an example of a memory, and includes, for example, a hard disk drive, a solid state drive, or the like. In this embodiment, the storage unit 12 stores various information such as a learning program 120, one or more first learning images 121, a plurality of additional images 123, a plurality of evaluation images 127, and learning result data 129.

The learning program 120 is a program for causing the learning device 1 to execute machine learning processing (FIGS. 7 and 8), which will be described later, for constructing the trained generative model 5. The learning program 120 includes a series of instructions for the information processing. One or more first learning images 121 are used for machine learning of the generative model 5. Each additional image 123 is a learning image candidate other than the first learning image 121 that is used for machine learning of the generative model 5. Each evaluation image 127 shows the product R and defects. Each evaluation image 127 is used to evaluate the performance of defect detection processing using the restored image generated by the generative model 5. The learning result data 129 indicates information regarding the learned generative model 5 constructed by machine learning. Learning result data 129 is obtained as an execution result of learning program 120. Details will be described later.

The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 13, the learning device 1 can perform data communication with other information processing devices (for example, the detection device 2) via the network. The learning device 1 may acquire a plurality of additional images 123 via the communication interface 13.

The input device 14 is, for example, a device for performing input such as a mouse or a keyboard. Further, the display device 15 is an example of an output device, and is, for example, a display. An operator can operate the learning device 1 via the input device 14 and the display device 15. Note that the display device 15 may be a touch panel display. In this case, the input device 14 may be omitted.

The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 91. The type of drive 16 may be selected as appropriate depending on the type of storage medium 91. At least one of the learning program 120, one or more first learning images 121, a plurality of additional images 123, and a plurality of evaluation images 127 may be stored in this storage medium 91.

The storage medium 91 stores information such as a recorded program by electrical, magnetic, optical, mechanical, or chemical action so that a computer, other device, machine, etc. can read the recorded program information. It is a medium for The learning device 1 may acquire at least one of the learning program 120, one or more first learning images 121, a plurality of additional images 123, and a plurality of evaluation images 127 from this storage medium 91.

Here, in FIG. 2, a disk-type storage medium such as a CD or a DVD is illustrated as an example of the storage medium 91. However, the type of storage medium 91 is not limited to the disk type, and may be other than the disk type. An example of a storage medium other than a disk type is a semiconductor memory such as a flash memory.

Note that regarding the specific hardware configuration of the learning device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA (field-programmable gate array), DSP (digital signal processor), or the like. The storage unit 12 may be configured by a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, input device 14, display device 15, and drive 16 may be omitted. The learning device 1 may include an output device other than the display device 15, such as a speaker. The learning device 1 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not match. Further, the learning device 1 may be an information processing device designed exclusively for the provided service, or may be a general-purpose server device, a general-purpose PC (Personal Computer), or the like.

<Detection device>
Next, an example of the hardware configuration of the detection device 2 according to this embodiment will be described using FIG. 3. FIG. 3 is a diagram schematically illustrating an example of the hardware configuration of the detection device 2 according to the present embodiment.

As shown in FIG. 3, in the detection device 2 according to the present embodiment, a control section 21, a storage section 22, a communication interface 23, an input device 24, a display device 25, a drive 26, and an external interface 27 are electrically connected. It is a computer. In addition, in FIG. 3, the external interface is described as "external I/F." The control unit 21 to drive 26 of the detection device 2 may be configured similarly to the control unit 11 to drive 16 of the learning device 1, respectively.

That is, the control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 22 includes, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as a detection program 220 and learning result data 129.

The detection program 220 uses the learned generative model 5 trained by the learning device 1 to cause the detection device 2 to perform information processing (FIG. 15), which will be described later, to determine the quality of the product R shown in the observed image 221. This is a program for The detection program 220 includes a series of instructions for processing the information. Details will be described later.

The communication interface 23 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 23, the detection device 2 can perform data communication with another information processing device (for example, the learning device 1) via the network.

The input device 24 is, for example, a device for performing input such as a mouse or a keyboard. The display device 25 is an example of an output device, and is, for example, a display. An operator can operate the detection device 2 via the input device 24 and the display device 25. The input device 24 and the display device 25 may be replaced with touch panel displays.

The drive 26 is, for example, a CD drive, a DVD drive, etc., and is a drive device for reading a program stored in the storage medium 92. At least one of the detection program 220 and the learning result data 129 may be stored in the storage medium 92. Further, the detection device 2 may acquire at least one of the detection program 220 and the learning result data 129 from the storage medium 92. The type of storage medium 92 may be a disk type or a type other than a disk type.

The external interface 27 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 27 may be selected as appropriate depending on the type and number of external devices to be connected. In this embodiment, the detection device 2 is connected to the camera CA via an external interface 27.

The camera CA is used to obtain an observation image 221 of the product R to be subjected to visual inspection. The type and location of camera CA do not need to be particularly limited, and may be determined as appropriate depending on the embodiment. For example, a general digital camera, depth camera, infrared camera, etc. may be used as the camera CA. Further, the camera CA may be appropriately placed so as to be able to observe the product R being transported through the production line. Camera CA may be placed, for example, near a production line that transports products R. Note that when the camera CA includes a communication interface, the detection device 2 may be connected to the camera CA via the communication interface 23 instead of the external interface 27.

Note that regarding the specific hardware configuration of the detection device 2, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be comprised of a microprocessor, FPGA, DSP, etc. The storage unit 22 may be configured by a RAM and a ROM included in the control unit 21. At least one of the communication interface 23, the input device 24, the display device 25, the drive 26, and the external interface 27 may be omitted. The detection device 2 may include an output device other than the display device 25, such as a speaker. The detection device 2 may be composed of multiple computers. In this case, the hardware configurations of the computers may or may not match. Further, the detection device 2 may be an information processing device designed exclusively for the provided service, or may be a general-purpose server device, a general-purpose PC, or the like.

[Software configuration]
<Learning device>
Next, an example of the software configuration of the learning device 1 according to this embodiment will be described using FIGS. 4 and 5. 4 and 5 are diagrams schematically illustrating an example of the software configuration of the learning device 1 according to the present embodiment.

The control unit 11 of the learning device 1 loads the learning program 120 stored in the storage unit 12 into the RAM. The control unit 11 then uses the CPU to interpret and execute instructions included in the learning program 120 developed in the RAM, thereby controlling each component. As a result, as shown in FIGS. 4 and 5, the learning device 1 according to the present embodiment includes a first acquisition unit 110, a learning unit 111, a second acquisition unit 112, a generation unit 113, an evaluation unit 114, an extraction unit 115 , an additional learning section 116, a third acquisition section 117, a display control section 118, a selection reception section 119, and a storage processing section 1110 as software modules. That is, in this embodiment, each software module of the learning device 1 is realized by the control unit 11 (CPU).

As shown in FIG. 4, the first acquisition unit 110 acquires one or more first learning images 121 in which the product R appears. The learning unit 111 performs machine learning of the generative model 5 using the acquired one or more first learning images 121. The generative model 5 is configured to, when an input image is given, convert the given input image into a feature amount, and generate an image that is a restoration of the given input image from the feature amount obtained by the conversion. By machine learning, when one or more first learning images 121 are given, the learning unit 111 uses a generative model 5 to generate a restored image that matches each of the given one or more first learning images 121. train.

In detail, the generative model 5 is a module (learning model ). The configuration of the generative model 5 does not need to be particularly limited, and may be appropriately selected depending on the embodiment. For example, as the generation model 5, a known learning model such as a neural network (for example, an autoencoder, etc.) or an eigenvector derived by principal component analysis can be used. In machine learning, when a first learning image 121 is given, the value of this calculation parameter M is adjusted so as to generate a restored image that matches the given first learning image 121. In this embodiment, the generation model 5 is configured by a neural network such as an autoencoder.

The second acquisition unit 112 acquires a plurality of additional images 123 in which the product R appears. In this embodiment, each additional image 123 depicts a product R that does not include defects (that is, a non-defective product). That is, each additional image 123 does not include any defects. The generation unit 113 generates each of the plurality of restored additional images 124 by restoring each of the plurality of additional images 123 by giving each of the plurality of acquired additional images 123 to the trained generation model 5. The evaluation unit 114 calculates the evaluation value 125 of each additional image 123 according to the difference between each additional image 123 and each generated restored additional image 124, and calculates the evaluation value 125 of each additional restored image 124. An evaluation value 125 for the degree of restoration of the original image 123 is calculated.

In this embodiment, the evaluation unit 114 calculates the difference between each additional image 123 and each restored additional image 124. Thereby, the evaluation unit 114 generates each of a plurality of difference images 182 indicating the difference between each of the plurality of additional images 123 and each of the plurality of restored and additional images 124. The more different a pixel is between the additional image 123 and the restored additional image 124, the larger the pixel value is in the difference image 182. In other words, the more pixels with large pixel values in the difference image 182, the lower the degree of restoration of the additional image 123 in the additional restoration image 124. On the contrary, the fewer pixels with large pixel values in the difference image 182, the higher the degree of restoration of the additional image 123 in the restored additional image 124. Based on this viewpoint, the evaluation unit 114 uses each generated difference image 182 to calculate each evaluation value 125 for the degree of restoration of each additional image 123.

The extraction unit 115 extracts one or more additional images 126 that are evaluated as having a low degree of restoration from the plurality of additional images 123 based on each calculated evaluation value 125 . The criteria for evaluating that the degree of restoration is low does not need to be particularly limited, and may be set as appropriate depending on the embodiment.

The learning device 1 according to the present embodiment is configured to execute processing for checking whether the one or more extracted additional images 126 are appropriate as images to be used for additional learning. Specifically, the display control unit 118 causes the display device 15 to display each of the one or more extracted additional images 126. The selection accepting unit 119 accepts, for each of the one or more additional images 126 displayed on the display device 15, a selection as to whether to use each of the one or more additional images 126 for machine learning.

The additional learning unit 116 determines one or more images included in the one or more extracted additional images 126 as the second learning image 126A according to the selection result received by the selection receiving unit 119. That is, the additional learning unit 116 determines the additional image 126 selected to be used for machine learning as the second learning image 126A to be the training target. The additional learning unit 116 excludes the additional image 126 selected not to be used for machine learning from the training target.

The additional learning unit 116 uses machine learning to generate restored images that match each of the one or more second learning images 126A, when each of the determined one or more second learning images 126A is given. Then, generative model 5 is further trained.

Furthermore, in the present embodiment, as shown in FIG. 5, before extracting the additional image 126, the learning device 1 performs defect detection processing using the restored image generated by the constructed generative model 5. configured to evaluate the performance of. Specifically, the third acquisition unit 117 acquires a plurality of evaluation images 127 in which the product R and the defect L are respectively captured. That is, each evaluation image 127 shows a product R including a defect L (that is, a defective product). The generation unit 113 further generates each of the plurality of restored evaluation images 128 by restoring each of the plurality of evaluation images 127 by giving each of the plurality of acquired evaluation images 127 to the trained generation model 5.

The evaluation unit 114 further determines whether or not the defect L is reflected in each additional image 123 based on the degree of difference between each additional image 123 and each restored additional image 124 (that is, whether or not each additional image 123 includes a defect L). It is determined whether the defect L exists in the product R shown in the image. Furthermore, the evaluation unit 114 determines whether or not the defect L is reflected in each evaluation image 127 based on the degree of difference between each evaluation image 127 and each restored evaluation image 128 (that is, the product R (Whether or not the defect L exists) is determined.

In this embodiment, the evaluation unit 114 uses each difference image 182 to determine whether the defect L is included in each additional image 123. Similarly, by calculating the difference between each evaluation image 127 and each restoration evaluation image 128, the evaluation unit 114 calculates a plurality of images indicating the difference between each of the plurality of evaluation images 127 and each of the restoration evaluation images 128. Each difference image 183 is generated. The evaluation unit 114 uses each difference image 183 to determine whether the defect L is included in each evaluation image 127.

Detection parameters for determining whether the defect L is included in each image (123, 127) may be given as appropriate. For example, the value of the detection parameter may be specified by operator input or may be given as a set value within the program. In this embodiment, a plurality of detection parameter candidates 60 are provided. The detection parameter candidate 60 is an example of a "parameter candidate" of the present invention. Each detected parameter candidate 60 indicates a value of a detected parameter candidate. The value of each detection parameter candidate 60 may be given as appropriate.

The evaluation unit 114 uses each detection parameter candidate 60 to determine whether the defect L is shown in each additional image 123 and to determine whether the defect L is shown in each evaluation image 127. do. Based on the results of each determination, the evaluation unit 114 selects an additional image 123 that is determined to include the defect L and an evaluation image 127 that is determined to not include the defect L from among the plurality of detection parameter candidates 60. One detection parameter candidate 61 with the smallest number of is identified. Thereby, the evaluation unit 114 automatically optimizes the detection parameters used for detecting the defect L.

Then, the evaluation unit 114 determines the number of additional images 123 that are determined to include the defect L (hereinafter, the number of additional images 123 that are erroneously determined) in the determination using the one identified detection parameter candidate 61. ) is greater than or equal to a first threshold value. In addition, the evaluation unit 114 determines the number of evaluation images 127 that are determined to not include the defect L (hereinafter also referred to as the number of erroneously determined evaluation images 127 in the determination using the one identified detection parameter candidate 61). ) is greater than or equal to a second threshold. The first threshold value and the second threshold value may each be determined as appropriate.

Alternatively, the evaluation unit 114 determines the percentage of additional images 123 that are determined to include the defect L (hereinafter referred to as erroneous It may also be determined whether the ratio of the additional image 123 for determination) is greater than or equal to a first threshold value. In addition, the evaluation unit 114 determines the percentage of evaluation images 127 that are determined to not include the defect L (hereinafter referred to as erroneous determination) among the plurality of evaluation images 127 in the determination using the one identified detection parameter candidate 61. It may be determined whether the ratio of the evaluation image 127 (also referred to as the ratio of the evaluation image 127) is greater than or equal to a second threshold value.

The learning device 1 adds images for addition when the number or proportion of additional images 123 of false determination is equal to or greater than a first threshold, or when the number or proportion of evaluation images 127 of false determination is equal to or greater than a second threshold. 126 is executed. That is, the evaluation unit 114 evaluates whether the number or proportion of additional images 123 of false determinations is equal to or greater than the first threshold in the results of each determination using one identified detection parameter candidate 61, or the evaluation of false determinations. When the number or proportion of images 127 is equal to or greater than the second threshold, an evaluation value 125 is calculated for one or more additional images 123 that have been incorrectly determined.

Based on the calculated evaluation value 125, the extraction unit 115 extracts one or more additional images 123 that are evaluated to have a low degree of restoration from among the one or more additional images 123 that are determined to include the defect L. image 126 is extracted. The additional learning unit 116 determines one or more images included in the one or more extracted additional images 126 as the second learning image 126A according to the selection result received by the selection receiving unit 119. The additional learning unit 116 uses machine learning to generate restored images that match each of the one or more second learning images 126A, when each of the determined one or more second learning images 126A is given. Then, generative model 5 is further trained.

Furthermore, in this embodiment, each image (121, 123) is divided into a plurality of patch images (P1, P2) as shown in FIGS. 4 and 5. The generative model 5 is given for each patch image (P1, P2). For each patch image P1, when one or more first learning images 121 are given, the learning unit 111 generates a restored image that matches each of the given one or more first learning images 121 by machine learning. Each generative model 5 is trained as follows.

The generation unit 113 generates each restored additional image 124 corresponding to each additional image 123 for each patch image P2, using the corresponding generation model 5. Similarly, each evaluation image 127 is also divided into a plurality of patch images P3. The generation unit 113 generates each restored evaluation image 128 corresponding to each evaluation image 127 for each patch image P3 using the corresponding generation model 5.

For each patch image P2, the evaluation unit 114 uses each difference image 182 to determine whether the defect L is included in each additional image 123. Similarly, for each patch image P3, the evaluation unit 114 uses each difference image 183 to determine whether the defect L is included in each evaluation image 127. Furthermore, the evaluation unit 114 calculates the evaluation value 125 of each additional image 123 for each patch image P2. In the present embodiment, the evaluation unit 114 evaluates patches for which the number or proportion of additional images 123 of false determination is equal to or greater than a first threshold, or the number or proportion of evaluation images 127 of false determination is equal to or greater than a second threshold. Regarding the image P2, an evaluation value 125 of each additional image 123 is calculated.

The extraction unit 115 extracts one or more additional images 126 from the plurality of additional images 123 for each patch image P2 based on the calculated evaluation value 125. The number of additional images 126 extracted may be different for each patch image P2. The additional learning unit 116 determines one or more images included in the one or more extracted additional images 126 as the second learning image 126A. When given each of the one or more second learning images 126A determined by machine learning for each patch image P2, the additional learning unit 116 adapts to each of the one or more given second learning images 126A. Each generative model 5 is further trained to generate a restored image.

Note that in FIGS. 4 and 5, each image (121, 123, 127) is equally divided into 16 patch images (P1, P2, P3). In this way, by dividing each image (121, 123, 127) into equal parts, a plurality of patch images (P1, P2, P3) with uniform dimensions may be generated. However, the method of dividing the image is not limited to this example as long as it is unified among the images (121, 123, 127), and may be determined as appropriate depending on the embodiment. . At least some dimensions of the plurality of patch images (P1, P2, P3) may be different from other dimensions. Further, the number of patch images (P1, P2, P3) in one image (121, 123, 127) does not need to be limited to 16, and may be appropriately selected depending on the embodiment.

The storage processing unit 1110 performs a process of storing data in the storage unit 12. In this embodiment, the storage processing unit 1110 generates information regarding the finally constructed trained generative model 5 as the learning result data 129. The storage processing unit 1110 then stores the generated learning result data 129 in a predetermined storage area. The predetermined storage area may be, for example, the RAM in the control unit 11, the storage unit 12, the storage medium 91, an external storage device, or a combination thereof.

<Detection device>
Next, an example of the software configuration of the detection device 2 according to this embodiment will be described using FIG. 6. FIG. 6 is a diagram schematically illustrating an example of the software configuration of the detection device 2 according to the present embodiment.

The control unit 21 of the detection device 2 loads the detection program 220 stored in the storage unit 22 into the RAM. The control unit 21 then uses the CPU to interpret and execute instructions included in the detection program 220 loaded in the RAM, thereby controlling each component. As a result, as shown in FIG. 6, the detection device 2 according to the present embodiment is configured as a computer including an acquisition section 211, a generation section 212, a detection section 213, and an output section 214 as software modules. That is, in this embodiment, each software module of the detection device 2 is also realized by the control unit 21 (CPU), similarly to the learning device 1 described above.

The acquisition unit 211 acquires an observation image 221 obtained by observing the product R. The generation unit 212 is equipped with a trained generation model 5 by holding learning result data 129. The generation unit 212 refers to the learning result data 129 and sets the learned generation model 5. Then, the generation unit 212 generates a restored observation image 223 by restoring the observation image 221 by inputting the acquired observation image 221 into the learned generation model 5.

Based on the difference between the generated restored observation image 223 and the observation image 221, the detection unit 213 determines whether or not the defect is shown in the observation image 221 together with the product R (that is, whether the product R shown in the observation image 221 is defective). ) is detected. In this embodiment, the detection unit 213 indicates the difference between the observed image 221 and the restored observed image 223 by calculating the difference between the observed image 221 and the restored observed image 223, as in the learning device 1 described above. A difference image 225 is generated. The detection unit 213 uses the difference image 225 to determine whether a defect exists in the product R shown in the observed image 221.

In this embodiment, the observed image 221 is divided into a plurality of patch images P4 according to the division of each image (121, 123). The generation unit 212 sets the generation model 5 for each patch image P4. Then, the generation unit 212 inputs each patch image P4 of the observed image 221 to the corresponding generation model, thereby generating a restored observed image 223, which is the restored observed image 221, for each patch image P4. The detection unit 213 determines, for each patch image P4, whether a defect L exists in the product R shown in the observed image 221, based on the difference between the observed image 221 and the restored observed image 223. The output unit 214 outputs information regarding the result of the detection, that is, the result of determining whether the product R is good or bad.

<Others>
Each software module of the learning device 1 and the detection device 2 will be explained in detail in the operation example described later. In this embodiment, an example is described in which each software module of the learning device 1 and the detection device 2 is both implemented by a general-purpose CPU. However, some or all of the software modules described above may be implemented by one or more dedicated processors. Further, regarding the software configurations of the learning device 1 and the detection device 2, software modules may be omitted, replaced, or added as appropriate depending on the embodiment.

§3 Operation example [Learning device]
Next, an example of the operation of the learning device 1 will be described using FIGS. 7 and 8. 7 and 8 are flowcharts showing an example of the processing procedure of the learning device 1 according to the present embodiment. The processing procedure described below is an example of a learning method. However, the processing procedure described below is only an example, and each process may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, or added as appropriate depending on the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the first acquisition unit 110 and acquires one or more first learning images 121 in which the product R is captured.

In the present embodiment, basically, an image in which the product R is shown but no defects are shown, in other words, an image in which the product R (good product) with no defects is shown is used as the first learning image 121. However, as long as it is possible to construct a trained generative model 5 that generates a restored image that can be used to detect defects, an image in which the product R including the defect appears may be used as a part of the first learning image 121. .

The method of generating the first learning image 121 does not need to be particularly limited, and may be selected as appropriate depending on the embodiment. For example, a camera of the same type as camera CA and a defect-free product R are prepared, and the product R is photographed using the prepared camera under various conditions depending on the environment in which the product R is inspected. Thereby, the first learning image 121 in which the product R appears can be generated.

The first learning image 121 may be automatically generated by a computer operation, or may be manually generated by an operator operation. Further, the first learning image 121 may be generated by the learning device 1 or by another computer other than the learning device 1. When the learning device 1 generates the first learning image 121, the control unit 11 automatically or manually performs the above series of processes through the input device 14 to generate one or more images. A first learning image 121 is acquired. On the other hand, when the first learning image 121 is generated by another computer, the control unit 11 generates one or more first learning images 121 generated by the other computer, for example, via the network, the storage medium 91, etc. get. When acquiring a plurality of first learning images 121, some of them may be generated by the learning device 1, and the rest may be generated by another computer.

The number of first learning images 121 to be acquired does not need to be particularly limited, and may be determined as appropriate depending on the embodiment. For example, the number of first learning images 121 to be acquired may be determined to the extent that a provisional trained generative model 5 can be constructed. After acquiring one or more first learning images 121, the control unit 11 advances the process to the next step S102.

(Step S102)
In step S102, the control unit 11 operates as the learning unit 111 and performs machine learning of the generative model 5 using the acquired one or more first learning images 121. In this machine learning process, when each of the one or more first learning images 121 is given, the control unit 11 generates a restored image that matches each of the one or more given first learning images 121. Train generative model 5.

The generation model 5 is constituted by a learning model including a calculation parameter M used for calculation processing to convert an input image into a feature amount and generate an image obtained by restoring the input image from the feature amount obtained by the conversion. The machine learning method may be selected as appropriate depending on the type of learning model used as the generative model 5. In this embodiment, a machine learning method is employed to generate the generative model 5 configured by a neural network. As an example of machine learning, the control unit 11 constructs a learned generative model 5 using the following processing procedure.

FIG. 9 is a diagram schematically illustrating an example of the generative model 5 configured by a neural network. An autoencoder is a typical example of a neural network. An autoencoder is a neural network intended for dimensionality reduction.

The neural network including the generative model 5 illustrated in FIG. 9 includes an input layer 51, an intermediate (hidden) layer 52, and an output layer 53. However, the structure of the neural network does not need to be limited to such an example, and may be determined as appropriate depending on the embodiment. For example, the number of intermediate layers is not limited to one, and may be two or more.

The number of neurons (nodes) included in the input layer 51 corresponds to the number of pixels of the input image. The number of neurons included in the intermediate layer 52 corresponds to the number of dimensions of the feature amount. The number of neurons included in the output layer 53 corresponds to the number of pixels of the restored image. The number of neurons included in the intermediate layer 52 may be set smaller than the number of neurons included in each of the input layer 51 and the output layer 53. In the case of an autoencoder aimed at dimensional compression, the number of neurons included in the intermediate layer 52 is set to be smaller than the number of neurons included in each of the input layer 51 and the output layer 53. Accordingly, the number of dimensions of the feature amount is set smaller than the number of dimensions of the input image and restored image.

In a neural network, neurons in adjacent layers are appropriately connected, and a weight (connection weight) is set for each connection. In the generative model 5 illustrated in FIG. 9, each neuron is connected to all neurons in adjacent layers. However, the connection of neurons does not need to be limited to this example, and may be set as appropriate depending on the embodiment. A threshold value is set for each neuron, and basically, the output of each neuron is determined depending on whether the sum of the products of each input and each weight exceeds the threshold value. The weight of the connection between each neuron included in each layer 51 to 53 and the threshold value of each neuron are examples of the calculation parameter M of the generative model.

The control unit 11 prepares a neural network that constitutes the generative model 5. Neural network structure (e.g., number of layers, number of neurons included in each layer, connection relationship between neurons in adjacent layers, etc.), initial values of parameters (e.g., initial values of connection weights between each neuron, each The initial value of the neuron threshold value may be given by a template or may be given by an operator's input.

Next, the control unit 11 uses each learning image as training data and teacher data to execute learning processing for each neural network. This learning process may use a batch gradient descent method, a stochastic gradient descent method, a mini-batch gradient descent method, or the like. For example, in the first step, the control unit 11 inputs each learning image to the input layer 51 of each neural network, and executes the arithmetic processing of each neural network. That is, the control unit 11 inputs each learning image to the input layer 51 of each neural network, and determines firing of each neuron included in each layer 51 to 53 in order from the input side. Through this calculation process, the control unit 11 converts each learning image into a feature amount, and obtains from the output layer 53 an output value corresponding to the result of generating an image in which each learning image is restored from the feature amount obtained by the conversion. do. In the process of this calculation process, the output obtained from the intermediate layer 52 corresponds to the result of converting the input image into a feature amount.

In the second step, the control unit 11 calculates the error between the output value obtained from the output layer 53 and each learning image based on the loss function. The type of loss function does not need to be particularly limited, and may be selected as appropriate depending on the embodiment. A known loss function may be used as the loss function. In the third step, the control unit 11 uses the error of the calculated output value by the error back propagation method to calculate the calculation parameter M of each neural network, that is, between each neuron in each layer 51 to 53. The connection weights and the errors of the thresholds of each neuron are calculated. In the fourth step, the control unit 11 updates the weights of connections between neurons in each layer 51 to 53 and the threshold values of each neuron, based on each calculated error.

By repeating the first to fourth steps above, the control unit 11 adjusts the calculation parameter M so that the sum of the errors between the output value (restored image) output from the output layer 53 and each learning image becomes small. Adjust values. For example, the control unit 11 may repeat the adjustment of the calculation parameter M in the first to fourth steps until the sum of the errors between the output value output from the output layer 53 and each learning image becomes equal to or less than the threshold value. good. The threshold value may be set as appropriate depending on the embodiment.

Thereby, the control unit 11 constructs a trained neural network as a trained generative model 5 so that when each learning image is input to the input layer 51, a restored image matching each learning image is output from the output layer 53. can do. In the neural network, each input learning image is converted into a feature amount during the calculation process from the input layer 51 to the intermediate layer 52. Then, a restored image is generated from the obtained feature amounts through the calculation process from the intermediate layer 52 to the output layer 53. After constructing the trained generative model 5, the control unit 11 advances the process to the next step S103.

Note that in this embodiment, the control unit 11 divides each of the acquired one or more first learning images 121 into a plurality of patch images P1 before executing step S102. The division method and the number of patch images P1 do not need to be particularly limited, and may be determined as appropriate depending on the embodiment. For example, the control unit 11 may divide the first learning image 121 into equal parts by a specified number in each of the vertical and horizontal directions. Thereby, the first learning image 121 may be divided into a plurality of patch images P1 in a grid pattern. The number of divisions may be specified by operator input, or may be given in advance within the program. Furthermore, when dividing into a grid, some dimensions of the plurality of patch images P1 may be different from other dimensions. Note that if the first learning image 121 has already been divided into patch images P1, this division process may be omitted.

Then, in step S102, the control unit 11 constructs a learned generation model 5 for each patch image P1 through the machine learning process described above. That is, in this embodiment, when the control unit 11 is given the corresponding patch image P1 of the first learning image 121, it converts the given patch image P1 into a feature amount, and calculates the corresponding feature amount from the feature amount obtained by the conversion. A generative model 5 trained to restore an image matching the patch image P1 is constructed for each patch image P1. As a result, a trained generative model 5 is provided for each patch image P1.

However, when each image is divided into a plurality of patch images, a neural network does not necessarily have to be prepared for each patch image. For example, by adding a neuron that inputs the class of patch images to the input layer of the neural network, it is possible to configure a neural network that restores images corresponding to different patch images. Therefore, a common neural network may be used as a generation model for at least two of the plurality of patch images.

The type of neural network constituting the generative model 5 does not have to be limited to such a fully connected neural network with a multilayer structure. The generative model 5 may use a convolutional neural network including a convolution layer and a pooling layer. Further, as the generative model 5, an autoencoder or a neural network having a similar structure may be used.

(Step S103)
In step S103, the control unit 11 operates as the second acquisition unit 112 and acquires a plurality of additional images 123, each of which shows the product R.

In this embodiment, each additional image 123 is basically an image showing a product R (good product) with no defects, similarly to the first learning image 121 described above. Each additional image 123 may be generated in the same manner as the first learning image 121 described above. Furthermore, a part of the image generated by the above processing and showing the product R without defects may be used as the first learning image 121, and the rest may be used as the additional image 123. For example, the control unit 11 may acquire a plurality of images showing the product R without defects. Then, in step S101, a part of the acquired images may be acquired as the first learning image 121, and in step S103, the remaining of the acquired images may be acquired as the additional image 123. .

Furthermore, similarly to the first learning image 121, the additional image 123 may be generated by the learning device 1 or by another computer other than the learning device 1. The control unit 11 may acquire the plurality of additional images 123 by automatically or manually executing the series of processes described above through an operator's operation via the input device 14. Alternatively, the control unit 11 may acquire a plurality of additional images 123 generated by another computer, for example, via a network, the storage medium 91, or the like. Some of the plurality of additional images 123 may be generated by the learning device 1, and the rest may be generated by another computer.

The number of additional images 123 to be acquired does not need to be particularly limited, and may be determined as appropriate depending on the embodiment. After acquiring the plurality of additional images 123, the control unit 11 advances the process to the next step S104.

(Step S104)
In step S104, the control unit 11 operates as the third acquisition unit 117 and acquires a plurality of evaluation images 127, each of which shows the product R including the defect L.

The method for generating each evaluation image 127 may be the method for generating each image (121, 123) described above, except that a product R including a defect L is prepared. That is, a camera of the same type as camera CA and a product R including the defect L are prepared, and the product R is photographed using the prepared camera under various conditions depending on the environment in which the product R is inspected. Thereby, each evaluation image 127 in which the product R including the defect L is captured can be generated. Alternatively, the defect L may be added by image processing. That is, each evaluation image 127 may be generated by adding the defect L to an image of the product R without defects through image processing. In this case, the image showing the defect-free product R may be selected from each image (121, 123).

Like each image (121, 123), each evaluation image 127 may be automatically generated by a computer operation, or may be manually generated by an operator's operation. Further, the generation of each evaluation image 127 may be performed by the learning device 1 or by another computer other than the learning device 1. The control unit 11 may acquire the plurality of evaluation images 127 by automatically or manually performing the above series of processes through an operator's operation via the input device 14. Alternatively, the control unit 11 may acquire a plurality of evaluation images 127 generated by another computer, for example, via a network, the storage medium 91, or the like. Some of the plurality of evaluation images 127 may be generated by the learning device 1, and the rest may be generated by another computer.

The number of evaluation images 127 to be acquired does not need to be particularly limited, and may be determined as appropriate depending on the embodiment. After acquiring the plurality of evaluation images 127, the control unit 11 advances the process to the next step S105.

(Step S105 and Step S106)
In step S105, the control unit 11 operates as the generation unit 113, and provides each of the acquired plurality of additional images 123 to the generative model 5 trained in step S102, thereby generating each of the plurality of additional images 123. A plurality of restored images 124 for restoration and addition are each generated. In step S106, the control unit 11 operates as the generation unit 113, and restores each of the plurality of evaluation images 127 by giving each of the plurality of acquired evaluation images 127 to the generative model 5 trained in step S102. Each of the plurality of restoration evaluation images 128 is generated.

Specifically, the control unit 11 refers to the machine learning result in step S102 and prepares the learned generative model 5. In step S105, the control unit 11 inputs each additional image 123 to the trained generative model 5. Thereby, the control unit 11 generates each restored additional image 124 by restoring each additional image 123. Similarly, in step S106, the control unit 11 inputs each evaluation image 127 to the trained generative model 5. Thereby, the control unit 11 generates each restored evaluation image 128 by restoring each evaluation image 127.

Note that in this embodiment, the generation model 5 is prepared for each patch image. Accordingly, in this embodiment, the control unit 11 divides each additional image 123 into a plurality of patch images P2. Further, the control unit 11 divides each evaluation image 127 into a plurality of patch images P3. The method of dividing the image is the same as that for the first learning image 121. If each image (123, 127) has already been divided into patch images (P2, P3), these division processes may be omitted.

In step S105, the control unit 11 uses the corresponding generation model 5 for each patch image P2 to generate each restored additional image 124 corresponding to each additional image 123. That is, the control unit 11 generates each patch image of each restored additional image 124 by providing each patch image P2 of each additional image 123 to the corresponding generation model 5. Similarly, in step S106, the control unit 11 generates each restored evaluation image 128 corresponding to each evaluation image 127 using the corresponding generation model 5 for each patch image P3. That is, the control unit 11 generates each patch image of each restored evaluation image 128 by providing each patch image P3 of each evaluation image 127 to the corresponding generation model 5. After generating each restoration addition image 124 and each restoration evaluation image 128, the control unit 11 advances the process to the next step S107.

(Step S107 and Step S108)
In step S107, the control unit 11 operates as the evaluation unit 114, and determines whether the defect L is reflected in each additional image 123 based on the degree of difference between each additional image 123 and each restored additional image 124. Determine whether In step S108, the control unit 11 operates as the evaluation unit 114, and determines whether the defect L is captured in each evaluation image 127 based on the degree of difference between each evaluation image 127 and each restoration evaluation image 128. do.

The method of detecting the presence or absence of the defect L depending on the degree of the difference may be set as appropriate depending on the embodiment. In this embodiment, in step S107, the control unit 11 generates each difference image 182 by calculating the difference between each additional image 123 and each restored additional image 124. The control unit 11 uses each difference image 182 to determine whether the defect L is included in each additional image 123. Similarly, the control unit 11 generates each difference image 183 by calculating the difference between each evaluation image 127 and each restored evaluation image 128. The control unit 11 uses each difference image 183 to determine whether the defect L is included in each evaluation image 127.

Furthermore, in this embodiment, a plurality of detection parameter candidates 60 are provided for determining whether the defect L is included in each image (123, 127). Each detected parameter candidate 60 indicates a value of a detected parameter candidate. The value of each detection parameter candidate 60 may be given as appropriate. For example, the value of each detection parameter candidate 60 may be obtained by input from an operator. Further, for example, the value of each detection parameter candidate 60 may be given in advance within the program. The detection parameters may be appropriately set according to the algorithm for detecting the presence or absence of the defect L from each difference image (182, 183).

Here, with further reference to FIG. 10, an example of a process for detecting the defect L and an example of a detection parameter will be described. FIG. 10 is a diagram schematically illustrating an example of the process of detecting the defect L. First, in the first step, the control unit 11 provides the image I10 showing the product R to the generation model 5 to generate the restored image I11. The image I10 corresponds to each of the above images (123, 127), and the restored image I11 corresponds to each of the above restored images (124, 128). Specifically, the image I10 corresponds to each patch image (P2, P3) of each image (123, 127), and the restored image I11 corresponds to each patch image of each restored image (124, 128). The generation process corresponds to each of steps S105 and S106 described above.

Next, in a second step, the control unit 11 generates a difference image I12 by calculating the difference between the image I10 and the restored image I11. The difference image I12 corresponds to each of the above-mentioned difference images (182, 183). Specifically, the difference image I12 corresponds to each patch image of the above-mentioned difference images (182, 183). A pixel that is different between the image I10 and the restored image I11 has a larger pixel value in the difference image I12. On the other hand, the pixel value in the difference image I12 becomes smaller when there is no difference between the image I10 and the restored image I11. In this embodiment, for convenience of explanation, the larger the difference between a pixel between the image I10 and the restored image I11, the larger the actual pixel value of the corresponding pixel in the difference image I12, and the smaller the difference is, the larger the actual pixel value of the corresponding pixel in the difference image I12. Assume that the actual pixel value of the corresponding pixel within is also smaller. However, "the pixel value is large" and "the pixel value is small" each indicate the relationship between the difference between the image I10 and the restored image I11, and the actual pixel value of the pixel in the difference image I12. It doesn't have to be compatible. For example, the difference image I12 may be calculated such that the larger the difference between pixels, the smaller the actual pixel value of the corresponding pixel, and the smaller the difference between the pixels, the larger the actual pixel value of the corresponding pixel.

In the example of FIG. 10, the pixels that are more different between the image I10 and the restored image I11 are whiter in the difference image I12, and the pixels that are less white are blacker. For example, when the value of each pixel is expressed in 256 gradations, the maximum value of the pixel value of the pixels of the difference image I12 may be "255", and the minimum value may be "0". In this case, the larger the pixel value of the pixel of the difference image I12, the more difference occurs between the image I10 and the restored image I11, and the smaller the pixel value of the pixel of the difference image I12, the more the difference between the image I10 and the restored image This shows that there is no difference between the image I11 and the image I11. However, the relationship between the difference that occurs between the image I10 and the restored image I11 and the pixel values of the difference image I12 is not limited to this example. For example, the relationship between the degree of difference between the image I10 and the restored image I11 and the pixel value of the difference image I12 may be the opposite.

If a feature (in this embodiment, defect L) that is not shown in the learning image used for machine learning of the generative model 5 or has a low possibility of appearing is shown in the image I10, the reproducibility of that feature in the restored image I11 is Since the image I10 is low, a relatively large difference may occur between the image I10 and the restored image I11. However, the cause of the difference between the image I10 and the restored image I11 is not limited to the fact that such a defect L appears in the image I10. Another cause of a relatively large difference between the image I10 and the restored image I11 is that, for example, the state of a predetermined feature (product R in this embodiment) appearing in a given input image is determined by machine learning. One example of this is that the appearance of the predetermined features is different from the appearance of the predetermined features in the training images used. In this case, the appearance of the product R that appears in the image I10 is not completely reproduced in the restored image I11, which may cause a relatively large difference between the image I10 and the restored image I11.

Furthermore, for example, the generative model 5 compresses the input image into low-dimensional features. At this time, information of the input image may be partially lost. Therefore, some error may occur between the input image and the restored image. This restoration noise can be an example of the cause of the difference between the image I10 and the restored image I11. Therefore, in addition to the difference caused by the defect L, differences based on a plurality of factors may appear in the difference image I12. However, the difference due to this noise is of a lower magnitude than the difference due to the above two factors. Therefore, these differences can be distinguished based on the pixel values of the difference image I12.

Therefore, in the third step, the control unit 11 uses the threshold value 65 to binarize each pixel of the difference image I12. For example, the control unit 11 converts the pixel values of pixels whose pixel values are equal to or greater than the threshold value 65 to "255", and converts the pixel values of pixels whose pixel values are less than the threshold value 65 to "0". “More than” may be replaced with “more than” and “less than” may be replaced with “less than”. The same applies to the following description. Thereby, the control unit 11 can generate the binarized image I13. By appropriately setting the threshold value 65, it is possible to obtain a binarized image I13 in which relatively low-level differences such as those caused by the noise are excluded from the original difference image I12.

Differences due to the defect L and differences due to insufficient learning may appear in the binarized image I13. Among the causes of these differences, the defect L may have attributes related to shape, such as area, width, height, circumference, aspect ratio, and circularity. That is, when a defect L exists in the product R shown in the image I10, the corresponding position in the binarized image I13 is an area where white "255" pixels (hereinafter also referred to as white pixels) are gathered, A region having the same shape-related attributes as the defect L appears. Therefore, by setting the threshold value 66 for the attribute related to the shape, it is possible to determine whether the defect L is included in the binarized image I13.

As an example of the process, in the fourth step, the control unit 11 identifies the area of continuous white pixels in the binarized image I13 as one area, and determines whether each area of white pixels satisfies the threshold value 66. Determine whether Then, the control unit 11 leaves the area that satisfies the threshold value 66 as it is, and converts the pixel values of pixels in the area that does not satisfy the threshold value 66 to "0". For example, when the threshold value 66 is set for area, the control unit 11 determines whether the area of each region of white pixels is equal to or larger than the threshold value 66. Then, the control unit 11 converts the pixel values of pixels within the area whose area is less than the threshold value 66 to "0". Thereby, the control unit 11 can generate the detected image I14. By appropriately setting the threshold value 66, it is possible to obtain a detected image I14 in which white areas that do not satisfy the attributes of the defect L are excluded from the binarized image I13.

In the fifth step, the control unit 11 determines whether the defect L is included in the image I10, depending on whether a white pixel area exists in the detected image I14. Specifically, when a region of white pixels exists in the detected image I14, the control unit 11 determines that the defect L is reflected in the corresponding region of the image I10. On the other hand, if there is no area of white pixels in the detected image I14, the control unit 11 determines that the defect L is not included in the image I10. Thereby, the control unit 11 ends the process related to detecting the defect L.

In the processing procedure for detecting the defect L described above, each threshold value (65, 66) is an example of a detection parameter. That is, in this embodiment, each detection parameter candidate 60 is configured by a combination of threshold values (65, 66). However, the method and detection parameters for detecting the defect L do not need to be limited to these examples, and may be appropriately selected depending on the embodiment. In step S107, the control unit 11 uses each detection parameter candidate 60 to execute the processing of the first to fifth steps regarding detection of the defect L, so that the defect L is reflected in each additional image 123. Determine whether or not the Similarly, in step S108, the control unit 11 uses each detection parameter candidate 60 to execute the processes of the first to fifth steps regarding detection of the defect L, so that the defect L is displayed in each evaluation image 127. Determine whether or not the image is captured.

Note that in this embodiment, each image (123, 127) is divided into a plurality of patch images (P2, P3). Accordingly, in step S107 according to the present embodiment, the control unit 11 selects each additional image based on the degree of difference between each additional image 123 and each restored additional image 124 for each patch image P2. It is determined whether the defect L is captured in the image 123. Similarly, in step S108 according to the present embodiment, the control unit 11 determines whether the defect L is present in each evaluation image 127 based on the degree of difference between each evaluation image 127 and each restoration evaluation image 128 for each patch image P3. Determine whether the image is visible or not. When each determination process is completed, the control unit 11 advances the process to the next step S109.

(Step S109)
In step S109, the control unit 11 operates as the evaluation unit 114. The control unit 11 optimizes detection parameters for determining whether the defect L is shown in the image based on the determination results of each step (S107, S108). Specifically, the control unit 11 selects the additional image 123 and the defect L determined to include the defect L from among the plurality of detection parameter candidates 60 based on the determination results of each step (S107, S108). One detection parameter candidate 61 with the least number of evaluation images 127 determined not to be captured is identified.

The number of additional images 123 determined to include defect L is the number of images erroneously determined to include product R that includes defect L, even though the product R that does not include defect L is included in the image. corresponds to the number of The number of evaluation images 127 that were determined not to contain the defect L is the number of images that were erroneously determined to contain the product R that does not contain the defect L, even though the product R that contains the defect L is shown. corresponds to a number. In the process related to the detection of the defect L, if the value of each threshold value (65, 66) is changed, the area of the white image appearing in the detected image I14 may change. That is, changing the value of the detection parameter may change the number of images (123, 127) that are incorrectly determined. Identifying one detection parameter candidate 61 with the least number of false determinations in step S109 corresponds to optimizing the detection parameters used to detect the defect L.

Here, an example of a process for identifying the optimal detection parameter candidate 61 will be described with further reference to FIG. 11. FIG. 11 is a diagram showing an example of hypothetical determination results in steps S107 and S108. In the example of FIG. 11, the determination results are expressed in a table format, and one record in the table corresponds to the determination result of one detection parameter candidate 60. However, the data format of the determination result is not limited to this example, and may be selected as appropriate depending on the embodiment.

Each record illustrated in FIG. 11 has fields related to detection parameters and determination results, respectively. “Number” indicates the identifier of each detection parameter candidate 60. "Binarization threshold" indicates the value of threshold 65, which is a reference for binarization. “Area threshold” indicates an example of a threshold 66 that is a reference for determining the presence or absence of a defect L. The "number of evaluation image errors" indicates the number of evaluation images 127 that were determined in step S108 to not include the defect L. The "number of additional images with errors" indicates the number of additional images 123 that were determined to include the defect L in step S107. “Determination result of each additional image” indicates the individual determination result of each additional image 123. "NG" indicates that it has been determined that the defect L is captured. "OK" indicates that it has been determined that the defect L is not captured. "Evaluation value of each additional image" indicates the evaluation value 125 of each additional image 123 calculated in step S112, which will be described later. Calculation of the evaluation value 125 will be described later.

The control unit 11 specifies the optimal detection parameter candidate 61 based on the determination result of each detection parameter candidate 60. As illustrated in FIG. 11, when two detection parameter candidates 60 with numbers "0" and "1" are given and the determination result shown in each record is obtained, the control unit 11 selects the detection parameter candidate with the least number of errors. The detection parameter candidate 60 with number "1" is identified as the optimal detection parameter candidate 61.

Note that in this embodiment, each image (123, 127) is divided into a plurality of patch images (P2, P3). Furthermore, each patch image P3 of the evaluation image 127 corresponds to the patch image P2 of the additional image 123. Accordingly, in step S109 according to the present embodiment, the control unit 11 determines the optimal detection parameter from the plurality of detection parameter candidates 60 for each patch image P2 (corresponding patch image P3) based on each determination result. Candidate 61 is identified. After identifying the optimal detection parameter candidate 61, the control unit 11 advances to the next step S110.

(Step S110 and Step S111)
In step S110, the control unit 11 operates as the evaluation unit 114, and determines that the number of falsely determined additional images 123 is greater than or equal to the first threshold in the determination result using the identified one detection parameter candidate 61. Determine whether or not. Furthermore, the control unit 11 determines whether or not the number of erroneously determined evaluation images 127 is equal to or greater than a second threshold value in the result of determination using the one identified detection parameter candidate 61.

The first threshold value and the second threshold value may each be determined as appropriate. For example, the first threshold value and the second threshold value may each be specified by operator input. Further, for example, the first threshold value and the second threshold value may each be given as set values within the program. The first threshold value and the second threshold value may be set to different values separately, or may be set to the same value.

In addition, in step S110, the control unit 11 determines that, in the result of the determination using the one identified detection parameter candidate 61, the proportion of the falsely determined additional images 123 among the plurality of additional images 123 is a first threshold value. It may be determined whether or not the value is greater than or equal to the value. Similarly, the control unit 11 determines whether or not the proportion of incorrectly determined evaluation images 127 among the plurality of evaluation images 127 is greater than or equal to a second threshold in the determination result using the one identified detection parameter candidate 61. It may be determined whether

In step S111, the control unit 11 determines the branch destination of the process based on the determination result in step S110. In the present embodiment, when the number or proportion of additional images 123 of false judgments is greater than or equal to the first threshold, or the number or proportion of evaluation images 127 of false judgment is greater than or equal to the second threshold, the control unit 11 , the process advances to step S112. On the other hand, if this is not the case, the control unit 11 advances the process to step S117. However, the branch criteria do not need to be limited to such an example, and may be set as appropriate depending on the embodiment. For example, when at least one of the following is satisfied: the number or proportion of additional images 123 of false judgments is equal to or greater than a first threshold, and the number or proportion of evaluation images 127 of false discrimination is equal to or greater than a second threshold. , the control unit 11 may proceed to step S112. If not, the control unit 11 may proceed to step S117. Also, for example, it is possible to determine whether the number or proportion of additional images 123 that are incorrectly determined is equal to or greater than a first threshold, and whether or not the number or proportion of evaluation images 127 that are falsely determined is equal to or greater than a second threshold. Either one of these determinations may be omitted.

Note that in this embodiment, each image (123, 127) is divided into a plurality of patch images (P2, P3). Accordingly, in the present embodiment, there is a patch image P2 in which the number or proportion of additional images 123 of false determinations is equal to or higher than the first threshold value, or the number or proportion of evaluation images 127 of false determinations is the second If there is a corresponding patch image P3 that is equal to or greater than the threshold value, the control unit 11 advances the process to step S112. On the other hand, if such patch images (P2, P3) do not exist, the control unit 11 advances the process to step S117.

(Step S112)
In step S112, the control unit 11 sets the evaluation value 125 of each additional image 123 according to the difference between each additional image 123 and each generated restored additional image 124, An evaluation value 125 for the degree of restoration of each additional image 123 in step 124 is calculated. In this embodiment, the control unit 11 uses each difference image 182 to calculate the evaluation value 125 of each additional image 123.

Here, an example of a method for calculating the evaluation value 125 of each additional image 123 will be described with further reference to FIG. 12. FIG. 12 is a diagram schematically illustrating an example of the process of calculating the evaluation value 125. In this embodiment, the control unit 11 generates each difference image 182 in step S107. A pixel with a large pixel value in each difference image 182 corresponds to a pixel with a large difference between each additional image 123 and each restored additional image 124. A pixel with a small pixel value in each difference image 182 corresponds to a pixel with a small difference between each additional image 123 and each restored additional image 124. The control unit 11 calculates an evaluation value 125 indicating an evaluation of the degree of restoration based on the degree of the difference appearing in the pixel value of each pixel.

In the present embodiment, as an example of a calculation method based on this viewpoint, the control unit 11 determines that the larger the pixel value of each pixel is, the higher the evaluation value 125 is, and the contribution to the evaluation value 125 of pixels that is equal to or higher than the reference pixel value. The evaluation value 125 is calculated so that it is larger than the contribution given to the evaluation value 125 of pixels whose degree is less than the reference pixel value. Specifically, the control unit 11 creates a frequency distribution (density histogram) regarding pixel values for the images forming the difference image 182, as shown in FIG. In the example of FIG. 12, it is assumed that the pixel values of each difference image 182 are expressed in 256 gradations. The "frequency" of the density histogram illustrated in FIG. 12 indicates the number of pixels having the same pixel value. Subsequently, the control unit 11 determines the minimum value of the pixel values occupying a predetermined upper rank in the created density histogram as the reference pixel value. Then, the control unit 11 extracts pixels having a reference pixel value or higher (that is, a predetermined percentage of pixels at the top) from the pixels in each difference image 182. The predetermined ratio may be specified by an operator's input, a set value in a program, or the like. Then, the control unit 11 calculates the average value or total value of the pixel values of the extracted pixels as the evaluation value 125. Note that the control unit 11 may calculate the average value or total value using a frequency greater than or equal to the reference pixel value in the density histogram. As a result, in this embodiment, the larger the calculated evaluation value 125, the larger the difference between the additional image 123 and the restored additional image 124. In other words, the difference between the additional image 123 and the restored additional image 124 is larger. Indicates that the degree of restoration is low.

However, the method for calculating the evaluation value 125 does not have to be limited to this example. The relationship between the difference between each additional image 123 and each restoration additional image 124 and the evaluation value 125 is that the evaluation value 125 indicates that the degree of restoration is low as the difference is large, and may be determined as appropriate so that the smaller the value, the higher the degree of restoration. For example, contrary to the calculation method described above, the evaluation value 125 may be calculated such that the lower the degree of restoration, the smaller the evaluation value 125, and the higher the degree of restoration, the larger the evaluation value 125. As an example of the calculation method, the control unit 11 may calculate the average value or the reciprocal of the total value of the pixel values of the pixels extracted by the above process as the evaluation value 125.

In this embodiment, in step S107, the control unit 11 determines whether the defect L is included in each additional image 123 based on the difference between each additional image 123 and each restored additional image 124. Judging. It is determined that the defect L is included because the difference between the additional image 123 and the restored additional image 124 is large. In other words, the degree of restoration of the additional image 123 in the restored additional image 124 is low. Show that. On the other hand, if it is determined that the defect L is not captured, the difference between the additional image 123 and the restored additional image 124 is small.In other words, the difference between the additional restored image 124 and the restored additional image 124 is small. Indicates that the degree is high.

Therefore, in the present embodiment, the control unit 11 calculates the evaluation value 125 for at least one or more additional images 123 that are determined to include the defect L among the plurality of additional images 123. On the other hand, the evaluation value 125 of the additional image 123 determined not to include the defect L may be calculated arbitrarily. That is, as shown in FIG. 11, the control unit 11 may calculate the evaluation value 125 also for the additional image 123 for which it is determined that the defect L does not appear. Alternatively, the control unit 11 may omit calculation of the evaluation value 125 for the additional image 123 determined to not include the defect L. When calculating the evaluation value 125 of each additional image 123 regardless of whether the defect L is captured, the control unit 11 executes the process of this step S112 before the process of the above step S107. Good too. The process of step S112 may be executed at any timing after the process of step S105 is executed.

Further, in this embodiment, each additional image 123 is divided into a plurality of patch images P2. Accordingly, in the present embodiment, the control unit 11 determines that the number or proportion of additional images 123 of false determination is equal to or greater than the first threshold, or that the number or proportion of evaluation images 127 of false determination is equal to or higher than the second The evaluation value 125 of each additional image 123 is calculated for each patch image P2 that is equal to or greater than the threshold value. After calculating the evaluation value 125 of each additional image 123, the control unit 11 advances the process to the next step S113.

(Step S113)
In step S113, the control unit 11 operates as the extraction unit 115, and selects one or more additional images 126 that are evaluated to have a low degree of restoration from the plural additional images 123 based on the calculated evaluation value 125. Extract from.

The criteria for evaluating that the degree of restoration is low does not need to be particularly limited, and may be set as appropriate depending on the embodiment. In this embodiment, the larger the evaluation value 125 calculated in step S112, the lower the degree of restoration of the additional image 123 in the additional restoration image 124. Therefore, for example, the control unit 11 may extract the additional image 123 with the highest evaluation value 125 (that is, the lowest evaluation for the degree of restoration) as the additional image 126. Further, for example, the control unit 11 may extract a predetermined number of additional images 123 as additional images 126 in order from those with higher evaluation values 125 (that is, those with lower evaluations). The predetermined number may be determined as appropriate based on operator input, set values in the program, etc. Further, for example, an additional image 123 whose evaluation value 125 is larger than a threshold (that is, whose evaluation is lower than the threshold) may be extracted as an additional image 126. The threshold value that serves as the criterion for determination may be determined as appropriate based on operator input, set values in the program, and the like.

In this embodiment, in step S107, the control unit 11 determines whether the defect L is included in each additional image 123 based on the difference between each additional image 123 and each restored additional image 124. Judging. Accordingly, in the present embodiment, the control unit 11 selects from the one or more additional images 123 determined to include the defect L based on the calculated evaluation value 125 in step S113. One or more additional images 126 that are evaluated to have a low degree of restoration are extracted.

Furthermore, in this embodiment, each additional image 123 is divided into a plurality of patch images P2. Accordingly, in the present embodiment, the control unit 11 extracts one or more additional images 126 from the plurality of additional images 123 for each patch image P2 based on the calculated evaluation value 125. After extracting one or more additional images 126, the control unit 11 advances the process to the next step S114.

(Step S114 and Step S115)
In step S<b>114 , the control unit 11 operates as the display control unit 118 and causes the display device 15 to display each of the extracted one or more additional images 126 . The display device 15 is an example of the "display device" of the present invention. In step S115, the control unit 11 operates as the selection reception unit 119, and for each of the one or more additional images 126 displayed on the display device 15, the control unit 11 selects each of the one or more additional images 126 in the steps described below. A selection of whether to use it for machine learning in S116 is accepted.

Here, an example of the display screen of the extracted additional image 126 will be described with further reference to FIGS. 13 and 14. 13 and 14 are diagrams schematically illustrating an example of a display screen according to this embodiment. An example of the display screen according to this embodiment has three areas 400 to 402. In the area 400, each additional image 123 including the extracted additional image 126 is displayed. In the area 401, a list of a plurality of additional images 123 including the extracted additional image 126 and the evaluation value 125 of each additional image 123 is displayed. In the area 402, a list of learning images used for machine learning in step S116, which will be described later, is displayed. Before accepting the selection in step S115, the list of area 402 includes the extracted additional image 126 in addition to the first learning image 121 used in step S102.

In this embodiment, the generative model 5 acquires the ability to restore an image of the product R that does not include the defect L, regardless of the presence or absence of the defect L, through machine learning. Thereby, the trained generative model 5 is used for detecting the defect L. Therefore, it is preferable that the additional image 123 added to use machine learning does not include the defect L. However, for example, due to human error or the like, there is a possibility that an image in which a defect L that is not desired to be reproduced in the generation model 5 is captured as the additional image 123 in step S103. Similarly to the case where the ability of the generation model 5 to reproduce the state of the product R without the defect L appearing in the additional image 123 is insufficient, also when the defect L is shown in the additional image 123, the evaluation value 125 The evaluation of the degree of restoration indicated by may be low. Therefore, when the image in which the defect L is captured is acquired as the additional image 123, there is a possibility that the additional image 123 in which the defect L is captured is extracted as the additional image 126 by the process of step S113.

Regarding this point, in this embodiment, the control unit 11 displays the extracted additional image 126 in the area 400 in step S114. Thereby, the operator can visually confirm the state of the product R appearing in the additional image 126 displayed in the area 400. Then, by deleting an additional image 126 that is desired not to be used for machine learning from the list in the area 402 due to the fact that the defect L is included in the image, the operator can perform machine learning on the target additional image 126. You can choose not to use it. That is, in the present embodiment, in step S115, the control unit 11 selects each of the one or more additional images 126 for each of the extracted one or more additional images 126 via the list display in the area 402. Accepts the selection of whether or not to use it for machine learning.

Further, an example of the display screen according to this embodiment has two operation buttons (405, 406). The operation button 405 is used to add an additional image 123 to be added to the list of learning images to be used displayed in the area 402 from the list of additional images 123 displayed in the area 401. The operator can add the desired additional image 123 to the list in the area 402 by selecting the desired additional image 123 from the list displayed in the area 401 and operating the operation button 405. The operation button 406 is used to perform machine learning on the generative model 5 using the images displayed in the list in the area 402. The control unit 11 advances the process to step S116 in response to the operation of the operation button 406. However, the display screen for displaying the extracted additional image 126 and accepting the selection of whether or not to use the extracted additional image 126 for machine learning is not limited to this example. It may be designed as appropriate depending on the embodiment.

Note that in this embodiment, each additional image 126 (123) is divided into a plurality of patch images P2. Accordingly, in this embodiment, as shown in FIG. 13, the control unit 11 further displays grid lines G1 at the boundaries of each patch image P2 in the additional image 126 (123) displayed in the area 400. You may let them. Thereby, the visibility of patch division in each additional image 126 (123) can be improved. In addition, as shown in FIG. 14, the control unit 11 may further display a mark G5 indicating the corresponding patch image P2 extracted as the additional image 126 based on the evaluation value 125. Thereby, the visibility of the patch image P2, which has been evaluated as having a low degree of restoration, can be improved.

(Step S116)
In step S116, the control unit 11 operates as the additional learning unit 116. The control unit 11 determines the image selected to be used for machine learning among the one or more extracted additional images 126 as the second learning image 126A. When the control unit 11 is given each of the one or more second learning images 126A determined by machine learning, the control unit 11 generates a restored image that matches each of the one or more given second learning images 126A. Further train generative model 5.

In this embodiment, the control unit 11 uses the first learning image 121 and the second learning image 126A to derive the calculation parameter M of the neural network in the same manner as in step S102 above. Thereby, when the first learning image 121 and the second learning image 126A are given, the control unit 11 generates restored images that match the given first learning image 121 and the second learning image 126A, respectively. A trained and learned generative model 5 can be constructed.

Note that in this embodiment, the additional image 126 (123) is divided into a plurality of patch images P2 in accordance with the patch division of the first learning image 121. Accordingly, in the present embodiment, when each of the one or more second learning images 126A determined by machine learning is given for each patch image P2, the control unit 11 The generative model 5 is further trained to generate restored images suitable for each of the second learning images 126A. That is, when the trained generative model 5 constructed for each patch image P1 of the first learning image 121 is given the corresponding patch image P2 of the second learning image 126A, the trained generative model 5 can be restored to fit the given patch image P2. It is further trained to generate images. Thereby, the control unit 11 constructs a learned generation model 5 for each patch image (P1, P2) corresponding to each image (121, 123, 126).

As described above, in the present embodiment, the control unit 11 displays each of the one or more extracted additional images 126 through the list display of the area 402 on the example display screen illustrated in FIGS. 13 and 14. , a selection as to whether to use each of the one or more additional images 126 for machine learning is accepted. Accordingly, in this embodiment, the control unit 11 excludes the additional image 126 selected not to be used for machine learning from the training target in step S116. More specifically, in this embodiment, the control unit 11 performs machine learning of the generative model 5 using the images included in the list of the area 402 at the time the operation button 406 is operated.

When further training of the generative model 5 is completed, the control unit 11 repeats the process from step S105 using the trained generative model 5 obtained in step S116. By repeating the processes from step S105 to step S116, additional images 126 that are likely to contribute to appropriate improvement of the ability of the generative model 5 to reproduce images of the product R are extracted from the plurality of additional images 123. be able to. Then, one or more second learning images 126A included in the extracted additional images 126 are used for further machine learning of the generative model 5 to determine the appropriateness of the ability of the generative model 5 to reproduce the image of the product R. It is possible to make significant improvements.

Note that when the processing of steps S105 to S116 is repeated multiple times, the values of the first threshold value and the second threshold value may be changed as appropriate each time the processing of step S110 is repeated. Furthermore, when comparing the performance of the generative model 5 before and after executing the processes in steps S112 to S116, each image (123, 127) that is incorrectly determined in steps S107 and S108 after executing these processes Assume that the number of In this case, the control unit 11 may discard the processing results of steps S112 to S116 related to the increase in the number of erroneously determined images (123, 127), and proceed to step S117. Thereby, the control unit 11 may adopt the trained generative model 5 constructed immediately before as the final learning result.

(Step S117)
In step S117, the control unit 11 operates as the storage processing unit 1110 and generates information regarding the finally constructed trained generative model 5 as learning result data 129. In this embodiment, the learning images used for final machine learning are the first learning image 121 and the additional image according to the results of the conditional branching in step S111, the processing in step S113, and the processing in step S115. 123 (126). Note that in this embodiment, a learned generation model 5 is constructed for each patch image of the learning image. Therefore, the control unit 11 generates information regarding the learned generative model 5 constructed for each patch image of the learning image as the learning result data 129.

The learning result data 129 includes information that allows the values of the calculation parameters M forming the generative model 5 to be reproduced. In this embodiment, the generative model 5 is configured by a neural network. Therefore, for example, the learning result data 129 includes information that can reproduce the value of the calculation parameter M of the finally constructed trained neural network (generation model 5). Further, for example, the learning result data 129 may include a calculation parameter M derived for each patch image of the learning image. The control unit 11 stores the generated learning result data 129 in a predetermined storage area.

The predetermined storage area may be, for example, the RAM in the control unit 11, the storage unit 12, the storage medium 91, an external storage device, or a combination thereof. The external storage device may be, for example, a data server such as NAS (Network Attached Storage). In this case, the control unit 11 may use the communication interface 13 to store the learning result data 129 in the data server via the network. Further, the external storage device may be, for example, an external storage device connected to the learning device 1. After storing the learning result data 129, the control unit 11 ends the series of processes related to this operation example.

Note that the learning result data 129 generated by the learning device 1 may be provided to the detection device 2 at any timing. For example, the control unit 11 may transfer the learning result data 129 to the detection device 2 as the process of step S117 or separately from step S117. The detection device 2 may acquire the learning result data 129 by accepting this transfer. Further, for example, the detection device 2 may obtain the learning result data 129 by accessing the learning device 1 or the data server via the network using the communication interface 23. Further, for example, the detection device 2 may acquire the learning result data 129 via the storage medium 92. Further, for example, the learning result data 129 may be incorporated into the detection device 2 in advance.

[Detection device]
Next, an example of the operation of the detection device 2 will be described using FIG. 15. FIG. 15 is a flowchart showing an example of the processing procedure of the detection device 2 according to this embodiment. The processing procedure described below is an example of a detection method. However, the processing procedure described below is only an example, and each process may be changed as much as possible. The detection method may include the learning method and model selection method described above. Further, regarding the processing procedure described below, steps can be omitted, replaced, or added as appropriate depending on the embodiment.

(Step S301)
In step S301, the control unit 21 operates as the acquisition unit 211 and acquires the observation image 221 obtained by observing the product R. In this embodiment, a camera CA is connected to the detection device 2 via an external interface 27. Therefore, the control unit 21 acquires the observed image 221 from the camera CA. This observed image 221 may be a moving image or a still image.

Note that the route for acquiring the observed image 221 does not need to be limited to such an example, and may be appropriately selected depending on the embodiment. For example, the camera CA may be connected to another information processing device different from the detection device 2. In this case, the control unit 21 may acquire the observed image 221 via another information processing device. After acquiring the observation image 221, the control unit 21 advances the process to the next step S302.

(Step S302)
In step S302, the control unit 21 operates as the generation unit 212, refers to the learning result data 129, and sets the learned generation model 5. Then, the control unit 21 inputs the acquired observation image 221 into the trained generative model 5 and executes the arithmetic processing of the learned generative model 5, thereby creating a restored observation image 223 that is the restored observation image 221. generate. The restored observation image 223 is obtained as an output of the trained generative model 5, that is, as a calculation result of the trained generative model 5.

This step S302 may be processed in the same manner as steps S105 and S106 described above. That is, in the present embodiment, the control unit 21 refers to the learning result data 129 and prepares the trained generative model 5. The control unit 21 inputs the observed image 221 to the generation model 5. Thereby, the control unit 21 generates a restored observation image 223.

Note that in this embodiment, a learned generation model 5 is constructed for each patch image of the learning image. Accordingly, in the present embodiment, the control unit 21 may divide the acquired observation image 221 into a plurality of patch images P4 in the same manner as the patch division of each learning image (121, 123). Alternatively, in step S301 above, the control unit 21 may acquire the observed image 221 divided into a plurality of patch images P4. In step S302, the control unit 21 sets the generation model 5 for each patch image P4 by referring to the learning result data 129. Then, the control unit 21 inputs each patch image P4 of the observed image 221 to the corresponding generation model 5, and executes arithmetic processing of each generation model 5. Thereby, the control unit 21 generates a restored observation image 223, which is the restored observation image 221, for each patch image P4. In other words, the control unit 21 generates each patch image of the restored observation image 223 corresponding to each patch image P4 of the observation image 221. After generating the restored observation image 223, the control unit 21 advances the process to the next step S303.

(Step S303)
In step S303, the control unit 21 operates as the detection unit 213, and determines whether a defect is shown in the observation image 221 together with the product R based on the difference between the generated restored observation image 223 and the observation image 221 ( That is, it is detected whether or not there is a defect in the product R shown in the observation image 221.

This step S303 may be processed in the same manner as steps S107 and S108 described above. That is, in the first step, the control unit 21 generates the difference image 225 by calculating the difference between the observed image 221 and the restored observed image 223. In the second step, the control unit 21 binarizes each pixel of the difference image 225 using a binarization threshold. In the third step, the control unit 21 identifies the area of continuous white pixels in the obtained binarized image as one area, and attributes (for example, area, width, etc.) regarding the shape of each area of white pixels. It is determined whether the height, circumference, aspect ratio, circularity, etc.) satisfy a threshold value. The control unit 21 leaves the area that satisfies the threshold as it is, and converts the pixel values of pixels in the area that does not satisfy the threshold to "0". Thereby, the control unit 21 can generate a defect detection image. In the fourth step, the control unit 21 determines whether a defect exists in the product R shown in the observed image 221, depending on whether a white pixel area exists in the detected image. In this embodiment, through these series of processes, the control unit 21 determines whether or not there is a defect in the product R shown in the observation image 221 based on the difference between the observation image 221 and the restored observation image 223. can be detected.

The value of each threshold value in the detection process may be determined as appropriate depending on the embodiment. For example, the value of each threshold may be specified by operator input. Further, for example, the value of each threshold value may be given as a set value within the program. Further, for example, the detection parameter candidate 61 identified in step S109 above may be used for each threshold value. In this case, the detection parameter candidate 61 may be provided to the detection device 2 together with the learning result data 129 or at a timing different from the learning result data 129. The method for applying the detection parameter candidates 61 to the detection device 2 may be the same as that for the learning result data 129.

Note that in this embodiment, the observed image 221 is divided into a plurality of patch images P4. Accordingly, in the present embodiment, the control unit 21 performs a process based on the difference between the target patch image P4 of the observed image 221 and the corresponding patch image of the restored observed image 223 for each patch image P4 of the observed image 221. , it may be detected whether or not there is a defect in the product R shown in the target patch image P4. Upon detecting whether or not there is a defect in the product R shown in the observation image 221, the control unit 21 advances the process to the next step S304.

(Step S304)
In step S304, the control unit 21 operates as the output unit 214 and outputs information regarding the result of defect detection, that is, the result of determining whether the product R is good or bad.

The output format of the result of determining the quality of the product R is not particularly limited, and may be appropriately selected depending on the embodiment. For example, the control unit 21 may output the result of determining the quality of the product R to the display device 25 as is. Furthermore, if the presence of a defect is detected in step S303, the control unit 21 may issue a warning to notify that a defect has been discovered as an output process in step S304. At this time, the control unit 21 may notify the location of the defect as well as the fact that the defect has been discovered.

Further, the control unit 21 may execute a predetermined control process according to the result of determining whether the product R is good or bad as the output process in step S304. As a specific example, the detection device 2 may be connected to a manufacturing line where the product R is transported. In this case, the control unit 21 performs processing in step S304 to send a command to the manufacturing line to transport the product R determined to have a defect by a route different from that for the product R determined to have no defect. It may also be performed as output processing.

When the output process of the result of determining the quality of the product R is completed, the control unit 21 ends the process related to this operation example. Note that the control unit 21 may execute the series of processes from steps S301 to S304 described above each time the product R being transported on the production line enters the photographing range of the camera CA. Thereby, the detection device 2 can continuously perform an external appearance inspection of the product R transported on the production line.

[Features]
As described above, in this embodiment, the learned generative model 5 is provisionally constructed by machine learning using the first learning image 121 in step S102. Then, through the processing in step S112 and step S113, based on the results of the provisionally constructed trained generative model 5 demonstrating the restoration ability for each additional image 123, the restoration ability is appropriately improved. One or more additional images 126 that are likely to contribute can be automatically extracted from the plurality of additional images 123. Therefore, according to this embodiment, learning images can be appropriately selected. Accordingly, in the series of processes described above, the detection device 2 uses the trained generative model 5 constructed using the learning images appropriately selected in this way to detect the existence of a defect in the product R. It is possible to accurately detect whether or not to do so. As a result, the cost of detecting defects in product R can be reduced.

Furthermore, when a generative model is prepared for each patch image, it becomes more difficult to appropriately select a learning image to be used for machine learning of the generative model. In contrast, according to the present embodiment, at least part of the process of selecting learning images used for machine learning of the generative model 5 prepared for each patch image can be automated. Therefore, the cost associated with this selection can be significantly reduced. Furthermore, the detection device 2 can improve the accuracy of detecting defects by using the generation model 5 that is optimized for each patch image.

In addition, in this embodiment, the performance of defect detection using the constructed and trained generative model 5 is evaluated through the processing of steps S105 to S111, and if the performance of defect detection is evaluated to be low, the generation The performance can be improved by further machine learning of Model 5. Furthermore, when evaluating the performance of defect detection using the generative model 5, the detection parameters used for defect detection are automatically optimized in accordance with the restoration ability of the generative model 5 in the processing of steps S107 to S109. can do. As a result, the learning images used for machine learning are optimized to be suitable for defect detection processing. Accordingly, the detection device 2 can improve the accuracy of detecting defects by using the trained generative model 5 constructed using the learning images appropriately selected in this way.

Furthermore, in this embodiment, the results of the operator's visual inspection can be reflected in the selection of learning images used for machine learning through the processing in steps S114 and S115. Thereby, the additional image 126 in which the defect appears can be excluded from the machine learning target. As a result, it can be expected that the learning images used in machine learning will be selected in a manner suitable for defect detection processing. Accordingly, the detection device 2 can improve the accuracy of detecting defects by using the trained generative model 5 constructed by machine learning in which the learning images used are optimized. .

§4 Experimental Examples Next, experimental examples of the present invention will be explained. Using a general-purpose computer, we conducted an experiment to construct a trained generative model according to the following experimental conditions.

<Experimental conditions>
・Number of first learning images: 10 ・Number of additional images: 3435 ・Each first learning image and each additional image: Images showing products without defects ・Number of evaluation images: 42 ・Evaluation images: Image of product with defect/prescribed characteristics: Product (LED: light emitting diode)
・Other characteristics: Defects (contamination with foreign matter)
・Size of each image: 350 x 350
・Pixel value of each image: 256 gradations ・Pixel value range of difference image: 0 to 255
・Binarization threshold (threshold 65) value: 10 to 70
- Value of threshold (threshold 66) for area (number of pixels): 10 to 1000.

The processes of steps S101 to S109 and steps S112 to S116 were performed in the same manner as in the above embodiment except that the image was not divided into patch images. The trained generative model was constructed using a neural network. In step S112, the top 10% of pixels in terms of pixel values were extracted, and the average value of the pixel values of the extracted pixels was calculated as an evaluation value. Further, in step S113, the additional image with the highest evaluation value (that is, the lowest degree of restoration) was extracted. The extracted additional image was determined as the second learning image. A series of processes from step S105 to step S109 and step S112 to step S116 were repeated, and the number of images that were incorrectly determined was calculated in step S107 and step S108. In this way, we evaluated the performance of the trained generative model that was built through the process of repeating a series of machine learning processes.

FIG. 16 shows the results of evaluating the performance of a trained generative model constructed in the process of repeating machine learning processing using the experimental example. The "evaluation value of newly learned learning image" field indicates the evaluation value of the second learning image added to the machine learning target. The "results" field indicates the number of images that were incorrectly determined in step S107 and step S108. The "detection parameter" field indicates the optimal value of each threshold value specified in step S109.

As shown in FIG. 16, the restoring ability of the generative model can be appropriately improved by repeating the series of processes from step S105 to step S109 and step S112 to step S116 from the first time to the fifth time. Ta. This made it possible to reduce the number of images that were erroneously determined in step S107 and step S108. From this result, it was found that according to the present invention, it is possible to extract additional images that are highly likely to contribute to an appropriate improvement in restoration ability.

Furthermore, in the learned generation model constructed by the sixth processing, the number of additional images that are incorrectly determined in step S107 has increased. As a result, it was found that an excessive number of training images may lead to a decrease in the performance of defect detection using a trained generative model. In the case of this example experiment, the number of learning images can be optimized to "14" by the time the fifth process is completed, and 3439 out of 3487 images have defects. We were able to construct a generative model that has the ability to restore the product corresponding to the ability to correctly detect it. As a result, it was found that the trained generative model constructed using 14 training images can detect defects in products with high precision, and significantly reduce the cost of the detection.

§5 Modifications The embodiments of the present invention have been described in detail above, but the above descriptions are merely illustrative of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the invention. For example, the following changes are possible. In addition, below, the same code|symbol is used regarding the same component as the said embodiment, and description is abbreviate|omitted suitably about the same point as the said embodiment. The following modified examples can be combined as appropriate.

<5.1 Modification 1>
In the embodiment described above, an example is shown in which the present invention is applied to a situation where it is detected whether or not a defect exists in the product R using an image of the product R. Product R (particularly a product that is transported on a production line and does not contain any defects) is an example of the predetermined characteristic. Defects that may occur in product R are an example of other characteristics. However, the scope of application of the present invention is not limited to such an example of visual inspection. The present invention is applicable to all situations where some characteristics are detected from an image of a subject, such as a situation where it is detected whether a predetermined object is included in an image.

The predetermined feature may be other than the product R as long as it can be seen in the image. The predetermined feature may be a feature that itself can be extracted, a background that is not extracted, or a combination thereof. Similarly, other features other than the above defects may be used as long as they can be seen in the image. As an example other than the embodiments described above, the predetermined feature may be a movable object such as a person or a vehicle (eg, a car). Correspondingly, the other features may be, for example, the three-dimensional shape of the object, the optical flow (direction of movement), etc., which may change due to the movement. When the pixel value of each pixel in an observed image from a depth camera etc. includes the distance in the depth direction, the three-dimensional shape of the target object can be estimated from the difference in each pixel between the observed image and the restored image. . Further, when the state before movement is restored in the restored image, the optical flow of the target object can be estimated from the difference between the observed image and the restored image.

Furthermore, the predetermined feature may be in two states: a state in which it includes other features and a state in which it does not include other features. When a predetermined feature and other features have such a relationship, it is desirable that the probabilities of each state be biased. When the possible probabilities of each state are biased, it means that the possible probabilities of any state are higher than the possible probabilities of other states to the extent that it affects the reproducibility of the generative model. In addition, it is desirable that a feature not included in a state with a high probability of occurrence of a predetermined feature but included in a state with a low probability of occurrence is selected as the other feature. Furthermore, it is desirable that the predetermined features appearing in each learning image used for machine learning do not include the other features. As a result, by using a training image that shows a predetermined feature that does not include other features for machine learning of a generative model, the generative model can determine whether or not the predetermined feature that appears in the input image contains other features. Regardless, it is trained to generate a restored image that shows a predetermined feature that does not include the other features. Therefore, based on the difference between the restored image generated by the generative model and the input image, it is possible to detect whether a predetermined feature appearing in the input image includes another feature.

<5.2 Modification 2>
In the embodiment described above, the generative model 5 is configured by a neural network. However, the configuration of the generative model 5 does not have to be limited to this example. The type of generation model 5 is not particularly limited as long as it can convert images, and may be selected as appropriate depending on the embodiment. For example, the generative model 5 may be configured by an eigenvector matrix derived by principal component analysis (KL expansion), as in Non-Patent Document 3, for example.

The process of generating the restored images (124, 128) in steps S105 and S106 is replaced from the neural network calculation process to the eigenvector matrix calculation process. Further, in step S117, the control unit 11 generates information capable of reproducing the eigenvector matrix derived by principal component analysis (KL expansion) as learning result data 129, and stores the generated learning result data 129 in a predetermined memory. Save to area. Except for these points, the learning device 1 according to the embodiment described above can appropriately select learning images to be used for machine learning using the same processing procedure as in the embodiment described above. Note that, similarly to the above embodiment, the generated learning result data 129 may be provided to the detection device 2 at any timing.

The same applies to the detection device 2 according to the above embodiment. That is, the process of generating the restored observed image 223 in step S302 above replaces the neural network calculation process with the eigenvector matrix calculation process. Except for this point, the detection device 2 according to the embodiment described above can detect whether a defect exists in the product R shown in the observation image 221 using the same processing procedure as in the embodiment described above.

<5.3 Modification 3>
In the embodiment described above, the control unit 11 outputs information such as the extracted additional image 126 to the display device 15. However, the output destination of the information does not need to be limited to this example, and may be appropriately selected depending on the embodiment. For example, the control unit 11 may output information such as the extracted additional image 126 to a display device other than the display device 15, or may output the information to an output destination other than the display device (e.g., memory, output device).

<5.4 Modification 4>
In the above embodiment, each image (121, 123, 127, 221) is divided into patch images (P1, P2, P3, P4). However, the form of the image does not have to be limited to this example. Each image (121, 123, 127, 221) does not need to be divided into patch images. In this case, each process of the above embodiment may be applied to the entire image.

<5.5 Modification 5>
In the embodiment described above, the learning device 1 performs machine learning of the generative model 5 in the steps S102 and S116. However, the machine learning process does not necessarily have to be executed by the learning device 1. At least one of the machine learning processes in step S102 and step S116 may be executed by another computer. For example, the machine learning process in step S102 may be executed by another computer. When the processes of step S102 and step S116 are executed by another computer, the learning device 1 may be referred to as a learning image selection device, for example.

FIG. 17 is a diagram schematically illustrating an example of the software configuration of the learning device 1B according to the fifth modification. The software configuration of the learning device 1B according to this modification is the same as the learning device 1 according to the above embodiment, except that the first acquisition section 110 and the learning section 111 are replaced with a model acquisition section 111B. The model acquisition unit 111B acquires the generative model 5 trained by machine learning using one or more first learning images 121. Further, the hardware configuration of the learning device 1B according to this modification is the same as that of the learning device 1 according to the embodiment described above. That is, the learning device 1B is a computer to which a control section 11, a storage section 12, a communication interface 13, an input device 14, a display device 15, and a drive 16 are electrically connected (not shown).

In this modification, the control unit 11 of the learning device 1B omits the processing of steps S101 and S102, and operates as the model acquisition unit 111B. That is, the control unit 11 acquires the generative model 5 trained by machine learning using one or more first learning images 121. Similar to the above embodiment, when an input image is given, the generative model 5 converts the given input image into a feature amount, and generates an image that restores the given input image from the feature amount obtained by the conversion. configured to do so. Through machine learning, the generative model 5 is trained to, when each of the one or more first learning images 121 is given, generate a restored image that matches each of the given one or more first learning images 121. There is. This machine learning process may be executed by another computer.

Similar to the learning result data 129 of the above embodiment, information regarding the generative model 5 trained by machine learning may be provided to the learning device 1B as learning result data. Furthermore, the learning result data generated by another computer may be provided to the learning device 1B in the same manner as the method of providing the learning result data 129 generated by the learning device 1 to the detection device 2 in the above embodiment. . After acquiring the generative model 5 trained by machine learning using one or more first learning images 121, the learning device 1B may execute the processes from step S103 to step S117 in the above embodiment. Thereby, the learning device 1B according to this modification can appropriately select learning images to be used for machine learning, similarly to the above embodiment. Note that in this modification, the second acquisition unit 112 is an example of an image acquisition unit.

As in the above modification, when the machine learning process is executed by another computer, the machine learning process in the above processing procedure is replaced by acquiring the machine learning process result from the other computer. Furthermore, when another computer executes the machine learning process in step S116, the learning device 1 notifies the other computer of the extraction result in step S113. Notifying the extraction result may mean notifying the identifier of the extracted additional image 126, or may mean transmitting the extracted additional image 126 itself.

<5.6 Modification 6>
In the above embodiment, in step S115, the learning device 1 determines whether each of the one or more additional images 126 displayed on the display device 15 is used for machine learning. We accept the choice of “no” or “no”. The process of accepting this selection may be omitted. In this case, the process of step S115 may be omitted from the processing procedure of the learning device 1 according to the embodiment, and each of the one or more additional images 126 may be automatically determined as the second learning image 126A. The selection reception unit 119 may be omitted from the software configuration of the learning device 1 according to the above embodiment. In addition, when step S115 is omitted, the process of step S114 for outputting the additional image 126 may be further omitted. In this case, the display control unit 118 may be further omitted from the software configuration of the learning device 1 according to the above embodiment.

Further, in the embodiment described above, the performance of defect detection using the constructed and trained generative model 5 is evaluated through the processing of steps S105 to S111. This process of evaluating defect detection performance may be omitted. In this case, steps S106 to S111 may be omitted from the processing procedure of the learning device 1 according to the above embodiment. The third acquisition unit 117 may be omitted from the software configuration of the learning device 1 according to the above embodiment.

Furthermore, in the embodiments described above, the processing order of each step may be changed as appropriate. For example, the processing order of step S103 and step S104 may be reversed. The process of step S103 may be executed at any timing before the process of step S105 is executed. The process of step S104 may be executed at any timing before the process of step S106 is executed. Furthermore, for example, the processing order of step S105 and step S106 may be reversed. The process of step S105 may be executed at any timing before the process of step S107 is executed. The process of step S106 may be executed at any timing before the process of step S108 is executed. Furthermore, for example, the processing order of step S107 and step S108 may be reversed. Each process of step S107 and step S108 may be executed at any timing before executing the process of step S109. If the determination result of either step S107 or step S108 is not used in the process of step S109, the determination process may be executed at any timing before executing step S110.

<5.7 Modification 7>
In the embodiment described above, a plurality of detection parameter candidates 60 are provided. In step S<b>109 , the control unit 11 identifies one detection parameter candidate 61 with the least number of false determinations from the plurality of detection parameter candidates 60 . However, the way the detection parameters are given does not have to be limited to this example.

For example, the control unit 11 may extract from the plurality of detection parameter candidates 60 one or more detection parameter candidates for which the number or proportion of images (123, 127) that are erroneously determined is equal to or less than a predetermined threshold. . Alternatively, the control unit 11 may extract from the plurality of detection parameter candidates 60 a predetermined number (for example, five) of detection parameter candidates with a low number or proportion of each image (123, 127) that is erroneously determined.

In the production line, it is important to avoid shipping defective products as non-defective products as much as possible. Therefore, the control unit 11 detects at least one of the plurality of evaluation images 127 in which the proportion of the evaluation images 127 that are erroneously determined is equal to or less than a predetermined threshold (hereinafter referred to as "acceptable missed rate"). Parameter candidates 60 are extracted. The allowable missed rate may be given by operator input, a set value in a program, etc. If there are two or more detection parameter candidates 60 for which the proportion of evaluation images 127 that are erroneously determined is less than or equal to the allowable oversight rate, the control unit 11 extracts the two or more detection parameter candidates 60. Alternatively, the control unit 11 may extract a predetermined number (for example, five) of detection parameter candidates 60 with a small number or percentage of erroneously determined evaluation images 127.

The control unit 11 displays each of the extracted one or more or a predetermined number of detection parameter candidates 60 and the number or proportion of each image (123, 127) that is incorrectly determined when each detection parameter candidate is used. It is displayed on the device 15. Then, the control unit 11 may select one detection parameter candidate 61 from one or more or a predetermined number of detection parameter candidates 60 displayed according to the selection instruction input to the input device 14. The number or percentage of evaluation images 127 that are erroneously determined is the number or percentage of defective products that are determined to be non-defective products. The smaller the number or proportion of evaluation images 127 that are incorrectly determined, the more it is possible to suppress the outflow of defective products. On the other hand, the number or ratio of additional images 123 that are incorrectly determined is the number or ratio at which non-defective products are determined to be defective. The smaller the number or proportion of additional images 123 that are incorrectly determined, the better the yield. The operator may select the detection parameter candidate 61 while considering the outflow rate of defective products and the yield rate required for the production line.

Alternatively, the value of the detection parameter may be determined in advance. The value of the detection parameter may be given by an operator's input, a set value in a program, or the like. In this case, the process of step S109 may be omitted from the process procedure of the learning device 1 according to the above embodiment. Further, the control unit 11 may execute the respective processes of step S107 and step S108 using the determined detection parameters. Further, in step S110, the control unit 11 determines whether or not the number or proportion of additional images 123 with false determinations is equal to or higher than a first threshold in the determination results using the determined detection parameters. Good too. Furthermore, the control unit 11 may determine whether the number or proportion of evaluation images 127 that are incorrectly determined is greater than or equal to a second threshold value in the result of determination using the determined detection parameters.

<5.8 Modification 8>
In the embodiment described above, the control unit 11 (third acquisition unit 117) acquires a plurality of evaluation images 127 in which the product R, which is a predetermined feature, and the defect L, which is another feature, are respectively captured. However, the control unit 11 may acquire a plurality of first evaluation images in which the product R and the defect L are respectively shown, and a plurality of second evaluation images in which the product R is shown and the defect L is not shown. That is, the first evaluation image shows the product R, which is a non-defective product, and the second evaluation image shows the product R, which is a defective product.

FIG. 18 is a diagram schematically illustrating an example of the software configuration of a learning device according to modification 8. The learning device 1C illustrated in FIG. 18 differs from the learning device 1 illustrated in FIG. 5 in that it includes a third acquisition unit 117C instead of the third acquisition unit 117.

The third acquisition unit 117C acquires a plurality of first evaluation images 127A in which the product R and the defect L are respectively shown, and a plurality of second evaluation images 127B in which the product R is shown and the defect L is not shown.

The plurality of first evaluation images 127A may be generated in the same manner as the evaluation image 127 described above. That is, like each image (121, 123), each first evaluation image 127A may be automatically generated by a computer operation, or may be manually generated by an operator's operation. Furthermore, the generation of each first evaluation image 127A may be performed by the learning device 1, or may be performed by another computer other than the learning device 1. The control unit 11 may acquire the plurality of first evaluation images 127A by automatically or manually executing the series of processes described above through an operator's operation via the input device 14. Alternatively, the control unit 11 may acquire the plurality of first evaluation images 127A generated by another computer, for example, via the network, the storage medium 91, or the like. Some of the plurality of first evaluation images 127A may be generated by the learning device 1, and the rest may be generated by another computer. The number of first evaluation images 127A to be acquired does not need to be particularly limited and may be determined as appropriate.

The plurality of second evaluation images 127B may be generated in the same manner as the above images (121, 123). The generation of each second evaluation image 127B may be performed by the learning device 1, or may be performed by another computer other than the learning device 1. The control unit 11 may acquire the plurality of second evaluation images 127B by automatically or manually executing the series of processes described above through an operator's operation via the input device 14. Alternatively, the control unit 11 may acquire a plurality of second evaluation images generated by another computer, for example, via a network, the storage medium 91, or the like. Some of the plurality of second evaluation images 127B may be generated by the learning device 1, and the rest may be generated by another computer. The number of second evaluation images to be acquired does not need to be particularly limited and may be determined as appropriate.

In this modification, the control unit 11 (generation unit 113) restores each of the plurality of second evaluation images 127B by giving each of the plurality of acquired second evaluation images 127B to the generation model 5 in step S105. A plurality of second restoration evaluation images 128B are generated respectively. Further, in step S106, the control unit 11 provides a plurality of first restored evaluation images obtained by restoring each of the plurality of first evaluation images 127A by providing each of the plurality of acquired first evaluation images 127A to the generation model 5. 128A respectively.

In step S107, the control unit 11 (evaluation unit 114) determines whether the defect L is reflected in each second evaluation image based on the degree of difference between each second evaluation image 127B and each second restoration evaluation image 128B. Determine whether or not. That is, the control unit 11 (evaluation unit 114) generates each of the plurality of difference images 183B indicating the difference between each of the plurality of second evaluation images 127B and each of the plurality of second restored evaluation images 128B. The control unit 11 (evaluation unit 114) uses each difference image 183B to determine whether the defect L is included in each second evaluation image 127B.

In step S108, the control unit 11 (evaluation unit 114) determines whether the defect L is reflected in each first evaluation image 127A based on the degree of difference between each first evaluation image 127A and each first restoration evaluation image 128A. Determine whether or not there is. That is, the control unit 11 (evaluation unit 114) generates each of the plurality of difference images 183A indicating the difference between each of the plurality of first evaluation images 127A and each of the plurality of first restored evaluation images 128A. The control unit 11 (evaluation unit 114) uses each difference image 183A to determine whether the defect L is included in each first evaluation image 127A.

In step S109, the control unit 11 optimizes the detection parameters for determining whether the defect L is shown in the image, based on the determination results of each step (S107, S108). Specifically, the control unit 11 counts the number of second evaluation images 127B determined to include the defect L (hereinafter also referred to as the number of erroneously determined second evaluation images). The control unit 11 counts the number of first evaluation images 127A in which it is determined that the defect L is not captured (hereinafter referred to as the number of erroneously determined first evaluation images). The control unit 11 identifies one detection parameter candidate 61 that has the smallest number of first evaluation images that are incorrectly determined and the least number of second evaluation images that are incorrectly judged.

Alternatively, as in Modification Example 7, the control unit 11 controls the control unit 11 so that the proportion of the first evaluation images 127A determined not to include the defect L among the plurality of first evaluation images 127A is at least equal to or less than the allowable overlook rate. One detection parameter candidate 60 may be extracted. Hereinafter, the ratio of first evaluation images 127A determined to include the defect L among the plurality of first evaluation images 127A will also be referred to as the ratio of first evaluation images that are incorrectly determined. The control unit 11 selects each of the one or more extracted detection parameter candidates 60 and the second evaluation image 127B that is determined to include the defect L when each of the detection parameter candidates is used. The ratio of the 2-evaluation image 127B is displayed on the display device 15. Hereinafter, the ratio of second evaluation images 127B determined to include the defect L among the plurality of second evaluation images 127B will also be referred to as the ratio of second evaluation images that are incorrectly determined. Then, the control unit 11 may select one detection parameter candidate 61 from the one or more extracted detection parameter candidates 60 according to the selection instruction input to the input device 14.

In step S110 described above, the control unit 11 operates as the evaluation unit 114, and in the result of the determination using the one identified detection parameter candidate 61, the number or proportion of the second evaluation images that are incorrectly determined is the same as the first one. Determine whether or not the threshold value is greater than or equal to the threshold value. Furthermore, the control unit 11 determines whether or not the number or proportion of first evaluation images that are incorrectly determined is equal to or greater than a second threshold value in the result of the determination using the one identified detection parameter candidate 61.

In step S<b>112 described above, the control unit 11 operates as the generation unit 113 and provides each of the acquired plurality of additional images 123 to the generation model 5 to generate a plurality of restorations in which each of the plurality of additional images 123 is restored. Each additional image 124 is generated. Then, the control unit 11 determines the evaluation value 125 of each additional image 123 according to the difference between each additional image 123 and each generated restored additional image 124, and the evaluation value 125 of each additional restored image 124. An evaluation value 125 for the degree of restoration of each additional image 123 may be calculated.

<5.9 Modification 9>
As described above, the appearance of the product R reflected in the observation image 221 may also vary due to factors such as individual differences in the product R and fluctuations in imaging conditions. The larger the fluctuation range of the appearance of the product R reflected in the observed image 221, the larger the number of learning images required for machine learning. It is preferable that the one or more second learning images 126A be selected according to the range of variation in the appearance of the product R reflected in the observed image 221. Therefore, in this modification, the control unit 11 (second acquisition unit 112) acquires the observed image 221 acquired by the detection device 2 as one of the plurality of additional images 123.

The control unit 11 may acquire the observation image 221 from the detection device 2 via the communication interface 13 . Alternatively, the detection device 2 may store the observation image 221 acquired from the camera CA in the storage medium 92. Then, the control unit 11 may read the observation image 221 from the storage medium 92 via the drive 16.

The detection device 2 acquires an observation image 221 for each product R transported on the production line. Generally, the number of observation images 221 acquired by the detection device 2 becomes enormous in a short period of time. Since the storage unit 12 in the learning device has a limited capacity, the storage unit 12 cannot store all observed images 221 acquired from the detection device 2. Therefore, in the learning device according to modification example 9, the control unit 11 (extraction unit 115) selects a predetermined number of observation images 221 (additional images 123) obtained from the detection device 2 in order from the one with the highest evaluation value 125. A number of observed images 221 (additional images 123) are extracted as additional images 126. The predetermined number is an integer of 1 or more. Then, the control unit 11 (storage processing unit 1110) stores only the extracted additional image 126 from among the plurality of additional images 123 in the storage unit 12.

The observed image 221 does not necessarily show the product R, which is a good product. That is, the control unit 11 can acquire the observation image 221 in which the defective product R is captured as the additional image 123. Therefore, the control unit 11 (display control unit 118) causes the display device 15 to display each of the predetermined number of additional images 126 extracted from the plurality of additional images 123 (observation images 221). Then, the control unit 11 (selection reception unit 119) receives a selection as to whether or not to use each of the predetermined number of additional images 126 for machine learning. Specifically, the control unit 11 (selection accepting unit 119) accepts a selection as to whether or not the defect L is included in each of the predetermined number of additional images 126. The control unit 11 (additional learning unit 116) determines the additional image 126 selected as not showing the defect L as the second learning image 126A.

Modification 9 may be combined with Modification 8 above. That is, the control unit 11 (third acquisition unit 117C) generates a plurality of first evaluation images 127A in which the product R and the defect L are respectively shown, and a plurality of second evaluation images 127B in which the product R is shown and the defect L is not shown. get. The control unit 11 may acquire each additional image 126 selected to include the defect L as one of the plurality of first evaluation images 127A. Furthermore, the control unit 11 may acquire each additional image 126 selected as not showing the defect L as one of the plurality of second evaluation images 127B.

An example of the operation of the learning device according to modification example 9 will be described with reference to FIGS. 19 to 22. FIGS. 19 and 20 are flowcharts illustrating an example of a procedure for extracting images for addition in the learning device according to Modification 9. FIG. Note that the following processing procedure is performed after acquiring the generative model 5 trained by machine learning using the first learning image 121. The generative model 5 trained by machine learning using the first learning image 121 may be obtained by the processing in step S101 and step S102 described above. Alternatively, the generative model 5 trained by machine learning using the first learning image 121 may be obtained by a method similar to the fifth modification.

(Step S401)
In step S401, the control unit 11 operates as the second acquisition unit 112 and acquires the observation image 221 acquired by the detection device 2 as the additional image 123. After acquiring the additional image 123, the control unit 11 advances the process to step S402.

(Step S402)
In step S402, the control unit 11 operates as the generation unit 113, and generates a restored additional image 124 by restoring the additional image 123 by providing the acquired additional image 123 to the generation model 5. Step S4023 is similar to step S105 above. Therefore, detailed explanation of step S402 will be omitted. After generating the restoration addition image 124, the control unit 11 advances the process to step S403.

(Step S403)
In step S403, the control unit 11 determines the evaluation value 125 of the additional image 123 according to the difference between the additional image 123 and the generated restored additional image 124, and the additional restored image 124. An evaluation value 125 for the degree of restoration of the image 123 is calculated. Step S403 is similar to step S112 above. Therefore, detailed explanation of step S403 will be omitted. After generating the evaluation value 125, the control unit 11 advances the process to step S404.

(Step S404)
In step S404, the control unit 11 determines whether the lower limit LL of the evaluation value 125 for the additional image 126 to be stored in the storage unit 12 has been set. The lower limit value LL is appropriately set by operator input, a set value in the program, etc., as necessary. If YES in step S404, the control unit 11 advances the process to step S405. If NO in step S404, the control unit 11 advances the process to step S407.

(Step S405)
In step S405, the control unit 11 compares the evaluation value 125 calculated by the process in step S403 and the lower limit value LL. If the evaluation value 125 is less than or equal to the lower limit value LL (NO in step S405), the control unit 11 advances the process to step S406. If the evaluation value 125 exceeds the lower limit LL (YES in step S405), the control unit 11 advances the process to step S407.

(Step S406)
In step S406, the control unit 11 discards the observed image 221 (additional image 123) acquired by the process in step S401. When the additional image 123 is discarded, the control unit 11 ends the process of extracting the additional image 126.

(Step S407)
In step S407, the control unit 11 determines whether an upper limit HL for the number of additional images 126 to be stored in the storage unit 12 has been set. The upper limit value HL is appropriately set according to operator input, a set value in the program, etc., as necessary. If NO in step S407, the control unit 11 advances the process to step S412. If YES in step S407, the control unit 11 advances the process to step S408.

(Step S408)
In step S408, the control unit 11 determines whether the number of additional images 126 stored in the storage unit 12 has reached the upper limit HL. If NO in step S408, the control unit 11 advances the process to step S412. If YES in step S408, the control unit 11 advances the process to step S409.

(Step S409)
In step S409, the control unit 11 identifies the minimum value of the evaluation values 125 for a predetermined number (same number as the upper limit HL) of additional images 126 stored in the storage unit 12. As will be described later in step S412, the additional image 126 and the evaluation value 125 are stored in association with each other in the storage unit 12. The control unit 11 identifies the minimum value by checking the evaluation value 125 associated with the additional image 126. After specifying the minimum value, the control unit 11 advances the process to step S410.

(Step S410)
In step S410, the control unit 11 determines whether the evaluation value 125 calculated in the process in step S403 exceeds the minimum value specified in the process in step S409. If YES in step S410, the control unit 11 advances the process to step S411. If NO in step S410, the control unit 11 advances the process to step S406, and discards the observed image 221 (additional image 123) acquired by the process in step S401.

(Step S411)
In step S411, the control unit 11 selects one additional image 126 with the smallest evaluation value 125 from the storage unit 12 among the plurality of additional images 126 (same number as the upper limit HL) stored in the storage unit 12. delete. After deleting the one additional image 126 with the smallest evaluation value 125 from the storage unit 12, the control unit 11 advances the process to step S412.

(Step S412)
In step S412, the control unit 11 stores the observation image 221 (additional image 123) acquired through the process in step S401 as a new addition image 126 in the storage unit 12. At this time, the control unit 11 stores the evaluation value 125 calculated by the process in step S403 in the storage unit 12 in association with the new additional image 126. After storing the new additional image 126 in the storage unit 12, the control unit 11 ends the process of extracting the additional image 126.

A series of processes from step S401 to step S412 shown in FIGS. 19 and 20 are performed every time the observation image 221 is acquired from the detection device 2. As a result, when the upper limit value HL is set, the storage unit 12 of the learning device 1 stores a predetermined number of observation images 221 (additional images 123) acquired in the past in order from the one with the highest evaluation value 125. number (=upper limit HL) of observation images 221 are stored in the storage unit 12 as additional images 126.

21 and 22 are flowcharts illustrating an example of the procedure of additional learning processing in the learning device according to Modification 9. FIG. The additional learning process shown in FIGS. 21 and 22 is performed after the above-described process for extracting the additional image 126 is executed. Note that, below, an example will be described in which a predetermined number (=upper limit value HL) of additional images 126 are stored in the storage unit 12.

(Step S501)
In step S501, the control unit 11 sorts a predetermined number of additional images 126 stored in the storage unit 12 according to the degree of restoration. Specifically, the control unit 11 sorts the predetermined number of additional images 126 in descending order of evaluation value 125. When the sorting is completed, the control unit 11 advances the process to step S502.

(Step S502)
In step S502, the control unit 11 operates as the display control unit 118 and causes the display device 15 to display a display screen including a predetermined number of additional images 126.

FIG. 23 is a diagram showing an example of the display screen displayed in step S502. The display screen 70 illustrated in FIG. 23 includes check boxes 71 and 73 and input fields 72 and 74. The check box 71 is an object for enabling the setting of the lower limit value LL of the evaluation value 125 for the additional image 126 stored in the storage unit 12. When the check box 71 is checked, the setting of the lower limit value LL becomes valid. In the input field 72, the lower limit value LL is input. The check box 73 is an object for enabling the setting of the upper limit HL of the number of additional images 126 to be saved in the storage unit 12. When the check box 73 is checked, the setting of the upper limit value HL becomes valid. In the input field 74, the upper limit value HL is input.

Display screen 70 includes areas 75 and 78. In the area 75, a list of a predetermined number of additional images 126 stored in the storage unit 12 is displayed. Specifically, in the area 75, a predetermined number of thumbnail images 76 corresponding to the predetermined number of additional images 126 are displayed. The predetermined number of thumbnail images 76 are arranged in the order sorted by the process in step S501. Furthermore, the evaluation value 125 corresponding to each additional image 126 is also displayed in the area 75. A cursor 77 for selecting one of a predetermined number of thumbnail images 76 is displayed in the area 75. The cursor 77 moves according to input to the input device 14. In the area 78, an additional image 126 corresponding to the thumbnail image 76 selected by the cursor 77 is displayed.

Display screen 70 further includes areas 79-81 and buttons 82-89. In the area 79, a list of additional images 126 classified as defective product images (hereinafter also referred to as a defective product image list) is displayed. In the area 80, a list of additional images 126 classified as non-defective images (hereinafter also referred to as a non-defective image list) is displayed. In the area 81, a list of additional images 126 selected as the second learning images 126A (hereinafter also referred to as "learning image list") is displayed. The defective product image list, the non-defective product image list, and the learning image list are lists of file names of images. Buttons 82 to 85 are buttons for classifying the additional image 126 into either a defective product image or a non-defective product image. The button 86 is a button for determining the second learning image 126A. Button 87 is a button for executing additional learning. The button 88 is a button for optimizing detection parameters. Button 89 is a button for closing display screen 70.

Display screen 70 includes an area 90 where various information is displayed. In the area 90, the "previous lowest score" indicates the minimum value among the predetermined number of evaluation values 125 corresponding to the predetermined number of additional images 126 extracted before the previous additional learning. The "previous average score" indicates the average value of a predetermined number of evaluation values 125 corresponding to the predetermined number of additional images 126 extracted before the previous additional learning. The "lowest score" indicates the minimum value among the predetermined number of evaluation values 125 corresponding to the predetermined number of additional images 126 extracted before the current additional learning. The "average score" indicates the average value of the predetermined number of evaluation values 125 corresponding to the predetermined number of additional images 126 extracted before the current additional learning. “Remaining storage capacity” indicates the free capacity in the storage unit 12. “Number of images that can be saved” indicates the maximum number of additional images 126 that can be saved in the storage unit 12.

When the display of the display screen 70 illustrated in FIG. 23 is completed, the control unit 11 advances the process to step S503.

(Step S503)
In step S503, the control unit 11 operates as the selection reception unit 119 and determines whether or not an operation for classification into non-defective images and defective images has been accepted. In response to the operation of the buttons 82 to 85 included in the display screen 70, the control unit 11 determines that an operation for classifying images into non-defective images and defective images has been accepted. If the buttons 82 to 85 are not operated (NO in step S503), the control unit 11 advances the process to step S505. If the buttons 82 to 85 are operated (YES in step S503), the control unit 11 advances the process to step S504.

(Step S504)
In step S504, the control unit 11 operates as the selection reception unit 119, and classifies the selected additional image 126 into either a defective product image or a non-defective product image according to the operations on the buttons 82 to 85.

When the button 82 is operated, the control unit 11 classifies the additional image 126 displayed in the area 78 as a non-defective image. The control unit 11 deletes the thumbnail image 76 corresponding to the additional image 126 classified as a non-defective image from the area 75. The control unit 11 stores the additional image 126 classified as a non-defective image in the storage unit 12 as a new second evaluation image 127B. The control unit 11 adds the new second evaluation image 127B to the non-defective image list displayed in the area 80.

When the button 83 is operated, the control unit 11 classifies the additional image 126 displayed in the area 78 as a defective product image. The control unit 11 deletes the thumbnail image 76 corresponding to the additional image 126 classified as a defective product image from the area 75. The control unit 11 stores the additional image 126 classified as a defective product image in the storage unit 12 as a new first evaluation image 127A. The control unit 11 adds the new first evaluation image 127A to the defective product image list displayed in the area 79.

The operator may operate the buttons 82, 83 incorrectly. For example, the operator may mistakenly operate button 83 when button 82 should have been operated. In this case, the additional image 126 that should be classified as a non-defective image is classified as a defective image and added to the defective image list displayed in area 79. Buttons 84 and 85 are provided on the display screen 70 so that even if such an erroneous operation is performed, it can be repaired. The button 84 is a button for moving an image selected from the defective product image list to the non-defective product image list. The button 85 is a button for moving an image selected from the non-defective image list to the defective image list.

The control unit 11 accepts selection of one image included in either the defective product image list or the non-defective product image list. When one image included in either the defective product image list or the non-defective product image list is selected, the control unit 11 displays the selected image in the area 78. Thereby, the operator can confirm whether or not the buttons 82 and 83 have been operated incorrectly by checking the image displayed in the area 78.

When the button 84 is operated with one first evaluation image 127A in the defective image list selected, the control unit 11 reclassifies the selected first evaluation image 127A as a non-defective image. The control unit 11 saves the first evaluation image 127A classified as a non-defective image in the storage unit 12 as a new second evaluation image 127B. The control unit 11 adds the new second evaluation image 127B to the non-defective image list displayed in the area 80.

When the button 85 is operated with one second evaluation image 127B in the non-defective image list selected, the control unit 11 reclassifies the selected second evaluation image 127B as a defective image. The control unit 11 saves the second evaluation image 127B classified as a defective image in the storage unit 12 as a new first evaluation image 127A. The control unit 11 adds the new first evaluation image 127A to the defective product image list displayed in the area 79.

When the process corresponding to the operated button is completed, the control unit 11 advances the process to step S505.

(Step S505)
In step S505, the control unit 11 determines whether an instruction to select one or more of the additional images 126 classified as non-defective images as the second learning image 126A has been received. The control unit 11 determines that the instruction to select the second learning image 126A has been received in response to the button 86 being operated with one or more second evaluation images 127B in the non-defective image list being selected. . If the button 86 is operated (YES in step S504), the control unit 11 advances the process to step S506. If the button 86 is not operated (NO in step S505), the control unit 11 advances the process to step S507.

(Step S506)
In step S506, the control unit 11 determines the selected second evaluation image 127B as the second learning image 126A in response to the operation of the button 86. The control unit 11 adds the determined second learning image 126A to the learning image list displayed in the area 81. Note that the second evaluation image 127B determined as the second learning image 126A is also left in the non-defective image list. After completing the process in step S506, the control unit 11 advances the process to step S507.

(Step S507)
In step S507, the control unit 11 determines whether an instruction to perform additional learning has been received. The control unit 11 determines that an instruction to perform additional learning has been received in response to the button 87 included in the display screen 70 being operated. If the button 87 is operated (YES in step S506), the control unit 11 advances the process to step S508. If the button 87 is not operated (NO in step S507), the control unit 11 advances the process to step S510.

(Step S508)
In step S508, the control unit 11 operates as the additional learning unit 116, and when each of the second learning images 126A included in the learning image list is given by machine learning, the control unit 11 adapts to each given second learning image 126A. The generative model 5 is further trained to generate a restored image. Step S508 is similar to step S116 above. Therefore, detailed explanation of step S508 will be omitted. When training of the generative model 5 is completed, the control unit 11 advances the process to step S509.

(Step S509)
In step S509, the control unit 11 operates as the storage processing unit 1110 and generates information regarding the constructed trained generative model 5 as the learning result data 129. The control unit 11 stores the generated learning result data 129 in a predetermined storage area. Note that the generated learning result data 129 may be provided to the detection device 2 at any timing. The process in step S509 is similar to the process in step S117 above. After storing the learning result data 129, the control unit 11 advances the process to step S510.

(Step S510)
In step S510, the control unit 11 determines whether an instruction to perform optimization of detection parameters has been received. The control unit 11 determines that an instruction to optimize the detection parameters has been received in response to the button 88 included in the display screen 70 being operated. If the button 88 is operated (YES in step S510), the control unit 11 advances the process to step S511. If the button 88 is not operated (NO in step S510), the control unit 11 advances the process to step S514.

(Step S511)
In step S511, the control unit 11 operates as the generation unit 113 and the evaluation unit 114, and optimizes the detection parameters. Specifically, the control unit 11 identifies the optimal detection parameter candidate 61 from among the plurality of detection parameter candidates 60. The process in step S511 is similar to steps S105 to S109 described above. However, similarly to Modification Example 8, the control unit 11 uses a plurality of first evaluation images 127A and a plurality of second evaluation images 127B instead of a plurality of evaluation images 127 and one or more second learning images 126A. Then, optimize the detection parameters. Note that the control unit 11 may exclude the image determined as the second learning image 126A from among the plurality of second evaluation images 127B, or may exclude the image determined as the second learning image 126A from among the plurality of second evaluation images 127B. It's okay. After completing the process in step S511, the control unit 11 advances the process to step S512.

(Step S512)
In step S512, the control unit 11 operates as the evaluation unit 114, and in the result of the determination using the identified one detection parameter candidate 61, the number or proportion of the second evaluation images 127B that are incorrectly determined is equal to the first threshold value. It is determined whether or not the value is greater than or equal to the value. Further, the control unit 11 determines whether or not the number or proportion of the first evaluation images 127A that are incorrectly determined is equal to or higher than a second threshold value in the result of the determination using the one identified detection parameter candidate 61. . If YES in step S512, the control unit 11 advances the process to step S513. If NO in step S512, the control unit 11 advances the process to step S514.

(Step S513)
In step S513, the control unit 11 selects one or more second evaluation images 127B with a low degree of restoration from among the plurality of second evaluation images 127B (excluding images already selected as the second learning images 126A). This is determined as the learning image 126A. After completing the process in step S513, the control unit 11 returns the process to step S508. Thereby, machine learning is repeated using one or more second evaluation images 127B with a low degree of restoration among the plurality of second evaluation images 127B as the second learning image 126A.

(Step S514)
In step S514, the control unit 11 determines whether an instruction to close the display screen 70 has been received. The control unit 11 determines that an instruction to close the display screen 70 has been received in response to the button 89 included in the display screen 70 being operated. If the button 89 has not been operated (NO in step S514), the control unit 11 returns the process to step S503. If the button 89 is operated (YES in step S514), the control unit 11 ends a series of processes related to additional learning.

As described above, in the learning device according to modification 9, the control unit 11 (second acquisition unit 112) acquires the observation image 221 acquired by the detection device 2 as one of the plurality of additional images 123. Thereby, the operator can omit the effort of preparing the additional image 123.

Based on the evaluation value 125, the control unit 11 (extraction unit 115) extracts a predetermined number of additional images 126 that are evaluated as having a low degree of restoration from the plurality of additional images 123. Then, the control unit 11 (selection reception unit 119) selects, for each of the predetermined number of additional images 126 displayed on the display device 15, whether or not to use each of the predetermined number of additional images 126 for machine learning. accept. Specifically, the control unit 11 (selection reception unit 119) receives a selection as to whether or not the defect L is included in each of the predetermined number of additional images 126. The control unit 11 (additional learning unit 116) determines the additional image 126 selected to be used for machine learning (that is, the additional image 126 selected as not showing the defect L) as the second learning image 126A. .

The learning device can acquire the observation image 221 showing the defective product R as the additional image 123. However, according to the above configuration, the additional image 126 that does not include the defect L is determined as the second learning image 126A in accordance with the operator's selection. Thereby, it is possible to prevent the performance of the generation model 5 based on machine learning using the image containing the defect L from deteriorating.

The control unit 11 (third acquisition unit 117C) acquires a plurality of first evaluation images 127A in which the product R and the defect L are shown, and a plurality of second evaluation images 127B in which the product R is shown and the defect L is not shown. . The control unit 11 (evaluation unit 114) is provided with a plurality of detection parameter candidates 60 for determining whether the defect L is shown in the image. The control unit 11 (evaluation unit 114) uses each of the plurality of detection parameter candidates 60 to determine whether or not the defect L appears in each first evaluation image 127A, and determines whether the defect L appears in each second evaluation image 127B. A determination is made as to whether or not the image is captured in the image. The control unit 11 (evaluation unit 114) optimizes detection parameters for determining whether the defect L is shown in the image based on the results of each determination. The control unit 11 (selection accepting unit 119) accepts a selection as to whether or not the defect L is included in each of the predetermined number of additional images 126 displayed on the display device 15. The control unit 11 (third acquisition unit 117) acquires each additional image 126 selected to include the defect L as one of the plurality of first evaluation images 127A. The control unit 11 (third acquisition unit 117) acquires each additional image 126 selected as not showing the defect L as one of the plurality of second evaluation images 127B. Thereby, the operator can omit the effort of preparing the first evaluation image 127A and the second evaluation image 127B.

The control unit 11 (storage processing unit 1110) stores a predetermined number of additional images 126 extracted from the plurality of additional images 123 in the storage unit 12. Generally, the number of observation images 221 acquired by the detection device 2 becomes enormous in a short period of time. However, according to the above configuration, although the observed image 221 is acquired as the additional image 123, a predetermined number of observed images 221 (additional images 123) are extracted as the additional images 126 in order from the one with the highest evaluation value 125. Only the predetermined number of extracted additional images 126 are stored in the storage unit 12. Therefore, a situation in which the free space in the storage unit 12 becomes insufficient can be avoided.

§6 Supplementary notes As described above, this embodiment includes the following disclosures.

(Configuration 1)
a first acquisition unit (11, 110) that acquires one or more first learning images (121) in which predetermined features are captured;
A learning unit (11, 111) that performs machine learning of the generative model (5) using the one or more first learning images (121),
The generative model (5) is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
By the machine learning, when each of the one or more first learning images (121) is given, the machine learning generates a restored image that matches each of the one or more given first learning images (121). training the generative model (5);
Learning department (11, 111) and
a second acquisition unit (11, 112) that acquires a plurality of additional images (123) showing the predetermined characteristics;
a generation unit that generates each of a plurality of restored additional images (124) obtained by restoring each of the plurality of additional images (123) by supplying each of the plurality of additional images (123) to the generation model (5); (11,113) and
An evaluation value of each additional image (124) according to the difference between each additional image (123) and each restored additional image (124), which is an evaluation value of each additional restored image (124). an evaluation unit (11, 114) that calculates an evaluation value for the degree of restoration of each of the additional images (124);
an extraction unit (11, 115) that extracts one or more additional images (126) that are evaluated as having a low degree of restoration from the plurality of additional images (123) based on the evaluation value;
When each of the one or more second learning images (126A) included in the one or more extracted additional images (126) is given by machine learning, the one or more given second learning images (126A) an additional learning unit (11, 116) that further trains the generative model (5) to generate a restored image that fits each;
Equipped with
Learning device (1, 1C).

(Configuration 2)
Calculating the evaluation value according to the difference includes:
generating each of a plurality of difference images (182) indicating the difference between each of the plurality of additional images (123) and each of the plurality of restored additional images (124);
In each of the difference images (182), the larger the pixel value of each pixel, the higher the evaluation value, and the contribution of pixels whose contribution to the evaluation value is equal to or greater than the reference pixel value is less than the reference pixel value. Calculating the evaluation value so that it is larger than the contribution given to the evaluation value;
Consisting of
The learning device (1, 1C) according to Configuration 1.

(Configuration 3)
further comprising a display control unit (11, 118) that causes the display device (15) to display each of the one or more extracted additional images (126);
The learning device (1, 1C) according to Configuration 1 or 2.

(Configuration 4)
For each of the one or more additional images (126) displayed on the display device (15), a selection that accepts a selection of whether to use each of the one or more additional images (126) for machine learning. Further equipped with a reception department (11, 119),
The additional learning unit (11, 116) determines the additional image (126) selected for use in the machine learning as the one or more second learning images (126A),
The learning device (1, 1C) according to configuration 3.

(Configuration 5)
further comprising a third acquisition unit (11, 117) that acquires a plurality of evaluation images (127) in which the predetermined feature and other features other than the predetermined feature are captured;
The other features are not included in the plurality of additional images (123);
The generation unit (11, 113) provides a plurality of restored evaluation images (127) for each of the plurality of evaluation images (127) by giving each of the plurality of acquired evaluation images (127) to the trained generation model (5). Further generate each restoration evaluation image (128) of
The evaluation unit (11, 114) further includes:
Based on the degree of difference between each of the additional images (123) and the generated restored additional images (124), it is determined whether the other features are reflected in each of the additional images (123). and determining, based on the degree of difference between each of the evaluation images (127) and each of the generated restoration evaluation images (128), whether or not the other features are reflected in each of the evaluation images (127). Determine,
The evaluation unit (11, 114) determines that the number or proportion of the additional images (123) determined to include the other features among the plurality of additional images (123) is equal to or higher than a first threshold value. or, when the number or proportion of the evaluation images (127) determined to not include the other features among the plurality of evaluation images (127) is equal to or higher than a second threshold, Calculate the evaluation value for one or more additional images (123) that are determined to include the characteristics of
The extraction unit (11, 115) determines the degree of restoration of the one or more additional images (123) determined to include the other features based on the calculated evaluation value. extracting images that are evaluated as low as the one or more additional images (126);
The learning device (1) according to any one of configurations 1 to 4.

(Configuration 6)
A plurality of parameter candidates are provided for determining whether or not the other features are shown in each of the additional images (123) and whether or not the other features are shown in each of the evaluation images (127). is,
The evaluation unit (11, 114)
Using each of the plurality of parameter candidates, it is determined whether or not the other feature is shown in each of the additional images (123), and whether the other feature is shown in each of the evaluation images (127). Execute a determination of whether or not
Based on the results of each of the determinations, optimization of parameters for determining whether or not the other features are included in the image is performed, and in the results of each of the determinations using the optimized parameters, The number or proportion of additional images (123) determined to include the other feature is equal to or greater than the first threshold, or an evaluation image (127) determined to not include the other feature. calculating the evaluation value for one or more additional images (123) that are determined to include the other features when the number or ratio of the other features is greater than or equal to the second threshold;
Learning device (1) according to configuration 5.

(Configuration 7)
Further comprising a third acquisition unit (11, 117C) that acquires a plurality of first evaluation images (127A) and a plurality of second evaluation images (127B),
The plurality of first evaluation images (127A) include the predetermined feature and features other than the predetermined feature,
The plurality of second evaluation images (127B) include the predetermined feature and do not include the other feature,
The evaluation unit (11, 114)
A plurality of parameter candidates are given for determining whether the other feature is included in the image,
Using each of the plurality of parameter candidates, it is determined whether or not the other feature appears in each of the first evaluation images (127A), and whether or not the other feature appears in each of the second evaluation images (127B). Execute a judgment as to whether the image is visible or not,
Based on the results of each of the determinations, optimizing parameters for determining whether the other features are included in the image;
The selection accepting unit (11, 119) further accepts a selection as to whether or not the other characteristics are included in each of the one or more additional images (126) displayed on the display device (15). ,
The third acquisition unit (11, 117C) is
each additional image (126) selected as showing the other characteristics is acquired as one of the plurality of first evaluation images (127A);
acquiring each additional image (126) selected as not showing the other features as one of the plurality of second evaluation images (127B);
Learning device (1C) according to configuration 4.

(Configuration 8)
The second acquisition unit (11, 112) communicates with the detection device (2),
The detection device (2) includes:
Obtaining an observation image obtained by observing the predetermined features,
By inputting the observed image (221) to the generative model (5), a restored observed image (223) corresponding to the observed image (221) is generated;
Based on the difference between the generated restored observation image (223) and the observation image (221), it is detected whether or not other features other than the predetermined features are reflected in the observation image (221). ,
The second acquisition unit (11, 112) acquires the observation image (221) acquired by the detection device (2) as one of the plurality of additional images.
The learning device (1, 1C) according to configuration 4 or 7.

(Configuration 9)
A storage processing unit (11, 1110),
The learning device (1, 1C) according to any one of configurations 1 to 8.

(Configuration 10)
Each of the first learning images (121) and each additional image (123) is divided into a plurality of patch images,
The generative model (5) is given for each patch image,
When each of the one or more first learning images (121) is given, the learning unit (11, 111) performs machine learning for each of the patch images to determine the one or more given first learning images. (121) training the generative model (5) to generate restored images that suit each;
The generation unit (11, 113) generates each restored additional image (124) corresponding to each additional image (123) for each patch image,
The evaluation unit (11, 114) calculates the evaluation value of each additional image (123) for each patch image,
The extraction unit (11, 115) extracts the one or more additional images (126) from the plurality of additional images (123) based on the calculated evaluation value for each patch image. ,
The additional learning unit (11, 116) performs machine learning for each of the patch images, when each of the one or more second learning images (126A) is given, the one or more given second learning images. further training the generative model (5) to generate restored images that match each of the images (126A);
The learning device (1, 1C) according to any one of configurations 1 to 6.

(Configuration 11)
The generative model (5) is configured by a neural network.
The learning device according to any one of configurations 1 to 10.

(Configuration 12)
The machine learning includes principal component analysis,
The generative model (5) is composed of eigenvectors derived by the principal component analysis,
The learning device (1, 1C) according to any one of Configurations 1 to 10.

(Configuration 13)
The predetermined characteristic is a product transported on a production line that does not contain defects;
The learning device (1, 1C) according to any one of configurations 1 to 12.

(Configuration 14)
an acquisition unit (21) that acquires an observation image (221) obtained by observing the predetermined characteristics;
By inputting the observed image (221) to the generative model (5) trained by the learning device (1, 1C) according to any one of configurations 1 to 13, the observed image (221) is restored corresponding to the observed image (221). a generation unit (22) that generates an observation image (223);
Based on the difference between the generated restored observation image (223) and the observation image (221), it is detected whether features other than the predetermined features are included in the observation image (221). a detection section (23);
Equipped with
Detection device (2).

(Configuration 15)
A model acquisition unit (11, 111B) that acquires a generative model (5) trained by machine learning using one or more first learning images (121) in which predetermined features are captured,
The generative model (5) is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
Through the machine learning, when each of the one or more first learning images (121) is given, the generative model (5) adapts to each of the one or more given first learning images (121). trained to generate reconstructed images,
A model acquisition unit (11, 111B),
an image acquisition unit (11, 112) that acquires a plurality of additional images (123) showing the predetermined characteristics;
a generation unit that generates each of a plurality of restored additional images (124) obtained by restoring each of the plurality of additional images (123) by supplying each of the plurality of additional images (123) to the generation model (5); (11,113) and
An evaluation value of each additional image (124) according to the difference between each additional image (123) and each restored additional image (124), which is an evaluation value of each additional restored image (124). an evaluation unit (11, 114) that calculates an evaluation value for the degree of restoration of each of the additional images (124);
an extraction unit (11, 115) that extracts one or more additional images (126) that are evaluated as having a low degree of restoration from the plurality of additional images (123) based on the evaluation value;
When each of the one or more second learning images (126A) included in the one or more extracted additional images (126) is given by machine learning, the one or more given second learning images (126A) an additional learning unit (11, 116) that further trains the generative model (5) to generate a restored image that fits each;
Equipped with
Learning device (11B).

(Configuration 16)
The computer is
a step of acquiring one or more first learning images (121) in which predetermined features are captured;
performing machine learning of the generative model (5) using the one or more first learning images (121),
The generative model (5) is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
By the machine learning, when each of the one or more first learning images (121) is given, the machine learning generates a restored image that matches each of the one or more given first learning images (121). training the generative model (5);
step and
acquiring a plurality of additional images (123) showing the predetermined characteristics;
generating each of a plurality of restored additional images (124) obtained by restoring each of the plurality of additional images (123) by supplying each of the plurality of additional images (123) to the generation model (5); ,
An evaluation value of each additional image (123) according to the difference between each additional image (123) and each restored additional image (124), which is an evaluation value of each additional restored image (124). calculating an evaluation value for the degree of restoration of each of the additional images (123);
extracting one or more additional images (126) that are evaluated as having a low degree of restoration from the plurality of additional images (123) based on the evaluation value;
When each of the one or more second learning images (126A) included in the one or more extracted additional images (126) is given by machine learning, the one or more given second learning images (126A) further training the generative model (5) to generate restored images that fit each;
execute,
How to learn.

(Configuration 17)
to the computer,
a step of acquiring one or more first learning images (121) in which predetermined features are captured;
performing machine learning of the generative model (5) using the one or more first learning images (121),
The generative model (5) is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
By the machine learning, when each of the one or more first learning images (121) is given, the machine learning generates a restored image that matches each of the one or more given first learning images (121). training the generative model (5);
step and
acquiring a plurality of additional images (123) showing the predetermined characteristics;
generating each of a plurality of restored additional images (124) obtained by restoring each of the plurality of additional images (123) by supplying each of the plurality of additional images (123) to the generation model (5); ,
An evaluation value of each additional image (123) according to the difference between each additional image (123) and each restored additional image (124), which is an evaluation value of each additional restored image (124). calculating an evaluation value for the degree of restoration of each of the additional images (123);
extracting one or more additional images (126) that are evaluated as having a low degree of restoration from the plurality of additional images (123) based on the evaluation value;
When each of the one or more second learning images (126A) included in the one or more extracted additional images (126) is given by machine learning, the one or more given second learning images (126A) further training the generative model (5) to generate restored images that fit each;
In order to execute
learning program.

Although the embodiments of the present invention have been described, the embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and it is intended that all changes within the meaning and range equivalent to the claims are included.

1, 1B, 1C learning device, 2 detection device, 5 generation model, 11, 21 control unit, 12, 22 storage unit, 13, 23 communication interface, 14, 24 input device, 15, 25 display device, 16, 26 drive , 27 external interface, 51 input layer, 52 intermediate layer, 53 output layer, 60, 61 detection parameter candidate, 65 (binarization) threshold, 66 (judgment) threshold, 70 display screen, 71, 73 radio button, 72, 74 input field, 75, 78-81, 90, 400-402 area, 76 thumbnail image, 77 cursor, 82-89 button, 91, 92 storage medium, 100 detection system, 110 first acquisition section, 111 learning section , 111B model acquisition unit, 112 second acquisition unit, 113 generation unit, 114 evaluation unit, 115 extraction unit, 116 additional learning unit, 117, 117C third acquisition unit, 118 display control unit, 119 selection reception unit, 120 learning program , 121 first learning image, 123 image for addition, 124 image for restoration and addition, 125 evaluation value, 126 (extracted) image for addition, 126A second learning image, 127 evaluation image, 127A first evaluation image, 127B 2 evaluation image, 128 restoration evaluation image, 128A first restoration evaluation image, 128B second restoration evaluation image, 129 learning result data, 182, 183, 183A, 183B, 225 difference image, 211 acquisition unit, 212 generation unit, 213 detection section, 214 output section, 220 detection program, 221 observation image, 223 restored observation image, 405, 406 operation button, 1110 storage processing section, CA camera, I10 image, I11 restored image, I12 difference image, I13 binarized image, I14 Detection image, L defect, M calculation parameter, P1, P2, P3, P4 patch image, R product.

Claims (16)

a first acquisition unit that acquires one or more first learning images showing predetermined features;
A learning unit that performs machine learning of a generative model using the one or more first learning images,
The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
Training the generative model by the machine learning so that, when each of the one or more first learning images is given, a restored image matching each of the one or more given first learning images is generated;
Learning department and
a second acquisition unit that acquires a plurality of additional images showing the predetermined characteristics;
a generation unit that generates each of a plurality of restored and additional images obtained by restoring each of the plurality of additional images by supplying each of the plurality of additional images to the generation model;
An evaluation value of each additional image according to the difference between each additional image and each restored additional image, which is an evaluation of the degree of restoration of each additional image in each restored additional image. an evaluation section that calculates a value;
an extraction unit that extracts one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value;
By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, a restored image that matches each of the one or more given second learning images is generated. an additional learning unit that further trains the generative model to generate
a third acquisition unit that acquires a plurality of evaluation images showing the predetermined feature and other features other than the predetermined feature;
The other features are not included in the plurality of additional images;
The generation unit further generates each of a plurality of restored evaluation images obtained by restoring each of the plurality of evaluation images by giving each of the acquired evaluation images to the trained generation model,
The evaluation unit further includes:
Determining whether or not the other feature is reflected in each of the additional images based on the degree of difference between each of the additional images and the generated restored and additional images, and
Determining whether or not the other feature is reflected in each of the evaluation images based on the degree of difference between each of the evaluation images and each of the generated restored evaluation images;
The evaluation unit is configured to determine whether, among the plurality of additional images, the number or proportion of additional images determined to include the other features is equal to or higher than a first threshold, or the Among them, if the number or proportion of evaluation images determined not to include the other feature is equal to or higher than the second threshold, one or more additional images determined to include the other feature. Calculate the evaluation value for
The extraction unit selects an image that is evaluated to have a low degree of restoration from among the one or more additional images determined to include the other features based on the calculated evaluation value. Extract as one or more additional images,
learning device. Calculating the evaluation value according to the difference includes:
generating each of a plurality of difference images indicating the difference between each of the plurality of additional images and each of the plurality of restored and additional images;
In each of the difference images, the larger the pixel value of each pixel is, the higher the evaluation value is, and the contribution of pixels having a reference pixel value or more to the evaluation value is greater than the evaluation value of pixels less than the reference pixel value. Calculating the evaluation value so that it is larger than the contribution given;
Consisting of
The learning device according to claim 1. further comprising a display control unit that causes a display device to display each of the one or more extracted additional images;
The learning device according to claim 1 or 2. For each of the one or more additional images displayed on the display device, the method further includes a selection reception unit that accepts a selection as to whether or not to use each of the one or more additional images for machine learning,
The additional learning unit determines the additional image selected to be used for the machine learning as the one or more second learning images.
The learning device according to claim 3. A plurality of parameter candidates are provided for determining whether or not the other features are shown in each of the additional images and whether or not the other features are shown in each of the evaluation images,
The evaluation department is
Using each of the plurality of parameter candidates, determine whether or not the other feature is depicted in each of the additional images, and determine whether or not the other feature is depicted in each of the evaluation images. death,
Based on the results of each of the determinations, optimization of parameters for determining whether or not the other features are included in the image is performed, and in the results of each of the determinations using the optimized parameters, The number or percentage of additional images determined to include the other feature is equal to or greater than the first threshold, or the number or percentage of evaluation images determined not to include the other feature is the calculating the evaluation value for one or more additional images that are determined to include the other feature when the evaluation value is equal to or higher than a second threshold;
The learning device according to any one of claims 1 to 4 . further comprising a third acquisition unit that acquires a plurality of first evaluation images and a plurality of second evaluation images,
The plurality of first evaluation images include the predetermined feature and features other than the predetermined feature,
The plurality of second evaluation images include the predetermined feature and do not include the other feature,
The evaluation department is
A plurality of parameter candidates are given for determining whether the other feature is included in the image,
Using each of the plurality of parameter candidates, determining whether or not the other feature is depicted in each of the first evaluation images, and determining whether or not the other feature is depicted in each of the second evaluation images. perform the judgment,
Based on the results of each of the determinations, optimizing parameters for determining whether the other features are included in the image;
The selection accepting unit further accepts a selection as to whether or not the other feature is included in each of the one or more additional images displayed on the display device,
The third acquisition unit is
acquiring each additional image selected as showing the other features as one of the plurality of first evaluation images;
acquiring each additional image selected as not showing the other features as one of the plurality of second evaluation images;
The learning device according to claim 4. The second acquisition unit communicates with a detection device,
The detection device includes:
Obtaining an observation image obtained by observing the predetermined features,
generating a restored observation image corresponding to the observation image by inputting the observation image to the generative model;
Based on the difference between the generated restored observation image and the observation image, detecting whether or not other features other than the predetermined feature appear in the observation image;
The second acquisition unit acquires the observation image acquired by the detection device as one of the plurality of additional images.
The learning device according to claim 4 or 6 . further comprising a storage processing unit that stores the one or more additional images extracted by the extraction unit among the plurality of additional images in a storage unit;
The learning device according to any one of claims 1 to 7 . Each of the first learning images and each additional image is divided into a plurality of patch images,
The generative model is given for each patch image,
The learning unit generates a restored image that matches each of the one or more given first learning images by machine learning for each of the patch images, when each of the one or more first learning images is given. training the generative model to
The generation unit generates each restored additional image corresponding to each additional image for each patch image,
The evaluation unit calculates the evaluation value of each additional image for each patch image,
The extraction unit extracts the one or more additional images from the plurality of additional images based on the calculated evaluation value for each patch image,
For each of the patch images, when each of the one or more second learning images is given, the additional learning unit generates a restored image that matches each of the one or more given second learning images by machine learning. further training the generative model to generate;
The learning device according to any one of claims 1 to 5 . The generative model is configured by a neural network,
The learning device according to any one of claims 1 to 9 . The machine learning includes principal component analysis,
The learning device according to any one of claims 1 to 9 , wherein the generative model is composed of eigenvectors derived by the principal component analysis. The predetermined characteristic is a product transported on a production line that does not contain defects;
The learning device according to any one of claims 1 to 11 . an acquisition unit that acquires an observation image obtained by observing the predetermined feature;
a generation unit that generates a restored observed image corresponding to the observed image by inputting the observed image to the generative model trained by the learning device according to claim 1 ;
a detection unit that detects whether a feature other than the predetermined feature is included in the observation image based on a difference between the generated restored observation image and the observation image;
Equipped with
Detection device. A model acquisition unit that acquires a generative model trained by machine learning using one or more first learning images in which predetermined features are captured,
The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
Through the machine learning, the generative model is trained to, when each of the one or more first learning images is given, generate a restored image that matches each of the one or more given first learning images. ing,
a model acquisition section;
a first image acquisition unit that acquires a plurality of additional images showing the predetermined characteristics;
a generation unit that generates each of a plurality of restored and additional images obtained by restoring each of the plurality of additional images by supplying each of the plurality of additional images to the generation model;
An evaluation value of each additional image according to the difference between each additional image and each restored additional image, which is an evaluation of the degree of restoration of each additional image in each restored additional image. an evaluation section that calculates a value;
an extraction unit that extracts one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value;
By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, a restored image that matches each of the one or more given second learning images is generated. an additional learning unit that further trains the generative model to generate
a second image acquisition unit that acquires a plurality of evaluation images showing the predetermined feature and other features other than the predetermined feature;
The other features are not included in the plurality of additional images;
The generation unit further generates each of a plurality of restored evaluation images obtained by restoring each of the plurality of evaluation images by giving each of the acquired evaluation images to the trained generation model,
The evaluation unit further includes:
Determining whether or not the other feature is reflected in each of the additional images based on the degree of difference between each of the additional images and the generated restored and additional images, and
Determining whether or not the other feature is reflected in each of the evaluation images based on the degree of difference between each of the evaluation images and each of the generated restored evaluation images;
The evaluation unit is configured to determine whether, among the plurality of additional images, the number or proportion of additional images determined to include the other features is equal to or higher than a first threshold, or the Among them, if the number or proportion of evaluation images determined not to include the other feature is equal to or higher than the second threshold, one or more additional images determined to include the other feature. Calculate the evaluation value for
The extraction unit selects an image that is evaluated to have a low degree of restoration from among the one or more additional images determined to include the other features based on the calculated evaluation value. Extract as one or more additional images,
learning device. The computer is
obtaining one or more first learning images showing predetermined features;
performing machine learning of a generative model using the one or more first learning images, the step of:
The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
Training the generative model by the machine learning so that, when each of the one or more first learning images is given, a restored image matching each of the one or more given first learning images is generated;
step and
acquiring a plurality of additional images showing the predetermined characteristics;
generating each of a plurality of restored additional images obtained by restoring each of the plurality of additional images by supplying each of the plurality of additional images to the generation model;
An evaluation value of each additional image according to the difference between each additional image and each restored additional image, which is an evaluation of the degree of restoration of each additional image in each restored additional image. a step of calculating a value;
extracting one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value;
By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, a restored image that matches each of the one or more given second learning images is generated. further training the generative model to generate
acquiring a plurality of evaluation images showing the predetermined feature and features other than the predetermined feature;
Run
The other features are not included in the plurality of additional images;
The generating step includes a step of further generating each of a plurality of restored evaluation images obtained by restoring each of the plurality of evaluation images by giving each of the plurality of acquired evaluation images to the trained generation model,
The step of calculating is
determining whether or not the other feature is reflected in each of the additional images, based on the degree of difference between each of the additional images and the generated restored and additional images;
determining whether or not the other feature is reflected in each of the evaluation images based on the degree of difference between each of the evaluation images and each of the generated restored evaluation images;
Among the plurality of additional images, the number or proportion of additional images determined to include the other feature is equal to or higher than the first threshold, or among the plurality of evaluation images, the other If the number or proportion of evaluation images determined not to include the feature is equal to or higher than the second threshold, the evaluation value for one or more additional images determined to include the other feature. and a step of calculating
The extracting step includes selecting an image that is evaluated to have a low degree of restoration from among the one or more additional images determined to include the other features based on the calculated evaluation value. including the step of extracting the one or more additional images;
How to learn. to the computer,
obtaining one or more first learning images showing predetermined features;
performing machine learning of a generative model using the one or more first learning images, the step of:
The generative model is configured to, when an image is given, convert the given image into a feature amount, and generate an image that restores the given image from the feature amount,
Training the generative model by the machine learning so that, when each of the one or more first learning images is given, a restored image matching each of the one or more given first learning images is generated;
step and
acquiring a plurality of additional images showing the predetermined characteristics;
generating each of a plurality of restored additional images obtained by restoring each of the plurality of additional images by supplying each of the plurality of additional images to the generation model;
An evaluation value of each additional image according to the difference between each additional image and each restored additional image, which is an evaluation of the degree of restoration of each additional image in each restored additional image. a step of calculating a value;
extracting one or more additional images that are evaluated as having a low degree of restoration from the plurality of additional images based on the evaluation value;
By machine learning, when each of the one or more second learning images included in the one or more extracted additional images is given, a restored image that matches each of the one or more given second learning images is generated. further training the generative model to generate
acquiring a plurality of evaluation images showing the predetermined feature and features other than the predetermined feature;
A learning program for executing
The other features are not included in the plurality of additional images;
The generating step includes a step of further generating each of a plurality of restored evaluation images obtained by restoring each of the plurality of evaluation images by giving each of the plurality of acquired evaluation images to the trained generation model,
The step of calculating is
determining whether or not the other feature is reflected in each of the additional images, based on the degree of difference between each of the additional images and the generated restored and additional images;
determining whether or not the other feature is reflected in each of the evaluation images based on the degree of difference between each of the evaluation images and each of the generated restored evaluation images;
Among the plurality of additional images, the number or proportion of additional images determined to include the other feature is equal to or higher than the first threshold, or among the plurality of evaluation images, the other If the number or proportion of evaluation images determined not to include the feature is equal to or higher than the second threshold, the evaluation value for one or more additional images determined to include the other feature. and a step of calculating
The extracting step includes selecting an image that is evaluated to have a low degree of restoration from among the one or more additional images determined to include the other features based on the calculated evaluation value. including the step of extracting the one or more additional images;
learning program.

Images