JP7059883B2 - Learning device, image generator, learning method, and learning program - Google Patents

Description

The present invention relates to a learning device, an image generation device, a learning method, and a learning program.

Conventionally, in the scene of manufacturing a product such as a production line, a technique of photographing the manufactured product with an imaging device and inspecting the quality of the product based on the obtained image data has been used. For example, in Patent Document 1, it is determined whether the inspection target imaged in the image is normal or abnormal based on the first learned neural network, and when it is determined that the inspection target is abnormal, it is determined. An inspection device that classifies the type of the anomaly based on the learned second neural network has been proposed.

Japanese Unexamined Patent Publication No. 2012-026892

Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua, "CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training", 2017 IEEE International Conference on Computer Vision, 2764-2773, 2017

The inventors of the present invention have the following problems in the conventional technique of determining the quality of a product from image data by using a classifier composed of a learning model such as a neural network as in Patent Document 1. I found that it could happen. That is, when machine learning is performed to make the learning model acquire the ability to judge the quality of the product, a sample image is prepared as training data. If the number of sample images is small, the accuracy of the pass / fail judgment by the trained learning model (discriminator) becomes insufficient. On the other hand, it is costly to prepare a sufficient number of sample images in order to improve the accuracy of the judgment by the classifier.

Therefore, the inventors of the present invention considered using a generator (generation model) to mass-produce a plurality of images different from the prepared images, and to use the mass-produced images as training data for machine learning. For example, Non-Patent Document 1 proposes a method of constructing a generator (generative model) from prepared learning images and classes by machine learning. This generator is trained by machine learning to generate an image corresponding to the training image of the specified class. Therefore, if the image of the product is used as a learning image and the defects contained in the product are specified by class, the generator is trained to generate a possible image of the product containing the specified defects. With this trained generator, it will be possible to automatically generate images of products containing a specified class of defects. That is, it will be possible to automatically generate sample images that can be used for machine learning to equip the learning model with the ability to identify defects contained in the product. Therefore, the cost of preparing a sample image can be reduced. Then, by using the automatically generated sample image for machine learning, the number of sample images used for machine learning can be increased, and as a result, a trained learning model with a high ability to judge the quality of the product is learned. Will be able to build.

However, the present inventors have found that the following problems may occur in the situation of using this generator. That is, when training a learning model to identify defects contained in a product and detect the extent of the defect, in addition to the label indicating the type of defect, boundary information indicating the extent of the defect in the sample image (that is, For example, the ground truth of the bounding box) is prepared as teacher data. Conventionally, the range of this defect is manually specified in the sample image, and information indicating the specified range is added to the sample image as boundary information. Therefore, it takes time and effort to prepare learning data to be used for machine learning for the process of designating the range of defects in the sample image and the process of adding to the boundary information. In addition, it has been difficult for workers who do not have the knowledge to identify defects and identify the extent of the defects to add boundary information to the sample image. Therefore, it took time and effort to make the worker fully acquire the knowledge. Therefore, there may be a problem that it is costly to prepare learning data to be used for machine learning to make a learning model acquire the ability to detect a range of defects.

In addition, such a problem is not only a situation where the learning model learns the ability to detect the range of defects as well as the ability to judge the quality of the product, but also the learning model learns the ability to detect the range of some object. It can happen in any situation. For example, assume a scene in which machine learning is performed to make a learning model acquire the ability to detect the range of an organ such as an eye in a face image of a subject's face. In this scene, if the range in which the eyes are visible in the sample image is manually specified and the information indicating the specified range is manually added to the sample image as boundary information, learning data to be used for machine learning is prepared. However, it costs a lot.

The present invention, on the one hand, has been made in view of such circumstances, and an object thereof is to prepare learning data used for machine learning to acquire the ability to detect a range of an object. It is to provide a technique for reducing such a cost.

The present invention adopts the following configuration in order to solve the above-mentioned problems.

That is, the learning device according to one aspect of the present invention is based on a combination of a learning image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and boundary information indicating the range of the defect. By performing the first machine learning and the data acquisition unit that acquires a plurality of learning data sets configured respectively, the input of the label is input to the input of the label included in each of the learning data sets. A first learning processing unit that constructs a generator trained to generate an image corresponding to the training image associated with the above, and a second machine learning are included in the training data set of each of the above cases. The type of defect contained in the product and reflected in the input learning image so as to match the label and the boundary information associated with the input learning image in response to the input of the learning image. It comprises a second learning processing unit that builds an estimator trained to estimate the range.

According to the learning device according to the configuration, a generator trained to generate an image corresponding to the training image associated with the specified label is constructed, and the type and range of defects appearing in the training image are estimated. You can build an estimator trained to do so. Of these, by using a generator, it is possible to automatically generate a sample image of a product containing a specified defect. In addition, by using an estimator, it is possible to estimate the range of defects contained in the generated sample image and automatically add information indicating the estimated range of defects to the sample image as boundary information. Therefore, according to the configuration, when preparing the training data, it is possible to omit the trouble of manually specifying the range of defects in the sample image. Therefore, it is possible to reduce the cost of preparing the learning data used for machine learning to acquire the ability to detect the range of the object.

The estimator and the generator are each composed of a learning model capable of performing machine learning. For the learning model, for example, a neural network or the like may be used. The order in which the first machine learning and the second machine learning are executed may not be particularly limited and may be appropriately selected depending on the embodiment. The first machine learning and the second machine learning may be executed simultaneously or separately. When the first and second machine learning are executed separately, the first machine learning may be executed before the second machine learning or may be executed after the second machine learning. good.

Further, the type of the product is not particularly limited as long as it can be subject to visual inspection, and may be appropriately selected according to the embodiment. The product may be, for example, a product transported on a production line for electronic devices, electronic parts, automobile parts, chemicals, foods, and the like. The electronic device is, for example, a mobile phone or the like. Electronic components are, for example, boards, chip capacitors, liquid crystals, relay windings, and the like. Automotive parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. The chemical may be, for example, a packaged tablet, an unpackaged tablet, or the like. Further, the type of the defect may be not particularly limited as long as it may be related to the quality of the product, and may be appropriately selected according to the embodiment. Defects include, for example, scratches, stains, cracks, dents, burrs, color unevenness, foreign matter contamination, and the like.

In the learning apparatus according to the above aspect, the input of the generator may be connected to the output of the encoder, and the output of the generator may be connected to the input of the estimator and the input of the discriminator. Then, performing the first machine learning means that the input image input to the discriminator is the image generated by the generator or the training image obtained from the plurality of training data sets. The generator is provided so that the first training step of training the discriminator to discriminate is present and the image generated by the generator is such that the discriminator misleads the discriminator. The second training step to be trained and the image generated by the generator so that the result of estimating the type of defect contained in the product shown in the image by the estimator matches the label. In addition, the third training step for training the generator and the image generated by the generator by inputting the label and the latent variable into the generator are restored to the training image. In the fourth training step of training the encoder and the generator, the latent variables are derived based on the output obtained from the encoder by inputting the label and the training image into the encoder. It may include a fourth training step and a fifth training step in which the encoder is trained so that the distribution of the latent variables derived from the output obtained from the encoder follows a predetermined prior distribution. According to this configuration, a trained generator and estimator can be appropriately constructed, thereby preparing learning data to be used for machine learning to acquire the ability to detect a range of an object. It is possible to reduce the cost required for. The order in which the first to fifth training steps are executed may not be particularly limited, and may be appropriately selected depending on the embodiment.

In the learning apparatus according to the above aspect, the output of the generator may be connected to the input of the discriminator. Then, performing the first machine learning means that the input image input to the discriminator is the image generated by the generator or the learning image obtained from the plurality of training data sets. The generator so that the first training step of training the discriminator and the image generated by the generator are such that the discriminator misleads the discriminator. It may include alternating with the second training step of training. According to this configuration, a trained generator can be properly constructed, and thus the cost of preparing training data to be used for machine learning to acquire the ability to detect a range of an object. Can be reduced.

In the learning device according to the above aspect, the range of the defect may be represented by a bounding box. According to this configuration, it is possible to reduce the cost of preparing learning data used for machine learning to acquire the ability to detect a bounding box as a range of an object.

Further, the image generator according to one aspect of the present invention includes the defect by giving an input value indicating the type of the defect to the generator constructed by the learning device according to any one of the above embodiments. By giving the generated image generated to the estimator constructed by the learning device and the generation unit that generates the generated image showing the product, the type of the defect contained in the product reflected in the generated image and the type of the defect and the defect. An estimation unit for estimating the range, a determination unit for determining whether or not the result of estimating the type of the defect included in the product shown in the generated image matches the type of the defect indicated by the input value, and a determination unit. When the determination unit determines that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image is associated with the result of estimating the range of the defect. It is provided with an image storage unit for storing in the storage area of. According to this configuration, it is possible to generate learning data that can be used for machine learning to acquire the ability to detect the range of an object without having to manually specify the range of the object.

Further, the learning device and the image generation device according to each of the above forms are not only capable of learning the learning model the ability to detect the range of defects as well as the ability to judge the quality of the product, but also the ability to detect the range of some object. May be applied to any situation that makes the learning model learn. For example, even if the learning device and the image generation device according to each of the above forms are applied to a scene in which the learning model learns the ability to detect the range of an organ such as an eye in a face image of a subject's face. good.

For example, the learning device according to one aspect of the present invention is configured by a combination of a learning image showing an object, a label indicating the type of the object, and boundary information indicating the range of the object in the learning image. By performing the first machine learning with the data acquisition unit that acquires a plurality of learned data sets, the input of the label included in each of the learning data sets is associated with the input label. The first learning processing unit for constructing a generator trained to generate an image corresponding to the learned image and the second machine learning are included in each of the learning data sets. To estimate the type and range of the object appearing in the input learning image so as to match the label and the boundary information associated with the input learning image with respect to the input of the learning image. It includes a second learning processing unit that builds a trained estimator.

The type of the "object" is not particularly limited as long as it can be an object to be detected from the image, and may be appropriately selected according to the embodiment. The object may be, for example, a product to be visually inspected, a defect contained in the product, a person, a body part of the person (for example, a face, etc.), an organ of the person, or the like. Further, when the present invention is used for a monitoring image of road traffic, the object may be, for example, a car, a pedestrian, a sign, or the like.

The image generation system according to one aspect of the present invention may be configured by the learning device and the image generation device according to any one of the above forms. Further, the estimation system according to one aspect of the present invention uses the learning device according to any one of the above forms, the image generation device, and the generated image to estimate the range of the object to be captured in the image. It may be configured by an estimator generator to be constructed and an estimator to estimate the range of an object to be captured in an image by using the constructed estimator. Further, as another form of each of the learning device, the image generation device, the image generation system, and the estimation system according to each of the above forms, one aspect of the present invention may be an information processing method that realizes each of the above configurations. However, it may be a program, or it may be a storage medium that stores such a program and can be read by a computer or the like. Here, the storage medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

For example, in the learning method according to one aspect of the present invention, the computer uses a learning image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and boundary information indicating the range of the defect. By performing the step of acquiring a plurality of learning data sets configured by the combination of the above and the first machine learning, the input of the label included in the learning data set of each of the above cases is input. The learning included in each of the training datasets by performing a step of building a generator trained to generate an image corresponding to the training image associated with the label and a second machine learning. For the input of the image, the type and range of the defect contained in the product reflected in the input learning image is estimated so as to match the label and the boundary information associated with the input learning image. It is an information processing method that performs, with the steps of building an estimator trained to do.

Further, for example, the learning program according to one aspect of the present invention shows on a computer a learning image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and a range of the defect. By performing the step of acquiring a plurality of learning data sets configured by the combination of boundary information and the first machine learning, it is input to the input of the label included in each of the learning data sets. A step of constructing a generator trained to generate an image corresponding to the training image associated with the label and a second machine learning are included in each of the training data sets. The type and range of the defect contained in the product reflected in the input training image so as to match the label and the boundary information associated with the input training image with respect to the input of the training image. It is a program to build and execute an estimator trained to estimate.

Further, for example, in the learning method according to one aspect of the present invention, the computer uses a learning image showing an object, a label indicating the type of the object, and boundary information indicating the range of the object in the learning image. By performing the step of acquiring a plurality of learning data sets configured by the combination of the above and the first machine learning, the input of the label included in the learning data set of each of the above cases is input. The learning included in each of the training datasets by performing a step of building a generator trained to generate an image corresponding to the training image associated with the label and a second machine learning. Training to estimate the type and range of the object appearing in the input learning image so that the input of the image matches the label and the boundary information associated with the input learning image. It is an information processing method that executes the steps of constructing the estimator.

Further, for example, in the learning program according to one aspect of the present invention, a learning image showing an object, a label indicating the type of the object, and boundary information indicating the range of the object in the learning image are shown on a computer. By performing the step of acquiring a plurality of learning data sets configured by the combination of the above and the first machine learning, the input of the label included in the learning data set of each of the above cases is input. The learning included in each of the training datasets by performing a step of building a generator trained to generate an image corresponding to the training image associated with the label and a second machine learning. Training to estimate the type and range of the object appearing in the input learning image so that the input of the image matches the label and the boundary information associated with the input learning image. It is a program to execute the steps to build the estimator.

According to the present invention, it is possible to reduce the cost of preparing learning data used for machine learning to acquire the ability to detect a range of an object.

FIG. 1 schematically illustrates an example of a situation in which the present invention is applied. FIG. 2 schematically illustrates an example of boundary information indicating the range of defects specified in the learning image according to the embodiment. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device according to the embodiment. FIG. 4 schematically illustrates an example of the hardware configuration of the image generator according to the embodiment. FIG. 5 schematically illustrates an example of the hardware configuration of the estimator generator according to the embodiment. FIG. 6 schematically illustrates an example of the hardware configuration of the inspection device according to the embodiment. FIG. 7 schematically illustrates an example of the software configuration of the learning device according to the embodiment. FIG. 8A schematically illustrates an example of the machine learning process of the learning network according to the embodiment. FIG. 8B schematically illustrates an example of the machine learning process of the learning network according to the embodiment. FIG. 9 schematically illustrates an example of the software configuration of the image generator according to the embodiment. FIG. 10 schematically illustrates an example of the software configuration of the estimator generator according to the embodiment. FIG. 11 schematically illustrates an example of the software configuration of the inspection device according to the embodiment. FIG. 12 illustrates an example of the processing procedure of the learning device according to the embodiment. FIG. 13 illustrates an example of the processing procedure of the image generator according to the embodiment. FIG. 14 illustrates an example of the processing procedure of the estimator generator according to the embodiment. FIG. 15 illustrates an example of the processing procedure of the inspection device according to the embodiment. FIG. 16 schematically illustrates an example of the configuration of the learning network according to the modified example. FIG. 17A schematically illustrates an example of the machine learning process of the learning network according to the modified example. FIG. 17B schematically illustrates an example of the machine learning process of the learning network according to the modified example. FIG. 17C schematically illustrates the process of assuming machine learning of the estimator according to the modified example. FIG. 18 illustrates an example of the processing procedure of the first machine learning according to the modified example. FIG. 19 schematically illustrates a modified example of the scene to which the present invention is applied. FIG. 20 schematically illustrates an example of the software configuration of the detection device according to the modified example.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the present embodiment are described in natural language, but more specifically, they are specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application example First, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a situation in which the present invention is applied to a visual inspection of a product R. The defect that may be contained in the product R is an example of the "object" of the present invention. However, the scope of application of the present invention is not limited to the examples of visual inspection illustrated below. The present invention is applicable to all situations where the range of any object in an image is detected.

As illustrated in FIG. 1, the inspection system 100 according to the present embodiment includes a learning device 1, an image generation device 2, an estimator generation device 3, and an inspection device 4 connected via a network. As a result, the inspection system 100 is configured to inspect the quality of the product R. The type of network between the learning device 1, the image generator 2, the estimator generator 3, and the inspection device 4 may be not particularly limited, and may be, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, and the like. It may be appropriately selected from a dedicated network or the like.

In the example of FIG. 1, the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 are separate computers. However, the configuration of the inspection system 100 does not have to be limited to such an example. At least one pair of the learning device 1, the image generator 2, the estimator generator 3, and the inspection device 4 may be an integrated computer. Further, the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 may each be composed of a plurality of computers.

The learning device 1 according to the present embodiment is a generator 50 for generating an image of the product R to be visually inspected, and a defect included in the product R with respect to the image generated by the generator 50. It is a computer configured to construct an estimator 52 for estimating a type and a range. Specifically, the learning device 1 is formed by a combination of a learning image 61 showing the product R to be visually inspected, a label 63 indicating the type of defect included in the product R, and boundary information 65 indicating the range of defects. A plurality of training data sets 60 configured for each are acquired.

Subsequently, the learning device 1 executes the first machine learning to correspond to the learning image 61 associated with the input label 63 with respect to the input of the label 63 included in each learning data set 60. Build a generator 50 trained to generate an image. The image corresponding to the learning image 61 is an image similar to the learning image 61 and in which the product R can be captured. The generator 50 is trained to generate an image that follows (follows) the distribution of the training image 61. Further, by executing the second machine learning, the learning device 1 receives the input of the learning image 61 included in each learning data set 60, and the label 63 and the boundary information associated with the input learning image 61. Construct an estimator 52 trained to estimate the type and range of defects contained in the product R in the input learning image 61 so as to match 65.

Here, an example of a method of designating a defect range will be described with reference to FIG. FIG. 2 schematically illustrates an example of boundary information 65 indicating a range of defects specified in the learning image 61 according to the present embodiment. In the example of FIG. 2, the learning image 61 shows the product R including the elliptical defect 612. The width of the training image 61 is W pixel and the height is H pixel, whereas the width of the defect 612 is w pixel and the height is h pixel. The range of the defect 612 may be appropriately specified so as to include the area in which the defect 612 appears in the learning image 61.

The method for expressing the range of the defect 612 does not have to be particularly limited, and may be appropriately selected depending on the embodiment. In this embodiment, the range of defects 612 is represented by a bounding box. As shown in FIG. 2, the bounding box can be defined by the center coordinates (cx, cy), width (w pixels), and height (h pixels) of the defect 612. That is, the rectangular range (the range shown by the dotted line in the figure) surrounding the outer circumference of the defect 612 is an example of the bounding box. However, the format that defines the bounding box does not have to be limited to such an example, and may be appropriately selected depending on the embodiment. For example, the bounding box may be defined by the center coordinates and the radius of the circle (eg, the circumscribed circle) designated to include the defect 612. In this case, the range of the circular shape including the defect 612 becomes the bounding box. According to the learning device 1 according to the present embodiment, the generator 50 trained to generate the image corresponding to the learning image 61 is constructed, and the range of defects represented by such a bounding box is defined as the learning image. An estimator 52 trained to estimate within 61 can be constructed.

Returning to FIG. 1, the image generation device 2 according to the present embodiment uses the trained generator 50 and the estimator 52 constructed by the learning device 1 to generate and generate a possible image of the product R. It is a computer configured to estimate the extent of defects in the image. Specifically, the image generation device 2 gives the generator 50 constructed by the learning device 1 an input value 71 indicating the type of the defect, thereby generating a generated image 73 showing the product R including the defect. .. The generated generated image 73 corresponds to the learning image 61.

Next, the image generation device 2 estimates the type and range of defects included in the product R reflected in the generated image 73 by giving the generated image 73 generated to the estimator 52 constructed by the learning device 1. Subsequently, the image generation device 2 determines whether or not the result of estimating the type of defect included in the product shown in the generated image 73 matches the type of defect indicated by the input value 71. Then, when the image generation device 2 determines that the result of estimating the defect type matches the defect type indicated by the input value 71, the generated image 73 is associated with the result of estimating the defect range. Store in a predetermined storage area.

The estimator generator 3 according to the present embodiment is a computer configured to construct an estimator for estimating the quality of the product R. Specifically, the estimator generator 3 acquires a plurality of learning data sets each composed of a combination of a sample image showing the product R and correct answer data. The correct answer data shows the result (that is, the correct answer) of determining the quality of the product R shown in the sample image. In the present embodiment, the correct answer data includes boundary information indicating the range of defects included in the product R shown in the sample image. Then, the estimator generator 3 performs machine learning using a plurality of learning data sets to determine the quality of the product R appearing in the given image, and if the product R contains a defect, the estimator generator 3 determines the quality of the product R. Build a trained estimator that has acquired the ability to detect the extent of the defect. The estimator generator 3 uses the generated image 73 generated by the image generator 2 as a sample image, and uses information indicating the result of estimating the range of defects associated with the generated image 73 as correct answer data. can do.

On the other hand, the inspection device 4 according to the present embodiment is a computer configured to judge the quality of the product R by using the trained estimator constructed by the estimator generation device 3. The inspection device 4 is an example of an estimation device that estimates the range of an object to be captured in an image. Specifically, the inspection device 4 acquires a target image of the product R to be visually inspected. In the present embodiment, the camera CA is connected to the inspection device 4. The inspection device 4 acquires a target image by photographing the product R with this camera CA. Next, the inspection device 4 inputs the acquired target image to the trained estimator, and executes the arithmetic processing of the trained estimator. As a result, the inspection device 4 acquires an output value corresponding to the result of determining the quality of the product R and the result of detecting the range of defects included in the product R from the trained estimator. Then, the inspection device 4 outputs information regarding the result of determining the quality of the product R based on the output value obtained from the learned estimator.

As described above, according to the learning device 1 according to the present embodiment, the generator 50 trained to generate an image corresponding to the learning image 61 associated with the label 63 that specifies the type of defect is constructed. , An estimator 52 trained to estimate the type and extent of defects in the learning image 61 can be constructed. Of these, by using the generator 50, it is possible to automatically generate a sample image of the product including the specified defect. In addition, by using the estimator 52, the range of defects contained in the product R reflected in the generated sample image is estimated, and information indicating the estimated range of defects is automatically added to the sample image as boundary information. can do. Therefore, according to the present embodiment, when preparing the training data, it is possible to omit the trouble of manually specifying the range of defects in the sample image. Therefore, it is possible to reduce the cost of preparing the learning data used for machine learning to acquire the ability to detect the range of defects.

Further, in the image generation device 2 according to the present embodiment, by using the generator 50 and the estimator 52 constructed by the learning device 1, it is possible to save the trouble of manually specifying the range of the defect and to eliminate the trouble of manually specifying the defect. It is possible to generate learning data that can be used for machine learning to acquire the ability to detect a range. In addition, in the present embodiment, the generated image 73 generated by the image generation device 2 is used as a sample image, and the information indicating the result of estimating the range of defects associated with the generated image 73 is used as correct answer data. be able to. This makes it possible to reduce the cost of collecting the training data set in the estimator generator 3. Further, in the present embodiment, the generated image 73 generated by the image generation device 2 is used as a sample image, and the information indicating the result of estimating the range of defects associated with the generated image 73 is used as correct answer data. Therefore, the number of training data sets used for machine learning of the estimator can be increased. This makes it possible to improve the accuracy of estimating the range of defects that may be contained in the product R in the inspection device 4.

The product R to be inspected for appearance may not be particularly limited and may be appropriately selected according to the embodiment. The product R may be, for example, a product transported on a production line such as an electronic component or an automobile component. Electronic components are, for example, boards, chip capacitors, liquid crystals, relay windings, and the like. Automotive parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. Further, the quality determination may be simply determining whether or not the product R has a defect, and in addition to determining whether or not the product R has a defect, the type of the defect is identified. May include doing. The defects are, for example, scratches, stains, cracks, dents, dust, burrs, color unevenness, and the like.

§2 Configuration example [Hardware configuration]
<Learning device>
Next, an example of the hardware configuration of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device 1 according to the present embodiment.

As shown in FIG. 3, the learning device 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected. .. In FIG. 3, the communication interface is described as "communication I / F".

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors, and is configured to execute information processing based on a program and various data. To. The storage unit 12 is an example of a memory, and is composed of, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage unit 12 stores various information such as the learning program 121, a plurality of learning data sets 60, the first learning result data 123, and the second learning result data 125.

The learning program 121 builds a generator 50 trained to generate a viable image of product R and an estimator 52 trained to estimate the types and extents of defects that may be contained in product R appearing in the image. This is a program for causing the learning device 1 to execute information processing (FIG. 12) described later. The learning program 121 includes a series of instructions for the information processing. Each training data set 60 is composed of a combination of a training image 61 showing the product R to be visually inspected, a label 63 indicating the type of defect included in the product R, and boundary information 65 indicating the range of defects. .. Each training data set 60 is used for machine learning of the generator 50 and the estimator 52. The number of training data sets 60 may be appropriately determined according to the embodiment. The first learning result data 123 is data for setting the trained generator 50 constructed by machine learning. The second learning result data 125 is data for setting the trained estimator 52 constructed by machine learning. The first learning result data 123 and the second learning result data 125 are generated as the execution result of the learning program 121. Details will be described later.

The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 13, the learning device 1 can perform data communication via a network with another information processing device (for example, an image generation device 2 and an estimator generation device 3).

The input device 14 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 15 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the learning device 1 by using the input device 14 and the output device 15.

The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 91. The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. At least one of the learning program 121 and the plurality of learning data sets 60 may be stored in the storage medium 91.

The storage medium 91 transfers the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the computer or other device, the machine or the like can read the information of the recorded program or the like. It is a medium to accumulate. The learning device 1 may acquire at least one of the learning program 121 and the plurality of learning data sets 60 from the storage medium 91.

Here, in FIG. 3, as an example of the storage medium 91, a disc-type storage medium such as a CD or a DVD is illustrated. However, the type of the storage medium 91 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories.

Regarding the specific hardware configuration of the learning device 1, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The learning device 1 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the learning device 1 may be a general-purpose server device, a PC (Personal Computer), or the like, in addition to an information processing device designed exclusively for the provided service.

<Image generator>
Next, an example of the hardware configuration of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 4 schematically illustrates an example of the hardware configuration of the image generation device 2 according to the present embodiment.

As shown in FIG. 4, the image generation device 2 according to the present embodiment is a computer to which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected. be. Each of the control unit 21 to the drive 26 of the image generation device 2 according to the present embodiment may be configured in the same manner as each of the control unit 11 to the drive 16 of the learning device 1.

That is, the control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as the image generation program 221, the first learning result data 123, the second learning result data 125, or a plurality of data sets.

The image generation program 221 uses the trained generator 50 to generate the generated image 73, and also uses the trained estimator 52 to estimate the range of defects in the generated image 73, which will be described later. FIG. 13) is a program for causing the image generation device 2 to execute. The image generation program 221 includes a series of instructions for the information processing. Each data set is composed of a combination of the generated image 73, the label 75, and the boundary information 77. The label 75 corresponds to the input value 71 given to the generator 50 when the generated image 73 is generated. The boundary information 77 shows the result of estimating the range of defects by the estimator 52. Each data set is generated as a result of executing the image generation program 221. Details will be described later.

The communication interface 23 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 23, the image generation device 2 can perform data communication via a network with another information processing device (for example, a learning device 1 and an estimator generation device 3).

The input device 24 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 25 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the image generation device 2 by using the input device 24 and the output device 25.

The drive 26 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 92. At least one of the image generation program 221, the first learning result data 123, and the second learning result data 125 may be stored in the storage medium 92. Further, the image generation device 2 may acquire at least one of the image generation program 221, the first learning result data 123, and the second learning result data 125 from the storage medium 92.

Regarding the specific hardware configuration of the image generation device 2, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 21 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 22 may be composed of a RAM and a ROM included in the control unit 21. At least one of the communication interface 23, the input device 24, the output device 25, and the drive 26 may be omitted. The image generation device 2 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the image generation device 2 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Estimator generator>
Next, an example of the hardware configuration of the estimator generator 3 according to the present embodiment will be described with reference to FIG. FIG. 5 schematically illustrates an example of the hardware configuration of the estimator generator 3 according to the present embodiment.

As shown in FIG. 5, the estimator generator 3 according to the present embodiment is a computer to which a control unit 31, a storage unit 32, a communication interface 33, an input device 34, an output device 35, and a drive 36 are electrically connected. Is. Each of the control unit 31 to the drive 36 of the estimator generation device 3 according to the present embodiment may be configured in the same manner as each of the control unit 11 to the drive 16 of the learning device 1.

That is, the control unit 31 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 32 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 32 stores various information such as the estimator generation program 321, the learning data set 322, and the third learning result data 325.

The estimator generation program 321 is a program for causing the estimator generation device 3 to execute information processing (FIG. 14) described later, which constructs an estimator for determining the quality of the product R. The estimator generation program 321 includes a series of instructions for the information processing. The training data set 322 is used for machine learning of this estimator. The third learning result data 325 is data for setting the trained estimator constructed by machine learning. The third learning result data 325 is generated as an execution result of the estimator generation program 321. Details will be described later.

The communication interface 33 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 33, the estimator generation device 3 can perform data communication via a network with other information processing devices (for example, a learning device 1, an image generation device 2, and an inspection device 4). ..

The input device 34 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 35 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the estimator generator 3 by using the input device 34 and the output device 35.

The drive 36 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 93. At least one of the estimator generation program 321 and the learning data set 322 may be stored in the storage medium 93. Further, the estimator generation device 3 may acquire at least one of the estimator generation program 321 and the learning data set 322 from the storage medium 93.

Regarding the specific hardware configuration of the estimator generator 3, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 31 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 32 may be composed of a RAM and a ROM included in the control unit 31. At least one of the communication interface 33, the input device 34, the output device 35, and the drive 36 may be omitted. The estimator generator 3 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the estimator generator 3 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Inspection equipment>
Next, an example of the hardware configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 6 schematically illustrates an example of the hardware configuration of the inspection device 4 according to the present embodiment.

As shown in FIG. 6, in the inspection device 4 according to the present embodiment, the control unit 41, the storage unit 42, the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47 are electrically connected to each other. It is a computer. In FIG. 6, the external interface is described as "external I / F". The control units 41 to 46 of the inspection device 4 may be configured in the same manner as the control units 11 to 16 of the learning device 1, respectively.

That is, the control unit 41 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 42 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 42 stores various information such as the inspection program 421 and the third learning result data 325.

The inspection program 421 uses the trained estimator constructed by the estimator generation device 3 to cause the inspection device 4 to execute information processing (FIG. 15) described later for determining the quality of the product R shown in the target image. It is a program for. The inspection program 421 includes a series of instructions for the information processing. Details will be described later.

The communication interface 43 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 43, the inspection device 4 can perform data communication via a network with another information processing device (for example, an estimator generation device 3).

The input device 44 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 45 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the inspection device 4 by using the input device 44 and the output device 45.

The drive 46 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 94. At least one of the inspection program 421 and the third learning result data 325 may be stored in the storage medium 94. Further, the inspection device 4 may acquire at least one of the inspection program 421 and the third learning result data 325 from the storage medium 94.

The external interface 47 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of the external interfaces 47 may be appropriately selected according to the type and number of connected external devices. In this embodiment, the inspection device 4 is connected to the camera CA via the external interface 47.

The camera CA is used to acquire a target image of the product R. The type and location of the camera CA are not particularly limited and may be appropriately determined according to the embodiment. As the camera CA, for example, a known camera such as a digital camera or a video camera may be used. Further, the camera CA may be arranged in the vicinity of the production line on which the product R is conveyed. When the camera CA includes a communication interface, the inspection device 4 may be connected to the camera CA via the communication interface 43 instead of the external interface 47.

Regarding the specific hardware configuration of the inspection device 4, the components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 41 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 42 may be composed of a RAM and a ROM included in the control unit 41. At least one of the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47 may be omitted. The inspection device 4 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, as the inspection device 4, in addition to an information processing device designed exclusively for the provided service, a general-purpose server device, a general-purpose desktop PC, a notebook PC, a tablet PC, a mobile phone including a smartphone, or the like may be used.

[Software configuration]
<Learning device>
Next, an example of the software configuration of the learning device 1 according to the present embodiment will be described with reference to FIG. 7. FIG. 7 schematically illustrates an example of the software configuration of the learning device 1 according to the present embodiment.

The control unit 11 of the learning device 1 expands the learning program 121 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the learning program 121 expanded in the RAM by the CPU, and controls each component. As a result, as shown in FIG. 7, the learning device 1 according to the present embodiment is a computer including a data acquisition unit 111, a first learning processing unit 112, a second learning processing unit 113, and a storage processing unit 114 as software modules. Works as. That is, in the present embodiment, each software module of the learning device 1 is realized by the control unit 11 (CPU).

The data acquisition unit 111 is composed of a combination of a learning image 61 showing the product R to be visually inspected, a label 63 indicating the type of defect included in the product R, and boundary information 65 indicating the range of defects. Acquire a plurality of training data sets 60. By executing the first machine learning, the first learning processing unit 112 corresponds to the learning image 61 associated with the input label 63 with respect to the input of the label 63 included in each learning data set 60. Build a generator 50 trained to generate an image. By executing the second machine learning, the second learning processing unit 113 receives the input of the training image 61 included in each learning data set 60 with respect to the label 63 and the boundary associated with the input learning image 61. An estimator 52 trained to estimate the type and range of defects contained in the product R appearing in the input learning image 61 is constructed so as to match the information 65. The storage processing unit 114 stores information about the constructed trained generator 50 and information about the constructed trained estimator 52 in a predetermined storage area.

(Learning network)
Next, an example of the configuration of the learning network 500 including the generator 50 and the estimator 52 according to the present embodiment will be described with reference to FIGS. 8A and 8B. 8A and 8B schematically illustrate an example of a first machine learning process that trains the generator 50 and a second machine learning process that trains the estimator 52.

As shown in each figure, in the present embodiment, the learning network 500 is configured by the encoder 56, the generator 50, the discriminator 54, and the estimator 52. The first learning processing unit 112 and the second learning processing unit 113 hold a learning network 500 for carrying out this machine learning. In the learning network 500, the encoder 56, the generator 50, the discriminator 54, and the estimator 52 are arranged in this order from the input side. The discriminator 54 and the estimator 52 are arranged in parallel on the output side. The encoder 56 is configured to accept the inputs of the training image 61 and the label 63 and output the output values corresponding to the latent variables along the prior distribution. The output of the encoder 56 is connected to the input of the generator 50. The output of the generator 50 is connected to the input of the estimator 52 and the input of the discriminator 54.

The encoder 56, the generator 50, the estimator 52, and the discriminator 54 according to the present embodiment are each composed of a multi-layered neural network used for so-called deep learning. The encoder 56 includes an input layer 571, an intermediate layer (hidden layer) 572, and an output layer 573. The generator 50 includes an input layer 511, an intermediate layer (hidden layer) 512, and an output layer 513. The estimator 52 includes an input layer 531, an intermediate layer (hidden layer) 532, and an output layer 533. The discriminator 54 includes an input layer 551, an intermediate layer (hidden layer) 552, and an output layer 553. However, the configurations of the encoder 56, the generator 50, the estimator 52, and the discriminator 54 are not limited to such an example, and may be appropriately set according to the embodiment. For example, the encoder 56, the generator 50, the estimator 52, and the discriminator 54 may each include two or more intermediate layers. The configurations of the encoder 56, the generator 50, the estimator 52, and the discriminator 54 may be different from each other.

The number of neurons (nodes) contained in each layer (571 to 573, 511 to 513, 513 to 533, 551 to 553) may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the examples of FIGS. 8A and 8B, each neuron is associated with all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment. A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold.

The weight of the connection between each neuron and the threshold value of each neuron included in each layer 571 to 573 of the encoder 56 are examples of the parameters of the encoder 56 used for arithmetic processing. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 511 to 513 of the generator 50 are examples of the parameters of the generator 50 used for arithmetic processing. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 531 to 533 of the estimator 52 are examples of the parameters of the estimator 52 used for arithmetic processing. The weight of the connection between each neuron included in each layer 551 to 553 of the discriminator 54 and the threshold value of each neuron are examples of the parameters of the discriminator 54 used for arithmetic processing.

The first learning processing unit 112 and the second learning processing unit 113 carry out machine learning of the learning network 500. The first learning processing unit 112 carries out the first machine learning regarding the generator 50 among them. On the other hand, the second learning processing unit 113 carries out the second machine learning regarding the estimator 52.

In this embodiment, performing the first machine learning includes the following first to fifth training steps. In the first training step, in the first training processing unit 112, the input image input to the discriminator 54 is an image 69 generated by the generator 50 or a learning image 61 obtained from a plurality of training data sets 60. The discriminator 54 is trained to discriminate whether or not it is. That is, the discriminator 54 is trained to discriminate whether the given input image is derived from the learning data (learning image 61) or the generator 50. In the second training step, the first learning processing unit 112 trains the generator 50 so that the image 69 generated by the generator 50 misleads the discrimination by the discriminator 54. In the examples of FIGS. 8A and 8B, the fact that the data is derived from the training data is expressed as "true", and the fact that the data is derived from the generator 50 is expressed as "false". However, the method for expressing each origin is not limited to such an example, and may be appropriately selected depending on the embodiment.

In the third training step, the first learning processing unit 112 uses the estimator 52 to estimate the type of defect contained in the product R, which is the image 69 generated by the generator 50, and the corresponding label 63. Train the generator 50 to be consistent with. In the fourth training step, in the first learning processing unit 112, the image 69 generated by the generator 50 by inputting the label 63 and the latent variable to the generator 50 restored the training image 61 (that is, learning). The encoder 56 and the generator 50 are trained so as to be (the image corresponding to the image 61). Latent variables are derived based on the output obtained from the encoder 56 by inputting the label 63 and the training image 61 to the encoder 56. In the fifth training step, the first learning processing unit 112 trains the encoder 56 so that the distribution of the latent variables derived based on the output obtained from the encoder 56 follows a predetermined prior distribution. The type of prior distribution may not be particularly limited and may be appropriately selected depending on the embodiment. As the prior distribution, a known distribution such as a Gaussian distribution may be used. The order in which the first to fifth training steps are executed may not be particularly limited, and may be appropriately determined according to the embodiment.

Specifically, as shown in FIG. 8A, the second learning processing unit 113 inputs the learning image 61 to the input layer 531 of the estimator 52 for each learning data set 60, and executes the arithmetic processing of the estimator 52. .. As a result, the second learning processing unit 113 acquires an output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the learning image 61 from the output layer 533. The second learning processing unit 113 calculates an error ( _LC ) between the output value corresponding to the result of estimating the defect type and the value of the label 63. Further, the second learning processing unit 113 calculates an error ( _LBB ) between the output value corresponding to the result of estimating the defect range and the value of the boundary information 65. For the calculation of the error ( _LBB ), for example, a known index such as IOU (Intersection Over Union) may be used.

Further, the first learning processing unit 112 inputs the learning image 61 and the label 63 to the input layer 571 of the encoder 56 for each learning data set 60, and executes the arithmetic processing of the encoder 56. As a result, the first learning processing unit 112 acquires an output value (latent variable) corresponding to the result of deriving some feature amount from the output layer 573 of the encoder 56. This output value may be set to correspond to any random variable of prior distribution (eg, mean and covariance). The first learning processing unit 112 calculates an error (L _KL ) between this output value and the value derived from the prior distribution.

Next, the first learning processing unit 112 inputs the output value obtained from the encoder 56 and the corresponding label 63 to the input layer 511 of the generator 50, and executes the arithmetic processing of the generator 50. As a result, the first learning processing unit 112 acquires the output (image 69) corresponding to the result of generating the image corresponding to the learning image 61 from the label 63 for each learning data set 60 from the output layer 513.

Subsequently, the first learning processing unit 112 inputs each image 69 generated by the generator 50 to the input layer 551 of the discriminator 54, and executes the arithmetic processing of the discriminator 54. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 (that is, the learning image 61 itself) from the output layer 553. In this scene, since each image 69 generated by the generator 50 is an input image, the correct answer is that the discriminator 54 discriminates as "false". The first learning processing unit 112 calculates an error ( _LD ) between the output value obtained from the output layer 553 and the correct answer for each image 69 generated by the generator 50.

Similarly, the first learning processing unit 112 inputs the learning image 61 to the input layer 551 of the discriminator 54 for each learning data set 60, and executes the arithmetic processing of the discriminator 54. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 from the output layer 553. In this scene, since the learning image 61 itself is an input image, the correct answer is that the discriminator 54 determines that it is “true”. The first learning processing unit 112 calculates an error ( _LD ) between the output value obtained from the output layer 553 and the correct answer for each learning data set 60.

Further, as shown in FIG. 8B, the first learning processing unit 112 causes the discriminator 54 to make an error in discriminating the output value obtained by inputting each image 69 generated by the generator 50 into the discriminator 54. Calculate the error (L _GD ) for training the generator 50 to produce such an image. Specifically, in the training of the generator 50, the correct answer is to mislead the result of the discrimination by the discriminator 54. That is, the correct answer is that the output value obtained from the output layer 553 corresponds to "true". The first learning processing unit 112 calculates an error (L _GD ) between the output value obtained from the output layer 553 by a series of processing and the correct answer (that is, “true”) for each learning data set 60.

Next, the first learning processing unit 112 inputs each image 69 generated by the generator 50 to the input layer 531 of the estimator 52, and executes the arithmetic processing of the estimator 52. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of estimating the type of the defect included in the product R shown in each image 69 from the output layer 533 of the estimator 52. The first learning processing unit 112 calculates an error ( _LGC ) between the output value corresponding to the result of estimating the defect type and the value of the corresponding label 63.

Next, as shown in FIG. 8A, the first training processing unit 112 has an error for training the generator 50 to function as a decoder for restoring the training image 61 from the output of the encoder 56 for each training data set 60. Is calculated. Specifically, the first learning processing unit 112 calculates an error ( _LG ) between the image 69 generated by the generator 50 and the corresponding learning image 61 for each training data set 60.

Then, as shown in FIGS. 8A and 8B, the first learning processing unit 112 and the second learning processing unit 113 are calculated for each error (LC, _{L BB} , L _KL , _{LD, L GD ,} L _GC ). , _LG ) are adjusted so that the value of each parameter of the learning network 500 is small. Specifically, the second learning processing unit 113 adjusts the value of the parameter of the estimator 52 so that the sum of the calculated errors (LC, _{LB B} ) becomes small. The first learning processing unit 112 adjusts the value of the parameter of the discriminator 54 so that the sum of the calculated errors ( _LD ) becomes small. The first learning processing unit 112 adjusts the value of the parameter of the generator 50 so that the sum of the calculated errors (LG, L _GD , _{LG C} ) becomes small. The first learning processing unit 112 adjusts the value of the parameter of the encoder 56 so that the sum of the calculated errors (L _KL , _LG ) becomes small.

In the first learning processing unit 112 and the second learning processing unit 113, until the sum of the calculated errors ( _LC , L _BB , L _KL , L _D , L _GD , L _GC , _LG ) becomes equal to or less than the threshold value. , The adjustment of the value of each parameter is repeated by the above series of processes. As a result, the first learning processing unit 112 trains the discriminator 54, the generator 50, and the encoder 56, respectively, and the second learning processing unit 113 trains the estimator 52. Of these, the process of adjusting the parameters of the estimator 52 so that the sum of each error (LC, _{L BB} ) becomes small is an example of the second machine learning process. The process of adjusting the value of the parameter of the encoder 56 so that the sum of the errors (L _KL ) becomes small is an example of the fifth training step of the first machine learning. The process of adjusting the value of the parameter of the discriminator 54 so that the sum of the errors ( _LD ) becomes small is an example of the first training step. The process of adjusting the value of the parameter of the generator 50 so that the sum of the errors (L _GD ) becomes small is an example of the second training step. The process of adjusting the value of the parameter of the generator 50 so that the sum of the errors (L _GC ) becomes small is an example of the third training step. The process of adjusting the parameter values of the generator 50 and the encoder 56 so that the sum of the errors ( _LG ) becomes small is an example of the fourth training step. The criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the first learning processing unit 112 and the second learning processing unit 113 may repeat the adjustment of the value of each parameter by the above-mentioned series of processing for a designated number of times.

After this machine learning is completed, the conservation processing unit 114 determines the configuration of the constructed generator 50 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, the transmission function of each neuron). And the first learning result data 123 showing the calculation parameters (for example, the weight of the connection between each neuron, the threshold value of each neuron) is generated. Further, the preservation processing unit 114 includes the configuration of the constructed estimator 52 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, the transmission function of each neuron), and the arithmetic parameters (for example, each). The second learning result data 125 showing the weight of the connection between neurons, the threshold value of each neuron) is generated. Then, the storage processing unit 114 stores the generated first learning result data 123 and the second learning result data 125 in a predetermined storage area.

<Image generator>
Next, an example of the software configuration of the image generation device 2 according to the present embodiment will be described with reference to FIG. 9. FIG. 9 schematically illustrates an example of the software configuration of the image generation device 2 according to the present embodiment.

The control unit 21 of the image generation device 2 expands the image generation program 221 stored in the storage unit 22 into the RAM. Then, the control unit 21 interprets and executes the image generation program 221 expanded in the RAM by the CPU to control each component. As a result, as shown in FIG. 9, the image generation device 2 according to the present embodiment operates as a computer including a generation unit 211, an estimation unit 212, a determination unit 213, and an image storage unit 214 as software modules. That is, in the present embodiment, each software module of the image generation device 2 is also realized by the control unit 21 (CPU) in the same manner as the learning device 1.

The generation unit 211 includes a trained generator 50 constructed by the learning device 1 by holding the first learning result data 123. The generation unit 211 generates an generated image 73 showing the product R including the defect by giving an input value 71 indicating the type of the defect to the generator 50. In the present embodiment, the generation unit 211 sets the trained generator 50 with reference to the first learning result data 123. Next, the generation unit 211 acquires noise (latent variable) from a predetermined probability distribution. The predetermined probability distribution may be, for example, a Gaussian distribution or the like. Then, the generation unit 211 inputs the acquired noise and the input value 71 to the input layer 511 of the generator 50, and executes the arithmetic processing of the generator 50. As a result, the generation unit 211 acquires the generated image 73 from the output layer 513 of the generator 50.

The estimation unit 212 includes a trained estimator 52 constructed by the learning device 1 by holding the second learning result data 125. The estimation unit 212 estimates the type and range of defects included in the product R shown in the generated image 73 by giving the generated generated image 73 to the estimator 52. In the present embodiment, the estimation unit 212 sets the trained estimator 52 with reference to the second learning result data 125. Then, the estimation unit 212 inputs the generated image 73 generated by the generator 50 to the input layer 531 of the estimator 52, and executes the arithmetic processing of the estimator 52. As a result, the estimation unit 212 acquires an output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the generated image 73 from the output layer 533 of the estimator 52.

The determination unit 213 determines whether or not the result of estimating the type of defect included in the product R shown in the generated image 73 matches the type of defect indicated by the input value 71. When the determination unit 213 determines that the result of estimating the defect type matches the defect type indicated by the input value 71, the image storage unit 214 determines the generated generated image 73 and the result of estimating the range of the defect. It is associated and stored in a predetermined storage area. In the present embodiment, the image storage unit 214 generates information indicating the result of estimating the range of defects as the boundary information 77. Further, the image storage unit 214 generates information indicating the type of defect as the label 75 corresponding to the input value 71. Then, the image storage unit 214 generates a data set by combining the generated image 73, the label 75, and the boundary information 77, and stores the generated data set in a predetermined storage area. Of these, the label 75 may be omitted.

<Estimator generator>
Next, an example of the software configuration of the estimator generator 3 according to the present embodiment will be described with reference to FIG. FIG. 10 schematically illustrates an example of the software configuration of the estimator generator 3 according to the present embodiment.

The control unit 31 of the estimator generation device 3 expands the estimator generation program 321 stored in the storage unit 32 into the RAM. Then, the control unit 31 controls each component by interpreting and executing the instruction included in the estimator generation program 321 expanded in the RAM by the CPU. As a result, as shown in FIG. 10, the estimator generator 3 according to the present embodiment is configured as a computer including a data acquisition unit 311, a learning processing unit 312, and a storage processing unit 313 as software modules. That is, in the present embodiment, each software module of the estimator generation device 3 is also realized by the control unit 31 (CPU) in the same manner as the learning device 1.

The data acquisition unit 311 is composed of a combination of a sample image 3221 showing the product R, a label 3222 indicating the types of defects that can be included in the product R, and boundary information 3223 indicating the range of defects that can be included in the product R. Acquire a plurality of training data sets 322. The sample image 3221 is used as input data (training data) for machine learning. On the other hand, the label 3222 and the boundary information 3223 are used as correct answer data (teacher data) for machine learning. That is, the label 3222 and the boundary information 3223 indicate the result (that is, the correct answer) of determining the quality of the product R shown in the corresponding sample image 3221. When the product R shown in the sample image 3221 does not contain a defect, the values of the label 3222 and the boundary information 3223 may be appropriately set to indicate that the product R does not contain a defect.

Here, the data set generated by the image generation device 2 may be used as the training data set 322. That is, the sample image 3221, the label 3222, and the boundary information 3223 of at least a part of the training data set 322 may be the generated image 73, the label 75, and the boundary information 77, respectively. Further, the data acquisition unit 311 may transmit the learning data set 322 to the learning device 1 and use the learning data set 322 as the learning data set 60 to cause the learning device 1 to execute the machine learning process. That is, the data acquisition unit 311 uses the sample image 3221 as the learning image 61, the label 3222 as the label 63, and the boundary information 3223 as the boundary information 65 to generate an image corresponding to the sample image 3221. The learning device 1 may be constructed with a generator 50 for estimating the type and range of defects included in the product R with respect to the generated image. Then, the data acquisition unit 311 may make the image generation device 2 use the constructed generator 50 and the estimator 52 to generate a plurality of data sets corresponding to the learning data set 322 in the image generation device 2. .. The data acquisition unit 311 can increase the number of learning data sets 322 used for machine learning by receiving the generated plurality of data sets as the training data set 322.

The learning processing unit 312 holds an estimator 80 that is a target of machine learning. The learning processing unit 312 constructs a trained estimator 80 that has acquired the ability to judge the quality of the product R reflected in a given image by performing machine learning using each acquired learning data set 322. .. In other words, the learning processing unit 312 constructs an estimator 80 for each training data set 322 that, when the sample image 3221 is input, outputs an output value that matches the label 3222 and the boundary information 3223. The storage processing unit 313 stores information about the constructed learned estimator 80 in a predetermined storage area.

In the relationship between the learning device 2 and the estimator generation device 3, the learning processing unit 312 may be referred to as a third learning processing unit. The data acquisition unit 111 and the storage processing unit 114 may be referred to as a first data acquisition unit and a first storage processing unit, respectively. The training data set 60 may be referred to as a first training data set. Accordingly, the data acquisition unit 311 and the storage processing unit 313 may be referred to as a second data acquisition unit and a second storage processing unit, respectively. The training data set 322 may be referred to as a second training data set.

(Estimator)
Next, an example of the configuration of the estimator 80 according to the present embodiment will be described. As shown in FIG. 10, the estimator 80 according to the present embodiment is configured by a multi-layered neural network used for so-called deep learning, like the generator 50 and the like. Specifically, the estimator 80 includes an input layer 801 and an intermediate layer (hidden layer) 802, and an output layer 803. However, the configuration of the estimator 80 does not have to be limited to such an example, and may be appropriately set according to the embodiment. For example, the estimator 80 may include two or more intermediate layers. The configuration of the estimator 80 may be different from that of the generator 50 and the like.

The number of neurons (nodes) included in each layer 801 to 803 may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the example of FIG. 10, each neuron is connected to all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment. A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 801 to 803 are examples of the parameters of the estimator 80 used for arithmetic processing.

The learning processing unit 312 inputs the sample image 3221 to the input layer 801 of the estimator 80 for each learning data set 322, and executes the arithmetic processing of the estimator 80. As a result of this arithmetic processing, the learning processing unit 312 determines whether the product R shown in the sample image 3221 is good or bad, in other words, an output value corresponding to the result of estimating the type and range of defects that can be included in the product R. Obtained from the output layer 803.

Subsequently, the learning processing unit 312 calculates an error between the acquired output value and each value of the label 3222 and the boundary information 3223. Then, the learning processing unit 312 adjusts the value of the parameter of the estimator 80 so that the sum of the calculated errors becomes small for each learning data set 322. For example, the learning processing unit 312 performs the parameter values of the estimator 80 by the above series of processing until the sum of the errors between the output value obtained from the output layer 803 and the values of the label 3222 and the boundary information 3223 becomes equal to or less than the threshold value. Repeat the adjustment of. As a result, when the sample image 3221 is input to the input layer 801 for each training data set 322, the learning processing unit 312 outputs an output value that matches the label 3222 and the boundary information 3223 associated with the input sample image 3221. An estimator 80 trained to output from 803 can be constructed. The criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the learning processing unit 312 may repeat the adjustment of the parameter value of the estimator 80 by the above-mentioned series of processing a specified number of times.

After the machine learning process is completed, the conservation processing unit 313 configures the constructed trained estimator 80 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, and each neuron. 3rd learning result data 325 showing the transmission function) and the arithmetic parameters (for example, the weight of the connection between each neuron, the threshold value of each neuron). Then, the storage processing unit 313 stores the generated third learning result data 325 in a predetermined storage area.

<Inspection equipment>
Next, an example of the software configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 11 schematically illustrates an example of the software configuration of the inspection device 4 according to the present embodiment.

The control unit 41 of the inspection device 4 expands the inspection program 421 stored in the storage unit 42 into the RAM. Then, the control unit 41 controls each component by interpreting and executing the instruction included in the inspection program 421 expanded in the RAM by the CPU. As a result, as shown in FIG. 11, the inspection device 4 according to the present embodiment is configured as a computer including the target data acquisition unit 411, the pass / fail determination unit 412, and the output unit 413 as software modules. In the present embodiment, each software module of the inspection device 4 is also realized by the control unit 41 (CPU) in the same manner as the learning device 1.

The target data acquisition unit 411 acquires the target image 422 of the product R that is the target of the visual inspection. In the present embodiment, the target data acquisition unit 411 acquires the target image 422 by photographing the product R with the camera CA. The pass / fail determination unit 412 includes the estimator 80 constructed by the estimator generator 3 but has been learned by holding the third learning result data 325. The quality determination unit 412 determines the quality of the product R shown in the target image 422 by using the trained estimator 80.

Specifically, the pass / fail determination unit 412 sets the trained estimator 80 with reference to the third learning result data 325. Next, the pass / fail determination unit 412 inputs the acquired target image 422 to the input layer 801 of the estimator 80, and executes the arithmetic processing of the estimator 80. As a result, the quality determination unit 412 estimates the output value corresponding to the result of determining the quality of the product R shown in the target image 422 (specifically, the result of estimating the type and range of defects that can be included in the product R). Obtained from the output layer 803 of the device 80. In the present embodiment, obtaining this output value corresponds to determining the quality of the product R. The output unit 413 outputs information regarding the result of determining the quality of the product R.

<Others>
Each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 will be described in detail in an operation example described later. In this embodiment, an example is described in which each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 is realized by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, with respect to the software configurations of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4, software modules may be omitted, replaced, or added as appropriate according to the embodiment.

§3 Operation example [Learning device]
Next, an operation example of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 12 shows an example of the processing procedure of the learning device 1 according to the present embodiment. The processing procedure described below is an example of the "learning method" of the present invention. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as a data acquisition unit 111, and displays a learning image 61 showing the product R to be visually inspected, a label 63 indicating the type of defect included in the product R, and a range of defects. A plurality of training data sets 60 configured by the combination of the boundary information 65 shown are acquired.

The method for acquiring the training data set 60 does not have to be particularly limited, and may be appropriately determined according to the embodiment. For example, a camera and a defective product R are prepared, and the product R is photographed by the camera so that the defective portion is captured. As a result, the learning image 61 can be generated. Next, the label 63 indicating the type of the defect included in the product R shown in the learning image 61 and the boundary information 65 indicating the range of the defect are appropriately generated. For example, the label 63 may be generated by selecting a defect to be reflected in the target learning image 61 from the list of defect types. Further, the boundary information 65 may be generated by designating a bounding box so as to include a range of defects on the learning image 61. Then, the generated label 63 and the boundary information 65 are associated with the corresponding learning image 61. This makes it possible to generate each training data set 60.

The generation of the training data set 60 may be automatically performed by the operation of the computer, or may be manually performed by the operation of the operator. Further, the information processing that generates the learning data set 60 may be executed in the learning device 1 or may be executed in a computer other than the learning device 1.

When the learning device 1 generates the learning data set 60, the control unit 11 acquires a plurality of learning data sets 60 by automatically or manually executing the above information processing by the operation of the operator. On the other hand, when the learning data set 60 is generated by another computer, the control unit 11 acquires a plurality of learning data sets 60 generated by the other computer via, for example, a network, a storage medium 91, or the like. The learning device 1 may generate a part of the training data set 60, and another computer may generate the remaining training data set 60.

The number of learning data sets 60 to be acquired may not be particularly limited and may be appropriately determined according to the embodiment. When the plurality of training data sets 60 are acquired, the control unit 11 proceeds to the next step S102.

(Steps S102 and S103)
In step S102, the control unit 11 operates as the first learning processing unit 112, and by executing the first machine learning, the input label with respect to the input of the label 63 included in each learning data set 60. Build a generator 50 trained to generate an image corresponding to the training image 61 associated with 63. In step S103, the control unit 11 operates as the second learning processing unit 113, and by executing the second machine learning, the control unit 11 is input to the input of the learning image 61 included in each learning data set 60. Constructs an estimator 52 trained to estimate the type and range of defects contained in the product R appearing in the input training image 61 so as to match the label 63 and boundary information 65 associated with the training image 61. do.

The order in which steps S102 and S103 are processed may not be particularly limited and may be appropriately determined according to the embodiment. The processes of steps S102 and S103 may be executed simultaneously or separately. In the present embodiment, the generator 50 and the estimator 52 together with the encoder 56 and the discriminator 54 form a learning network 500. Therefore, in steps S102 and S103, the control unit 11 trains the generator 50 and the estimator 52 by performing machine learning of the learning network 500.

Specifically, first, the control unit 11 prepares the learning network 500 to be processed. The configuration of the learning network 500 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, in the case of re-learning, the control unit 11 may prepare the learning network 500 based on the learning result data obtained by performing the past machine learning.

Next, the control unit 11 uses the learning image 61 of each learning data set 60 as input data, and uses the corresponding label 63 and the boundary information 65 as teacher data to execute the learning process of the estimator 52. The control unit 11 uses the training image 61 and the label 63 of each training data set 60 as input data, and uses the values derived from the prior distribution as teacher data to execute the learning process of the encoder 56. The control unit 11 inputs the output value and the label 63 obtained from the encoder 56 to the generator 50 for each learning data set 60, and executes the arithmetic processing of the generator 50. As a result, the control unit 11 acquires each image 69 from the generator 50. The control unit 11 uses each image 69 and each learning image 61 as input data, and uses the origin of each image (61, 69) as teacher data to execute the learning process of the discriminator 54. For each learning data set 60, the control unit 11 uses the output value and the label 63 obtained from the encoder 56 as input data, and makes the discriminator 54 make a mistake in discrimination as teacher data. Execute the learning process. The control unit 11 uses the output value and the label 63 obtained from the encoder 56 as input data for each training data set 60, and uses the fact that the estimator 52 correctly estimates the type of defect as teacher data. The learning process of the generator 50 is executed. The control unit 11 uses the training image 61 and the label 63 as input data for each training data set 60, and uses the training image 61 as teacher data to execute the learning process of the encoder 56 and the generator 50. A stochastic gradient descent method or the like may be used for these learning processes.

For example, the control unit 11 inputs the learning image 61 included in each learning data set 60 to the input layer 531 of the estimator 52, and determines the firing of each neuron included in each layer 531 to 533 in order from the input side. As a result, the control unit 11 acquires an output value corresponding to the result of estimating the type and range of the defect included in the product R reflected in each learning image 61 from the output layer 533. The control unit 11 calculates an error ( _LC ) between the output value corresponding to the result of estimating the defect type and the value of the label 63. Further, the control unit 11 calculates an error ( _LBB ) between the output value corresponding to the result of estimating the defect range and the value of the boundary information 65.

Next, the control unit 11 inputs the learning image 61 and the label 63 included in each learning data set 60 to the input layer 571 of the encoder 56, and determines the firing of each neuron included in each layer 571 to 573 in order from the input side. conduct. As a result, the control unit 11 acquires an output value (latent variable) corresponding to the result of deriving some feature quantity from the output layer 573 of the encoder 56. The control unit 11 calculates an error (L _KL ) between this output value and the value derived from the prior distribution. The prior distribution may be, for example, a Gaussian distribution.

Next, the control unit 11 inputs the output value obtained from the encoder 56 and the corresponding label 63 to the input layer 511 of the generator 50, and determines the firing of each neuron included in each layer 511 to 513 in order from the input side. conduct. As a result, the control unit 11 acquires each image 69 from the output layer 513 as a result of generating an image corresponding to each learning image 61 from the label 63.

Subsequently, the control unit 11 inputs each image 69 generated by the generator 50 to the input layer 551 of the discriminator 54, and determines the firing of each neuron included in each layer 551 to 553 in order from the input side. As a result, the control unit 11 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 (that is, the learning image 61 itself) from the output layer 553. In this scene, since each image 69 generated by the generator 50 is an input image, the correct answer is that the discriminator 54 discriminates as "false". The control unit 11 calculates an error ( _LD ) between the output value obtained from the output layer 553 and the correct answer for each image 69 generated by the generator 50.

Further, the control unit 11 inputs the learning image 61 included in each learning data set 60 to the input layer 551 of the discriminator 54, and determines the firing of each neuron included in each layer 551 to 553 in order from the input side. As a result, the control unit 11 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 from the output layer 553. In this scene, since the learning image 61 itself is an input image, the correct answer is that the discriminator 54 determines that it is “true”. The control unit 11 calculates an error ( _LD ) between the output value obtained from the output layer 553 and the correct answer for each learning data set 60.

Next, the control unit 11 generates an image that causes the discriminator 54 to make a mistake in discriminating the output value obtained by inputting each image 69 generated by the generator 50 into the discriminator 54. Calculate the error (L _GD ) for training the vessel 50. Specifically, when each image 69 generated by the generator 50 is input to the discriminator 54, the correct answer is that the output value obtained from the discriminator 54 corresponds to "true". The control unit 11 calculates an error (L _GD ) between the output value obtained from the output layer 553 by a series of processes and this correct answer (that is, “true”) for each learning data set 60.

Next, the control unit 11 inputs each image 69 generated by the generator 50 to the input layer 531 of the estimator 52, and determines the firing of each neuron included in each layer 531 to 533 in order from the input side. As a result, the control unit 11 acquires an output value corresponding to the result of estimating the type of the defect included in the product R shown in each image 69 from the output layer 533 of the estimator 52. The control unit 11 calculates an error ( _LGC ) between the output value corresponding to the result of estimating the defect type by a series of processes and the value of the label 63 corresponding to each learning data set 60.

Next, the control unit 11 calculates an error for training the generator 50 to function as a decoder that restores the training image 61 from the output of the encoder 56 for each training data set 60. Specifically, the control unit 11 calculates an error ( _LG ) between the image 69 generated by the generator 50 and the corresponding training image 61 for each training data set 60.

Then, the control unit 11 uses each error ( _{LC, LB B, L KL , LD ,} L _GD , L _GC , _LG ) calculated by the back propagation method for each neuron. The weight of the connection between them and the error of each threshold value of each neuron are calculated. That is, the control unit 11 calculates the error of the connection weight between each neuron of the estimator 52 and the error of each threshold value of each neuron by using each calculated error (LC, _{LB B} ). The control unit 11 calculates the error of the connection weight between each neuron of the discriminator 54 and the error of each threshold value of each neuron by using the calculated error ( _LD ). The control unit 11 calculates the error of the connection weight between each neuron of the generator 50 and the error of each threshold value of each neuron by using each calculated error (LG, L _GD , _{LG C} ). The control unit 11 calculates the error of the connection weight between each neuron of the encoder 56 and the error of each threshold value of each neuron by using each calculated error (L _KL , _LG ). Based on each calculated error, the control unit 11 updates the weight of the connection between each neuron and the value of each threshold value of each neuron in each of the estimator 52, the discriminator 54, the generator 50, and the encoder 56.

The control unit 11 performs the above-mentioned series of processing until the sum of the calculated errors ( _LC , _LB , L _KL , L _D , L _GD , L _GC , _LG ) becomes equal to or less than the threshold value, and the control unit 11 determines each parameter. Repeat the adjustment of the value. Each threshold value may be appropriately set according to the embodiment. As a result, the control unit 11 includes each training data set 60 in the product R reflected in the input training image 61 so as to match the corresponding label 63 and the boundary information 65 with respect to the input of the training image 61. An estimator 52 trained to estimate the type and extent of defects can be constructed. The control unit 11 can construct a discriminator 54 trained so that it can appropriately discriminate whether the input input image is derived from the generator 50 or the learning data set 60. The control unit 11 is trained so that the discriminator 54 makes a discriminant error, the estimator 52 estimates the type of defect so as to match the corresponding label 63, and the image 69 that can correspond to the learning image 61 can be generated. The generator 50 can be constructed. The control unit 11 can cause the generator 50 to generate an image corresponding to the training image 61, which is an output value according to the prior distribution for the input of the training image 61 and the label 63 for each training data set 60. An encoder 56 trained to output a high output value can be constructed. When the machine learning process is completed, the control unit 11 proceeds to the next step S104. The criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the control unit 11 may repeat the adjustment of the value of each parameter by the above-mentioned series of processes for a specified number of times.

(Step S104)
In step S104, the control unit 11 operates as a storage processing unit 114, and stores information about the constructed trained generator 50 and information about the constructed trained estimator 52 in a predetermined storage area.

In the present embodiment, the control unit 11 generates information indicating the configuration and parameters of the trained generator 50 constructed by step S102 as the first learning result data 123. Further, the control unit 11 generates information indicating the configuration and parameters of the trained estimator 52 constructed by step S103 as the second learning result data 125. Then, the control unit 11 stores the generated first learning result data 123 and the second learning result data 125 in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 11, a storage unit 12, an external storage device, a storage medium, or a combination thereof.

The storage medium may be, for example, a CD, a DVD, or the like, and the control unit 11 may store the first learning result data 123 and the second learning result data 125 in the storage medium via the drive 16. The external storage device may be, for example, a data server such as NAS (Network Attached Storage). In this case, the control unit 11 may store the first learning result data 123 and the second learning result data 125 in the data server via the network. Further, the external storage device may be, for example, an external storage device connected to the learning device 1.

The handling of the encoder 56 and the discriminator 54 does not have to be particularly limited, and may be appropriately determined according to the embodiment. For example, the control unit 11 also generates information indicating the respective configurations and parameters of the encoder 56 and the discriminator 54 as learning result data, as in the generator 50 and the estimator 52, and determines the generated learning result data. It may be stored in the storage area of. Alternatively, the control unit 11 may omit the process of generating and storing the learning result data.

When the first learning result data 123 and the second learning result data 125 are saved, the control unit 11 ends the process according to this operation example. After constructing the trained generator 50 and the estimator 52, the control unit 11 may transfer the generated first learning result data 123 and the second learning result data 125 to the image generator 2 at an arbitrary timing. .. The image generation device 2 may acquire the first learning result data 123 and the second learning result data 125 by accepting the transfer from the learning device 1. The image generation device 2 may acquire the first learning result data 123 and the second learning result data 125 by accessing the learning device 1 or the data server. Alternatively, the first learning result data 123 and the second learning result data 125 may be incorporated in the image generation device 2 in advance.

Further, the control unit 11 may periodically update at least one of the first learning result data 123 and the second learning result data 125 by periodically repeating the processes of steps S101 to S104. When this is repeated, the training data set 60 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 11 transfers at least one of the updated first learning result data 123 and the second learning result data 125 to the image generation device 2 every time machine learning is executed, so that the image generation device 2 holds the image generation device 2. At least one of the first learning result data 123 and the second learning result data 125 may be updated periodically.

[Image generator]
Next, an operation example of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 13 shows an example of the processing procedure of the image generation device 2 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S201)
In step S201, the control unit 21 operates as the generation unit 211, and uses the trained generator 50 constructed by the learning device 1 to generate a generated image 73 in which the product R including the designated defect can be captured. do.

In the present embodiment, the control unit 21 sets the trained generator 50 with reference to the first learning result data 123. Next, the control unit 21 acquires noise (latent variable) from a predetermined probability distribution. The predetermined probability distribution may be, for example, a Gaussian distribution or the like. Further, the control unit 21 appropriately acquires an input value 71 indicating the type of defect. For example, information (not shown) showing a list of defect types may be stored in the image generation program 221, a storage unit 22, an external storage device, a storage medium, or the like. In this case, the control unit 21 acquires the information indicating the list of the types of the defect, refers to the acquired information, and selects the defect to be designated as the target for generating the image from the list, thereby inputting the input value 71. You may get it. Further, for example, the control unit 21 may randomly select an input value 71 from a predetermined numerical range. Then, the control unit 21 inputs the acquired noise and the input value 71 to the input layer 511 of the generator 50, and determines the firing of each neuron included in each layer 511 to 513 in order from the input side. As a result, the control unit 21 acquires the generated image 73 from the output layer 513 of the generator 50. When the generation of the generated image 73 is completed, the control unit 21 proceeds to the next step S202.

(Step S202)
In step S202, the control unit 21 operates as the estimation unit 212, and by using the learned estimator 52 constructed by the learning device 1, the type and range of defects included in the product R that can be captured in the generated image 73. To estimate.

In the present embodiment, the control unit 21 sets the trained estimator 52 with reference to the second learning result data 125. Then, the control unit 21 inputs the generated image 73 generated by the generator 50 to the input layer 531 of the estimator 52, and determines the firing of each neuron included in each layer 531 to 533 in order from the input side. As a result, the control unit 21 acquires an output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the generated image 73 from the output layer 533 of the estimator 52. When the estimation of the defect type and range is completed, the control unit 21 proceeds to the next step S203.

(Steps S203 and S204)
In step S203, the control unit 21 operates as the determination unit 213, and determines whether or not the result of estimating the type of defect included in the product R shown in the generated image 73 matches the type of defect indicated by the input value 71. judge. In step S204, the control unit 21 processes the conditional branch according to the determination result of step S203.

The fact that the result of estimating the defect type by the estimator 52 matches the input value 71 means that the generator 50 was able to generate an image (generated image 73) of the product R including the specified defect. Show that. Therefore, when it is determined that the result of estimating the defect type matches the defect type indicated by the input value 71, the control unit 21 proceeds to the next step S205 for storing the generated image 73.

On the other hand, the fact that the result of estimating the defect type by the estimator 52 does not match the input value 71 means that the generator 50 could not generate an image of the product R including the specified defect. show. Therefore, when it is determined that the result of estimating the defect type does not match the defect type indicated by the input value 71, the control unit 21 omits the processing of the next step S205 and ends the processing related to this operation example. do.

In this case, the control unit 21 may discard the generated image 73 generated in step S201. Alternatively, the control unit 21 may accept the designation of at least one of the defect type and range for the generated generated image 73. Then, the control unit 21 may generate a data set by associating the information indicating at least one of the designated defect types and ranges with the generated image 73, and store the generated data set in a predetermined storage area. .. The predetermined storage area may be, for example, a RAM in the control unit 21, a storage unit 22, an external storage device, a storage medium, or a combination thereof.

(Step S205)
In step S205, the control unit 21 operates as an image storage unit 214, and stores the generated generated image 73 and the result of estimating the range of defects in a predetermined storage area in association with each other. In the present embodiment, the control unit 21 generates information indicating the result of estimating the defect range as the boundary information 77. Further, the control unit 21 generates information indicating the type of defect as a label 75 corresponding to the input value 71. Then, the control unit 21 generates a data set by combining the generated image 73, the label 75, and the boundary information 77, and stores the generated data set in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 21, a storage unit 22, an external storage device, a storage medium, or a combination thereof. In this series of processing, the label 75 may be omitted. That is, the control unit 21 may omit the generation of the label 75, generate a data set by combining the generated image 73 and the boundary information 77, and store the generated data set in a predetermined storage area.

As a result, the control unit 21 ends the process related to this operation example. The control unit 21 may generate a plurality of data sets by repeatedly executing the steps S201 to S205. The number of data sets to be generated is not particularly limited and may be appropriately determined according to the embodiment. Further, the control unit 21 may transfer the generated data set to the estimator generation device 3 in order to use the generated data set as the learning data set 322.

[Estimator generator]
Next, an operation example of the estimator generator 3 according to the present embodiment will be described with reference to FIG. FIG. 14 is a flowchart illustrating an example of the processing procedure of the estimator generator 3 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S301)
In step S301, the control unit 31 operates as a data acquisition unit 311 and has a sample image 3221 showing the product R, a label 3222 indicating the types of defects that can be included in the product R, and a range of defects that can be included in the product R. A plurality of training data sets 322 configured by the combination of the boundary information 3223 indicating the above are acquired.

The method for acquiring the training data set 322 does not have to be particularly limited, and may be appropriately determined according to the embodiment. For example, a camera and a product R are prepared, and a defective or non-defective product R is photographed by the camera. This makes it possible to generate a sample image 3221. Next, a label 3222 indicating the type of defect that can be included in the product R shown in the sample image 3221 and boundary information 3223 indicating the range of the defect are appropriately generated. The method for generating the label 3222 and the boundary information 3223 may be the same as the method for generating the label 63 and the boundary information 65. The boundary information 3223 may be appropriately generated by the bounding box to indicate the range of defects. Further, when the product R shown in the sample image 3221 does not contain a defect, the values of the label 3222 and the boundary information 3223 may be appropriately set so as to indicate that the product R does not contain a defect. Then, the generated label 3222 and the boundary information 3223 are associated with the corresponding sample image 3221. As a result, each training data set 322 can be generated.

The generation of the learning data set 322 may be automatically performed by the operation of the computer, or may be manually performed by the operation of the operator. Further, the information processing for generating the learning data set 322 may be executed by the estimator generator 3 or may be performed by a computer other than the estimator generator 3.

When the estimator generator 3 generates the learning data set 322, the control unit 31 acquires a plurality of learning data sets 322 by automatically or manually executing the above information processing by the operation of the operator. On the other hand, when the learning data set 322 is generated by another computer, the control unit 31 acquires a plurality of learning data sets 322 generated by the other computer via, for example, a network, a storage medium 93, or the like. The estimator generator 3 may generate a part of the training data set 322, and another computer may generate the remaining training data set 322.

Here, at least a part of the learning data set 322 acquired may be a data set generated by the image generation device 2. That is, the sample image 3221, the label 3222, and the boundary information 3223 of at least a part of the training data set 322 may be the generated image 73, the label 75, and the boundary information 77, respectively. The control unit 31 may acquire the data set generated by the image generation device 2 as the learning data set 322 via the network, the storage medium 93, or the like.

The number of learning data sets 322 to be acquired may not be particularly limited and may be appropriately determined according to the embodiment. When the plurality of training data sets 322 are acquired, the control unit 31 proceeds to the next step S302.

(Step S302)
In step S302, the control unit 31 operates as the learning processing unit 312, and performs machine learning of the estimator 80 by using the plurality of learning data sets 322. In this machine learning, when the sample image 3221 is input to the input layer 801 for each learning data set 322, the control unit 31 outputs an output value matching the corresponding label 3222 and the boundary information 3223 from the output layer 803. Train the estimator 80. As a result, the control unit 31 constructs the trained estimator 80 that has acquired the ability to determine the quality of the product R.

This machine learning process may be basically executed in the same manner as the machine learning process by the learning device 1. That is, the control unit 31 prepares the estimator 80 to be processed. The configuration of the estimator 80 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, in the case of re-learning, the control unit 31 may prepare the estimator 80 based on the learning result data obtained by performing the past machine learning.

Next, the control unit 31 uses the sample image 3221 included in each learning data set 322 acquired in step S301 as input data, and uses the corresponding label 3222 and the boundary information 3223 as teacher data to use the estimator 80. Executes the learning process of. A stochastic gradient descent method or the like may be used for this learning process.

For example, the control unit 31 inputs the sample image 3221 to the input layer 801 for each learning data set 322, and determines the firing of each neuron included in each layer 801 to 803 in order from the input side. As a result, the control unit 31 acquires an output value corresponding to the result of estimating the type and range of defects that can be included in the product R shown in the sample image 3221 from the output layer 803. Next, the control unit 31 calculates an error between the output value corresponding to each result and each value of the label 3222 and the boundary information 3223. Subsequently, the control unit 31 calculates the error of the connection weight between each neuron and the error of each threshold value of each neuron in the estimator 80 by using the error of the calculated output value by the error back propagation method. Then, the control unit 31 updates the weight of the connection between each neuron and the value of each threshold value of each neuron in the estimator 80 based on each calculated error.

By repeating the above series of processes, the control unit 31 outputs the output value that matches the corresponding label 3222 and the boundary information 3223 when the sample image 3221 is input for each training data set 322. Adjust the value of the parameter of. For example, the control unit 31 estimates for each learning data set 322 by the above series of processes until the sum of the errors between the output value obtained from the output layer 803 and the values of the label 3222 and the boundary information 3223 becomes equal to or less than the threshold value. Repeat the adjustment of the parameter value of the vessel 80. The threshold value may be appropriately set according to the embodiment. As a result, the control unit 31 is trained to output the output value matching the corresponding label 3222 and the boundary information 3223 from the output layer 803 when the sample image 3221 is input to the input layer 801 for each training data set 322. The estimator 80 can be constructed. When the machine learning process of the estimator 80 is completed, the control unit 31 proceeds to the next step S303. The criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the control unit 31 may repeat the adjustment of the parameter value of the estimator 80 by the above-mentioned series of processes for a specified number of times.

(Step S303)
In step S303, the control unit 31 operates as a storage processing unit 313 and stores information about the constructed learned estimator 80 in a predetermined storage area. In the present embodiment, the control unit 31 generates information indicating the configuration and parameters of the trained estimator 80 constructed by step S302 as the third learning result data 325. Then, the control unit 31 stores the generated third learning result data 325 in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 31, a storage unit 32, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof. As a result, the control unit 31 ends the process related to this operation example.

After constructing the trained estimator 80, the control unit 31 may transfer the generated third learning result data 325 to the inspection device 4 at an arbitrary timing. The inspection device 4 may acquire the third learning result data 325 by accepting the transfer from the estimator generator 3, or may acquire the third learning result data 325 by accessing the estimator generator 3 or the data server. You may get it. The third learning result data 325 may be incorporated in the inspection device 4 in advance.

Further, the control unit 31 may periodically update the third learning result data 325 by periodically repeating the processes of steps S301 to 303. When this is repeated, the training data set 322 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 31 periodically updates the third learning result data 325 held by the inspection device 4 by transferring the updated third learning result data 325 to the inspection device 4 every time the machine learning is executed. May be good.

Further, the control unit 31 may evaluate the determination performance of the constructed estimator 80 by using the evaluation data set. The evaluation data set can be configured in the same manner as each training data set 322 described above. That is, the evaluation data set may be composed of a combination of a sample image showing the product R, a label indicating the types of defects that can be contained in the product R, and boundary information indicating the range of defects that can be contained in the product R. .. Similar to step S402 described later, the control unit 31 uses the estimator 80 to determine the quality of the product R shown in the sample image of the evaluation data set. The control unit 31 can evaluate the determination performance of the estimator 80 by collating the determination result with the correct answer indicated by the label and the boundary information.

When the determination performance of the estimator 80 is equal to or less than a predetermined standard (for example, the correct answer rate is equal to or less than the threshold value), the control unit 31 may use one or a plurality of learning data sets selected from the plurality of learning data sets 322. 322 may be transmitted to the learning device 1. Next, the control unit 31 may use the transmitted learning data set 322 as the learning data set 60 to cause the learning device 1 to perform machine learning processing of the generator 50 and the estimator 52. As a result, the learning device 1 is provided with a generator 50 for generating an image corresponding to the sample image 3221 and an estimator 52 for estimating the type and range of defects included in the product R with respect to the generated image. Can be built. The control unit 31 may transfer the trained generator 50 and the estimator 52 to the image generator 2 by the learning device 1. Then, the control unit 31 uses the trained generator 50 and the estimator 52 to generate one or a plurality of data sets composed of a combination of the generated image 73, the label 75, and the boundary information 77, respectively. May be executed by the image generation device 2.

In response to this, the control unit 31 may acquire one or a plurality of data sets generated by the image generation device 2 as a learning data set 322. Then, the control unit 31 may add the acquired one or more learning data sets 322 to the original learning data group. As a result, the control unit 31 can increase the number of learning data sets 322 used for machine learning. The control unit 31 may use this new learning data group to perform machine learning of the estimator 80 again. By this series of re-learning processes, the determination performance of the constructed trained estimator 80 can be improved.

[Inspection equipment]
Next, an operation example of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart illustrating an example of the processing procedure of the inspection device 4 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S401)
In step S401, the control unit 41 operates as the target data acquisition unit 411 to acquire the target image 422 of the product R to be visually inspected. In the present embodiment, the inspection device 4 is connected to the camera CA via the external interface 47. Therefore, the control unit 41 acquires the target image 422 from the camera CA. The target image 422 may be moving image data or still image data. When the target image 422 is acquired, the control unit 41 proceeds to the next step S402.

However, the route for acquiring the target image 422 does not have to be limited to such an example, and may be appropriately selected according to the embodiment. For example, another information processing device different from the inspection device 4 may be connected to the camera CA. In this case, the control unit 41 may acquire the target image 422 via another information processing device.

(Step S402)
In step S402, the control unit 41 operates as a quality determination unit 412, and uses the trained estimator 80 to determine the quality of the product R shown in the target image 422.

Specifically, the control unit 41 sets the trained estimator 80 with reference to the third learning result data 325. Next, the pass / fail determination unit 412 inputs the acquired target image 422 to the input layer 801 of the estimator 80, and determines the firing of each neuron included in each layer 801 to 803 in order from the input side. As a result, the control unit 41 acquires an output value corresponding to the result of estimating the type and range of defects that can be included in the product R shown in the target image 422 from the output layer 803 of the estimator 80.

As a result, the control unit 41 determines the quality of the product R reflected in the target image 422 based on the output value acquired from the estimator 80. When the output value obtained from the estimator 80 indicates that the product R contains a defect, the control unit 41 can determine from this output value that the product R is not a good product (that is, contains a defect). .. In this case, the control unit 41 can specify the type and range of the defect from the output value acquired from the estimator 80. The extent of the defect is indicated by the bounding box. On the other hand, when the output value obtained from the estimator 80 indicates that the product R does not contain a defect, the control unit 41 determines from this output value that the product R is a good product (that is, does not contain a defect). can do.

The quality of the determination may be appropriately determined according to the output format of the estimator 80. For example, the estimator 80 may be configured to output the probability of occurrence of the defect according to each type of defect. In this case, the control unit 41 can determine the quality of the product R by comparing the output value of the estimator 80 with the threshold value. That is, when there is an output value exceeding (or higher than) the threshold value, the control unit 41 determines that the product R is not a non-defective product and that the product R contains a defect of the type corresponding to the output value. be able to. On the other hand, if this is not the case, the control unit 41 can determine that the product R is a non-defective product.

Further, for example, the estimator 80 may be configured to output a class corresponding to a defect type (including not including a defect). In this case, the inspection device 4 may hold reference information (not shown) such as a table format in which the output value obtained from the estimator 80 and the type of defect are associated with each other in the storage unit 42. In response to this, the control unit 41 can determine the quality of the product R shown in the target image 422 according to the output value obtained from the estimator 80 by referring to the reference information.

As described above, the control unit 41 can determine the quality of the product R shown in the target image 422 by using the estimator 80. When the quality determination of the product R is completed, the control unit 41 proceeds to the next step S403.

(Step S403)
In step S403, the control unit 41 operates as the output unit 413 and outputs the result of determining the quality of the product R in step S402.

The output format as a result of determining the quality of the product R does not have to be particularly limited, and may be appropriately selected according to the embodiment. For example, the control unit 41 may output the result of determining the quality of the product R to the output device 45 as it is. Further, when it is determined in step S402 that the product R has a defect, the control unit 41 may issue a warning for notifying that the defect has been found as the output process of this step S403. At this time, the control unit 41 can indicate the range in which the found defect exists by the bounding box.

Further, the control unit 41 may execute a predetermined control process according to the result of determining the quality of the product R as the output process of this step S403. As a specific example, the inspection device 4 may be connected to a production line for transporting products. In this case, when the control unit 41 determines that the product R is defective, the control unit 41 sends a command to the production line to convey the defective product R by a route different from that of the non-defective product, in this step 403. It may be performed as an output process of.

When the output processing as a result of determining the quality of the product R is completed, the control unit 41 ends the processing related to this operation example. The control unit 41 may execute a series of processes of steps S401 to S403 each time the product R conveyed on the production line enters the shooting range of the camera CA. As a result, the inspection device 4 can inspect the appearance of the product R transported on the production line.

[feature]
As described above, the learning device 1 according to the present embodiment constructs the generator 50 trained to generate the image corresponding to the learning image 61 by the processing of steps S102 and S103, and is reflected in the learning image 61. An estimator 52 trained to estimate the type and extent of defects can be constructed. Of these, by using the generator 50, it is possible to automatically generate a sample image of the product including the specified defect. In addition, by using the estimator 52, the range of defects contained in the product R reflected in the generated sample image is estimated, and information indicating the estimated range of defects is automatically added to the sample image as boundary information. can do. Therefore, according to the present embodiment, when preparing the training data, it is possible to omit the trouble of manually specifying the range of defects in the sample image. Therefore, it is possible to reduce the cost of preparing the learning data used for machine learning to acquire the ability to detect the range of defects.

Further, in the image generator 2 according to the present embodiment, the range of defects is manually specified by using the generator 50 and the estimator 52 constructed by the learning device 1 by a series of processes of steps S201 to S205. It is possible to generate a data set that can be used for machine learning to acquire the ability to detect the range of defects without the trouble of doing so. In addition, in the present embodiment, the data set generated by the image generation device 2 can be used as the learning data set 322. As a result, in the estimator generator 3 according to the present embodiment, the cost of collecting the learning data set 322 can be reduced in the step S301. Further, in the present embodiment, the number of learning data sets 322 used for machine learning of the estimator 80 can be increased by causing the image generation device 2 to generate data sets. As a result, in the inspection device 4 according to the present embodiment, the accuracy of determining the quality of the product R can be improved.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, a so-called multi-layered fully connected neural network is used for each of the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80. However, the structure and type of the neural network constituting each of the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80 may not be limited to such an example, and may vary depending on the embodiment. It may be appropriately selected. For example, a convolutional neural network may be used for each of the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80.

Further, in the above embodiment, a neural network is used as a learning model constituting the generator 50, the estimator 52, and the estimator 80. However, the type of the learning model constituting each of the generator 50, the estimator 52, and the estimator 80 may not be particularly limited as long as machine learning of the image can be performed, depending on the embodiment. It may be appropriately selected.

<4.2>
In the above embodiment, the first learning result data 123 includes information indicating the configuration of the generator 50. The second learning result data 125 includes information indicating the configuration of the estimator 52. The third learning result data 325 contains information indicating the configuration of the estimator 80. However, the configuration of each learning result data (123, 125, 325) does not have to be limited to such an example, and may be appropriately determined according to the embodiment. For example, when the configuration of the neural network used for each is common to each device, each learning result data (123, 125, 325) does not include information indicating the configuration of the neural network. good.

<4.3>
In the above embodiment, the boundary information (65, 77, 3223) indicates the range of defects by the bounding box. However, the method for indicating the range of defects does not have to be limited to such an example, and may be appropriately selected depending on the embodiment. The extent of the defect may be indicated in a format other than the bounding box.

<4.4>
In the above embodiment, the estimator generator 3 is trained to estimate the type and range of defects that may be contained in the product R shown in the image by using the label 3222 and the boundary information 3223 as teacher data. We are building a vessel 80. However, the ability acquired by the estimator 80 does not have to be limited to such an example, and may be appropriately determined according to the embodiment. For example, label 3222 may be omitted from this series of processes. This allows the estimator generator 3 to build an estimator 80 trained to estimate only the extent of the defect.

<4.5>
In the learning network 500 according to the above embodiment, the output of the encoder 56 is connected to the input of the generator 50, and the output of the generator 50 is connected to the input of the estimator 52 and the input of the discriminator 54. Then, in the machine learning process, each parameter of the learning network 500 is set so that the sum of the calculated errors ( _LC , L _BB , L _KL , L _D , L _GD , L _GC , _LG ) becomes small. The value is adjusted. However, the configuration of the learning network 500 does not have to be limited to such an example, and may be appropriately set according to the embodiment. Each machine learning process may be appropriately selected according to the configuration of the generator 50 and the estimator 52. The generator 50 and the estimator 52 may each be trained independently.

FIG. 16 schematically illustrates an example of the software configuration of the learning device 1 that holds the learning network 500A according to this modification. The learning device 1 according to the present modification is configured in the same manner as the above embodiment, except that the learning network 500 according to the above embodiment is replaced with the learning network 500A.

The learning network 500A according to this modification includes a generator 50A and a discriminator 54A. The output of the generator 50A is connected to the input of the discriminator 54A. On the other hand, the estimator 52A is not connected to the generator 50A and is disconnected from the learning network 500A. In this variant, the estimator 52A is trained individually.

Next, an example of the machine learning process of the generator 50A and the estimator 52A will be described with reference to FIGS. 17A to 17C. 17A and 17B schematically illustrate an example of the machine learning process of the learning network 500A including the generator 50A. FIG. 17C schematically illustrates an example of the machine learning process of the estimator 52A.

In this modification, the first learning processing unit 112 performs machine learning of the learning network 500A. Performing machine learning of the learning network 500A includes alternately performing a first training step for training the discriminator 54A and a second training step for training the generator 50A. The configurations of the generator 50A and the discriminator 54A according to the present modification may be the same as those of the generator 50 and the discriminator 54 according to the above embodiment. The first training step and the second training step according to the present modification are substantially the same as the first training step and the second training step according to the above embodiment, respectively. That is, in the first training step, in the first training processing unit 112, the input image input to the discriminator 54A is the image 69 generated by the generator 50A, or the learning obtained from the plurality of training data sets 60. The discriminator 54A is trained to discriminate whether it is the image 61. In the second training step, the first learning processing unit 112 trains the generator 50A so that the image 69 generated by the generator 50A misleads the discrimination by the discriminator 54A.

Specifically, as shown in FIG. 17A, in the first training step, the first learning processing unit 112 extracts noise (latent variable) from a predetermined probability distribution, and combines the extracted noise with each label 63. To generate a plurality of first data sets. The predetermined probability distribution may be, for example, a Gaussian distribution, a uniform distribution, or the like. Any probability distribution may be used for noise extraction. Subsequently, the first learning processing unit 112 inputs each first data set (each label 63 and noise) to the generator 50A, and executes arithmetic processing of the generator 50A. As a result, the first learning processing unit 112 acquires the output (image 69) corresponding to the result of generating the image from each label 63 from the generator 50A. The first learning processing unit 112 generates a plurality of second data sets by combining the generated images 69 and the labels 63.

Next, the first learning processing unit 112 inputs each second data set (each image 69 generated by the generator 50A and each label 63) to the discriminator 54A, and executes arithmetic processing of the discriminator 54A. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50A or the learning data set 60 from the discriminator 54A. In this scene, since each image 69 generated by the generator 50A is an input image, the correct answer is that the discriminator 54A discriminates as "false". The first learning processing unit 112 calculates an error ( _LDA ) between the output value obtained from the discriminator 54A and the correct answer for each second data set.

Similarly, the first learning processing unit 112 inputs the learning image 61 and the label 63 included in each learning data set 60 to the discriminator 54A, and executes the arithmetic processing of the discriminator 54A. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50A or the learning data set 60 from the discriminator 54A. In this scene, since each learning image 61 is an input image, the correct answer is that the discriminator 54A determines that it is “true”. The first learning processing unit 112 calculates an error ( _LDA ) between the output value obtained from the discriminator 54A and the correct answer for each learning data set 60.

Then, the first learning processing unit 112 adjusts the value of the parameter of the discriminator 54A so that the sum of the calculated errors ( _LDA ) becomes small. For example, the first learning processing unit 112 adjusts the value of the parameter of the discriminator 54A by the above series of processing until the sum of the error between the output value obtained from the discriminator 54 and the correct answer of truth is equal to or less than the threshold value. repeat. As a result, in the first training step, in the first training processing unit 112, the input image input to the discriminator 54A is the image 69 generated by the generator 50A or obtained from a plurality of training data sets 60. The discriminator 54A is trained so as to discriminate whether the image is the learning image 61.

As shown in FIG. 17B, in the second training step, the first learning processing unit 112 extracts noise (latent variable) from a predetermined probability distribution, combines the extracted noise with each label 63, and a plurality of data. Generate a set. Each data set may be the same as or different from the first data set described above. Subsequently, the first learning processing unit 112 inputs each data set (each label 63 and noise) to the generator 50A, and executes arithmetic processing of the generator 50A. As a result, the first learning processing unit 112 acquires the output (image 69) corresponding to the result of generating the image from each label 63 from the generator 50A.

Next, the first learning processing unit 112 inputs the combination of each generated image 69 and each corresponding label 63 to the discriminator 54A, and executes the arithmetic processing of the discriminator 54A. As a result, the first learning processing unit 112 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50A or the learning data set 60 from the discriminator 54A. In the training of the generator 50A, the correct answer is that the result of the discrimination by the discriminator 54A is incorrect. That is, the correct answer is that the output value obtained from the discriminator 54A corresponds to "true". The first learning processing unit 112 calculates an error ( _LGDA ) between the output value obtained from the discriminator 54A by a series of processing and the correct answer (that is, “true”) for each data set.

Then, the first learning processing unit 112 adjusts the value of the parameter of the generator 50A so that the sum of the calculated errors ( _LGDA ) becomes small. For example, for each data set, the first learning processing unit 112 performs the parameter of the generator 50A by the above series of processing until the sum of the error between the output value obtained from the discriminator 54A and the "true" becomes equal to or less than the threshold value. Repeat the adjustment of the value of. As a result, in the second training step, the first learning processing unit 112 trains the generator 50A so that the image 69 generated by the generator 50A misleads the discrimination by the discriminator 54A.

In this modification, the first learning processing unit 112 alternately repeats the first training step and the second training step, so that the input of the label 63 included in each learning data set 60 is represented by the input label. A generator 50A trained to generate an image corresponding to the training image 61 associated with 63 is constructed. The storage processing unit 114 generates the first learning result data 123A showing the configuration and parameters of the constructed generator 50A, and stores the generated first learning result data 123A in a predetermined storage area.

On the other hand, as shown in FIG. 17C, the estimator 52A according to this modification is configured by a so-called convolutional neural network. Specifically, the estimator 52A according to this modification has one or more convolutional layers 524, a region proposal network (RPN) 525, a ROI (region of interest) pooling layer 526, or one or more fully coupled. It includes a layer 527, a classification unit 528, and a regression unit 529.

The convolution layer 524 is a layer that performs an image convolution operation. Image convolution corresponds to a process of calculating the correlation between an image and a predetermined filter. When the estimator 52A includes a plurality of convolution layers 524, the input image is input to the convolution layer 524 arranged on the input side most, and the output of the convolution layer 524 arranged on the output side is the region proposal network 525 and ROI pooling. Connected to layer 526.

The region proposal network 525 is configured to propose a plurality of bounding boxes with scores indicating object-likeness for each local region of the feature map output from the convolution layer 524 arranged on the output side. The region proposal network 525 is composed of, for example, one or more fully connected layers, a regression network that predicts the parameters (position and aspect ratio) of the bounding box, and a classification network that predicts the presence or absence of an object. One or more fully connected layers are arranged on the input side. The output of the fully connected layer located on the output side is connected to the regression network and the classification network. The regression network and the classification network are each composed of, for example, one or more fully connected layers.

The ROI pooling layer 526 converts an object region candidate of an arbitrary size proposed based on the output of the region proposal network 525 into a vector of a constant size by pooling the object region candidate of an arbitrary size in the feature map output from the convolution layer 524. It is configured as follows. The output of the ROI pooling layer 526 is connected to the fully coupled layer 527 (located closest to the input side).

Fully connected layer 527 is a layer that connects all neurons between adjacent layers. When the estimator 52A includes a plurality of fully connected layers 527, the output of the fully connected layer 527 arranged on the output side is connected to the classification unit 528 and the regression unit 529. The classification unit 528 is configured to classify the types of defects included in the product R shown in the image from the output of the fully connected layer 527 arranged on the output side. The classification unit 528 is composed of, for example, a fully connected layer. The regression unit 529 is configured to regress from the output of the fully connected layer 527 arranged on the output side to the range of defects included in the product R shown in the image (for example, a bounding box). The regression unit 529 is composed of, for example, a fully connected layer.

In the second machine learning, the second learning processing unit 113 alternately executes the first training step for training the region proposal network 525 and the second training step for training the other parts. In the first training step, the second learning processing unit 113 adjusts the value of the parameter of the region proposal network 525 so that the error regarding the classification of the presence or absence of the object and the error regarding the position and aspect ratio of the bounding box are small. In the second training step, the second learning processing unit 113 utilizes the learned area proposal network 525 and makes each parameter small so that each error (LC, _{LB B} ) becomes small as in the above embodiment. Adjust the value of.

In this modification, the second learning processing unit 113 is input to the input of the learning image 61 included in each learning data set 60 by alternately repeating the first training step and the second training step. Constructs an estimator 52A trained to estimate the type and range of defects contained in the product R appearing in the input training image 61 so as to match the label 63 and boundary information 65 associated with the training image 61. do. The storage processing unit 114 generates the second learning result data 125A showing the configuration and parameters of the constructed estimator 52A, and stores the generated second learning result data 125A in a predetermined storage area. However, the configuration of the estimator 52A does not have to be limited to such an example, and may be appropriately determined according to the embodiment. The configuration of the estimator 52A may be the same as that of the estimator 52.

(Operation example)
Next, an operation example of the learning device 1 according to this modification will be described. The learning device 1 according to this modification can operate in the same manner as in the above embodiment, except that a part of the processing process of each machine learning is different from the above embodiment. That is, in step S101, the control unit 11 acquires a plurality of learning data sets 60. In the next step S102, the control unit 11 operates as the first learning processing unit 112 and executes the first machine learning regarding the generator 50A.

An example of the processing procedure of the first machine learning will be described further with reference to FIG. FIG. 18 illustrates an example of the processing procedure of the first machine learning according to this modification. Performing the first machine learning according to this modification includes the following steps S601 to S603. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

Before executing step S601, the control unit 11 prepares the generator 50A and the discriminator 54A to be processed, as in the above embodiment. In step S601, the control unit 11 adjusts the value of the parameter of the discriminator 54A so that the sum of the calculated errors ( _LDA ) becomes small as described above. As a result, the control unit 11 determines whether the input image input to the discriminator 54A is the image 69 generated by the generator 50A or the training image 61 obtained from the plurality of training data sets 60. As such, the discriminator 54A is trained.

In step S602, the control unit 11 adjusts the value of the parameter of the generator 50A so that the sum of the calculated errors ( _LGDA ) becomes small as described above. As a result, the control unit 11 trains the generator 50A so that the image 69 generated by the generator 50A is such that the image 69 generated by the generator 50A is erroneously discriminated by the discriminator 54A. The adjustment of the value of the parameter in each step S601 and S602 may be processed in the same manner as in the above embodiment.

In step S603, the control unit 11 determines whether or not to repeat the execution of the training steps of steps S602 and S603. Criteria for repeating the execution of the training step may be appropriately determined depending on the embodiment. For example, the number of times the training steps of steps S602 and S603 are executed may be preset. In this case, the control unit 11 determines whether or not the number of times the training steps of steps S602 and S603 have been executed has reached the set number of times. If it is determined that the number of times the training steps of steps S602 and S603 have been executed has not reached the set number of times, the control unit 11 returns the process to step S601. On the other hand, when it is determined that the number of times the training steps of steps S602 and S603 have been executed has reached the set number of times, the control unit 11 completes the first machine learning process according to this modification, and the next step. Proceed to S103. As a result, the control unit 11 is trained to generate an image corresponding to the training image 61 associated with the input label 63 for the input of the label 63 included in each training data set 60. Build 50.

In step S103, the control unit 11 operates as the second learning processing unit 113 and executes the second machine learning regarding the estimator 52A. In the second machine learning, as described above, the control unit 11 alternately executes the first training step for training the region proposal network 525 and the second training step for training the other parts. In the first training step, the control unit 11 adjusts the values of the parameters of the region proposal network 525 so that the error regarding the classification of the presence or absence of the object and the error regarding the position and aspect ratio of the bounding box are small. In the second training step, the control unit 11 uses the learned region proposal network 525 to adjust the value of each parameter so that each error (LC, _{LB B} ) becomes small. The adjustment of the value of the parameter in each step may be processed in the same manner as in the above embodiment. As a result, the control unit 11 includes each training data set 60 in the product R reflected in the input training image 61 so as to match the corresponding label 63 and the boundary information 65 with respect to the input of the training image 61. Build an estimator 52 trained to estimate the type and extent of defects.

In the next step S104, the control unit 11 stores the information about the constructed trained generator 50A and the information about the constructed trained estimator 52A in a predetermined storage area, as in the above embodiment. .. As described above, in the present modification, the learning device 1 can generate the generator 50A and the estimator 52A having the same functions as the generator 50 and the estimator 52 according to the above embodiment.

<4.6>
In the above embodiment, an example in which the present invention is applied to a scene where an appearance inspection of a product R shown in an image is performed is shown. However, the scope of application of the present invention is not limited to such an example of visual inspection. The present invention is applicable to all situations in which the range of an object in a given image is estimated. By changing the image handled by the inspection system 100 from the image of the product R to the image of some object, it is possible to configure an estimation system that estimates the range of the object in the given image.

FIG. 19 schematically illustrates an example of an application scene of the estimation system 100B according to this modification. As shown in FIG. 19, the estimation system 100B according to the present modification includes a learning device 1B, an image generation device 2B, an estimator generation device 3B, and an estimation device 4B connected via a network. Except for changing the image handled from the image of the product R to the image of some object, the hardware configuration and software configuration of each of the devices 1B to 4B are the hardware configurations of the devices 1 to 4 according to the above embodiment. And may be the same as the software configuration. Further, the devices 1B to 4B may operate in the same manner as the devices 1 to 4 according to the above embodiment.

That is, the learning device 1B according to this modification is a combination of the learning image 61B showing the object RB, the label 63B indicating the type of the object RB, and the boundary information 65B indicating the range of the object RB in the learning image 61B. Acquire a plurality of training data sets 60B configured by the above. Next, the learning device 1B performs the first machine learning, and for the input of the label 63B included in each learning data set 60B, the image corresponding to the learning image 61B associated with the input label 63B. Build a generator 50B trained to generate. Further, by performing the second machine learning, the learning device 1B performs the label 63B and the boundary information 65B associated with the input learning image 61B with respect to the input of the learning image 61B included in each learning data set 60B. The estimator 52B trained to estimate the type and range of the object RB reflected in the input learning image 61B is constructed so as to match with. The configuration of the generator 50B and the estimator 52B may be the same as that of the generator 50 and the estimator 52. Further, the machine learning process of the generator 50B and the estimator 52B may be the same as the machine learning process of the generator 50 and the estimator 52.

On the other hand, the image generation device 2B generates a generated image 73B that can include the object RB by giving an input value 71B indicating the type of the object RB to the generator 50B constructed by the learning device 1B. .. Next, the image generation device 2B estimates the type and range of the object RB that can be captured in the generated image 73B by giving the generated image 73B to the estimator 52B constructed by the learning device 1B. Subsequently, the image generation device 2B determines whether or not the result of estimating the type of the object RB reflected in the generated image 73B matches the type of the object RB indicated by the input value 71B. Then, when the image generation device 2B determines that the result of estimating the type of the object RB matches the type of the object RB indicated by the input value 71B, the image generation device 2B estimates the range of the generated generated image 73B and the object RB. It is stored in a predetermined storage area in association with the result.

Further, the estimator generator 3B has a plurality of learnings configured by a combination of a sample image 3221B including the object RB, a label 3222B indicating the type of the object RB, and boundary information 3223B indicating the range of the object RB. Acquire the data set 322B. Next, the estimator generator 3B uses the plurality of learning data sets 322B to perform machine learning of the estimator 80B. The configuration of the estimator 80B may be the same as that of the estimator 80. Further, the machine learning process of the estimator 80B may be the same as the machine learning process of the estimator 80. Thereby, the estimator generator 3B can construct the estimator 80B trained to input the sample image 3221B and output the output value corresponding to the corresponding label 3222B and the boundary information 3223B. Then, the estimator generator 3B generates information indicating the configuration and parameters of the learned estimator 80B as the learning result data 325B. The estimator generator 3B can use the data set generated by the image generator 2B as the learning data set 322B.

FIG. 20 schematically illustrates an example of the software configuration of the estimation device 4B according to this modification. The estimation device 4B according to this modification acquires an object image 422B in which the object RB can be captured. In this modification, the camera CA is connected to the estimation device 4B. The estimation device 4B acquires the target image 422B by photographing the range in which the target object RB can exist with this camera CA. Next, the control unit of the estimation device 4B operates as the estimation unit 412B, and sets the trained estimator 80B with reference to the learning result data 325B. Subsequently, the estimation device 4B inputs the acquired target image 422B to the trained estimator 80B, and executes arithmetic processing of the trained estimator 80B. As a result, the estimation device 4B acquires an output value corresponding to the result of estimating the type and range of the object RB (that is, the result of detecting the object RB) from the estimator 80B. Then, the estimation device 4B outputs information regarding the result of detecting the object RB based on the output value obtained from the estimator 80B. As a result, the estimation system 100B according to the present modification is configured to estimate the range in which the object RB appears in the given image.

The type of the object RB may not be particularly limited as long as it can be a target to be detected from the image, and may be appropriately selected according to the embodiment. The object RB may be, for example, a product, a person, a body part of the person (for example, a face, etc.) to be subject to the visual inspection. Further, when the present modification is used for monitoring road traffic, the object RB may be, for example, a car, a pedestrian, a sign, or the like.

100 ... Inspection system,
1 ... Learning device,
11 ... control unit, 12 ... storage unit, 13 ... communication interface,
14 ... input device, 15 ... output device, 16 ... drive,
111 ... (1st) data acquisition unit, 112 ... 1st learning processing unit,
113 ... 2nd learning processing unit, 114 ... (1st) storage processing unit,
121 ... learning program, 123 ... first learning result data,
125 ... Second learning result data,
2 ... Image generator,
21 ... Control unit, 22 ... Storage unit, 23 ... Communication interface,
24 ... Input device, 25 ... Output device, 26 ... Drive,
211 ... Generation unit, 212 ... Estimating unit, 213 ... Judgment unit,
214 ... Image storage unit,
221 ... Image generation program,
50 ... generator, 52 ... estimator, 54 ... discriminator,
56 ... Encoder,
60 ... (1st) training data set,
61 ... learning image, 63 ... label, 65 ... boundary information,
71 ... Input value, 73 ... Generated image, 75 ... Label,
77 ... Boundary information,
3 ... Estimator generator,
31 ... control unit, 32 ... storage unit, 33 ... communication interface,
34 ... Input device, 35 ... Output device, 36 ... Drive,
311 ... (2nd) Data acquisition unit,
312 ... (3rd) Learning Processing Department,
313 ... (2nd) Preservation processing unit,
321 ... Estimator generation program,
322 ... (2nd) training data set,
3221 ... sample image, 3222 ... label,
3223 ... Boundary information,
325 ... Third learning result data,
4 ... Inspection device (estimation device),
41 ... Control unit, 42 ... Storage unit, 43 ... Communication interface,
44 ... Input device, 45 ... Output device, 46 ... Drive,
47 ... External interface,
411 ... Target data acquisition unit, 412 ... Good / bad judgment unit,
413 ... Output section,
421 ... Inspection program,
80 ... Estimator,
91/92/93/94 ... Storage medium

Claims (6)

Data for acquiring a plurality of training data sets each composed of a combination of a training image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and boundary information indicating the range of the defect. Acquisition department and
It is the first machine learning using a discriminator, and by performing the first machine learning using the information of the pixel of the training image included in the training data set of each case and the label , each of the above cases. With a first learning processing unit that constructs a generator trained to generate an image corresponding to the training image associated with the input label in response to the input of the label included in the training data set of. ,
By performing the second machine learning, the input of the training image included in each of the training data sets is made to match the label and the boundary information associated with the input training image. A second learning processing unit that constructs an estimator trained to estimate the type and range of the defect contained in the product reflected in the input learning image.
A generator that generates a generated image of the product including the defect by giving an input value indicating the type of the defect to the generator constructed by the first machine learning.
By giving the generated image generated to the estimator constructed by the second machine learning, an estimation unit for estimating the type and range of the defect contained in the product reflected in the generated image, and an estimation unit.
A determination unit for determining whether or not the result of estimating the type of the defect contained in the product shown in the generated image matches the type of the defect indicated by the input value.
When the determination unit determines that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image is associated with the result of estimating the range of the defect. Image storage unit stored in the storage area of
Equipped with
The output of the generator is connected to the input of the discriminator.
Performing the first machine learning is
The discriminator is trained to discriminate whether the input image input to the discriminator is the image generated by the generator or the training image obtained from the plurality of training data sets. The first training step to do and
A second training step in which the generator is trained so that the image generated by the generator is such that the discrimination by the discriminator is erroneous.
Including alternating
Inspection system . The input of the generator is connected to the output of the encoder.
The output of the generator is further connected to the input of the estimator.
Performing the first machine learning is
The first training step and
The second training step and
The generator is trained so that the image generated by the generator is such that the result of estimating the type of defect contained in the product shown in the image by the estimator matches the label. The third training step and
A fourth training step of training the encoder and the generator so that the image generated by the generator by inputting the label and the latent variable into the generator is a restored image of the learning image. The fourth training step, wherein the latent variable is derived based on the output obtained from the encoder by inputting the label and the learning image into the encoder.
A fifth training step in which the encoder is trained so that the distribution of the latent variables derived from the output obtained from the encoder follows a predetermined prior distribution.
including,
The inspection system according to claim 1. The extent of the defect is represented by the bounding box.
The inspection system according to claim 1 or 2 . The computer
A step of acquiring a plurality of training data sets each composed of a combination of a training image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and boundary information indicating the range of the defect. When,
It is the first machine learning using a discriminator, and by performing the first machine learning using the information of the pixel of the training image included in the training data set of each case and the label , each of the above cases. For the input of the label contained in the training data set of, a step of constructing a generator trained to generate an image corresponding to the training image associated with the input label.
By performing the second machine learning, the input of the training image included in each of the training data sets is made to match the label and the boundary information associated with the input training image. , Steps to build an estimator trained to estimate the type and extent of the defect contained in the product in the input learning image.
A generation step of generating a generated image showing the product including the defect by giving an input value indicating the type of the defect to the generator constructed by the first machine learning.
By giving the generated image generated to the estimator constructed by the second machine learning, an estimation step for estimating the type and range of the defect contained in the product reflected in the generated image, and an estimation step.
A determination step for determining whether or not the result of estimating the type of the defect contained in the product shown in the generated image matches the type of the defect indicated by the input value.
When it is determined in the determination step that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image is associated with the result of estimating the range of the defect. Image storage step to store in a predetermined storage area,
And run
The output of the generator is connected to the input of the discriminator.
Performing the first machine learning is
The discriminator is trained to discriminate whether the input image input to the discriminator is the image generated by the generator or the training image obtained from the plurality of training data sets. The first training step to do and
A second training step in which the generator is trained so that the image generated by the generator is such that the discrimination by the discriminator is erroneous.
Including alternating
Inspection method. On the computer
A step of acquiring a plurality of training data sets each composed of a combination of a training image showing a product to be visually inspected, a label indicating the type of defect contained in the product, and boundary information indicating the range of the defect. When,
It is the first machine learning using a discriminator, and by performing the first machine learning using the information of the pixel of the training image included in the training data set of each case and the label , each of the above cases. For the input of the label contained in the training data set of, a step of constructing a generator trained to generate an image corresponding to the training image associated with the input label.
By performing the second machine learning, the input of the training image included in each of the training data sets is made to match the label and the boundary information associated with the input training image. , Steps to build an estimator trained to estimate the type and extent of the defect contained in the product in the input learning image.
A generation step of generating a generated image showing the product including the defect by giving an input value indicating the type of the defect to the generator constructed by the first machine learning.
By giving the generated image generated to the estimator constructed by the second machine learning, an estimation step for estimating the type and range of the defect contained in the product reflected in the generated image, and an estimation step.
A determination step for determining whether or not the result of estimating the type of the defect contained in the product shown in the generated image matches the type of the defect indicated by the input value.
When it is determined in the determination step that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image is associated with the result of estimating the range of the defect. Image storage step to store in a predetermined storage area,
To execute ,
The output of the generator is connected to the input of the discriminator.
Performing the first machine learning is
The discriminator is trained to discriminate whether the input image input to the discriminator is the image generated by the generator or the training image obtained from the plurality of training data sets. The first training step to do and
A second training step in which the generator is trained so that the image generated by the generator is such that the discrimination by the discriminator is erroneous.
Including alternating
Inspection program. A data acquisition unit that acquires a plurality of learning data sets each composed of a combination of a learning image showing an object, a label indicating the type of the object, and boundary information indicating the range of the object in the learning image. When,
It is the first machine learning using a discriminator, and by performing the first machine learning using the information of the pixel of the training image included in the training data set of each case and the label , each of the above cases. With a first learning processing unit that constructs a generator trained to generate an image corresponding to the training image associated with the input label in response to the input of the label included in the training data set of. ,
By performing the second machine learning, the input of the training image included in each of the training data sets is made to match the label and the boundary information associated with the input training image. , A second learning processing unit that constructs an estimator trained to estimate the type and range of the object reflected in the input learning image.
A generator that generates a generated image of the product including the defect by giving an input value indicating the type of the defect to the generator constructed by the first machine learning.
By giving the generated image generated to the estimator constructed by the second machine learning, an estimation unit for estimating the type and range of the defect contained in the product reflected in the generated image, and an estimation unit.
A determination unit for determining whether or not the result of estimating the type of the defect contained in the product shown in the generated image matches the type of the defect indicated by the input value.
When the determination unit determines that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image is associated with the result of estimating the range of the defect. Image storage unit stored in the storage area of
Equipped with
The output of the generator is connected to the input of the discriminator.
Performing the first machine learning is
The discriminator is trained to discriminate whether the input image input to the discriminator is the image generated by the generator or the training image obtained from the plurality of training data sets. The first training step to do and
A second training step in which the generator is trained so that the image generated by the generator is such that the discrimination by the discriminator is erroneous.
Including alternating
Inspection system .

Images