JP2020060879A - Learning device, image generator, method for learning, and learning program - Google Patents

Abstract

To reduce the cost for preparing learning data used for a mechanical learning for acquiring an ability to detect the range of an object.SOLUTION: The learning device according to an aspect of the present invention forms a generator trained to acquire plural learning data sets each made of a combination of a learning image of an object, the label showing the type of the object, and boundary information showing the range of the object and to generate an image corresponding to the learning image for an input of each label by performing a first mechanical learning, and also forms an estimation unit trained to estimate the type and the range of the object in the learning image so that the label and the boundary information will conform to each other for input of each learning image by performing a second mechanical learning.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a learning device, an image generation device, a learning method, and a learning program.

  2. Description of the Related Art Conventionally, in a scene of manufacturing a product such as a manufacturing line, a technique of photographing a manufactured product with a photographing device and inspecting the quality of the product based on the obtained image data is used. For example, in Patent Document 1, it is determined whether the inspection target in the image is normal or abnormal based on the learned first neural network, and when it is determined that the inspection target is abnormal, An inspection apparatus has been proposed which classifies the type of abnormality based on the learned second neural network.

JP 2012-026982A

Jianmin Bao, Dong Chen, Fang Wen, Houqiang Li, Gang Hua, "CVAE-GAN: Fine-Grained Image Generation through Asymmetric Training", 2017 IEEE International Conference on Computer Vision, 2764-2773, 2017

  The inventors of the present invention have the following problems in the conventional technology for determining the quality of a product from image data by using a classifier configured by a learning model such as a neural network as disclosed in Patent Document 1. It has been found that this can happen. That is, when machine learning is performed to allow a learning model to acquire the ability to determine the quality of a product, sample images are prepared as training data. If the number of sample images is small, the accuracy of the quality determination by the learned model (discriminator) that has been learned becomes insufficient. On the other hand, it is costly to prepare a sufficient number of sample images in order to improve the accuracy of determination by the classifier.

  Therefore, the inventors of the present invention considered using a generator (generation model) to mass-produce a plurality of different images from prepared images and use the mass-produced images as training data for machine learning. For example, Non-Patent Document 1 proposes a method of constructing a generator (generation model) from prepared learning images and classes by machine learning. This generator is trained by machine learning to generate an image corresponding to a specified class of learning images. Therefore, if the image of the product is used as a learning image and the defects included in the product are designated by the class, the generator is trained to generate a visible image of the product including the designated defect. This trained generator can be used to automatically generate an image of a product containing a specified class of defects. That is, it becomes possible to automatically generate a sample image that can be used for machine learning for making a learning model acquire the ability to identify a defect included in a product. Therefore, the cost of preparing the sample image can be reduced. By using the automatically generated sample images for machine learning, the number of sample images used for machine learning can be increased, and as a result, a learned learning model with high ability to judge the quality of the product can be obtained. Will be able to build.

  However, the inventors of the present invention have found that the following problems may occur in the scene where this generator is used. That is, when identifying a defect included in a product and training the learning model to detect the range of the defect, in addition to the label indicating the type of the defect, boundary information indicating the range of the defect in the sample image ( For example, the ground truth of the bounding box) is prepared as teacher data. Conventionally, the range of this defect has been manually specified in the sample image, and information indicating the specified range has been added to the sample image as boundary information. Therefore, it takes a lot of time to prepare the learning data to be used for machine learning by the steps of designating the range of the defect in the sample image and the step of adding the boundary information. In addition, it is difficult for an operator who does not have knowledge to identify a defect and specify the range of the defect to attach the boundary information to the sample image. Therefore, it took some time to make the workers fully acquire the knowledge. Therefore, it may be costly to prepare the learning data to be used for the machine learning for making the learning model acquire the ability to detect the defect range.

  It is to be noted that such a problem is not limited to the case where the learning model learns the ability to judge the quality of the product and the ability to detect the range of defects, and the learning model to learn the ability to detect the range of some object. It can occur in any situation. For example, assume a case where machine learning is performed to make a learning model acquire the ability to detect a range in which an organ such as an eye appears in a face image showing a face of a subject. In this scene, the range in which the eyes are visible is manually specified in the sample image, and the information indicating the specified range is manually added to the sample image as boundary information. Therefore, the learning data used for machine learning is prepared. It costs a lot of money.

  The present invention, in one aspect, has been made in view of such circumstances, and an object thereof is to prepare learning data used for machine learning for acquiring the ability to detect a range of an object. It is to provide a technique for reducing the cost.

  The present invention adopts the following configurations in order to solve the problems described above.

  That is, the learning device according to one aspect of the present invention uses a learning image obtained by copying the product to be subjected to the visual inspection, a label indicating the type of defect included in the product, and boundary information indicating the range of the defect. By performing the first machine learning with a data acquisition unit that acquires a plurality of learning data sets that are each configured, the label that is input with respect to the input of the label that is included in each of the learning data sets A first learning processing unit that constructs a generator trained to generate an image corresponding to the learning image associated with For the input of the learning image to be input, the product reflected in the input learning image is matched with the label and the boundary information associated with the input learning image. Comprising a second learning processing unit for constructing a trained estimator to estimate the type and scope of Murrell the defect, the.

  According to the learning device according to the configuration, a generator trained to generate an image corresponding to a learning image associated with a specified label is constructed, and the type and range of defects appearing in the learning image are estimated. An estimator trained to do so can be constructed. Of these, by using the generator, it is possible to automatically generate a sample image in which a product including the designated defect can be captured. In addition, by using the estimator, the range of defects included in the generated sample image can be estimated, and information indicating the estimated range of defects can be automatically added to the sample image as boundary information. Therefore, according to the configuration, it is possible to omit the trouble of manually designating the defect range in the sample image when preparing the learning data. Therefore, it is possible to reduce the cost required to prepare the learning data used for machine learning for acquiring the ability to detect the range of the object.

  The estimator and the generator are each composed of a learning model capable of performing machine learning. For the learning model, for example, a neural network or the like may be used. The order of executing the first machine learning and the second machine learning may not be particularly limited, and may be appropriately selected according to the embodiment. The first machine learning and the second machine learning may be executed simultaneously or separately. When the first and second machine learnings are executed separately, the first machine learning may be executed before the second machine learning or may be executed after the second machine learning. Good.

  Further, the type of the product is not particularly limited as long as it can be subjected to the visual inspection, and may be appropriately selected according to the embodiment. The product may be, for example, a product conveyed on a production line for electronic devices, electronic parts, automobile parts, chemicals, foods, and the like. The electronic device is, for example, a mobile phone or the like. The electronic component is, for example, a substrate, a chip capacitor, a liquid crystal, a winding of a relay, or the like. The automobile parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. The drug may be, for example, a packaged tablet, an unpackaged tablet, or the like. Further, the type of the defect is not particularly limited as long as it can relate to the quality of the product, and may be appropriately selected according to the embodiment. The defects include, for example, scratches, stains, cracks, dents, burrs, color unevenness, and foreign matter mixed.

  In the learning device according to the above aspect, the input of the generator may be connected to the output of the encoder, and the output of the generator may be connected to the input of the estimator and the input of the discriminator. Then, performing the first machine learning means that the input image input to the discriminator is the image generated by the generator or the learning image obtained from the plurality of learning data sets. A first training step of training the discriminator to discriminate whether or not there is, and the generator so that the image generated by the generator causes the discrimination by the discriminator to be erroneous. The second training step of training and the image generated by the generator are such that the result of estimating the type of defects included in the product shown in the image by the estimator matches the label. And a third training step of training the generator, and the image generated by the generator by inputting the label and latent variables into the generator restores the learning image. A fourth training step for training the encoder and the generator such that the latent variable is based on an output obtained from the encoder by inputting the label and the training image to the encoder. And a fifth training step of training the encoder so that the distribution of the latent variable derived based on the output obtained from the encoder follows a predetermined prior distribution. Good. According to this configuration, it is possible to appropriately construct a learned generator and an estimator, and thereby prepare learning data used for machine learning for acquiring the ability to detect the range of an object. It is possible to reduce the cost involved. The order in which the first to fifth training steps are executed may not be particularly limited and may be appropriately selected according to the embodiment.

  In the learning device according to the above aspect, the output of the generator may be connected to the input of the discriminator. Then, performing the first machine learning means that the input image input to the discriminator is the image generated by the generator or the learning image obtained from the plurality of learning data sets. A first training step of training the discriminator to determine if there is, and the generator so that the image generated by the generator is such that the discriminator by the discriminator is erroneous. Alternating with a second training step of training According to this configuration, it is possible to appropriately construct a learned generator, and thus the cost of preparing learning data used for machine learning to acquire the ability to detect the range of an object. Can be reduced.

  In the learning device according to the above aspect, the range of the defect may be represented by a bounding box. With this configuration, it is possible to reduce the cost of preparing learning data used for machine learning for acquiring the ability to detect a bounding box as a range of an object.

  Further, the image generation device according to one aspect of the present invention includes the defect including the defect by giving an input value indicating the type of the defect to the generator constructed by the learning device according to any one of the above aspects. By providing a generation unit that generates a generated image of a product and the generated image generated by the estimator constructed by the learning device, the defect type and the defect included in the product reflected in the generated image, and An estimation unit that estimates a range, and a determination unit that determines whether or not the result of estimating the type of the defect included in the product shown in the generated image matches the type of the defect indicated by the input value, When the determination unit determines that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image generated and the result of estimating the range of the defect are associated with each other. Comprising only an image storage unit for storing in a predetermined storage area, the. According to the configuration, it is possible to generate learning data that can be used for machine learning for acquiring the ability to detect the range of the object while omitting the trouble of manually specifying the range of the object.

  Further, the learning device and the image generation device according to each of the above-described embodiments are not limited to the case where the learning model is made to acquire the ability to determine the quality of the product and the ability to detect the range of defects, and the ability to detect the range of some object May be applied to any situation in which the learning model is mastered. For example, the learning device and the image generation device according to each of the above embodiments may be applied to a scene in which a learning model acquires the ability to detect a range in which an organ such as an eye appears in a face image showing a face of a subject. Good.

  For example, the learning device according to one aspect of the present invention is configured by a combination of a learning image of a target object, a label indicating the type of the target object, and boundary information indicating the range of the target object in the learning image. A data acquisition unit that acquires a plurality of learned learning data sets, and performs the first machine learning to associate the input of the labels included in the learning data sets of the respective cases with the input labels. The first learning processing unit that constructs a generator trained to generate an image corresponding to the learned image that has been obtained, and the second machine learning are performed, so that the learning data set included in each of the learning data sets With respect to the input of the learning image, the type and the type of the target object appearing in the input learning image are matched so as to match the label and the boundary information associated with the input learning image. Comprising a second learning processing unit for constructing a trained estimator to estimate the range of.

  The “object” may be of any type as long as it can be an object to be detected from an image, and may be appropriately selected according to the embodiment. The object may be, for example, a product to be subjected to the appearance inspection, a defect included in the product, a person, a human body part (for example, face), a human organ, or the like. When the present invention is applied to a road traffic monitoring image, the object may be, for example, a car, a pedestrian, or a sign.

  The image generation system according to one aspect of the present invention may be configured by the learning device and the image generation device according to any one of the above modes. Further, an estimation system according to one aspect of the present invention includes a learning device according to any one of the above modes, the image generation device, and an estimator that estimates a range of an object captured in the image by using the generated image. It may be configured by an estimator generation device that is constructed and an estimation device that estimates the range of an object shown in an image by using the constructed estimator. Furthermore, as another mode of each of the learning device, the image generation device, the image generation system, and the estimation system according to each of the above modes, one aspect of the present invention may be an information processing method that realizes each of the above configurations. However, it may be a program or a computer-readable storage medium that stores such a program. Here, a computer-readable storage medium is a medium that stores information such as programs by electrical, magnetic, optical, mechanical, or chemical action.

  For example, in a learning method according to one aspect of the present invention, a computer includes a learning image in which a product to be subjected to a visual inspection is copied, a label indicating a type of defect included in the product, and boundary information indicating a range of the defect. The step of acquiring a plurality of learning data sets each configured by a combination of and the first machine learning is performed, and the input of the label included in the learning data set of each case is performed. Constructing a generator trained to generate an image corresponding to the learning image associated with a label, and performing a second machine learning to include the learning included in each of the learning data sets. With respect to the input of the image, the product reflected in the input learning image so as to match the label and the boundary information associated with the input learning image. A step of constructing a trained estimator to estimate the type and extent of the defects included in the execution, an information processing method.

  Further, for example, a learning program according to one aspect of the present invention indicates to a computer a learning image showing a product to be subjected to a visual inspection, a label indicating a type of defect included in the product, and a range of the defect. A step of acquiring a plurality of learning data sets each configured by a combination of boundary information and a first machine learning are performed to input the labels included in the learning data sets of the respective cases. Constructing a generator trained to generate an image corresponding to the learning image associated with the label, and performing a second machine learning to include the learning data set in each case. With respect to the input of the learning image, the input learning image so as to match the label and the boundary information associated with the input learning image For executing the steps of constructing a trained estimator to estimate the type and extent of the defect the included in the product Utsuru, a, a program.

  Further, for example, in a learning method according to one aspect of the present invention, a computer includes a learning image in which an object is copied, a label indicating a type of the object, and boundary information indicating a range of the object in the learning image. The step of acquiring a plurality of learning data sets each configured by a combination of and the first machine learning is performed, and the input of the label included in the learning data set of each case is performed. Constructing a generator trained to generate an image corresponding to the learning image associated with a label, and performing a second machine learning to include the learning included in each of the learning data sets. With respect to the input of the image, the pair captured in the input learning image so as to match the label and the boundary information associated with the input learning image. Performing the steps of constructing a trained estimator to estimate the type and range of the object, and an information processing method.

  Further, for example, the learning program according to one aspect of the present invention, a computer, a learning image of the object, a label indicating the type of the object, and boundary information indicating the range of the object in the learning image. The step of acquiring a plurality of learning data sets each configured by a combination of and the first machine learning is performed, and the input of the label included in the learning data set of each case is performed. Constructing a generator trained to generate an image corresponding to the learning image associated with a label, and performing a second machine learning to include the learning included in each of the learning data sets. When the image is input, it is reflected in the input learning image so as to match the label and the boundary information associated with the input learning image. For executing the steps of constructing a trained estimator to estimate the type and extent of the serial object, and a program.

  According to the present invention, it is possible to reduce the cost required to prepare learning data used for machine learning for acquiring the ability to detect the range of an object.

FIG. 1 schematically illustrates an example of a scene to which the present invention is applied. FIG. 2 schematically illustrates an example of boundary information indicating a range of a defect designated in the learning image according to the embodiment. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device according to the embodiment. FIG. 4 schematically illustrates an example of the hardware configuration of the image generation apparatus according to the embodiment. FIG. 5 schematically illustrates an example of the hardware configuration of the estimator generation device according to the embodiment. FIG. 6 schematically illustrates an example of the hardware configuration of the inspection device according to the embodiment. FIG. 7 schematically illustrates an example of the software configuration of the learning device according to the embodiment. FIG. 8A schematically illustrates an example of a machine learning process of the learning network according to the embodiment. FIG. 8B schematically illustrates an example of a machine learning process of the learning network according to the embodiment. FIG. 9 schematically illustrates an example of the software configuration of the image generating apparatus according to the embodiment. FIG. 10 schematically illustrates an example of the software configuration of the estimator generation device according to the embodiment. FIG. 11 schematically illustrates an example of the software configuration of the inspection device according to the embodiment. FIG. 12 illustrates an example of a processing procedure of the learning device according to the embodiment. FIG. 13 illustrates an example of a processing procedure of the image generation apparatus according to the embodiment. FIG. 14 illustrates an example of the processing procedure of the estimator generation device according to the embodiment. FIG. 15 illustrates an example of a processing procedure of the inspection device according to the embodiment. FIG. 16 schematically illustrates an example of the configuration of the learning network according to the modification. FIG. 17A schematically illustrates an example of a process of machine learning of a learning network according to the modification. FIG. 17B schematically illustrates an example of a process of machine learning of the learning network according to the modification. FIG. 17C schematically illustrates a hypothetical process of machine learning of the estimator according to the modification. FIG. 18 illustrates an example of a processing procedure of the first machine learning according to the modification. FIG. 19 schematically illustrates a modified example of a scene to which the present invention is applied. FIG. 20 schematically illustrates an example of the software configuration of the detection device according to the modification.

  Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “this embodiment”) will be described with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. That is, in implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. It should be noted that although the data that appears in this embodiment is described in natural language, more specifically, it is specified by a computer-recognizable pseudo language, command, parameter, machine language, or the like.

§1 Application Example First, an example of a scene to which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a situation in which the present invention is applied to an appearance inspection of a product R. The defects that can be included in the product R are an example of the “object” of the present invention. However, the application range of the present invention is not limited to the example of the visual inspection described below. INDUSTRIAL APPLICABILITY The present invention can be applied to all situations in which the range of any object in an image is detected.

  As illustrated in FIG. 1, the inspection system 100 according to the present embodiment includes a learning device 1, an image generation device 2, an estimator generation device 3, and an inspection device 4 which are connected via a network. Accordingly, the inspection system 100 is configured to inspect the quality of the product R. The type of network among the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 may not be particularly limited, and examples thereof include the Internet, a wireless communication network, a mobile communication network, and a telephone network. It may be appropriately selected from a dedicated network or the like.

  In the example of FIG. 1, the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 are separate computers. However, the configuration of the inspection system 100 may not be limited to such an example. At least one pair of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 may be an integrated computer. The learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 may each be configured by a plurality of computers.

  The learning device 1 according to the present embodiment includes a generator 50 for generating an image of a product R to be subjected to a visual inspection, and a defect included in the product R with respect to the image generated by the generator 50. A computer configured to build an estimator 52 for estimating type and range. Specifically, the learning device 1 uses a combination of a learning image 61 showing a product R to be subjected to an appearance inspection, a label 63 indicating a type of defect included in the product R, and boundary information 65 indicating a range of the defect. A plurality of configured learning data sets 60 are acquired.

  Subsequently, the learning device 1 executes the first machine learning to correspond to the learning image 61 associated with the input label 63 with respect to the input of the label 63 included in each learning data set 60. Construct a generator 50 trained to generate an image. The image corresponding to the learning image 61 is an image similar to the learning image 61 and in which the product R can be captured. The generator 50 is trained to generate an image according to the distribution of the learning image 61. Further, the learning device 1 executes the second machine learning, so that for the input of the learning image 61 included in each learning data set 60, the label 63 and the boundary information associated with the input learning image 61. The estimator 52 trained to estimate the type and range of the defect included in the product R shown in the input learning image 61 is constructed so as to match 65.

  Here, an example of a method of designating a defect range will be described with reference to FIG. FIG. 2 schematically illustrates an example of the boundary information 65 indicating the range of the defect designated in the learning image 61 according to this embodiment. In the example of FIG. 2, the learning image 61 shows the product R including the elliptical defect 612. The learning image 61 has a horizontal width of W pixels and a vertical width of H pixels, whereas the defect 612 has a horizontal width of w pixels and a vertical width of h pixels. The range of this defect 612 may be appropriately designated so as to include the region in which the defect 612 appears in the learning image 61.

  The method of expressing the range of the defect 612 is not particularly limited and may be appropriately selected according to the embodiment. In this embodiment, the range of the defect 612 is represented by a bounding box. As shown in FIG. 2, the center coordinates (cx, cy) of the defect 612, the width (w pixels), and the height (h pixels) can define the bounding box. That is, the rectangular range surrounding the outer circumference of the defect 612 (the range indicated by the dotted line in the figure) is an example of the bounding box. However, the format that defines the bounding box is not limited to such an example, and may be appropriately selected according to the embodiment. For example, the bounding box may be defined by the radius of a circle (eg, a circumscribing circle) designated to include the center coordinate and the defect 612. In this case, the circular range including the defect 612 becomes the bounding box. According to the learning device 1 according to the present embodiment, the generator 50 that is trained to generate the image corresponding to the learning image 61 is constructed, and the range of the defect represented by such a bounding box is used as the learning image. An estimator 52 trained to estimate in 61 can be constructed.

  Returning to FIG. 1, the image generation device 2 according to the present embodiment uses the learned generator 50 and the estimator 52 constructed by the learning device 1 to generate an image in which the product R can be captured and generate the image. Is a computer configured to estimate the extent of a defect in the captured image. Specifically, the image generation device 2 generates the generated image 73 in which the product R including the defect is copied by giving the generator 50 constructed by the learning device 1 an input value 71 indicating the type of the defect. . The generated image 73 generated corresponds to the learning image 61.

  Next, the image generation device 2 gives the generated image 73 to the estimator 52 constructed by the learning device 1 to estimate the type and range of the defect included in the product R shown in the generated image 73. Subsequently, the image generation device 2 determines whether or not the result of estimating the type of the defect included in the product shown in the generated image 73 matches the type of the defect indicated by the input value 71. Then, when the image generation device 2 determines that the result of estimating the type of defect matches the type of defect indicated by the input value 71, the generated image 73 and the result of estimating the range of the defect are associated with each other. Store in a predetermined storage area.

  The estimator generation device 3 according to the present embodiment is a computer configured to construct an estimator for estimating the quality of the product R. Specifically, the estimator generation device 3 acquires a plurality of learning data sets each configured by a combination of a sample image of the product R and correct answer data. The correct answer data indicates the result (that is, correct answer) of the quality of the product R shown in the sample image. In this embodiment, the correct answer data includes boundary information indicating the range of defects included in the product R shown in the sample image. Then, the estimator generation device 3 determines whether the product R shown in the given image is good or bad by performing machine learning using a plurality of learning data sets, and when the product R includes a defect, Build a trained estimator that has acquired the ability to detect the range of the defect. Note that the estimator generation device 3 uses the generated image 73 generated by the image generation device 2 as a sample image, and uses information indicating the result of estimating the range of the defect associated with the generated image 73 as the correct answer data. can do.

  On the other hand, the inspection device 4 according to the present embodiment is a computer configured to determine the quality of the product R using the learned estimator constructed by the estimator generation device 3. The inspection device 4 is an example of an estimation device that estimates the range of an object shown in an image. Specifically, the inspection device 4 acquires a target image of the product R that is the target of the visual inspection. In the present embodiment, a camera CA is connected to the inspection device 4. The inspection device 4 acquires the target image by photographing the product R with the camera CA. Next, the inspection device 4 inputs the acquired target image to the learned estimator, and executes the arithmetic processing of the learned estimator. Thereby, the inspection device 4 acquires the output value corresponding to the result of determining the quality of the product R and the result of detecting the range of defects included in the product R from the learned estimator. Then, the inspection device 4 outputs information regarding the result of determining the quality of the product R based on the output value obtained from the learned estimator.

  As described above, the learning device 1 according to the present embodiment builds the generator 50 trained to generate the image corresponding to the learning image 61 associated with the label 63 that specifies the defect type, and It is possible to construct the estimator 52 that has been trained to estimate the type and range of the defect shown in the learning image 61. Of these, by using the generator 50, it is possible to automatically generate a sample image in which a product including the designated defect can be captured. In addition, by using the estimator 52, the range of defects included in the product R shown in the generated sample image is estimated, and information indicating the estimated range of defects is automatically added to the sample image as boundary information. can do. Therefore, according to the present embodiment, when preparing the learning data, it is possible to omit the trouble of manually specifying the defect range in the sample image. Therefore, it is possible to reduce the cost required to prepare learning data used for machine learning for acquiring the ability to detect the range of defects.

  Further, in the image generating apparatus 2 according to the present embodiment, by using the generator 50 and the estimator 52 constructed by the learning apparatus 1, the trouble of manually specifying the defect range is omitted, and Learning data that can be used for machine learning to acquire the ability to detect a range can be generated. In addition, in the present embodiment, the generated image 73 generated by the image generation device 2 is used as a sample image, and the information indicating the result of estimating the range of the defect associated with the generated image 73 is used as the correct answer data. be able to. As a result, it is possible to reduce the cost of collecting the learning data set in the estimator generation device 3. Furthermore, in the present embodiment, the generated image 73 generated by the image generation device 2 is used as a sample image, and the information indicating the result of estimating the range of the defect associated with the generated image 73 is used as the correct answer data. Can increase the number of training data sets used for machine learning of the estimator. Thereby, in the inspection device 4, the accuracy of estimating the range of defects that can be included in the product R can be improved.

  Note that the product R to be subjected to the visual inspection is not particularly limited, and may be appropriately selected according to the embodiment. The product R may be, for example, a product conveyed on a manufacturing line for electronic components, automobile components, and the like. The electronic component is, for example, a substrate, a chip capacitor, a liquid crystal, a winding of a relay, or the like. The automobile parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. Further, the determination of pass / fail may be performed simply by determining whether or not the product R has a defect, or in addition to determining whether or not the product R has a defect, the type of the defect is identified. It may include doing. The defects are, for example, scratches, dirt, cracks, dents, dust, burrs, color unevenness, and the like.

§2 Configuration example [Hardware configuration]
<Learning device>
Next, an example of the hardware configuration of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 3 schematically illustrates an example of the hardware configuration of the learning device 1 according to the present embodiment.

  As shown in FIG. 3, the learning device 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected. . Note that, in FIG. 3, the communication interface is described as “communication I / F”.

  The control unit 11 includes a CPU (Central Processing Unit) that is a hardware processor, a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and is configured to execute information processing based on a program and various data. It The storage unit 12 is an example of a memory, and includes, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage unit 12 stores various information such as the learning program 121, the plurality of learning data sets 60, the first learning result data 123, the second learning result data 125.

  The learning program 121 constructs a generator 50 trained to generate a visible image of the product R and an estimator 52 trained to estimate the type and range of defects that may be included in the product R in the image. It is a program for causing the learning device 1 to execute information processing (FIG. 12) described later. The learning program 121 includes a series of instructions for the information processing. Each learning data set 60 is composed of a combination of a learning image 61 showing a product R to be subjected to an appearance inspection, a label 63 showing a type of defect included in the product R, and boundary information 65 showing a range of the defect. . Each learning data set 60 is used for machine learning of the generator 50 and the estimator 52. The number of learning data sets 60 may be appropriately determined according to the embodiment. The first learning result data 123 is data for setting the learned generator 50 constructed by machine learning. The second learning result data 125 is data for setting the learned estimator 52 constructed by machine learning. The first learning result data 123 and the second learning result data 125 are generated as execution results of the learning program 121. Details will be described later.

  The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The learning device 1 can perform data communication via the network with another information processing device (for example, the image generation device 2 and the estimator generation device 3) by using the communication interface 13.

  The input device 14 is a device for inputting, for example, a mouse or a keyboard. The output device 15 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the learning device 1 by using the input device 14 and the output device 15.

  The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 91. The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. At least one of the learning program 121 and the plurality of learning data sets 60 may be stored in the storage medium 91.

  The storage medium 91 stores information such as a recorded program by an electrical, magnetic, optical, mechanical, or chemical action so that a computer, other device, machine, or the like can read the recorded program or other information. It is a storage medium. The learning device 1 may acquire at least one of the learning program 121 and the plurality of learning data sets 60 from the storage medium 91.

  Here, in FIG. 3, a disk-type storage medium such as a CD or a DVD is illustrated as an example of the storage medium 91. However, the type of the storage medium 91 is not limited to the disc type, and may be other than the disc type. As a storage medium other than the disk type, for example, a semiconductor memory such as a flash memory can be cited.

  Regarding the specific hardware configuration of the learning device 1, it is possible to appropriately omit, replace, and add the constituent elements depending on the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be configured by a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage unit 12 may include a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The learning device 1 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the learning device 1 may be a general-purpose server device, a PC (Personal Computer), or the like, in addition to the information processing device designed for the provided service.

<Image generation device>
Next, an example of the hardware configuration of the image generating apparatus 2 according to this embodiment will be described with reference to FIG. FIG. 4 schematically illustrates an example of the hardware configuration of the image generating apparatus 2 according to this embodiment.

  As shown in FIG. 4, the image generation device 2 according to the present embodiment is a computer in which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected. is there. Each of the control unit 21 to the drive 26 of the image generation device 2 according to the present embodiment may be configured similarly to each of the control unit 11 to the drive 16 of the learning device 1.

  That is, the control unit 21 includes a CPU that is a hardware processor, a RAM, a ROM, and the like, and is configured to execute various types of information processing based on programs and data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as the image generation program 221, the first learning result data 123, the second learning result data 125, one or a plurality of data sets.

  The image generation program 221 uses the learned generator 50 to generate the generated image 73, and uses the learned estimator 52 to estimate the range of defects in the generated image 73. 13) is a program for causing the image generating apparatus 2 to execute (FIG. 13). The image generation program 221 includes a series of instructions for the information processing. Each data set is composed of a combination of the generated image 73, the label 75, and the boundary information 77. The label 75 corresponds to the input value 71 given to the generator 50 when generating the generated image 73. The boundary information 77 indicates the result of estimating the defect range by the estimator 52. Each data set is generated as the execution result of the image generation program 221. Details will be described later.

  The communication interface 23 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The image generation device 2 can perform data communication via the network with another information processing device (for example, the learning device 1 and the estimator generation device 3) by using the communication interface 23.

  The input device 24 is a device for inputting, for example, a mouse or a keyboard. The output device 25 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the image generating apparatus 2 by using the input device 24 and the output device 25.

  The drive 26 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 92. At least one of the image generation program 221, the first learning result data 123, and the second learning result data 125 may be stored in the storage medium 92. Further, the image generating device 2 may acquire at least one of the image generating program 221, the first learning result data 123, and the second learning result data 125 from the storage medium 92.

  Regarding the specific hardware configuration of the image generating apparatus 2, the constituent elements can be omitted, replaced, and added as appropriate according to the embodiment. For example, the control unit 21 may include a plurality of hardware processors. The hardware processor may be configured by a microprocessor, FPGA, DSP or the like. The storage unit 22 may include a RAM and a ROM included in the control unit 21. At least one of the communication interface 23, the input device 24, the output device 25, and the drive 26 may be omitted. The image generation device 2 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the image generating device 2 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Estimator generator>
Next, an example of the hardware configuration of the estimator generation device 3 according to the present embodiment will be described with reference to FIG. FIG. 5 schematically illustrates an example of the hardware configuration of the estimator generation device 3 according to this embodiment.

  As shown in FIG. 5, the estimator generation device 3 according to the present embodiment is a computer in which a control unit 31, a storage unit 32, a communication interface 33, an input device 34, an output device 35, and a drive 36 are electrically connected. Is. Each of the control unit 31 to the drive 36 of the estimator generation device 3 according to the present embodiment may be configured similarly to each of the control unit 11 to the drive 16 of the learning device 1.

  That is, the control unit 31 includes a CPU that is a hardware processor, a RAM, a ROM, and the like, and is configured to execute various types of information processing based on programs and data. The storage unit 32 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 32 stores various kinds of information such as the estimator generation program 321, the learning data set 322, and the third learning result data 325.

  The estimator generation program 321 is a program for causing the estimator generation device 3 to execute the information processing (FIG. 14) described later that constructs an estimator for determining the quality of the product R. The estimator generation program 321 includes a series of instructions for the information processing. The learning data set 322 is used for machine learning of this estimator. The third learning result data 325 is data for setting the learned estimator constructed by machine learning. The third learning result data 325 is generated as the execution result of the estimator generation program 321. Details will be described later.

  The communication interface 33 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 33, the estimator generation device 3 can perform data communication via the network with another information processing device (for example, the learning device 1, the image generation device 2, the inspection device 4). .

  The input device 34 is a device for inputting, for example, a mouse or a keyboard. The output device 35 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the estimator generation device 3 by using the input device 34 and the output device 35.

  The drive 36 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 93. At least one of the estimator generation program 321 and the learning data set 322 may be stored in the storage medium 93. Further, the estimator generation device 3 may acquire at least one of the estimator generation program 321 and the learning data set 322 from the storage medium 93.

  Regarding the specific hardware configuration of the estimator generation device 3, it is possible to appropriately omit, replace, and add the constituent elements depending on the embodiment. For example, the control unit 31 may include a plurality of hardware processors. The hardware processor may be configured by a microprocessor, FPGA, DSP or the like. The storage unit 32 may include a RAM and a ROM included in the control unit 31. At least one of the communication interface 33, the input device 34, the output device 35, and the drive 36 may be omitted. The estimator generation device 3 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. The estimator generation device 3 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Inspection device>
Next, an example of the hardware configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 6 schematically illustrates an example of the hardware configuration of the inspection device 4 according to the present embodiment.

  As shown in FIG. 6, the inspection device 4 according to the present embodiment is electrically connected to the control unit 41, the storage unit 42, the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47. It is a computer. In FIG. 6, the external interface is described as “external I / F”. The control unit 41 to the drive 46 of the inspection device 4 may be configured similarly to the control unit 11 to the drive 16 of the learning device 1, respectively.

  That is, the control unit 41 includes a CPU that is a hardware processor, a RAM, a ROM, and the like, and is configured to execute various types of information processing based on programs and data. The storage unit 42 includes, for example, a hard disk drive, a solid state drive, or the like. The storage unit 42 stores various information such as the inspection program 421 and the third learning result data 325.

  The inspection program 421 uses the learned estimator constructed by the estimator generation device 3 to cause the inspection device 4 to execute the information processing (FIG. 15) described below for determining the quality of the product R shown in the target image. It is a program for. The inspection program 421 includes a series of instructions for the information processing. Details will be described later.

  The communication interface 43 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. The inspection device 4 can perform data communication via the network with another information processing device (for example, the estimator generation device 3) by using the communication interface 43.

  The input device 44 is a device for inputting, for example, a mouse or a keyboard. The output device 45 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the inspection device 4 by using the input device 44 and the output device 45.

  The drive 46 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 94. At least one of the inspection program 421 and the third learning result data 325 may be stored in the storage medium 94. Further, the inspection device 4 may acquire at least one of the inspection program 421 and the third learning result data 325 from the storage medium 94.

  The external interface 47 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of external interfaces 47 may be appropriately selected according to the type and number of external devices to be connected. In the present embodiment, the inspection device 4 is connected to the camera CA via the external interface 47.

  The camera CA is used to acquire a target image of the product R. The type and location of the camera CA may not be particularly limited and may be appropriately determined according to the embodiment. As the camera CA, a known camera such as a digital camera or a video camera may be used. Further, the camera CA may be arranged in the vicinity of the manufacturing line on which the product R is conveyed. If the camera CA has a communication interface, the inspection device 4 may be connected to the camera CA via the communication interface 43 instead of the external interface 47.

  Regarding the specific hardware configuration of the inspection device 4, it is possible to appropriately omit, replace, and add components according to the embodiment. For example, the control unit 41 may include a plurality of hardware processors. The hardware processor may be configured by a microprocessor, FPGA, DSP or the like. The storage unit 42 may include a RAM and a ROM included in the control unit 41. At least one of the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47 may be omitted. The inspection device 4 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, as the inspection device 4, a general-purpose server device, a general-purpose desktop PC, a notebook PC, a tablet PC, a mobile phone including a smartphone, and the like may be used in addition to the information processing device designed for the provided service.

[Software configuration]
<Learning device>
Next, an example of the software configuration of the learning device 1 according to the present embodiment will be described using FIG. 7. FIG. 7 schematically illustrates an example of the software configuration of the learning device 1 according to the present embodiment.

  The control unit 11 of the learning device 1 loads the learning program 121 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the learning program 121 expanded in the RAM by the CPU to control each component. As a result, as shown in FIG. 7, the learning device 1 according to the present embodiment is a computer that includes the data acquisition unit 111, the first learning processing unit 112, the second learning processing unit 113, and the storage processing unit 114 as software modules. To work as. That is, in the present embodiment, each software module of the learning device 1 is realized by the control unit 11 (CPU).

  The data acquisition unit 111 is configured by a combination of a learning image 61 showing a product R to be subjected to an appearance inspection, a label 63 indicating a defect type included in the product R, and boundary information 65 indicating a defect range. Acquire a plurality of learning data sets 60. By executing the first machine learning, the first learning processing unit 112 corresponds to the learning image 61 associated with the input label 63 with respect to the input of the label 63 included in each learning data set 60. Construct a generator 50 trained to generate an image. By executing the second machine learning, the second learning processing unit 113, with respect to the input of the learning image 61 included in each learning data set 60, the label 63 and the boundary associated with the input learning image 61. The estimator 52 that is trained to estimate the type and range of defects included in the product R shown in the input learning image 61 is constructed so as to match the information 65. The saving processing unit 114 saves the information about the constructed learned generator 50 and the information about the constructed learned estimator 52 in a predetermined storage area.

(Learning network)
Next, an example of the configuration of the learning network 500 including the generator 50 and the estimator 52 according to the present embodiment will be described by further using FIGS. 8A and 8B. 8A and 8B schematically illustrate an example of a first machine learning process for training the generator 50 and a second machine learning process for training the estimator 52.

  As shown in each drawing, in the present embodiment, the learning network 500 is configured by the encoder 56, the generator 50, the discriminator 54, and the estimator 52. The first learning processing unit 112 and the second learning processing unit 113 hold a learning network 500 that is a target for carrying out this machine learning. In the learning network 500, the encoder 56, the generator 50, the discriminator 54, and the estimator 52 are arranged in this order from the input side. The discriminator 54 and the estimator 52 are arranged in parallel on the output side. The encoder 56 is configured to receive the input of the learning image 61 and the label 63 and output the output value corresponding to the latent variable along the prior distribution. The output of encoder 56 is connected to the input of generator 50. The output of the generator 50 is connected to the input of the estimator 52 and the input of the discriminator 54.

  The encoder 56, the generator 50, the estimator 52, and the discriminator 54 according to this embodiment are each configured by a neural network having a multilayer structure used for so-called deep learning. The encoder 56 includes an input layer 571, an intermediate layer (hidden layer) 572, and an output layer 573. The generator 50 includes an input layer 511, an intermediate layer (hidden layer) 512, and an output layer 513. The estimator 52 includes an input layer 531, an intermediate layer (hidden layer) 532, and an output layer 533. The discriminator 54 includes an input layer 551, an intermediate layer (hidden layer) 552, and an output layer 553. However, the configurations of the encoder 56, the generator 50, the estimator 52, and the discriminator 54 are not limited to such an example, and may be set appropriately according to the embodiment. For example, the encoder 56, the generator 50, the estimator 52, and the discriminator 54 may each include two or more intermediate layers. The configurations of the encoder 56, the generator 50, the estimator 52, and the discriminator 54 may be different from each other.

  The number of neurons (nodes) included in each of the layers (571 to 573, 511 to 513, 531 to 533, 551 to 553) may be appropriately set according to the embodiment. The neurons in the adjacent layers are appropriately connected to each other, and a weight (connection weight) is set for each connection. In the example of FIGS. 8A and 8B, each neuron is connected to all neurons in the adjacent layers. However, the connection of neurons need not be limited to such an example, and may be set appropriately according to the embodiment. A threshold value is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of products of each input and each weight exceeds the threshold value.

  The weights of the connections between the neurons and the thresholds of the neurons included in the layers 571 to 573 of the encoder 56 are examples of the parameters of the encoder 56 used for the arithmetic processing. The weights of the connections between the neurons and the thresholds of the neurons included in the layers 511 to 513 of the generator 50 are examples of the parameters of the generator 50 used for the arithmetic processing. The weights of the connections between the neurons and the thresholds of the neurons included in the layers 531 to 533 of the estimator 52 are examples of the parameters of the estimator 52 used for the arithmetic processing. The weights of the connections between the neurons and the thresholds of the neurons included in the layers 551 to 553 of the discriminator 54 are examples of the parameters of the discriminator 54 used in the arithmetic processing.

  The first learning processing unit 112 and the second learning processing unit 113 carry out machine learning of the learning network 500. The first learning processing unit 112 executes the first machine learning regarding the generator 50 among these. On the other hand, the second learning processing unit 113 carries out the second machine learning regarding the estimator 52.

  In the present embodiment, performing the first machine learning includes the following first to fifth training steps. In the first training step, the first learning processing unit 112 determines whether the input image input to the discriminator 54 is the image 69 generated by the generator 50 or the learning image 61 obtained from the plurality of learning data sets 60. The discriminator 54 is trained to discriminate whether or not That is, the discriminator 54 is trained to determine whether the given input image is derived from the learning data (learning image 61) or the generator 50. In the second training step, the first learning processing unit 112 trains the generator 50 so that the image 69 generated by the generator 50 causes the discrimination by the discriminator 54 to be erroneous. In the examples of FIGS. 8A and 8B, the fact that the learning data is derived is expressed as “true”, and the origin of the generator 50 is expressed as “false”. However, the method of expressing each origin is not limited to such an example, and may be appropriately selected according to the embodiment.

  In the third training step, in the first learning processing unit 112, the label 63 corresponding to the result of the image 69 generated by the generator 50 estimating the type of defect included in the product R shown in the image 69 by the estimator 52. Train the generator 50 to be such that In the fourth training step, the first learning processing unit 112 inputs the label 63 and the latent variable to the generator 50, and the image 69 generated by the generator 50 is a reconstructed version of the learning image 61 (that is, learning). The encoder 56 and the generator 50 are trained to be (image corresponding to the image 61). The latent variable is derived based on the output obtained from the encoder 56 by inputting the label 63 and the learning image 61 to the encoder 56. In the fifth training step, the first learning processing unit 112 trains the encoder 56 so that the distribution of the latent variable derived based on the output obtained from the encoder 56 follows a predetermined prior distribution. The type of prior distribution may not be particularly limited and may be appropriately selected according to the embodiment. A known distribution such as a Gaussian distribution may be used as the prior distribution. The order in which the first to fifth training steps are executed may not be particularly limited, and may be appropriately determined according to the embodiment.

Specifically, as shown in FIG. 8A, the second learning processing unit 113 inputs the learning image 61 into the input layer 531 of the estimator 52 for each learning data set 60, and executes the arithmetic processing of the estimator 52. . As a result, the second learning processing unit 113 acquires, from the output layer 533, an output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the learning image 61. The second learning processing unit 113 calculates the error (L _C ) between the output value corresponding to the result of estimating the defect type and the value of the label 63. The second learning processing unit 113 also calculates an error (L _BB ) between the output value corresponding to the result of estimating the defect range and the value of the boundary information 65. A known index such as IoU (Intersection Over Union) may be used for the calculation of the error (L _BB ).

Further, the first learning processing unit 112 inputs the learning image 61 and the label 63 to the input layer 571 of the encoder 56 for each learning data set 60, and executes the arithmetic processing of the encoder 56. As a result, the first learning processing unit 112 acquires the output value (latent variable) corresponding to the result of deriving a certain feature amount from the output layer 573 of the encoder 56. This output value may be set to correspond to any random variable of the prior distribution (eg, mean and covariance). The first learning processing unit 112 calculates the error (L _KL ) between this output value and the value derived from the prior distribution.

  Next, the first learning processing unit 112 inputs the output value obtained from the encoder 56 and the corresponding label 63 to the input layer 511 of the generator 50, and executes the arithmetic processing of the generator 50. As a result, the first learning processing unit 112 acquires, from the output layer 513, an output (image 69) corresponding to the result of generating the image corresponding to the learning image 61 from the label 63 for each learning data set 60.

Subsequently, the first learning processing unit 112 inputs each image 69 generated by the generator 50 to the input layer 551 of the discriminator 54, and executes the arithmetic processing of the discriminator 54. As a result, the first learning processing unit 112 acquires, from the output layer 553, an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 (that is, the learning image 61 itself). In this scene, since each image 69 generated by the generator 50 is an input image, it is the correct answer for the discriminator 54 to determine “false”. The first learning processing unit 112 calculates an error (L _D ) between the output value obtained from the output layer 553 and the correct answer for each image 69 generated by the generator 50.

Similarly, for each learning data set 60, the first learning processing unit 112 inputs the learning image 61 to the input layer 551 of the discriminator 54 and executes the arithmetic processing of the discriminator 54. Thereby, the first learning processing unit 112 acquires the output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 from the output layer 553. In this scene, since the learning image 61 itself is the input image, it is the correct answer for the discriminator 54 to determine “true”. The first learning processing unit 112 calculates the error (L _D ) between the output value obtained from the output layer 553 and this correct answer for each learning data set 60.

Further, as shown in FIG. 8B, the first learning processing unit 112 causes the discriminator 54 to make an erroneous determination regarding the output value obtained by inputting each image 69 generated by the generator 50 to the discriminator 54. Calculate the error (L _GD ) for training the generator 50 to generate such an image. Specifically, in the training of the generator 50, the correct answer is to make the discrimination result by the discriminator 54 erroneous. That is, the correct answer is that the output value obtained from the output layer 553 corresponds to “true”. The first learning processing unit 112 calculates, for each learning data set 60, an error (L _GD ) between the output value obtained from the output layer 553 through a series of processes and this correct answer (that is, “true”).

Next, the first learning processing unit 112 inputs each image 69 generated by the generator 50 into the input layer 531 of the estimator 52, and executes the arithmetic processing of the estimator 52. Accordingly, the first learning processing unit 112 acquires the output value corresponding to the result of estimating the type of defect included in the product R shown in each image 69 from the output layer 533 of the estimator 52. The first learning processing unit 112 calculates an error (L _GC ) between the output value corresponding to the result of estimating the defect type and the value of the corresponding label 63.

Next, as shown in FIG. 8A, the first learning processing unit 112, for each learning data set 60, an error for training the generator 50 to function as a decoder that restores the learning image 61 from the output of the encoder 56. To calculate. Specifically, the first learning processing unit 112 calculates the error (L _G ) between the learning image 61 and the image 69 generated by the generator 50 for each learning data set 60.

Then, as shown in FIGS. 8A and 8B, the first learning processing unit 112 and the second learning processing unit 113, the error (L _{C calculated, L BB, L KL, L} D, L GD, L GC , L _G ), the value of each parameter of the learning network 500 is adjusted. Specifically, the second learning processing unit 113, the error (L _C, L _BB) which is calculated as the sum of the decreases, adjusting the value of the parameter estimator 52. The first learning processing unit 112 adjusts the parameter values of the discriminator 54 so that the sum of the calculated errors (L _D ) becomes small. The first learning processing unit 112 adjusts the values of the parameters of the generator 50 so that the sum of the calculated errors (L _G , L _GD , L _GC ) becomes small. The first learning processing unit 112 adjusts the parameter values of the encoder 56 so that the sum of the calculated errors (L _KL , L _G ) becomes small.

The first learning processing unit 112 and the second learning processing unit 113 until the sum of the calculated errors (L _C , L _BB , L _KL , L _D , L _GD , L _GC , L _G ) becomes less than or equal to the threshold value. By the series of processes described above, the adjustment of the value of each parameter is repeated. Thereby, the first learning processing unit 112 trains each of the discriminator 54, the generator 50, and the encoder 56, and the second learning processing unit 113 trains the estimator 52. Of these, the process of adjusting parameters of the estimator 52 so that the sum is less of each error (L _C, L _BB) is an example of the processing of a second machine learning. The process of adjusting the parameter values of the encoder 56 so that the sum of the errors (L _KL ) is small is an example of the fifth training step of the first machine learning. The process of adjusting the parameter values of the discriminator 54 so that the sum of the errors (L _D ) becomes small is an example of the first training step. The process of adjusting the parameter values of the generator 50 so that the sum of the errors (L _GD ) becomes small is an example of the second training step. The process of adjusting the parameter values of the generator 50 so that the sum of the errors (L _GC ) becomes small is an example of the third training step. The process of adjusting the parameter values of the generator 50 and the encoder 56 so that the sum of the errors (L _G ) becomes small is an example of the fourth training step. It should be noted that the criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the first learning processing unit 112 and the second learning processing unit 113 may repeat the adjustment of the value of each parameter by the designated number of times by the series of processes described above.

  After this machine learning is completed, the storage processing unit 114 configures the constructed generator 50 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, the transfer function of each neuron), And the first learning result data 123 indicating the calculation parameters (for example, the weight of the connection between the neurons and the threshold value of the neurons). Further, the storage processing unit 114 configures the constructed estimator 52 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, the transfer function of each neuron), and the operation parameter (for example, each The second learning result data 125 indicating the weight of connection between neurons and the threshold value of each neuron is generated. Then, the storage processing unit 114 stores the generated first learning result data 123 and second learning result data 125 in a predetermined storage area.

<Image generation device>
Next, an example of the software configuration of the image generating apparatus 2 according to this embodiment will be described with reference to FIG. FIG. 9 schematically illustrates an example of the software configuration of the image generating apparatus 2 according to this embodiment.

  The control unit 21 of the image generation device 2 loads the image generation program 221 stored in the storage unit 22 into the RAM. Then, the control unit 21 controls the components by interpreting and executing the image generation program 221 expanded in the RAM by the CPU. As a result, as shown in FIG. 9, the image generation device 2 according to the present embodiment operates as a computer including the generation unit 211, the estimation unit 212, the determination unit 213, and the image storage unit 214 as software modules. That is, in the present embodiment, each software module of the image generation device 2 is also realized by the control unit 21 (CPU) as in the learning device 1.

  The generation unit 211 includes the learned generator 50 constructed by the learning apparatus 1 by holding the first learning result data 123. The generation unit 211 supplies the input value 71 indicating the type of the defect to the generator 50 to generate the generated image 73 in which the product R including the defect is copied. In the present embodiment, the generation unit 211 refers to the first learning result data 123 and sets the learned generator 50. Next, the generation unit 211 acquires noise (latent variable) from a predetermined probability distribution. The predetermined probability distribution may be, for example, a Gaussian distribution or the like. Then, the generation unit 211 inputs the acquired noise and the input value 71 to the input layer 511 of the generator 50, and executes the arithmetic processing of the generator 50. As a result, the generation unit 211 acquires the generated image 73 from the output layer 513 of the generator 50.

  The estimation unit 212 includes the learned estimator 52 constructed by the learning device 1 by holding the second learning result data 125. The estimation unit 212 estimates the type and range of the defect included in the product R shown in the generated image 73 by supplying the generated generated image 73 to the estimator 52. In the present embodiment, the estimation unit 212 refers to the second learning result data 125 and sets the learned estimator 52. Then, the estimation unit 212 inputs the generated image 73 generated by the generator 50 to the input layer 531 of the estimator 52, and executes the arithmetic processing of the estimator 52. Thereby, the estimation unit 212 acquires the output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the generated image 73 from the output layer 533 of the estimator 52.

  The determination unit 213 determines whether or not the result of estimating the defect type included in the product R shown in the generated image 73 matches the defect type indicated by the input value 71. When the determination unit 213 determines that the result of estimating the type of defect matches the type of defect indicated by the input value 71, the image storage unit 214 determines the generated image 73 generated and the result of estimating the range of the defect. The data is associated and stored in a predetermined storage area. In the present embodiment, the image storage unit 214 generates, as the boundary information 77, information indicating the result of estimating the defect range. Further, the image storage unit 214 generates, as the label 75, information indicating the type of defect, corresponding to the input value 71. Then, the image storage unit 214 generates a data set by combining the generated image 73, the label 75, and the boundary information 77, and stores the generated data set in a predetermined storage area. Of these, the label 75 may be omitted.

<Estimator generator>
Next, an example of the software configuration of the estimator generation device 3 according to the present embodiment will be described using FIG. 10. FIG. 10 schematically illustrates an example of the software configuration of the estimator generation device 3 according to this embodiment.

  The control unit 31 of the estimator generation device 3 loads the estimator generation program 321 stored in the storage unit 32 into the RAM. Then, the control unit 31 controls the respective components by interpreting and executing the instruction included in the estimator generation program 321 expanded in the RAM by the CPU. As a result, as shown in FIG. 10, the estimator generation device 3 according to the present embodiment is configured as a computer including the data acquisition unit 311, the learning processing unit 312, and the storage processing unit 313 as software modules. That is, in the present embodiment, each software module of the estimator generation device 3 is also realized by the control unit 31 (CPU) as in the learning device 1.

  The data acquisition unit 311 is configured by a combination of a sample image 3221 that shows the product R, a label 3222 that indicates the type of defects that can be included in the product R, and boundary information 3223 that indicates the range of defects that can be included in the product R. A plurality of learning data sets 322 are acquired. The sample image 3221 is used as input data (training data) for machine learning. On the other hand, the label 3222 and the boundary information 3223 are used as correct data (teaching data) for machine learning. That is, the label 3222 and the boundary information 3223 indicate the result (that is, correct answer) of the quality of the product R shown in the corresponding sample image 3221. When the product R shown in the sample image 3221 does not include a defect, each value of the label 3222 and the boundary information 3223 may be appropriately set so as to indicate that the product R does not include a defect.

  Here, the data set generated by the image generation device 2 may be used as the learning data set 322. That is, the sample image 3221, the label 3222, and the boundary information 3223 of at least a part of the learning data set 322 may be the generated image 73, the label 75, and the boundary information 77, respectively. In addition, the data acquisition unit 311 may transmit the learning data set 322 to the learning device 1 and cause the learning data set 322 to be used as the learning data set 60 to cause the learning device 1 to execute the machine learning process. That is, the data acquisition unit 311 uses the sample image 3221 as the learning image 61, the label 3222 as the label 63, and the boundary information 3223 as the boundary information 65 to generate an image corresponding to the sample image 3221. The learning apparatus 1 may be configured to construct the generator 50 for performing the above, and the estimator 52 for estimating the type and range of the defect included in the product R with respect to the generated image. Then, the data acquisition unit 311 may cause the image generator 2 to use the constructed generator 50 and the estimator 52 to cause the image generator 2 to generate a plurality of data sets corresponding to the learning data set 322. . The data acquisition unit 311 can increase the number of learning data sets 322 used for machine learning by receiving the generated plurality of data sets as the learning data sets 322.

  The learning processing unit 312 holds the estimator 80 that is the target of machine learning. The learning processing unit 312 constructs a learned estimator 80 that has acquired the ability to determine the quality of the product R shown in a given image by performing machine learning using each acquired learning data set 322. . In other words, the learning processing unit 312 constructs the estimator 80 trained to output the output values that match the label 3222 and the boundary information 3223 when the sample image 3221 is input for each learning data set 322. The save processing unit 313 saves the constructed information about the learned estimator 80 in a predetermined storage area.

  Note that in the relationship between the learning device 2 and the estimator generation device 3, the learning processing unit 312 may be referred to as a third learning processing unit. The data acquisition unit 111 and the storage processing unit 114 may be referred to as a first data acquisition unit and a first storage processing unit, respectively. The learning data set 60 may be referred to as a first learning data set. Accordingly, the data acquisition unit 311 and the storage processing unit 313 may be referred to as a second data acquisition unit and a second storage processing unit, respectively. The learning data set 322 may be referred to as a second learning data set.

(Estimator)
Next, an example of the configuration of the estimator 80 according to this embodiment will be described. As shown in FIG. 10, the estimator 80 according to the present embodiment is configured by a neural network having a multilayer structure used for so-called deep learning, like the generator 50 and the like. Specifically, the estimator 80 includes an input layer 801, an intermediate layer (hidden layer) 802, and an output layer 803. However, the configuration of the estimator 80 is not limited to such an example, and may be set appropriately according to the embodiment. For example, the estimator 80 may include two or more intermediate layers. The configuration of the estimator 80 may be different from that of the generator 50 and the like.

  The number of neurons (nodes) included in each of the layers 801 to 803 may be set appropriately according to the embodiment. The neurons in the adjacent layers are appropriately connected to each other, and a weight (connection weight) is set for each connection. In the example of FIG. 10, each neuron is connected to all neurons in the adjacent layer. However, the connection of neurons need not be limited to such an example, and may be set appropriately according to the embodiment. A threshold value is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of products of each input and each weight exceeds the threshold value. The weights of the connections between the neurons and the thresholds of the neurons included in the layers 801 to 803 are examples of parameters of the estimator 80 used for the arithmetic processing.

  The learning processing unit 312 inputs the sample image 3221 into the input layer 801 of the estimator 80 for each learning data set 322, and executes the arithmetic processing of the estimator 80. As a result of this arithmetic processing, the learning processing unit 312 determines whether the product R shown in the sample image 3221 is good or bad, in other words, outputs the output value corresponding to the result of estimating the type and range of defects that can be included in the product R. It is acquired from the output layer 803.

  Subsequently, the learning processing unit 312 calculates an error between the acquired output value and each value of the label 3222 and the boundary information 3223. Then, the learning processing unit 312 adjusts the value of the parameter of the estimator 80 for each learning data set 322 so that the sum of the calculated errors becomes small. For example, the learning processing unit 312 causes the value of the parameter of the estimator 80 by the series of processes described above until the sum of the errors between the output value obtained from the output layer 803 and the values of the label 3222 and the boundary information 3223 becomes equal to or less than the threshold value. Repeat the adjustment of. Accordingly, when the sample image 3221 is input to the input layer 801 for each learning data set 322, the learning processing unit 312 outputs an output value that matches the label 3222 and the boundary information 3223 associated with the input sample image 3221. An estimator 80 trained to output from 803 can be constructed. It should be noted that the criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the learning processing unit 312 may repeat the adjustment of the value of the parameter of the estimator 80 by the designated number of times by the series of processes described above.

  After the machine learning process is completed, the storage processing unit 313 configures the structure of the constructed estimator 80 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, each neuron). 3) and third calculation result data 325 indicating the calculation parameter (for example, the weight of connection between neurons, the threshold value of each neuron). Then, the storage processing unit 313 stores the generated third learning result data 325 in a predetermined storage area.

<Inspection device>
Next, an example of the software configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 11 schematically illustrates an example of the software configuration of the inspection device 4 according to the present embodiment.

  The control unit 41 of the inspection device 4 loads the inspection program 421 stored in the storage unit 42 into the RAM. Then, the control unit 41 controls and interprets each component by causing the CPU to interpret and execute the command included in the inspection program 421 expanded in the RAM. As a result, as shown in FIG. 11, the inspection device 4 according to the present embodiment is configured as a computer including the target data acquisition unit 411, the quality determination unit 412, and the output unit 413 as software modules. In the present embodiment, each software module of the inspection device 4 is also realized by the control unit 41 (CPU) like the learning device 1.

  The target data acquisition unit 411 acquires a target image 422 of the product R that is a target of the visual inspection. In the present embodiment, the target data acquisition unit 411 acquires the target image 422 by photographing the product R with the camera CA. The pass / fail judgment unit 412 holds the third learning result data 325, and thus includes the estimator 80 that has been constructed by the estimator generation device 3 but has already been learned. The quality determination unit 412 determines the quality of the product R shown in the target image 422 using the learned estimator 80.

  Specifically, the quality determination unit 412 refers to the third learning result data 325 to set the learned estimator 80. Next, the quality determination unit 412 inputs the acquired target image 422 to the input layer 801 of the estimator 80, and executes the arithmetic processing of the estimator 80. Thereby, the quality determination unit 412 estimates the output value corresponding to the result of determining the quality of the product R shown in the target image 422 (specifically, the result of estimating the type and range of defects that may be included in the product R). From the output layer 803 of the container 80. In this embodiment, obtaining this output value corresponds to determining the quality of the product R. The output unit 413 outputs information related to the result of determining the quality of the product R.

<Other>
Each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 will be described in detail in an operation example described later. In addition, in this embodiment, an example in which each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 is realized by a general-purpose CPU is described. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, regarding the software configurations of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4, depending on the embodiment, omission, replacement, and addition of software modules may be appropriately performed.

§3 Example of operation [Learning device]
Next, an operation example of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 12 shows an example of the processing procedure of the learning device 1 according to the present embodiment. The processing procedure described below is an example of the “learning method” of the present invention. However, the processing procedure described below is merely an example, and each processing may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as the data acquisition unit 111, and displays a learning image 61 showing a product R to be subjected to an appearance inspection, a label 63 indicating a defect type included in the product R, and a defect range. A plurality of learning data sets 60 each configured by a combination of the boundary information 65 shown are acquired.

  The method of acquiring the learning data set 60 does not have to be particularly limited and may be appropriately determined according to the embodiment. For example, a camera and a defective product R are prepared, and the product R is photographed by the camera so that the defective portion can be seen. Thereby, the learning image 61 can be generated. Next, the label 63 indicating the type of defect included in the product R shown in the learning image 61 and the boundary information 65 indicating the range of the defect are appropriately generated. For example, the label 63 may be generated by selecting a defect that appears in the target learning image 61 from the list of defect types. The boundary information 65 may be generated by designating the bounding box so as to include the defect range on the learning image 61. Then, the generated label 63 and boundary information 65 are associated with the corresponding learning image 61. Thereby, each learning data set 60 can be generated.

  The generation of the learning data set 60 may be automatically performed by the operation of the computer or may be manually performed by the operation of the operator. Further, the information processing for generating the learning data set 60 may be executed in the learning device 1 or may be executed in a computer other than the learning device 1.

  When the learning device 1 generates the learning data sets 60, the control unit 11 acquires a plurality of learning data sets 60 by executing the information processing automatically or manually by an operator's operation. On the other hand, when the learning data set 60 is generated by another computer, the control unit 11 acquires the plurality of learning data sets 60 generated by the other computer via, for example, the network or the storage medium 91. The learning device 1 may generate a part of the learning data set 60, and another computer may generate the remaining learning data set 60.

  The number of learning data sets 60 to be acquired does not have to be particularly limited, and may be appropriately determined according to the embodiment. When the plurality of learning data sets 60 are acquired, the control unit 11 advances the processing to the next step S102.

(Steps S102 and S103)
In step S102, the control unit 11 operates as the first learning processing unit 112 and executes the first machine learning, so that the label 63 input to the input of the label 63 included in each learning data set 60 is input. Construct a generator 50 trained to generate an image corresponding to the training image 61 associated with 63. In step S103, the control unit 11 operates as the second learning processing unit 113 and executes the second machine learning to input the learning images 61 included in each learning data set 60. Constructs an estimator 52 trained to estimate the type and range of defects included in the product R shown in the input learning image 61 so as to match the label 63 and the boundary information 65 associated with the learning image 61. To do.

  The order of processing steps S102 and S103 may not be particularly limited and may be appropriately determined according to the embodiment. The processes of steps S102 and S103 may be executed simultaneously or separately. In the present embodiment, the generator 50 and the estimator 52 constitute the learning network 500 together with the encoder 56 and the discriminator 54. Therefore, in steps S102 and S103, the control unit 11 trains the generator 50 and the estimator 52 by performing the machine learning of the learning network 500.

  Specifically, first, the control unit 11 prepares the learning network 500 to be processed. The configuration of the learning network 500 to be prepared, the initial value of the connection weight between neurons, and the initial value of the threshold value of each neuron may be given by a template or may be given by an operator's input. Further, when performing re-learning, the control unit 11 may prepare the learning network 500 based on the learning result data obtained by performing the past machine learning.

  Next, the control unit 11 uses the learning image 61 of each learning data set 60 as input data and the corresponding label 63 and boundary information 65 as teacher data to execute the learning process of the estimator 52. The control unit 11 uses the learning image 61 and the label 63 of each learning data set 60 as input data, and uses the value derived from the prior distribution as teacher data to execute the learning process of the encoder 56. The control unit 11 inputs the output value obtained from the encoder 56 and the label 63 for each learning data set 60 to the generator 50, and executes the arithmetic processing of the generator 50. Thereby, the control unit 11 acquires each image 69 from the generator 50. The control unit 11 uses each image 69 and each learning image 61 as input data, and uses the origin of each image (61, 69) as teacher data to execute the learning process of the discriminator 54. For each learning data set 60, the control unit 11 uses the output value and the label 63 obtained from the encoder 56 as input data, and uses the discriminator 54 to make an erroneous determination as teacher data, and uses the output value and the label 63 of the generator 50. Execute the learning process. For each learning data set 60, the control unit 11 uses the output value and the label 63 obtained from the encoder 56 as input data, and uses the estimator 52 to correctly estimate the defect type as teacher data, The learning process of the generator 50 is executed. For each learning data set 60, the control unit 11 uses the learning image 61 and the label 63 as input data and the learning image 61 as teacher data to execute the learning process of the encoder 56 and the generator 50. A stochastic gradient descent method or the like may be used for these learning processes.

For example, the control unit 11 inputs the learning image 61 included in each learning data set 60 to the input layer 531 of the estimator 52, and determines the firing of each neuron included in each layer 531 to 533 in order from the input side. Thereby, the control unit 11 acquires, from the output layer 533, an output value corresponding to the result of estimating the type and range of the defect included in the product R shown in each learning image 61. The control unit 11 calculates an error (L _C ) between the output value corresponding to the result of estimating the defect type and the value of the label 63. The control unit 11 also calculates an error (L _BB ) between the output value corresponding to the result of estimating the defect range and the value of the boundary information 65.

Next, the control unit 11 inputs the learning image 61 and the label 63 included in each learning data set 60 to the input layer 571 of the encoder 56, and determines firing of each neuron included in each layer 571 to 573 in order from the input side. To do. As a result, the control unit 11 acquires the output value (latent variable) corresponding to the result of deriving some kind of characteristic amount from the output layer 573 of the encoder 56. The control unit 11 calculates the error (L _KL ) between this output value and the value derived from the prior distribution. The prior distribution may be, for example, a Gaussian distribution or the like.

  Next, the control unit 11 inputs the output value obtained from the encoder 56 and the corresponding label 63 to the input layer 511 of the generator 50, and determines the firing determination of each neuron included in each layer 511 to 513 in order from the input side. To do. Thereby, the control unit 11 acquires each image 69 from the output layer 513 as a result of generating the image corresponding to each learning image 61 from the label 63.

Subsequently, the control unit 11 inputs each image 69 generated by the generator 50 to the input layer 551 of the discriminator 54, and performs firing determination of each neuron included in each layer 551 to 553 in order from the input side. Thereby, the control unit 11 acquires from the output layer 553 an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60 (that is, the learning image 61 itself). In this scene, since each image 69 generated by the generator 50 is an input image, it is the correct answer for the discriminator 54 to determine “false”. The control unit 11 calculates an error (L _D ) between the output value obtained from the output layer 553 and the correct answer for each image 69 generated by the generator 50.

Further, the control unit 11 inputs the learning image 61 included in each learning data set 60 to the input layer 551 of the discriminator 54, and sequentially determines the firing of each neuron included in each layer 551 to 553 from the input side. Thereby, the control unit 11 acquires, from the output layer 553, an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning data set 60. In this scene, since the learning image 61 itself is the input image, it is the correct answer for the discriminator 54 to determine “true”. The control unit 11 calculates the error (L _D ) between the output value obtained from the output layer 553 and the correct answer for each learning data set 60.

Next, the control unit 11 generates an image that causes the discrimination by the discriminator 54 to be erroneous with respect to the output value obtained by inputting each image 69 generated by the generator 50 into the discriminator 54. Calculate the error (L _GD ) for training vessel 50. Specifically, it is a correct answer that the output value obtained from the discriminator 54 corresponds to “true” when each image 69 generated by the generator 50 is input to the discriminator 54. The control unit 11 calculates an error (L _GD ) between the output value obtained from the output layer 553 through a series of processes and the correct answer (that is, “true”) for each learning data set 60.

Next, the control unit 11 inputs each image 69 generated by the generator 50 to the input layer 531 of the estimator 52, and performs firing determination of each neuron included in each layer 531 to 533 in order from the input side. Thereby, the control unit 11 acquires the output value corresponding to the result of estimating the type of the defect included in the product R shown in each image 69 from the output layer 533 of the estimator 52. For each learning data set 60, the control unit 11 calculates an error (L _GC ) between the output value corresponding to the result of estimating the defect type by a series of processes and the value of the corresponding label 63.

Next, the control unit 11 calculates, for each learning data set 60, an error for training the generator 50 to function as a decoder that restores the learning image 61 from the output of the encoder 56. Specifically, the control unit 11 calculates, for each learning data set 60, an error (L _G ) between the image 69 generated by the generator 50 and the corresponding learning image 61.

Then, the control unit 11, by the error backpropagation (Back propagation), the error calculated _{(L C, L BB, L} KL, L D, L GD, L GC, L G) with, each neuron The error of each of the connection weight and the threshold value of each neuron is calculated. That is, the control unit 11 calculates the weight of the coupling between the neurons of the estimator 52 and the error of the threshold value of each neuron by using each calculated error (L _C , L _BB ). The control unit 11 uses the calculated error (L _D ) to calculate the weight of the connection between the neurons of the discriminator 54 and the error of the threshold value of each neuron. The control unit 11 uses each calculated error (L _G , L _GD , L _GC ) to calculate the weight of the coupling between the neurons of the generator 50 and the error of each threshold value of each neuron. The control unit 11 uses the calculated errors (L _KL , L _G ) to calculate the weight of the coupling between the neurons of the encoder 56 and the error of the threshold of each neuron. The control unit 11 updates the weight of the coupling between the neurons in each of the estimator 52, the discriminator 54, the generator 50, and the encoder 56 and the value of each threshold value of each neuron based on each calculated error.

Control unit 11, the error calculated _{(L C, L BB, L} KL, L D, L GD, L GC, L G) to the sum of less than or equal to the threshold value, the above-described series of processes, each parameter Repeat the value adjustment. Each threshold may be appropriately set according to the embodiment. As a result, the control unit 11 includes, for each learning data set 60, the product R shown in the input learning image 61 so that the input of the learning image 61 matches the corresponding label 63 and boundary information 65. An estimator 52 may be constructed that is trained to estimate the type and range of defects that will be exposed. The control unit 11 can construct a discriminator 54 that has been trained so as to be able to appropriately discriminate whether the input image input is from the generator 50 or the learning data set 60. The control unit 11 has been trained so that the discriminator 54 makes a discrimination error, the estimator 52 estimates the defect type so as to match the corresponding label 63, and can generate the image 69 that can correspond to the learning image 61. The generator 50 can be constructed. The control unit 11 can cause the generator 50 to generate an image corresponding to the learning image 61, which is an output value according to the prior distribution with respect to the inputs of the learning image 61 and the label 63 for each learning data set 60. An encoder 56 trained to output various output values can be constructed. When the machine learning process is completed, the control unit 11 advances the process to the next step S104. It should be noted that the criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the control unit 11 may repeat the adjustment of the value of each parameter by the specified number of times through the series of processes described above.

(Step S104)
In step S104, the control unit 11 operates as the storage processing unit 114 and stores the information regarding the constructed learned generator 50 and the information regarding the constructed learned estimator 52 in a predetermined storage area.

  In the present embodiment, the control unit 11 generates, as the first learning result data 123, information indicating the configuration and parameters of the learned generator 50 constructed in step S102. The control unit 11 also generates information indicating the configuration and parameters of the learned estimator 52 constructed in step S103 as the second learning result data 125. Then, the control unit 11 saves the generated first learning result data 123 and second learning result data 125 in a predetermined storage area. The predetermined storage area may be, for example, the RAM in the control unit 11, the storage unit 12, the external storage device, the storage medium, or a combination thereof.

  The storage medium may be, for example, a CD, a DVD, or the like, and the control unit 11 may store the first learning result data 123 and the second learning result data 125 in the storage medium via the drive 16. The external storage device may be, for example, a data server such as NAS (Network Attached Storage). In this case, the control unit 11 may store the first learning result data 123 and the second learning result data 125 in the data server via the network. The external storage device may be, for example, an external storage device connected to the learning device 1.

  The handling of the encoder 56 and the discriminator 54 is not particularly limited, and may be appropriately determined according to the embodiment. For example, the control unit 11 also generates information indicating the configuration and parameters of the encoder 56 and the discriminator 54, as the learning result data, as with the generator 50 and the estimator 52, and sets the generated learning result data to a predetermined value. May be stored in the storage area. Alternatively, the control unit 11 may omit the process of generating and storing the learning result data.

  When the first learning result data 123 and the second learning result data 125 are saved, the control unit 11 ends the process according to this operation example. After constructing the learned generator 50 and the estimator 52, the control unit 11 may transfer the generated first learning result data 123 and second learning result data 125 to the image generating device 2 at arbitrary timing. . The image generation device 2 may acquire the first learning result data 123 and the second learning result data 125 by accepting the transfer from the learning device 1. The image generation device 2 may acquire the first learning result data 123 and the second learning result data 125 by accessing the learning device 1 or the data server. Alternatively, the first learning result data 123 and the second learning result data 125 may be incorporated in the image generating device 2 in advance.

  Moreover, the control unit 11 may periodically update at least one of the first learning result data 123 and the second learning result data 125 by periodically repeating the processes of steps S101 to S104. When this is repeated, the learning data set 60 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 11 transfers the updated first learning result data 123 and / or the second learning result data 125 to the image generating apparatus 2 each time the machine learning is executed, so that the image generating apparatus 2 holds the same. At least one of the first learning result data 123 and the second learning result data 125 may be updated periodically.

[Image generation device]
Next, an operation example of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 13 shows an example of the processing procedure of the image generating apparatus 2 according to this embodiment. However, the processing procedure described below is merely an example, and each processing may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S201)
In step S201, the control unit 21 operates as the generation unit 211, and uses the learned generator 50 constructed by the learning device 1 to generate the generated image 73 of the product R including the designated defect. To do.

  In the present embodiment, the control unit 21 refers to the first learning result data 123 and sets the learned generator 50. Next, the control unit 21 acquires noise (latent variable) from a predetermined probability distribution. The predetermined probability distribution may be, for example, a Gaussian distribution or the like. Further, the control unit 21 appropriately acquires the input value 71 indicating the type of defect. For example, information (not shown) indicating a list of defect types may be stored in the image generation program 221, the storage unit 22, the external storage device, the storage medium, or the like. In this case, the control unit 21 acquires the information indicating the list of defect types, refers to the acquired information, and selects the defect to be designated as an image generation target from the list, thereby setting the input value 71. You may get it. Further, for example, the control unit 21 may randomly select the input value 71 from a predetermined numerical range. Then, the control unit 21 inputs the acquired noise and the input value 71 to the input layer 511 of the generator 50, and determines the firing of each neuron included in each of the layers 511 to 513 in order from the input side. Thereby, the control unit 21 acquires the generated image 73 from the output layer 513 of the generator 50. When the generation of the generated image 73 is completed, the control unit 21 advances the processing to the next step S202.

(Step S202)
In step S202, the control unit 21 operates as the estimation unit 212 and uses the learned estimator 52 constructed by the learning device 1 to use the type and range of defects included in the product R that can be shown in the generated image 73. To estimate.

  In the present embodiment, the control unit 21 sets the learned estimator 52 with reference to the second learning result data 125. Then, the control unit 21 inputs the generated image 73 generated by the generator 50 to the input layer 531 of the estimator 52, and performs firing determination of each neuron included in each layer 531 to 533 in order from the input side. Thereby, the control unit 21 acquires the output value corresponding to the result of estimating the type and range of the defect included in the product R shown in the generated image 73 from the output layer 533 of the estimator 52. When the estimation of the defect type and range is completed, the control unit 21 advances the process to the next step S203.

(Steps S203 and S204)
In step S <b> 203, the control unit 21 operates as the determination unit 213 and determines whether or not the result of estimating the defect type included in the product R shown in the generated image 73 matches the defect type indicated by the input value 71. judge. In step S204, the control unit 21 processes the conditional branch according to the determination result of step S203.

  The fact that the result of estimating the defect type by the estimator 52 matches the input value 71 means that the image (generated image 73) showing the product R including the specified defect can be generated by the generator 50. Indicates that. Therefore, when it is determined that the result of estimating the defect type matches the defect type indicated by the input value 71, the control unit 21 advances the process to the next step S205 for storing the generated image 73.

  On the other hand, the fact that the result of estimating the defect type by the estimator 52 does not match the input value 71 means that the image of the product R including the specified defect cannot be generated by the generator 50. Show. Therefore, when it is determined that the result of estimating the defect type does not match the defect type indicated by the input value 71, the control unit 21 omits the process of the next step S205 and ends the process according to the present operation example. To do.

  In this case, the control unit 21 may discard the generated image 73 generated in step S201. Alternatively, the control unit 21 may accept designation of at least one of the defect type and the range for the generated generated image 73. Then, the control unit 21 may generate a data set by associating information indicating at least one of the type and range of the designated defect with the generated image 73, and may store the generated data set in a predetermined storage area. . The predetermined storage area may be, for example, a RAM in the control unit 21, a storage unit 22, an external storage device, a storage medium, or a combination thereof.

(Step S205)
In step S205, the control unit 21 operates as the image storage unit 214 and stores the generated image 73 and the result of estimating the range of the defect in a predetermined storage area in association with each other. In the present embodiment, the control unit 21 generates, as the boundary information 77, information indicating the result of estimating the defect range. Further, the control unit 21 generates information indicating the type of defect as the label 75, corresponding to the input value 71. Then, the control unit 21 generates a data set by combining the generated image 73, the label 75, and the boundary information 77, and stores the generated data set in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 21, a storage unit 22, an external storage device, a storage medium, or a combination thereof. The label 75 may be omitted in this series of processes. That is, the control unit 21 may omit the generation of the label 75, generate a data set by combining the generated image 73 and the boundary information 77, and store the generated data set in a predetermined storage area.

  As described above, the control unit 21 ends the process according to this operation example. The control unit 21 may generate a plurality of data sets by repeatedly executing the above steps S201 to S205. The number of data sets to be generated does not have to be particularly limited and may be appropriately determined according to the embodiment. Further, the control unit 21 may transfer the generated data set to the estimator generation device 3 in order to use the generated data set as the learning data set 322.

[Estimator generator]
Next, an operation example of the estimator generation device 3 according to the present embodiment will be described using FIG. 14. FIG. 14 is a flowchart illustrating an example of the processing procedure of the estimator generation device 3 according to this embodiment. However, the processing procedure described below is merely an example, and each processing may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S301)
In step S301, the control unit 31 operates as the data acquisition unit 311 and a sample image 3221 showing the product R, a label 3222 indicating the type of defect that can be included in the product R, and a range of defects that can be included in the product R. A plurality of learning data sets 322 each constituted by a combination of boundary information 3223 indicating

  The method for acquiring the learning data set 322 may not be particularly limited and may be appropriately determined according to the embodiment. For example, a camera and a product R are prepared, and a defective or non-defective product R is photographed by the camera. Thereby, the sample image 3221 can be generated. Next, a label 3222 indicating the type of defect that can be included in the product R shown in the sample image 3221 and boundary information 3223 indicating the range of the defect are appropriately generated. The method of generating the label 3222 and the boundary information 3223 may be the same as the method of generating the label 63 and the boundary information 65. The boundary information 3223 may be appropriately generated by the bounding box so as to indicate the range of the defect. Further, when the product R shown in the sample image 3221 does not include a defect, each value of the label 3222 and the boundary information 3223 may be appropriately set so as to indicate that the product R does not include a defect. Then, the generated label 3222 and boundary information 3223 are associated with the corresponding sample image 3221. Thereby, each learning data set 322 can be generated.

  The generation of the learning data set 322 may be automatically performed by the operation of the computer or may be manually performed by the operation of the operator. The information processing for generating the learning data set 322 may be executed by the estimator generation device 3 or may be performed by a computer other than the estimator generation device 3.

  When the estimator generation device 3 generates the learning data sets 322, the control unit 31 acquires the plurality of learning data sets 322 by executing the above information processing automatically or manually by the operation of the operator. On the other hand, when the learning data set 322 is generated by another computer, the control unit 31 acquires the plurality of learning data sets 322 generated by the other computer via, for example, the network, the storage medium 93, or the like. The estimator generation device 3 may generate a part of the learning data set 322, and the remaining learning data set 322 may be generated by another computer.

  Here, at least a part of the acquired learning data set 322 may be a data set generated by the image generation device 2. That is, the sample image 3221, the label 3222, and the boundary information 3223 of at least a part of the learning data set 322 may be the generated image 73, the label 75, and the boundary information 77, respectively. The control unit 31 may acquire the data set generated by the image generation device 2 as the learning data set 322 via the network, the storage medium 93, or the like.

  The number of learning data sets 322 to be acquired does not have to be particularly limited, and may be appropriately determined according to the embodiment. After acquiring the plurality of learning data sets 322, the control unit 31 advances the processing to the next step S302.

(Step S302)
In step S302, the control unit 31 operates as the learning processing unit 312, and uses the plurality of learning data sets 322 to perform the machine learning of the estimator 80. In this machine learning, when the sample image 3221 is input to the input layer 801 for each learning data set 322, the control unit 31 outputs an output value that matches the corresponding label 3222 and boundary information 3223 from the output layer 803. Train the estimator 80. As a result, the control unit 31 builds a learned estimator 80 that has acquired the ability to determine the quality of the product R.

  This machine learning process may be basically executed in the same manner as the machine learning process by the learning device 1. That is, the control unit 31 prepares the estimator 80 to be processed. The configuration of the estimator 80 to be prepared, the initial value of the connection weight between the neurons, and the initial value of the threshold value of each neuron may be given by a template or may be given by an operator's input. When performing re-learning, the control unit 31 may prepare the estimator 80 based on the learning result data obtained by performing the past machine learning.

  Next, the control unit 31 uses the sample image 3221 included in each learning data set 322 acquired in step S301 as input data, uses the corresponding label 3222 and boundary information 3223 as teacher data, and uses the estimator 80. The learning process of is executed. A stochastic gradient descent method or the like may be used for this learning processing.

  For example, the control unit 31 inputs the sample image 3221 to the input layer 801 for each learning data set 322, and determines the firing of each neuron included in each layer 801 to 803 in order from the input side. As a result, the control unit 31 acquires from the output layer 803 an output value corresponding to the result of estimating the type and range of defects that can be included in the product R shown in the sample image 3221. Next, the control unit 31 calculates an error between the output value corresponding to each result and each value of the label 3222 and the boundary information 3223. Subsequently, the control unit 31 uses the error of the calculated output value by the error back propagation method to calculate the weight of the coupling between the neurons in the estimator 80 and the error of the threshold value of each neuron. Then, the control unit 31 updates the weight of the coupling between the neurons in the estimator 80 and the value of the threshold value of each neuron based on each calculated error.

  The control unit 31 repeats the series of processes described above, and when the sample image 3221 is input for each learning data set 322, the estimator 80 outputs the output value that matches the corresponding label 3222 and the boundary information 3223. Adjust the value of the parameter. For example, the control unit 31 performs estimation by the above series of processes until the sum of the errors between the output value obtained from the output layer 803 and the respective values of the label 3222 and the boundary information 3223 becomes equal to or less than the threshold value for each learning data set 322. The adjustment of the parameter values of the container 80 is repeated. The threshold value may be appropriately set according to the embodiment. Accordingly, the control unit 31 is trained to output the output values that match the corresponding label 3222 and the boundary information 3223 from the output layer 803 when the sample image 3221 is input to the input layer 801 for each learning data set 322. The estimator 80 can be constructed. When the machine learning process of the estimator 80 is completed, the control unit 31 advances the process to the next step S303. It should be noted that the criterion for repeating the parameter adjustment is not limited to the example using such a threshold value. For example, the control unit 31 may repeat the adjustment of the value of the parameter of the estimator 80 by the above-described series of processes for the designated number of times.

(Step S303)
In step S303, the control unit 31 operates as the storage processing unit 313, and stores the constructed information about the learned estimator 80 in a predetermined storage area. In the present embodiment, the control unit 31 generates information indicating the configuration and parameters of the learned estimator 80 constructed in step S302 as the third learning result data 325. Then, the control unit 31 saves the generated third learning result data 325 in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 31, a storage unit 32, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof. As a result, the control unit 31 ends the process according to this operation example.

  In addition, after constructing the learned estimator 80, the control unit 31 may transfer the generated third learning result data 325 to the inspection device 4 at an arbitrary timing. The inspection apparatus 4 may acquire the third learning result data 325 by accepting the transfer from the estimator generation apparatus 3, or may access the third learning result data 325 by accessing the estimator generation apparatus 3 or the data server. You may get it. The third learning result data 325 may be incorporated in the inspection device 4 in advance.

  In addition, the control unit 31 may periodically update the third learning result data 325 by repeating the processes of steps S301 to S303. When repeating this, the learning data set 322 may be appropriately changed, modified, added, deleted, or the like. Then, the control unit 31 transfers the updated third learning result data 325 to the inspection device 4 each time the machine learning is performed, thereby periodically updating the third learning result data 325 held by the inspection device 4. Good.

  Furthermore, the control unit 31 may evaluate the determination performance of the constructed estimator 80 using the evaluation data set. The evaluation data set can be configured in the same manner as each learning data set 322 described above. That is, the evaluation data set may be configured by a combination of a sample image of the product R, a label indicating the type of defects that can be included in the product R, and boundary information that indicates the range of defects that can be included in the product R. . The control unit 31 uses the estimator 80 to determine the quality of the product R shown in the sample image of the evaluation data set, as in step S402 described below. The control unit 31 can evaluate the determination performance of the estimator 80 by collating the determination result with the correct answer indicated by the label and the boundary information.

  When the determination performance of the estimator 80 is less than or equal to a predetermined reference (for example, the correct answer rate is less than or equal to a threshold), the control unit 31 selects one or more learning data sets selected from the plurality of learning data sets 322. 322 may be transmitted to the learning device 1. Next, the control unit 31 may cause the transmitted learning data set 322 to be used as the learning data set 60 and cause the learning device 1 to execute the machine learning process of the generator 50 and the estimator 52. As a result, the learning device 1 includes the generator 50 for generating an image corresponding to the sample image 3221 and the estimator 52 for estimating the type and range of the defect included in the product R with respect to the generated image. Can be built. The controller 31 may cause the learning device 1 to transfer the learned generator 50 and the estimator 52 to the image generating device 2. Then, the control unit 31 uses the learned generator 50 and the estimator 52 to generate one or a plurality of data sets each configured by the combination of the generated image 73, the label 75, and the boundary information 77. May be executed by the image generating device 2.

  In response to this, the control unit 31 may acquire one or a plurality of data sets generated by the image generation device 2 as the learning data set 322. Then, the control unit 31 may add the acquired one or more learning data sets 322 to the original learning data group. Accordingly, the control unit 31 can increase the number of learning data sets 322 used for machine learning. The control unit 31 may perform the machine learning of the estimator 80 again using this new learning data group. By this series of re-learning processing, the determination performance of the constructed estimator 80 that has been learned can be improved.

[Inspection equipment]
Next, an operation example of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart illustrating an example of the processing procedure of the inspection device 4 according to the present embodiment. However, the processing procedure described below is merely an example, and each processing may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S401)
In step S401, the control unit 41 operates as the target data acquisition unit 411 and acquires the target image 422 of the product R that is the target of the visual inspection. In the present embodiment, the inspection device 4 is connected to the camera CA via the external interface 47. Therefore, the control unit 41 acquires the target image 422 from the camera CA. The target image 422 may be moving image data or still image data. When the target image 422 is acquired, the control unit 41 advances the processing to the next step S402.

  However, the route for acquiring the target image 422 may not be limited to such an example, and may be appropriately selected according to the embodiment. For example, another information processing device different from the inspection device 4 may be connected to the camera CA. In this case, the control unit 41 may acquire the target image 422 via another information processing device.

(Step S402)
In step S402, the control unit 41 operates as the quality determination unit 412 and determines the quality of the product R shown in the target image 422 using the learned estimator 80.

  Specifically, the control unit 41 sets the learned estimator 80 with reference to the third learning result data 325. Next, the quality determination unit 412 inputs the acquired target image 422 to the input layer 801 of the estimator 80, and determines the firing of each neuron included in each layer 801 to 803 in order from the input side. Accordingly, the control unit 41 acquires, from the output layer 803 of the estimator 80, an output value corresponding to the result of estimating the types and ranges of defects that can be included in the product R shown in the target image 422.

  Thereby, the control unit 41 determines the quality of the product R shown in the target image 422 based on the output value acquired from the estimator 80. When the output value acquired from the estimator 80 indicates that the product R includes a defect, the control unit 41 can determine from the output value that the product R is not a good product (that is, includes a defect). . In this case, the control unit 41 can specify the type and range of the defect from the output value acquired from the estimator 80. The extent of the defect is indicated by the bounding box. On the other hand, when the output value acquired from the estimator 80 indicates that the product R does not include a defect, the control unit 41 determines from the output value that the product R is a good product (that is, does not include a defect). can do.

  Note that this pass / fail determination may be appropriately performed according to the output format of the estimator 80. For example, the estimator 80 may be configured to output the probability of occurrence of a defect according to each type of defect. In this case, the control unit 41 can determine the quality of the product R by comparing the output value of the estimator 80 and the threshold value. That is, when there is an output value that exceeds (or is greater than) the threshold value, the control unit 41 determines that the product R is not a non-defective product and that the product R contains a defect of the type corresponding to the output value. be able to. On the other hand, if not, the control unit 41 can determine that the product R is a non-defective product.

  Further, for example, the estimator 80 may be configured to output the class corresponding to the type of defect (including not including the defect). In this case, the inspection device 4 may store reference information (not shown) in a table format or the like in which the output value obtained from the estimator 80 and the defect type are associated with each other in the storage unit 42. In response to this, the control unit 41 can determine the quality of the product R shown in the target image 422 according to the output value obtained from the estimator 80 by referring to the reference information.

  As described above, the control unit 41 can determine the quality of the product R shown in the target image 422 by using the estimator 80. When the determination of the quality of the product R is completed, the control unit 41 advances the processing to the next step S403.

(Step S403)
In step S403, the control unit 41 operates as the output unit 413, and outputs the result of determining the quality of the product R in step S402.

  The output format of the result of determining the quality of the product R is not particularly limited and may be appropriately selected according to the embodiment. For example, the control unit 41 may directly output the result of the quality determination of the product R to the output device 45. Further, when it is determined in step S402 that the product R has a defect, the control unit 41 may issue a warning for notifying that the defect is found as the output process of step S403. At this time, the control unit 41 can indicate the range in which the found defect exists by a bounding box.

  Further, the control unit 41 may execute a predetermined control process according to the result of the quality determination of the product R as the output process of step S403. As a specific example, the inspection device 4 may be connected to a manufacturing line that conveys products. In this case, when it is determined that the product R has a defect, the control unit 41 performs a process of transmitting to the manufacturing line a command to convey the defective product R through a route different from that of the product having no defect (step 403). May be performed as the output processing of.

  When the output process of the result of determining the quality of the product R is completed, the control unit 41 ends the process according to this operation example. The control unit 41 may execute a series of processes of steps S401 to S403 each time the product R conveyed on the manufacturing line enters the photographing range of the camera CA. Thereby, the inspection device 4 can perform the appearance inspection of the product R conveyed on the manufacturing line.

[Characteristic]
As described above, the learning device 1 according to the present embodiment constructs the generator 50 trained to generate the image corresponding to the learning image 61 by the processing of steps S102 and S103, and also appears in the learning image 61. An estimator 52 trained to estimate the defect type and range can be constructed. Of these, by using the generator 50, it is possible to automatically generate a sample image in which a product including the designated defect can be captured. In addition, by using the estimator 52, the range of defects included in the product R shown in the generated sample image is estimated, and information indicating the estimated range of defects is automatically added to the sample image as boundary information. can do. Therefore, according to the present embodiment, when preparing the learning data, it is possible to omit the trouble of manually specifying the defect range in the sample image. Therefore, it is possible to reduce the cost required to prepare learning data used for machine learning for acquiring the ability to detect the range of defects.

  Further, in the image generation device 2 according to the present embodiment, the defect range is manually designated by using the generator 50 and the estimator 52 constructed by the learning device 1 through the series of processes of steps S201 to S205. It is possible to generate a data set that can be used for machine learning to acquire the ability to detect the range of defects while omitting the work to be performed. In addition, in the present embodiment, the data set generated by the image generation device 2 can be used as the learning data set 322. As a result, in the estimator generation device 3 according to the present embodiment, the cost of collecting the learning data set 322 in step S301 can be reduced. Furthermore, in the present embodiment, the number of learning data sets 322 used for machine learning of the estimator 80 can be increased by causing the image generation device 2 to generate a data set. As a result, in the inspection device 4 according to the present embodiment, the accuracy with which the quality of the product R is determined can be increased.

§4 Modifications The embodiments of the present invention have been described in detail above, but the above description is merely an example of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. For example, the following changes are possible. In addition, below, the same code | symbol is used about the same component as the said embodiment, and the description about the same point as the said embodiment was abbreviate | omitted suitably. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80 each use a so-called multilayer fully-connected neural network. However, the structures and types of the neural networks forming the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80 may not be limited to such an example, and may depend on the embodiment. May be appropriately selected. For example, a convolutional neural network may be used for each of the generator 50, the estimator 52, the discriminator 54, the encoder 56, and the estimator 80.

  Further, in the above-described embodiment, a neural network is used as a learning model forming the generator 50, the estimator 52, and the estimator 80. However, the types of the learning models constituting the generator 50, the estimator 52, and the estimator 80 are not particularly limited as long as the machine learning of the image can be implemented, and the learning model is not limited to the embodiment. May be appropriately selected.

<4.2>
In the above embodiment, the first learning result data 123 includes information indicating the configuration of the generator 50. The second learning result data 125 includes information indicating the configuration of the estimator 52. The third learning result data 325 includes information indicating the configuration of the estimator 80. However, the structure of each learning result data (123, 125, 325) is not limited to such an example, and may be appropriately determined according to the embodiment. For example, when the configuration of the neural network used for each is common to each device, each learning result data (123, 125, 325) does not need to include information indicating the configuration of the neural network. Good.

<4.3>
In the above embodiment, the boundary information (65, 77, 3223) indicates the range of the defect by the bounding box. However, the method of indicating the defect range is not limited to such an example, and may be appropriately selected according to the embodiment. Defect extents may be indicated in formats other than bounding boxes.

<4.4>
In the above-described embodiment, the estimator generation device 3 uses the label 3222 and the boundary information 3223 as teacher data to estimate the type and range of defects that can be included in the product R shown in the image. The container 80 is constructed. However, the ability learned by the estimator 80 is not limited to such an example, and may be appropriately determined according to the embodiment. For example, the label 3222 may be omitted from this series of processes. Thereby, the estimator generation device 3 can construct the estimator 80 trained to estimate only the range of defects.

<4.5>
In the learning network 500 according to the above embodiment, the output of the encoder 56 is connected to the input of the generator 50, and the output of the generator 50 is connected to the input of the estimator 52 and the input of the discriminator 54. Then, in the process of machine learning, the error is calculated as _{(L C, L BB, L} KL, L D, L GD, L GC, L G) is the sum of the decreases, the learning network 500 for each parameter The value is adjusted. However, the configuration of the learning network 500 is not limited to such an example, and may be set appropriately according to the embodiment. The processing of each machine learning may be appropriately selected according to the configurations of the generator 50 and the estimator 52. The generator 50 and the estimator 52 may each be trained independently.

  FIG. 16 schematically illustrates an example of the software configuration of the learning device 1 that holds the learning network 500A according to the present modification. The learning device 1 according to the present modification is configured similarly to the above embodiment, except that the learning network 500 according to the above embodiment is replaced with the learning network 500A.

  The learning network 500A according to this modification includes a generator 50A and a discriminator 54A. The output of the generator 50A is connected to the input of the discriminator 54A. On the other hand, the estimator 52A is not connected to the generator 50A but is disconnected from the learning network 500A. In this modification, the estimator 52A is individually trained.

  Next, an example of a machine learning process of the generator 50A and the estimator 52A will be described with reference to FIGS. 17A to 17C. 17A and 17B schematically illustrate an example of a machine learning process of the learning network 500A including the generator 50A. FIG. 17C schematically illustrates an example of the machine learning process of the estimator 52A.

  In this modification, the first learning processing unit 112 performs machine learning of the learning network 500A. Performing machine learning of the learning network 500A includes alternating first training steps for training the discriminator 54A and second training steps for training the generator 50A. The configurations of the generator 50A and the discriminator 54A according to the present modification may be the same as those of the generator 50 and the discriminator 54 according to the above embodiment. The first training step and the second training step according to the present modification are substantially the same as the first training step and the second training step according to the above embodiment, respectively. That is, in the first training step, the first learning processing unit 112 learns whether the input image input to the discriminator 54A is the image 69 generated by the generator 50A or is obtained from a plurality of learning data sets 60. The discriminator 54A is trained so as to discriminate whether or not it is the image 61. In the second training step, the first learning processing unit 112 trains the generator 50A so that the image 69 generated by the generator 50A causes the discrimination by the discriminator 54A to be erroneous.

  Specifically, as shown in FIG. 17A, in the first training step, the first learning processing unit 112 extracts noise (latent variable) from a predetermined probability distribution and combines the extracted noise and each label 63. To generate a plurality of first data sets. The predetermined probability distribution may be, for example, a Gaussian distribution, a uniform distribution, or the like. An arbitrary probability distribution may be used to extract noise. Subsequently, the first learning processing unit 112 inputs each first data set (each label 63 and noise) to the generator 50A, and executes the arithmetic processing of the generator 50A. As a result, the first learning processing unit 112 acquires an output (image 69) corresponding to the result of generating an image from each label 63 from the generator 50A. The first learning processing unit 112 generates a plurality of second data sets by combining the generated images 69 and the labels 63.

Next, the first learning processing unit 112 inputs each second data set (each image 69 and each label 63 generated by the generator 50A) to the discriminator 54A, and executes the arithmetic processing of the discriminator 54A. Accordingly, the first learning processing unit 112 acquires, from the discriminator 54A, an output value corresponding to the result of discriminating whether the input image is derived from the generator 50A or the learning data set 60. In this scene, since each image 69 generated by the generator 50A is an input image, it is the correct answer for the discriminator 54A to determine “false”. The first learning processing unit 112 calculates the error (L _DA ) between the output value obtained from the discriminator 54A and this correct answer for each second data set.

Similarly, the first learning processing unit 112 inputs the learning image 61 and the label 63 included in each learning data set 60 to the discriminator 54A, and executes the arithmetic processing of the discriminator 54A. Accordingly, the first learning processing unit 112 acquires, from the discriminator 54A, an output value corresponding to the result of discriminating whether the input image is derived from the generator 50A or the learning data set 60. In this scene, since each learning image 61 is an input image, it is the correct answer for the discriminator 54A to determine “true”. The first learning processing unit 112 calculates the error (L _DA ) between the output value obtained from the discriminator 54A and this correct answer for each learning data set 60.

Then, the first learning processing unit 112 adjusts the parameter values of the discriminator 54A so that the sum of the calculated errors (L _DA ) becomes small. For example, the first learning processing unit 112 adjusts the value of the parameter of the discriminator 54A through the series of processes until the sum of the errors between the output value obtained from the discriminator 54 and the true / false correct answer becomes equal to or less than the threshold value. repeat. As a result, in the first training step, the first learning processing unit 112 obtains the input image input to the discriminator 54A is the image 69 generated by the generator 50A or is obtained from the plurality of learning data sets 60. The discriminator 54A is trained to discriminate whether or not it is the learning image 61.

  As shown in FIG. 17B, in the second training step, the first learning processing unit 112 extracts noise (latent variable) from a predetermined probability distribution, combines the extracted noise and each label 63, and outputs a plurality of data. Generate a set. Each data set may be the same as or different from the first data set. Subsequently, the first learning processing unit 112 inputs each data set (each label 63 and noise) to the generator 50A, and executes the arithmetic processing of the generator 50A. As a result, the first learning processing unit 112 acquires an output (image 69) corresponding to the result of generating an image from each label 63 from the generator 50A.

Next, the first learning processing unit 112 inputs the generated combination of the images 69 and the corresponding labels 63 to the discriminator 54A, and executes the arithmetic processing of the discriminator 54A. Accordingly, the first learning processing unit 112 acquires, from the discriminator 54A, an output value corresponding to the result of discriminating whether the input image is derived from the generator 50A or the learning data set 60. In the training of the generator 50A, the correct answer is that the discrimination result by the discriminator 54A is incorrect. That is, the correct answer is that the output value obtained from the discriminator 54A corresponds to "true". The first learning processing unit 112 calculates an error (L _GDA ) between the output value obtained from the discriminator 54A and the correct answer (that is, “true”) by a series of processes for each data set.

Then, the first learning processing unit 112 adjusts the value of the parameter of the generator 50A so that the sum of the calculated errors (L _GDA ) becomes small. For example, for each data set, the first learning processing unit 112 performs the above series of processes until the sum of the error between the output value obtained from the discriminator 54A and the “true” becomes equal to or less than the threshold, and the parameters of the generator 50A are processed. Repeat the adjustment of the value of. As a result, in the second training step, the first learning processing unit 112 trains the generator 50A so that the image 69 generated by the generator 50A causes the discrimination by the discriminator 54A to be erroneous.

  In the present modification, the first learning processing unit 112 alternately repeats the first training step and the second training step, so that the input label for the label 63 included in each learning data set 60 is input. Construct a generator 50A trained to generate an image corresponding to the training image 61 associated with 63. The storage processing unit 114 generates first learning result data 123A indicating the configuration and parameters of the constructed generator 50A, and stores the generated first learning result data 123A in a predetermined storage area.

  On the other hand, as shown in FIG. 17C, the estimator 52A according to the present modification is configured by a so-called convolutional neural network. Specifically, the estimator 52A according to the present modification includes one or more convolutional layers 524, a region proposal network (RPN) 525, an ROI (region of interest) pooling layer 526, and one or more total combinations. The layer 527, the classification unit 528, and the regression unit 529 are provided.

  The convolutional layer 524 is a layer that performs a convolutional operation on an image. The convolution of an image corresponds to the process of calculating the correlation between the image and a predetermined filter. When the estimator 52A includes a plurality of convolutional layers 524, the input image is input to the convolutional layer 524 arranged on the most input side, and the output of the convolutional layer 524 arranged on the most output side is the region proposal network 525 and the ROI pooling. Connected to layer 526.

  The area proposal network 525 is configured to propose a plurality of bounding boxes to which a score indicating the object-likeness is given for each local area of the feature map output from the convolutional layer 524 arranged on the most output side. The area proposal network 525 is configured by, for example, one or more fully connected layers, a regression network that predicts parameters (position and aspect ratio) of the bounding box, and a classification network that predicts the presence or absence of an object. One or more fully coupled layers are located on the input side. The outputs of the most connected layers located on the most output side are connected to the regression network and the classification network. The regression network and the classification network are each composed of, for example, one or a plurality of fully connected layers.

  The ROI pooling layer 526 pools the object region candidates of arbitrary size proposed based on the output of the region proposal network 525 in the feature map output from the convolutional layer 524, thereby converting the object region candidate into a vector of a fixed size. Is configured as follows. The output of the ROI pooling layer 526 is connected to the fully coupled layer 527 (located on the most input side).

  The fully connected layer 527 is a layer in which all neurons between adjacent layers are connected. When the estimator 52A includes a plurality of fully-connected layers 527, the output of the most-connected fully-connected layer 527 is connected to the classification unit 528 and the regression unit 529. The classifying unit 528 is configured to classify the types of defects included in the product R shown in the image from the output of the full coupling layer 527 arranged on the most output side. The classification unit 528 includes, for example, a fully connected layer. The regression unit 529 is configured to regress a range of defects (for example, a bounding box) included in the product R shown in the image from the output of the fully coupled layer 527 arranged closest to the output side. The regression unit 529 is composed of, for example, a fully connected layer.

In the second machine learning, the second learning processing unit 113 alternately executes the first training step for training the area proposal network 525 and the second training step for training the other portions. In the first training step, the second learning processing unit 113 adjusts the parameter values of the region proposal network 525 so that the error regarding the classification of the presence or absence of the object and the error regarding the position and aspect ratio of the bounding box are small. In the second training step, the second learning processing unit 113 uses the learned region suggestion network 525 to set each parameter so that each error (L _C , L _BB ) becomes small as in the above embodiment. Adjust the value of.

  In this modification, the second learning processing unit 113 alternately inputs the learning image 61 included in each learning data set 60 by repeating the first training step and the second training step. Construct an estimator 52A trained to estimate the type and range of defects included in the product R shown in the input learning image 61 so as to match the label 63 and the boundary information 65 associated with the learning image 61. To do. The storage processing unit 114 generates second learning result data 125A indicating the configuration and parameters of the constructed estimator 52A, and stores the generated second learning result data 125A in a predetermined storage area. However, the configuration of the estimator 52A may not be limited to such an example, and may be appropriately determined according to the embodiment. The configuration of the estimator 52A may be the same as that of the estimator 52.

(Operation example)
Next, an operation example of the learning device 1 according to this modification will be described. The learning device 1 according to the present modification can operate in the same manner as in the above-described embodiment, except that a part of the processing process of each machine learning is different from that in the above-described embodiment. That is, in step S101, the control unit 11 acquires a plurality of learning data sets 60. In the next step S102, the control unit 11 operates as the first learning processing unit 112 and executes the first machine learning regarding the generator 50A.

  An example of the processing procedure of the first machine learning will be described further using FIG. FIG. 18 illustrates an example of a processing procedure of the first machine learning according to the present modification. Performing the first machine learning according to the present modification includes the following steps S601 to S603. However, the processing procedure described below is merely an example, and each processing may be changed as much as possible. Further, regarding the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

Before executing step S601, the control unit 11 prepares the generator 50A and the discriminator 54A to be processed, as in the above embodiment. In step S601, the control unit 11 adjusts the value of the parameter of the discriminator 54A so that the sum of the calculated errors (L _DA ) becomes small as described above. Accordingly, the control unit 11 determines whether the input image input to the discriminator 54A is the image 69 generated by the generator 50A or the learning image 61 obtained from the plurality of learning data sets 60. Thus, the discriminator 54A is trained.

In step S602, the control unit 11 adjusts the value of the parameter of the generator 50A so that the sum of the calculated errors (L _GDA ) becomes small as described above. As a result, the control unit 11 trains the generator 50A so that the image 69 generated by the generator 50A causes the discrimination by the discriminator 54A to be erroneous. Note that the adjustment of the parameter values in steps S601 and S602 may be processed in the same manner as in the above embodiment.

  In step S603, the control unit 11 determines whether to repeat the execution of the training steps of steps S602 and S603. The criterion for repeating the execution of the training step may be appropriately determined depending on the embodiment. For example, the number of times to execute the training steps of steps S602 and S603 may be set in advance. In this case, the control unit 11 determines whether the number of times the training steps of steps S602 and S603 have been executed has reached the set number. When determining that the number of times the training steps of steps S602 and S603 have been executed has not reached the set number of times, the control unit 11 returns the process to step S601. On the other hand, when it is determined that the number of times the training steps of steps S602 and S603 have been executed has reached the set number, the control unit 11 completes the first machine learning process according to this modification, and the next step The process proceeds to S103. Thereby, the control unit 11 is trained to generate an image corresponding to the learning image 61 associated with the input label 63, for the input of the label 63 included in each learning data set 60. Build 50.

In step S103, the control unit 11 operates as the second learning processing unit 113 and executes the second machine learning regarding the estimator 52A. In the second machine learning, as described above, the control unit 11 alternately executes the first training step for training the area proposal network 525 and the second training step for training other portions. In the first training step, the control unit 11 adjusts the value of the parameter of the region proposal network 525 so that the error regarding the classification of the presence or absence of the object and the error regarding the position and aspect ratio of the bounding box become small. In the second training step, the control unit 11 uses the learned region proposed network 525, each error (L _C, L _BB) so decrease, adjusting the value of each parameter. The adjustment of the parameter value in each step may be processed in the same manner as in the above embodiment. As a result, the control unit 11 includes, for each learning data set 60, the product R shown in the input learning image 61 so that the input of the learning image 61 matches the corresponding label 63 and boundary information 65. Construct an estimator 52 that has been trained to estimate the type and extent of the defect to be treated.

  In the next step S104, the control unit 11 saves the information about the constructed learned generator 50A and the constructed information about the learned estimator 52A in a predetermined storage area, as in the above embodiment. . As described above, in the present modification, the learning device 1 can generate the generator 50A and the estimator 52A having the same functions as the generator 50 and the estimator 52 according to the above-described embodiment.

<4.6>
The above embodiment shows an example in which the present invention is applied to a scene in which the appearance inspection of the product R shown in the image is performed. However, the scope of application of the present invention is not limited to such an example of the visual inspection. INDUSTRIAL APPLICABILITY The present invention can be applied to all situations in which the range in which an object appears in a given image is estimated. By changing the image handled by the inspection system 100 from the image in which the product R appears to the image in which some object appears, it is possible to configure an estimation system that estimates the range in which some object appears in the given image.

  FIG. 19 schematically illustrates an example of an application scene of the estimation system 100B according to this modification. As shown in FIG. 19, the estimation system 100B according to the present modification includes a learning device 1B, an image generation device 2B, an estimator generation device 3B, and an estimation device 4B that are connected via a network. The hardware configuration and software configuration of each of the devices 1B to 4B are the same as the hardware configuration of each of the devices 1 to 4 according to the above-described embodiment, except that the image to be handled is changed from the image of the product R to the image of some object. And the software configuration may be the same. Further, each of the devices 1B to 4B may operate similarly to each of the devices 1 to 4 according to the above-described embodiment.

  That is, the learning device 1B according to the present modification is a combination of the learning image 61B that shows the target RB, the label 63B that indicates the type of the target RB, and the boundary information 65B that indicates the range of the target RB in the learning image 61B. A plurality of learning data sets 60B each configured by are acquired. Next, the learning device 1B performs the first machine learning, and with respect to the input of the label 63B included in each learning data set 60B, the image corresponding to the learning image 61B associated with the input label 63B. Construct a generator 50B trained to generate Further, the learning device 1B performs the second machine learning, so that for the input of the learning image 61B included in each learning data set 60B, the label 63B and the boundary information 65B associated with the input learning image 61B. The estimator 52B trained to estimate the type and range of the object RB appearing in the input learning image 61B is constructed so as to match The configurations of the generator 50B and the estimator 52B may be the same as the generator 50 and the estimator 52 described above. The machine learning process of the generator 50B and the estimator 52B may be the same as the machine learning process of the generator 50 and the estimator 52.

  On the other hand, the image generation device 2B generates the generated image 73B that may include the target RB by giving the input value 71B indicating the type of the target RB to the generator 50B constructed by the learning device 1B. . Next, the image generation device 2B estimates the type and range of the target object RB that can be reflected in the generated image 73B by giving the generated image 73B to the estimator 52B constructed by the learning device 1B. Subsequently, the image generation device 2B determines whether or not the result of estimating the type of the target object RB shown in the generated image 73B matches the type of the target object RB indicated by the input value 71B. Then, when the image generation device 2B determines that the result of estimating the type of the target object RB matches the type of the target object RB indicated by the input value 71B, it estimates the range of the generated generated image 73B and the target object RB. The result is stored in a predetermined storage area in association with the result.

  Further, the estimator generation device 3B includes a plurality of learning images each configured by a combination of a sample image 3221B that may include the target RB, a label 3222B that indicates the type of the target RB, and boundary information 3223B that indicates the range of the target RB. Acquire the data set 322B. Next, the estimator generation device 3B performs machine learning of the estimator 80B using the plurality of learning data sets 322B. The configuration of the estimator 80B may be the same as that of the estimator 80. The machine learning process of the estimator 80B may be the same as the machine learning process of the estimator 80. Accordingly, the estimator generation device 3B can construct the estimator 80B that has been trained to output the output value that matches the corresponding label 3222B and the boundary information 3223B when the sample image 3221B is input. Then, the estimator generation device 3B generates information indicating the configuration and parameters of the learned estimator 80B as learning result data 325B. Note that the estimator generation device 3B can use the data set generated by the image generation device 2B as the learning data set 322B.

  FIG. 20 schematically illustrates an example of the software configuration of the estimation device 4B according to this modification. The estimation device 4B according to the present modification acquires a target image 422B that can capture the target RB. In this modification, a camera CA is connected to the estimation device 4B. The estimation device 4B acquires the target image 422B by capturing a range in which the target RB can exist with the camera CA. Next, the control unit of the estimation device 4B operates as the estimation unit 412B, refers to the learning result data 325B, and sets the learned estimator 80B. Subsequently, the estimation device 4B inputs the acquired target image 422B to the learned estimator 80B, and executes the arithmetic processing of the learned estimator 80B. As a result, the estimation device 4B acquires an output value corresponding to the result of estimating the type and range of the target object RB (that is, the result of detecting the target object RB) from the estimator 80B. Then, the estimation device 4B outputs information related to the result of detecting the object RB based on the output value obtained from the estimator 80B. Thereby, the estimation system 100B according to the present modification is configured to estimate the range in which the target object RB appears in the given image.

  It should be noted that the type of the object RB is not particularly limited as long as it can be an object to be detected from the image, and may be appropriately selected according to the embodiment. The target object RB may be, for example, a product, a person, a body part of the person (for example, a face or the like) which is a target of the visual inspection. When using this modification for monitoring road traffic, the object RB may be, for example, a car, a pedestrian, a sign, or the like.

100 ... inspection system,
1 ... learning device,
11 ... Control unit, 12 ... Storage unit, 13 ... Communication interface,
14 ... Input device, 15 ... Output device, 16 ... Drive,
111 ... (first) data acquisition unit, 112 ... first learning processing unit,
113 ... Second learning processing unit, 114 ... (First) storage processing unit,
121 ... Learning program, 123 ... First learning result data,
125 ... Second learning result data,
2 ... Image generation device,
21 ... Control unit, 22 ... Storage unit, 23 ... Communication interface,
24 ... Input device, 25 ... Output device, 26 ... Drive,
211 ... Generation unit, 212 ... Estimation unit, 213 ... Judgment unit,
214 ... an image storage unit,
221 ... Image generation program,
50 ... Generator, 52 ... Estimator, 54 ... Discriminator,
56 ... Encoder,
60 ... (first) learning data set,
61 ... Learning image, 63 ... Label, 65 ... Boundary information,
71 ... Input value, 73 ... Generated image, 75 ... Label,
77 ... boundary information,
3 ... Estimator generation device,
31 ... Control unit, 32 ... Storage unit, 33 ... Communication interface,
34 ... Input device, 35 ... Output device, 36 ... Drive,
311 ... (second) data acquisition unit,
312 ... (Third) learning processing unit,
313 ... (Second) storage processing unit,
321 ... Estimator generation program,
322 ... (Second) learning data set,
3221 ... sample image, 3222 ... label,
3223 ... boundary information,
325 ... Third learning result data,
4 ... Inspection device (estimation device),
41 ... Control unit, 42 ... Storage unit, 43 ... Communication interface,
44 ... Input device, 45 ... Output device, 46 ... Drive,
47 ... external interface,
411 ... Target data acquisition unit, 412 ... Pass / fail judgment unit,
413 ... Output unit,
421 ... inspection program,
80 ... Estimator,
91, 92, 93, 94 ... Storage medium

Claims (8)

Data for acquiring a plurality of learning data sets each configured by a combination of a learning image showing a product to be subjected to visual inspection, a label indicating the type of defect included in the product, and boundary information indicating the range of the defect. The acquisition part,
Training to generate an image corresponding to the learning image associated with the input label for the input of the label included in each of the learning data sets by performing the first machine learning. A first learning processing unit for constructing the generated generator,
By performing the second machine learning, the input of the learning image included in the learning data set of each case is matched with the label and the boundary information associated with the input learning image. A second learning processing unit that constructs an estimator trained to estimate the type and range of the defect included in the product captured in the input learning image,
With
Learning device. The input of the generator is connected to the output of the encoder,
The output of the generator is connected to the input of the estimator and the input of the discriminator,
Performing the first machine learning is
Train the discriminator to discriminate whether the input image input to the discriminator is the image generated by the generator or the learning image obtained from the plurality of learning data sets. A first training step,
A second training step of training the generator such that the image generated by the generator is such that the discrimination by the discriminator is erroneous;
Train the generator so that the image generated by the generator is such that the result of estimating the type of defect contained in the product shown in the image by the estimator matches the label. A third training step,
A fourth training step of training the encoder and the generator such that the image generated by the generator by inputting the label and the latent variable to the generator is a reconstructed version of the learning image. Wherein the latent variable is derived based on an output obtained from the encoder by inputting the label and the learning image to the encoder, a fourth training step,
A fifth training step of training the encoder such that the distribution of the latent variable derived based on the output obtained from the encoder follows a predetermined prior distribution;
including,
The learning device according to claim 1. The output of the generator is connected to the input of the discriminator,
Performing the first machine learning is
The discriminator is trained so as to discriminate whether the input image input to the discriminator is the image generated by the generator or the learning image obtained from the plurality of learning data sets. A first training step to
A second training step of training the generator such that the image generated by the generator is such that the discrimination by the discriminator is erroneous;
Including alternating
The learning device according to claim 1. The range of the defect is represented by a bounding box,
The learning device according to any one of claims 1 to 3. An input value indicating the type of the defect is given to the generator constructed by the learning device according to any one of claims 1 to 4 to generate a generated image in which the product including the defect is copied. A generator that
An estimation unit that estimates the type and range of the defect included in the product shown in the generated image by giving the generated image generated to the estimator constructed by the learning device,
A determination unit that determines whether the result of estimating the type of the defect included in the product shown in the generated image matches the type of the defect indicated by the input value,
When the determination unit determines that the result of estimating the type of the defect matches the type of the defect indicated by the input value, the generated image generated and the result of estimating the range of the defect are associated with each other and predetermined. Image storage unit to store in the storage area of
With
Image generation device. Computer
A step of acquiring a plurality of learning data sets each formed by a combination of a learning image showing a product to be subjected to a visual inspection, a label indicating a type of defect included in the product, and boundary information indicating a range of the defect. When,
Training to generate an image corresponding to the learning image associated with the input label for the input of the label included in each of the learning data sets by performing the first machine learning. Constructing the generated generator,
By performing the second machine learning, the input of the learning image included in the learning data set of each case is matched with the label and the boundary information associated with the input learning image. Constructing an estimator trained to estimate the type and range of the defect contained in the product in the input learning image,
Run the
Learning method. On the computer,
A step of acquiring a plurality of learning data sets each formed by a combination of a learning image showing a product to be subjected to a visual inspection, a label indicating a type of defect included in the product, and boundary information indicating a range of the defect. When,
Training to generate an image corresponding to the learning image associated with the input label for the input of the label included in each of the learning data sets by performing the first machine learning. Constructing the generated generator,
By performing the second machine learning, the input of the learning image included in the learning data set of each case is matched with the label and the boundary information associated with the input learning image. Constructing an estimator trained to estimate the type and range of the defect contained in the product in the input learning image,
To execute
Learning program. A data acquisition unit that acquires a plurality of learning data sets each configured by a combination of a learning image that captures an object, a label that indicates the type of the object, and boundary information that indicates the range of the object in the learning image. When,
Training to generate an image corresponding to the learning image associated with the input label for the input of the label included in each of the learning data sets by performing the first machine learning. A first learning processing unit for constructing the generated generator,
By performing the second machine learning, the input of the learning image included in the learning data set of each case is matched with the label and the boundary information associated with the input learning image. A second learning processing unit that constructs an estimator trained to estimate the type and range of the object shown in the input learning image,
With
Learning device.