JP7059889B2 - Learning device, image generator, learning method, and learning program - Google Patents

Description

The present invention relates to a learning device, an image generation device, a learning method, and a learning program.

Conventionally, in the scene of manufacturing a product such as a production line, a technique of photographing the manufactured product with an imaging device and inspecting the quality of the product based on the obtained image data has been used. For example, in Patent Document 1, it is determined whether the inspection target imaged in the image is normal or abnormal based on the first learned neural network, and when it is determined that the inspection target is abnormal, it is determined. An inspection device that classifies the type of the anomaly based on the learned second neural network has been proposed.

Japanese Unexamined Patent Publication No. 2012-026892

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, "Generative Adversarial Networks", Advances in neural information processing systems, 2672-2680, 2014

The inventors of the present invention have the following problems in the conventional technique of determining the quality of a product from image data by using an estimator composed of a learning model such as a neural network as in Patent Document 1. I found that it could happen. That is, when performing machine learning to make a learning model acquire the ability to judge the quality of a product, a sample image is prepared as training data. If the number of sample images is small, the accuracy of the pass / fail judgment by the trained learning model (estimator) becomes insufficient. On the other hand, it is costly to prepare a sufficient number of sample images in order to improve the accuracy of the judgment by the estimator.

Therefore, the inventors of the present invention consider using a generator (generative model) to mass-produce a plurality of different images from the prepared images, and to use the mass-produced multiple images as training data for machine learning. bottom. For example, Non-Patent Document 1 proposes a method of constructing a generator (generative model) from a prepared learning image by machine learning. This generator is trained by machine learning to generate an image corresponding to the trained image. That is, if the image of the product is used as the learning image, the generator is trained to produce a possible image of the product. Prepare sample images because this trained generator will be able to automatically generate sample images that can be used for machine learning to acquire the ability to judge the quality of a product. The cost can be reduced.

However, the present inventors have found that the following problems may occur in the situation of using this generator. That is, the larger the size of the training image, the more complicated the structure of the generator. For example, when the generator is configured by a neural network, the larger the size of the training image, the larger the number of layers such as the intermediate layer (hidden layer) and the number of neurons in each layer. As a result, the calculation cost required to construct the generator by machine learning becomes large, a memory shortage occurs during the calculation process of machine learning, and the calculation process takes a long time. There may be a problem that a problem such as the calculation process not being completed within a predetermined time occurs.

It should be noted that such a problem is not limited to the scene where a generator (generation model) for generating an image of a product is constructed by machine learning, but also a generator for generating an image of an object. Can occur in any situation that builds. For example, when constructing a generator for generating a face image showing a subject's face, if a high-resolution image is used as a learning image, the calculation cost for constructing the generator becomes large. As a result, the above-mentioned problem may occur.

The present invention, on the one hand, has been made in view of such circumstances, and an object of the present invention is to provide a technique for reducing the computational cost required to construct a generator for generating an image. be.

The present invention adopts the following configuration in order to solve the above-mentioned problems.

That is, the learning device according to one aspect of the present invention includes an image acquisition unit that acquires a learning image of a product to be visually inspected, an image division unit that divides the learning image into a plurality of patch images, and the above. Depending on the position of each patch image in the training image, it is associated with each patch image by performing machine learning with a labeling unit that associates one of a plurality of labels with each patch image. A learning processing unit for constructing a generator trained to generate an image corresponding to each patch image from the label is provided.

The learning device according to the configuration divides the learning image showing the product to be visually inspected into a plurality of patch images, and performs machine learning of the generator using each patch image and each corresponding label. .. By this machine learning, the generator is trained to generate an image corresponding to each corresponding patch image from each label. Combining each image generated by a trained generator according to each label produces a full-size image corresponding to the training image that can capture the product subject to visual inspection. Can be done.

That is, the learning device according to the configuration causes the generator to learn the generation of an image similar to each patch image obtained by dividing the learning image. This causes a full-size image that resembles the training image through the generation of an image that resembles each patch image, rather than letting the generator learn the ability to directly generate an image that resembles the full-size training image. To teach the generator the ability to indirectly generate. Therefore, in this configuration, the size of the image to be trained by the generator can be suppressed, so that the configuration of the generator can be simplified. Therefore, according to the configuration, it is possible to reduce the calculation cost required to construct the generator by machine learning. As a result, it is possible to reduce the amount of memory used in the calculation process of machine learning and shorten the time required for the calculation process.

In the learning device according to the one aspect, the image dividing unit may calculate the complexity of the learning image, may determine the patch size according to the calculated complexity, and may determine the patch size. The learning image may be divided into the plurality of patch images each having the above. According to this configuration, the training image can be divided with a patch size appropriate for the generator to acquire the ability to generate an image similar to the training image. Therefore, while reducing the computational cost required to construct the generator by machine learning, the machine learning can make the generator acquire the ability to generate an appropriate image corresponding to (similar to) the trained image.

In the learning device according to the one aspect, the labeling unit may calculate the complexity of each patch image, and based on the calculated complexity, two of the plurality of patch images that are similar to each other. The above patch images may be integrated, or the two or more integrated patch images may be given the same label. According to this configuration, by integrating two or more similar patch images, the types of patch images to be learned by the generator can be reduced, so that the configuration of the generator can be further simplified. Therefore, the calculation cost required to construct the generator by machine learning can be further reduced.

The learning device according to the above aspect may further include a storage processing unit that stores information about the constructed generator in a predetermined storage area. According to this configuration, the generator built by machine learning can be made available to other computers.

In the learning device according to the one aspect, the generator may be connected to the discriminator, and performing the machine learning means that the input image input to the discriminator is generated by the generator. From the first training step of training the discriminator to discriminate whether it is an image or the patch image divided from the learning image, and from the label associated with each patch image, the discriminator is used. It may include alternating second training steps to train the generator to generate the image such that the discrimination is erroneous. According to this configuration, a generator trained to generate an image corresponding to each corresponding patch image from each label can be appropriately constructed.

In the learning device according to the one aspect, the generator may be connected to an encoder, and performing the machine learning means an output value obtained from the encoder by inputting each patch image to the encoder and the above. It may include training the generator to generate an image corresponding to each patch image from the label associated with each patch image. According to this configuration, a generator trained to generate an image corresponding to each corresponding patch image from each label can be appropriately constructed.

Further, the image generation device according to one aspect of the present invention is a generation unit that generates a plurality of images from each of the plurality of labels by using the generator constructed by the learning device according to any one of the above modes. And a joining portion for generating a combined image by arranging each of the generated plurality of images at a position corresponding to each of the plurality of labels and then combining them. According to this configuration, by using the simplified generator, it is possible to reduce the calculation cost required for mass-producing an image corresponding to the learning image, that is, an image (combined image) in which the product can be captured. ..

Further, the learning device and the image generation device according to each of the above modes are not only used for constructing a generator for generating an image of a product to be visually inspected, but also, for example, a face of a subject. It may be applied to any scene in which a generator for generating an image of an object is constructed, such as a scene in which a generator for generating a face image is constructed.

For example, the learning device according to one aspect of the present invention includes an image acquisition unit that acquires a learning image of an object, an image division unit that divides the learning image into a plurality of patch images, and the above-mentioned in the learning image. Depending on the position of each patch image, a label assigning unit that associates one of a plurality of labels with each patch image and machine learning are performed to obtain the label associated with each patch image. It comprises a learning processing unit that builds a generator trained to generate an image corresponding to each patch image. The type of the object is not particularly limited as long as it can be an object for identifying some attribute, and may be appropriately selected according to the embodiment. The object may be, for example, a product to be inspected for appearance, a person, a body part of the person (for example, a face, etc.), characters, a situation outside the vehicle in a vehicle, or the like. The feature to be estimated does not have to be particularly limited, and may be appropriately selected depending on the object. When the object is a product to be visually inspected, the feature to be estimated may be, for example, a defect such as a stain, a stain, or a scratch. When the object is a person's face, the features to be estimated may be, for example, the type of facial expression, the state of facial parts, the attributes of the person on the face, and the like. When the object is a character, the feature to be estimated may be attributes such as print sharpness, distortion, and character string deviation. When the object is a situation outside the vehicle, the feature to be estimated may be an attribute of the road condition such as a lane condition and a traffic condition.

The image generation system according to one aspect of the present invention may be configured by the learning device and the image generation device according to any one of the above forms. Further, the estimation system according to one aspect of the present invention is an estimator that estimates some features of an object to be captured in an image by using the learning device, the image generation device, and the generated image according to any one of the above forms. It may be configured by an estimator generator for constructing an image and an estimator for estimating the characteristics of an object in an image by using the constructed estimator. Further, as another form of each of the learning device, the image generation device, the image generation system, and the estimation system according to each of the above forms, one aspect of the present invention may be an information processing method that realizes each of the above configurations. However, it may be a program, or it may be a storage medium that stores such a program and can be read by a computer or the like. Here, the storage medium that can be read by a computer or the like is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

For example, the learning method according to one aspect of the present invention includes a step in which a computer acquires a learning image showing a product to be visually inspected, a step of dividing the learning image into a plurality of patch images, and the learning. Depending on the position of each patch image in the image, the step of associating one of the plurality of labels with each patch image and the label associated with each patch image by performing machine learning. It is an information processing method that executes a step of constructing a generator trained to generate an image corresponding to each of the patch images.

Further, for example, the learning program according to one aspect of the present invention includes a step of acquiring a learning image of a product to be visually inspected on a computer, a step of dividing the learning image into a plurality of patch images, and a step of dividing the learning image into a plurality of patch images. Depending on the position of each patch image in the training image, one of a plurality of labels is associated with each patch image, and machine learning is performed to associate one of the plurality of labels with each patch image. A program for constructing a generator trained to generate an image corresponding to each of the patch images from the label.

According to the present invention, it is possible to reduce the computational cost required to construct a generator for generating an image.

FIG. 1 schematically illustrates an example of a situation in which the present invention is applied. FIG. 2 schematically illustrates an example of the hardware configuration of the learning device according to the embodiment. FIG. 3 schematically illustrates an example of the hardware configuration of the image generator according to the embodiment. FIG. 4 schematically illustrates an example of the hardware configuration of the estimator generator according to the embodiment. FIG. 5 schematically illustrates an example of the hardware configuration of the inspection device according to the embodiment. FIG. 6 schematically illustrates an example of the software configuration of the learning device according to the embodiment. FIG. 7A schematically illustrates an example of the machine learning process of the network according to the embodiment. FIG. 7B schematically illustrates an example of the machine learning process of the network according to the embodiment. FIG. 8 schematically illustrates an example of the software configuration of the image generator according to the embodiment. FIG. 9 schematically illustrates an example of the software configuration of the estimator generator according to the embodiment. FIG. 10 schematically illustrates an example of the software configuration of the inspection device according to the embodiment. FIG. 11 illustrates an example of the processing procedure of the learning device according to the embodiment. FIG. 12 illustrates an example of the patch division processing procedure according to the embodiment. FIG. 13 illustrates an example of the labeling processing procedure according to the embodiment. FIG. 14 illustrates an example of the machine learning processing procedure according to the embodiment. FIG. 15 illustrates an example of the processing procedure of the image generator according to the embodiment. FIG. 16 illustrates an example of the processing procedure of the estimator generator according to the embodiment. FIG. 17 illustrates an example of the processing procedure of the inspection device according to the embodiment. FIG. 18 schematically illustrates an example of the software configuration of the learning device according to the modified example. FIG. 19 schematically illustrates a modified example of the scene to which the present invention is applied. FIG. 20 schematically illustrates an example of the software configuration of the estimation device according to the modified example.

Hereinafter, an embodiment according to one aspect of the present invention (hereinafter, also referred to as “the present embodiment”) will be described with reference to the drawings. However, the embodiments described below are merely examples of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. That is, in carrying out the present invention, a specific configuration according to the embodiment may be appropriately adopted. The data appearing in the present embodiment are described in natural language, but more specifically, they are specified in a pseudo language, a command, a parameter, a machine language, etc. that can be recognized by a computer.

§1 Application example First, an example of a situation in which the present invention is applied will be described with reference to FIG. FIG. 1 schematically illustrates an example of a situation in which the present invention is applied to a visual inspection of a product R. However, the scope of application of the present invention is not limited to the examples of visual inspection illustrated below. The present invention is applicable to all situations in which some feature of an object in an image is estimated.

As illustrated in FIG. 1, the inspection system 100 according to the present embodiment includes a learning device 1, an image generation device 2, an estimator generation device 3, and an inspection device 4 connected via a network. As a result, the inspection system 100 is configured to inspect the quality of the product R. The type of network between the learning device 1, the image generator 2, the estimator generator 3, and the inspection device 4 may be not particularly limited, and may be, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, and the like. It may be appropriately selected from a dedicated network or the like.

In the example of FIG. 1, the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 are separate computers. However, the configuration of the inspection system 100 does not have to be limited to such an example. At least one pair of the learning device 1, the image generator 2, the estimator generator 3, and the inspection device 4 may be an integrated computer. Further, the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 may each be composed of a plurality of computers.

The learning device 1 according to the present embodiment is a computer configured to construct a generator 50 for generating an image of a product R to be visually inspected. Specifically, the learning device 1 acquires a learning image 60 showing the product R to be visually inspected, and divides the acquired learning image 60 into a plurality of patch images 61. Next, the learning device 1 associates (assigns) any label 62 of the plurality of labels 62 with each patch image 61 according to the position of each patch image 61 in the learning image 60. Each label 62 indicates the position of each patch image 61 in the training image 60.

Then, the learning device 1 adjusts the parameters of the generator 50 by performing machine learning of the generator 50 using each patch image 61 and each label 62. As a result, the learning device 1 constructs a generator 50 trained to generate an image corresponding to each patch image 61 from the label 62 associated with each patch image 61. The image corresponding to each patch image 61 is an image similar to each patch image 61 and capable of showing an object to be reflected in each patch image 61. The generator 50 is trained to generate an image that follows (along with) the distribution of each patch image 61.

Note that "P1" in FIG. 1 indicates the type of the patch image 61, and "C1" indicates the type of the label 62. In the example of FIG. 1, the label "C1" is attached to the patch image "P1" in the upper left corner. However, the type of the patch image 61, the type of the label 62, and the correspondence between the patch image 61 and the label 62 do not have to be limited to such an example, and may be appropriately determined according to the embodiment.

On the other hand, the image generation device 2 according to the present embodiment is a computer configured to generate a possible image of the product R by using the trained generator 50 constructed by the learning device 1. be. Specifically, the image generation device 2 uses the generator 50 constructed by the learning device 1 to generate each of the plurality of images 67 from each of the plurality of labels 62. The generated image 67 is an image corresponding to each of the patch images 61.

Then, the image generation device 2 generates a combined image 69 by arranging each of the generated plurality of images 67 at a position corresponding to each of the plurality of labels 62 and then combining them. The generated combined image 69 is an image corresponding to the learning image 60 showing the product R. Note that "Q1" in FIG. 1 indicates the type of the image 67 generated by the generator 50. In the example of FIG. 1, the image "Q1" generated from the label "C1" is placed in the upper left corner and then combined with another image 67 generated from the other label 62.

The estimator generator 3 according to the present embodiment is a computer configured to construct an estimator for estimating the quality of the product R. Specifically, the estimator generator 3 acquires a plurality of learning data sets each composed of a combination of a sample image showing the product R and correct answer data. The correct answer data shows the result (that is, the correct answer) of determining the quality of the product R shown in the sample image. Then, the estimator generator 3 constructs a trained estimator that has acquired the ability to judge the quality of the product R reflected in a given image by performing machine learning using a plurality of learning data sets. .. The estimator generation device 3 can use the combined image 69 generated by the image generation device 2 as a sample image.

On the other hand, the inspection device 4 according to the present embodiment is a computer configured to judge the quality of the product R by using the trained estimator constructed by the estimator generation device 3. The inspection device 4 is an example of an estimation device for estimating some feature of an object shown in an image. Specifically, the inspection device 4 acquires a target image of the product R to be visually inspected. In the present embodiment, the camera CA is connected to the inspection device 4. The inspection device 4 acquires a target image by photographing the product R with this camera CA. Next, the inspection device 4 inputs the acquired target image to the trained estimator, and executes the arithmetic processing of the trained estimator. As a result, the inspection device 4 acquires an output value corresponding to the result of determining the quality of the product R from the trained estimator. Then, the inspection device 4 outputs information regarding the result of determining the quality of the product R based on the output value obtained from the learned estimator.

As described above, the learning device 1 according to the present embodiment causes the generator 50 to learn the generation of the image corresponding to each patch image 61 obtained by dividing the learning image 60. As a result, the generator 50 does not acquire the ability to directly generate the image corresponding to the full-size learning image 60, but corresponds to the learning image 60 through the generation of the image corresponding to each patch image 61. The generator 50 is trained in the ability to indirectly generate a full-size image. Therefore, in the present embodiment, the size of the image to be learned by the generator 50 can be suppressed, so that the configuration of the generator 50 can be simplified. Therefore, according to the present embodiment, it is possible to reduce the calculation cost required to construct the generator 50 by machine learning. As a result, it is possible to reduce the amount of memory used in the calculation process of machine learning and shorten the time required for the calculation process.

Further, in the image generation device 2 according to the present embodiment, by using the generator 50 constructed by the learning device 1, the calculation cost required for mass-producing the image (combined image 69) in which the product R can be captured is suppressed. be able to. In addition, in the present embodiment, the combined image 69 mass-produced by the image generation device 2 can be used as a sample image. This makes it possible to reduce the cost of collecting the training data set in the estimator generator 3. Further, in the present embodiment, by using the combined image 69 mass-produced by the image generation device 2 as a sample image, the number of sample images used for machine learning of the estimator can be increased. This makes it possible to improve the accuracy of determining the quality of the product R in the inspection device 4.

The product R to be inspected for appearance may not be particularly limited and may be appropriately selected according to the embodiment. The product R may be, for example, a product transported on a production line such as an electronic component or an automobile component. Electronic components are, for example, boards, chip capacitors, liquid crystals, relay windings, and the like. Automotive parts are, for example, connecting rods, shafts, engine blocks, power window switches, panels and the like. Further, the quality determination may be simply determining whether or not the product R has a defect, and in addition to determining whether or not the product R has a defect, the type of the defect is identified. May include doing. The defects are, for example, scratches, stains, cracks, dents, dust, burrs, color unevenness, and the like.

§2 Configuration example [Hardware configuration]
<Learning device>
Next, an example of the hardware configuration of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 2 schematically illustrates an example of the hardware configuration of the learning device 1 according to the present embodiment.

As shown in FIG. 2, the learning device 1 according to the present embodiment is a computer to which a control unit 11, a storage unit 12, a communication interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected. .. In FIG. 2, the communication interface is described as "communication I / F".

The control unit 11 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, which are hardware processors, and is configured to execute information processing based on a program and various data. To. The storage unit 12 is an example of a memory, and is composed of, for example, a hard disk drive, a solid state drive, or the like. In the present embodiment, the storage unit 12 stores various information such as the learning program 121, the learning image 60, the label information 621, and the first learning result data 125.

The learning program 121 is a program for causing the learning device 1 to execute information processing (FIGS. 11 to 14) described later, which constructs a generator 50 for generating a possible image of the product R. The learning program 121 includes a series of instructions for the information processing. The learning image 60 is used for machine learning of the generator 50. The number of learning images 60 may be appropriately determined according to the embodiment. The label information 621 shows the correspondence between each patch image 61 and each label 62. The first learning result data 125 is data for setting the trained generator 50 constructed by machine learning. The first learning result data 125 is generated as an execution result of the learning program 121. Details will be described later.

The communication interface 13 is, for example, a wired LAN (Local Area Network) module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 13, the learning device 1 can perform data communication via a network with another information processing device (for example, an image generation device 2 and an estimator generation device 3).

The input device 14 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 15 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the learning device 1 by using the input device 14 and the output device 15.

The drive 16 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 91. The type of the drive 16 may be appropriately selected according to the type of the storage medium 91. At least one of the learning program 121 and the learning image 60 may be stored in the storage medium 91.

The storage medium 91 transfers the information of the program or the like by electrical, magnetic, optical, mechanical or chemical action so that the computer or other device, the machine or the like can read the information of the recorded program or the like. It is a medium to accumulate. The learning device 1 may acquire at least one of the learning program 121 and the learning image 60 from the storage medium 91.

Here, in FIG. 2, as an example of the storage medium 91, a disc-type storage medium such as a CD or a DVD is illustrated. However, the type of the storage medium 91 is not limited to the disc type, and may be other than the disc type. Examples of storage media other than the disk type include semiconductor memories such as flash memories.

Regarding the specific hardware configuration of the learning device 1, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 11 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, an FPGA (field-programmable gate array), a DSP (digital signal processor), or the like. The storage unit 12 may be composed of a RAM and a ROM included in the control unit 11. At least one of the communication interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The learning device 1 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the learning device 1 may be a general-purpose server device, a PC (Personal Computer), or the like, in addition to an information processing device designed exclusively for the provided service.

<Image generator>
Next, an example of the hardware configuration of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 3 schematically illustrates an example of the hardware configuration of the image generation device 2 according to the present embodiment.

As shown in FIG. 3, the image generation device 2 according to the present embodiment is a computer to which a control unit 21, a storage unit 22, a communication interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected. be. Each of the control unit 21 to the drive 26 of the image generation device 2 according to the present embodiment may be configured in the same manner as each of the control unit 11 to the drive 16 of the learning device 1.

That is, the control unit 21 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 22 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 22 stores various information such as the image generation program 221, the label information 621, the first learning result data 125, and the combined image 69.

The image generation program 221 is a program for causing the image generation device 2 to execute information processing (FIG. 15) described later to generate a combined image 69 by using the trained generator 50. The image generation program 221 includes a series of instructions for the information processing. The combined image 69 is generated as an execution result of the image generation program 221. Details will be described later.

The communication interface 23 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 23, the image generation device 2 can perform data communication via a network with another information processing device (for example, a learning device 1 and an estimator generation device 3).

The input device 24 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 25 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the image generation device 2 by using the input device 24 and the output device 25.

The drive 26 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 92. At least one of the image generation program 221 and the label information 621 and the first learning result data 125 may be stored in the storage medium 92. Further, the image generation device 2 may acquire at least one of the image generation program 221, the label information 621, and the first learning result data 125 from the storage medium 92.

Regarding the specific hardware configuration of the image generation device 2, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 21 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 22 may be composed of a RAM and a ROM included in the control unit 21. At least one of the communication interface 23, the input device 24, the output device 25, and the drive 26 may be omitted. The image generation device 2 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the image generation device 2 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Estimator generator>
Next, an example of the hardware configuration of the estimator generator 3 according to the present embodiment will be described with reference to FIG. FIG. 4 schematically illustrates an example of the hardware configuration of the estimator generator 3 according to the present embodiment.

As shown in FIG. 4, the estimator generator 3 according to the present embodiment is a computer to which a control unit 31, a storage unit 32, a communication interface 33, an input device 34, an output device 35, and a drive 36 are electrically connected. Is. Each of the control unit 31 to the drive 36 of the estimator generation device 3 according to the present embodiment may be configured in the same manner as each of the control unit 11 to the drive 16 of the learning device 1.

That is, the control unit 31 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 32 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 32 stores various information such as the estimator generation program 321, the learning data set 322, and the second learning result data 325.

The estimator generation program 321 is a program for causing the estimator generation device 3 to execute information processing (FIG. 16) described later, which constructs an estimator for determining the quality of the product R. The estimator generation program 321 includes a series of instructions for the information processing. The training data set 322 is used for machine learning of this estimator. The second learning result data 325 is data for setting the trained estimator constructed by machine learning. The second learning result data 325 is generated as an execution result of the estimator generation program 321. Details will be described later.

The communication interface 33 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 33, the estimator generation device 3 can perform data communication via a network with other information processing devices (for example, a learning device 1, an image generation device 2, and an inspection device 4). ..

The input device 34 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 35 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the estimator generator 3 by using the input device 34 and the output device 35.

The drive 36 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 93. At least one of the estimator generation program 321 and the learning data set 322 may be stored in the storage medium 93. Further, the estimator generation device 3 may acquire at least one of the estimator generation program 321 and the learning data set 322 from the storage medium 93.

Regarding the specific hardware configuration of the estimator generator 3, components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 31 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 32 may be composed of a RAM and a ROM included in the control unit 31. At least one of the communication interface 33, the input device 34, the output device 35, and the drive 36 may be omitted. The estimator generator 3 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, the estimator generator 3 may be a general-purpose server device, a general-purpose PC, or the like, in addition to an information processing device designed exclusively for the provided service.

<Inspection equipment>
Next, an example of the hardware configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 5 schematically illustrates an example of the hardware configuration of the inspection device 4 according to the present embodiment.

As shown in FIG. 5, in the inspection device 4 according to the present embodiment, the control unit 41, the storage unit 42, the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47 are electrically connected to each other. It is a computer. In FIG. 5, the external interface is described as "external I / F". The control units 41 to 46 of the inspection device 4 may be configured in the same manner as the control units 11 to 16 of the learning device 1, respectively.

That is, the control unit 41 includes a CPU, RAM, ROM, etc., which are hardware processors, and is configured to execute various information processing based on programs and data. The storage unit 42 is composed of, for example, a hard disk drive, a solid state drive, or the like. The storage unit 42 stores various information such as the inspection program 421 and the second learning result data 325.

The inspection program 421 uses the trained estimator constructed by the estimator generation device 3 to cause the inspection device 4 to execute information processing (FIG. 17) described later for determining the quality of the product R shown in the target image. It is a program for. The inspection program 421 includes a series of instructions for the information processing. Details will be described later.

The communication interface 43 is, for example, a wired LAN module, a wireless LAN module, or the like, and is an interface for performing wired or wireless communication via a network. By using this communication interface 43, the inspection device 4 can perform data communication via a network with another information processing device (for example, an estimator generation device 3).

The input device 44 is, for example, a device for inputting a mouse, a keyboard, or the like. Further, the output device 45 is, for example, a device for outputting a display, a speaker, or the like. The operator can operate the inspection device 4 by using the input device 44 and the output device 45.

The drive 46 is, for example, a CD drive, a DVD drive, or the like, and is a drive device for reading a program stored in the storage medium 94. At least one of the inspection program 421 and the second learning result data 325 may be stored in the storage medium 94. Further, the inspection device 4 may acquire at least one of the inspection program 421 and the second learning result data 325 from the storage medium 94.

The external interface 47 is, for example, a USB (Universal Serial Bus) port, a dedicated port, or the like, and is an interface for connecting to an external device. The type and number of the external interfaces 47 may be appropriately selected according to the type and number of connected external devices. In this embodiment, the inspection device 4 is connected to the camera CA via the external interface 47.

The camera CA is used to acquire a target image of the product R. The type and location of the camera CA are not particularly limited and may be appropriately determined according to the embodiment. As the camera CA, for example, a known camera such as a digital camera or a video camera may be used. Further, the camera CA may be arranged in the vicinity of the production line on which the product R is conveyed. When the camera CA includes a communication interface, the inspection device 4 may be connected to the camera CA via the communication interface 43 instead of the external interface 47.

Regarding the specific hardware configuration of the inspection device 4, the components can be omitted, replaced, or added as appropriate according to the embodiment. For example, the control unit 41 may include a plurality of hardware processors. The hardware processor may be composed of a microprocessor, FPGA, DSP and the like. The storage unit 42 may be composed of a RAM and a ROM included in the control unit 41. At least one of the communication interface 43, the input device 44, the output device 45, the drive 46, and the external interface 47 may be omitted. The inspection device 4 may be composed of a plurality of computers. In this case, the hardware configurations of the computers may or may not match. Further, as the inspection device 4, in addition to an information processing device designed exclusively for the provided service, a general-purpose server device, a general-purpose desktop PC, a notebook PC, a tablet PC, a mobile phone including a smartphone, or the like may be used.

[Software configuration]
<Learning device>
Next, an example of the software configuration of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 6 schematically illustrates an example of the software configuration of the learning device 1 according to the present embodiment.

The control unit 11 of the learning device 1 expands the learning program 121 stored in the storage unit 12 into the RAM. Then, the control unit 11 interprets and executes the learning program 121 expanded in the RAM by the CPU, and controls each component. As a result, as shown in FIG. 6, the learning device 1 according to the present embodiment includes an image acquisition unit 111, an image segmentation unit 112, a labeling unit 113, a learning processing unit 114, and a storage processing unit 115 as software modules. Works as a computer. That is, in the present embodiment, each software module of the learning device 1 is realized by the control unit 11 (CPU).

The image acquisition unit 111 acquires a learning image 60 showing the product R to be visually inspected. The image segmentation unit 112 divides the acquired learning image 60 into a plurality of patch images 61. In the example of FIG. 6, the image segmentation unit 112 divides the learning image 60 having a size of W × H into a patch image 61 having a patch size of w × h. The patch size (w × h) of each patch image 61 with respect to the size (W × H) of the training image 60 is not particularly limited and may be appropriately determined according to the embodiment. In this embodiment, the patch size of each patch image 61 is determined based on the complexity of the learning image 60 by information processing (FIG. 12) described later.

The labeling unit 113 associates the label 62 of the plurality of labels 62 with each patch image 61 according to the position of each patch image 61 in the learning image 60. The learning processing unit 114 constructs a generator 50 trained to generate an image corresponding to each patch image 61 from the label 62 associated with each patch image 61 by performing machine learning. The storage processing unit 115 stores information about the constructed trained generator 50 in a predetermined storage area.

(Learning network)
Next, an example of the configuration of the learning network 500 including the generator 50 according to the present embodiment will be described with reference to FIGS. 7A and 7B. As shown in each figure, in the present embodiment, the generator 50 is connected to the discriminator 52, and the learning network 500 is configured by the generator 50 and the discriminator 52. FIG. 7A schematically illustrates an example of the process of the first training step of training the discriminator 52 in the learning network 500. On the other hand, FIG. 7B schematically illustrates an example of the process of the second training step for training the generator 50.

The generator 50 according to the present embodiment is configured by a multi-layered neural network used for so-called deep learning. Specifically, the generator 50 includes an input layer 501, an intermediate layer (hidden layer) 502, and an output layer 503. However, the configuration of the generator 50 does not have to be limited to such an example, and may be appropriately set according to the embodiment. For example, the generator 50 may include two or more intermediate layers 502.

The number of neurons (nodes) included in each layer 501 to 503 may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the examples of FIGS. 7A and 7B, each neuron is associated with all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment. A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 501 to 503 are examples of the parameters of the generator 50 used for arithmetic processing.

Similarly, the discriminator 52 according to the present embodiment is also configured by a multi-layered neural network used for so-called deep learning. Specifically, the discriminator 52 includes an input layer 521, an intermediate layer (hidden layer) 522, and an output layer 523. However, the configuration of the discriminator 52 does not have to be limited to such an example, and may be appropriately set according to the embodiment. For example, the discriminator 52 may include two or more intermediate layers 522. The configuration of the discriminator 52 may be different from that of the generator 50.

The number of neurons (nodes) included in each layer 521 to 523 may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the examples of FIGS. 7A and 7B, each neuron is associated with all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment. A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 521 to 523 are examples of the parameters of the discriminator 52 used for arithmetic processing.

The learning processing unit 114 carries out machine learning of the learning network 500. Performing this machine learning includes alternating a first training step for training the discriminator 52 and a second training step for training the generator 50. In the first training step, the learning processing unit 114 determines whether the input image input to the discriminator 52 is an image generated by the generator 50 or each patch image 61 divided from the learning image 60. The discriminator 52 is trained to do so. That is, the discriminator 52 is trained to discriminate whether the given input image is derived from the learning data (learning image 60) or the generator 50. In the second training step, the learning processing unit 114 trains the generator 50 so as to generate an image in which discrimination by the discriminator 52 is erroneous. In the examples of FIGS. 7A and 7B, the fact that the data is derived from the training data is expressed as "true", and the fact that the data is derived from the generator 50 is expressed as "false". However, the method for expressing each origin is not limited to such an example, and may be appropriately selected depending on the embodiment.

Specifically, as shown in FIG. 7A, in the first training step, the learning processing unit 114 extracts noise from a predetermined probability distribution, combines the extracted noise with each label 62, and a plurality of first data. Generate a set. Subsequently, the learning processing unit 114 inputs each first data set (each label 62 and noise) to the input layer 501 of the generator 50, and executes arithmetic processing of the generator 50. As a result, the learning processing unit 114 acquires the output corresponding to the result of generating the image from each label 62 from the output layer 503. The learning processing unit 114 generates a plurality of second data sets by combining each generated image with each label 62. Further, the learning processing unit 114 generates a plurality of third data sets by combining each patch image 61 with each corresponding label 62.

Next, the learning processing unit 114 inputs each second data set (each image generated by the generator 50 and each label 62) to the input layer 521 of the discriminator 52, and executes arithmetic processing of the discriminator 52. As a result, the learning processing unit 114 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning image 60 from the output layer 523. In this scene, since the input image is each image generated by the generator 50, the correct answer is that the discriminator 52 discriminates as "false". The learning processing unit 114 calculates an error between the output value obtained from the output layer 523 and this correct answer for each second data set.

Similarly, the learning processing unit 114 inputs each third data set (each patch image 61 and each label 62) to the input layer 521 of the discriminator 52, and executes arithmetic processing of the discriminator 52. As a result, the learning processing unit 114 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning image 60 from the output layer 523. In this scene, since the input image is each patch image 61, the correct answer is that the discriminator 52 determines that it is “true”. The learning processing unit 114 calculates an error between the output value obtained from the output layer 523 and this correct answer for each third data set.

Then, the learning processing unit 114 adjusts the value of the parameter of the discriminator 52 so that the sum of the calculated errors becomes small. The learning processing unit 114 repeats the adjustment of the parameter value of the discriminator 52 by the above series of processing until the sum of the error between the output value obtained from the output layer 523 and the true / false correct answer becomes equal to or less than the threshold value. As a result, in the first training step, the learning processing unit 114 trains the discriminator 52 so as to discriminate whether the image is an image generated by the generator 50 or each patch image 61 divided from the learning image 60. do.

On the other hand, as shown in FIG. 7B, in the second training step, the learning processing unit 114 extracts noise from a predetermined probability distribution and combines the extracted noise with each label 62 to generate a plurality of data sets. .. Each data set may be the same as or different from the first data set described above. Subsequently, the learning processing unit 114 inputs each data set (each label 62 and noise) to the input layer 501 of the generator 50, and executes arithmetic processing of the generator 50. As a result, the learning processing unit 114 acquires the output corresponding to the result of generating the image from each label 62 from the output layer 503.

Next, the learning processing unit 114 inputs each generated image and a combination of the corresponding labels 62 to the input layer 521 of the discriminator 52, and executes the arithmetic processing of the discriminator 52. As a result, the learning processing unit 114 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning image 60 from the output layer 523. In the training of the generator 50, the correct answer is that the result of the discrimination by the discriminator 52 is erroneous. That is, the correct answer is that the output value obtained from the output layer 523 corresponds to "true". The learning processing unit 114 calculates an error between the output value obtained from the output layer 523 by a series of processing and this correct answer (that is, “true”) for each data set.

Then, the learning processing unit 114 adjusts the value of the parameter of the generator 50 so that the sum of the calculated errors becomes small. For each data set, the learning processing unit 114 performs the parameter of the generator 50 by the above-mentioned series of processing until the sum of the errors between the output value obtained from the output layer 523 and the "true" by the series of processing becomes equal to or less than the threshold value. Repeat the adjustment of the value of. As a result, in the second training step, the learning processing unit 114 trains the generator 50 so as to generate an image from each label 62 that is erroneously discriminated by the discriminator 52.

The learning processing unit 114 alternately improves the accuracy of the discriminator 52 and the generator 50 by alternately performing the first training step and the second training step. As a result, the generator 50 can acquire the ability to appropriately generate an image corresponding to each patch image 61 divided from the training image 60.

After this machine learning is completed, the conservation processing unit 115 determines the configuration of the constructed generator 50 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, the transmission function of each neuron). And the first learning result data 125 showing the arithmetic parameters (eg, the weight of the connection between each neuron, the threshold of each neuron) is generated. Then, the storage processing unit 115 stores the generated first learning result data 125 in a predetermined storage area.

<Image generator>
Next, an example of the software configuration of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 8 schematically illustrates an example of the software configuration of the image generation device 2 according to the present embodiment.

The control unit 21 of the image generation device 2 expands the image generation program 221 stored in the storage unit 22 into the RAM. Then, the control unit 21 interprets and executes the image generation program 221 expanded in the RAM by the CPU to control each component. As a result, as shown in FIG. 8, the image generation device 2 according to the present embodiment operates as a computer including a generation unit 211, a coupling unit 212, and an image storage unit 213 as software modules. That is, in the present embodiment, each software module of the image generation device 2 is also realized by the control unit 21 (CPU) in the same manner as the learning device 1.

The generation unit 211 includes a trained generator 50 constructed by the learning device 1 by holding the first learning result data 125. The generation unit 211 uses the trained generator 50 constructed by the learning device 1 to generate each of the plurality of images 67 from each of the plurality of labels 62. In the present embodiment, the generation unit 211 sets the trained generator 50 with reference to the first learning result data 125. Then, the generation unit 211 acquires noise from a predetermined probability distribution, inputs the acquired noise and each label 62 to the input layer 501 of the generator 50, and executes arithmetic processing of the generator 50. As a result, the generation unit 211 acquires each image 67 generated from each label 62 as an output from the output layer 503. The joining portion 212 generates a bound image 69 by arranging each of the generated images 67 at a position corresponding to each label 62 and then joining them. The image storage unit 213 stores the generated combined image 69 in a predetermined storage area.

<Estimator generator>
Next, an example of the software configuration of the estimator generator 3 according to the present embodiment will be described with reference to FIG. 9. FIG. 9 schematically illustrates an example of the software configuration of the estimator generator 3 according to the present embodiment.

The control unit 31 of the estimator generation device 3 expands the estimator generation program 321 stored in the storage unit 32 into the RAM. Then, the control unit 31 controls each component by interpreting and executing the instruction included in the estimator generation program 321 expanded in the RAM by the CPU. As a result, as shown in FIG. 9, the estimator generator 3 according to the present embodiment is configured as a computer including a learning data acquisition unit 311, a learning processing unit 312, and a storage processing unit 313 as software modules. That is, in the present embodiment, each software module of the estimator generation device 3 is also realized by the control unit 31 (CPU) in the same manner as the learning device 1.

The learning data acquisition unit 311 acquires a plurality of learning data sets 322 each composed of a combination of a sample image 3221 showing a product and correct answer data 3222. The correct answer data 3222 shows the result (that is, the correct answer) of determining the quality of the product R shown in the corresponding sample image 3221. The sample image 3221 is used as input data (training data) for machine learning, and the correct answer data 3222 is used as teacher data.

Here, as the sample image 3221, the combined image 69 generated by the image generation device 2 may be used. That is, the sample image 3221 of at least a part of the training data set 322 may be the combined image 69. Further, the learning data acquisition unit 311 may transmit the sample image 3221 as the learning image 60 to the learning device 1 to construct a generator 50 for generating an image corresponding to the sample image 3221. Then, the learning data acquisition unit 311 may use the constructed generator 50 in the image generator 2 to generate a plurality of combined images 69 corresponding to the sample image 3221. The learning data acquisition unit 311 can increase the number of learning data sets 322 used for machine learning by receiving the generated plurality of combined images 69 as sample images 3221.

The learning processing unit 312 constructs a trained estimator 70 that has acquired the ability to judge the quality of the product R reflected in a given image by performing machine learning using each acquired learning data set 322. .. In other words, the learning processing unit 312 constructs an estimator 70 for each learning data set 322 that, when the sample image 3221 is input, outputs an output value that matches the correct answer data 3222. The storage processing unit 313 stores information about the constructed learned estimator 70 in a predetermined storage area.

In the relationship between the learning device 1 and the estimator generation device 3, the learning processing unit 114 and the storage processing unit 115 may be referred to as a first learning processing unit and a first storage processing unit, respectively. Accordingly, the learning processing unit 312 and the storage processing unit 313 may be referred to as a second learning processing unit and a second storage processing unit, respectively.

(Estimator)
Next, an example of the configuration of the estimator 70 according to the present embodiment will be described. As shown in FIG. 9, the estimator 70 according to the present embodiment is configured by a multi-layered neural network used for so-called deep learning, like the generator 50. Specifically, the estimator 70 includes an input layer 701, an intermediate layer (hidden layer) 702, and an output layer 703. However, the configuration of the estimator 70 does not have to be limited to such an example, and may be appropriately set according to the embodiment. For example, the estimator 70 may include two or more intermediate layers 702. The configuration of the estimator 70 may be different from that of the generator 50 and the discriminator 52.

The number of neurons (nodes) included in each layer 701 to 703 may be appropriately set according to the embodiment. Neurons in adjacent layers are appropriately connected to each other, and a weight (bonding load) is set for each connection. In the example of FIG. 9, each neuron is connected to all neurons in adjacent layers. However, the connection of neurons does not have to be limited to such an example, and may be appropriately set according to the embodiment. A threshold is set for each neuron, and basically, the output of each neuron is determined by whether or not the sum of the products of each input and each weight exceeds the threshold. The weight of the connection between each neuron and the threshold value of each neuron included in each layer 701 to 703 are examples of the parameters of the estimator 70 used for arithmetic processing.

The learning processing unit 312 inputs the sample image 3221 to the input layer 701 of the estimator 70 for each learning data set 322, and executes the arithmetic processing of the estimator 70. As a result of this arithmetic processing, the learning processing unit 312 acquires an output value corresponding to the result of determining the quality of the product R shown in the sample image 3221 from the output layer 703. Subsequently, the learning processing unit 312 calculates an error between the acquired output value and the correct answer data 3222. Then, the learning processing unit 312 adjusts the value of the parameter of the estimator 70 so that the sum of the calculated errors becomes small for each learning data set 322. The learning processing unit 312 repeats the adjustment of the parameter value of the estimator 70 by the above series of processing until the sum of the errors of the output value obtained from the output layer 703 and the correct answer data 3222 becomes equal to or less than the threshold value. As a result, when the sample image 3221 is input to the input layer 701 for each training data set 322, the learning processing unit 312 outputs an output value matching the correct answer data 3222 associated with the input sample image 3221 from the output layer 703. An estimator 70 trained to do so can be constructed.

After the processing of this machine learning is completed, the preservation processing unit 313 configures the constructed trained estimator 70 (for example, the number of layers of the neural network, the number of neurons in each layer, the connection relationship between neurons, and each neuron. (Transmission function of Then, the storage processing unit 313 stores the generated second learning result data 325 in a predetermined storage area.

<Inspection equipment>
Next, an example of the software configuration of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 10 schematically illustrates an example of the software configuration of the inspection device 4 according to the present embodiment.

The control unit 41 of the inspection device 4 expands the inspection program 421 stored in the storage unit 42 into the RAM. Then, the control unit 41 controls each component by interpreting and executing the instruction included in the inspection program 421 expanded in the RAM by the CPU. As a result, as shown in FIG. 10, the inspection device 4 according to the present embodiment is configured as a computer including the target data acquisition unit 411, the pass / fail determination unit 412, and the output unit 413 as software modules. In the present embodiment, each software module of the inspection device 4 is also realized by the control unit 41 (CPU) in the same manner as the learning device 1.

The target data acquisition unit 411 acquires the target image 422 of the product R that is the target of the visual inspection. In the present embodiment, the target data acquisition unit 411 acquires the target image 422 by photographing the product R with the camera CA. The pass / fail determination unit 412 includes the estimator 70 which has been constructed by the estimator generator 3 but has been learned by holding the second learning result data 325. The quality determination unit 412 determines the quality of the product R shown in the target image 422 by using the trained estimator 70.

Specifically, the pass / fail determination unit 412 sets the trained estimator 70 with reference to the second learning result data 325. Next, the pass / fail determination unit 412 inputs the acquired target image 422 to the input layer 701 of the estimator 70, and executes the arithmetic processing of the estimator 70. As a result, the quality determination unit 412 acquires an output value corresponding to the result of determining the quality of the product R shown in the target image 422 from the output layer 703 of the estimator 70. In the present embodiment, obtaining this output value corresponds to determining the quality of the product R. The output unit 413 outputs information regarding the result of determining the quality of the product R.

<Others>
Each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 will be described in detail in an operation example described later. In this embodiment, an example is described in which each software module of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4 is realized by a general-purpose CPU. However, some or all of the above software modules may be implemented by one or more dedicated processors. Further, with respect to the software configurations of the learning device 1, the image generation device 2, the estimator generation device 3, and the inspection device 4, software modules may be omitted, replaced, or added as appropriate according to the embodiment.

§3 Operation example [Learning device]
Next, an operation example of the learning device 1 according to the present embodiment will be described with reference to FIG. FIG. 11 shows an example of the processing procedure of the learning device 1 according to the present embodiment. The processing procedure described below is an example of the "learning method" of the present invention. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S101)
In step S101, the control unit 11 operates as an image acquisition unit 111 to acquire a learning image 60 showing the product R to be visually inspected.

The method for acquiring the learning image 60 is not particularly limited, and may be appropriately selected depending on the embodiment. For example, a camera and a product R are prepared, and a defective or non-defective product R is photographed by the camera. As a result, the learning image 60 can be generated. The generation of the learning image 60 may be automatically performed by the operation of the computer, or may be manually performed by the operation of the operator. Further, the information processing for generating the learning image 60 may be executed in the learning device 1 or may be performed by a computer other than the learning device 1.

When the learning device 1 generates the learning image 60, the control unit 11 acquires one or a plurality of learning images 60 by automatically or manually executing the information processing by the operation of the operator. On the other hand, when the learning image 60 is generated by another computer, the control unit 11 acquires one or a plurality of learning images 60 generated by the other computer via, for example, a network, a storage medium 91, or the like. In this case, the control unit 11 may acquire the sample image 3221 as the learning image 60 from the estimator generator 3 or another computer.

The number of learning images 60 to be acquired may not be particularly limited and may be appropriately determined according to the embodiment. When one or a plurality of learning images 60 are acquired, the control unit 11 proceeds to the next step S102.

(Step S102)
In step S102, the control unit 11 operates as an image segmentation unit 112 and divides the acquired learning image 60 into a plurality of patch images 61. Specifically, the control unit 11 appropriately determines the patch size, and divides the learning image 60 by the determined patch size. As a result, the control unit 11 can obtain a plurality of patch images 61 each having a patch size from the learning image 60.

The method for determining the patch size is not particularly limited and may be appropriately set according to the embodiment. For example, the patch size may be fixed by a specified value, a set value, or the like. However, if the patch size is fixed, the learning image 60 may not be divided into patch images 61 having a patch size suitable for generation. Therefore, in the present embodiment, as a preferable mode of patch division, the control unit 11 determines the patch size by the method shown in FIG. 12 below, and divides the learning image 60 by the determined patch size.

FIG. 12 illustrates an example of the patch division processing procedure by the learning device 1 according to the present embodiment. Step S102 according to the present embodiment includes the following steps S201 to S203. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S201)
In step S201, the control unit 11 calculates the complexity of the learning image 60. The complexity is an index showing the complexity of the training image 60. The complexity may be derived as appropriate. In the present embodiment, the control unit 11 uses five indexes of information entropy (E), boundary ratio (R), contrast (CT), correlation (CL), and energy (EN) to complicate the training image 60. Derive the degree. However, the index of complexity does not have to be limited to these examples, and may be appropriately selected depending on the embodiment. For example, in deriving the complexity of the training image 60, at least one of these five indicators may be omitted.

The information entropy (E) indicates an expected value of the amount of information possessed by the learning image 60. The control unit 11 can calculate the information entropy (E) by the following equation 1. The boundary ratio (R) indicates the ratio of the boundary points to the total number of pixels of the training image 60. Boundary points are points that belong to the boundaries of different image areas. For example, an edge detected by the Sobel method, the Canny method, the Prewitt method, or the like can be used as a boundary point. The control unit 11 can calculate the boundary ratio (R) by the following equation 2. Contrast (CT indicates the degree of difference between a pixel and nearby pixels. Contrast (CT) may also be referred to as "dispersion" or "inertia." Contrast (CT) can be calculated by (CL) can be calculated. Energy (EN) indicates the sum of squares of the elements of the Gray-level Co-occurrence Matrix (GLCM). Energy (EN) is "uniformity". The energy (EN) can be calculated by the control unit 11 by the following equation 5. The simultaneous occurrence matrix of the gray level is the gray level (gray scale). Intensity) Derives the frequency with which a pixel with a value i appears adjacent to a pixel with a value j in a specified direction. The four boundary ratios (R), contrast (CT), correlation (CL), and energy (EN). The index is derived from this gray scale simultaneous occurrence matrix. The control unit 11 calculates the complexity (CP) from these five indexes by the following equation 6.

ここで、ｋは、学習画像６０内に存在する画素値の種類数を示す。ｎ_iは、ｉ番目の種類の画素数を示す。Ｎは、学習画像６０内に存在する画素数を示す。Ｐｅは、境界点の数を示す。Ｗは、学習画像６０の横方向の画素数を示す。Ｈは、学習画像６０の縦方向の画素数を示す。ＷとＨとの積によりＮを算出することができる。ｐ（ｉ，ｊ）は、同時生起行列の要素を示す。ｕ_i、ｕ_j、ａ_i、及びａ_jはそれぞれ、以下の式７～式１０に示される。

Here, k indicates the number of types of pixel values existing in the learning image 60. n _i indicates the number of pixels of the i-th type. N indicates the number of pixels existing in the learning image 60. Pe indicates the number of boundary points. W indicates the number of pixels in the horizontal direction of the learning image 60. H indicates the number of pixels in the vertical direction of the learning image 60. N can be calculated from the product of W and H. p (i, j) indicates an element of the simultaneous occurrence matrix. u _i , u _j , a _i , and a _j are represented by the following equations 7 to 10, respectively.

When determining a patch size common to a plurality of learning images 60, the control unit 11 acquires the averaged learning image (average image) 60 by averaging the plurality of learning images 60. .. Then, the control unit 11 calculates the complexity of the averaged learning image 60 by the above method. After calculating the complexity of the learning image 60, the control unit 11 proceeds to the next step S202.

(Step S202)
In step S202, the control unit 11 determines the patch size according to the calculated complexity. In the present embodiment, the control unit 11 determines the patch size so that the higher the complexity, the smaller the patch size, and the lower the complexity, the larger the patch size. For example, patch sizes may be prepared in a plurality of stages such as 32 × 32, 64 × 64, 128 × 128, 256 × 256, and the like. Accordingly, a numerical range corresponding to each stage of patch size may be set for complexity. The patch size of each stage may be appropriately set according to the size of the training image 60 and the number of patch images 61 to be generated. For example, as shown in Table 1 below, a numerical range of complexity may be set for each stage of patch size.

この場合、制御部１１は、算出した複雑度がいずれの数値範囲に属するかを判定してよい。そして、制御部１１は、算出した複雑度の属する数値範囲に対応して、パッチサイズを決定してもよい。パッチサイズを決定すると、制御部１１は、次のステップＳ２０３に処理を進める。

In this case, the control unit 11 may determine which numerical range the calculated complexity belongs to. Then, the control unit 11 may determine the patch size according to the numerical range to which the calculated complexity belongs. After determining the patch size, the control unit 11 proceeds to the next step S203.

(Step S203)
In step S203, the control unit 11 divides the learning image 60 into a plurality of patch images 61 each having a determined patch size. For example, when the size of the training image 60 is 1280 × 1024 and the patch size is determined to be 256 × 256 in step S202, the training image 60 is divided into 20 patch images 61. When a plurality of learning images 60 are acquired in step S101, the control unit 11 divides each learning image 60 by a determined patch size. When the division of the learning image 60 is completed, the control unit 11 ends the patch division process and proceeds to the next step S103.

(Step S103)
Returning to FIG. 11, in step S103, the control unit 11 operates as the label giving unit 113, and depending on the position of each patch image 61 in the learning image 60, one of the plurality of labels 62 is labeled 62. Associated with each patch image 61. Each label 62 indicates the position of each patch image 61 in the training image 60.

The correspondence between each patch image 61 and each label 62 may be appropriately determined according to the embodiment. The control unit 11 may assign a different label 62 to each patch image 61. However, in this case, the number of types of the label 62 increases, and the configuration of the generator 50 may be complicated accordingly. In the present embodiment, the learning image 60 is an image showing the product R to be visually inspected, and each patch image 61 corresponding to the background portion thereof may be the same (that is, there is no difference). expensive. If each matching patch image 61 is integrated into the same label 62, the number of types of the label 62 can be reduced, which can further simplify the configuration of the generator 50. Therefore, in the present embodiment, as a preferred embodiment of labeling, the control unit 11 determines the correspondence between each patch image 61 and each label 62 by the method shown in FIG. 13 below.

FIG. 13 illustrates an example of the labeling processing procedure by the learning device 1 according to the present embodiment. Step S103 according to this embodiment includes the following steps S301 to 305. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S301)
In step S301, the control unit 11 calculates the complexity of each patch image 61. The method of calculating the complexity of each patch image 61 may be the same as in step S201. However, the method of calculating the complexity of the training image 60 and the method of calculating the complexity of each patch image 61 do not necessarily have to match. For example, the control unit 11 derives the complexity of the learning image 60 from the above five indexes, whereas the control unit 11 omits at least one of the above five indexes in deriving the complexity of each patch image 61. May be good. Thereby, the method of calculating the complexity of the training image 60 and the method of calculating the complexity of each patch image 61 may be different. After calculating the complexity of each patch image 61, the control unit 11 proceeds to the next step S302.

(Steps S302 to S304)
In steps S302 to S304, the control unit 11 integrates two or more patch images 61 similar to each other among the plurality of patch images 61 into the same label 62 based on the calculated complexity.

Specifically, in step S302, the control unit 11 determines whether or not there are two or more patch images 61 that are similar to each other based on the calculated complexity. The complexity of two similar patch images 61 is close to each other. That is, if the contents of the two patch images 61 are substantially the same, the complexity of the two patch images 61 is substantially the same. Therefore, the control unit 11 calculates the difference in complexity between the two patch images 61, and determines whether or not the calculated difference is equal to or less than the threshold value. The threshold value may be appropriately set according to the embodiment. Then, when the calculated difference is equal to or less than the threshold value, the control unit 11 determines that the two target patch images 61 are similar to each other. On the other hand, when the calculated difference is not equal to or less than the threshold value, the control unit 11 determines that the two target patch images 61 are not similar. The control unit 11 executes this determination process for all the patch images 61. Thereby, based on the calculated complexity, it is possible to specify whether or not there are two or more patch images similar to each other.

The complexity level represents the content of the image based on each of the above indexes. Therefore, although the difference in complexity is relatively small, the two patch images 61 may differ greatly from each other. Therefore, in step S302, the control unit 11 may further use an index other than the above complexity as an index for determining whether or not the two target patch images 61 are similar to each other. The processing order of the determination based on the complexity and the determination based on other indicators may be appropriately determined according to the embodiment. As an example of the determination method based on other indexes, for example, a determination method based on a monochrome histogram can be mentioned. When the determination method based on the monochrome histogram is adopted, the control unit 11 generates the monochrome histogram of each patch image 61. The monochrome histogram shows the number of pixels of each luminance in the image, that is, the statistical information of the luminance of the image. Next, the control unit 11 calculates the difference between the monochrome histograms of the two patch images 61, and determines whether or not the calculated difference is equal to or less than the threshold value. The threshold value may be appropriately determined according to the embodiment. The threshold value for the difference in complexity may be referred to as a first threshold value, and the threshold value for the difference in monochrome histogram may be referred to as a second threshold value. When the difference in the calculated complexity is less than or equal to the first threshold value and the difference in the calculated monochrome histogram is less than or equal to the second threshold value, the control unit 11 determines that the two patch images 61 of the target are similar to each other. Is determined. On the other hand, if this is not the case, the control unit 11 determines that the two target patch images 61 are not similar. The control unit 11 executes these determination processes for all the patch images 61. As described above, in step S302, the control unit 11 may execute the determination process based on other indexes in addition to the determination process based on the complexity.

In step S303, the control unit 11 processes the conditional branch according to the determination result of step S302. If it is determined in step S302 that two or more patch images 61 similar to each other exist, the control unit 11 proceeds to the next step S304. On the other hand, if it is determined in step S302 that two or more patch images 61 similar to each other do not exist, the control unit 11 omits step S304 and proceeds to the next step S305.

In step S304, the control unit 11 integrates two or more patch images 61 that are similar to each other. That is, the control unit 11 is set so that the same label 62 is given to two or more patch images 61 that are similar to each other. There may be a plurality of groups of two or more patch images 61 that are similar to each other. In this case, the control unit 11 is set to give the same label 62 to each patch image 61 in the same group, but is set to give different labels 62 to the patch images 61 of different groups. do. When the integration process of the patch image 61 is completed, the control unit 11 proceeds to the next step S305.

(Step S305)
In step S305, the control unit 11 associates (assigns) the label 62 with each patch image 61. In this process, the control unit 11 assigns the same label 62 to the two or more patch images 61 integrated in step S304. On the other hand, the control unit 11 assigns different labels 62 to different patch images 61 that are not integrated.

In the process of this labeling, the control unit 11 generates label information 621 indicating the correspondence between each patch image 61 and each label 62. The control unit 11 stores the generated label information 621 in a predetermined storage area. The data format of the label information 621 and the predetermined storage area may not be particularly limited, and may be appropriately selected depending on the embodiment. The predetermined storage area may be, for example, a RAM in the control unit 11, a storage unit 12, an external storage device, a storage medium, or a combination thereof. The storage medium may be, for example, a CD, a DVD, or the like, and the control unit 11 may store the label information 621 in the storage medium via the drive 16. The external storage device may be, for example, an external storage device connected to the learning device 1. In this case, the control unit 11 may store the label information 621 in an external storage device. Further, the external storage device may be, for example, a data server such as NAS (Network Attached Storage). In this case, the control unit 11 may store the label information 621 in the data server via the network.

When all the patch images 61 have been given any of the labels 62, the control unit 11 ends the labeling process according to the present embodiment, and proceeds to the next step S104.

(Step S104)
Returning to FIG. 11, the control unit 11 operates as a learning processing unit 114, and performs machine learning of the generator 50 by using each patch image 61 and each label 62. As a result, the control unit 11 constructs a generator 50 trained to generate an image corresponding to each patch image 61 from the label 62 associated with each patch image 61. In the present embodiment, the output of the generator 50 is connected to the input of the discriminator 52, and the learning network 500 is configured by the generator 50 and the discriminator 52. Therefore, in the present embodiment, the control unit 11 carries out machine learning of the learning network 500 by the method shown in FIG. 14 below.

FIG. 14 illustrates an example of a machine learning processing procedure by the learning device 1 according to the present embodiment. Step S104 according to the present embodiment includes the following steps S401 to 403. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

Before executing step S401, the control unit 11 prepares a generator 50 and a discriminator 52 to be processed. The configuration of each of the generator 50 and the discriminator 52 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or given by the input of the operator. May be good. Further, in the case of re-learning, the control unit 11 may prepare the generator 50 and the discriminator 52 based on the learning result data obtained by performing the past machine learning. When this preprocessing is completed, the control unit 11 executes the machine learning process from the next step S401.

(Step S401)
In step S401, the control unit 11 performs machine learning of the discriminator 52, so that the input image input to the discriminator 52 is an image generated by the generator 50 or divided from the learning image 60. The discriminator 52 is trained to discriminate whether it is each patch image 61. Step S401 is an example of the "first training step" of the present invention.

Specifically, the control unit 11 extracts noise (latent variable) from a predetermined probability distribution, combines the extracted noise with each label 62, and generates a plurality of first data sets. The predetermined probability distribution may be, for example, a Gaussian distribution, a uniform distribution, or the like. Subsequently, the control unit 11 inputs each first data set (each label 62 and noise) to the input layer 501 of the generator 50, and determines the firing of each neuron included in each layer 501 to 503 in order from the input side. .. As a result, the control unit 11 acquires the output corresponding to the result of generating the image from each label 62 from the output layer 503. The control unit 11 generates a plurality of second data sets by combining each generated image and each label 62. Further, by combining each patch image 61 and each corresponding label 62, a plurality of third data sets are generated.

Next, the control unit 11 uses each second data set and each third data set as input data (training data), and uses the origin of the image included in each data set as teacher data (correct answer data). , The learning process of the discriminator 52 is executed. A stochastic gradient descent method or the like may be used for this learning process.

For example, the control unit 11 inputs each second data set (each image generated by the generator 50 and each label 62) to the input layer 521 of the discriminator 52, and is included in each of the layers 521 to 523 in order from the input side. Judge the firing of neurons. As a result, the control unit 11 acquires an output value corresponding to the result of determining whether the input image included in the data set is derived from the generator 50 or the learning image 60 from the output layer 523. In this scene, since the input image is each image generated by the generator 50, the correct answer is that the discriminator 52 discriminates as "false". The learning processing unit 114 calculates an error between the output value obtained from the output layer 523 and this correct answer for each second data set.

Similarly, the control unit 11 inputs each third data set (each patch image 61 and each label 62) to the input layer 521 of the discriminator 52, and fires each neuron included in each layer 521 to 523 in order from the input side. Make a judgment. As a result, the control unit 11 acquires an output value corresponding to the result of determining whether the input image included in the data set is derived from the generator 50 or the learning image 60 from the output layer 523. In this scene, since the input image is each patch image 61, the correct answer is that the discriminator 52 determines that it is “true”. The learning processing unit 114 calculates an error between the output value obtained from the output layer 523 and this correct answer for each third data set.

Then, the control unit 11 calculates the weight of the connection between each neuron in the discriminator 52 and the error of each threshold value of each neuron by using the error of the output value calculated by the error back propagation method (Back propagation). The control unit 11 updates the weight of the connection between each neuron and the value of each threshold value of each neuron in the discriminator 52 based on each calculated error. For each data set, the control unit 11 repeats the adjustment of the parameter values by this series of processing until the sum of the errors between the output value obtained from the output layer 523 and the correct answer from which the input image is derived becomes equal to or less than the threshold value. The threshold value may be appropriately set according to the embodiment. Thereby, the control unit 11 can appropriately determine whether the input input image for each data set is an image generated by the generator 50 or each patch image 61 divided from the learning image 60. A trained discriminator 52 can be constructed. When the machine learning of the discriminator 52 is completed, the control unit 11 proceeds to the next step S402.

(Step S402)
In step S402, the control unit 11 trains the generator 50 so as to generate an image in which the discriminator 52 makes a mistake in discrimination by performing machine learning of the generator 50. Step S402 is an example of the "second training step" of the present invention.

Specifically, the control unit 11 extracts noise from a predetermined probability distribution, combines the extracted noise with each label 62, and generates a plurality of data sets. Each data set may be the same as or different from the first data set described above. Subsequently, the control unit 11 inputs each data set (each label 62 and noise) to the input layer 501 of the generator 50, and determines the firing of each neuron included in each layer 501 to 503 in order from the input side. As a result, the control unit 11 acquires the output corresponding to the result of generating the image from each label 62 from the output layer 503.

Next, the control unit 11 uses each data set as input data (training data), and uses the misjudgment of the discriminator 52 as teacher data (correct answer data) to execute the learning process of the generator 50. do. Similar to step S401, a stochastic gradient descent method or the like may be used for this learning process.

For example, the control unit 11 inputs the generated image and the combination of the corresponding labels 62 into the input layer 521 of the discriminator 52, and determines the firing of each neuron included in each layer 521 to 523 in order from the input side. .. As a result, the control unit 11 acquires an output value corresponding to the result of determining whether the input image is derived from the generator 50 or the learning image 60 from the output layer 523. In the training of the generator 50, the correct answer is that the result of the discrimination by the discriminator 52 is erroneous. That is, the correct answer is that the output value obtained from the output layer 523 corresponds to "true". The control unit 11 calculates an error between the output value obtained from the output layer 523 by a series of processes and this correct answer (that is, “true”) for each data set.

Then, the control unit 11 calculates the error of the connection weight between each neuron in the generator 50 and the error of each threshold value of each neuron by using the error of the output value calculated by the error back propagation method. The control unit 11 updates the weight of the connection between each neuron in the generator 50 and the value of each threshold value of each neuron based on each calculated error. For each data set, the control unit 11 repeats the adjustment of the parameter value by this series of processing until the sum of the error between the output value obtained from the output layer 523 and the “true” by the series of processing becomes equal to or less than the threshold value. .. The threshold value may be appropriately set according to the embodiment. As a result, the control unit 11 can construct a generator 50 trained so that an image that is erroneously discriminated by the discriminator 52 can be generated from each label 62 for each data set. When the machine learning of the generator 50 is completed, the control unit 11 proceeds to the next step S403.

(Step S403)
In step S403, the control unit 11 determines whether or not to repeat the machine learning process of steps S401 and S402. The criteria for repeating the machine learning process may be appropriately determined according to the embodiment. For example, the number of times to perform machine learning in steps S401 and S402 may be set. In this case, the control unit 11 determines whether or not the number of times the machine learning of steps S401 and S402 is performed reaches the set number of times. If it is determined that the number of executions of steps S401 and S402 has not reached the set number of times, the control unit 11 returns the process to step S401. On the other hand, when it is determined that the number of executions of steps S401 and S402 has reached the set number of times, the control unit 11 ends the machine learning process according to the present embodiment and proceeds to the next step S105.

In this way, the control unit 11 alternately improves the accuracy of the discriminator 52 and the generator 50 by alternately performing the processes of steps S401 and S401. As a result, the generator 50 can acquire the ability to appropriately generate an image corresponding to each patch image 61 divided from the training image 60.

(Step S105)
Returning to FIG. 11, in step S105, the control unit 11 operates as the storage processing unit 115, and stores information about the constructed trained generator 50 in a predetermined storage area. In the present embodiment, the control unit 11 generates information indicating the configuration and parameters of the trained generator 50 constructed by step S104 as the first learning result data 125. The first learning result data 125 may or may not include information indicating the configuration and parameters of the discriminator 52. Then, the control unit 11 stores the generated first learning result data 125 in a predetermined storage area. The predetermined storage area is, for example, a RAM in the control unit 11, a storage unit 12, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof, similarly to the storage destination of the label information 621. It may be there. As a result, the control unit 11 ends the process related to this operation example.

After constructing the trained generator 50, the control unit 11 may transfer the generated first learning result data 125 to the image generator 2 at an arbitrary timing. The image generation device 2 may acquire the first learning result data 125 by accepting the transfer from the learning device 1, or may acquire the first learning result data 125 by accessing the learning device 1 or the data server. May be good. The first learning result data 125 may be incorporated in the image generation device 2 in advance.

Further, the control unit 11 may periodically update the first learning result data 125 by periodically repeating the processes of steps S101 to S105. When this is repeated, the learning image 60 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 11 periodically updates the first learning result data 125 held by the image generation device 2 by transferring the updated first learning result data 125 to the image generation device 2 every time the machine learning is executed. You may.

[Image generator]
Next, an operation example of the image generation device 2 according to the present embodiment will be described with reference to FIG. FIG. 15 shows an example of the processing procedure of the image generation device 2 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S501)
In step S501, the control unit 21 operates as the generation unit 211, and uses the trained generator 50 constructed by the learning device 1 to generate each of the plurality of images 67 from each of the plurality of labels 62.

In the present embodiment, the control unit 21 sets the trained generator 50 with reference to the first learning result data 125. The control unit 21 identifies the label 62 assigned to each patch image 61 with reference to the label information 621. Subsequently, the control unit 21 acquires noise (latent variable) from a predetermined probability distribution. The predetermined probability distribution may be, for example, a Gaussian distribution, a uniform distribution, or the like, as described above. The control unit 21 inputs the acquired noise and each label 62 to the input layer 501 of the generator 50, and determines the firing of each neuron included in each layer 501 to 503 in order from the input side. As a result, each image 67 generated from each label 62 is acquired as an output from the output layer 503. When each image 67 is generated for each label 62, the control unit 21 proceeds to the next step S502.

(Step S502)
In step S502, the control unit 21 operates as the coupling unit 212, and with reference to the label information 621, arranges each generated image 67 at a position corresponding to each label 62. After that, the control unit 21 combines the images 67 arranged at the positions corresponding to the labels 62. As a result, the control unit 21 generates the combined image 69. The combined image 69 corresponds to the learning image 60. When the generation of the combined image 69 is completed, the control unit 21 proceeds to the next step S503.

(Step S503)
In step S503, the control unit 21 operates as the image storage unit 213 and stores the generated combined image 69 in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 21, a storage unit 22, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof. As a result, the control unit 21 ends the process related to this operation example.

The control unit 21 may generate a plurality of combined images 69 by repeatedly executing the steps S502 and S502. The number of combined images 69 to be generated may not be particularly limited and may be appropriately selected depending on the embodiment. Further, the control unit 21 may transfer the generated combined image 69 to the estimator generation device 3 in order to use the generated combined image 69 as the sample image 3221.

[Estimator generator]
Next, an operation example of the estimator generator 3 according to the present embodiment will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of the processing procedure of the estimator generator 3 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S701)
In step S701, the control unit 31 operates as a learning data acquisition unit 311 and acquires a plurality of learning data sets 322 each composed of a combination of a sample image 3221 showing a product and correct answer data 3222. The correct answer data 3222 shows the result (that is, the correct answer) of determining the quality of the product R shown in the corresponding sample image 3221.

The method for acquiring the training data set 322 does not have to be particularly limited, and may be appropriately determined according to the embodiment. For example, a camera and a product R are prepared, and a defective or non-defective product R is photographed by the camera. This makes it possible to generate a sample image 3221. Then, the correct answer data 3222 indicating the quality of the product R reflected in the sample image 3221 is appropriately generated, and the generated correct answer data 3222 is associated with the corresponding sample image 3221. As a result, each training data set 322 can be generated.

The generation of the learning data set 322 may be automatically performed by the operation of the computer, or may be manually performed by the operation of the operator. Further, the information processing for generating the learning data set 322 may be executed by the estimator generator 3 or may be performed by a computer other than the estimator generator 3.

When the estimator generator 3 generates the learning data set 322, the control unit 31 acquires a plurality of learning data sets 322 by automatically or manually executing the above information processing by the operation of the operator. On the other hand, when the learning data set 322 is generated by another computer, the control unit 31 acquires a plurality of learning data sets 322 generated by the other computer via, for example, a network, a storage medium 93, or the like. The estimator generator 3 may generate a part of the training data set 322, and another computer may generate the remaining training data set 322.

Here, the sample image 3221 included in at least a part of the acquired training data set 322 may be a combined image 69 generated by the image generation device 2. The control unit 31 may acquire the combined image 69 generated by the image generation device 2 as a sample image 3221 via a network, a storage medium 93, or the like. The generation of the training data set 322 including the combined image 69 (sample image 3221) may be performed by the image generation device 2. That is, the image generation device 2 is an example of the above-mentioned other computer.

The data format of the correct answer data 3222 may be appropriately determined according to the output format of the estimator 70. For example, the correct answer data 3222 may be set so as to indicate the quality of the product R by two values. Further, for example, the correct answer data 3222 may be set so as to indicate the probability that the product R is a non-defective product (that is, there is no defect) or the probability that the product R has a defect as a continuous value. Further, for example, the correct answer data 3222 may be set to indicate the index (class) of the quality or defect type of the product R. Further, for example, the correct answer data 3222 may be set to indicate a defective portion or the like.

The number of learning data sets 322 to be acquired may not be particularly limited and may be appropriately determined according to the embodiment. When the plurality of training data sets 322 are acquired, the control unit 31 proceeds to the next step S702.

(Step S702)
In step S702, the control unit 31 operates as the learning processing unit 312, and performs machine learning of the estimator 70 by using the plurality of learning data sets 322. In this machine learning, the control unit 31 trains the estimator 70 so that when the sample image 3221 is input to the input layer 701 for each training data set 322, an output value matching the correct answer data 3222 is output from the output layer 703. do. As a result, the control unit 31 constructs a trained estimator 70 that has acquired the ability to determine the quality of the product R.

This machine learning may be executed in the same manner as the machine learning by the learning device 1. That is, the control unit 31 prepares the estimator 70 to be processed. The configuration of the estimator 70 to be prepared, the initial value of the weight of the connection between each neuron, and the initial value of the threshold value of each neuron may be given by the template or by the input of the operator. Further, in the case of re-learning, the control unit 31 may prepare the estimator 70 based on the learning result data obtained by performing the past machine learning.

Next, the control unit 31 uses the sample image 3221 included in each learning data set 322 acquired in step S701 as input data (training data), and uses the corresponding correct answer data 3222 as teacher data to be an estimator. 70 learning processes are executed. A stochastic gradient descent method or the like may be used for this learning process.

For example, in the first step, the control unit 31 inputs the sample image 3221 to the input layer 701 for each learning data set 322, and determines the firing of each neuron included in each layer 701 to 703 in order from the input side. As a result, the control unit 31 acquires an output value corresponding to the result of determining the quality of the product R shown in the sample image 3221 from the output layer 703. In the second step, the control unit 31 calculates an error between the acquired output value and the correct answer data 3222. In the third step, the control unit 31 calculates the error of the connection weight between each neuron in the estimator 70 and the error of each threshold value of each neuron by using the error of the calculated output value by the error back propagation method. In the fourth step, the control unit 31 updates the weight of the connection between each neuron and the value of each threshold value of each neuron in the estimator 70 based on each calculated error.

By repeating the first to fourth steps, the control unit 31 outputs an output value that matches the corresponding correct answer data 3222 when the sample image 3221 is input for each training data set 322. Adjust the values of 70 parameters. In other words, for each learning data set 322, the control unit 31 is an estimator according to the first to fourth steps until the sum of the errors between the output value obtained from the output layer 703 and the correct answer data 3222 becomes equal to or less than the threshold value. Repeat the adjustment of the value of the parameter of 70. The threshold value may be appropriately set according to the embodiment. As a result, the control unit 31 trains the estimator 70 to output an output value matching the correct answer data 3222 from the output layer 703 when the sample image 3221 is input to the input layer 701 for each training data set 322. Can be built. When the machine learning of the estimator 70 is completed, the control unit 31 proceeds to the next step S703.

(Step S703)
In step S703, the control unit 31 operates as a storage processing unit 313 and stores information about the constructed learned estimator 70 in a predetermined storage area. In the present embodiment, the control unit 31 generates information indicating the configuration and parameters of the trained estimator 70 constructed by step S702 as the second learning result data 325. Then, the control unit 31 stores the generated second learning result data 325 in a predetermined storage area. The predetermined storage area may be, for example, a RAM in the control unit 31, a storage unit 32, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof. As a result, the control unit 31 ends the process related to this operation example.

After constructing the trained estimator 70, the control unit 31 may transfer the generated second learning result data 325 to the inspection device 4 at an arbitrary timing. The inspection device 4 may acquire the second learning result data 325 by accepting the transfer from the estimator generator 3, or may acquire the second learning result data 325 by accessing the estimator generator 3 or the data server. You may get it. The second learning result data 325 may be incorporated in the inspection device 4 in advance.

Further, the control unit 31 may periodically update the second learning result data 325 by periodically repeating the processes of steps S701 to 703. When this is repeated, the training data set 322 may be changed, modified, added, deleted, or the like as appropriate. Then, the control unit 31 periodically updates the second learning result data 325 held by the inspection device 4 by transferring the updated second learning result data 325 to the inspection device 4 every time the machine learning is executed. May be good.

Further, the control unit 31 may evaluate the determination performance of the constructed estimator 70 by using the evaluation data set. The evaluation data set can be configured in the same manner as each training data set 322 described above. That is, the evaluation data set may be composed of a combination of a sample image showing the product R and correct answer data indicating the quality of the product R shown in the sample image. As described in step S902 described later, the control unit 31 uses the estimator 70 to determine the quality of the product R shown in the sample image of the evaluation data set. The control unit 31 can evaluate the determination performance of the estimator 70 by collating the determination result with the correct answer indicated by the correct answer data.

When the determination performance of the estimator 70 is equal to or less than a predetermined standard (for example, the correct answer rate is equal to or less than the threshold value), the control unit 31 may use one or a plurality of learning data sets selected from the plurality of learning data sets 322. Each sample image 3221 included in 322 may be transmitted to the learning device 1. Next, the control unit 31 may cause the learning device 1 to construct a generator 50 for generating an image corresponding to each transmitted sample image 3221. Further, the control unit 31 transfers the trained generator 50 to the image generator 2 by the learning device 1, and executes a process of generating the combined image 69 by using the trained generator 50 in the image generator 2. You may let me.

Accordingly, one or more combined images 69 generated by the image generator 2 may be received as the sample image 3221. Then, the control unit 31 may generate a learning data set 322 including the received combined image 69 (sample image 3221), and may add the generated learning data set 322 to the original learning data group. As a result, the control unit 31 can increase the number of learning data sets 322 used for machine learning. The control unit 31 may perform machine learning of the estimator 70 again by using this new learning data group. By this series of re-learning processes, the determination performance of the constructed trained estimator 70 can be improved.

[Inspection equipment]
Next, an operation example of the inspection device 4 according to the present embodiment will be described with reference to FIG. FIG. 17 is a flowchart illustrating an example of the processing procedure of the inspection device 4 according to the present embodiment. However, the processing procedure described below is only an example, and each processing may be changed as much as possible. Further, with respect to the processing procedure described below, steps can be omitted, replaced, and added as appropriate according to the embodiment.

(Step S901)
In step S901, the control unit 41 operates as the target data acquisition unit 411 to acquire the target image 422 of the product R to be visually inspected. In the present embodiment, the inspection device 4 is connected to the camera CA via the external interface 47. Therefore, the control unit 41 acquires the target image 422 from the camera CA. The target image 422 may be moving image data or still image data. When the target image 422 is acquired, the control unit 41 proceeds to the next step 902.

However, the route for acquiring the target image 422 does not have to be limited to such an example, and may be appropriately selected according to the embodiment. For example, another information processing device different from the inspection device 4 may be connected to the camera CA. In this case, the control unit 41 may acquire the target image 422 via another information processing device.

(Step S902)
In step S902, the control unit 41 operates as a quality determination unit 412, and uses the trained estimator 70 to determine the quality of the product R shown in the target image 422.

Specifically, the control unit 41 sets the trained estimator 70 with reference to the second learning result data 325. Next, the control unit 41 inputs the acquired target image 422 to the input layer 701 of the estimator 70, and determines the firing of each neuron included in each layer 701 to 703 in order from the input side. As a result, the control unit 41 acquires an output value corresponding to the result of determining the quality of the product R reflected in the target image 422 from the output layer 703 of the estimator 70.

As a result, the control unit 41 determines the quality of the product R reflected in the target image 422 based on the output value acquired from the output layer 703 of the estimator 70. It should be noted that the quality determination may be appropriately performed according to the output format of the estimator 70. For example, when the output value obtained from the estimator 70 indicates the quality of the product R by two values, the control unit 41 can specify the quality of the product R according to the output value obtained from the estimator 70. Further, for example, when the output value obtained from the estimator 70 indicates the probability that the product R is a non-defective product or the probability that the product R has a defect as a continuous value, the control unit 41 has the output value obtained from the estimator 70. By comparing with the threshold value, the quality of the product R can be determined. Further, for example, when the output value obtained from the estimator 70 indicates a defective portion, the control unit 41 determines the quality of the product R reflected in the target image 422 based on the output value obtained from the estimator 70. At the same time, if a defect exists, the location of the defect can be identified.

Further, for example, when the output value obtained from the estimator 70 indicates the index of the quality or defect type of the product R, the inspection device 4 determines the output value obtained from the estimator 70 and the quality or defect type of the product R. Reference information (not shown) such as a table format associated with may be stored in the storage unit 42. In this case, the control unit 41 can determine the quality of the product R shown in the target image 422 according to the output value obtained from the estimator 70 by referring to this reference information.

As described above, the control unit 41 can determine the quality of the product R shown in the target image 422 by using the estimator 70. When the quality determination of the product R is completed, the control unit 41 proceeds to the next step S903.

(Step S903)
In step S903, the control unit 41 operates as an output unit 413 and outputs the result of determining the quality of the product R in step S902.

The output format as a result of determining the quality of the product R does not have to be particularly limited, and may be appropriately selected according to the embodiment. For example, the control unit 41 may output the result of determining the quality of the product R to the output device 45 as it is. Further, when it is determined in step S902 that the product R has a defect, the control unit 41 may issue a warning for notifying that the defect has been found as the output process of this step S903. Further, the control unit 41 may execute a predetermined control process according to the result of determining the quality of the product R as the output process of this step S903. As a specific example, when the inspection device 4 is connected to the production line for transporting the product, when the control unit 41 determines that the product R has a defect, the control unit 41 regards the defective product R as a defect-free product. May perform the process of transmitting a command to be conveyed by a different route to the production line as the output process of this step S903.

When the output processing as a result of determining the quality of the product R is completed, the control unit 41 ends the processing related to this operation example. The control unit 41 may execute a series of processes of steps S901 to S903 each time the product R conveyed on the production line enters the shooting range of the camera CA. As a result, the inspection device 4 can inspect the appearance of the product R transported on the production line.

[feature]
As described above, the learning device 1 according to the present embodiment is trained to divide the learning image 60 into a plurality of patch images 61 in step S102 and generate an image corresponding to each patch image 61 in step S104. Build the vessel 50. As a result, the generator 50 does not acquire the ability to directly generate the image corresponding to the full-size learning image 60, but corresponds to the learning image 60 through the generation of the image corresponding to each patch image 61. The generator 50 is trained in the ability to indirectly generate a full-size image. Therefore, in the present embodiment, the size of the image to be learned by the generator 50 can be suppressed, so that the configuration of the generator 50 can be simplified. Therefore, according to the present embodiment, it is possible to reduce the calculation cost required to construct the generator 50 by machine learning in step S104. As a result, the amount of memory required to process step S104 can be reduced, and the calculation time of step S104 can be shortened.

Further, the image generator 2 according to the present embodiment uses the generator 50 constructed by the learning device 1 to mass-produce images (combined images 69) in which the product R can be captured in steps S501 and S502. Such calculation cost can be suppressed. In addition, in the present embodiment, the combined image 69 mass-produced by the image generation device 2 can be used as a sample image. This makes it possible to reduce the cost of collecting the learning data set 322 used for machine learning in the estimator generator 3. Further, in the present embodiment, by using the combined image 69 mass-produced by the image generation device 2 as the sample image 3221, the number of sample images 3221 used for machine learning of the estimator 70 can be increased. As a result, in the inspection device 4 according to the present embodiment, the accuracy of determining the quality of the product R can be improved by step S902.

Further, the learning device 1 according to the present embodiment determines the patch size according to the complexity of the learning image 60 by the processing of steps S201 to S203. The high complexity of the training image 60 means that each pixel value in the training image 60 is rich in variation. In this case, if the size of the patch image 61 is increased, the amount of change in the patch image 61 to be learned in order to generate the image corresponding to the patch image 61 increases, so that the configuration of the generator 50 is complicated. Can be transformed into. On the other hand, when the complexity of the learning image 60 is low and the size of the patch image 61 is reduced, useless processing increases. Therefore, in step S202, the learning device 1 according to the present embodiment is set so that the patch size of each patch image 61 is reduced when the complexity of the learning image 60 is high, and the complexity of the learning image 60 is increased. If is low, the patch size of each patch image 61 is set to be large. Thereby, the learning image 60 can be divided into each patch image 61 having a patch size suitable for generation. Therefore, according to the present embodiment, the trained generation capable of generating an appropriate image corresponding to the learning image 60 by the machine learning while reducing the calculation cost required to construct the generator 50 by the machine learning. The vessel 50 can be constructed.

Further, the learning device 1 according to the present embodiment integrates two or more patch images 61 similar to each other so as to give the same label 62 by the processing of steps S301 to S305. This makes it possible to reduce the number of types of labels 62 to be learned, which further simplifies the configuration of the generator 50. Therefore, according to the present embodiment, the calculation cost required to construct the generator 50 by machine learning in step S104 can be further reduced.

§4 Modifications Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. For example, the following changes can be made. In the following, the same reference numerals will be used for the same components as those in the above embodiment, and the same points as in the above embodiment will be omitted as appropriate. The following modifications can be combined as appropriate.

<4.1>
In the above embodiment, a so-called multi-layered fully connected neural network is used for the generator 50 and the estimator 70. However, the structure and type of the neural network constituting each of the generator 50 and the estimator 70 are not limited to such an example, and may be appropriately selected depending on the embodiment. For example, a convolutional neural network may be used for each of the generator 50 and the estimator 70.

Further, in the above embodiment, a neural network is used as a learning model constituting the generator 50 and the estimator 70. However, the type of the learning model constituting each of the generator 50 and the estimator 70 may not be particularly limited as long as machine learning of the image can be performed, and the learning model is appropriately selected according to the embodiment. good.

<4.2>
In the above embodiment, the first learning result data 125 includes information indicating the configuration of the generator 50. Similarly, the second learning result data 325 contains information indicating the configuration of the estimator 70. However, the configuration of each learning result data (125, 325) does not have to be limited to such an example, and may be appropriately determined according to the embodiment. For example, when the configuration of the neural network to be used is common to each device, each learning result data (125, 325) does not have to include information indicating the configuration of the neural network.

<4.3>
For each information processing (FIGS. 11 to 17) according to the above embodiment, steps can be omitted, replaced, and added as appropriate according to the embodiment. For example, the learning device 1 according to the above embodiment may divide the learning image 60 by the patch size specified by the set value instead of the processing of steps S201 to S203. Further, for example, the learning device 1 according to the above embodiment may assign a different label 62 to each patch image 61 instead of the processing of steps S301 to S305.

<4.4>
In the learning network 500 according to the above embodiment, the generator 50 is connected to the discriminator 52. As a result, the learning device 1 according to the above embodiment appropriately performs the image corresponding to each patch image 61 by alternately performing the first training step of the discriminator 52 and the second training step of the generator 50. We are constructing a generator 50 that has acquired the ability to generate. However, the configuration of the learning network 500 does not have to be limited to such an example, and may be appropriately set according to the embodiment. Also, instead of constructing such a learning network 500, the generator 50 may be trained independently.

FIG. 18 schematically illustrates an example of the software configuration of the learning device 1A according to this modification. The hardware configuration and software configuration of the learning device 1A according to the present modification may be the same as the learning device 1 according to the embodiment, except that the learning network 500 according to the above embodiment is replaced with the learning network 500A.

The learning network 500A according to this modification includes a generator 50A and an encoder 54. The generator 50A is connected to the encoder 54. Specifically, the output of the encoder 54 is connected to the input of the generator 50A. The encoder 54 is configured to derive a feature amount (latent variable) from a given image (patch image 61 and label 62). On the other hand, the generator 50A serves as a decoder. That is, the generator 50A is configured to restore an image from the derived features. Such a generator 50A and an encoder 54 may be configured by a so-called multi-layered fully connected neural network as in the above embodiment.

The learning device 1A according to the present modification may operate in the same manner as the learning device 1 according to the above embodiment. That is, in step S101, the control unit of the learning device 1A acquires the learning image 60. In step S102, the control unit divides the learning image 60 into a plurality of patch images 61. In step S103, the control unit assigns a label 62 to each patch image 61. Steps S101 to S103 may be processed in the same manner as in the above embodiment. Then, in step S104, the control unit operates as the learning processing unit 114A, and by performing machine learning, the output value obtained from the encoder 54 and each patch image 61 are obtained by inputting each patch image 61 to the encoder 54. The generator 50A is trained to generate an image corresponding to each patch image 61 from the associated label 62.

Specifically, the control unit uses a data set composed of a combination of patch images 61 and labels 62 corresponding to each other as input data (training data), and uses prior distribution as teacher data (correct answer data). The learning process of the encoder 54 is executed. Further, the control unit uses a data set composed of a combination of the patch image 61 and the label 62 corresponding to each other as input data (training data), and uses the corresponding patch image 61 as teacher data (correct answer data). , The learning process of the encoder 54 and the generator 50A is executed.

For example, the control unit inputs a data set composed of a combination of patch images 61 and labels 62 corresponding to each other to the encoder 54, and executes arithmetic processing of the encoder 54. As a result, the control unit acquires an output value corresponding to the result of deriving some feature amount from the encoder 54. The control unit calculates the first error between this output value and the prior distribution. The prior distribution does not have to be particularly limited and may be appropriately selected depending on the embodiment. The prior distribution may be, for example, a Gaussian distribution. The control unit adjusts the value of the parameter of the encoder 54 so that the sum of the first errors becomes small.

Further, the control unit inputs the output value obtained from the encoder 54 and the corresponding label 62 to the generator 50A, and executes the arithmetic processing of the generator 50A. As a result, the control unit acquires the output corresponding to the result of generating the image from each label 62 from the generator 50A. The control unit calculates a second error between each generated image and the corresponding patch image 61. The control unit adjusts the parameter values of the generator 50A and the encoder 54 so that the sum of the second errors becomes small.

Similar to the above embodiment, an error backpropagation method may be used for adjusting the value of each parameter. The control unit repeats the adjustment of the value of each parameter by the above series of processes until the sum of each error becomes equal to or less than the threshold value. As a result, the control unit is trained to generate an image corresponding to each patch image 61 from the output value obtained from the encoder 54 and the corresponding label 62 by inputting each patch image 61 to the encoder 54. A vessel 50A can be constructed.

In step S105, the control unit generates information indicating the configuration and parameters of the trained generator 50A constructed by step S104 as the first learning result data 125A. The first learning result data 125A may or may not include information indicating the configuration and parameters of the encoder 54. Then, the control unit stores the generated first learning result data 125A in a predetermined storage area. The predetermined storage area may be, for example, a RAM in a control unit, a storage unit, an external storage device (for example, a data server such as NAS), a storage medium, or a combination thereof. As a result, the learning device 1A according to the present modification can construct a generator 50A for generating an image corresponding to each patch image 61. The constructed generator 50A can be used in the same manner as the generator 50 according to the above embodiment. The image generator 2 according to the above embodiment may use the generator 50A to generate a combined image 69.

<4.5>
In the above embodiment, an example in which the present invention is applied to a scene where an appearance inspection of a product R shown in an image is performed is shown. However, the scope of application of the present invention is not limited to such an example of visual inspection. The present invention is applicable to all situations in which some feature of an object in an image is estimated. By changing the image handled by the inspection system 100 from the image of the product R to the image of the object, it is possible to configure an estimation system that estimates some features from the image of the object.

FIG. 19 schematically illustrates an example of an application scene of the estimation system 100B according to this modification. As shown in FIG. 19, the estimation system 100B according to the present modification includes a learning device 1B, an image generation device 2B, an estimator generation device 3B, and an estimation device 4B connected via a network. Except for changing the image handled from the image of the product R to the image of the object, the hardware configuration and software configuration of each of the devices 1B to 4B are the hardware configuration of each of the devices 1 to 4 and the hardware configuration of each of the devices 1 to 4 according to the above embodiment. It may be the same as the software configuration. Further, the devices 1B to 4B may operate in the same manner as the devices 1 to 4 according to the above embodiment.

That is, the learning device 1B according to the present modification acquires the learning image 60B in which the object RB is copied, and divides the acquired learning image 60B into a plurality of patch images 61B. Next, the learning device 1B associates the label 62B of any of the plurality of labels 62B with each patch image 61B according to the position of each patch image 61B in the learning image 60B. Then, the learning device 1B constructs a generator 50B trained to generate an image corresponding to each patch image 61B from the label 62B associated with each patch image 61B by performing machine learning.

On the other hand, the image generation device 2B uses the trained generator 50B constructed by the learning device 1B to generate each of the plurality of images 67B from each of the plurality of labels 62B. Then, the image generation device 2B generates the combined image 69B by arranging each of the generated plurality of images 67B at positions corresponding to each of the plurality of labels 62B and then combining them. The generated combined image 69B is an image in which the object RB can be captured, and is an image corresponding to the learning image 60B.

Further, the estimator generator 3B acquires a plurality of learning data sets 322B each composed of a combination of a sample image 3221B showing the object RB and correct answer data 3222B. The correct answer data 3222B shows the result (that is, the correct answer) of identifying the characteristics of the object RB reflected in the sample image 3221B. Then, the estimator generator 3B has acquired the ability to estimate the characteristics of the object RB appearing in a given image by performing machine learning using a plurality of learning data sets 322B. To build. In response to this, the estimator generator 3B generates information indicating the configuration and parameters of the estimator 70B that has been learned as the learning result data 325B. The estimator generation device 3B can use the combined image 69B generated by the image generation device 2B as the sample image 3221B.

FIG. 20 schematically illustrates an example of the software configuration of the estimation device 4B according to this modification. The estimation device 4B according to this modification acquires the target image 422B in which the target object RB is captured. In this modification, the camera CA is connected to the estimation device 4B. The estimation device 4B acquires the target image 422B by photographing the target object RB with this camera CA. Next, the control unit of the estimation device 4B operates as the estimation unit 412B, and sets the trained estimator 70B with reference to the learning result data 325B. Subsequently, the estimation device 4B inputs the acquired target image 422B to the trained estimator 70B, and executes arithmetic processing of the trained estimator 70B. As a result, the estimation device 4B acquires an output value corresponding to the result of estimating the characteristics of the object RB from the trained estimator 70B. Then, the estimation device 4B outputs information regarding the result of estimating the characteristics of the object RB based on the output value obtained from the learned estimator 70B. As a result, the estimation system 100B according to the present modification is configured to estimate some features from the image of the object RB.

The type of the object RB may not be particularly limited as long as it can be a target for identifying some attribute, and may be appropriately selected according to the embodiment. The object RB may be, for example, a product, a person, a person's body part (for example, a face, etc.), a character, a situation outside the vehicle in a vehicle, or the like, which is the subject of the visual inspection. The feature to be estimated does not have to be particularly limited, and may be appropriately selected according to the object RB. When the object RB is the face of a person, the features to be estimated may be, for example, the type of facial expression, the state of facial parts, the attributes of the person on the face, and the like. When the object RB is a character, the feature to be estimated may be attributes such as print sharpness, distortion, and character string deviation. When the object RB is a situation outside the vehicle, the feature to be estimated may be an attribute of the road condition such as a lane condition and a traffic condition.

100 ... Inspection system,
1 ... Learning device,
11 ... control unit, 12 ... storage unit, 13 ... communication interface,
14 ... input device, 15 ... output device, 16 ... drive,
111 ... image acquisition unit, 112 ... image segmentation unit,
113 ... Labeling unit, 114 ... (1st) Learning processing unit,
115 ... (1st) Preservation processing unit,
121 ... learning program, 125 ... first learning result data,
2 ... Image generator,
21 ... Control unit, 22 ... Storage unit, 23 ... Communication interface,
24 ... Input device, 25 ... Output device, 26 ... Drive,
211 ... Generation unit, 212 ... Coupling unit, 213 ... Image storage unit,
221 ... Image generation program,
50 ... generator, 52 ... discriminator,
60 ... learning image, 61 ... patch image,
62 ... Label, 621 ... Label information,
67 ... (patch) image, 69 ... combined image,
3 ... Estimator generator,
31 ... control unit, 32 ... storage unit, 33 ... communication interface,
34 ... Input device, 35 ... Output device, 36 ... Drive,
311 ... Learning data acquisition unit 312 ... (2nd) Learning processing unit,
313 ... (2nd) Preservation processing unit,
321 ... Estimator generation program,
322 ... Learning dataset,
3221 ... sample image, 3222 ... correct answer data,
325 ... Second learning result data,
4 ... Inspection device (estimation device),
41 ... Control unit, 42 ... Storage unit, 43 ... Communication interface,
44 ... Input device, 45 ... Output device, 46 ... Drive,
47 ... External interface,
411 ... Target data acquisition unit, 412 ... Good / bad judgment unit,
413 ... Output section,
421 ... Inspection program,
70 ... Estimator,
91/92/93/94 ... Storage medium

Claims (7)

An image acquisition unit that acquires learning images of products subject to visual inspection,
An image segmentation unit that divides the learning image into a plurality of patch images,
A labeling unit that associates one of a plurality of labels with each patch image according to the position of each patch image in the training image.
Machine learning using a discriminator, in which the information of the pixels of each patch image divided from the learning image input to the discriminator and the label associated with each patch image are input. To build a generator trained to generate an image corresponding to each patch image from the label associated with each patch image.
A generator that generates a plurality of images from each of the plurality of labels by using the generator constructed by the machine learning.
A joining portion that generates a combined image by arranging each of the generated plurality of images at a position corresponding to each of the plurality of labels and then combining them.
Equipped with
The generator is connected to the discriminator and
The machine learning is such that it is determined whether the input image input to the discriminator is the image generated by the generator or each patch image divided from the training image. From the first training step for training the discriminator, the pixel information of each patch image, and the label associated with each patch image, the generation so as to generate the image such that the discrimination by the discriminator is erroneous. The second training step of training the vessel, including alternating.
Inspection system . The image segmentation section
The complexity of the learning image is calculated and
Determine the patch size according to the calculated complexity,
The training image is divided into the plurality of patch images each having the determined patch size.
The inspection system according to claim 1. The labeling section is
Calculate the complexity of each patch image and
Based on the calculated complexity, two or more patch images similar to each other among the plurality of patch images are integrated.
Give the same label to the two or more integrated patch images.
The inspection system according to claim 1 or 2. Further provided with a storage processing unit that stores information about the constructed generator in a predetermined storage area.
The inspection system according to any one of claims 1 to 3. The computer
Steps to acquire a learning image of the product subject to visual inspection,
The step of dividing the training image into a plurality of patch images,
A step of associating one of a plurality of labels with each patch image according to the position of each patch image in the training image.
Machine learning using a discriminator, in which the information of the pixels of each patch image divided from the learning image input to the discriminator and the label associated with each patch image are input. To build a generator trained to generate an image corresponding to each patch image from the label associated with each patch image.
Using the generator constructed by the machine learning, a generation step of generating each of a plurality of images from each of the plurality of labels, and a generation step.
A combination step of generating a combined image by arranging each of the generated plurality of images at a position corresponding to each of the plurality of labels and then combining them.
And run
The generator is connected to the discriminator and
The machine learning is such that it is determined whether the input image input to the discriminator is the image generated by the generator or each patch image divided from the training image. From the first training step for training the discriminator, the pixel information of each patch image, and the label associated with each patch image, the generation so as to generate the image such that the discrimination by the discriminator is erroneous. The second training step of training the vessel, including alternating.
Inspection method. On the computer
Steps to acquire a learning image of the product subject to visual inspection,
The step of dividing the training image into a plurality of patch images,
A step of associating one of a plurality of labels with each patch image according to the position of each patch image in the training image.
Machine learning using a discriminator, in which the information of the pixels of each patch image divided from the learning image input to the discriminator and the label associated with each patch image are input. To build a generator trained to generate an image corresponding to each patch image from the label associated with each patch image.
Using the generator constructed by the machine learning, a generation step of generating each of a plurality of images from each of the plurality of labels, and a generation step.
A combination step of generating a combined image by arranging each of the generated plurality of images at a position corresponding to each of the plurality of labels and then combining them.
To execute ,
The generator is connected to the discriminator and
The machine learning is such that it is determined whether the input image input to the discriminator is the image generated by the generator or each patch image divided from the training image. From the first training step for training the discriminator, the pixel information of each patch image, and the label associated with each patch image, the generation so as to generate the image such that the discrimination by the discriminator is erroneous. The second training step of training the vessel, including alternating.
Inspection program. An image acquisition unit that acquires a learning image of an object,
An image segmentation unit that divides the learning image into a plurality of patch images,
A labeling unit that associates one of a plurality of labels with each patch image according to the position of each patch image in the training image.
Machine learning using a discriminator, in which the information of the pixels of each patch image divided from the learning image input to the discriminator and the label associated with each patch image are input. To build a generator trained to generate an image corresponding to each patch image from the label associated with each patch image.
A generator that generates a plurality of images from each of the plurality of labels by using the generator constructed by the machine learning.
A joining portion that generates a combined image by arranging each of the generated plurality of images at a position corresponding to each of the plurality of labels and then combining them.
Equipped with
The generator is connected to the discriminator and
The machine learning is such that it is determined whether the input image input to the discriminator is the image generated by the generator or each patch image divided from the training image. From the first training step for training the discriminator, the pixel information of each patch image, and the label associated with each patch image, the generation so as to generate the image such that the discrimination by the discriminator is erroneous. The second training step of training the vessel, including alternating.
Inspection system .

Images