JP2020046984A - Image generating method, image determining method, image generating apparatus, and image generating program - Google Patents

Abstract

To make classification of images using machine learning more accurate based on a small amount of collected images.SOLUTION: An image generating method according to an embodiment of the present invention includes a first learning process and a generating process. The first learning process carries out, by using collected images as sample data, first learning which machine-learns potential distribution of sample data indicative of classes into which the sample data are classified, thereby generating generated models obtained by modeling the distribution. The generating process uses the generated model to specify respective regions of the classes in a potential space where the distribution is supposed to exist, and generates an intermediate image which is an image at an intermediate point among classes based on the regions.SELECTED DRAWING: Figure 2

Description

  The embodiments of the present disclosure relate to an image generation method, an image determination method, an image generation device, and an image generation program.

  2. Description of the Related Art Conventionally, a technique for performing image analysis using machine learning is known. For example, the technology disclosed in Patent Literature 1 executes machine learning using a feature vector calculated for a plurality of images whose classification results are known, and the classification result is unknown based on the result obtained by the machine learning. This classifies a certain image.

  By using such a technique, it is possible to classify, for example, whether or not the product is a normal product from an image of the manufactured product.

JP-A-2015-011552

  However, the above-described conventional technology has room for further improvement in achieving high-precision image classification using machine learning based on a small amount of collected images.

  Specifically, in image classification using machine learning, it is possible to achieve high accuracy by learning a large number of learning images, but before the system is operated or in the initial stage of operation, a large number of images are needed. In many cases, the learning images cannot be collected.

  In such a case, a method of collecting images while actually operating the system, performing additional learning as needed, and updating the classification model used for image classification every time can be considered. However, such a method is complicated because it requires a manual check operation or the like every time additional learning is performed, and also requires a long time to improve the accuracy of the classification model.

  An aspect of the embodiment is made in view of the above, and is based on a small amount of collected images, and is capable of achieving high-precision image classification using machine learning, an image determination method, and an image determination method. It is an object to provide a generation device and an image generation program.

  An image generation method according to one aspect of an embodiment includes a first learning step and a generation step. The first learning step uses the collected images as sample data, and executes a first learning that machine-learns a potential distribution of the sample data indicating a class for classifying the sample data. Generate a modeled generation model. The generation step uses the generation model to specify an area of each of the classes in a latent space where the distribution is assumed to exist, and generates an intermediate image that is an image of an intermediate point between the classes based on the area. Generate.

  According to an aspect of the embodiment, it is possible to improve the accuracy of image classification using machine learning based on a small amount of collected images.

FIG. 1A is a schematic explanatory diagram of an image determination method according to a comparative example. FIG. 1B is a schematic explanatory diagram of the image determination method according to the embodiment. FIG. 2 is a block diagram of the image determination device according to the embodiment. FIG. 3A is an explanatory diagram (part 1) of the intermediate image generation process. FIG. 3B is an explanatory diagram (part 2) of the intermediate image generation process. FIG. 3C is an explanatory diagram (part 3) of the intermediate image generation process. FIG. 4 is an explanatory diagram of the determination processing. FIG. 5 is a flowchart illustrating a processing procedure executed by the image determination device according to the embodiment. FIG. 6 is a hardware configuration diagram illustrating an example of a computer that implements the functions of the image determination device according to the embodiment.

  Hereinafter, embodiments of an image generation method, an image determination method, an image generation device, and an image generation program disclosed in the present application will be described in detail with reference to the accompanying drawings. The present invention is not limited by the embodiments described below.

  First, an outline of an image determination method according to the embodiment will be described with reference to FIGS. 1A and 1B. FIG. 1A is a schematic explanatory diagram of an image determination method according to a comparative example. FIG. 1B is a schematic explanatory diagram of the image determination method according to the embodiment.

  In the following, a description will be given of an example in which a waffle that is a baked confectionery is manufactured as a product, and a non-defective or defective waffle is determined in a pre-shipment inspection or the like.

  As illustrated in FIG. 1A, in the image determination method according to the comparative example, first, machine learning is performed using a “learning image” of a waffle including non-defective products and defective products collected in advance as a learning data set (step S1 ′). ), And generates a classification model 12d. As the algorithm of machine learning, a known algorithm such as deep learning is used.

  In the image determination method according to the comparative example, the “determination target image” to be determined is input to the generated classification model 12d, and the “determination target image” is classified based on the output value output by the classification model 12d. (Step S2 '), and a classification result is obtained.

  However, according to the image determination method according to the comparative example, as shown in the figure, if the number of samples of the learning image is “small”, sufficient learning cannot be performed, and “accuracy” of the classification result by the classification model 12d is not obtained. Was low. "

  Therefore, as shown in FIG. 1B, in the image determination method according to the embodiment, first, first learning using “learning images” collected in advance is performed (step S1) to generate the generation model 12b. did.

  The generative model 12b is a so-called generative model, which is different from a discriminative model (classified model) that is modeled by learning the boundaries of classes for classifying sample data. Are modeled by learning the distribution of sample data that is assumed to potentially exist.

  Therefore, by using the generation model 12b, not only class classification but also pseudo data generation at an arbitrary position in an area corresponding to each class becomes possible.

  Then, in the image determination method according to the embodiment, by utilizing this, an "intermediate image" which is an image of an arbitrary intermediate point between classes is generated using the generation model 12b (step S2). This makes it possible to automatically generate a large number of images similar to the collected images, so that the amount of collected images to be collected in advance can be significantly reduced.

  Then, in the image determination method according to the embodiment, the generated “intermediate image” is added to the “learning image” (step S3), and the second learning is performed using the “learning image” (step S3). S4) The classification model 12d is generated.

  That is, in the image determination method according to the embodiment, the classification model 12d is generated as a highly accurate model that has been sufficiently learned by the second learning using the “learning image” to which the “intermediate image” has been added. Can be.

  Then, in the image determination method according to the embodiment, the “determination target image” is input to the classification model 12d, and the “determination target image” is classified based on the output value output by the classification model 12d (step S5). ), To obtain the classification result.

  Therefore, according to the image determination method according to the embodiment, it is possible to improve the accuracy of image classification using machine learning based on a small amount of collected images.

  Hereinafter, the configuration of the image determination device 10 to which the above-described image determination method is applied will be described more specifically.

  FIG. 2 is a block diagram of the image determination device 10 according to the embodiment. In FIG. 2, components necessary for explaining the features of the present embodiment are represented by functional blocks, and descriptions of general components are omitted.

  In other words, each component illustrated in FIG. 2 is a functional concept and does not necessarily need to be physically configured as illustrated. For example, the specific form of distribution / integration of each functional block is not limited to the illustrated one, and all or a part of the functional blocks may be functionally or physically distributed in arbitrary units according to various loads and usage conditions. -It is possible to integrate and configure.

  In the description using FIG. 2, the description of the components already described above may be simplified or omitted.

  As shown in FIG. 2, the image determination device 10 includes a control unit 11 and a storage unit 12. The control unit 11 includes an acquisition unit 11a, a first learning unit 11b, an intermediate image generation unit 11c, a second learning unit 11d, and a determination unit 11e.

  The storage unit 12 is a storage device such as a hard disk drive, a non-volatile memory, or a register, and stores a learning data set 12a, a generation model 12b, an intermediate image 12c, and a classification model 12d.

  As shown in FIG. 2, the control unit 11 includes at least an acquisition unit 11a, a first learning unit 11b, and an intermediate image generation unit 11c, and the storage unit 12 stores a learning data set 12a, a generation model By storing at least 12b, the image generating apparatus 50 that generates the intermediate image 12c can be configured.

  The control unit 11 performs overall control of the image determination device 10. The acquisition unit 11a acquires a learning image of a waffle containing a non-defective product or a defective product collected in advance, and stores the image in the learning data set 12a.

  Note that the acquisition unit 11a can receive a user operation from the operation unit 20 such as a keyboard, a mouse, and a touch panel connected to the image determination device 10 and acquire a learning image based on the user operation.

  The first learning unit 11b acquires the learning data set 12a, performs first learning using the data set 12a, and generates the generation model 12b. As an algorithm of the first learning, a hostile generation network (GAN: Generative Adversarial Network), VAE (Variational Auto Encoder), or the like can be used.

  Thus, the generator model 12b is a generator or variational self-encoder in the GAN. Since these algorithms are publicly known, detailed description thereof is omitted here.

  Further, the first learning unit 11b causes the display unit 30 such as a display or a touch panel connected to the image determination device 10 to display the generation result of the generation model 12b.

  The intermediate image generation unit 11c specifies, using the generation model 12b, an area of each class in a latent space where a distribution indicating each class of the learning image is assumed to exist, and determines an intermediate point between the classes based on the area. An intermediate image 12c is generated.

  The intermediate image generation processing will be specifically described with reference to FIGS. 3A to 3C. 3A to 3C are explanatory diagrams (part 1) to (part 3) of the intermediate image generation process.

  First, the generation model 12b is generated by the first learning unit 11b, and a class is set in the learning image based on, for example, the knowledge of the user. Here, it is assumed that a plurality of sample data exist in each class as learning images.

  Then, as shown in FIG. 3A, it is assumed that there is a distribution of data estimated by “each sample position” in the “latent space” indicated by the generation model 12b. In such a case, the intermediate image generation unit 11c specifies an area of each of the classes set as “Class A” to “Class E” based on, for example, a user operation from the operation unit 20.

  Then, the intermediate image generation unit 11c generates an intermediate image 12c based on the specified regions. Specifically, as shown in FIG. 3A, the intermediate image generation unit 11c calculates the “center position” of the sample data existing in each area, and sets an arbitrary position on a straight line connecting the center positions to the intermediate point. To generate the intermediate image 12c.

  For example, as shown in FIG. 3B, it is assumed that there are two “center positions”. The intermediate image generation unit 11c generates an intermediate image 12c as an arbitrary position on a straight line connecting the two points as an intermediate point, in other words, a “generation position”.

  For example, when the intermediate image generation unit 11c receives a user operation for specifying the number of generated intermediate images 12c = 3 from the operation unit 20 between the “center positions” shown in FIG. The intermediate image 12c is generated by setting each position obtained by equally dividing the above as the above-mentioned “generation position”.

  Thus, when the user wants to generate an intermediate image 12c between certain classes, it is possible to automatically generate a large number of intermediate images 12c by an easy operation without specifying the “generation position” in detail.

  Also, by arbitrarily changing the number of generations, that is, the “generation position” that is a latent variable, as shown in FIG. 3C, the intermediate image generation unit 11c can generate any intermediate image 12c assumed to exist on the straight line. Can be automatically generated.

  Therefore, the user does not specify the number of generations, but automatically generates the intermediate image 12c while changing the “generation position” from one “center position” to the other “center position” on the straight line, Unnecessary items may be thinned out based on the knowledge of the above.

  Alternatively, the generated intermediate image 12c may be newly classified, and a further intermediate image 12c may be generated by associating the center position with the center position of an existing class.

  Accordingly, it is possible to easily and easily generate a large number of intermediate images 12c similar to the collected images without previously collecting a large number of collected images. Therefore, it is possible to improve the accuracy of image classification using machine learning based on a small amount of collected images.

  Returning to the description of FIG. 2, the second learning unit 11d will be described. The second learning unit 11d performs a second learning which is a machine learning using the learning data set 12a to which the intermediate image 12c generated by the intermediate image generating unit 11c is added, thereby classifying the determination target image. Generate the model 12d.

  As the second learning algorithm, a known machine learning algorithm can be arbitrarily used.

  The determination unit 11e determines a class into which the determination target image is classified based on an output value of the classification model 12d obtained by inputting the determination target image to the classification model 12d. Further, the determination unit 11e causes the display unit 30 to display the determination result, for example.

  Here, an example of the determination process will be specifically described with reference to FIG. FIG. 4 is an explanatory diagram of the determination processing.

  In the case of the waffle production line according to the embodiment, as illustrated in FIG. 4, the determination unit 11e cuts out each image of the baked waffle and inputs the images one by one to the classification model 12d as determination target images.

  In response to the input, the classification model 12d outputs, for example, a class ID that is an identifier of a class into which the determination target image is classified. The determination unit 11e receives the class ID and, for example, if the class ID indicates “high-fire NG”, causes the display unit 30 to display the determination result of the content urging the “discard instruction” as an example.

  In addition, if the received class ID is, for example, a class ID indicating “low heat NG”, the determination unit 11e causes the display unit 30 to display a determination result of a content urging a “heat-up instruction” as an example. If the received class ID is, for example, a class ID corresponding to “non-defective product”, the determination unit 11e causes the display unit 30 to display a determination result of a content urging “shipping instruction” as an example.

  Next, a processing procedure executed by the image determination device 10 according to the embodiment will be described with reference to FIG. FIG. 5 is a flowchart illustrating a processing procedure executed by the image determination device 10 according to the embodiment.

  As shown in FIG. 5, the acquisition unit 11a prepares the collected learning images as a learning data set 12a (Step S101). Then, the first learning unit 11b generates the generation model 12b from the learning data set 12a by the first learning (Step S102).

  Then, a classification class is set for the learning image based on, for example, the user's knowledge (step S103).

  Then, the intermediate image generation unit 11c specifies the area of each class in the latent space using the generation model 12b, and generates the intermediate image 12c between the classes (Step S104). A classification class is set in the generated intermediate image 12c based on, for example, the knowledge of the user (step S105).

  Then, the intermediate image 12c is added to the learning data set 12a (step S106), and the second learning unit 11d generates a classification model 12d by the second learning from the learning data set 12a to which the intermediate image 12c is added (step S106). S107).

  Although not shown, the determination unit 11e classifies the determination target image based on an output value of the classification model 12d obtained by appropriately inputting the determination target image to the classification model 12d during operation of the system. Class is determined, and the determination result is notified.

  Note that the image determination device 10 according to the above-described embodiment is realized by, for example, a computer 60 having a configuration as illustrated in FIG. FIG. 6 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the image determination device 10 according to the embodiment. The computer 60 includes a CPU (Central Processing Unit) 61, a RAM (Random Access Memory) 62, a ROM (Read Only Memory) 63, a HDD (Hard Disk Drive) 64, a communication interface (I / F) 65, and an input / output interface (I / F) 66 and a media interface (I / F) 67.

  The CPU 61 operates based on a program stored in the ROM 63 or the HDD 64 and controls each unit. The ROM 63 stores a boot program executed by the CPU 61 when the computer 60 is started, a program depending on hardware of the computer 60, and the like.

  The HDD 64 stores a program executed by the CPU 61, data used by the program, and the like. The communication interface 65 receives data from another device via the communication network, sends the data to the CPU 61, and transmits the data generated by the CPU 61 to the other device via the communication network.

  The CPU 61 controls an output device such as a display and a printer and an input device such as a keyboard and a mouse via the input / output interface 66. The CPU 61 acquires data from an input device via the input / output interface 66. Further, the CPU 61 outputs the generated data to the output device via the input / output interface 66.

  The media interface 67 reads a program or data stored in the recording medium 68 and provides the program or data to the CPU 61 via the RAM 62. The CPU 61 loads the program from the recording medium 68 onto the RAM 62 via the media interface 67, and executes the loaded program. The recording medium 68 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc), a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. And so on.

  For example, when the computer 60 functions as the image determination device 10 according to the embodiment, the CPU 61 of the computer 60 implements each function of the control unit 11 by executing a program loaded on the RAM 62. The data in the storage unit 12 is stored in the HDD 64. Although the CPU 61 of the computer 60 reads these programs from the recording medium 68 and executes them, as another example, these programs may be obtained from another device via a communication network.

  As described above, the image generation device 50 according to the embodiment includes the first learning unit 11b and the intermediate image generation unit 11c (corresponding to an example of the “generation unit”). The first learning unit 11b sets the collected image as a learning image (corresponding to an example of “sample data”), and machine-learns a potential distribution of the learning image indicating a class into which the learning image is classified. By executing the first learning, a generation model 12b that models the distribution is generated. The intermediate image generation unit 11c specifies, using the generation model 12b, an area of each class in the latent space where the distribution is assumed to exist, and based on the area, an intermediate image 12c that is an image of an intermediate point between the classes. Generate

  Therefore, according to the image generation device 50 according to the embodiment, it is possible to improve the accuracy of image classification using machine learning based on a small amount of collected images.

  The image determination device 10 according to the embodiment includes a second learning unit 11d and a determination unit 11e in addition to the image generation device 50. The second learning unit 11d performs a second learning which is a machine learning using the learning data set 12a to which the intermediate image 12c generated by the image generating device 50 is added, thereby classifying the determination target image. Generate 12d. The determination unit 11e determines the class of the determination target image based on the output value of the classification model 12d obtained by inputting the determination target image to the classification model 12d.

  Therefore, according to the image determination device 10 according to the embodiment, image classification using machine learning can be performed with high accuracy based on a small amount of collected images.

  In the above-described embodiment, before generating the classification model 12d, the intermediate model 12c is generated by the generation model 12b, and the generated intermediate image 12c is added to the learning data set 12a to perform the second learning. Although generated, the classification model 12d may be generated first.

  That is, before adding the intermediate image 12c, the classification model 12d is generated in advance by performing the second learning using the learning data set 12a, and if the intermediate image 12c is generated, the classification model 12d is used. The classification model 12d may be updated by executing additional learning of the second learning.

  In addition, when a new collected image that needs to be additionally learned is collected during operation of the system, the generation model 12b may be updated based on the collected image, and a new intermediate image 12c may be generated. Further, in this case, the classification model 12d may be updated by performing additional learning of the second learning using the newly acquired image and the newly generated intermediate image 12c.

  In the above-described embodiment, the case where the image generation device 50 is included in the image determination device 10 has been described as an example. However, the image generation device 50 and the image determination device 10 may be separated.

  In the above-described embodiment, the algorithm of the deep learning is mainly described as an example of the algorithm of the machine learning, and the algorithm to be used is not limited. Therefore, machine learning may be performed by a regression analysis method such as support vector regression using a pattern classifier such as an SVM (Support Vector Machine) to generate, for example, the classification model 12d. Here, the pattern classifier is not limited to the SVM, and may be, for example, AdaBoost. Further, a random forest or the like may be used.

  Further effects and modifications can be easily derived by those skilled in the art. For this reason, the broader aspects of the present invention are not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

Reference Signs List 10 image determination device 11 control unit 11a acquisition unit 11b first learning unit 11c intermediate image generation unit 11d second learning unit 11e determination unit 12 storage unit 12a learning data set 12b generation model 12c intermediate image 12d classification model 50 image generation device

Claims (7)

Generating a model that models the distribution by executing a first learning that machine-learns a potential distribution of the sample data indicating a class that classifies the sample data with the collected images as sample data. A first learning step to be performed;
A generation step of using the generation model to specify an area of each of the classes in a latent space where the distribution is assumed to exist, and generating an intermediate image that is an image of an intermediate point between the classes based on the area; An image generation method comprising: The generation step includes:
The center image of the sample data existing in each of the regions is calculated, and the intermediate image is generated using an arbitrary position on a straight line connecting the center positions as the intermediate point. Image generation method. The generation step includes:
When an arbitrary number for generating the intermediate image is designated, the intermediate image is generated with each position obtained by equally dividing a straight line connecting the center positions according to the number as the intermediate point. The image generation method according to claim 1. The first learning step includes:
The image generation method according to claim 1, wherein the generation model is generated as a generation network in a hostile generation network or as a variational self-encoder. A second learning which is a machine learning using the sample data to which the intermediate image generated by the image generation method according to claim 1 is added, thereby classifying the determination target image. A second learning step of generating a classification model to perform,
A determination step of determining the class of the determination target image based on an output value of the classification model obtained by inputting the determination target image to the classification model. Generating a model that models the distribution by executing a first learning that machine-learns a potential distribution of the sample data indicating a class that classifies the sample data with the collected images as sample data. A first learning unit to
A generation unit that specifies an area of each class in the latent space where the distribution is assumed to exist by using the generation model, and generates an intermediate image that is an image of an intermediate point between the classes based on the area; An image generating apparatus comprising: Generating a model that models the distribution by executing a first learning that machine-learns a potential distribution of the sample data indicating a class that classifies the sample data with the collected images as sample data. A first learning procedure to be performed;
Using the generation model, specify an area of each of the classes in the latent space where the distribution is assumed to exist, and generate an intermediate image that is an image of an intermediate point between the classes based on the area. An image generation program for causing a computer to execute the following.