JP6647632B2 - Generating training data for machine learning - Google Patents

Description

  The present invention relates to generation of training data for machine learning.

  One image processing method uses a neural network to automatically detect a biological configuration in an image of a biological tissue sample (for example, see Patent Document 1).

  Generally, training data of a neural network to which an image is input is obtained by obtaining a sample image and manually attaching a label to the sample image.

JP-T-2017-516992

  However, machine learning systems such as deep neural networks require many training data sets (that is, many input and output pairs), and it takes a long time to generate appropriate training data. .

  As a method of generating a pseudo training data set from the original training data, there is a data extension. In data expansion, translation, mirror image inversion, rotation, predetermined geometric deformation, slight color change, noise addition, etc. are performed on the original sample image (input image for machine learning) of the original training data set Generates a new training dataset. Normally, in data extension, the label attached to the original sample image is used as it is in a new training data set. However, when the output of the machine learning system is an image (label image), in the data extension, the original label image (output image for machine image) of the original training data set is translated and mirror-inverted of the original sample image. , Rotation, and a predetermined geometric transformation, the image subjected to the similar translation, mirror image inversion, rotation, and the predetermined geometric transformation is used as a label image of a new training data set. In addition, in the case of data expansion by slight color change and noise addition, the same image as the original label image is used as a label image of a new training data set.

  However, when data extension is used, overtraining is less likely to occur than in the case of using only the training data set of the original image, but the sample image and label image of the new training data set generated by data extension are compared as described above. It is processed by a simple method, and retains many features of the original sample images and original label images actually acquired, so even if many training data sets are newly generated by data expansion, the original Due to the small number of sample images and original label images, overlearning is still likely to occur in machine learning systems such as deep neural networks where machine learning is performed on many training data sets.

  The present invention has been made in view of the above problems, and provides a training data generating system, a training data generating method, and a training data generating program that automatically generate many appropriate training data sets in a short time. Aim.

  The training data generation system according to the present invention is a training data generation system that automatically generates training data for a machine learning system that generates an output image from an input image, and a similar image of an original label image corresponding to an original sample image, A similar image generation unit that generates a pseudo-label image, and an analogy image generation unit that generates an image analogized from the pseudo-label image according to the conversion characteristics from the original label image to the original sample image as a pseudo-sample image corresponding to the pseudo-label image And Here, the pseudo sample image is an input image of the machine learning system in the training data, and the pseudo label image is an output image of the machine learning system in the training data.

  The training data generation method according to the present invention is a training data generation method for automatically generating training data for a machine learning system that generates an output image from an input image, and a similar image of an original label image corresponding to an original sample image, A similar image generating step of generating a pseudo-label image, and an analogy image generating step of generating an image analogized from the pseudo-label image according to a conversion characteristic from the original label image to the original sample image as a pseudo-sample image corresponding to the pseudo-label image And Here, the pseudo sample image is an input image of the machine learning system in the training data, and the pseudo label image is an output image of the machine learning system in the training data.

  The training data generation program according to the present invention is a training data generation program that causes a computer to automatically generate training data for a machine learning system that generates an output image from an input image. A similar image generation step of generating a similar image of the image as a pseudo-label image, and an image analogized from the pseudo-label image according to the conversion characteristic from the original label image to the original sample image as a pseudo-sample image corresponding to the pseudo-label image And an analogy image generation step to be generated. Here, the pseudo sample image is an input image of the machine learning system in the training data, and the pseudo label image is an output image of the machine learning system in the training data.

  According to the present invention, many suitable training data sets are automatically generated in a short time.

  The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description in conjunction with the accompanying drawings.

FIG. 1 is a block diagram illustrating a configuration of a training data generation system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a machine learning system that generates an output image from an input image. FIG. 3 is a flowchart illustrating a training data generation method in the system shown in FIG. FIG. 4 is a diagram for explaining the correspondence between the original training data set and the pseudo training data set in the system shown in FIG. FIG. 5 is a diagram illustrating a specific example of an original sample image and an original label image. FIG. 6 is a diagram illustrating a specific example of generating a pseudo label image. FIG. 7 is a diagram illustrating a specific example of generating a pseudo sample image. FIG. 8 is a diagram illustrating a specific example of an output image of the image generation engine after machine learning.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of a training data generation system according to an embodiment of the present invention. Although the training data generation system illustrated in FIG. 1 includes one information processing device, each processing unit described below may be distributed to a plurality of information processing devices that can perform data communication with each other.

  The training data generation system according to this embodiment includes a plurality (a large number) of pseudo sample images and a plurality of pseudo label images that can be used for a training data set used for learning of an image generation engine such as a deep neural network. Automatically generated from the original training data set (pair of original sample images and original label images).

  FIG. 2 is a diagram illustrating a machine learning system that generates an output image from an input image. As shown in FIG. 2, the machine learning system is a system for automatically generating an output image corresponding to the input image from an input image using an image generation engine such as a deep neural network. .

  In such a system, a known input image and its known input image are corresponded before starting use or at any time so that a desired output image is automatically generated from an unknown input image prepared by a user. Learning of an image generation engine such as a deep neural network is performed based on a training data set that is a pair with an output image to be executed.

  By using the training data generation system according to this embodiment, a large number of pseudo training data sets (pairs of pseudo sample images and pseudo label images) can be converted from original training data sets (pairs of original sample images and original label images). ) Is obtained, and the training of the image generation engine using the original training data set and the pseudo training data set is appropriately performed.

  The training data generation system shown in FIG. 1 includes a storage device 1, a communication device 2, an image reading device 3, and an arithmetic processing device 4.

  The storage device 1 is a nonvolatile storage device such as a flash memory and a hard disk, and stores various data and programs.

  Here, the storage device 1 stores the training data generation program 11, and the generated training data is stored as needed. The training data generation program 11 may be stored in a portable computer-readable recording medium such as a CD (Compact Disk). In that case, for example, the training data generation program 11 is installed in the storage device 1 from the recording medium. The training data generation program 11 may be a single program or an aggregate of a plurality of programs.

  The communication device 2 is a device capable of data communication such as a network interface, a peripheral device interface, and a modem, and performs data communication with another device as necessary. The image reading device 3 optically reads an image of a document on a physical medium such as a photograph including an original image as an input image, and generates image data of the original image. The communication device 2 and the image reading device 3 are used for acquiring an original training data set.

  The arithmetic processing device 4 is a computer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. By executing, it operates as various processing units.

  Here, by executing the training data generation program 11, the arithmetic processing device 4 operates as an original data acquisition unit 21, a similar image generation unit 22, and an analogy image generation unit 23.

  The original data acquisition unit 21 acquires one or a plurality of original training data sets. One original training data set includes one original sample image that can be used as training data for a machine learning system that generates an output image from an input image, and an original label image corresponding to the original sample image. The original sample image is an image such as an actually obtained photograph that can be used as an input image during machine learning, and the original label image is a manual image that can be used as an output image during machine learning. This is an appropriate label image generated by the above method.

  The original data acquisition unit 21 reads the original training data set stored in the storage device 1, receives the original training data set from an external device using the communication device 2, and uses the image reading device 3. Then, the image read by the image reading device 3 is acquired as an original training data set. The original training data set acquired using the communication device 2 and the image reading device 3 may be stored in the storage device 1.

  The similar image generation unit 22 generates a similar image of the acquired original label image as a pseudo label image. Here, the pseudo label image is used as an output image of the machine learning system during machine learning.

  In this embodiment, the similar image generation unit 22 receives one or a plurality of random numbers (random number vectors) having values within a predetermined range (for example, −1 to +1) as input values and generates an image corresponding to the input value. It has a generation engine, performs machine learning of the image generation engine using a plurality of original label images, and outputs an output image when another random value is input to the image generation engine as a similar image (that is, a pseudo-label image). And For example, the similar image generation unit 22 generates a similar image of the original label image as a pseudo label image based on a GAN (Generative Adversarial Network) such as DCGANS (Deep Convolutional Generative Adversarial Networks). This method of generating a similar image is different from normal data expansion (translation, mirror image inversion, rotation fluctuation, geometric deformation, density change, color fluctuation, addition of random noise, etc.). In other words, the normal data extension is obtained by converting the original image according to some rule or changing the original image somewhat irregularly. The image obtained by the data extension includes one original image. Although a certain degree of unique regularity remains, in this generation method, the regularity of a plurality of original images (original label images) is extracted by machine learning and reflected on similar images. Regularity is less likely to remain in the generated similar image.

  When generating a similar label image by GAN, the similar image generating unit 22 includes a generator and a discriminator. After performing machine learning of the discriminator and the generator, the generator generates a pseudo label image from the random number value vector. I do. The discriminator is a neural network that distinguishes an original label image from an image generated by a generator, and the generator is a neural network that generates an image from a random number vector. In the machine learning of the discriminator and the generator, (a) the discriminator is machine-learned based on the original label image so that the original label image is identified as the original label image, and (b) arbitrarily sampled. The generator is machine-learned such that the image is identified as the original label image for the image generated by the generator for the random number vector, and the image is identified as the image generated by the generator. The generator is machine-learned as is done.

  Note that the similar image generation unit 22 may generate a similar image of the original label image as a pseudo label image by another image generation engine that performs machine learning on the original label image.

  Further, the similar image generation unit 22 may perform a noise removal process on the generated pseudo label image. The noise removal processing is executed according to, for example, a Non-local Means Denoising algorithm.

  Further, if necessary, the similar image generation unit 22 performs normal data expansion (parallel movement, mirror image inversion, rotation fluctuation, geometric deformation, density change, color fluctuation, random fluctuation) on the generated pseudo label image. Noise, etc.) to increase the number of pseudo-label images.

  The analogy image generation unit 23 generates an image (analogous image) analogized from the pseudo label image according to the conversion characteristic from the original label image to the original sample image as a pseudo sample image corresponding to the pseudo label image. Here, the pseudo sample image is used as an input image of the machine learning system during machine learning.

  For example, the analogy image generation unit 23 sets each pixel of the pseudo-label image as a target pixel, and has, in the peripheral region, a pixel value distribution closest to a pixel value distribution of a peripheral region of the target pixel (a window of a predetermined size including the target pixel). A pseudo sample image is generated by searching for a pixel in the original label image, and setting the pixel value of the original sample image at the same position as the position of the found pixel to the pixel at the same position in the pseudo sample image.

  For example, based on an image analogy method (Image analogies), the analogy image generation unit 23 generates an image analogized from a pseudo label image as a pseudo sample image according to a conversion characteristic from an original label image to an original sample image.

  When there are a plurality of original training data sets, the analogy image generation unit 23 selects one original training data set from the plurality of original training data sets, and converts the original label image to the original sample image in the selected original training data set. An image inferred from the pseudo label image is generated as a pseudo sample image according to the conversion characteristics. At this time, the original training data set is selected, for example, at random or according to a predetermined rule.

  When there are a plurality of original training data sets, the analogy image generation unit 23 generates a pseudo label for each of the plurality of original training data sets according to the conversion characteristic from the original label image to the original sample image in the selected original training data set. An image inferred from the image is generated as a pseudo sample image candidate, and one pseudo sample image is selected from the plurality of pseudo sample image candidates, or a predetermined image process is performed on the plurality of pseudo sample image candidates to perform one image processing. One pseudo sample image may be generated. When one pseudo sample image is selected from a plurality of pseudo sample image candidates, the pseudo sample image is selected, for example, randomly or according to a predetermined rule.

  Next, the operation of the above system will be described. FIG. 3 is a flowchart illustrating a training data generation method in the system shown in FIG. FIG. 4 is a diagram for explaining the correspondence between the original training data set and the pseudo training data set in the system shown in FIG.

  First, the original data acquisition unit 21 acquires one or a plurality of original training data sets (pairs of original sample images and original label images corresponding to each other) as described above (step S1).

  Thereafter, the similar image generation unit 22 generates a plurality of (for example, hundreds of thousands to tens of millions) pseudo-label images from the original label images in the acquired original training data set as described above (step S2). . For example, when GAN is used, the similar image generation unit 22 generates a plurality of different random number vectors and obtains the same number of pseudo-label images as the random number vectors from the random number vectors.

  Then, the analogy image generation unit 23 generates a pseudo sample image corresponding to each pseudo label image from the pair of the original sample image and the original label image in the acquired original training data set as described above (step S3). ). In conventional data expansion such as translation, an original sample image is processed to obtain a pseudo sample image, but in the present embodiment, the relationship between both the original sample image and the original label image is Based on the pseudo label image, a pseudo sample image is generated.

  Thereby, a large number of pairs of the pseudo sample image and the pseudo label image (that is, the pseudo training data set) corresponding to each other are generated, and thereafter, for example, stored in the storage device 1 or transmitted by the communication device 2. Then, based on the pseudo training data set (or the original training data set and the pseudo training data set), machine learning of an image generation engine such as a deep neural network based on a large number of training data sets is performed.

  Here, a specific example will be described in which the original sample image is a cell tissue image and the original label image is an image showing a boundary line of cells in the cell tissue image.

  FIG. 5 is a diagram illustrating a specific example of an original sample image and an original label image.

  FIG. 5A shows a cell tissue image (gray scale image) as 12 original sample images, and FIG. 5B shows 12 original label images in FIG. 5 is a binary image in which cell boundaries respectively corresponding to the cell tissue images shown are indicated by edges (black-white boundaries).

  Such an original label image is created manually or the like from an original sample image. Usually, it is difficult to obtain a large number (for example, 100,000) of original sample images, and it also takes time to create an original label image from the original sample images. It is difficult to prepare as many training data sets as possible.

  FIG. 6 is a diagram illustrating a specific example of generating a pseudo label image.

  FIG. 6A shows the 12 original label images shown in FIG. 5B. FIG. 6B shows six pseudo-label images generated from the twelve original label images shown in FIG. 6A. Although six pseudo-label images are shown here, as described above, in the similar image generation processing, the input value of the image generation engine (the above-described random number, that is, the real value) is changed. Since a large number of pseudo-label images can be generated without an upper limit, for example, a large number of pseudo-label images such as 100,000 sheets and 10 million sheets different from each other can be generated. FIG. 6C shows the result of performing noise removal processing on the six pseudo-label images shown in FIG. 6B according to the Non-local Means Denoising algorithm. The noise removal processing may be applied as needed, and another algorithm may be used.

  FIG. 7 is a diagram illustrating a specific example of generating a pseudo sample image. A pseudo sample image as shown in FIG. 7 was obtained from the pseudo label image shown in FIG. 6 (C) based on one of the original training data sets shown in FIG.

  Next, a specific example of an output image of the image generation engine after machine learning using the pseudo training data set generated as described above will be described. FIG. 8 is a diagram illustrating a specific example of an output image of the image generation engine after machine learning.

  FIG. 8A shows a test cell tissue image (that is, one not included in the training data set).

  FIG. 8B shows a label image generated based on the three original training data sets and corresponding to the above-described test cell tissue image. Specifically, an image generation engine machine-learned with a training data set obtained by performing normal data expansion on three original training data sets, and using the cell tissue image shown in FIG. The label image shown in FIG. 8B was generated.

  FIG. 8C shows a label image generated based on the 12 original training data sets and corresponding to the above-described test cell tissue image. More specifically, an image generation engine machine-learned using a training data set obtained by performing normal data extension on 12 original training data sets, and using the cell tissue image shown in FIG. The label image shown in FIG. 8C was generated.

  FIG. 8D shows a label image corresponding to the above-described test cell tissue image generated based on the 12 original training data sets and the 12 pseudo training data sets. More specifically, an image generation engine machine-learned with a training data set obtained by performing normal data extension on these training data sets uses the image generation engine shown in FIG. The label image shown in FIG. 8 (D) was generated.

  From the situation of separation of regions corresponding to individual cells in the label image, it is clear that the label image shown in FIG. 8D is more appropriate than those in FIGS. 8B and 8C. . As described above, an appropriate label image can be obtained by generating the training data by the training data generating method according to the present embodiment rather than the normal data extension.

  As described above, according to the above embodiment, the similar image generation unit 22 generates a similar image of the original label image corresponding to the original sample image as a pseudo label image. The analogy image generation unit 23 generates an image analogized from the pseudo label image according to the conversion characteristic from the original label image to the original sample image as a pseudo sample image corresponding to the pseudo label image. Here, the pseudo sample image is an input image of the machine learning system in the training data, and the pseudo label image is an output image of the machine learning system in the training data.

  Thus, a large number of training data sets of new pseudo sample images and pseudo label images different from the original sample images and original label images of the original training data set can be automatically generated. That is, many appropriate training data sets are automatically generated in a short time.

  Various changes and modifications to the above-described embodiment will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. That is, such changes and modifications are intended to be included in the scope of the claims.

  For example, in the above embodiment, the label image is not limited to an image indicating a boundary, and may indicate a specific cell (such as a cancer cell). In the above embodiment, the original sample image may be a biological image other than the cell tissue image, or may be another captured image.

  Further, in the above embodiment, the above-described image generation engine is not limited to a deep neural network (that is, a hierarchical neural network having two or more hidden layers).

  The present invention is applicable to, for example, automatic generation of training data for a machine learning system.

11 Training Data Generation Program 22 Similar Image Generation Unit 23 Analogue Image Generation Unit

Claims (6)

In a training data generation system that automatically generates training data for a machine learning system that generates an output image from an input image,
A similar image generation unit that generates a similar image of the original label image corresponding to the original sample image as a pseudo label image,
An analogy image generation unit that generates an image inferred from the pseudo label image according to the conversion characteristics from the original label image to the original sample image as a pseudo sample image corresponding to the pseudo label image,
The pseudo sample image is an input image of the machine learning system in the training data, the pseudo label image is an output image of the machine learning system in the training data,
A training data generation system characterized by the following.   The similar image generation unit includes an image generation engine configured to generate an output image corresponding to the input value using one or more random number values as an input value, and machine learning of the image generation engine using a plurality of the original label images. 2. The training data according to claim 1, wherein an output image obtained when another random value is input to the image generation engine is acquired as a similar image to the original label image, and is used as the pseudo label image. Generation system.   The analogy image generation unit sets each pixel of the pseudo label image as a target pixel, and sets a pixel having a pixel value distribution closest to a pixel value distribution of a peripheral region having a predetermined size for the target pixel in the peripheral region to the original label image. Searching within, setting the pixel value of the original sample image at the same position as the position of the found pixel at the pixel at the same position in the pseudo sample image to generate the pseudo sample image. The training data generation system according to claim 1 or 2, wherein the training data is generated. The original sample image is a cell tissue image,
The original label image includes an image showing a boundary line of cells in the cell tissue image,
The training data generation system according to any one of claims 1 to 3, characterized in that: In a training data generation method for automatically generating training data for a machine learning system that generates an output image from an input image,
Generating a similar image of the original label image corresponding to the original sample image as a pseudo-label image;
An analogy image generating step of generating an image inferred from the pseudo label image according to the conversion characteristics from the original label image to the original sample image as a pseudo sample image corresponding to the pseudo label image,
The pseudo sample image is an input image of the machine learning system in the training data, the pseudo label image is an output image of the machine learning system in the training data,
A training data generation method characterized by the following. A training data generation program that causes a computer to automatically generate training data for a machine learning system that generates an output image from an input image,
On the computer,
Generating a similar image of the original label image corresponding to the original sample image as a pseudo-label image;
An analogy image generation step of generating an image inferred from the pseudo label image according to the conversion characteristics from the original label image to the original sample image as a pseudo sample image corresponding to the pseudo label image,
The pseudo sample image is an input image of the machine learning system in the training data, the pseudo label image is an output image of the machine learning system in the training data,
A training data generation program characterized by the following.