JP7423905B2 - Machine learning model training method, data generation device, and trained machine learning model - Google Patents

Description

The present specification relates to a machine learning model and a technique for generating data using the machine learning model.

Generative adversarial networks (GANs) are known as machine learning models that have received attention in recent years. A generative adversarial network uses two networks called a discriminator and a generator. The discriminator is trained for the purpose of distinguishing between real data and false data, and the generator is trained for the purpose of generating false data such that the discriminator makes the above-mentioned false identification.

Non-Patent Document 1 describes a method of converting an image (e.g., a line drawing) belonging to a specific category (called a domain) into an image (e.g., a photo) belonging to another category using a conditional generative adversarial network. The technology has been disclosed. In this technology, the generator includes an encoder that encodes an input image, and a decoder that decodes the encoded data to generate a converted image.

Isola, P. et al “Image-to-Image Translation with Conditional Adversarial Networks.” IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017) Taigman, Y. et al “Unsupervised cross-domain image generation.” arXivpreprint arXiv:1611.02200 (2016).

In the above technology, training of the generator is performed using input data, false data, and a generator using identification results by a discriminator, but there is a need for further improvement in training of the generator.

This specification discloses a technique for effectively training a machine learning model and using the machine learning model to generate data that appropriately reflects the characteristics of input data.

The technology disclosed in this specification has been made to solve at least part of the above-mentioned problems, and can be realized as the following application examples.

[Application example 1] A method for training a machine learning model, the first step of inputting input data to a first machine learning model to output fake data corresponding to the input data, The first machine learning model includes an encoder that performs dimension reduction processing to reduce the number of dimensions on the input data to generate first feature data, and an encoder that reduces the number of dimensions for the first feature data. a decoder that generates the fake data by performing a dimension restoration process to restore, the first machine learning model performing the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters. inputting a plurality of data pairs including the first pair and the second pair into a second machine learning model, and inputting a plurality of data pairs corresponding to the plurality of data pairs. A second step of outputting identification data, wherein the first pair is a pair of data consisting of the input data and actual data corresponding to the input data, and the second pair is: A pair of data consisting of the input data and the fake data corresponding to the input data, and the identification data indicates whether the corresponding data pair is the first pair or the second pair. the second step of generating the identification data by performing a calculation using a plurality of second calculation parameters; and teacher data indicating a target value of the identification data, a third step of adjusting the plurality of second calculation parameters so that the difference between the identification data and the teacher data becomes small; a fourth step of performing the dimension reduction process by the encoder on the fake data to generate second feature data; and the identification data, the teacher data, the first feature data, and the second feature. data, so that the difference between the identification data and the teacher data becomes large and the difference between the first feature data and the second feature data becomes small. a fifth step of adjusting calculation parameters of the first machine learning model and the second machine learning model in parallel by repeating the first to fifth steps a plurality of times. How to train.

According to the above configuration, the first calculation parameter of the first machine learning model is not only adjusted so that the difference between the identification data and the training data becomes large, but also The difference between these feature data is adjusted using the data. As a result, the first calculation parameters can be adjusted so that the characteristics of the input data are appropriately reflected in the first characteristic data. Therefore, the first machine learning model can be effectively trained, and by using the machine learning model trained using the above method, it is possible to generate appropriate fake data that reflects the characteristics of the input data.
[Application example 2]
The method described in Application Example 1, further comprising:
Using the first feature data and the second feature data, without using the identification data and the teacher data, the difference between the first feature data and the second feature data is reduced. a sixth step of adjusting the plurality of first calculation parameters,
After training the first machine learning model by repeating the first step, the fourth step, and the sixth step multiple times, by repeating the first step to the fifth step multiple times, A method of training the first machine learning model and the second machine learning model in parallel.

[Application Example 3 ] A method for training a machine learning model, the first step of inputting input data to a first machine learning model to output fake data corresponding to the input data, The first machine learning model includes an encoder that performs dimension reduction processing to reduce the number of dimensions on the input data to generate first feature data, and an encoder that reduces the number of dimensions for the first feature data. a decoder that generates the fake data by performing a dimension restoration process to restore, the first machine learning model performing the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters. inputting a plurality of data pairs including the first pair and the second pair into a second machine learning model, and inputting a plurality of data pairs corresponding to the plurality of data pairs. A second step of outputting identification data, wherein the first pair is a pair of data consisting of the input data and actual data corresponding to the input data, and the second pair is: A pair of data consisting of the input data and the fake data corresponding to the input data, and the identification data indicates whether the corresponding data pair is the first pair or the second pair. the second step of generating the identification data by performing a calculation using a plurality of second calculation parameters; and teacher data indicating a target value of the identification data, a third step of adjusting the plurality of second calculation parameters so that the difference between the identification data and the teacher data becomes small; a fourth step of performing the dimension reduction process by the encoder on the fake data to generate second feature data; and using the identification data and the teacher data, a fifth step of adjusting the plurality of first calculation parameters so that the difference between the identification data and the teacher data is increased; a sixth step of adjusting the plurality of first calculation parameters so that the difference between the first feature data and the second feature data is small, without using the first feature data, After training the first machine learning model by repeating the first step, the fourth step, and the sixth step multiple times, the first machine learning model is trained by repeating the first to fifth steps multiple times. A method for training a first machine learning model and a second machine learning model in parallel.

According to the above configuration, after the first machine learning model is trained using the first feature data and the second feature data so that the difference between these feature data becomes small, the first machine learning model is trained using the first feature data and the second feature data. and a second machine learning model in parallel. As a result, the first machine learning model can be effectively trained so that the features of the input data are appropriately reflected in the first feature data, so if the machine learning model trained using the above method is used, , it is possible to generate appropriate output data that reflects the characteristics of input data.
[Application example 4]
The method according to either Application Example 2 or 3,
In the sixth step, the first calculation parameter is further adjusted using the real data and the fake data corresponding to the specific input data so that the difference between the real data and the fake data is small. How to adjust.
[Application example 5]
The method according to any one of Application Examples 1 to 4,
In the fifth step, the first calculation parameter is further adjusted using the real data and the fake data corresponding to the specific input data so that the difference between the real data and the fake data is small. How to adjust.

[Application Example 6 ] A data generation device, comprising a generation unit that generates output data corresponding to input data using a trained first machine learning model, and an output unit that outputs the output data. , the first machine learning model includes an encoder that performs dimension reduction processing to reduce the number of dimensions on the input data to generate first feature data; , a decoder that performs a dimension restoration process to restore the number of dimensions and generates the output data, and a model that executes the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters. and the plurality of first calculation parameters of the trained first machine learning model have been adjusted by a training process, and the training process inputs the input data to the first machine learning model. a first process of outputting the output data corresponding to the input data as fake data; and inputting a plurality of data pairs including the first pair and the second pair to a second machine learning model. , a second process of outputting a plurality of identification data corresponding to the plurality of data pairs, wherein the first pair is a pair consisting of the input data and actual data corresponding to the input data. , the second pair is a pair of data consisting of the input data and the fake data corresponding to the input data, and the identification data is such that the corresponding data pair is the first pair of data. and the second pair, and the second machine learning model executes a calculation using a plurality of second calculation parameters to obtain the identification data. Using the second process to generate the identification data and the teacher data indicating the target value of the identification data, the plurality of second a fourth process of performing the dimension reduction process by the encoder on the fake data to generate second feature data; Using the first feature data and the second feature data, the difference between the identification data and the teacher data becomes large, and the difference between the first feature data and the second feature data increases. a fifth process of adjusting the plurality of first calculation parameters such that A data generation device that performs a process of training the second machine learning model in parallel.

According to the above configuration, the first calculation parameter of the first machine learning model is not only adjusted so as to increase the difference between the identification data and the teacher data in the training process, but also adjusted to increase the difference between the identification data and the teacher data. Adjustment is made using the second feature data so that the difference between these feature data becomes small. As a result, the first machine learning model is effectively trained so that the features of the input data are appropriately reflected in the first feature data. Therefore, the data generation device can use the machine learning model to generate appropriate output data that reflects the characteristics of the input data.

[Application Example 7 ] A data generation device, comprising a generation unit that generates output data corresponding to input data using a trained first machine learning model, and an output unit that outputs the output data. , the first machine learning model includes an encoder that performs dimension reduction processing to reduce the number of dimensions on the input data to generate first feature data; , a decoder that performs a dimension restoration process to restore the number of dimensions and generates the output data, and a model that executes the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters. and the plurality of first calculation parameters of the trained first machine learning model have been adjusted by a training process, and the training process inputs the input data to the first machine learning model. a first process of outputting the output data corresponding to the input data as fake data; and inputting a plurality of data pairs including the first pair and the second pair to a second machine learning model. , a second process of outputting a plurality of identification data corresponding to the plurality of data pairs, wherein the first pair is a pair consisting of the input data and actual data corresponding to the input data. , the second pair is a pair of data consisting of the input data and the fake data corresponding to the input data, and the identification data is such that the corresponding data pair is the first pair of data. and the second pair, and the second machine learning model executes a calculation using a plurality of second calculation parameters to obtain the identification data. Using the second process to generate the identification data and the teacher data indicating the target value of the identification data, the plurality of second a fourth process of performing the dimension reduction process by the encoder on the fake data to generate second feature data; a fifth process of adjusting the plurality of first calculation parameters so that the difference between the identification data and the teacher data becomes large using and, without using the identification data and the teacher data, calculate the plurality of first calculation parameters so that the difference between the first feature data and the second feature data becomes small. and a sixth process for adjusting, and after training the first machine learning model by repeating the first process, the fourth process, and the sixth process multiple times, the first process to the sixth process are performed. The data generation device is a process of training the first machine learning model and the second machine learning model in parallel by repeating the fifth process a plurality of times.

According to the above configuration, the first machine learning model is trained using the first feature data and the second feature data so that the difference between these feature data becomes small, and then the second machine learning model The learning model is further trained in parallel. As a result, the first machine learning model is effectively trained so that the features of the input data are appropriately reflected in the first feature data. Therefore, the data generation device can generate appropriate fake data that reflects the characteristics of the input data using the machine learning model.
[Application example 8]
The data generation device according to Application Example 6 or 7,
The input data is image data having a first attribute and a second attribute and not having a third attribute,
The output data is image data having the first attribute and the third attribute and not having the second attribute.

Note that the technology disclosed in this specification can be realized in various forms, such as a training device that executes the above training method, a trained machine learning model, and a device that implements these methods and devices. This can be realized in the form of a computer program for, a recording medium on which the computer program is recorded, and the like.

FIG. 2 is a block diagram showing the configuration of a data generation device 200 of this embodiment. It is a flowchart of image generation processing. FIG. 3 is a diagram showing an example of an input image and an output image. FIG. 2 is a block diagram showing the configuration of a generation network GN. FIG. 2 is a block diagram showing the configuration of an encoder EC. FIG. 2 is a block diagram showing the configuration of a decoder DC. FIG. 1 is a block diagram showing the configuration of a training device 100 according to the present embodiment. FIG. 3 is a diagram showing an example of an input image and a real image. FIG. 1 is a conceptual diagram of a network system 1000 according to the present embodiment. FIG. 2 is a block diagram showing the configuration of an identification network DN. It is a flowchart of training processing. It is a flowchart of pre-processing. It is a flowchart of main processing.

A. Example A-1. Configuration of data generation device Next, an embodiment will be described based on an example. FIG. 1 is a block diagram showing the configuration of a data generation device 200 of this embodiment.

The data generation device 200 is a computer such as a personal computer or a smartphone. The data generation device 200 includes a CPU 210 as a controller of the data generation device 200, a volatile storage device 220 such as a RAM, a nonvolatile storage device 230 such as a hard disk drive or a flash memory, and a display unit 240 such as a liquid crystal display. It includes an operation unit 250 such as a keyboard and a mouse, and a communication interface (IF) 270. Communication interface 270 is an interface for connecting to an external device (for example, printer 300). The communication interface 270 includes, for example, a wired or wireless interface for connecting to the network NW to which the printer 300 is connected.

The volatile storage device 220 provides a buffer area that temporarily stores various intermediate data generated when the CPU 210 performs processing. The nonvolatile storage device 230 stores a computer program PGg and an input data group IG including a plurality of input data IDs, which will be described later. The volatile storage device 220 and the nonvolatile storage device 230 are internal memories of the data generation device 200.

The computer program PGg is provided, for example, in the form of being downloaded from a server operated by the manufacturer of the printer 300. Alternatively, the computer program PGg may be provided in a form stored on a DVD-ROM or the like. The CPU 210 executes image generation processing, which will be described later, by executing the computer program PGg.

The computer program PGg includes as a module a generation network program GNM, which is a computer program that causes the CPU 210 to implement the functions of a generation network (generator) GN, which will be described later.

The printer 300 is an inkjet printing device or an electrophotographic printing device, and prints an image on a printing medium such as paper using ink or toner as a printing material.

A-2. Image Generation Processing FIG. 2 is a flowchart of the image generation processing. This image generation process is started, for example, in the data generation device 200 based on a start instruction from the user.

In S10, the CPU 210 obtains one input data ID. This input data ID is, for example, one piece of data selected from the input data group IG stored in the nonvolatile storage device 230 based on the user's designation. The input data ID is image data indicating the input image II.

FIG. 3 is a diagram showing an example of an input image and an output image. The input image II of this embodiment is an image showing specific characters in a first type of font. Input image II includes (H×W) pixels arranged in a matrix in the vertical and horizontal directions. H is the number of pixels in the vertical direction of the input image II, and W is the number of pixels in the horizontal direction of the input image II. In this example, H=W=256. For example, the input image IIa in FIG. 3A is an image showing the character "A" in the first type font. The input image IIb in FIG. 3B is an image showing the character "B" in the first type font.

In this embodiment, the input data ID is bitmap data representing an image including a plurality of pixels, and specifically, RGB image data representing the color of each pixel using RGB values. The RGB value is a color value of the RGB color system including gradation values of three color components (hereinafter also referred to as component values), that is, an R value, a G value, and a B value. The R value, G value, and B value are, for example, gradation values of a predetermined number of gradations (for example, 256).

In S20, the CPU 210 inputs the input data ID to the generation network GN to generate output data OD corresponding to the input data ID. The output data OD is image data indicating the output image OI, and is RGB image data like the input data ID. The output image OI of this embodiment is an image showing the same characters as the corresponding input image II in a second type font different from the first type font. For example, the output image OIa in FIG. 3A is an output image corresponding to the input image IIa, and is an image showing the character "A" in the second type font. The output image OIb in FIG. 3B is an output image corresponding to the input image IIb, and is an image showing the character "B" in the second type font. In this embodiment, the size of the output image OI is the same as the size of the input image II. The specific configuration of the generation network GN will be described later.

In S30, the CPU 210 outputs the generated output data OD. For example, the CPU 210 uses the output data OD to generate print data indicating the output image OI, and sends it to the printer 300. The printer 300 prints the output image OI on a print medium using print data. Alternatively, the CPU 210 displays the output image OI on the display unit 240 using the output data OD.

A-3. Configuration of Generation Network GN FIG. 4 is a block diagram showing the configuration of generation network GN. The generation network GN includes an encoder EC and a decoder DC.

A-3-1. Configuration of encoder EC The encoder EC performs dimension reduction processing on the input data ID using a plurality of calculation parameters Pe, and calculates the characteristics representing the characteristics of the input data ID (that is, the characteristics of the input image II). Generate a data CD. In this example, the input data ID includes three component values (R value, G value, B value) for each of (256 x 256) pixels, so it has (256 x 256 x 3) values. , that is, (256×256×3) dimensional data. In this embodiment, the feature data CD is data including (1×1×512) values, that is, 512-dimensional data. In this way, in the dimension reduction process, the number of dimensions of the input data ID is reduced.

FIG. 5 is a block diagram showing the configuration of encoder EC. The encoder EC is a neural network having an input layer EL_0 and a plurality of convolutional layers EL_1 to EL_8.

The input layer EL_0 is a layer into which input data ID is input. The input data ID input to the input layer EL_0 is input as is to the first convolutional layer EL_1. The convolutional layer EL_1 performs arithmetic processing, which will be described later, on the (256×256×3)-dimensional input data ID to generate (A ₁ ×B ₁ ×C ₁ )-dimensional data (A ₁ , B ₁ and C ₁ are positive integers).

The kth (k is an integer from 2 to 8) convolutional layer EL_k has (A _k-1 ×B _k- 1) generated by the (k-1)th convolutional layer EL_(k-1). Processed data of dimensions (A _k-1 , B _k-1 , C _k-1 ) obtained by performing predetermined post-processing (described later) on data of dimensions × _C k-1 is input. Ru. The convolutional layer EL_k performs arithmetic processing to be described later on the (A _k-1 ×B _k-1 ×C _k-1 )-dimensional processed data to obtain (A _k ×B _k ×C _k )-dimensional processed data. (A _k , B _k , C _k are positive integers).

The arithmetic processing executed by each of the convolutional layers EL_1 to EL_8 includes a convolution process and a bias addition process. Convolution processing is a process in which s filters of (p x q x r) dimensions are sequentially applied to input data to calculate a correlation value indicating the correlation between the input data and the filters. be. In the process of applying each filter, a plurality of correlation values are sequentially calculated while sliding the filter. One filter includes (p×q×r) weights. The bias addition process is a process of adding one bias prepared for each filter to the calculated correlation value. The (p×q×r×s) weights included in the s filters and the s biases corresponding to the s filters are the plurality of calculation parameters Pe described above, and the training Updated in processing.

Each value of data generated by each convolutional layer EL_1 to EL_8 is a value obtained by adding a bias to the above-mentioned correlation value. The number of data included in the data generated by each convolutional layer EL_1 to EL_8 (for example, (A ₁ ×B ₁ ×C ₁ ) in the case of convolutional layer EL_1) is the stride (sliding the filter) in the convolution process. s) and the number s of filters.

Each value of the data generated by the convolutional layer EL_1 is input to the activation function and converted as the above-mentioned post-processing. In this embodiment, a so-called LeakyReLU (Leaky Rectified Linear Unit) is used as the activation function.

Each value of the data generated by the convolutional layer EL_2 to convolutional layer EL_8 is converted by batch normalization as the above-mentioned post-processing, and then further input to the activation function and converted. Batch normalization is a process of normalizing each value by calculating the average and variance of each value for a set (batch) of input data used in the training process described later. In the image generation process, each value is normalized using the moving average value of the mean and variance calculated for each batch during the training process.

Processed data obtained by performing the above-described post-processing on the data generated by the convolutional layer EL_8 is the above-mentioned feature data CD.

In this example, the number of dimensions (A ₁ ×B ₁ ×C ₁ ) to (A ₈ ×B ₈ ×C ₈ ) of data generated by each convolutional layer EL_1 to EL_8 is as follows. be.
(A ₁ ×B ₁ ×C ₁ )=(128×128×64)
(A ₂ ×B ₂ ×C ₂ )=(64×64×128)
(A ₃ ×B ₃ ×C ₃ )=(32×32×256)
(A ₄ ×B ₄ ×C ₄ )=(16×16×512)
(A ₅ × B ₅ × C ₅ ) = (8 × 8 × 512)
(A ₆ ×B ₆ ×C ₆ )=(4×4×512)
(A ₇ ×B ₇ ×C ₇ )=(2×2×512)
(A ₈ ×B ₈ ×C ₈ )=(1×1×512)

A-3-2. Configuration of Decoder DC The decoder DC performs dimension restoration processing on the feature data CD generated by the encoder EC using a plurality of calculation parameters Pd to generate the above-mentioned output data OD. In this embodiment, the feature data CD is data including (1×1×512) values, that is, 512-dimensional data, as described above. In this embodiment, the output data OD, like the input data ID, is data including (256×256×3) values, that is, (256×256×3) dimensional data. In this embodiment, in the dimension restoration process of this embodiment, the number of dimensions of the feature data CD is restored in this way.

FIG. 6 is a block diagram showing the configuration of decoder DC. The decoder DC is a neural network having a plurality of transposed convolutional layers DL_1 to transposed convolutional layers DL_8.

Feature data CD is input to the first transposed convolution layer DL_1. The transposed convolution layer DL_1 performs arithmetic processing to be described later on the feature data CD to generate (D ₁ ×E ₁ ×F ₁ )-dimensional data (D ₁ , E ₁ , and F ₁ are positive integer).

The m-th (m is an integer from 2 to 8) transposed convolutional layer DL_m includes (D _m-1 , E _m Processed data of dimensions (D _m -1 , E _{m-1 , F m-1 ) obtained by performing predetermined post-processing (described later) on data of dimensions (D m-1} , E _m-1 , F _{m -1} ) is input. be done. Furthermore, the m-th transposed convolutional layer DL_m is obtained by performing the above-mentioned post-processing on the data generated by the (9-m)th convolutional layer EL_(9-m) of the encoder EC. (A _9-m , B _9-m , C _9-m )-dimensional processed data is input. For example, as shown in FIGS. 5 and 6, processed data based on the data generated by EL_7 of the encoder EC is input to the transposed convolution layer DL_2 of the decoder DC, and the transposed convolution layer DL_4 is inputted to the transposed convolution layer DL_2 of the encoder DC. Processed data based on data generated by EC EL_5 is input. Therefore, the m-th (m is an integer from 2 to 8) transposed convolutional layer DL_m has {(D _m-1 , E _m-1 , F _m-1 )+(A _9-m , B _{9- m} , C _9-m )}-dimensional processed data is input. The transposed convolutional layer DL_m performs arithmetic processing, which will be described later, on the input processed data to generate (D _m ×E _m ×F _m )-dimensional data (D _m , E _m , F _m is a positive integer).

The arithmetic processing executed by each transposed convolution layer DL_1 to DL_8 includes transposed convolution processing and bias addition processing. In the transposed convolution process, after increasing the number of dimensions by adding an appropriate value (for example, a value of zero) to the input data according to the stride, the transpose convolution process is performed in the same way as the convolution process described above. This is a process of performing a convolution operation using a q×r)-dimensional filter. The bias addition process is a process of adding one bias prepared for each filter to the correlation value calculated by the transposed convolution operation. The (p×q×r×s) weights included in the s filters and the s biases corresponding to the s filters are the plurality of calculation parameters Pd described above, and the training Updated in processing.

Each value of data generated by each of the transposed convolutional layers DL_1 to DL_8 is a value obtained by adding a bias to the above-mentioned correlation value. The number of data included in the data generated by each of the transposed convolutional layers DL_1 to DL_8 (for example, (D ₁ ×E ₁ ×F ₁ ) in the case of the transposed convolutional layer DL_1) is the stride ( (the amount to add a value such as zero) and the number of filters s.

Each value of the data generated by the transposed convolutional layers DL_1 to DL_3 is transformed by the batch normalization described above as the post-processing described above. In the training process, each value transformed by batch normalization is further transformed by dropout as post-processing, and then input to the activation function and transformed. Dropout is a process of invalidating (setting to 0) some randomly selected values in order to suppress overfitting. A so-called ReLU (Rectified Linear Unit) is used as the activation function. In the image generation process, dropout processing is not performed, and each value converted by batch normalization is input to the activation function and converted.

Each value of the data generated by the transposed convolutional layers DL_4 to DL_7 is transformed by the batch normalization described above as the post-processing described above, and then input to the activation function and transformed. In the post-processing of the transposed convolutional layers DL_4 to DL_7, dropout is not performed in either the training process or the image generation process.

The (D ₈ ×E ₈ ×F ₈ )-dimensional data generated by the transposed convolutional layer DL_8 is the above-mentioned output data OD. Therefore, the number of dimensions (D ₈ ×E ₈ ×F ₈ ) of the data generated by the transposed convolutional layer DL_8 is equal to the number of dimensions (256 × 256 × 3) of the output data OD.

In this example, the number of dimensions (D ₁ ×E ₁ ×F ₁ ) to (D ₈ ×E ₈ ×F ₈ ) of data generated by each transposed convolution layer DL_1 to DL_8 is as follows. It is.
(D ₁ ×E ₁ ×F ₁ )=(2×2×512)
(D ₂ ×E ₂ ×F ₂ )=(4×4×512)
(D ₃ ×E ₃ ×F ₃ )=(8×8×512)
(D ₄ ×E ₄ ×F ₄ )=(16×16×512)
(D ₅ ×E ₅ ×F ₅ )=(32×32×256)
(D ₆ ×E ₆ ×F ₆ )=(64×64×128)
(D ₇ ×E ₇ ×F ₇ )=(128×128×64)
(D ₈ ×E ₈ ×F ₈ )=(256×256×3)

A-4. Training of Generation Network GN The generation network GN described above is trained so that it can generate desired output data OD when input data ID is input. Below, training of the generation network GN will be explained.

A-4-1. Configuration of Training Device FIG. 7 is a block diagram showing the configuration of the training device 100 that executes training of the generation network GN of this embodiment.

The training device 100 is a computer such as a personal computer or a server. The training device 100 includes a CPU 110 as a controller of the training device 100, a volatile storage device 120 such as a RAM, a non-volatile storage device 130 such as a hard disk drive or a flash memory, a display unit 140 such as a liquid crystal display, and a keyboard and the like. It includes an operation unit 150 such as a mouse, and a communication interface (IF) 170 for connecting to an external device (for example, a printer 300).

The volatile storage device 120 provides a buffer area that temporarily stores various intermediate data generated when the CPU 110 performs processing. The nonvolatile storage device 130 includes a computer program PGt, an input data group IG including a plurality of input data IDs for training, a real data group RG including a plurality of real data RD, and teacher data LD. Stored. The volatile storage device 120 and the nonvolatile storage device 130 are internal memories of the training device 100.

The computer program PGt is provided, for example, in the form of being downloaded from a server operated by the manufacturer of the printer 300. Alternatively, the computer program PGt may be provided in a form stored on a DVD-ROM or the like. The CPU 110 executes a training process, which will be described later, by executing the computer program PGt.

The input data ID used in the training process is RGB image data similar to the input data ID used in the image generation process described above. As described above, the input image II indicated by the input data ID is an image showing specific characters in the first typeface (font). The actual data RD is RGB image data similar to the output data OD generated in the image generation process described above. The plurality of real data RD correspond one-to-one to each of the plurality of input data IDs included in the input data group IG. In this embodiment, the real image RI indicated by the real data RD is an image showing the same characters as the corresponding input image II in the second font.

FIG. 8 is a diagram showing an example of an input image and a real image. Input image IIc in FIG. 8A is an image showing the character "C" in the first typeface. The real image RIc corresponding to the input image IIc in FIG. 8(A) is an image showing the character "C" in the second font. The input image IId in FIG. 8B is an image showing the character "d" in the first type font. The real image RId corresponding to the input image IId in FIG. 8(B) is an image showing the character "d" in the second font.

A predetermined number, for example, 1000 pairs of input data ID and actual data RD are prepared by the operator who executes the training.

The teacher data LD is data indicating a target value of identification data DD to be output by the identification network DN, which will be described later. The teacher data LD is prepared by the operator who executes the training. The teacher data LD will be described further later.

A-4-2. Configuration of Network System FIG. 9 is a conceptual diagram of the network system 1000 of this embodiment. The network system 1000 is a system used to train the generation network GN, and includes a discriminator DN in addition to the generation network GN described above. The generative network GN and the identification network DN constitute so-called generative adversarial networks (GANs).

A pair of data (also referred to as a real data pair Pr) consisting of an input data ID and real data corresponding to the input data ID is input to the identification network DN. Furthermore, a pair of data (also referred to as a false data pair Pf) consisting of input data ID and output data OD corresponding to the input data ID is input to the identification network DN. The output data OD corresponding to the input data ID means the output data OD generated by inputting the input data to the generation network GN. In addition, in a generative adversarial network, the output data OD is also called "fake data OD." Here, the fake data OD that is generated by inputting the input data ID corresponding to the specific real data RD into the generation network GN is also referred to as the fake data OD that corresponds to the specific real data RD.

When either the real data pair Pr or the false data pair Pf is input, the identification network DN identifies the authenticity of the input data pair. That is, the identification network DN performs arithmetic processing on the input data pair using a plurality of calculation parameters Pdn, and determines whether the input data pair is a real data pair or a fake data pair. The identification data DD indicating the result of identification is output.

In this embodiment, the identification data DD is (30×30×1)-dimensional data including (30×30×1) values. When the real data pair Pr is input, the identification network DN outputs identification data DD indicating that the input data pair is the real data pair Pr, and when the fake data pair Pf is input, the identification network DN outputs identification data DD indicating that the input data pair is the real data pair Pr. , is trained to output identification data DD indicating that the input data pair is a false data pair Pf.

For this reason, in this embodiment, two types of teacher data LD (FIG. 7) are prepared: teacher data LDr corresponding to the real data pair Pr and teacher data LDf corresponding to the false data pair Pf. The teacher data LDr is data indicating that the input data pair is a real data pair Pr, and is dimensional data (30×30×1) in which all values are “1”. The teacher data LDf is data indicating that the input data pair is a false data pair Pf, and is dimensional data (30×30×1) in which all values are “0”.

If the identification data DD is one-dimensional data, in the process of reducing the number of dimensions to one dimension in the identification network DN, the image represented by the input real data or fake data is changed to show global characteristics. Information may be lost. In this case, there is a possibility that the training of the identification network DN will not progress. Furthermore, when the identification data DD is one-dimensional data, the number of layers of the identification network DN and, by extension, the number of calculation parameters becomes excessively large, and the time required for training becomes excessively long. In this embodiment, by setting the identification data DD to (30 x 30 x 1) dimensional data, it is possible to suppress the loss of information for identifying the authenticity of input data pairs, and to quickly training can be achieved.

FIG. 10 is a block diagram showing the configuration of identification network DN. The identification network DN is a neural network having an input layer L_0 and a plurality of convolutional layers L_1 to L_5.

The input layer L_0 is a layer into which either the false data pair Pf or the real data pair Pr is input. The data pair input to the input layer L_0 is input as is to the first convolutional layer L_1. The number of dimensions of the input data ID making up the data pair is (256x256x3), and the number of dimensions of the real data RD or fake data OD making up the data pair is (256x256x3). Therefore, the number of dimensions of the data pair is (256×256×6). The convolution layer L_1 performs arithmetic processing, which will be described later, on the (256×256×6)-dimensional input data ID to generate (G ₁ ×H ₁ ×I ₁ )-dimensional data (G ₁ , H ₁ and I ₁ are positive integers).

The nth (n is an integer from 2 to 5) convolutional layer L_n has (G _n-1 ×H _n- 1) generated by the (n-1)th convolutional layer L_(n-1). Processed data of dimensions (G _{n -1} , H _n-1 , I _n-1 ) obtained by performing predetermined post-processing (described later) on data of dimensions ×I n-1 is input. Ru. The convolutional layer L_n executes the arithmetic processing described later on the (G _n-1 ×H _n-1 ×I _n-1 )-dimensional processed data to obtain (G _n ×H _n ×I _n )-dimensional processed data. (G _n , H _n , I _n are positive integers).

The arithmetic processing executed by each of the convolutional layers L_1 to L_5 includes convolution processing and bias addition processing, similar to the arithmetic processing of the convolutional layer of the encoder EC. The plurality of weights included in the filter used in the convolution process and the bias corresponding to each filter are the plurality of calculation parameters Pe described above, and are updated in the training process described later. Note that in the convolution process, zero padding is appropriately performed to supplement data in order to adjust the number of dimensions of generated data.

Each value of the data generated by the convolutional layer L_1 is input to the activation function and converted as the above-mentioned post-processing. In this embodiment, a so-called LeakyReLU (Leaky Rectified Linear Unit) is used as the activation function.

Each value of the data generated by the convolutional layers L_2 to L_4 is converted by batch normalization as the above-mentioned post-processing, and then further input to the activation function and converted. Batch normalization is as explained in the description of the decoder DC above.

The (G ₅ ×H ₅ ×I ₅ )-dimensional data generated by the convolutional layer L_5 is the above-mentioned identification data DD. Therefore, the number of dimensions (G ₅ ×H ₅ ×I ₅ ) of the data generated by the convolutional layer L_5 is equal to the number of dimensions (30 × 30 × 1) of the identification data DD.

In this example, the number of dimensions (G ₁ ×H ₁ ×I ₁ ) to (G ₅ ×H ₅ ×I ₅ ) of data generated by each convolutional layer L_1 to L_8 is as follows. be.
(G ₁ ×H ₁ ×I ₁ )=(128×128×64)
(G ₂ × H ₂ × I ₂ ) = (64 × 64 × 128)
(G ₃ × H ₃ × I ₃ ) = (32 × 32 × 256)
(G ₄ × H ₄ × I ₄ ) = (31 × 31 × 512)
(G ₅ × H ₅ × I ₅ ) = (30 × 30 × 1)

In this embodiment, the error value Egan between the identification data DD and the teacher data LD is used to train the identification network DN and the generation network GN. In this embodiment, a sigmoid cross entropy error is used as the error Egan. For example, Egan is calculated using the following equation (1). The error value Egan increases as the identification data DD approaches the corresponding teacher data LD. In other words, the error value Egan becomes larger as the difference between the identification data DD and the corresponding teacher data LD becomes smaller.

Here, p is the number of dimensions of the identification data DD and the teacher data LD (in this embodiment, p=(30×30×1)). a _i is a value obtained by normalizing each value of the identification data DD by inputting it into a sigmoid function, and t _i is each value (0 or 1 as described above) of the teacher data LD corresponding to the value a _i . be.

In this embodiment, in addition to the error value Egan, two types of error values E1 and E2 are used to train the generation network GN. The error value E1 indicates the error between the feature data CD of the input data ID described above and the feature data CDf of the false data OD corresponding to the input data ID. The feature data CDf of the fake data OD is data generated by inputting the fake data OD generated by inputting the input data ID into the generation network GN to the encoder EC of the generation network GN. That is, the feature data CDf is generated by performing the same operation on the fake data OD as the operation performed on the input data ID when generating the feature data CD. A mean squared error (MSE) is used for the error value E1. For example, the error value E1 is calculated using the following equation (2).

Here, q is the number of dimensions of the feature data CD and CDf (in this embodiment, q=(1×1×512)). b _i is each value of the feature data CD, and c _i is each value of the feature data CDf corresponding to the value b _i . The error value E1 becomes smaller as the feature data CDf is closer to the feature data CD. In other words, the error value E1 becomes smaller as the difference between the feature data CDf and the feature data CD becomes smaller.

The error value E2 indicates the error between the real data RD and the false data OD corresponding to the real data RD. For example, a mean absolute error (MAE) is used as the error value E2. For example, the error value E2 is calculated using the following equation (3).

Here, s is the number of dimensions of the real data RD and the fake data OD (in this embodiment, s=(256×256×3)). d _i is each value of the actual data RD, and e _i is each value of the output data OD corresponding to the value d _i . As can be seen from equation (3), the error value E2 is the pixel-by-pixel error between the image represented by the real data RD and the image represented by the false data OD. The error value E2 becomes smaller as the false data OD and the real data RD are closer. In other words, the error value E2 becomes smaller as the difference between the false data OD and the real data RD becomes smaller.

The discrimination network DN is trained such that the error value Egan becomes large. In other words, the identification network DN is trained for the purpose of correctly identifying the authenticity of input data pairs.

The generation network GN is trained so that the error value Egan is small. At the same time, the generation network GN is trained such that the error values E1, E2 are small. In other words, the generation network GN recognizes that the false data pair Pf is incorrectly identified as the real data pair Pr by the identification network DN, that the feature data CDf and the feature data CD approach each other, and that the false data pair Pf is incorrectly identified as the real data pair Pr. The purpose of training is to make the data OD closer to the real data RD.

A-4-3. Training Processing Below, specific training processing will be explained. The training process is performed by adjusting the plurality of calculation parameters Pe and Pd described above in the generation network GN and the plurality of calculation parameters Pdn described above in the identification network DN, so that the generation network GN can output appropriate output data (fake data) OD. This is a process of training to be able to output . The generation network GN incorporated in the computer program PGt stored in the nonvolatile storage device 230 of the data generation device 200 described above is a learned model trained by this training process.

FIG. 11 is a flowchart of the training process. In S100, CPU 110 executes pre-processing. The preprocessing is a process of adjusting a plurality of calculation parameters Pe and Pd of the generation network GN using only the generation network GN without using the identification network DN. In GANs, the task of the generation network GN (generating image data) is heavier than the task of the identification network DN (identifying image data). For this reason, it is preferable to train the generation network GN to some extent in advance.

In S200, CPU 210 executes main processing. The main process is a process of adjusting a plurality of calculation parameters Pdn of the identification network DN and a plurality of calculation parameters Pe and Pd of the generation network GN using the identification network DN and the generation network GN. When the main processing is completed, the training of the generation network GN is completed, and the trained model of the generation network GN is completed.

A-4-3-1. Pre-processing FIG. 12 is a flowchart of pre-processing. In S110, a plurality of calculation parameters Pe and Pd of the generation network GN and a plurality of calculation parameters Pdn of the identification network DN are initialized. For example, the initial values of these calculation parameters Pe, Pd, and Pdn are set to random numbers independently obtained from the same distribution (eg, normal distribution).

In S120, the CPU 210 selects a batch size of real data pairs Pr, for example, V (V is 2 or more) from a plurality of (for example, 1000) real data pairs Pr stored in the nonvolatile storage device 230. Select a real data pair Pr of an integer, eg, V=100. The plurality of real data pairs Pr stored in the nonvolatile storage device 230 are divided in advance into a plurality of groups (batches) each containing V real data pairs Pr. The CPU 210 selects V real data pairs Pr to be used by sequentially selecting one group from the plurality of groups. Alternatively, V real data pairs Pr may be randomly selected each time from a plurality of real data pairs Pr stored in the nonvolatile storage device 230.

In S130, the CPU 210 inputs the V input data IDs included in the V selected real data pairs Pr to the encoder EC of the generation network GN, and obtains the feature data CD of the V input data IDs. generate.

In S140, the CPU 210 inputs the V feature data CD to the decoder DC of the generation network GN to generate V real data RD and V false data OD corresponding to the input data ID.

In S150, the CPU 210 calculates, for each of the V pieces of real data RD, an error value E2 between the real data RD and the fake data OD corresponding to the real data RD.

In S160, the CPU 210 inputs each of the V pieces of fake data OD to the encoder EC of the generation network GN to generate feature data CDf of the V pieces of fake data OD.

In S170, the CPU 210 calculates the error value E1 between the feature data CD and the corresponding feature data CDf for each of the feature data CD of the V input data IDs.

In S180, the CPU 210 uses the V error values E1 and V error values E2 to adjust the plurality of calculation parameters Pe and Pd of the generation network GN. Specifically, the CPU 210 calculates the index value using a loss function Lpre shown in equation (4) below. The CPU 210 adjusts the calculation parameters Pe and Pd according to a predetermined algorithm so that the index value calculated using the loss function Lpre becomes small. As the predetermined algorithm, for example, an algorithm using error backpropagation and gradient descent is used.

Here, L1 is an average value of V error values E1, L2 is an average value of V error values E2, and α is a predetermined coefficient.

In S190, CPU 210 determines whether training has been completed. In this embodiment, if a completion instruction is input from the worker, it is determined that the training has been completed, and if an instruction to continue training is input, it is determined that the training is not completed. For example, the CPU 210 inputs a plurality of test input data IDs different from the input data ID used for training to the generation network GN, and generates a plurality of false data OD. The CPU 210 displays the image indicated by the generated fake data OD on the display unit 240. The worker checks the displayed image to see if a sufficient image has been generated for the pre-processing stage (for example, an image in which some characters can be recognized), and then , an instruction to complete or continue training is input via the operation unit 250. In a modified example, for example, it may be determined that training is completed when the processes of S120 to S180 are repeated a predetermined number of times.

If it is determined that the training has not been completed (S190: NO), the CPU 210 returns the process to S120. If it is determined that the training has been completed (S190: YES), the CPU 210 ends the pre-processing.

A-4-3-2. Main Processing FIG. 13 is a flowchart of the main processing. In S205, similar to S120 in FIG. 12, the CPU 210 extracts real data pairs Pr for the batch size, for example, V real data pairs Pr from the plurality of real data pairs Pr stored in the nonvolatile storage device 230. Select. Note that the batch size of the main process, that is, the number of real data pairs Pr selected in this step, may be different from the batch size of the pre-process.

In S210, similarly to S130 in FIG. 12, the CPU 210 inputs each of the V input data IDs included in the selected V real data pairs Pr to the encoder EC of the generation network GN. Generate feature data CD of input data ID.

In S215, similarly to S140 in FIG. 12, the CPU 210 inputs the V feature data CD to the decoder DC of the generation network GN, and inputs the V feature data CD corresponding to the V real data RD and the input data ID. Generate fake data OD.

In S220, the CPU 210 inputs the V real data pairs Pr to the identification network DN and generates V identification data DD corresponding to the V real data pairs Pr.

In S225, the CPU 210 inputs the V fake data pairs Pf to the identification network DN, and generates V identification data DD corresponding to the V fake data pairs Pf. Each of the V false data pairs Pf is a pair of each of the V input data IDs included in the V real data pairs Pr and the false data OD corresponding to the input data ID. The fake data OD corresponding to the input data ID means the fake data OD generated by inputting the input data ID to the generation network GN in S210 and S215.

In S230, the CPU 210 generates identification data for each of the V identification data DD generated for the real data pair Pr in S220 and the V identification data DD generated for the fake data pair Pf in S225. An error value Egan between DD and teacher data LD is calculated.

In S235, the CPU 210 uses the calculated (2×V) error values Egan to adjust the plurality of calculation parameters Pdn of the identification network DN. Specifically, the CPU 210 calculates the index value using a loss function Lgan shown in equation (5) below. The CPU 210 adjusts the calculation parameter Pdn according to a predetermined algorithm so that the index value calculated using the loss function Lgan becomes large. As the predetermined algorithm, for example, an algorithm using error backpropagation and gradient descent is used.

As shown in equation (5), the loss function Lgan is a function that indicates the average value of (2×V) error values Egan.

In S240, similarly to S150 in FIG. 12, the CPU 210 calculates, for each of the V pieces of real data RD, an error value E2 between the real data RD and the false data OD corresponding to the real data RD. .

In S245, similarly to S160 in FIG. 12, the CPU 210 inputs each of the V pieces of fake data OD to the encoder EC of the generation network GN to generate feature data CDf of the V pieces of fake data OD.

In S250, similarly to S170 in FIG. 12, the CPU 210 calculates the error value E1 between the feature data CD and the corresponding feature data CDf for each of the feature data CD of the V input data IDs.

In S255, the CPU 210 uses (2×V) error values Egan, V error values E1, and V error values E2 to adjust the plurality of calculation parameters Pe and Pd of the generation network GN. . Specifically, the CPU 210 calculates the index value using a loss function Lg shown in equation (6) below. The CPU 210 adjusts the calculation parameters Pe and Pd according to a predetermined algorithm so that the index value calculated using the loss function Lg becomes small. As the predetermined algorithm, for example, an algorithm using error backpropagation and gradient descent is used.

Here, as described above, Lgan is the average value of (2×V) error values Egan, L1 is the average value of V error values E1, and L2 is the average value of V error values E2. It is a value. β and γ are predetermined coefficients.

In S260, CPU 210 determines whether training has been completed. In this embodiment, if a completion instruction is input from the worker, it is determined that the training has been completed, and if an instruction to continue training is input, it is determined that the training is not completed. For example, the CPU 210 inputs a plurality of test input data IDs different from the input data ID used for training to the generation network GN, and generates a plurality of false data OD. The CPU 210 displays the image indicated by the generated fake data OD on the display unit 240. The operator checks the displayed image and confirms whether or not an image sufficient for the final image (for example, an image that accurately represents a specific character in the second font) has been generated, Depending on the confirmation result, an instruction to complete or continue the training is input via the operation unit 250. In a modified example, for example, it may be determined that training is completed when the processes of S205 to S255 are repeated a predetermined number of times.

If it is determined that the training has not been completed (S260: NO), the CPU 210 returns the process to S205. If it is determined that the training has been completed (S260: YES), the CPU 210 ends the training process. At the time when the training process described above is completed, the generation network GN has become a learned model with the calculation parameters Pe and Pd adjusted. Therefore, it can be said that the training process is a process of generating (manufacturing) a trained model.

According to the present embodiment described above, the main process (FIG. 13) of the training process (FIG. 11) is to output the fake data OD corresponding to the input data ID by inputting the input data ID to the generation network GN. 1 step (S210, S215 in FIG. 13), a plurality of data pairs including a real data pair Pr and a fake data pair Pf are input to the identification network DN, and a plurality of identifications corresponding to the plurality of data pairs are The second step of outputting the data DD (S220, S225 in FIG. 13) and the identification data DD and the teacher data LD are used to reduce the difference between the identification data DD and the teacher data LD (in this embodiment, , a third step (S230, S235 in FIG. 13) of adjusting a plurality of calculation parameters Pdn of the identification network DN (so that the error value Egan becomes large).

Furthermore, the main process includes a fourth step (S245 in FIG. 13) of performing dimension reduction processing using the encoder EC on the fake data OD to generate feature data CDf, and identifying data DD, teacher data LD, and feature data. By using CD and the feature data CDf, the difference between the identification data DD and the teacher data LD becomes large (in this example, the error value Egan becomes small), and the difference between the feature data CD and the feature data CDf becomes large. A fifth step (S250, S255 in FIG. 13) of adjusting a plurality of calculation parameters Pe, Pd of the generation network GN so that the error value E1 becomes smaller (in this embodiment, the error value E1 becomes smaller) is provided.

In the main process, the generation network GN and the discrimination network DN are trained in parallel by repeating the first to fifth steps multiple times (S260 in FIG. 13). According to this main processing, the plurality of calculation parameters Pe and Pd of the generation network GN are not only adjusted so that the difference between the identification data DD and the teacher data LD becomes large, but also are adjusted to increase the difference between the identification data DD and the teacher data LD. is used to adjust the difference between these feature data CD and CDf to be small. As a result, the calculation parameters Pe and Pd can be adjusted so that the characteristics of the input data ID are appropriately reflected in the characteristic data CD. Therefore, by using the generation network GN trained using the above training process, it is possible to generate appropriate fake data OD that reflects the characteristics of the input data ID.

For example, the encoder EC appropriately maps the features of the input data ID (for example, the essential features independent of the font of the characters shown in the input image II) into the space of feature data (also called latent space). It is assumed that the characteristics are appropriately reproduced in the fake data OD by the decoder DC. In this case, the difference between the feature data CD and the feature data CDf is considered to be sufficiently small. For this purpose, by training the generation network GN so that the difference between the feature data CD and the feature data CDf becomes small, the accuracy with which the encoder EC maps the features of the input data ID in the latent space can be improved, and the decoder The accuracy with which the DC reproduces its features in the fake data OD can be improved. In other words, for example, the generation network GN is trained so that a specific attribute of the input data ID (for example, the type of character to be expressed) can be maintained while converting another attribute to be converted (for example, the typeface). can do. In addition, we can create fake data OD with converted attributes (for example, font) in response to not only limited input data IDs but also various input data IDs (for example, input data IDs indicating various characters). The generation network GN can be trained to generate (ensure diversity). Furthermore, since it is possible to prevent the training from progressing in a direction in which the encoder EC performs incorrect mapping in the latent space, it is possible to stabilize the training of the generation network GN.

Furthermore, in the main processing of this embodiment, in the fifth step mentioned above, the difference between the real data RD and the fake data OD is further reduced by using the real data RD and the fake data OD corresponding to one input data. (in this embodiment, the plurality of calculation parameters Pe and Pd of the generation network GN are adjusted so that the error value E2 becomes small) (S240 and S255 in FIG. 13). As a result, the generation network GN can be trained to generate the desired false data OD. In other words, by training the generation network GN so that the difference between the real data RD and the false data OD becomes small, the generation network GN simply generates false data OD that would cause the identification network DN to misidentify the truth. It is trained not only to generate, but also to generate false data OD that is close to the real data RD. For example, in this embodiment, the generation network GN can be trained so that the typeface expressed in the real data RD is appropriately reproduced in the fake data OD.

Further, in the pre-processing (FIG. 12) of this embodiment, similarly to the main processing (FIG. 13), the first process outputs the fake data OD corresponding to the input data ID by inputting the input data ID to the generation network GN. (S130, S140 in FIG. 12), and a fourth step (S160 in FIG. 12) of performing dimension reduction processing by the encoder EC on the fake data OD to generate feature data CDf. The preprocessing is further performed using the feature data CD and the feature data CDf, without using the identification data DD and the teacher data LD, so that the difference between the feature data CD and the feature data CDf becomes small (this embodiment A sixth step (S180 in FIG. 12) of adjusting a plurality of calculation parameters Pe and Pd of the generation network GN so that the error value E1 becomes smaller) is provided. In the pre-processing, the first step, the fourth step, and the sixth step are repeated multiple times (S190 in FIG. 12) to train the generation network GN. After the pre-processing, the main processing described above is executed (FIG. 11). In this way, after training the generation network GN so that the difference between the feature data CD and CDf becomes small, the generation network GN and the discrimination network DN are trained in parallel. As a result, in the pre-processing, the accuracy with which the encoder EC maps the features of the input data ID onto the latent space can be guaranteed to some extent. Therefore, the generation network GN can be effectively trained so that the features of the input data ID are appropriately reflected in the first feature data, so if the generation network GN trained using the above training process is used, the input Appropriate fake data OD that reflects the characteristics of data ID can be generated.

Here, as described above, in GANs, the task of the generation network GN (generation of image data) is heavier than the task of the identification network DN (identification of image data). In the main process, the training of the generation network GN tends to lag behind the training of the discrimination network DN. If pre-processing is not performed, there is a possibility that training of the generation network GN will be excessively delayed with respect to training of the identification network DN in the main processing. In this case, when adjusting the plurality of calculation parameters Pe and Pd of the generation network GN, the gradient may disappear and the training of the generation network GN may not progress. According to this embodiment, in the pre-processing, a plurality of calculation parameters Pe and Pd of the generation network GN are adjusted so that the difference between the feature data CD and the feature data CDf becomes small, so that such inconveniences can be avoided. The occurrence can be suppressed. Therefore, according to this embodiment, it is possible to stabilize training and shorten training time.

Furthermore, in the pre-processing of this embodiment, in the sixth step, the real data RD and fake data OD corresponding to one input data are further used to reduce the difference between the real data RD and the fake data OD. A plurality of calculation parameters Pe and Pd of the generation network GN are adjusted (in this embodiment, so that the error value E2 is small) (S150 and S180 in FIG. 12). As a result, a plurality of calculation parameters Pe and Pd of the generation network GN can be adjusted before the main processing so that the desired false data OD is generated. As a result, the generation network GN can be efficiently trained to generate the desired false data OD. Further, it is possible to further suppress the inconvenience that the training progress of the generation network GN lags behind the training progress of the identification network DN.

Furthermore, in this embodiment, the input data ID has a first attribute (for example, an attribute that is the character "A") and a second attribute (for example, an attribute that is a first font), In addition, it is image data that does not have the third attribute (for example, the attribute that it is the second font). The output data (false data) OD is image data that has the first attribute and the third attribute, but does not have the second attribute. Therefore, according to this embodiment, the trained generation network GN converts the second attribute of the input data ID into the third attribute while maintaining the first attribute of the input data ID, thereby generating the output data. (Fake data) OD can be generated.

Furthermore, according to the training device 100 (FIG. 1) of the present embodiment and the trained generation network GN, the generation network GN is trained by the training process described above, so that the characteristics of the input data ID are not reflected. Appropriate fake data OD can be generated. In addition, we can create fake data OD with converted attributes (for example, font) in response to not only limited input data IDs but also various input data IDs (for example, input data IDs indicating various characters). Can be generated.

As can be seen from the above description, the real data pair Pr of this embodiment is an example of the first pair, and the false data pair Pf is an example of the second pair. Further, the feature data CD of the input data ID is an example of first feature data, and the feature data CDf of the fake data OD is an example of second feature data. Furthermore, the calculation parameters Pe and Pd of the generation network GN are examples of first calculation parameters, and the calculation parameters Pdn of the identification network DN are examples of second calculation parameters. The generation network GN is an example of a first machine learning model, and the discrimination network DN is an example of a second machine learning model.

B. Variant:
(1) In the above embodiment, when training the generation network GN, the error value E1 between the feature data CD and the feature data CDf is used in both the pre-processing and the main processing. The present invention is not limited to this, and the error value E1 may be used only in at least one of the pre-processing and the main processing. For example, in the above embodiment, in S255 of the main processing, the calculation parameters Pe and Pd of the generation network GN are adjusted using the error value E1, and in S180 of the pre-processing, the error value is adjusted without using the error value E1. The calculation parameters Pe and Pd of the generation network GN may be adjusted using only E2. Alternatively, in S255 of the main process, the calculation parameters Pe, Pd of the generation network GN are adjusted using only the error value Egan, E2 without using the error value E1, and in S180 of the pre-processing, the error value E1 is used. Then, the calculation parameters Pe and Pd of the generation network GN may be adjusted.

(2) In S255 of the main processing of the above embodiment, the training of the generation network GN, that is, the adjustment of the calculation parameters Pe and Pd, is performed using the error value Egan, the error value E1, and the error value E2. ing. Alternatively, in S255 of the main process, the calculation parameters Pe and Pd may be adjusted using only the error value Egan and the error value E1 without using the error value E2.

(3) In S180 of the pre-processing in the above embodiment, the error value E1 and the error value E2 are used to train the generation network GN, that is, adjust the calculation parameters Pe and Pd. Alternatively, in S180 of the pre-processing, the calculation parameters Pe and Pd may be adjusted using only the error value E1 without using the error value E2.

(4) In the training process of the above embodiment, the training of the generation network GN may be performed by omitting the pre-processing and only by the main process.

(5) In the above embodiment, a sigmoid cross entropy error is used for the error value Egan, a mean square error is used for the error value E1, and a mean absolute error is used for the error value E1. Instead, other types of error values may be used for the error values Egan, E1, and E2. For example, a softmax cross entropy error may be used for the error value Egan. An average absolute error may be used as the error value E1. A mean square error may be used for the error value E2.

(6) In the above embodiment, the generation network GN generally has a first attribute (for example, the attribute that it is the letter "A") and a second attribute (for example, the first image data that has the first attribute and the third attribute and does not have the third attribute (for example, the attribute that it is the second font) and does not have the third attribute (for example, the attribute that it is the second font) , is a model for converting into image data that does not have the second attribute. Such conversion is also commonly referred to as "style transfer" or "image-to-image translation." As another specific example, the generation network GN may generate a line drawing (an image drawn only with lines) showing a predetermined object (a still life such as a building, shoes, or an animal such as a horse or dog). It may also be a model that converts into a color image as shown. Furthermore, the generation network GN may be, for example, a model that converts an aerial photograph showing a predetermined location into a map showing the predetermined location. In these cases as well, the generation network GN has a first attribute (an attribute indicating a predetermined object, an attribute indicating a predetermined location) and a second attribute (an attribute indicating a line drawing, an attribute indicating an aerial photograph). ) and does not have the third attribute (the attribute of being a color image, the attribute of being a map), the image data that has the first attribute and the third attribute and does not have the third attribute (the attribute of being a color image, the attribute of being a map) is It can be said that this is a model for converting image data to image data that does not have attributes.

(7) In the above embodiment, in the generation network GN, both the input data ID and the false data OD are image data. The present invention is not limited to this, and both or one of the input data ID and the fake data OD may be data indicating text or audio. For example, the generation network GN may be a model that converts image data representing a specific object into text or audio representing the specific object. Alternatively, the generation network GN may be a model that converts text expressing a specific object into images or sounds expressing the specific object.

(8) The specific configurations of the generation network GN and identification network DN are not limited to the configurations shown in FIGS. 5 and 6, and may be various other configurations. For example, in the generation network GN and the identification network DN, the number of convolutional layers and transposed convolutional layers may be changed as appropriate. Further, the generation network GN and the identification network DN may include a pooling layer or a fully connected layer instead of all or part of the convolution layer or transposed convolution layer. Furthermore, various arbitrary configurations may be adopted for the post-processing performed on the values output in each layer. For example, the activation function used in post-processing may be any other function, such as PReLU, softmax, or sigmoid. Additionally, processes such as batch normalization and dropout may be added or omitted as post-processing as appropriate. Furthermore, in the generation network GN of the above embodiment, the m-th transposed convolutional layer DL_m of the decoder DC has the value output from the (9-m)th convolutional layer EL_(9-m) of the encoder EC. However, the input may be omitted.

Further, the identification network DN may be a machine learning model different from a neural network, and may be a machine learning model trained using teacher data, such as a support vector machine (SVM).

(9) The hardware configurations of the data generation device 200 in FIG. 1 and the training device 100 in FIG. 7 are examples, and are not limited thereto. For example, the processor of the data generation device 200 and the training device 100 is not limited to a CPU, but may be a GPU (Graphics Processing Unit), an ASIC (application specific integrated circuit), or a combination of these and a CPU. Further, the training device 100 and the data generation device 200 may be a plurality of computers (for example, a so-called cloud server) that can communicate with each other via a network.

(10) In each of the above embodiments, part of the configuration realized by hardware may be replaced with software, or conversely, part or all of the configuration realized by software may be replaced by hardware. You can do it like this. For example, the generation network GN and the identification network DN may be realized by a hardware circuit such as an ASIC (Application Specific Integrated Circuit) instead of a program module.

Although the present invention has been described above based on examples and modifications, the embodiments of the invention described above are for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention may be modified and improved without departing from the spirit and scope of the claims, and the present invention includes equivalents thereof.

100... Training device, 110... CPU, 120... Volatile storage device, 130... Non-volatile storage device, 140... Display section, 150... Operation section, 200... Data generation device, 210... CPU, 220... Volatile storage device, 230...Nonvolatile storage device, 240...Display unit, 250...Operation unit, 270...Communication interface, 300...Printer, 1000...Network system, EL_0, L_0...Input layer, EL_1 to EL_8, L_0 to L_5...Convolution layer, DL_1 to DL_8...Transposed convolution layer, EC...Encoder, DC...Decoder, ID...Input data, OD...Output data (fake data), CD, CDf...Feature data, LD...Teacher data, GN...Generation network, DN... Identification network, Pd, Pe, Pdn... calculation parameters, Pr... real data pair, PGg, PGt... computer program

Claims (4)

A method for training a machine learning model, the method comprising:
A first step of outputting fake data corresponding to the input data by inputting the input data to a first machine learning model, the first machine learning model an encoder that performs dimension reduction processing to reduce the number of dimensions to generate first feature data; and a decoder that performs dimension restoration processing to restore the number of dimensions to the first feature data to generate the fake data. and the first step, wherein the first machine learning model executes the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters;
A second step of inputting a plurality of data pairs including a first pair and a second pair into a second machine learning model and outputting a plurality of identification data corresponding to the plurality of data pairs. The first pair is a pair of data consisting of the input data and actual data corresponding to the input data, and the second pair is the input data and actual data corresponding to the input data. a pair of data consisting of the false data, the identification data indicating a result of identifying whether the corresponding data pair is the first pair or the second pair; The second machine learning model performs an operation using a plurality of second operation parameters to generate the identification data, and the task of the second machine learning model is to the second step, which is lighter than the task;
A third step of adjusting the plurality of second calculation parameters using the identification data and teacher data indicating a target value of the identification data so that the difference between the identification data and the teacher data becomes small. and,
a fourth step of performing the dimension reduction process by the encoder on the fake data to generate second feature data;
Using the identification data, the teaching data, the first feature data, and the second feature data, the difference between the identification data and the teaching data becomes large, and the first feature data and the a fifth step of adjusting the plurality of first calculation parameters so that the difference with the second feature data is small;
Using the first feature data and the second feature data, without using the identification data and the teacher data, the difference between the first feature data and the second feature data is reduced. a sixth step of adjusting the plurality of first calculation parameters,
Equipped with
The first machine learning model is created by repeating the first step, the fourth step, and the sixth step multiple times without performing the second step, the third step, and the fifth step. After training, the first machine learning model and the second machine learning model are trained in parallel by repeating the first to fifth steps multiple times. A method for training a machine learning model, the method comprising:
A first step of outputting fake data corresponding to the input data by inputting the input data to a first machine learning model, the first machine learning model an encoder that performs dimension reduction processing to reduce the number of dimensions to generate first feature data; and a decoder that performs dimension restoration processing to restore the number of dimensions to the first feature data to generate the false data. and the first step, wherein the first machine learning model executes the dimension reduction process and the dimension restoration process using a plurality of first calculation parameters;
A second step of inputting a plurality of data pairs including a first pair and a second pair into a second machine learning model and outputting a plurality of identification data corresponding to the plurality of data pairs. The first pair is a pair of data consisting of the input data and actual data corresponding to the input data, and the second pair is the input data and actual data corresponding to the input data. a pair of data consisting of the false data, the identification data indicating a result of identifying whether the corresponding data pair is the first pair or the second pair; The second machine learning model performs an operation using a plurality of second operation parameters to generate the identification data, and the task of the second machine learning model is to the second step, which is lighter than the task;
A third step of adjusting the plurality of second calculation parameters using the identification data and teacher data indicating a target value of the identification data so that the difference between the identification data and the teacher data becomes small. and,
a fourth step of performing the dimension reduction process by the encoder on the fake data to generate second feature data;
a fifth step of adjusting the plurality of first calculation parameters using the identification data and the teacher data so that the difference between the identification data and the teacher data becomes large;
Using the first feature data and the second feature data, without using the identification data and the teacher data, the difference between the first feature data and the second feature data is reduced. a sixth step of adjusting the plurality of first calculation parameters,
Equipped with
The first machine learning model is created by repeating the first step, the fourth step, and the sixth step multiple times without performing the second step, the third step, and the fifth step. After training, the first machine learning model and the second machine learning model are trained in parallel by repeating the first to fifth steps multiple times. 3. The method according to claim 1 or 2,
In the sixth step, the first calculation parameter is further adjusted using the real data and the fake data corresponding to the specific input data so that the difference between the real data and the fake data is small. How to adjust. The method according to any one of claims 1 to 3,
In the fifth step, the first calculation parameter is further adjusted using the real data and the fake data corresponding to the specific input data so that the difference between the real data and the fake data is small. How to adjust.

Images