JP2019220133A - Image generation device, image generator, image discriminator, image generation program, and image generation method - Google Patents

Abstract

To generate a high-resolution generated image including an object having a complex structure.SOLUTION: An image generator 28 and an image discriminator 30 are trained stepwise through multiple stages corresponding respectively to different resolutions, from the stage corresponding to the low resolution to the stage corresponding to the high resolution. In each stage, the image discriminator 30 is trained to discriminate which of a training image 22 and a generated image 50 is a generated image on the basis of the generated image 50 and of structure information 26 and the training image 22 converted to the resolution of the stage; and the image generator generates the generated image 50 of the resolution of the stage on the basis of potential vectors 24, and is trained such that the generated image 50 exhibits characteristics of the training image 22, on the basis of feedback (a result of the discrimination) from the image discriminator 30 and of the structure information 26 and the training image 22 converted to the resolution of the stage.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image generation device, an image generator, an image classifier, an image generation program, and an image generation method.

  In recent years, a technology for automatically generating an image by using a machine learning technology using a deep neural network or the like has been proposed.

  Non-Patent Document 1 discloses an image generator that generates an image based on a latent vector indicating a feature of an image, an image identifier that identifies whether an input image is a generated image generated by the image generator, A technology called GANs (Generative Adversarial Nets) composed of a set of GANs is disclosed. In GANs, the image classifier learns to better identify whether the input image is a generated image or not, while the image generator fools the image classifier (the image classifier is incorrectly identified). Learning) to generate a generated image close to the real image.

  Non-Patent Document 2 discloses PG (Progressive Growing of) GANs, which is an improved technology of GANs. In PGGANs, learning of an image generator and an image classifier is performed at a plurality of stages corresponding to different resolutions. Specifically, first, the image discriminator generates one of a generated image having a resolution of 4 × 4 [dpi] generated by the image generator and a real image (an image that is not an image generated by the image generator). Learning is performed to identify whether or not the image is an image, and the identification result is fed back to the image generator. The image generator learns to generate a generated image having the feature indicated by the latent vector in consideration of the identification result. When the learning of the stage corresponding to the resolution of 4 × 4 [dpi] is completed, the process proceeds to the stage of higher resolution (8 × 8 [dpi]), and the same processing is repeated stepwise.

  Non-Patent Document 3 discloses a technique for generating an image including an image of a person in the posture indicated by the pose information based on an image including the image of the person and pose information indicating the posture of the person. .

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, "Generative Adversarial Nets" In NIPS 2014. Tero Karras, Timo Aila, Samuli Laine, Jaakko Lehtinen, "Progressive Growing of GANs for Improved Quality, and Stability, and Variation", In ICLR 2018. Liqian Ma, Qianru Sun, Xu Jia, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool, "Pose Guided Person Image Generation", In NIPS 2017.

  Conventionally, it has been difficult to generate a high-resolution generated image including an object having a complicated structure (hereinafter, the structure of an object in an image may be simply referred to as “image structure”). Here, high resolution means a resolution of about 512 × 512 [dpi] or more.

  By using the PGGANs described in Non-Patent Document 2, it is possible to generate a high-resolution generated image having a simple structure (for example, only a face of a human image), but it is difficult to generate an image having a complicated structure. . For example, an image including a human image in which the body is facing backward but only the face is facing forward, an image of an animal having two faces, and the like may be generated (see FIGS. 14 to 17 of Non-Patent Document 2). Etc.).

  On the other hand, if the technology described in Non-Patent Document 3 is used, it is possible to generate a generated image after designating the structure, but the edges of the generated image are blurred (see Non-Patent Document 3). 4 to 6), it was more difficult to generate a high-resolution generated image.

  Therefore, a technique capable of generating a high-resolution generated image including an object having a complicated structure is desired. Further, for example, in an image classifier used in GANs, it is desirable that a high-resolution image having a complicated structure can be suitably identified as a generated image.

  An object of the present invention is to generate a high-resolution generated image including an object having a complicated structure. Alternatively, an object of the present invention is to make it possible to appropriately identify whether or not a high-resolution image including an object having a complicated structure is a generated image.

  The present invention is based on an image generator for learning such that a generated image generated from a latent vector shows the characteristics of a learning image, the learning image, and the generated image generated by the image generator. An image classifier that learns to identify which of the learning image and the generated image is the image generated by the image generator, wherein the image generator has a classification result of the image classifier. The image generator and the image classifier perform the learning stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution in a plurality of stages corresponding to different resolutions. Performing, at each stage, the image generator further determines the stage from the latent vector in consideration of the structural information indicating the structure of the learning image Learning to generate the generated image of the image resolution, the image classifier further considers the structural information, the learning image converted to the resolution determined in the stage and the learning image in the stage An image generating apparatus, wherein learning is performed so as to identify which of the generated images generated by the image generator is an image generated by the image generator.

  Preferably, the structure information is information indicating positions of a plurality of feature points of the object in the learning image.

  Preferably, the structure information is a structure image having coordinate information of a pixel corresponding to the feature point, and in each stage, the image generator and the image classifier are converted to a resolution determined for the stage. And learning in consideration of the structured image.

  Preferably, the latent vector is generated based on a predetermined probability distribution.

  Preferably, the latent vector is generated based on the learning image.

  In addition, the present invention performs learning at a plurality of stages corresponding to different resolutions, and for each stage, generates a generated image of the resolution determined for the stage generated from the latent vector for the learning of the resolution determined for the stage. An image generator for performing a process of learning to show features of an image stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution, and further comprising, in each stage, a structure of the learning image. And learning to generate the generated image in consideration of structural information indicating the following.

  Further, the present invention performs learning at a plurality of stages corresponding to different resolutions based on the learning image and the generated image generated from the latent vector by the image generator, and determines the stage for each stage. From a stage corresponding to a low resolution to a high resolution from a stage corresponding to a process for learning so as to identify which of the learning image and the generated image converted to a given resolution is an image generated by the image generator. An image discriminator that is performed stepwise toward a stage where the learning image and the generated image are further considered in each stage in consideration of structural information indicating the structure of the learning image. An image discriminator characterized in that learning is performed so as to discriminate an image generated by the image generator.

  Further, the present invention provides an image generator for learning a computer so that a generated image generated from a latent vector shows the characteristics of a learning image, the learning image, and the generated image generated by the image generator. Based on, the learning image and the generated image, an image classifier that learns to identify which is the image generated by the image generator, to function as, the image generator, Learning is further performed based on the identification result of the image classifier, and the image generator and the image classifier change from a stage corresponding to a low resolution to a stage corresponding to a high resolution in a plurality of stages corresponding to different resolutions. In each stage, the image generator further performs learning based on structural information indicating the structure of the learning image. Learning to generate the generated image of the resolution determined in the stage, the image classifier further considers the structural information, the learning for the converted to the resolution determined in the stage An image generation program for learning to identify which of an image and the generated image generated by the image generator at the stage is an image generated by the image generator.

  The present invention also provides an image generator for learning so that a generated image generated from a latent vector indicates the characteristics of a learning image, the learning image, and the generated image generated by the image generator. An image classifier that learns which of the learning image and the generated image is an image generated by the image generator, and an image classifier that uses the image generator. Learning is further performed based on the identification result of the image classifier, and the image generator and the image classifier each include a stage corresponding to a low resolution and a stage corresponding to a high resolution among a plurality of stages corresponding to different resolutions. Learning is performed stepwise toward each stage, and at each stage, the image generator further considers structural information indicating the structure of the learning image, Learning to generate the generated image of the resolution determined in the stage, the image classifier further considers the structural information, the learning for the converted to the resolution determined in the stage An image generation method, wherein learning is performed to identify which of an image and the generated image generated by the image generator in the stage is an image generated by the image generator.

  According to the present invention, it is possible to generate a high-resolution generated image after specifying a complicated structure of the image. Further, according to the present invention, it is possible to appropriately identify whether or not a high-resolution image including an object having a complicated structure is a generated image.

FIG. 2 is a schematic configuration diagram of an image generation device according to the present embodiment. It is a figure which shows the example for a learning image and structure information. It is a conceptual diagram showing the structure of the image generator and the image discriminator according to the present embodiment. FIG. 3 is a first diagram illustrating a character generated by the image generation device according to the embodiment. FIG. 5 is a second diagram illustrating a character generated by the image generation device according to the embodiment. It is a flowchart which shows the flow of the learning process concerning this embodiment.

<Configuration of image generation device>
Hereinafter, embodiments of the present invention will be described. FIG. 1 is a schematic configuration diagram of an image generation device 10 according to the present embodiment. The image generation device 10 may be, for example, a personal computer or a server, but may be any device as long as it can perform the functions described below.

  Although the details will be described later, the image generating apparatus 10 includes a learning device (the image generating device 28 and the image discriminating device 30), and learns the learning device to generate a latent vector 24 (described later) indicating the feature of the image. From the generated image. In particular, the image generating apparatus 10 generates a high-resolution generated image having a structure indicated by the structure information 26 based on structure information 26 (described later) indicating the structure of an object in the image.

  The control unit 12 includes, for example, a CPU and the like. The control unit 12 controls each unit of the image generation device 10 according to an image generation program stored in the storage unit 20 described below. In particular, the control unit 12 executes a learning process for learning an image generator 28 and an image classifier 30 described later. Thus, the control unit 12 also functions as a learning unit that executes the learning process.

  The display unit 14 includes a liquid crystal display, for example. Various screens are displayed on the display unit 14. In particular, the display unit 14 displays a generated image generated by the image generation device 10.

  The input unit 16 includes a mouse, a keyboard, a touch panel, and the like. The input unit 16 is used to input a user's instruction to the image generation device 10.

  The communication unit 18 includes, for example, a network adapter and the like. The communication unit 18 is used to communicate with another device via a communication line such as a LAN or the Internet.

  The storage unit 20 includes, for example, a RAM, a ROM, or a hard disk. The storage unit 20 stores an image generation program for operating each unit of the image generation device 10. As shown in FIG. 1, the storage unit 20 stores a learning image 22, a latent vector 24, structural information 26, an image generator 28, and an image classifier 30.

  The learning image 22 is an image used for learning the image generator 28 and the image classifier 30. The learning image 22 may include an object having a complicated structure. The object having a complicated structure is, for example, a human image including a body and limbs such that the posture is understandable, and a character image including a body and limbs such that the posture is understandable. Objects having a complicated structure are not limited to humanoid objects. Also, an object having a complicated structure can be said to be an object whose positional relationship between feature points of the object can change. For example, a person image can take various postures, and the positional relationship between feature points (for example, between the head and the right hand) is not fixed, and changes according to the posture.

  The latent vector 24 is information indicating characteristics of an image. The latent vector 24 is generated based on a predetermined probability distribution used for learning of the image generator 28 and the image classifier 30, the characteristic vector of the learning image 22, and a learned image generation. Some of the features that are used when the device 28 generates a generated image and that indicate the characteristics of the image that the user wants to generate.

  The latent vector 24 may be generated in any manner, but the learning latent vector 24 is generated from a predetermined probability distribution or based on the learning image 22. Specifically, when the probability distribution is generated from a predetermined probability distribution, the probability distribution is generated by sampling a specified number of dimensions. Examples of the predetermined probability distribution include a normal distribution and a uniform distribution. When the learning image 22 is generated from a learning image, the learning image 22 is input to an encoder that performs a process of converting the learning image 22 into a latent vector 24, so that the latent vector 24 indicating the feature of the learning image 22 is obtained. Can be generated.

  The latent vector 24 is a multidimensional vector expressing the features of the image. The latent space defined by the latent vector 24 is a space expressing the features of the learning image. The structure information 26 may be linked to the latent vector so that the latent space can be further conditioned by the features of the image.

  The structure information 26 is information indicating the structure of the image. The structure of the image is, for example, the posture of the person image if the image includes the person image. Alternatively, the structure of the image may include a position where an object exists in the image, a positional relationship between a plurality of objects, and the like. The structure information 26 indicates the structure of the learning image 22 used for learning of the image generator 28 and the image classifier 30 and is used when the learned image generator 28 generates a generated image. And the structure of an image that the user wants to generate.

  The structure information 26 used at the time of learning may be information indicating positions of a plurality of feature points of the object in the learning image 22. In the present embodiment, the structural information 26 used at the time of learning is a structural image having coordinate information of a plurality of pixels corresponding to a plurality of feature points of an object in the learning image 22.

  FIG. 2 shows an example of the learning image 22 and structure information 26 indicating the structure of the learning image 22. When the learning image 22 is a human image including the whole body as shown in FIG. 2A, the structure information 26 corresponds to a plurality of feature points of the human image as shown in FIG. 2B. The position of the pixel is shown. Specifically, the structural information 26 includes the right eye, left eye, right ear, left ear, nose, right shoulder, left shoulder, right elbow, left elbow, right hand, left hand, right knee, left knee, right foot, This is a structural image in which pixels corresponding to feature points such as the left foot are expressed in white (luminance 100%), and other pixels are expressed in black (luminance 0%). The structure information 26 also has information indicating which feature point of the human image corresponds to each pixel corresponding to the feature point (for example, whether the pixel is a right eye or a left eye). As described above, the structure information 26 may be an image in which information indicating a feature point is added to a pixel corresponding to a position of each feature point of an object in the image.

  Note that the structure information 26 is not limited to a structure image as shown in FIG. For example, a sentence describing the structure of the image may be used. In this case, the structure of the image represented by the structure information 26 can be acquired by the control unit 12 analyzing the sentence. The structure information 26 may include not only information on the position of the object in the image but also attribute information indicating the feature of the object. When the object is a person image, the attribute information includes, for example, the presence or absence of glasses, clothes, hair color, race, and the like. Note that the structure information 26 is not limited to the explicit information as described above. For example, it may be a simple numerical vector in which the above information is embedded.

  The image generator 28 generates an image having the content indicated by the latent vector 24 based on the latent vector 24. Hereinafter, the image generated by the image generator 28 is referred to as a “generated image”. In particular, the image generator 28 generates a generated image based on the structure information 26 as well. The image generator 28 is a learning device, and is specifically configured to include a convolutional neural network. The entity of the convolutional neural network is composed of various parameters related to the convolutional neural network (layer structure, neuron structure of each layer, number of filters in each layer, filter size, stride width, zero padding width, weight of each element of each filter, etc.), and This is a processing execution program for performing processing on input data (the latent vector 24 and the structure information 26). Therefore, the fact that the image generator 28 is stored in the storage unit 20 means that the various parameters and the processing execution program are stored in the storage unit 20.

  The image generator 28 determines the characteristics of the learning image 22 based on the learning image 22, the latent vector 24 for learning, the structure information 26 indicating the structure of the learning image 22, and the identification result of the image classifier 30. It is learned to generate a generated image having the structure indicated by the structure information 26. Details of the structure of the image generator 28 and the learning method will be described later.

  The image identifier 30 identifies whether the input image is a real image (an image that is not a generated image) or a generated image. In the present embodiment, both the real image and the generated image are input to the image classifier 30, and the image classifier 30 identifies which of the two input images is the generated image. The image discriminator 30 is a learning device like the image generator 28, and specifically includes a convolutional neural network. Therefore, storing the image classifier 30 in the storage unit 20 means that the various parameters and the processing execution program are stored in the storage unit 20.

  The image classifier 30 performs learning based on the learning image 22 and the generated image so as to identify which of the learning image 22 and the generated image is the generated image. Details of the structure of the image classifier 30 and the learning method will be described later.

  The configuration outline of the image generation device 10 is as described above. Hereinafter, the structures of the image generator 28 and the image classifier 30 and the details of the learning method will be described with reference to FIG. FIG. 3 shows a structure of the image generator 28 and the image classifier 30 (see the structure of the stage 8) and a conceptual diagram of a learning process in each learning stage. In FIG. 3, the upper side of the dashed line drawn substantially at the center in the up-down direction indicates the structure of the image generator 28, and the lower side of the dashed line indicates the structure of the image discriminator 30.

<Structure of image generator>
In FIG. 3, the white box indicates the convolutional layer 40. The convolutional layer 40 is a target of the learning process. N × N characters described in the convolutional layer 40 indicate that the convolutional layer 40 corresponds to a (spatial) resolution of N × N [dpi]. That is, the N × N convolutional layer 40 is learned so as to generate a generated image having a resolution of N × N [dpi].

  As shown in FIG. 3, the image generator 28 is configured to include a plurality of convolutional layers 40 having different resolutions. In the present embodiment, the image generator 28 has eight convolutional layers of 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64, 128 × 128, 256 × 256, and 512 × 512. 40.

  In FIG. 3, the gray boxes indicate the downsampling layer 42. The downsampling layer 42 downsamples (converts) the structure information 26 to reduce its resolution. The downsampling layer 42 may or may not be a learning target. N × N characters described in the downsampling layer 42 indicate that the downsampling layer 42 corresponds to a resolution of N × N [dpi]. That is, the N × N downsampling layer 42 downsamples the structure information 26 to an N × N resolution and outputs it.

  As shown in FIG. 3, the image generator 28 is configured to include a plurality of downsampling layers 42 each having a different resolution. In the present embodiment, the image generator 28 has a structure including eight downsampling layers 42 of 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64, and 128 × 128. I have.

<Structure of image classifier>
Like the image generator 28, the white boxes indicate the convolutional layer 44, and the convolutional layer 44 is the target of the learning process. N × N characters described in the convolution layer 44 indicate that the convolution layer 44 corresponds to a resolution of N × N [dpi]. That is, the N × N convolutional layer 44 is learned so as to identify which of the learning image 22 and the generated image whose resolution is N × N [dpi] is the generated image.

  The image classifier 30 includes the same number of convolution layers 44 as the convolution layers 40 of the image generator 28. In the present embodiment, the image discriminator 30 includes four convolutional layers of 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64, 128 × 128, 256 × 256, and 512 × 512. 44.

  The image classifier 30 also has a plurality of downsampling layers 42. In the present embodiment, the image classifier 30 has a structure including eight downsampling layers 42 of 4 × 4, 8 × 8, 16 × 16, 32 × 32, 64 × 64, and 128 × 128. I have. The downsampling layer 42 of the image classifier 30 downsamples not only the structure information 26 but also the learning image 22.

<Learning method of image generator and image classifier>
The image generator 28 and the image classifier 30 perform learning in a plurality of stages corresponding to different resolutions. Specifically, learning is performed stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution. In particular, in the present embodiment, the image generator 28 performs learning in consideration of not only the learning image 22 and the latent vector 24 but also the structure information 26 indicating the structure of the learning image 22, and the image classifier 30 also performs learning. The feature is that learning is performed in consideration of the structure information 26 indicating the structure of the image 22. The details will be described below.

  First, the image generator 28 and the image classifier 30 start learning from the stage 1 corresponding to the lowest resolution (4 × 4 [dpi] in the present embodiment). In stage 1, the image generator 28 learns the convolution layer 40 having a resolution of 4 × 4 [dpi], and the image discriminator 30 learns the convolution layer 44 having a resolution of 4 × 4 [dpi].

  Specifically, the image discriminator 30 includes the learning image 22a and the structure information 26a downsampled to 4 × 4 [dpi] by the 4 × 4 downsampling layer 42, and the 4 × 4 downsampling layer 42 of the image generator. Based on the 4 × 4 [dpi] generated image 50a generated by the convolutional layer 40, the 4 × 4 convolutional layer 44 is used so that the generated image 50a or the learning image 22a can be identified as a generated image. (That is, learn).

  Here, since the 4 × 4 convolution layer 44 is learned in consideration of the structure information 26a, for example, learning is performed in consideration of whether the generated image 50a has the structure indicated by the structure information 26a. be able to. That is, if the structure of the generated image 50a is not similar to the structure indicated by the structure information 26a, the 4 × 4 convolution layer 44 indicates that the generated image 50a is not the learning image 22a, that is, the generated image 50a. It is learned to determine.

  Further, the image classifier 30 outputs a classification result. For example, if the generated image 50a is identified as a generated image, “1” is output, and if the learning image 22a is identified as a generated image, “0” is output. The identification result of the image identifier 30 is fed back to the image generator 28. Note that the output of the image classifier 30 is not limited to “0” and “1” as described above. Different values corresponding to the generated image 50a and the learning image 22a may be output.

  The 4 × 4 convolution layer 40 of the image generator 28 generates a 4 × 4 [dpi] generated image 50 a based on the latent vector 24. Then, based on the learning image 22a and the structure information 26a downsampled to 4 × 4 [dpi], the generated image 50a, and the identification result fed back from the image classifier 30, the generated image 50a is converted into the learning image 22a. Is updated (that is, learning is performed) so that the parameter of the 4 × 4 convolutional layer 40 approaches (4).

  Here, since the 4 × 4 convolution layer 40 is learned in consideration of the structure information 26a, the 4 × 4 convolution layer 44 generates the generated image 50a while referring to the structure indicated by the structure information 26a. Can be. Thus, it can be said that the generated image 50a closer to the learning image 22a is easily generated.

  In stage 1, the image generator 28 (the 4 × 4 convolution layer 40) and the image discriminator 30 (the 4 × 4 convolution layer 44) repeat the above-described learning processing. In the process, the image generator 28 learns to “fool” the image classifier 30, ie, the image classifier 30 erroneously identifies that the generated image 50 a is not a generated image. 30 learns so as to correctly identify the learning image 22a and the generated image 50a. Accordingly, the image generator 28 can generate the generated image 50a closer to the learning image 22a, and the image classifier 30 can more accurately identify the learning image 22a and the generated image 50a.

  Then, the image generator 28 and the image discriminator 30 are in a Nash equilibrium, that is, even if the parameters of the image generator 28 are updated, the image discriminator 30 is not messed up and the parameters of the image discriminator 30 are updated If the accuracy of discrimination between the learning image 22a and the generated image 50a does not improve even after that, the learning process of the stage 1, that is, the learning process of the 4 × 4 convolution layers 40 and 44, is completed. The process proceeds to stage 2 which is a stage corresponding to a high resolution (8 × 8 [dpi] in the present embodiment).

  In stage 2, the learning process is basically performed in the same manner as in stage 1. In stage 2, the image generator 28 learns the convolution layer 40 having a resolution of 8 × 8 [dpi], and the image discriminator 30 learns the convolution layer 44 having a resolution of 8 × 8 [dpi].

  Specifically, the image classifier 30 includes the learning image 22b and the structure information 26b downsampled to 8 × 8 [dpi] by the 8 × 8 downsampling layer 42, and the 8 × 8 downsampler of the image generator. The 8 × 8 convolution layer 44 is formed based on the 8 × 8 [dpi] generated image 50b generated by the convolution layer 40 so that the generated image 50b or the learning image 22b can be identified as the generated image. Update the parameters of.

  The output of the 8 × 8 convolutional layer 44 is passed to the 4 × 4 convolutional layer 44 and outputs the identification result as the image classifier 30 in consideration of the processing result of the 4 × 4 convolutional layer 44. Also in stage 2, the identification result of the image identifier 30 is fed back to the image generator 28.

  The 8 × 8 convolutional layer 40 of the image generator 28 has an 8 × 8 [dpi] based on the latent vector 24 indicating the feature of the learning image 22 in consideration of the processing result of the 4 × 4 convolutional layer 40. The generated image 50b is generated. Then, based on the learning image 22b and the structural information 26b downsampled to 8 × 8 [dpi], the generated image 50b, and the identification result fed back from the image classifier 30, the generated image 50b is converted into the learning image 22b. Are updated so as to approach the following equation.

  Then, when the image generator 28 and the image classifier 30 are in Nash equilibrium, the process proceeds to the next stage.

  The image generator 28 and the image discriminator 30 are learned by performing the above-described learning process for all stages (up to stage 8 in the present embodiment).

  Based on the latent vector 24 and the structure information 26 indicating the features of the image, the learned image generator 28 has a high resolution (having the features indicated by the latent vector 24 and the structure indicated by the structure information 26). In the present embodiment, a generated image of 512 × 512 [dpi]) can be generated. Further, the learned image classifier 30 can more accurately determine whether the high-resolution image is a generated image or a real image based on the high-resolution image and the structure information 26.

  As described above, the image generator 28 according to the present embodiment is based on PGGANs capable of generating a high-resolution image, and is further learned in consideration of the structural information 26. Thus, it is possible to generate a generated image having a complicated structure. Similarly, since the image classifier 30 according to the present embodiment is based on PGGANs that enable high-resolution image classification and is further learned in consideration of the structural information 26, the image classifier 30 has high resolution and is complicated. An image having a simple structure can be identified.

  In this embodiment, the image generator 28 and the image classifier 30 are learned in parallel using GANs. However, the image generator 28 and the image classifier 30 may be individually learned. Good. Even in such a case, the image generator 28 and the image classifier 30 perform learning at a plurality of stages corresponding to different resolutions, and perform learning while considering the structural information 26.

  When learning the image generator 28 individually, the image generator 28 generates the generated image 50 based on the latent vector 24 and the structural information 26 of the resolution corresponding to the stage at each stage, and Based on the learning image 22 of the corresponding resolution, the generated image 50, and the structure information 26 of the resolution corresponding to the stage, the learning is performed so that the generated image 50 approaches the learning image 22.

  When the image classifier 30 is individually learned, the image classifier 30 performs the learning in each stage based on the learning image 22, the generated image 50, and the structure information 26 having the resolution corresponding to the stage. Learning is performed so as to identify which of the image for use 22 and the generated image 50 is the generated image.

<Image generation by trained image generator and image classifier>
The image generator 28 that has been sufficiently learned by the learning process described above generates a generated image having the features indicated by the latent vector 24 and the structure indicated by the structural information 26 based on the latent vector 24 and the structural information 26. Can be generated.

  The learned image generator 28 can be used in various ways. One of them can be used to automatically generate a character such as an animation or a game. In particular, high-resolution characters having various postures can be easily and automatically generated. In particular, in recent years, the animation industry has been suffering from a shortage of human resources, and the efficiency of animation production has been urgently required. Therefore, it is desired to be able to efficiently generate high-resolution anime characters having various postures.

  FIG. 4 shows automatically generated high-resolution characters 50 'in various postures. First, a latent vector 24 indicating a characteristic of a character to be automatically generated is generated from the reference image of the character by using the above-described encoder. In FIG. 4, the latent vector 24 is shown as a reference image of an actual character for convenience. Further, structure information 26 indicating the posture of the character is prepared. The structure information 26 may be created by the user himself, or may be automatically generated from various images using a technique such as Openpose.

  Then, by inputting the latent vector 24 and the structure information 26 to the learned image generator 28, a character 50 'having a feature indicated by the latent vector 24 and taking a posture indicated by the structure information 26 is automatically generated. Can be. By inputting a combination of the same latent vector 24 and various structural information 26 to the learned image generator 28, it is possible to easily automatically generate images of the same character in various postures as shown in FIG. Can be.

  Further, it is also possible to automatically generate a character (latent vector 24) by sampling the latent vector 24 from within a latent space expressing the features of the image constructed from the learning image 22. For example, in the latent space, the coordinates (a coordinates) corresponding to the latent vector 24a indicating the characteristics of the leftmost character shown in FIG. 5 correspond to the latent vectors 24b indicating the characteristics of the rightmost character shown in FIG. A line segment is defined between the coordinates (b coordinates). Then, the latent vectors 24c, 24d, 24e, and 24f corresponding to the coordinates (for example, the c coordinate, the d coordinate, the e coordinate, and the f coordinate) on the line segment are, as shown in FIG. This indicates a character having characteristics between the characters indicated by the latent vector 24b. In the example of FIG. 5, the clothes of the character indicated by the latent vector 24a are dark and the clothes of the character indicated by the latent vector 24b are light, so that the latent vectors 24c, 24d, 24e, and 24f are changed. As you go, the color of your clothes is gradually fading. Further, when the structure information 26 includes the attribute information and the latent space is constructed in consideration of the structure information 26, it is also possible to automatically generate an image having the specified feature by specifying the attribute information. It is. For example, when attribute information such as blue for men, black hair, and eyes is given, an image corresponding to the attribute information is generated.

  In the present embodiment, the image generator 28 and the image discriminator 30 are used in combination, but the image generator 28 can be used alone.

  The learned image classifier 30 can also be used alone. For example, by inputting the input image and the structure information 26 to the learned image classifier 30, the image classifier 30 determines whether the input image is a generated image generated by the image generator 28 or an actual image. Can be identified. At the same time, it is possible to identify whether or not the structure of the input image is the structure indicated by the structure information 26. Thus, the image classifier 30 can identify the input image as a generated image but not the structure indicated by the structure information 26.

<Flow of learning process>
Hereinafter, the flow of the learning process of the image generator 28 and the image classifier 30 will be described with reference to the flowchart shown in FIG.

  In step S10, the control unit 12 sets the variable n to 1. The variable n is a variable indicating the learning stage (see FIG. 3) of the image generator 28 and the image classifier 30. When the variable n is 1, it means that the learning process of stage 1 is performed.

  In step S12, the control unit 12 inputs the learning image 22 and the structure information 26 (structure image) to the downsampling layer 42, and converts the image into a resolution corresponding to the stage n.

  In step S14, based on the learning image 22 and the structure information 26 whose resolution has been converted in step S12, and the generated image having the resolution corresponding to stage n, the image classifier 30 determines the convolutional layer 44 corresponding to stage n. (That is, learn).

  In step S16, the control unit 12 inputs to the image generator 28 the learning image 22, the latent vector 24 whose resolution has been converted in step S12, and the structure information 26 whose resolution has been converted in step S12.

  In step S18, the image generator 28 generates a generated image 50 having a resolution corresponding to the stage n based on the latent vector and the structure information 26.

  In step S20, the image classifier 30 feeds back the classification result to the image generator 28.

  In step S22, the image generator 28 identifies the learning image 22 and the structure information 26 whose resolution has been converted in step S12, the generated image 50 generated in step S18, and the image identifier 30 fed back in step S20. Based on the result, the parameters of the convolutional layer 40 corresponding to the stage n are updated (that is, learned).

  In step S24, the control unit 12 determines whether or not the image generator 28 and the image classifier 30 are in Nash equilibrium. If the Nash equilibrium has not been reached, the process returns to step S14, and the learning process in the stage is repeated. If Nash equilibrium has been reached, the process proceeds to step S26.

  In step S26, the control unit 12 determines whether the learning process has been completed in all predetermined stages. If not, the process proceeds to step S28, the variable n is incremented by 1, and the process returns to step S12. That is, the learning process in the next stage is executed. When the learning process in all the stages is completed, the learning process ends.

  The embodiment according to the present invention has been described above, but the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention.

  Reference Signs List 10 image generation device, 12 control unit, 14 display unit, 16 input unit, 18 communication unit, 20 storage unit, 22 learning image, 24 latent vector, 26 structure information, 28 image generator, 30 image classifier, 40, 44 convolutional layer, 42 downsampling layer, 50 generated image.

Claims (9)

An image generator that learns such that a generated image generated from the latent vector shows the characteristics of the learning image,
Learning so as to identify which of the learning image and the generated image is an image generated by the image generator, based on the learning image and the generated image generated by the image generator. An image classifier that performs
With
The image generator further learns based on the identification result of the image classifier,
The image generator and the image discriminator perform learning stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution in a plurality of stages corresponding to different resolutions,
At each stage,
The image generator further learns to generate the generated image having a resolution determined in the stage from the latent vector in consideration of structure information indicating a structure of the learning image,
The image classifier further considers the structure information, which of the learning image converted to the resolution determined in the stage and the generated image generated by the image generator in the stage. Learn to identify whether the image is generated by the image generator,
An image generating apparatus, characterized in that: The structure information is information indicating positions of a plurality of feature points of an object in the learning image.
The image generating apparatus according to claim 1, wherein: The structure information is a structure image having coordinate information of a pixel corresponding to the feature point,
At each stage,
The image generator and the image classifier learn in consideration of the structural image converted to a resolution determined for the stage,
The image generating apparatus according to claim 2, wherein: The latent vector is generated based on a predetermined probability distribution,
The image generation device according to claim 1, wherein: The latent vector is generated based on the learning image,
The image generation device according to claim 1, wherein: Learning is performed at a plurality of stages corresponding to different resolutions, and for each stage, a generated image of the resolution determined at the stage generated from the latent vector indicates the characteristic of the learning image of the resolution determined at the stage. An image generator that performs a learning process stepwise from a stage corresponding to low resolution to a stage corresponding to high resolution,
At each stage, learning is further performed to generate the generated image in consideration of structure information indicating the structure of the learning image.
An image generator characterized in that: Based on the learning image and the generated image generated by the image generator from the latent vector, learning is performed at a plurality of stages corresponding to different resolutions, and each stage is converted to a resolution determined for the stage. A learning process to identify which of the learning image and the generated image is the image generated by the image generator, from a stage corresponding to a low resolution to a stage corresponding to a high resolution. Image discriminator to be performed,
At each stage, learning is performed so as to identify which of the learning image and the generated image is an image generated by the image generator, in consideration of structure information indicating a structure of the learning image. Do
An image classifier characterized in that: Computer
An image generator that learns such that a generated image generated from the latent vector shows the characteristics of the learning image,
Learning so as to identify which of the learning image and the generated image is an image generated by the image generator, based on the learning image and the generated image generated by the image generator. An image classifier that performs
Function
The image generator further learns based on the identification result of the image classifier,
The image generator and the image discriminator perform learning stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution in a plurality of stages corresponding to different resolutions,
At each stage,
The image generator further learns to generate the generated image having a resolution determined in the stage from the latent vector in consideration of structure information indicating a structure of the learning image,
The image classifier further considers the structure information, which of the learning image converted to the resolution determined in the stage and the generated image generated by the image generator in the stage. Learn to identify whether the image is generated by the image generator,
An image generation program, characterized in that: An image generator that learns such that a generated image generated from the latent vector shows the characteristics of the learning image,
Learning so as to identify which of the learning image and the generated image is an image generated by the image generator, based on the learning image and the generated image generated by the image generator. An image classifier that performs
An image generation method using
The image generator further learns based on the identification result of the image classifier,
The image generator and the image discriminator perform learning stepwise from a stage corresponding to a low resolution to a stage corresponding to a high resolution in a plurality of stages corresponding to different resolutions,
At each stage,
The image generator further learns to generate the generated image having a resolution determined in the stage from the latent vector in consideration of structure information indicating a structure of the learning image,
The image classifier further considers the structure information, which of the learning image converted to the resolution determined in the stage and the generated image generated by the image generator in the stage. Learn to identify whether the image is generated by the image generator,
An image generation method characterized by the above-mentioned.