RU2735148C1 - Training gan (generative adversarial networks) to create pixel-by-pixel annotation - Google Patents

Abstract

FIELD: physics.

SUBSTANCE: invention relates to a method and a computer-readable medium for combined synthesis of images and pixel-by-pixel annotations for display. Method comprises preliminarily instructing a GAN model on an unmapped target data set to create images from an existing data set based on random vectors, constructing a decoder by displaying outputs of intermediate layers of GAN in a semantic segmentation mask, annotating with human involvement several images created by GAN, masks of semantic segmentation, teaching with teacher decoder on pairs of introduced features and corresponding annotated masks to produce pairs of synthetic images with corresponding pixel by annotations, creating synthetic data set, wherein synthetic data set consists of pairs of images with corresponding segmentation masks, by selecting random vector from normal distribution and supplying random noise to GAN input, which displays it in synthetic image, a separate semantic segmentation network is trained under supervision on the created synthetic data set.

EFFECT: technical result is higher efficiency of deep training algorithms.

6 cl, 2 tbl, 6 dwg

Description

The present invention can be used in machine learning networks, computer vision and generative adversarial networks to create synthetic datasets in order to improve the quality of deep learning methods.

Description of the prior art

In recent years, the quality of neural generative models has improved dramatically. The first GANs [8] could only generate MNIST digits or low resolution images. The most modern models [3, 16] are capable of generating images with such high quality and resolution that people can often hardly distinguish them from real ones.

With the advent of modern neural network-based techniques, high-quality solutions to the problem of semantic image segmentation have also been obtained, which is critical for understanding scenes. A number of works describe that these two approaches can be used together in one teaching scheme to create a realistic image using a segmentation mask [13, 12] or for segmentation with partial involvement of a teacher.

Generative adversarial networks (GANs) usually consist of a generator network and a discriminator network. A generator trained to trick the discriminator generates an image from random noise, and the discriminator is trained to differentiate the real and generated images. The iterative training of both networks makes the generator capable of producing images that are indistinguishable from real images. The main challenges facing the machine learning community include creating high-quality images, creating images with a lot of variety, and stable learning.

The architecture of the first GAN [8] was very simple: the discriminator and generator consisted of only fully connected layers. This GAN was only capable of generating digits from the MNIST dataset or low-resolution images from CIFAR-10, and did not work on more complex datasets. With the advent of DCGAN [21], the quality and variety of image generation continued to improve through the use of convolutional layers and transposed convolutional layers. The authors of the above-mentioned article [21] analyze the latent space, demonstrating interpolation between images. This was one of the first works in which it was proposed to use the discriminator features for training the classifier. In [23], mini-batch discrimination, feature matching, and label smoothing were proposed. Since there is no explicit objective function for GANs, the performance of different models is difficult to compare. The authors of [23] proposed the Inception Score (IS) as an objective metric for assessing the quality of the generated images. They showed that IS correlates well with a person's subjective assessment. One of the disadvantages of IS is that it can misinterpret quality if the GAN only generates one image per class. In [11], a new target metric for GAN was proposed, called the Fr'echet Inception Distance (FID). The FID is intended to be an enhancement to IS because, instead of just evaluating the generated samples, it involves actually comparing the statistics of the generated samples with real samples. Both of these scores (FID and IS) are commonly used to measure the quality of the generated samples. ProGAN [15] has shown impressive results in generating high quality images up to 1024x1024 resolution. The proposed progressive learning strategy substantially stabilized learning for both networks. BigGAN [3] showed the highest level results on conditional generation using ImageNet. In other works, new losses were proposed [14, 18, 9], architectures and methods for implementing conditional information [19, 20].

A number of works have proposed methods for studying and manipulating the internal features of the GAN. For example, in GAN Dissection [2], the authors provided an analytical framework for visualizing and understanding GAN properties at the element, object and scene levels. In [4], the authors introduced the NeuralPhoto Editor as an interface for exploring the trained hidden space of generative models and making certain semantic changes to natural images.

Semantic segmentation solves the problem of classifying each pixel in an image into a specific set of categories. With the development of deep learning models, the quality of semantic segmentation has improved significantly compared to methods based on manually created features.

The Fully Convolution Network (FCN) [17] was the first architecture to support end-to-end learning for the image segmentation problem. The main network (AlexNet, VGG16) without fully connected layers was adapted to receive arbitrary image sizes. The features derived from the underlying image network are then scaled up through bilinear interpolation or through a series of transposed convolutions. The U-Net architecture [22] is an improvement over FCN: in it, part of the encoder extracts features from the image, and part of the decoder gradually upsamples feature maps and generates the final forecast. The main innovation in U-Net are skip connections between corresponding blocks of decoder and encoder parts. This optimizes the gradient flow, improving the aggregation of information at different scales. The Pyramid Scene Parsing Network (PSPNet) [26] introduced the Pyramid Pooling Module (PPM) to explicitly include information of different scales. This module performs pooling operation on feature maps using different kernel sizes in parallel. The PPM outputs are then scaled up and combined to form feature maps containing both global and local contextual information. This idea was explored in DeepLabV3 [5], and later on PPM was replaced by Atrous Spatial Pyramid Pooling (ASPP).

It uses extended (dilatational) convolutions with varying degrees of dilatation. In DeepLabV3, the last layers of convolution in the encoder's main network are replaced with extended convolution to prevent significant loss in image size. DeepLabV3 + [6] achieves up-to-date results on widely used test programs. In it, DeepLabV3 has been enhanced by the addition of a simple yet effective decoder module to obtain clearer segmentation masks. The present invention uses DeepLabV3 + as the base version.

The essence of the invention

Since GANs can generate high quality images based on some random vector, this vector and a set of intermediate layer pins contain high-level information about the rendered image. Therefore, a natural question arises whether it is possible to project the vector and intermediate features into the semantic segmentation mask and generate images together with pixel-by-pixel annotation. A number of experiments were carried out, which showed that this question can be answered in the affirmative.

StyleGAN [16] is a modern unsupervised image generation method with the best IS and FID scores on the FFHQ, Celeba-HQ and LSUN datasets. Applying a number of ideas from style transfer research, the authors proposed a new architecture in which the image synthesis process is controlled by Adaptive Instance Normalization (AdaIN). The generator starts with a trained constant tensor and adjusts the style in each convolutional block based on the hidden code. FIG. 2 shows the generator architecture. The display network consists of 8 fully connected layers, and each SBlock has an upsampling layer, 2 convolutions and 2 AdaINs.

The known algorithm of generative adversarial networks, which is taken as the basis for data generation; using it, it is possible to generate data from a specific distribution, but the known method is not capable of generating markup for this data. The proposed algorithm simultaneously generates data and markup for them. A method for joint synthesis of images and per-pixel annotations is proposed.

Brief Description of Drawings

The above and / or other aspects will become more apparent from the description of exemplary embodiments with reference to the accompanying drawings, in which:

fig. 1 shows an example of an image from StyleGAN-FFHQ and the corresponding generated annotation for hair segmentation;

fig. 2 - schematic network architecture presentation;

fig. 3 - comparison of the proposed method with the basic version on LSUN interiors for a different number of training samples (upper curve - the proposed method without pre-training on ImageNet; lower curve - DeepLabV3 + without pre-training on ImageNet);

fig. 4 - segmentation masks for cars from the LSUN dataset;

fig. 5 - on the left - an arbitrary image; in the center - the result of consistent application of the Image2StyleGAN and StyleGAN models; on the right - a segmentation mask created by the proposed decoder trained on 20 annotated synthetic (without real or unmarked) images;

fig. 6 shows synthetic images from StyleGAN trained on FFHQ datasets and proposed segmentation masks for the anterior right tooth.

Detailed description

A method for joint synthesis of images and per-pixel annotations (segmentation masks) using GAN is proposed. GAN has a good high-level representation of the target data, which can be easily projected into semantic segmentation masks. This method can be used to create a training dataset to train a single semantic segmentation network. Experiments have shown that such a segmentation network is successfully generalized to real data. It also outperforms supervised learning with a small number of teaching samples and works on a wide variety of different scenes and classes. The invention can be implemented in software and hardware for obtaining images on a screen.

An object of the invention is to co-generate data and corresponding synthetic markup using generative adversarial networks, which requires few examples of human markup. It is suggested to use this markup to train the subnet.

The proposed invention makes it possible to achieve the same quality in-depth learning algorithms with fewer samples, annotated human.

This description demonstrates that a separate semantic segmentation network trained on a synthetic dataset generalizes to real images. In addition, the description shows that the proposed method outperforms conventional supervised learning with a small number of annotated images.

Suppose you have a GAN model that is trained on some dataset. The GAN model is trained to generate images with characteristics similar to those from the target dataset. Since GAN training is time-consuming and of no interest for this invention, all experiments used pre-trained models, in particular StyleGAN [17]. GAN takes a random vector as input and outputs an image. The main idea of the proposed method is to add a lightweight decoder to this GAN. A lightweight model means a model that has a much smaller number of parameters than a GAN. This decoder is trained to create a pixel-by-pixel annotation for the GAN-generated image. To train the decoder, the GAN creates several images and the human annotator annotates them manually. Based on the image generated by the GAN, a person draws a pixel mask object of interest. Annotation is referred to as a segmentation mask or pixel map. The decoder is trained on the masks from the previous step with the corresponding intermediate GAN features. As with most research papers on semantic segmentation, cross-entropy between the predicted mask and the reference is minimized. To reduce computational costs, the GAN remains fixed during training. The authors have demonstrated that due to the lightweight nature of the decoder, only a few images are required to train it. The modified mesh is then used to generate a large image dataset along with annotation.

The generated synthetic dataset consists of image pairs and predicted segmentation masks, which can be viewed as a reference markup. Hence, this dataset can be used to train a segmentation network with a supervisor.

In the present invention, StyleGAN is used as the basic version of the image generation method, and DeepLabV3 + is used as the basic version of the image segmentation method.

In particular, there is proposed hardware containing software products that perform a method for jointly synthesizing images and pixel-by-pixel annotations by means of a GAN, including the following steps. Pretraining a GAN model on an unlabeled target dataset. The GAN is assumed to be a random vector-to-image mapping from the target dataset distribution (eg, StyleGAN, DCGAN, BigGAN, etc.). Extending the pretrained GAN model by adding a decoder network. This network maps the features of the inner layers of the GAN model into a semantic segmentation mask corresponding to the image generated by the GAN from the same features. Human annotation of multiple GAN-generated samples with semantic segmentation masks. Train the decoder on pairs of input features and corresponding annotated masks with a teacher. The GAN model remains fixed throughout the learning process. Generate a large synthetic dataset from the GAN and decoder by applying these models to random vectors. Supervised learning of a separate semantic segmentation network on the generated synthetic dataset.

As a result, it is possible to create a synthetic dataset that can be used to train a segmentation network using just a few human annotated semantic segmentation masks. Typically thousands of human annotated masks are required.

The semantic segmentation algorithm taken as a basis was described. The proposed algorithm was compared in accuracy with the main one using a different number of training examples.

In contrast to the prior art, the present invention provides:

- an algorithm for jointly generating data and associated annotation markup ;

- the ability to train semantic segmentation deep learning models with less annotated data;

- the ability to train on data generated by the GAN, and at the same time, this model successfully generalizes on real data.

In general terms, the present invention is as follows.

An example of implementation of the invention is its application to speed up interactive annotation to perform segmentation on a user dataset. This means that fewer annotated semantic segmentation masks are required to train an adversarial segmentation model that works well with random images from the Internet. Typically hundreds or thousands of annotated semantic segmentation masks are required to achieve the same precision.

The main idea of the proposed method is to modify the already trained GAN model by adding a special lightweight semantic segmentation network. The following describes the architecture of this network and the proposed method for joint synthesis of images and per-pixel annotations.

As a preliminary training GAN: GAN model pre-trained on the untagged set of target data (hereinafter will be described experiments for a number of data sets, including FFHQ, LSUN interiors, LSUN automobiles). The first step is to train the GAN model to generate images from the available dataset based on random vectors. GAN training is a time consuming and resource intensive process that is beyond the scope of this document. All experiments assume a pre-trained GAN model, so this work uses StyleGAN, although the ideas of the invention can be applied to any other architecture.

Construction of a decoder (assignment of graphs): FIG. 2 shows the architecture of the proposed decoder. The decoder maps the outputs of the GAN middleware to a semantic segmentation mask. The display network consists of 8 fully connected layers, and each SBlock contains an upsampling layer, 2 convolutions and 2 AdaINs. Each CBlock decoder accepts features from the corresponding SBlock StyleGAN as input. CBlock consists of a dropout layer, a convolution layer, and a batch normalization layer. This block maps features from StyleGAN to the decoder, decreasing their dimension.

The probability of turning off neurons in dropout layers is set to 50%, this value was chosen during the experiment. Each RBlock of the decoder has one residual block with two convolutional layers. Number of feature maps for each convolutional layer the decoder is set to 32 because initial experiments have shown that increasing the number of feature maps does not improve quality.

Annotation of several synthetic images: For further training of the constructed decoder, a small sample of synthetic images is annotated manually; for each picture from a small sample, the user defines the mask of the object of interest by drawing it with a computer mouse. Images are generated by mapping a random vector from a normal distribution via the GAN, and intermediate features are stored for further training.

Decoder training: The decoder is supervised (https://en.wikipedia.org/wiki/Supervised_learning) using the masks from the previous step with the corresponding intermediate GAN features. As with most semantic segmentation research, cross-entropy between the predicted mask and the reference is minimized. During training, the GAN remains fixed to reduce computational costs.

Generating a large synthetic dataset: After training the decoder, an arbitrary number of synthetic images with the corresponding segmentation masks are generated. To do this, a random vector is selected from a normal distribution and fed to the GAN input, which maps it into a synthetic image. The outputs of the required generator blocks are fed to the decoder as described above. Finally, the decoder creates a segmentation mask corresponding to the proposed synthetic image.

Segmentation network training: The generated synthetic dataset consists of image pairs and predicted segmentation masks, which can be considered as a reference. Hence, this dataset can be used to train a segmentation network with a supervisor.

Let's consider the steps of this method in more detail.

Decoder training

As shown in FIG. 1, at the training stage, the decoder is trained to jointly synthesize image 1 and mask 2 of semantic segmentation. As shown in FIG. 2, the decoder receives features from the GAN and provides a segmentation mask.

The decoder is trained with a teacher on pairs of input features and corresponding masks (see Fig. 2). In particular, during the training procedure, the cross-entropy loss is minimized through backpropagation, this is a standard procedure in training neural networks. Such pairs can be collected simply by annotating the generated images and storing the corresponding intermediate features from the GAN. It should be noted that the original GAN remains fixed (frozen). Intermediate features are taken after each Stylegan block before the upsampling layer, as shown in FIG. 2.

It is well known that the training time is proportional to the number of parameters being trained, so the GAN weights remain fixed (frozen) to reduce computational costs. The training takes a few minutes, and the decoder is successfully trained on a small number of training examples. Tutorial examples are randomly selected from generated images.

FIG. 2 shows a schematic diagram of a decoder with the original StyleGAN. StyleGAN takes a random vector from a normal distribution (normal distribution is a kind of continuous probability distribution for a real random variable (https://en.wikipedia.org/wiki/Normal_distribution)) as input and outputs an image. The decoder uses features from StyleGAN as input and issues a mask. Tags are taken after each StyleGAN block.

Training a segmentation network on synthetic data

The decoder is then trained to generate a large dataset from the pairs of generated GAN pictures and the corresponding masks predicted by the decoder. Training is done using DeepLabV3 + on this synthetic dataset. Experiments have shown that such a network is successfully generalized to real data.

Finding the segmentation mask without training the segmentation network

For this purpose, the proposed training pipeline for the segmentation model includes a separate neural network at the last stage. An attempt is made to check whether it is possible to remove the generation of a large synthetic dataset and the training of a separate segmentation network from the proposed pipeline and restrict it to a trained decoder. Both the decoder and the segmentation model output a segmentation mask, but the former takes a set of intermediate GAN functions as input, and the latter takes the picture itself. The last stage of the experiment in this formulation is the construction of a mapping from an arbitrary image into the space of intermediate GAN features. This topic has already been studied in the literature, in particular, in the work Image2StyleGAN [1], an optimization procedure was proposed that maps an arbitrary image into a set of input vectors for the StyleGAN generator. As a result of inputting the obtained vectors into this generator, a set of intermediate features for the corresponding image is obtained, from which a segmentation mask is obtained using the pre-trained proposed decoder. One of the results is shown in FIG. 5. The left image shows an arbitrary photo, the middle one shows its view after successive application of the Image2StyleGAN and StyleGAN generator, and the right one shows the segmentation mask created by the decoder described in the previous experiment.

The ability of the present model to generalize can be used for a very specific part of the face. For this purpose, the inventors conducted an experiment with marking only the right upper anterior tooth. Although more than 10 teeth are visible in the photograph of a human face, the proposed method has shown nearly ideal results using only 5 annotated synthetic images, as shown in FIG. 6 (only 5 synthetic images were annotated for training. It should be noted that the proposed model, as expected, segments only one of several equally textured teeth).

Experiments

LSUN cars. ("LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop". Link https://www.yf.io/p/lsun).

The inventors randomly selected a subset of 100 images from the validation part of LSUN vehicles and annotated them with vehicle masks. Then the dataset was randomly split into training and test parts, 20 samples were used for training and 80 samples for testing. For the basic version, the inventors used these 20 training samples to train DeepLabV3 + [7]. For the proposed method, the inventors also annotated 20 random images generated by StyleGAN and used them to train the decoder. Then 10,000 synthetic samples were generated and DeepLabV3 + trained on them. Both methods were tested on 80 real samples. The evaluation results are shown in Table 1. Examples of results are shown in FIG. 3. In FIG. 3 shows a comparison of the proposed method with the basic version on LSUN interiors for a different number of training samples (the upper curve is the proposed method without pre-training on ImageNet; the lower curve is DeepLabV3 + without pre-training on ImageNet): (a) - the main network without pre-training on ImageNet; accuracy; (b) - main network, pre-trained on ImageNet, accuracy; (c) - main network without pre-training on ImageNet, average IoU; (d) - main network, pretrained on ImageNet, average IoU. Accuracy and IoU are used as score metrics to compare the baseline version with the proposed method. Both methods were evaluated on 80 test images. The results are presented in Table 1. The proposed method gives an increase in accuracy of more than 2% and an increase in IoU by 5% compared to the basic version. Table 1 also shows a comparison of the proposed approach with the basic version in the case of training neural networks from scratch, that is, without using the main network pre-trained on ImageNet. In this experiment, only 20 annotated images were used in training, as opposed to the additional millions of images from ImageNet with multiclass semantic segmentation masks in the pretrained core network experiment. In this case, the increase in accuracy compared to the basic version becomes even greater (specifically, more than 10% in accuracy and 20% in IoU).

FIG. 4 shows a visualization of the results. The masks in the upper row obtained by the proposed method are more accurate than the masks in the lower row obtained by the basic version.

Table 1. Evaluation results on the LSUN vehicle dataset with two classes (background and vehicles)

Method Pretrained
main network Accuracy IoU DeepLabV3 + (Chen et.al.2018)
Proposed method (invention) -
- 0.8588
0.9787 0.6983
0.9408 DeepLabV3 + (Chen et.al.2018)
Proposed method (invention) ImageNet
ImageNet 0.9641
0.9862 0.9049
0.9609

The LSUN dataset [24] with a pretrained StyleGAN model was used for the car segmentation problem. LSUN is a dataset for classifying a million images, broken down into 10 scene categories and 20 object categories. Only images related to the category "cars" were selected. 100 512x384 pixel images with car segmentation masks were manually annotated. Of these images, 20 samples were used to train the DeepLabV3 + basic model, and the remaining 80 were used to evaluate the proposed method and basic version. Then the entire training pipeline was launched, including the generation of 10,000 synthetic images at the stage of generating a dataset and manual annotation of 20 of them. Both methods were evaluated on 80 test images. The results are shown in Table 1. The proposed method provided an increase in accuracy of more than 2% and an increase in IoU by 5% compared to the basic version.

Table 1 also shows a comparison of the proposed approach with the basic version in the case of training neural networks from scratch, that is, without using the main network pre-trained on ImageNet. In this experiment, only 20 annotated images were used in training, as opposed to the additional millions of images from ImageNet with multiclass semantic segmentation masks in the pretrained core network experiment. In this case, the increase in accuracy compared to the basic version has become even greater (specifically, more than 10% in accuracy and 20% in IoU). Additional discussion is provided in the subsection on experimenting with interior scenes.

Also in FIG. 4 shows the obtained segmentation masks for four images from the test subset to confirm the difference in quality between the proposed method and the basic version. FIG. 4 shows the segmentation masks for vehicles from the LSUN dataset. The top row is the proposed method, the bottom row is DeepLabV3 +. The training was carried out on 20 annotated images for both methods.

Assessment protocol.

Two variants of the main network for DeepLabV3 + [7] were tested: pre-trained on ImageNet and without pre-training. Pixel precision and class-averaged intersection (mIoU) were measured. DeepLabV3 +, like other segmentation models, uses a backbone pre-trained on ImageNet. ImageNet is a large-scale dataset containing 1 million human annotated images from 1000 classes. hence such methods imply implicit use of classification annotation in addition to segmentation. The inventors conducted experiments to compare the proposed method with the basic version using two protocols: with the main network, pre-trained on ImageNet, and without pre-training.

The inventors performed the same experiments on the FFHQ dataset. Flickr-Faces-HQ (FFHQ) is a high quality human face imaging dataset originally created as a benchmark for generative adversarial networks (GANs). This dataset consists of 70,000 high quality PNG images with a resolution of 1024 × 1024, varying greatly in age, ethnicity, and image background. It also has a good coverage of accessories such as glasses, sunglasses, hats, etc. Hair segmentation was used as a target to demonstrate the proposed method on this dataset. The inventors used 20 human annotated images for training and 80 human annotated images for testing.

The results are presented in Table 2 (pixel precision and merge intersection, averaged across classes (see evaluation protocol), they are dimensionless). The proposed method outperforms the base DeepLabV3 model by 7% in IoU and 1% in accuracy.

The inventors also experimented with Image2StyleGAN for StyleGAN-FFHQ. FIG. 5 shows an example of nesting and masking. The left image shows an example of a real photo. Then the inventors applied the Image2StyleGAN algorithm to find its spatial representation attachments. The image reconstructed from this attachment is shown in the center. Finally, the inventors applied a trained decoder to the features of this reconstructed image to obtain a hair segmentation mask. This mask is shown in the right image.

Table 2. Assessment results on the FFHQ dataset with two classes (background and hair)

Method Pretrained
main network Accuracy IoU DeepLabV3 + (Chen et.al.2018)
Proposed method (invention) ImageNet
ImageNet 0.8831
0.8967 0.7549
0.8243

LSUN interiors is a subset of the LSUN dataset that contains photographs of interiors. In this experiment, the proposed method was compared with the baseline version for different numbers of training samples to see the dynamics. Since there are no semantic segmentation masks for LSUN interiors, and annotation is a rather tedious procedure, the inventors used the segmentation network from the GluonCV package, pre-trained on the ADE20K, to create the annotation. Of the 150 classrooms in the ADE20K, only 13 were used, which corresponded to the interior scenes. The results are shown in FIG. 3.

Comparison of the proposed method with the basic version at different settings is shown in four graphs. The inventors compared IoU and accuracy metrics for different numbers of training examples. Graphs (a) and (b) compare the accuracy, while graphs (c) and (d) compare mIoU. The main network pretrained on ImageNet was used for experiments (a) and (c). Segmentation network trained from scratch for experiments (b) and (d). Experiments have shown that the proposed method works well with a small number of training samples and in this case outperforms the basic version by a large margin.

Implementation details

To implement the proposed algorithm, the MXNet Gluon platform was used [7]. Because StyleGAN training is time-consuming and of no interest to this invention, all experiments used models pre-trained by the original authors, which were converted to the format accepted by MXNet Gluon. To train the decoder, the Adam optimizer was used with its initial learning rate set to 1 × 10-4. The original DeepLabV3 + was trained using various training parameters. In particular, the SGD optimizer was used with the torque setting at 0.9, the initial learning rate at 0.01, and the regularization factor of the network weights at 1x10-4. In addition, ResNet-50 was taken as the primary network for DeepLabV3 + and was assumed to be pre-trained on ImageNet unless otherwise noted. Pixel fidelity and class-averaged intersection (IoU) were used to evaluate quality in all experiments.

Experiments have shown that the proposed method works well with a small number of training examples and, in this case, outperforms conventional supervised learning by a large margin. However, with an increase in the number of training examples, the difference in accuracy decreases (Fig. 3 (a), (c)). In the case of using the main network, pre-trained on ImageNet, the proposed method starts to work worse at some point (Fig. 3 (b), (d)). This can be explained by the fact that the GAN itself has limited capabilities: the quality of the generated images is not ideal and the GAN is often unable to generate some rare objects. Consequently, such rare objects are absent from the synthetic dataset. In addition, the internal GAN representation from which the semantic masks are projected may differ slightly from the actual high-level representation. For example, the same features are probably used to represent human hair and beard. As a result, the quality of hair segmentation deteriorates.

The inventors have proposed a method for generating images along with semantic segmentation masks using a pretrained GAN. It can be used to train a single segmentation network. Research has shown that such a segmentation network is successfully generalized to real data.

The limitations of the proposed method are due to two factors. The first is the lack of GAN diversity. The second is an imperfect internal representation of GANs.

The above described embodiments are exemplary and should not be construed as limiting. In addition, the description of the exemplary embodiments is intended to be illustrative, and not to limit the scope of the claims, and many alternatives, modifications, and variations will be apparent to those skilled in the art.

LITERATURE

[1] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? arXiv preprint arXiv: 1904.03189, 2019.

[2] David Bau, Jun-Yan Zhu, Hendrik Strobelt, Bolei Zhou, Joshua B Tenenbaum, William T Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. arXiv preprint arXiv: 1811.10597, 2018.

[3] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809.11096, 2018.

[4] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv: 1609.07093, 2016.

[5] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 1706.05587, 2017.

[6] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.

[7] Tianqi Chen, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang. Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv: 1512.01274, 2015.

[8] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, DavidWarde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.

[9] Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pages 5767-5777, 2017.

[10] Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. arXiv preprint arXiv: 1812.01187, 2018.

[11] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pages 6626-6637, 2017.

[12] Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018.

[13] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. arxiv, 2016.

[14] Alexia Jolicoeur-Martineau. The relativistic discriminator: a key element missing from standard gan. arXiv preprint arXiv: 1807.00734, 2018.

[15] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 1710.10196, 2017.

[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv: 1812.04948, 2018.

[17] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.

[18] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv: 1801.04406, 2018.

[19] Takeru Miyato and Masanori Koyama. cgans with projection discriminator. arXiv preprint arXiv: 1802.05637, 2018.

[20] Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2642-2651. JMLR. org, 2017.

[21] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1511.06434, 2015.

[22] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. Unet: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention, pages 234-241. Springer, 2015.

[23] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pages 2234-2242, 2016.

[24] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03365, 2015.

[25] Zhi Zhang, Tong He, Hang Zhang, Zhongyuan Zhang, Junyuan Xie, and Mu Li. Bag of freebies for training object detection neural networks. arXiv preprint arXiv: 1902.04103, 2019.

[26] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.

[27] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. arXiv preprint arXiv: 1608.05442, 2016.

[28] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.

Claims (12)

1. A method of joint synthesis of images and per-pixel annotations for display, which consists in the fact that - pre-train the GAN model on the unlabeled target dataset to create images from the available dataset based on random vectors; - constructing a decoder by mapping the outputs of the intermediate layers of the GAN into a semantic segmentation mask; - annotate with the involvement of a person several images created by the GAN, masks of semantic segmentation; - a decoder is trained with a teacher on pairs of introduced features and corresponding annotated masks to obtain pairs of synthetic images with corresponding per-pixel annotations; - create a synthetic dataset, and the synthetic dataset consists of pairs of images with corresponding segmentation masks, by choosing a random vector from the normal distribution and supplying random noise to the input of the GAN, which displays it in the synthetic image; - a separate semantic segmentation network is trained under supervision on the created synthetic dataset. 2. The method of claim 1, wherein the GAN is a random vector-to-image mapping from a target dataset distribution. 3. The method of claim 1, wherein the GAN model is a StyleGAN model. 4. The method of claim 1, wherein the decoder network maps features of the inner layers of the GAN model into a semantic segmentation mask corresponding to an image generated by the GAN from the same features. 5. The method of claim 1, wherein the GAN model remains fixed during training. 6. Computer-readable media, which stores computer-executable instructions for implementing the method according to PP. 1-5.

Images